We investigated the common difficulties that students have with concepts related to rotational and rolling motion covered in the introductory physics courses. We compared the performance of calculus- and algebra-based introductory physics students with physics juniors who had learned rotational and rolling motion concepts in an intermediate level mechanics course. Interviews were conducted with six physics juniors and ten introductory students using demonstration-based tasks. We also administered free-response and multiple-choice questions to a large number of students enrolled in introductory physics courses, and interviewed six additional introductory students on the test questions (during the test design phase). All students showed similar difficulties regardless of their background, and higher mathematical sophistication did not seem to help acquire a deeper understanding. We found that some difficulties were due to related difficulties with linear motion, while others were tied specifically to the more intricate nature of rotational and rolling motion.