Development of validated physics surveys on various topics is important for investigating the extent to which students master those concepts after traditional instruction and for assessing innovative curricula and pedagogies that can improve student understanding significantly. Here, we discuss the development and validation of a conceptual multiple-choice survey related to magnetism suitable for introductory physics courses. The survey was developed taking into account common students' difficulties with magnetism concepts covered in introductory physics courses found in our investigation and the incorrect choices to the multiple-choice questions were designed based upon those common student difficulties. After the development and validation of the survey, it was administered to introductory physics students in various classes in paper–pencil format before and after traditional lecture-based instruction in relevant concepts. We compared the performance of students on the survey in the algebra-based and calculus-based introductory physics courses before and after traditional lecture-based instruction in relevant magnetism concepts. We discuss the common difficulties of introductory physics students with magnetism concepts we found via the survey. We also administered the survey to upper-level undergraduates majoring in physics and PhD students to benchmark the survey and compared their performance with those of traditionally taught introductory physics students for whom the survey is intended. A comparison with the base line data on the validated magnetism survey from traditionally taught introductory physics courses and upper-level undergraduate and PhD students discussed in this paper can help instructors assess the effectiveness of curricula and pedagogies which is especially designed to help students integrate conceptual and quantitative understanding and develop a good grasp of the concepts.