Framework for Students’ Epistemological Development in Physics Experiments

In order to better understand the impact of lab courses and experiential learning on students’ views of professional physics and physics careers, we are developing tools that assess students’ epistemology specifically related to physics experiments. We developed an open-ended survey about doing physics experiments and administered the survey to students from introductory-level to graduate level students. We then constructed a framework of student epistemology of experimental physics, which includes topics such as link between theory and experiment, autonomy in experimentation, and justifications for the validity of experiments. We also identified key aspects of epistemological development from introductory physics students to graduate students.


I. INTRODUCTION
In the past few decades, much research has focused on students' attitudes and views about knowledge and learning, which may shape and are shaped by their learning experiences [1][2][3].A number of validated survey instruments have been developed to measure students' beliefs and attitudes towards physics and learning physics [4][5][6].Those surveys either focus on several aspects of the nature of science (e.g., nature of theories or scientific method) [3,5] or on specific aspects of learning science/physics (e.g., problem solving or conceptual understanding) [2,5,7].However, the experimental nature of physics has largely been ignored in these surveys.More recently, the Colorado Learning Attitudes about Science Survey for Experimental Physics (E-CLASS) was developed to assess students' perceptions of the nature of physics experiments in laboratory courses and professional research labs [8].
One shortcoming of much research on epistemology and nature of science has been lack of clear novice-expert developmental progressions [9].We are interested in understanding and measuring students' epistemological growth across the undergraduate and more advanced curriculum.Epistemology generally refers to the theory of knowledge, knowing and learning [10].Especially relevant to education, personal epistemology in physics is a set of views about the nature of knowledge, knowing and learning in physics [1].Halloun and Hestenes provide a helpful division of epistemology into two major dimensions: the scientific dimension and cognitive dimension [11].The scientific dimension consists of structure, methodology, and validity of knowledge in the discipline.The cognitive dimension consists of self-efficacy, reflective thinking, and personal relevance.Although there are still arguments about the distinction between epistemology and expectation, and whether metacognition should be a part of personal epistemology [12], we focus our attention on the scientific dimension, which is the narrowest definition, and most closely aligned with the way epistemology is defined in the philosophy of science [13].
In order to characterize students' epistemology of experimental physics, we developed, administered, and analyzed results from an open-ended survey that was given to physics students at all levels ranging from college freshmen to graduate students.The results of this research were used to construct a more detailed framework of student epistemology in experimental physics and to identify key areas of epistemological development.

A. Development and administration of survey
We created an 11 questions open-ended survey that was administered through Qualtrics online survey platform.The survey questions were aligned with the scientific dimension of epistemology, which includes the structure, methodology, and validity in the context of physics experiments.The structure component refers to the link between theory and experiment, abstract and real world, and lecture, lab and everyday life.The methodology component includes design of physics experiments, data collection, and data analysis.The validity component includes the validity and justification of experimental results.The two questions that were analyzed in this paper are the first two in the survey: (1) Why are experiments a common part of physics classes?(2) Why do scientists do experiments for their research?The survey was given to three different populations outlined in Table I.

B. Analysis of survey
All 73 responses were imported into NVivo qualitative data analysis software.The coding process took four steps: (1) Open coding: We highlighted key words and grouped similar words and phrases, and then made initial codes.(2) Focused coding: We identified emergent themes in the initial codes and created a hierarchical structure of codes.Steps 1 and 2 were done independently by two coders (the two authors).Table II shows how the emergent themes fit under the epistemological categories of structure, methodology, and validity.Questions 1 and 2 were not designed to directly probe validity, so the emergent themes have little overlap with that aspect of epistemology.

III. RESULTS
Tables III and IV show the percentage of students from each population (introductory-level, upper-level, and graduate) expressing each of the codes.The percentage agreement of two independent coders is above 80% on all of the codes.Given the small number of responses from upper-level physics majors, we focus our discussion on the comparison between introductory-level and graduate students.The p-values were calculated using the chi-square test comparing the introductory and graduate students' responses for each code.

A. Students' views of experiments as a common part of physics classes
Question 1 asked, "Why are experiments a common part of physics classes?"Table III shows significant differences in the language used by different levels of students in response to Question 1.
Introductory level students had a much higher percentage of statements in the categories of "link theory to real world", "better understanding", and "visualization/demonstration."This suggests that introductory students often perceive laboratory courses as an opportunity to link the abstract concepts to the physical world, or a venue for solidifying concepts and improving understanding.However, they made very few statements about the experimental nature of physics and developing scientific abilities in physics laboratory courses.One possible explanation is that the laboratory component of the algebra-based physics course was designed to follow the lecture so that students would have the opportunity to practice on the concepts they just learned.This "concepts first" approach, while encouraging students to make links between the abstract and real world, also obscures the empirical nature of physics and gaining scientific abilities.Overall, these intro-level students think experiments are supplemental experiences that aid in understanding the concepts or theories taught in lecture.
In contrast, graduate students frequently stated that experiments were important in their own right and they mentioned several additional roles of experiments in the curriculum: (1) Physics is experimental: Graduate students said physics is an experimental science and experimenting is how science is carried out.As one student mentioned, "Experiments are the primary way that physicists begin the scientific process."(2) Scientific abilities: Graduate students said taking laboratory courses offers an opportunity for developing scientific abilities.
(3) Theory testing: Through doing experiments, theories can be formed and validated.One student said, "Any new physical theory needs to be inspired or verified by experimental observation."(4) Science appreciation: Students should know how science works and how experimentation fits into the overall scientific process.Beyond these four codes, graduate students also expressed codes related to improving understanding, however they were less frequent.
In terms of the epistemology framework developed based on our data (see Table II), introductory students tend to describe the structure component of epistemology and neglect the methodology component.Also, graduate students and introductory students differ in their views of structure.While graduate students tend to emphasize that experimentation is the foundational aspect of scientific knowledge, introductory students see the classroom experiment as supplemental to understanding ideas discussed in lecture.Graduate students also emphasize the methodology component of epistemology as a key purpose of laboratories.Question 2 asked, "Why scientists do experiments for their research?"Table IV shows the percentage of responses in each population expressing certain codes in response to Question 2. Table IV shows much more similarity among the three populations.Introductory students and graduate students are not all that different in how they perceive the goals of professional experimental research.This finding is consistent with previous CLASS results that showed students were good at identifying the expert response, but their personal beliefs were much more novice [2].All three levels of students viewed doing physics experiments in a research lab as primarily to test or verify a theory.There are also a small portion of students in introductory and graduate levels mentioned the role of "theory testing" physics laboratory courses, how-ever it is not as significant as it is in research labs.In addition, introductory students also describe doing experiments in research labs as providing evidence for a theory (i.e., empirical evidence).This code is similar to "theory testing" code but students often use terms such as "support", "back up" instead of "prove" or "test".About half of graduate students also mention the role of "scientific investigation" in doing experimental research.

IV. CONCLUSIONS
There are several important findings from our data.We developed a framework of epistemology that explicitly integrates experimentation.We also identified significant differences between introductory-level students and graduate students on their views about the pedagogical value of experiments, which could form the foundation of a developmental progression of epistemology of physics experiments.Ongoing work includes analyzing the remaining nine questions, identifying significant introductory to graduate level differences on those items, administering the survey to broader populations of students to see how repeatable the results are, and developing easier to score versions of the survey as long as it continues to provide helpful insights into students' epistemological development.
(3) Inter-coder edited by Churukian, Jones, and Ding; Peer-reviewed, doi:10.1119/perc.2015.pr.031Published by the American Association of Physics Teachers under a Creative Commons Attribution 3.0 license.Further distribution must maintain attribution to the article's authors, title, proceedings citation, and DOI.2015 PERC Proceedings,

TABLE I .
Information of survey respondents.Scientific abilities: Through doing experiments, students acquire the knowledge of scientific practices, and develop scientific abilities.Scientific abilities consists of the knowledge of how to do science, which includes many specific skills (e.g., measurement skills, and data analysis skills).A typical student phrase was "they can learn what things are important in experimental design.
(4)parison: We compared our individual coding categories and developed a consensus coding scheme.(4)Inter-raterreliability:Thecodingschemewas given to a third person who coded the data independently, which was compared with the coding of the first author.A description of the major codes are shown below.Codes 1 through 10 were emergent from Question 1 and codes 11 through 14 were emergent from Question 2. The code "theory testing" were identified from student responses in questions 1 and 2.1.Link theory to real world: Doing experiments provides a way to link virtual and abstract physics ideas to real and concrete world.A typical student phrase was "experiments allow the theory to be applied to physical situations."2.Better understanding: Experiments can improve understanding, solidify concepts, or help explain an idea.A typical student phrase was "it is easier to understand certain things."3.Visualization/demonstration: " 10.Career preparation Experiments provide career preparation and make students aware of careers in physics.A typical student phrase was "experimental physics is a huge part of both academia and industry jobs, so exposure to labs is a crucial process for those who will go on to pursue those jobs."11.Theory development/modification: New

TABLE II .
A description of epistemology framework based on current data analysis.

TABLE IV .
Percentage of responses within each population towards Question 2 "Why do scientists do experiments for their research?"