Developing expertlike epistemologies about physics empirical discovery using virtual reality

The process and role of developing new scientiﬁc models experimentally is a cornerstone of physics – and arguably an important learning objective of laboratory courses – but research in physics education suggests that the content-focused, “cookbook” paradigm of lab instruction common in the last 2-3 decades does not effectively meet this objective. These instructional practices often leave students believing that in-class experiments exist to supplement conceptual learning and that the sole purpose of real-world experiments is to test theories. As a result, many students frame lab work as an exercise in knowledge conﬁrmation, and often develop the expectation that all physics that appears in the labs can be found with a Google search. Framing a real-world lab such that its results are Google-proof enough to engage students in the process of scientiﬁc discovery requires great creativity. This paper describes an alternative approach, using virtual reality to integrate the process of authentic scientiﬁc discovery into the lab curriculum. We developed labs in which students explore and develop mathematical models for unknown force laws between new kinds of particles that we invented. We ﬁnd that this intervention produced signiﬁcant shifts in students’ epistemologies about experimental physics toward being more expertlike – speciﬁcally regarding the central role experimentation plays for research physicists in making new discoveries and developing theories, and the role of laboratory experiments in teaching a nuanced and unique scientiﬁc way of knowing.


I. INTRODUCTION
Understanding how to shift students' epistemologies to reflect the importance of experimentation in the discovery of new physics is an open problem in physics education research on labs. Creation of new scientific models is an essential element of the scientific process [1], but students rarely get experience doing so in the "cookbook" style of introductory physics labs common in the last 2-3 decades [2,3], many of which remain in use today. Improving the match between instruction and authentic science practices requires overcoming barriers: • Great creativity is required to design lab activities in such a way that students don't see their findings as something that has been discovered before and can be Googled. • Students overwhelmingly believe that the role of experiments in physics courses is to confirm something they already know [3,4]. • While there are exceptions, physics lab curricula in large-enrollment courses generally do not reflect the iterative nature of scientific inquiry, leaving students with a poor understanding of the complex interplay between experiment and theory involved in the creation of new scientific models [3,5]. In this paper, we document effects of a novel intervention on students' epistemologies about experimental physics. This intervention provides the opportunity to generate models of new force laws through experimentation using a virtual reality (VR) environment [6]. We code free-response survey data to answer the following research questions: (RQ1) If students engage in experimentation as a process for theory development, how will it effect their epistemologies about experimental physics? (RQ2) How do students compare their experience in labs that mimic experimental research to other kinds of laboratory experiences? II. BACKGROUND

A. Student Epistemologies
To better understand student epistemologies about physics experimentation, we replicate a study reported on by Hu and Zwickl [7], hereafter referred to as the "original study", or "original." These researchers developed an instrument to explore beliefs about experimentation, theory, and their interplay held by physics lab students. They administered the instrument to physics students with a range of experience: 28 introductory calculus-based students in a combined studio/lab course, 20 upper-division undergraduates in standalone modern physics lab courses, and 31 PhD students. This instrument is composed of four open-ended questions, accompanied by a codebook used to identify certain themes in question responses in a reproducible way. ples or any evidence to support your answer. Introductory students in the original study exhibit an almost singular focus on the idea that experiments exist to supplement conceptual learning. This contrasts with upperdivision undergraduates and PhD students, who acknowledge the role of in-class experimentation in the development of scientific abilities, and as a means to better understand the scientific process. Likewise, introductory students describe the relationship between experimentation and theory almost universally as experimentation serving to test or prove theories. More experienced students described the same relationship as an iterative, holistic process in which experiments can give rise to new theories, theories can explain experimental findings, and theories can guide the development of experiments.
We describe our use of Hu and Zwickl's instrument to assess the impact of a novel curriculum on students' perception of experimentation in physics in Section III C.

B. Hypothesis-Generating Curriculum
Our framework for designing curriculum to teach model generation is the Investigative Science Learning Environment (ISLE) framework [8]. In ISLE, model generation happens in hypothesis-generating experiments, where students engage in open-minded exploration with the goal of developing a model for an unknown phenomenon. This phase of the ISLE cycle, itself a simplified representation of the scientific process, is followed by testing the models created and further revising, refining, and applying them.
Hypothesis-generating experiments involving novel scenarios are difficult to create, especially in introductory courses. In many cases, experimental physics questions that could reasonably be investigated at the introductory level are well-known with easily-Googled answers. In presence of known answers, students tend to hold those in highest regard, seeking to confirm known answers above anything and everything else, even in the face of contradictory data [9]. Thus, any access to a "right answer" derails efforts to engage students in authentic model generation.

C. NOMR Labs
Our approach to this problem uses VR labs to simulate physical laws that do not exist in nature or the Internet. We include the constraint that they be consistent with known physics that actually does exist so that students can rely on their physics knowledge when reasoning in the VR space. All of the "fictitious" physical laws in this intervention can be construed as hypothetical mathematical variations on Coulomb's Law.
We call these NOMR labs [6], short for Novel Observations in Mixed Reality. In NOMR labs, students explore fictitious physical phenomena in an immersive 3D environment. The virtual apparatus is designed such that it does not give perfect answers; experimental uncertainty is still very much present even in the simulation. The "right answers" programmed into the simulation are never shared with students, such that the only "right" answer is the one that students can make the best case for.

A. Virtual Learning Environment
We refer to the VR application running on the headsets as the virtual learning environment (VLE). The VLE offers students a common set of tools in every lab, including measuring tapes, force meters, the ability to anchor individual particles or pause physics entirely, and a source of the particles they are experimenting upon. The NOMR VLE design is described in detail in a prior publication [6]; the VLE used in this study includes visual and quality-of-life upgrades over the previous version but no notable changes to its controls or capabilities.

B. Instructional Context
Our data come from the Winter 2021 administration of the University of Washington's honors introductory electromagnetism lab; 37 students, mostly freshmen, worked in persistent groups of 3-4, meeting weekly in 2-hour remote sessions. Five of nine weeks of lab activities were NOMR labs; the other four weeks went to a 1-week pen-and-paper uncertainty lab, a 2-week PhET-powered circuits lab, and a week of final presentations. NOMR labs were run remotely by students with Oculus Quest headsets (either their own or or borrowed from the Physics Department). Each group had one such headset operator, who streamed the display to the rest of their group over Zoom.
The first VR lab, called Charge and Mint, lasted two weeks. Groups designed and conducted an experiment to test whether virtual analogues of electric charges follow some re-scaled manifestation of Coulomb's law. Then, they took qualitative and quantitative data to create an empirical model for the interaction between fictitious minty particles, which behave according to an unknown force law.
The second VR lab, called the Manifold Lab, lasted three weeks. Students explored several sets of new types of particles that were discovered in the Manifold, each set obeying a unique fictitious physical force law. Each group was assigned at random one of four sets of particles. In the first week, they performed a hypothesis-generating experiment to create a testable model of the unique force law. Each group submitted a report describing their experiment and model to a class-wide repository.
During class in the second week, each group selected a report describing a model of a set of particles they had not yet seen. They wrote and submitted a proposal before the third week of lab, describing an experiment to test the other group's model. These experiments were carried out in the third week, and their results presented in a symposium in the fourth week.
The Manifold Lab was presented to students with a gamelike narrative in which they function as research scientists. They explore different "pocket" universes with novel forms of matter obeying fundamental force laws unknown to our universe. Before performing the second experiment, students wrote a "grant proposal" that summarized another group's findings, proposed an experiment to test their model, and requested additional or improved equipment within the VR lab. Each group had a few points to spend on equipment, e.g. more precise measurement tools, a larger workspace, tools that "snap" to more convenient configurations, or the ability to automatically pause physics after a set amount of time.
This design seeks to emulate the experience of working within a scientific collaboration: The class as a whole collaborates by sharing data and designing experiments to and test and revise each other's models. The privilege to conduct the second experiment with better equipment depends on applying (non-competitively) for grant funds.

C. Research Methodology
We administered a survey following the course's final exam for a small amount of participation credit. This survey included the four questions from the original study [7] as well as a fifth question, presented in another section of the survey, asking students about their experience of the Manifold Lab: Q5 The framing around Lab 4 (the "Manifold Lab") was designed to put you and your group in a setting that emulates the experience of working within a scientific collaboration. Did you feel that this improved or detracted from your experience? Why? What, if anything, would you change about it?
Responses to Q1-Q4 were coded independently by the first author and by a colleague using the published codebook. After discussion and some context-specific clarification of our codebook we reached 99% inter-rater agreement across all codes. We present here on a subset of codes identified in responses to Q1, Q2, and Q4. In the interest of constraining the scope of our discussion to student views of experimentation, we do not present on the responses to Q3 in this paper.
Responses to Q5 are used to gauge students' perceptions of the Manifold Lab's similarity to their previous lab experiences and to real-world science.  1. These charts show the percentage of responses from each population of students that received each code. "H&Z" in the legend indicates data from Hu and Zwickl [7]. Cool colors represent introductory calculus students (the "novice" population) and warm colors upper-division undergraduates and PhD students (the "expert" population).

IV. ATTITUDES QUESTIONNAIRE
Results from our students are compared to reported data from the original study's introductory calculus-based students (the "novice" population) in Figure 1. We also show results from the original study's upper-division undergraduate and PhD student populations, which we consider to be the "expert" populations. We note that our honors intro physics students likely represent a more advanced population in terms of their scientific epistemology, perhaps falling in a space between the original's introductory students and their upperdivision undergraduates.
We make quantitative comparisons of qualitative data, which is inherently less precise than quantitative data. We therefore refrain from making claims about their statistical significance until these effects can explored in greater depth [10]. We find the patterns to be interesting on their own.
Responses to Q1, Why are experiments a common part of physics classes?, were assigned up to two codes: Supplemental learning: Experiments provide supplemental learning experiences for concepts and theories. Scientific abilities: Experiments help cultivate students' scientific abilities, such as experimental design, data collection, and data analysis skills. We find that the students in our study think about in-class experimentation with less of a singular focus on supplemental learning, and a stronger belief that labs are meant to develop their scientific abilities. Even compared to experts, they are more likely to talk about in-class experiments developing scientific abilities.
Responses to Q2, Why do scientists do experiments in professional research?, were assigned up to three codes: Theory testing: The purpose of doing physics experiments is to prove a theory or test a hypothesis. Discovery: Experiments help investigate unknowns. Theory development: Experiments inspire the development or improvement of theories.
Compared to intro students in the original study, we found that the students in our study have less of a singular focus on theory testing than any other population. Further, they mention discovery and theory development at a rate comparable to experts in the original study, well above that of intro calculus students.
Responses to Q4, How do theory and experiment relate?, were assigned up to three codes: Experiment tests theory: Experiments are conducted to test, support, or back up a theory. Experiment inspires theory: Experiments can inspire the development or modification of theories. Theory guides experiment: Theories often provide the basis for the design of experiments.
We again find that students in our study do not universally believe that experiments exist just to test theories. We see more frequent description of experiment-inspired theory development and in theory as a basis for experimental designs. Comparing to the original study's experts, students in our study discussed the latter two ideas at a rate solidly between the two expert categories, upper-division undergraduates and PhD students. With the assumption that PhD students' responses are informed by experience working in research labs while the upper-division undergraduates' are generally not, these data suggest that the NOMR labs had a similar (but weaker) effect upon the epistemologies of students in our study as real-world research experience.

V. MANIFOLD LAB FEEDBACK
Student responses to Q5 were positive nearly across the board. Complaints spoke to the organizational and administrative hiccups commonly associated with pilot testing a new set of activities, rather than to the content and design of the Manifold Lab. The most negative response described the experience as unremarkable: "It was fine. It felt somewhat like a normal lab." Many responses remarked on the Manifold Lab's similarity to the real-world practice of experimentation: ". . . having our group test hypotheses, collect data by doing tests, and creating our own model for novel physical phenomena was a great way to teach the process of "doing science," especially by emphasizing the importance of connecting the data to math. I also enjoyed the process of needing to test further with better equipment (and needing to save up for said equipment), which is also a realistic situation that happens in the real world." ". . . It really felt like what I imagine it feels like to work in the academic community, and was unlike any lab I had done previously. In particular, the concept of working with completely new phenomena without any previous model was a really fantastic experience. . . " Students described significant changes to their thinking about lab activities, brought about by the absence of a known answer: " . . . we had to do a lot of troubleshooting and ended up with wild results, which I think was beneficial, because it required us to reflect on our process and the uncertainty of our experiment more." " Being able to work with a pocket universe and using experimentation to describe it was the best experience in physics courses I've ever experienced. I preferred this VR experience to any physical lab for the sole reason of it being entirely new and having to get every ounce of info about it through experimentation and collaboration." " . . . we were able to explore physics in the pockets instead of focusing on getting a grade for an assignment; in addition, we could design our own experiments instead of following a set list of instructions which made it enjoyable. In essence, it felt more like an experiment than other labs which was nice." Overall, students in our study responded positively to the game-like narrative structure of the Manifold Lab and shared the belief that it emulates the real-world practice of experimental physics to a greater degree than lab curriculum they had previously experienced.

VI. CONCLUSIONS
These findings raise the question: Was VR strictly necessary to achieve this outcome? We predict that embodied cognition and VR's improved similarity to hands-on experimentation as opposed to "flat" desktop simulations play a key role. Without a greater understanding of differences in students' perception and experience of VR and desktop simulations, we are not yet in a position to answer this question.
A comparison of the original study's results with ours revealed notable features of the epistemologies of students who experienced NOMR labs. The view held by nearly all of the introductory students in the Hu and Zwickl study -that the purpose of experimentation is to provide supplemental conceptual learning -is less commonly held by the students in our study. Many did not mention supplemental learning at all in response to Q1, instead focusing on the development of scientific abilities.
The students who completed NOMR labs exhibit a more nuanced understanding of the role of experiments and of the iterative relationship between experiment and theory that underpin real-world science. They acknowledge the role of experimentation in the investigation of unknowns and the development of new theories (Q2) and identify more facets of the interplay between experiment and theory (Q4) with frequencies comparable to experts.
This gives us an answer to RQ1: Engaging students in experimentation as a process for theory development may shift their epistemologies about experimental physics to reflect a more expertlike understanding of the complex role of experimentation in development and improvement of theories.
Feedback from the Manifold Lab suggests that students perceived it as an authentic simulation of the process of scientific theory development. Some reported that this approach gave them an unprecedented feeling of connection with realworld science, and had a transformative effect on how they think about experimentation as budding scientists. These insights answer RQ2: Students generally believe they learned more about the practice of science in activities that mimic experimental research than in their previous laboratory experiences.
We have presented findings suggesting that immersion in a novel universe as a research scientist on the cutting edge can shift students' epistemologies about scientific theory development toward expertlike views, reflecting their role in the narrative. With or without VR, we hope this approach aids instructional designers in tackling theory development as a learning goal in their lab curriculum.