Towards an analytic framework for characterizing student use of models

Engaging with models has been considered central to the practice of doing science as it facilitates sense-making of the world around us. Therefore, engaging students in the practice of using models is an important component of their science education. But to do so effectively, we also need to understand how students use models in their work. Consequently, we require a way to analyzing students’ use of models. In the current work, we present an analytical framework which characterizes students’ use of models by considering common themes from the existing literature on modeling in physics. These themes present themselves as ﬁve components: (i) Presence of a real-world phenomenon, (ii) Use of representation(s) depicting the phenomenon, (iii) Invoking of conceptual knowledge organized around representation(s), (iv) Presence of explanation/prediction about the phenomenon and (v) Linking the explanation/prediction to a representation through appropriate reasoning. Analysis of students’ written and verbal responses to physics problems through these components indicate that students seldom link the predictions made to the representations through reasoning, and, when they do, representations are often mathematical equations even though diagrams are present in their solution.


I. INTRODUCTION
Creating and refining models to predict and explain the world around us is central to the practice of doing science [1][2][3][4]. Over the past couple of decades, the physics community has grown increasingly interested in engaging students in using models, often times seeing modeling as a desirable form of physics problem solving (e.g. References [1,[5][6][7]). While modeling is not identical to scientific reasoning, it does connect to many of the ways scientists reason [8]. This importance as a learning tool has been recognized as one of the Scientific Practices of 3-Dimensional Learning (3DL) by the National Research Council [9] and the Next Generation Science Standards [10].
Given the importance of models in both doing and learning science, we are developing an analytic framework to characterize how students employ models in their written work. This would be a significant (and novel) tool, particularly in regards to student written work, as most evidence of students' physics reasoning is exhibited in written responses to homework and exams. Creating this framework would provide tools to help identify and distinguish how students use models. This would allow instructors to better view how students are reasoning with physics, and researchers easier analysis of differences in learning outcomes and instructional methods; however, we leave these details for a later paper.
Here, we present our preliminary results towards developing this analytic framework for characterizing student use of models. Specifically, we address two research questions: 1. What model components do we need in order to identify evidence of student use of models in their work? 2. How can we use this analytic framework to identify and distinguish between student uses of models? In the following sections, we'll address these questions by first briefly reviewing the literature on modeling to determine its components. We'll then apply these components to student think-aloud interviews what that application can tell us about how student reason with models.

II. LITERATURE REVIEW
A full inventory of the literature on models and modeling is beyond the scope of this paper. While not exhaustive or comprehensive, our review captures the themes of a rich discourse investigating students' use of models in physics. These investigations range from analyzing students' models during problem solving [11,12] to proposing frameworks to facilitate modeling in classrooms [6,13] & laboratories [7], modeling expert engagement with models [14], and characterizing the potential for assessments to engage students with models [15]. However, there is considerable diversity in the ways researchers define models and the process of engaging with them (modeling). Nonetheless, amongst these accounts, we find an overarching agreement on the purpose of a model to abstractly represent and reason about a target system [16].
The primary use of models is to make sense of the world around us. Consequently, the real-world context is inherent to how the model is employed [3,8,17,18], and which parts of the phenomena that are abstracted determine the details of the model [5,19,20]. In order for a model to be scientifically meaningful, there must be some phenomena or situation to be the target of the model's abstraction [13,15]. Such target systems can encompass a wide array of real world situations and systems [1,7,11,21,22]. Because of this, curricula emphasizing engagement with models or modeling take a phenomena centered perspective to instruction [3,6,18,23,24].
Models need a phenomenon if they are to describe the world, but they function through an abstract representation of the phenomenon. Representations depict aspects of the target system [19], and simplifications are made to facilitate their usage [5,7,14]. While much work has been done with mathematical representations [1,11,18,21,25,26], they are by no means the only ones [7,22,27]. Often, multiple representations are engaged in concert, forming increasingly sophisticated models [1,6,8,17,[21][22][23][24]. Many curricula and frameworks focus heavily on constructing representations of the phenomena or systems they are modeling as an entry to the modeling process [6,7,23]. Both "model" and "representation" are used interchangeably in everyday speech, but discussions of models indicate that the representational form remains distinct from the model itself [8,13,15,17].
Representations allow physics ideas and concepts to be ordered by the model [1,14,21,22]. Models are "not simply of phenomena, they are tools to be used for reasoning about the phenomena" (emphasis in original) [17], capable of being leveraged as powerful sensemaking tools [8,26]. This requires models to align with physical laws and known concepts to be able to be useful [6,7,22].
All of which foregrounds using the model as a sensemaking tool. For a model to be present in student work, it needs to be used to explain or predict something about the situation being depicted by its representation [5,8,13,15,22]. Use of mathematics in physics models, in particular, has been noted for its unique relationship to explanations and predictions [1,11,19,21,25,26]. This is also a key component in the process of constructing, validating, and refining models as the model must be compared to the actual target system [6,7,23]. It is through application of a model that students engage with it in their reasoning process [17].
However, if a model is to be used as a sensemaking tool, there must be reasoning that connects the representation to the explanation or prediction [1,8,13,15,21,22]. Only through linking the representation together with the explanation or predictions made by the model, can the the valuable insights the model is capable of yielding be achieved [18]. These connections involve the resources used to understand the representation [14], and, in the case of mathematical representations, are facilitated through the use of elements of cognitive resources known as 'symbolic forms' [25].

III. THE ANALYTIC FRAMEWORK
Using the common threads presented from various studies in Section II, we now create a set of 5 components for identifying a model in student work. By this we mean to characterize the model expressed [28] by the students, rather than their personal mental conceptions [29]. However, the terms used vary greatly amongst the literature and don't necessarily lend themselves to transparent usage. Additionally, the varied nature of the literature includes descriptions that do not necessarily have a 1:1 mapping with each other. Because of this, we note that the names we choose for each component is somewhat arbitrary, but the definitions behind them are not.
We propose the following 5 components: 1. Situation: Presence of a real world situation, such as an event, observation, or phenomenon around which a model can be formed. [1, 3, 5-8, 11, 13, 15, 17-24] A real-world situation should be present if the model is to make sense of the world. This can include hypothetical or theoretical phenomena so long as it is not an abstraction. For example, light propagation through the luminiferous aether would count, but an abstract distribution of point charges would not (though charged pith balls would). For student work, this is often set up by the problem statement.
2. Representation: Use of one or more representations that depict the real world situation and guide the organization of ideas. [1, 5-8, 11, 13-15, 17-19, 21-27] The representation can be a mathematical, pictorial, or written description of the real world situation. Unlike a representation as part of a procedural step, depiction of the phenomenon is crucial for it to be relevant to a model. For example, a Free Body Diagram of a real-world object counts as a representation, but a triangle used as part of vector decomposition does not. We also count mathematical statements describing a system, relations, and definitions of quantities as representations. Multiple representations may be linked in student work (e.g. Free Body Diagram and equations) or used together to form a complete representation (e.g. multiple equations).
3. Concepts: Physics ideas about the phenomenon are used by the student and organized by the representation. [1, 6-8, 14, 17, 18, 21, 22, 24, 26] Models are not merely representations, but use representations to abstract and make sense of a situation. The representation achieves this by organizing ideas about the physics of the situation. For example, both the gravitational and frictional forces can be concepts needed to understand an object's motion, and appear as variables on a Free Body Diagram that informs how the forces relate to one another. Variables can be both concepts and representations of those aspects of a system described by those concepts.
4. Explanation or Prediction: Presence of an explanation or prediction about the phenomena. [1, 5-8, 11, 13, 15, 17, 19, 21-23, 25, 26] Models are sensemaking tools that tell us something about the situation they describe. Therefore, an important component of a model is the existence of one or more explanations or predictions -statements that provide a You are asked to design a Gravitron for the country fair, an amusement park ride where the riders enters a hollow cylinder, radius of 4.6 m, the rider leans against the wall and the room spins until it reaches angular velocity, at which point the floor lowers. The coefficient of static friction is 0.2. You need this ride to sustain mass between 25-160 kg to be able to ride safely and not slide off the wall. If the minimum ω is 3 rad/s will anyone slide down and off the wall at these masses? Explain your reasoning using diagrams, equations, and words.
FIG. 1. Assessment task given to students during the interviews. The question is aligned with the 3D-LAP [15] for the practice of Developing and Using Models. details for why an event occurred/what will happen without providing justifications using scientific principles. Different models can be used for the same situation, providing different explanations or predictions about the situation.
5. Linking Reasoning: reasoning that provides justification for the explanation or prediction and connects it to a representation. [1, 8, 13-15, 18, 21, 22, 25] Providing reasoning is important as it provides principles that support the explanation. Without reasoning underlying the prediction or explanation, student concepts are not being leveraged.
We've identified 5 components from the literature on modeling that can be used to characterize how students use models. These 5 components answer our first research question listed at the end of Section I. They capture the context in which the model can be applied, the use of the model as a sensemaking tool, the centrality of its representational form, and the knowledge it allows its user to put into use.

IV. METHODOLOGY
The data consists of video of and written solutions collected from think-aloud interviews of 9 engineering majors (7 men & 2 women) from a large Midwestern US university. Interviewees were recruited from introductory physics courses irrespective of grades, and volunteers were compensated with the equivalent of $20. Interviewees were given a set of physics problems on topics commonly found in introductory mechanics courses in an untimed, exam-like setting. The interviewer interjected only to answer clarification questions or prompt the interviewee to articulate their thoughts after long pauses. Interviews were conducted at the end of the semester after all topics in the problem set had been covered; one interviewee had completed first semester mechanics while the rest were co-enrolled. The problem set consisted of a blend of questions with and without the potential to elicit the Scientific Practice of Developing and Using Models according to the 3D-LAP [15]. All 9 students were unfamiliar with 3DL assessments and the situation shown in the task presented in Figure 1. As this study is ongoing and a larger data set is needed, we have not completed IRR. Instead we will fully present a single student solution to the task in Figure 1, along with the coding of it. Our analysis consists of coding the completely solved written solutions of the problem in addition to video and audio recordings (henceforth referred to as "verbal"). The coding is done in terms of the components elucidated in Section III. For the written solutions, we went line-by-line, while following the sequence of student work as much as possible. As coding progressed, an electronic copy of the work was annotated to indicate where certain codes occurred (see, for example, Figure 2). For the verbal data, the video of the segment was iteratively viewed and the corresponding transcript was taken into account to find evidence allowing us to code for the components a model. Verbal coding also took into account student gestures, pauses, etc. to understand how they were referring to their thinking and work. We'll illustrate the coding process below with an excerpt taken from the written data of a student we'll refer to as Michael (see Figure 2). How we identify evidence is otherwise the same, though requiring explicit statements from the student about what they are doing to identify evidence of each component; due to space constraints, we will not give an example of coding verbal data.
Michael's approach to the problem is initiated by the construction of a Free Body Diagram (FBD) -a representation, with labels of forces (acting on a person in the Gravitron) that indicate concepts being leveraged (presence of components 1, 2, and 3). To its right, Michael begins a series of calculations, each starting with relations for F N and F fr , which are representations of the forces and their corresponding concepts. We see a similar set of algebraic representations and concepts for torque and force towards the end of Michael's work.
Directly following a calculation of force from torque, Michael then gives both an answer to the question and a justification for it, "Force via the torque is larger than the gravitation and frictional forces so it'll work" (a prediction, component 4). Furthermore, the reasoning references the representation used for the calculation ("Force via the torque"), allowing us to identify component 5. This representations relates force and torque, and the claim is made by comparing the calculated values. With all the five components being identified, we conclude that Michael used his (mathematical) model to reason about the Gravitron carnival ride.
After coding each interview, we compared codes for those solutions that had evidence of all 5 components. This allowed us to see in which data (verbal or written) evidence was appearing. From there, we focused on codes given for the 5th component (linking reasoning) as we found the most variation between written and verbal solutions. Those codes allowed us to gain a full picture of the model by tracing its connections to evidence of the other components. This allowed us to characterize how students were reasoning with their model and therefore what differences existed.

V. RESULTS
By going through the common themes from the existing literature, we identified five components as part of our framework to identify students' engagement with models. These include: (1) presence of a real-world phenomenon, (2) use of representations depicting the phenomenon, (3) employing physics concepts and ideas, (4) presence of an explanation or prediction, and (5) linking generated explanation or prediction to the representation through appropriate reasoning.
Among the nine interviews from the data set, we find six of them completely solved the problem (irrespective of the correctness) and thus being eligible to be coded in terms of our five components. Of those six completed written solutions, only two of them (including the one in Figure 2) exhibited all the five components, fully aligning with the practice of using a model. Including verbal data, we find 4 of the 6 completed solutions exhibited evidence for all 5 components.
Notably, we find that linking reasoning (component 5) was the least likely component evidenced by students' written work. Among various representations available, 3 students referred to mathematical equations, but not to any of their diagrams while justifying their prediction. While diagrams were present, they appear to be representations used as a source for their calculations, whereas reasoning made use of subsequent representations that were employed during their solution. This highlights the applicability of the presented framework in capturing students' diverse engagement with models while solving problems. This finding also calls out the necessity of making it explicit in the problem statement that reasoning must be provided that involves a representation.

VI. DISCUSSION
The components we have developed have helped us understand what students are doing when they use a model to answer a question. This is most notable in how the linking reasoning component frames student solutions. Not only is this the predominantly missing components, but when it is present it allows us to more clearly see what representation form is being used by the model.
In Figure 2, we can see 3 of the 4 sets of representations employed by Michael: various force expressions, an expression for torque (another torque equation not shown), and a Free Body Diagram. As discussed in the coding of Michael's work in Section IV, reasoning links this prediction directly to the mathematical representations for the torque and individual forces. However, we do not see the force representations being organized by the Free Body Diagram. For instance, the relations between the forces are not consistent with the diagram, and, in the reasoning itself, both the frictional and gravitational forces remain as independent entities in the Michael's reasoning. This was even more explicit in Michael's verbal response, as he indicated his shift away from the Free Body Diagram and toward kinematics almost immediately. He does this stating he is "Change[ing] to the kinematic equations to be angular... to get an acceleration", and then, after computing the values for the gravitational and frictional forces, and that these results are part of his attempt to, "Still trying to figure out how to find [angular] acceleration [...]". This tells us that the Free Body Diagram, whose presence we might otherwise take to be indicative of its usage in student reasoning, is in fact not a part of the model employed by the student. This does not have to be the case. For example, we can easily imagine students' reasoning referencing how forces superimpose as indicated by the Free Body Diagram they constructed.
The use of multiple representations in modeling is well known [1,19,24], and curricula that include models often describe navigation of multiple representations as a goal [6,23,30]. However, using our components, we can clearly see how student use representations as a part of model-based reasoning, and that not all representations are necessarily being utilized. Linking reasoning clearly indicates this, but that was the least likely component to be evidenced in our data. This may be because unfamiliarity with 3DL assessments means that providing reasoning is not a normalized skill, or that more explicit scaffolding is needed beyond "explain your reasoning using diagrams, [etc.]". Further study is necessary, but we recommend that instructors be intentional in asking students for their reasoning.

VII. CONCLUSIONS
In this paper we present initial steps towards an analytic framework for characterizing how students use models as reasoning tools. We've synthesized major threads of the litera-ture on models to understand what components could form a basis for a full analytic tool. Doing so, we've adopted a 3DL-informed perspective in aligning the varied descriptions of models given by different authors. However, at this stage, these are still merely components, and more work with a larger data set is required to create an analytic tool. Our results are also limited by the fact we have confined ourselves to only one dimension whereas 3DL posits the 3 dimensions are intertwined, each contributing to knowledge organization and reasoning.
We then applied these components to think-aloud interviews with students solving an introductory-level mechanics problem. Doing so illuminated the richness in student modelbased reasoning. Using our components, we could see how students shift between representational forms and how these are built into a model that students use to reason about a physical situation. We also found clearly presented Free Body Diagrams that did not play a role in how the model was used as a reasoning tool. Free Body Diagrams are commonly found in introductory physics courses, and we might expect many instructors would want their students to be able to reason with them. However, our results indicate that the situation is more complex than we might suspect as instructors when reviewing student work.
While more work is still to be done and a larger data set is needed, these components appear viable for building an analytic framework for characterizing students' use of models. Model-based reasoning is an important aspect of scientific reasoning, and is a major theme in a growing number of physics curricula and teaching frameworks. Consequently, instructors will be interested in understanding what choices their students' make in deploying models to make sense of the world. Similarly, researchers may want to investigate those choices and their implications, and what the impact of reformed curricula have on this aspect of student thinking. For both instructors and researchers who have an interest in student use of models, such an analytic framework we hope to report back on would be a boon.

VIII. ACKNOWLEDGEMENTS
We would like to thank M. Cooper and her group, C.V. Schwarz, P. Nelson, and D. McPadden for early feedback. We'd also like to thank B.R. Wilcox for discussing the development of the ACER framework. This work is supported by the NSF under Grant Nos. 1725520 and 1726360.