A methodology for identifying task features that facilitate sensemaking

Assessments provide opportunities for students to make sense of curriculum by connecting their existing ideas to generate new knowledge. As educators, we would like to be able to engage students in this process of sensemaking to help their learning. Through a case study, we present an approach to identifying assessment task features that facilitate sensemaking. The context involves an introductory student’s attempt to a physics problem. We analyze the student’s transcript through the lens of sensemaking epistemic game and note the student’s actions (‘moves’) while solving the problem. By analyzing these moves, we argue that the assessment features that played a role in the student’s sensemaking can be identiﬁed. We ﬁnd that the presence of a real-world context, engaging with multiple representations, and requiring a physical interpretation of a numerical answer to be instrumental in student’s sensemaking. Identifying task features in this way can be instrumental in designing future assessment tasks that increase the chances students engage in sensemaking while working on them.


I. INTRODUCTION
Sensemaking [1], the process of making sense of real-life scenarios through formal knowledge, has attracted considerable attention in recent years [2][3][4][5]. It has been shown to assist students in generating new knowledge in addition to establishing connections between their existing ideas [6,7]. It has also been associated with the ways disciplinary experts engage in knowledge construction [8,9].
Given this significance, researchers have investigated task features and discourse markers that promote sensemaking. Vexing questions during interactions [10], blending modeland evidence-based arguments during classroom investigations [11], computational reasoning of physics scenarios [12], solving quantitative problems through conceptual insights and vice-versa [5], and reasoning about the physical features of systems through mathematical tools (such as graphs) [13] are a few such findings. The results from this study contribute to the literature on eliciting sensemaking, particularly through assessments.
In the rest of the paper, we demonstrate the viability of an approach to identify the task features that promote sensemaking. We achieve this objective by qualitatively analyzing a case study of an introductory physics student's sensemaking through the framework of the sensemaking epistemic game [14]. By navigating through the transcript, we identify the student's problem solving 'moves' associated with specific stages of the epistemic game. The moves include reasoning around a constructed representation and physically interpreting a numerical value. Recognition of such features takes us a step closer to purposefully elicit sensemaking in our classrooms, thereby strengthening the 'pedagogical tool box' [14].

II. THEORETICAL FRAMEWORK
The primary objective of sensemaking is to 'figure something out'. It involves building or revising explanations surrounding the underlying mechanism of a phenomenon. These explanations are often a blend of one's everyday knowledge and formal knowledge [1].
Various studies have adopted the framework of epistemic games [15] to investigate problem solving in physics [16][17][18]. Epistemic games correspond to the set of rules employed when undertaking a scientific inquiry towards the goal of the task called the target epistemic form. These epistemic forms vary with the task under investigation. When playing any game, we abide by a certain set of rules which cannot be violated. In epistemic games, these are referred to as the constraints. The circumstances under which a person begins to play the game are called entry conditions. These conditions are often triggered by the very nature of the inquiry. Moves are the set of actions taken at various stages of the game and transfers are the subsets of those moves which change the direction of the game towards another epistemic game.
Based on data from clinical interviews of pairs of students, Odden and Russ proposed a 'sensemaking epistemic game' [14]. According to this model, the process of sensemaking involves navigation across all the following four stages of the epistemic game: Stage 0: Assembling a knowledge framework: Prior to sensemaking, students prime their existing knowledge on the domain. The priming of knowledge is often influenced by the contextual cues such as interaction with instructor or peers. Stage 1: Noticing a gap or inconsistency: This stage corresponds to noticing an inconsistency between what students know (students' existing knowledge) and what is demanded from the task (generation of new knowledge). The noticing of the inconsistency marks the entry condition into the epistemic game. Stage 2: Generating an explanation: Upon noticing an inconsistency, students attempt the task by iteratively building diverse explanations. These explanations (or moves of the epistemic game) are often a blend of everyday and formal knowledge. Stage 3: Resolution: This marks the final stage of the sensemaking epistemic game. At this point the student reaches the target epistemic form of the game. This is often accompanied with confident articulation of the explanation supported by due justifications. In the present work, we adopt this framework to address the research question: What are the assessment task features which influence the student's problem solving 'moves' across the stages of the sensemaking epistemic game? Answering this question provides a methodology moving forward for identifying task features which increase the likelihood that a student engages in sensemaking while working on a task.

III. METHODOLOGY
The present case study is derived from a pool of video data taken from a Midwestern US university to investigate students' engagement in scientific practices [19]. It involves students participating in think-aloud interviews [20] while solving a set of physics problems [21]. The problem set included questions specifically designed using the Three-Dimensional Learning Assessment Protocol (3D-LAP) [22] to elicit the scientific practice of 'Developing and Using Models' [19]. This set consisted of problems from introductory college physics and the participants were asked to treat the set of tasks as an exam. Students were allowed to use calculators and were provided with an equation sheet.
Out of nine problems these students were asked to solve, we focus on Matthew's response to problem seven, which we henceforth refer to as the 'Gravitron problem', shown in Fig. 1. We chose to focus on this case for three reasons. First, Matthew did not engage in sensemaking (as defined in Section II) on the six problems prior to this one. Second, unlike the first six problems, the Gravitron problem was written to You are asked to design a Gravitron for the county fair, an amusement park ride where the rider enters a hollow cylinder, radius of 4.6 m, the rider leans against the wall and the room spins until it reaches angular velocity, at which point the floor lowers. The coefficient of static friction is 0.2. You need this ride to sustain mass between 25-160 kg to be able to ride safely and not slide off the wall. If the minimum ω is 3 rad/s, will anyone slide down and off the wall at these masses? Explain your reasoning using diagrams, equations and words. We transcribed the selected, 7.5 minute episode by taking into account the participant's physical gestures and written solution. As part of the initial investigation, our analysis primarily focuses on the verbal arguments. We refrain from providing the student's written work due to space constraints.
We attempted to identify each stage of the sensemaking epistemic game by looking for behavioral and verbal cues [10,11] associated with that stage while watching the video recording, reading the transcript, and examining his written solution. For example, we looked for instances of Matthew 'being stuck' as an evidence for the 'noticing a gap' stage of the epistemic game. Instances preceding this stage where Matthew contemplated the problem were used as evidence of the 'assembling knowledge' stage. The segmented transcript (in terms of the stages of the epistemic game) was then discussed and validated with an external expert. Upon identifying the various stages of the epistemic game in student's approach, we focus on Matthew's actions ('moves') across each of those stages. Informed by these moves, we then hypothesize the task features responsible for Matthew's engagement in sensemaking.
In the following section, we summarize Matthew's overall approach to solving the Gravitron problem. Once we have identified the four stages and Matthew's moves within them, we then demonstrate a methodology of identifying the task features influential in Matthew's sensemaking (Section V).

IV. ANALYSIS OF MATTHEW'S TRANSCRIPT
Matthew's attempt to the Gravitron problem begins with him noting the parameters given in the problem statement. Thereafter, he realizes an inconsistency between his existing knowledge of linear motion and the expected knowledge about rotational motion. Upon noticing this inconsistency, he constructs a Free Body Diagram (FBD) that guides his formulation of a solution. In the process, he realizes that for people in the Gravitron not to fall, the force of friction needs to be greater than or equal to the force of gravity. The result of this inequality makes him conclude that the occupants of the Gravitron would slide under the specified conditions. We summarize the moves Matthew makes within each stage in the second column of Table I.
Stage 0. Assembling a knowledge framework: Matthew begins by circling the magnitudes of physical quantities from the problem statement. He then states: So, I just went through and circled all the important information.
Here it talks about the mass of the rider. These statements reflect the first stage of the epistemic game, i.e., assembling one's knowledge framework. The assembling is marked by identification and pooling of relevant information from the problem statement. The role of the problem statement in cuing what information to gather during problem solving has been noted by other works [23][24][25].
Stage 1. Noticing a gap or inconsistency: After marking the provided information, Matthew focuses on the mass of the rider. Invoking his intuitive knowledge, he makes a passing reference on the ease of heavier people falling down. After a brief pause, he resumes his argument with mentioning of angular momentum.
If [Pauses for 17 seconds]. The above episode reflects the entry condition into the sensemaking epistemic game. Matthew's sense of puzzlement ("is that right?") suggests he is noticing an inconsistency in his thinking. We suspect it is between his everyday knowledge on linear motion (i.e., free fall by a heavier person) and the expected knowledge on rotational motion (reference to 'angular momentum'), as this is consistent with the existing literature on students' difficulties with rotational dynamics [26]. The noticing of inconsistency is also accompanied with hand gesture, a 20 second pause and initial attempts to construct a representation.
Stage 2. Generating explanations: Resuming after a 17 second pause, Matthew shifts his line of reasoning from free fall and angular momentum to the domain of forces. According to his revised explanation, in order to avoid people from sliding down the Gravitron, the normal force would need to exceed the downward pull of gravity. Matthew then pauses for a prolonged period of time after making this claim.
Since the riders are going to be thrown into the wall, you need a normal force that exceeds the downward force of gravity. [Pauses for 38 seconds, looks back and forth between problem statement and equation sheet].

Matthew's Moves
Gravitron Task features Assembling the knowledge framework

Identifies information from the problem statement
Given numerical values of physical quantities, particularly mass range of the rider.

Noticing inconsistency
Invokes his intuitive knowledge about falling. Gravitron as a real-world context.

Generation of explanations Constructs a representation and translates the diagram based arguments into mathematical equations
Solution approach requires 'mathematizing' the arguments from constructed diagram.

Resolution
Physically interprets a numerical value to make a claim about Gravitron.
Asks for a prediction about people in the Gravitron.
TABLE I: Summary of the stages of the Sensemaking epistemic game, the moves Matthew takes within them, and the features of the Gravitron task that were needed for Matthew to make those moves.
The student's transcript until this point highlights the mental modeling [27,28] of the situation. Without using external representations (diagrams and equations), Matthew articulates what he thinks is the necessary mathematical condition for people not to slide down the Gravitron. The articulation of this condition along with the hand-gesturing (in the previous stage) reflects the complexity of the student's thought process. Other studies too have associated such gesturing with mental modeling [29][30][31][32].
At the end of the 38 second pause, the interviewer intervenes and asks Matthew, What are you thinking? Matthew replies that he is trying to recall an approach to solving the problem. He then proceeds with drawing a FBD and calculating the magnitude of the angular acceleration.
I . This segment of Matthew's transcript is significant for two reasons. First, he mentions trying to recall the procedure for solving the given problem, which suggests a transferring move to a different epistemic game [18]. However, in the rest of the transcript we do not find any explicit mention of him recalling a prior solution. Second, it includes the construction of a representation (FBD) depicting the forces acting on a person inside the Gravitron. Through this representation, Matthew mathematically models [27] the given context by equating frictional force and gravity. He constructs and then manipulates the equation with the help of the provided equation sheet.
In the following transcript (which happens a little while later in his solution attempt), he translates his empirical claim, guided by the diagram, into mathematical equations.
Assuming that it's 90 degree, all of their... all of that will be converted into frictional force [looking towards the equation sheet]. Yeah... that's the normal force.. So F = ma. [......] So it's the the normal force into the wall, the force of friction, has to equal the force of gravity and both have mass, and this mass is negligible, so you have just ..... a equals.... and then this a has to be bigger than or equal to [erases]. So, the force of friction is normal force times mu equals force of gravity. Normal force in this equation is the angular acceleration times mass. Stage 3. Resolution: Continuing the 'cranking' of equations, Matthew arrives at a mathematical inequality consisting of centripetal acceleration, the coefficient of static friction, and acceleration due to gravity. The result of this inequality makes him infer that under the given parameters, people would slide off the wall.
For F = ma times mu equals mass times gravitational acceleration and then both have mass [cancels], so yeah 13.8 times the 0.2-coefficient of static friction and that has to be greater than or equal to 9.8 for the Gravitron to be able to hold people up. This exercise reflects the last stage of the epistemic game. The student reaches the target epistemic form, i.e., whether people in the Gravitron would slide off the walls under the given conditions. With the help of a mathematical inequality, Matthew makes the required claim.

V. IDENTIFICATION OF TASK FEATURES
After analyzing Matthew's approach through the lens of the sensemaking epistemic game, we now shift our focus to identify the task features facilitating sensemaking. To achieve this objective, we identify the task features Matthew needs and/or uses during his moves within each stage of the epistemic game. A summary of these observations can be found in Table I. Informed by these observations, we claim that the following features of the task were influential in nudging Matthew towards sensemaking (summarized in Figure 2): Numerical values from the problem statement. Matthew focuses on the numerical values from the problem statement during the knowledge-assembling stage of the epistemic game. Even though this stage is prior to sensemaking, it can considerably influence on students' initial approach to the problem. For instance, Matthew begins his approach by focusing on the mass of the rider (whose values range from 25 to 160kg). This choice by Matthew suggests that the range of plausible values from the problem statement can influence students' initial attempts during sensemaking.
Presence of a real-world scenario. In the Stage 1 of the epistemic game, Matthew notices the need to reason from formal knowledge (on rotational motion) after arguing intuitively on the free fall by a heavier person. We suspect that the presence of a real-world context, in this case Gravitron, was instrumental in invoking Matthew's intuitions which further informed an inadequacy in his approach (and thus sustaining sensemaking). Our argument is also consistent with recent findings on the real-world contexts promoting students to generate explanations from their lived experiences [3].
Solution requires navigation across multiple representations. In the explanation-generation stage, Matthew predominantly reasons with equations which are informed by the constructed FBD (and the equation sheet). This navigation across multiple representations, i.e., from a constructed diagram to mathematical equations is a key feature in the Stage 2 of the epistemic game. From this observation, we claim that a solution approach requiring navigation across multiple representations can engage students in sensemaking.
Seeking a prediction about the real-world context. In the 'Resolution' stageMatthew used the result of a mathemat-ical inequality to make a claim about the Gravitron. A target epistemic form which requires extracting the real-world implication of the numerical value (unlike its calculation) can engage students in making sense of the task.

VI. DISCUSSION AND FUTURE WORK
We analyzed an introductory student's attempt at solving the 'Gravitron problem' (refer Section III) across the four stages of the sensemaking epistemic game. Before sensemaking, Matthew's priming of knowledge is mainly cued by the information from the problem statement. The noticing of inconsistency in his understanding is associated with invoking intuitive knowledge, recurring pauses, a sense of puzzlement, a hand gesture, and initial attempts to construct a representation. While generating explanations, we noted Matthew expressed the diagram-based arguments into equations and manipulating them. In the end, Matthew uses a mathematical inequality to make his claim.
The significance of this work lies in the application of the sensemaking epistemic game to a new context, and in using this approach to identify task features that may have triggered the student to engage in sensemaking. Unlike studies that involve students' collaborative work [3,14], our context is a student working alone on a paper-based assessment. We use this context to identify task features that may have promoted the student to engage in sensemaking (Fig. 2). Future work will identify more task features and the role(s) they play in eliciting sensemaking.
We do not claim that the above mentioned task features are the only ones responsible for nudging Matthew towards making sense of the problem. Neither do we claim that these task features necessarily engage students in sensemaking. The objective of the current work is to elucidate an approach to qualitatively analyzing student work to identify the task features the student needed to engage in sensemaking. Future work will focus on extending this approach to a larger data set consisting of other students' approaches to the Gravitron as well as other problems from the set. Given that the Gravitron problem was crafted to elicit the scientific practice of 'Developing and Using Models' [22], analyzing a larger data set could also shed valuable insights on eliciting sensemaking by engaging students in modeling. We believe that such explorations can lead to an approach to assessment design that increases the chances students engage in sensemaking.

VII. ACKNOWLEDGMENTS
We would like to thank Katherine C. Ventura for creating the tests and conducting the interviews. Also thanks to Tor O.B. Odden, Paul D.O. Bergeron, Matthew Mikota and members of KSUPER for their valuable insights. This material is based upon work supported by the National Science Foundation under Grant No. 1726360.