Physics students ’ epistemic framings for a conceptual test question

Quality assessment of student learning is important in physics education. However, it can be challenging to construct assessments measuring students’ understanding of concepts and elicit deeper levels of cognition during problem solving. Tests are common assessments, but students assume test questions will have a numeric or symbolic answer. This study analyzes test responses and reflections on a question without a numeric or symbolic answer from a calculus-based physics class. Though designed to test conceptual understanding, students’ assumptions and expectations about test questions hindered their ability to accurately represent their understanding. We analyzed students’ responses and short reflections on the problem by looking at ways students tried to solve the problem and their uncertainties when confronted by a non-traditional test question. Using a framework of epistemic resources, we found that students used a variety of resources to pursue a solution; though common patterns emerge that can inform instructors and physics education researchers.


I. INTRODUCTION
Epistemic beliefs are an important facet of student development and success in introductory physics [1,2].But are sophisticated and flexible epistemic beliefs being developed through assessments?A goal of assessment in a classroom is to accurately reflect the goals and emphases of the course.However, this is not always the case.Elby [3] found that physics students see "trying to understand physics well" as different than "trying to do well in the course."Students recommended altering study habits if the goal was only "to understand physics more deeply."This presents a problem to educators trying to develop in their students a more sophisticated and flexible epistemology of physics.Bing and Redish [4] argued that epistemic flexibility in problem solving is a hallmark of the development of expertise.
As any professor who has been asked "Will this be on the test?" knows, tests are the primary focus of most students.Though physics educators might try to develop the ability of their students to mature in their epistemic beliefs, many students only see value in what will help them succeed on assessments.
Therefore, this study seeks to better understand how students make sense of physics exams and problems.The work is guided by two research questions: Q1.How do physics students' framings of exams affect their responses to exam questions?
Q2.What epistemic resources do students use when attempting to solve a question without a numeric or symbolic answer?
Here the term "framing" is defined as the answer to the question, "what kind of activity is going on here?"[5].The first research question seeks to understand what students expect during formal assessments and if those expectations help or hinder the goal of epistemic development.The second question focuses on what different perspectives students take when solving a problem that does not meet their expectations.

II. METHODS
In order to answer the research questions, this study took a qualitative approach, where the students' test solutions and reflections were considered and triangulated to get a more holistic view of their problem solving approaches on the problem of interest.
The data from this study come from the following two sources: student answers on an exam question and their post-exam reflection essays.The 13 students in this study were enrolled in a calculus-based, introductory physics course.Most students were biology or chemistry majors with a few math, computer science, and pre-engineering also.Many were double majors in two of these fields.The first unit of the Spring 2015 semester included discussion and a lab about oscillatory motion.Therefore, the unit exam contained the following two part question: The angle that a swinging, simple pendulum makes with the vertical obeys the equation Θ(t) = (0.150rad) cos[(2.85rad/s)t+ 1.66] A) What is the length of the pendulum?B) What is the mass of the swinging bob at the end of the pendulum?
The test was administered in the sixth week of the semester and students were permitted to bring an equation sheet of their own making.
The first part of the question is fairly easy to solve given the pendulum's equation of motion.However, the second part has no numeric or symbolic solution.Both parts of this problem were suggested by an introductory physics textbook's instructor materials as multiple-choice questions.The second part included a multiple choice option, "It cannot be determined from the information given."The instructor of this course had discussed the independence of motion from mass, and it had been the topic of a laboratory objective completed during the unit.Therefore, for the unit test, the multiple choice options were omitted with the hope that students would have the conceptual and experiential knowledge to be confident in their answers without them.
Unfortunately, the students struggled with the fact that no numeric or symbolic solution existed.Though most would have probably gotten the question correct in the multiple choice format, only three did in this scenario.
After the exam, the instructor offered full credit on the problem plus extra credit if the students would write a short reflection describing their thoughts and feelings while trying to solve the problem.In particular, they were asked about what methods they used to try to solve the problem and, if they conceptually knew the answer, what were their reasons for questioning their conceptual understanding.The test answers and the students' reflections offer insight into their epistemic framings both of the test as a whole and during a problem solving process.
The student reflections were coded using directed content analysis [6], meaning a set of preexisting codes (described later in this section) were used while also allowing for emergent codes.Two of the authors independently coded each of the student reflections along with the student's test solutions.They then compared codes to check for inter-coder agreement.An emergent theme of test framing was identified [7].
For our prexisting codes, we used the resource framework of Bing and Redish [5] who identified four epistemic resources that correspond to different framings.
Calculation: "Algorithmically following a set of established computational steps should lead to a trustable result." Physical Mapping: "A mathematical symbolic representation faithfully characterizes some feature of the physical or geometric system it is intended to represent." Invoking Authority: "Information that comes from an authoritative source can be trusted." Math Consistency: "Mathematics and mathematical manipulations have a regularity and reliability and are consistent across different situations." These were identified in the student written reflections by coding for characteristics of these epistemic resources.
For example, Ulysses'1 comment, "My initial plan was to use my equation sheet [to] find an equation linking mass and length" was coded as invoking authority.
In this context, students often invoked the authority of their equation sheets.Therefore, it is important to note the distinction between invoking authority and calculation.If a student looked for an equation that only linked two variables, it was coded as invoking authority.If the student looked for equations that could be manipulated or used in conjunction with other equations to produce a solution, this was coded as calculation.
The emergent theme of test framing was organized into categories which describe common test framing schemas.These schemas fell in one of these four categories: (1) test questions have a numeric or symbolic solution, (2) test questions always have one right answer, (3) short answer test questions always have enough information to find a numerical solution (4) test questions use equations from the equation sheet.

III. RESULTS AND DISCUSSION
In answer to the first research question, our data clearly show that the students struggled with this problem due to their framing of the physics exam.When approaching a physics exam, students expect the problems to have a numeric or symbolic answer unless notified otherwise.In her reflection, Jackie said, Although I did know that the period [of the pendulum] is independent of the mass. . .I figured that since it was a short answer question, there had to be a way to calculate mass.The fact that I assumed we wouldn't be given a question incapable of being solved, I didn't give up on the problem until I had a numerical answer.[emphasis in original] Similarly, Hannah wrote in her reflection, Originally, I wrote. . .'the motion of a simple pendulum is independent of mass.'However, I changed my answer because "you can't" is never an acceptable answer on a test. . .I wasn't confident enough to say, "there is no way to do this" because usually if that is an acceptable answer, that is explicitly stated either in the directions or by the professor.[emphasis added] Around half of the students made statements explicitly stating that they were confused that this question did not align with their notion of what a physics problem should look like.This confusion was implied in the reflections of most of the others.It is important to note that even the students who figured out the problem wrote reflections similar to the others.This framing comes from past experiences which have taught them what to expect [7].One student, Calvin, even wrote in his reflection, The thought crossed my mind that there was no answer, but I quickly shot it down because I had never had a question with that type of answer on anything in my academic career.
It is clear that for these students, their framing of physics exams is that "real" answers are numeric or symbolic.Other options must be explicitly allowed to be valid.This causes conflict if a goal of physics education is to develop scientific thinking which allows multiple or ambiguous solutions.Physics educators might tell students this or even engage students in activities or laboratory experiments illustrating grey areas in the of interpretations of science (as was done in this course), but by years of training, students have learned most assessments require numeric answers.Jackie even described the "tug of war going on in my brain" that eventually persuaded her to "write down a number, even though I was pretty sure I was wrong."The students' framings of the test as numeric overwhelmed correct conceptual understanding, leading many students to fruitless attempts at calculating a non-existent value.
The second research question of this study is more specific, asking what epistemic resources (calculation, physical mapping, invoking authority, or math consistency) students used while trying to solve this problem.Clearly, the two research questions are connected.A student's epistemic framing of this problem will depend on how he or she frames the test.If the students in this study frame the test as requiring numeric or symbolic answers by default, they will probably rely on mathematical related framing to solve the problem.
One of the first results to emerge was that while every single student invoked authority in the problem solving process, these invocations fell into two categories.Recall that in an invoking authority framing a student accepts that "information that comes from an authoritative source can be trusted" [5].One type of authority invoked was course related.Six of the students remembered that pendulum motion was independent of mass from the laboratory experiment, class lecture, or an uncited source.
The second type of authority used was the equation sheet.Here, students looked for a simple connection between variables that could lead them to the numeric or symbolic solution they sought.For example, in the previously quoted reflection, Ulysses looked for "an equation linking mass and length."Since he had a numeric value for pendulum length from part A of the problem, he looked for the authority of an equation to tell him what the pendulum's mass would be if its length was 1.21 m.Some of the students used both types of authority.These, after rejecting their original conclusion-derived from classroom authority-because of their test framing, proceeded to use their equation sheets as authority.
The next most common framing, used by all but a few students, was calculation.This usually followed on the heels of invoking authority.When the authority invoked failed-because it was second guessed or because it did not contain a solution-students proceeded to try calculations.Though we expected it to be difficult to distinguish between an invoking authority frame and a calculation, usually the students clearly described the transition from one to the other.For instance, Helen reflected, I recall looking through my theorem sheet, looking for equations for pendulums that involved mass.When I couldn't find any equations, I began trying to manipulate equations in efforts to re-write them in terms of mass.
Here, the authority of the equation sheet failed as there was no equation directly linking pendulum motion to mass.This led Helen to consider manipulating other equations by chaining, the characteristic of a calculation framing [5].
Physical mapping was less common than invoking authority or calculation.Several drew pictures on their test paper, though it probably was just the force of habit as only a handful appeared to use the picture in attempting to solve the problem.Only two students wrote reflections that indicated an attempt to use physical mapping.In these cases, it arose when students reasoned through gravity's effect on the mass of the pendulum.The clearest case of physical mapping was actually a case of incorrect reasoning.Martin wrote in his reflection, It just seemed to make sense that its weight would change the period and amplitude, heavier pendulums decreasing both.This was somewhat surprising as we thought that using more resources might allow the students to better triangulate to the correct conclusion.However, from Martin's case it is clear that framing flexibility was not sufficient for successful problem solving.
Math consistency framings were harder to code as they went hand-in-hand with calculations.There were no clear cases of math consistency framing that were separate from a calculation.Because the problem involved simple harmonic motion, two students tried solving the problem by using equations for spring oscillations.
The nearly exclusive focus of the students on invoking authority and calculations is perhaps not surprising given the results of our first research question.If students expect test answers to be numeric or symbolic, physical mapping and math consistency resources would be utilized least.This also corresponds to other work showing that physics students use mostly equations and definitions on their note sheets for tests [8].McCaskey [8] found only a small percentage of note sheets included diagrams that might classify as a physical mapping epistemic resource.

IV. CONCLUSIONS AND FUTURE WORK
This study used responses and reflections on a test question to investigate student framings of physics tests and the epistemic resources they used while trying to solve a problem without a numeric or symbolic answer.
Most students could not successfully answer the problem because their framings of a physics test did not account for a question without a numeric or symbolic answer.They expected that any other answer would be explicitly suggested by multiple-choice options or legitimized by the instructor.This raises problems for physics educators attempting to reinforce an "expert" epistemology of physics.
Future work should more deliberately focus on student framings of physics tests.The potential conflict between frames and goals of physics education is rooted in years of experience throughout the education system.Therefore, we anticipate that solutions will not be quick or easy.It will likely take much explicit scaffolding to break students and professors of this habit.In this case, an entire laboratory experiment was not enough to help most students arrive at the right conclusion about pendulum masses.Physics educators will need to be deliberate and persistent if they desire to develop in their students the ability to think outside of their traditional test framings.
The data also showed that students primarily focused on invoking authority and calculations as their epistemic resources.This is not surprising as we found that student expectations for exams are that solutions are normally numeric or symbolic.A potential benefit to breaking students-and teachers-of their traditional test framings might be that students become more flexible in their use of epistemic resources.Perhaps if students no longer expect only numeric answers, they will be more likely to employ physical mapping or math consistency frames.
Another area for future study is the extension of the epistemic resource framework to more formal assessments.The work of Bing and Redish [4,5] focused on problem solving during study sessions usually focused on homework problems.However, since tests often are a significant focus of introductory physics courses, it is important to understand how students approach problem solving in that context.This could provide insight into the value of the open-ended question in revealing students' conceptual understanding, student responses on the open-ended question and post-test reflections could be compared to responses and reflections for a traditional multiple choice version of the question.Concept inventory questions relevant to the topic could be given to quantitatively assess students' conceptual understanding, allowing researchers to triangulate students' self-reported understanding in post-test reflections.
The connections between the epistemic resource framework and other frameworks of problem solving are other area for investigation.Epistemology, metacognition, and self-regulation are interconnected [9].It is likely that metacognition and self-regulation play a significant role in the process of choosing and changing frames while problem solving.Understanding the role these connections play in problem solving would be a valuable goal for future research.