Using multiple survey questions about energy to uncover elements of middle school student reasoning

One power of middle school physics teaching is its focus on conceptual understanding, rather than mathematical modeling. Teaching energy in middle school allows one to focus on the conceptual ideas, metaphors, and analogies we use to make sense of the topic. In the Next Generation Science Standards, energy is both a core disciplinary idea in the physical sciences and a crosscutting concept. In this paper, we provide several examples of seeming contradictions in student responses to similar questions. For example, students think differently about energy flow to the air or the ground. They also think differently about energy flow in cold and hot situations, though not necessarily as expected. Analyzing these results carefully, in particular when comparing and contrasting seemingly similar questions, may help both researchers and teachers listen for ideas, target instruction, and recognize learning more effectively.


I. INTRODUCTION
In the Next Generation Science Standards, energy is one of the four Disciplinary Core Ideas in the Physical Sciences and the flow of energy is a crosscutting concept throughout the sciences [1]. As part of the Maine Physical Sciences Partnership, we have been studying the ways in which middle school teachers and students understand the concept of energy [2][3][4][5][6]. As part of our work, we developed a survey containing questions primarily taken from the assessment website of the American Association for the Advancement of Science (AAAS) [7], as well as the Misconceptions-Oriented Standards-Based Assessment Resources for Teachers (MOSART) project [8] and with some questions written locally. By looking at how similar questions are answered differently, we are able to investigate students' understanding of subtle issues around energy and energy flow, helping us provide feedback to the teachers who can then make use of this knowledge to teach their students more effectively.
We present two examples in which it is possibly misleading to carry out an analysis based on most common student difficulties. In the first, we show how student understanding of energy may not be due to the welldocumented statement that energy is "used up" [9], as suggested by each individual question, and may instead be due to a wholly separate difficulty. In the second, we show that nearly all students use the idea of "coldness" as a kind of energy when reasoning about energy flow, consistent with the literature [10,11], but that they are more likely to do so in some scenarios than in others. In both examples, we find evidence that student reasoning depends on the context and system represented in each problem. Examining the contextual factors that influence students' reasoning about energy may therefore be a productive area for further study of students' thinking about energy.

II. EXAMPLE 1: WHICH OBJECTS HAVE ENERGY?
Two questions taken from the AAAS Achievement website were chosen to investigate how students think about energy loss due to frictional effects, but were modified locally [6]. In question NG079002 (Fig. 1), a box slides across the floor and comes to rest. Students are asked about the loss of kinetic energy -was energy used up, or transformed into thermal energy in the box and floor? In question NG065004 (Fig. 2), a pendulum swings and comes to rest. Students are asked about the loss of energy of the pendulum. For both questions, answer A is correct.
We analyzed the results of 1175 middle school students who answered these questions before instruction in two consecutive years. Data were taken from multiple schools in Maine, with multiple teachers, using an online survey tool. When considering each question individually, we grouped responses into categories consistent with three ideas: energy is conserved, energy is used up, or "other." These interpretations were driven by the multiple-choice responses in each question. We recognize that this oversimplifies the ideas that students might be using to answer these questions, but made the choice in part because it is consistent with the research literature on the most common difficulty students have with each question ("used up") and is also consistent with what teachers we work with highlighted as the most problematic response in their classrooms.
Results show that far more students chose an answer consistent with energy conservation for the Box question (43%) than the Pendulum question (29%) (Fig. 3). Also, far more students chose an "energy is used up" response on the Pendulum question (59%) than on the Box question (25%). We wish to understand the large difference in how many students give the "used up" response to each question.   We compared student responses to the two questions by looking at the differences between the questions. One difference is where the energy might flow as the motion of the object changes. In the Box question, the "floor gets a little warmer," while in the Pendulum question the energy might be "transferred somewhere else, like the air." The ground (or floor) is a readily understandable object that students can imagine energy flowing into, but air is not, for many students at this age [12]. In other words, it is highly possible that students accurately recognize that energy is supposed to be going somewhere, but do not believe that the air is an object that can have energy. As a result, they may be more inclined to answer "used up" not because of issues related to the nature of energy but because of issues related to objects. Answer A for the Pendulum question ( Fig. 2) might seem absurd to them, so they pick another answer.
To illustrate this point, we compare student responses to the two questions in Table 1. We note that 70% of students gave an "energy used up" response at least once but only 13% of students do so consistently. Furthermore, twice as many students who answered "used up" on the Pendulum question gave the correct answer (26%) as opposed to the "used up" answer (13%) on the Box question. They are not treating these situations identically, and are more likely to answer correctly (or with other difficulties) than consistently saying "used up." Further research would be needed to determine their exact reasoning.
We note one shortcoming of this analysis. In the Box question, energy transfer to the floor is stated in the stem of the question. The Pendulum question lacks clarity because the option of energy flowing into the air is only given in the first offered response and not in the stem. Nevertheless, we believe students reading this idea in answer A would be able to understand it and keep it in mind while thinking through the other responses as well.
In conclusion, in our first example, we find that a simple analysis of student responses (in terms of energy being used up) is not sufficient for understanding the differences in their responses to two different questions. Students need to understand the issue of energy degradation [13,14] but it is highly likely that something else is leading to students' incorrect answers, in addition, namely, the manner in which they think about objects in a system. This shows the complexity of reasoning about energy, even in middle school science. A student shoves a box, and it slides across the floor. As the box slides across the floor, the box slows down and both the box and the floor get a little warmer. What happens to the energy of the box as it slides across the floor and comes to a stop and why?
A. The motion energy (kinetic energy) of the box decreases and its thermal energy increases because the motion energy is converted into thermal energy. B. The motion energy (kinetic energy) of the box decreases and its thermal energy stays the same because motion energy is used up and is not converted into thermal energy. C. The motion energy (kinetic energy) of the box stays the same and its thermal energy increases because new energy in the form of thermal energy is made. D. Both the motion energy (kinetic energy) and the thermal energy of the box decrease to zero because the box is no longer moving and an object has energy only when it is moving.
A student is playing with a pendulum (a ball attached to the end of a string). He gives the ball a push and watches the ball as it swings from side to side. After a while, the ball stops swinging. Why does the ball stop swinging?
A. The motion energy (kinetic energy) of the ball is transferred somewhere else, like the air, as the ball swings from side to side. B. A little bit of the ball's motion energy (kinetic energy) is used up each time it swings from side to side, but the motion energy is not transferred anywhere else. C. Some motion energy (kinetic energy) is transferred somewhere else, like the air, and some motion energy is used up. D. An object only has motion energy (kinetic energy) when a person makes it move, and the student is no longer pushing the ball

III. EXAMPLE 2: WHAT IS COLDNESS?
In our second example, we describe a similar result, in which the most obvious issue might not be the most relevant for a teacher to attend to.
In another year of giving a different group of middle school students the energy survey before instruction, 345 answered three questions dealing with thermal energy flow due to temperature imbalances. We refer to these three questions as the Soda Can, Sue, and Frying pan questions. In the Soda Can question [15], a warm can of soda is placed in cold water and its temperature changes. In the Sue question [8], a metal rod is placed in ice water and "Sue" grabs the end not in ice water and notices that it is cold. In the Frying Pan question [16], a hot frying pan is placed on a counter at room temperature and eventually cools down. In each of the three questions, as modified from their AAAS or MOSART sources locally, students could choose answers where only thermal energy transfers, only coldness transfers, or a mixture of thermal energy and coldness transfers. Because heat is a highly ambiguous term in K-12 education [17], we used the term thermal energy, consistent with the materials students used in MainePSP classrooms.
In our data, 342 of the 345 students (99%) used the idea of "coldness" at least once across the three questions. The literature on student ideas emphasizes the difficulties with distinguishing heat and temperature [18,19] while mentioning cold far more rarely. We were thus surprised by the prevalence of the coldness response, and wished to understand its use in each of the three questions.
Our expectation had been that the Frying Pan question would be answered in terms of thermal energy and the Soda Can and Sue question would be answered in terms of coldness, but the issue was not so clear (see Table 2). On the Frying Pan question, 66% of student answers included coldness in some fashion, with 15% answering with only coldness. On the Soda Can and the Sue questions, 63% of students gave an answer including thermal energy, though only 13% or 6%, respectively, chose the answer using only thermal energy. It is tempting to say that students pick the answer that matches their perception, e.g., thermal energy for the hot pan, coldness for the cold soda can or metal rod in ice. But, we can account for a larger share of the data by saying that students rarely pick only the answer that contradicts perception, e.g., they rarely pick only thermal energy for a cold can or only coldness for a hot pan. The fact that more than 50% of students on each question answer using both coldness and thermal energy requires further explanation. As part of previous work [4], we engaged with teachers about the ways in which coldness is and is not a problematic model. When talking about coldness, students are talking about the flow of this quantity from one object to another, consistent with the crosscutting concept of the NGSS regarding energy flow. Also, they are using the metaphor of energy as a substance-like quantity, something that has been shown to be of value to students [20] and experts [21] and is consistent with the NGSS [22]. Both of these ideas are of great value, and it seems that there is some intuitive power in what students are doing. The problem is thus not with coldness in and of itself, but that it is the wrong kind of substance, though one that makes perfect sense in Maine, where one might yell "close the door, you're letting the cold in!" on a winter evening. One challenge for teachers is how to have students keep ideas of transfer and substance while helping them refine the idea of which substance to use in their modeling. This conclusion is consistent with our previously published work [4].
A second issue takes a different look at the data. In each question, slightly more than half the students use both coldness and thermal energy in their thinking. Thermal energy flows one way, and coldness flows the other. This answer complicates the simple suggestion that students need to refine their thinking about which energy substance to consider. For example, we have observed students saying that equilibrium (say, with room temperature) is arrived at when all the hot and cold energies have canceled out. We are in the process of carrying out interviews with 8 th grade students, and are finding that their thinking about this topic is far more complex than expected. We observe that students may think about thermal energy in terms of the motion of particles in the system (NGSS standard MS-PS3.A), but treat rigid, cold objects, like ice, differently, in part because ice molecules that are rigid don't wiggle and therefore have no thermal energy. Perhaps the issue is not only a misunderstanding about the substance of energy, but also the connection between microscopic models (of motion) and macroscopic descriptions (of thermal energy), and the consistency of models one uses in describing the world [23]. Again, further research is required.

IV. DISCUSSION
"Energy gets used up" is described as a common misconception, and "Students talk about coldness as a kind of energy" is an observation made by many teachers. In both our examples, these easy and obvious conclusions may not be the most productive for a teacher to consider. They diagnose the difficulty, but do not suggest a path toward addressing the problem.
In the first example, thinking of energy being used up was the easy response, and made sense of the data for both questions, individually. When considering the two questions together, though, it became clear that energy being used up was not the only issue to consider. Student performance on the Pendulum question was worse in a way that suggests that the issue is not only energy being used up but also that air is not an object that can absorb energy from the pendulum. This is a problem of how to understand the objects in the system, not how to understand energy.
Similarly, the nature of energy might not be the only problem for students when answering thermal energy flow questions in terms of coldness. Instead, as made clear by the majority of students using both thermal energy flow and coldness flow, the problem may be that students do not have a coherent and consistent microscopic and macroscopic model of energy and the mechanisms by which energy flows. Though the NGSS discusses temperature in terms of "the average kinetic energy of particles of matter" (MS-PS3.A) and energy transfer "out of hotter regions and into colder ones" (MS-PS3.B), there is little language about how to connect these two. It seems absurd to suggest the use of kinetic theory to 8 th grade classrooms, but a sufficiently detailed description does seem appropriate and necessary for students struggling with the idea of coldness.
In both our examples, we note that the preliminary analysis in terms of the most common incorrect idea (i.e., energy used up, coldness) does not necessarily suggest a way for teachers to address that student response. The more detailed analysis, though, suggests ways for responding to students, sensitive to the role that context plays in their thinking and using that awareness to help guide student learning.

V. CONCLUSIONS
Middle school physical science is a rich area for studying student conceptual understanding of energy. We find that using multiple questions to interpret a single concept provides us with greater insight into student knowledge, the role of context, and issues related to coherence and consistency across contexts. After interpreting multiple questions on a single topic, we are able to suggest analyses of student thinking that may prove useful for teachers who wish to address their students' difficulties in the classroom.