Investigating causal inference difficulties with a simple, qualitative force-and-motion problem

.


I. INTRODUCTION
Understanding the nature of causality is a key component of conceptual understanding in science [1][2][3][4]. diSessa and Sherin [1] proposed that part of conceptual understanding is knowledge that allows one to draw causal inferences from information read out from physical situations. We investigated the hypothesis of whether certain types of causal inference are more challenging than others. If correct, particularly challenging causal inferences may provide a unified explanation for different conceptual difficulties across physics content areas.
A physical system's causal structure determines the types of causal inferences that can be made. The simplest, interesting causal structure is a common-effect structure where two independent causes impact one effect. Consider this experimental situation: a cart with an attached sail is placed in a wind tunnel where the fan blows the wind, the force of the wind acts on the sail, and the cart accelerates. The causal diagram representing this system ( Fig. 1) indicates that the force of the fan and the mass of the cart are two independent causes that impact an effect, the acceleration of the cart. Fig. 1 also indicates that each cause is linked to a source object, either the fan used or the cart used. This system can be altered by intervening on the source objects (e.g., the fan can be changed to increase the force of the wind on the sail). Although there are other potential source objects that could be altered to affect the causal factors, here we work with a simplified system. For this system, there are three types of causal inferences that can be made from changes in the system (Table I): • cause-to-effect (CE) inference: a change in a cause (force or mass) is used to infer whether there is a change in the effect (acceleration). • effect-to-cause (EC) inference: a change in the effect (acceleration) is used to infer whether there is a change in a cause (force or mass). • cause-to-cause (CC) inference: a change in one cause is used to infer whether there is a change in another cause.
For independent causes, changes in one cause will not affect the other cause and recognizing this independence between causes is part of physical conceptual understanding. Errors in making correct cause-cause inferences may describe a subset of conceptual difficulties seen in physics education. The classic Newton's 3 rd law error of inferring that, in a collision between two objects, a less massive object will feel a larger force [5,6] can be interpreted as an incorrect cause-cause inference: using information about the difference in masses to infer a difference in forces. Redish and Kuo [7] discussed an introductory E&M problem where a test charge in an external electric field feels a force. Students commonly answered incorrectly that changing the test charge would change the external electric field, even FIG. 1. The causal structure of the sail-cart situation. The + sign indicates that if the force increases, acceleration will increase. The sign indicates that if the mass increases, acceleration will decrease. though the test charge and the external electric field are independent causes that impact the force on the test charge. In a previous study [8] a question about a spring attached to the ceiling being stretched out by a hanging block was used. A common error was inferring that increasing the mass of the block changed the "spring stiffness," even though the weight of the block and spring constant are independent causes that impact the distance stretched. Motivated by these examples, the current study focuses on CC inferences, testing if these are easier or more difficult than CE and EC inferences.
Some conceptual difficulties in physics can also be modeled as the ability to recognize and integrate the presence of alternative causes in causal inference [9,10]. In a CE or EC inference, information about an alternative cause can affect the normative inferences that can be made. For instance, if the fan is replaced with another fan to increase the force of the wind, the acceleration may not increase, since mass could increase to offset or overcome the impact of the changing force. Similarly, if the acceleration increases, one may not be able to infer that the force of the fan increases since the change to acceleration may be due to a decrease in the mass. Therefore, this study also investigates the impact of different alternative cause information on the difficulty of causal inferences.

II. RESEARCH QUESTIONS
This paper investigates two aspects of qualitative physics problems that may impact the difficulty of the causal inferences required. The first is the type of causal inference asked: CE, EC, or CC. Here, we focus on whether CC inferences are easier or more difficult than CE and EC inferences, since, as discussed earlier, they may describe many known conceptual difficulties in physics.
The second aspect under investigation is alternative cause information. This study includes causal inferences when the alternative source that determines the alternative cause is explicitly held constant or is explicitly allowed to vary. The question is whether one of these alternative source conditions is more difficult than the other. We also investigate two competing hypotheses of how people make causal inferences when information about the alternative source object is left ambiguous. One hypothesis follows from the pragmatic, conversational assumption that all relevant information is provided [11]. From this conversational pragmatics perspective, the absence of explicit information about an alternative source could lead people to assume-implicitly or explicitly-that this alternative cause is held constant. A second hypothesis follows a more formal logic perspective: an unmentioned alternative source could vary, leading one to the assumption that the alternative cause status is unknown, so there is no way to make a definite causal inference. This difference in perspectives is perhaps why students can view some questions as "trick questions," as students may make pragmatic conversational assumptions while they are graded from a logical perspective.

III. METHODS
The data collection and analysis plan were preregistered on the Open Science Framework [https://osf.io/95g72/]. Participants were recruited online from Amazon Mechanical Turk and were all located in the US. Data were collected until 900 participants successfully completed the study and the embedded attention and verification checks. Two participants who failed to complete the study were included in data collection by error. They were excluded from the analysis, leaving N = 898 participants.
The experimental design was a 6 (causal question) x 3 (alternative source information) between-subjects design, with ~50 participants in each condition. Each participant answered one inference question about the sail-cart. We label the causes and effects using subscripts-for example, EaCm is the EC question where information about the acceleration (a) is used to draw an inference about the mass (m). There are 6 causal question permutations: two CE questions (CFEa, CmEa), two EC questions (EaCF, EaCm), and two CC questions (CFCm, CmCF). There are also 3 alternative source information types: Explicitly Constant, Explicitly Unknown, or Ambiguous.
Participants first read a brief passage about how force (F) and mass (m) can impact an object's acceleration (a). Then, participants were presented with a sail-cart question. Table  II provides the full question text for one CE, EC, and CC question, with three possible alternative source information statements. Each question included: • An introduction laying out the physical situation of a sail-cart accelerating in a wind tunnel. • A change to the system-an increase to either the force, mass, or acceleration. • Alternative source information in either the explicitly constant, explicitly varying, or ambiguous condition. • A question asking participants to infer how another factor of the system changes: is it greater, the same, less, or indeterminate? In a second trial, the sail-cart's acceleration was 3 times greater than before.
In a second trial, the fan was replaced with another fan that exerts 3 times more force than before.

of 3 Alternative
Source Information Options

Explicitly Constant
The cart was kept the same as before.
The fan was kept the same as before.
The cart was kept the same as before.

Explicitly Unknown
It's unknown if the cart changed or was kept the same as before.
It's unknown if the fan changed or was kept the same as before.
It's unknown if the cart changed or was kept the same as before.

Ambiguous None None None
Inference Question In this case, the acceleration of the sail-cart would be?
In this case, the mass of the cart would be?
In this case, the mass of the cart would be?
Response Choices Greater than before; Less than before; The same as before; It cannot be determined from the information given. Table III shows the normative responses given Explicitly Constant and Explicitly Unknown alternative source information. For the Explicitly Constant condition, the normative response corresponds to the direct connection between factors, as shown in Figure 1. In the Explicitly Unknown condition, the normative response is always "it cannot be determined" (denoted with ?), because when it is unknown whether the alternative source changes, definite qualitative inferences about a target factor cannot be made.

A. Coding responses
For example, the CFEA question states that the fan is replaced such that the force of the wind increases and asks about the acceleration. In the Explicitly Constant condition, the cart stays the same, so the mass is held constant, and an increase to force means the acceleration will be greater than before (>). In the Explicitly Unknown condition, it is unknown whether the cart stays the same or changes, so the mass can vary, and it cannot be determined what happens to acceleration (?). The other responses are less than before (<) or the same as before (=).
For each of the 18 different conditions, responses were coded into these three categories: normative for explicitly constant, Normative for explicitly unknown, or other responses. For the Explicitly Constant and Explicitly Unknown conditions, we are interested in how frequently participants provide the normative response for the question given. For the Ambiguous condition, conversational or logical assumptions both be appropriate depending on context, so we do not label one response as being normative. Rather, we are interested in whether the distribution of responses aligns more with that for the Explicitly Constant condition (in line with conversational assumptions) or the Explicitly Unknown condition (in line with the formal logic perspective).

IV. RESULTS
We aggregated our results from 6 causal questions into the 3 causal inference types: CE, EC, and CC. Although there were some differences between questions within causal inference type, they were not germane to the overall results here.  Figure 2 shows the proportion of normative responses for the Explicitly Constant and Explicitly Unknown conditions. We performed a 3 (causal inference type: CE, EC, CC) x 2 (alternative source information: explicitly constant, explicitly unknown) logistic regression using log odds-ratio of normative responses as the dependent variable. CCexplicitly constant was chosen as the reference group because of our focus on whether CC inferences are easier or more difficult CE or EC inference.  Table IV shows the results of the logistic regression model. Odds ratios greater than one indicate more normative responses (and odds ratios less than one indicate fewer normative responses). The significant effects of CE and EC indicate that, in the Explicitly Constant condition, CE and EC inferences had more normative responses than CC inferences. CC inferences have greater rates of "other" responses (as shown in Fig. 3)-such as, increasing the force of the fan changes the mass of the cart, even though the cart used is explicitly stated to be kept the same.

A. Does the type of causal inference or explicitly constant / explicitly varying alternative source information impact the difficulty of causal inferences?
The significant effect of Explicitly Unknown indicates that, for CC inferences, Explicitly Unknown questions had fewer normative responses than Explicitly Constant questions. This appears to be due to responses on explicitly unknown questions that say that a cause stays the same, even though it is explicitly unknown if its source object changes (as shown in Fig. 3)-such as, responding that the mass of the cart stays the same even though it is unknown whether the cart used changes or stays the same. The significant interactions of CE x Explicitly Unknown and EC x Explicitly Unknown indicate that the difference in normative responses between explicitly constant and explicitly unknown is greater for CE and EC inferences than for CC inferences.

B. Is ambiguous alternative source information treated
as constant or unknown? Figure 3 shows the distribution of responses for ambiguous questions as compared to explicitly constant and explicitly unknown ones.

V. DISCUSSION
In the current study, we investigated how different aspects of causal inference questions impact how easy or difficult normative inferences are. One aspect was the type of causal inference asked: CE, EC, or CC. For explicitly constant alternative source information, we found that CC inferences here are more difficult than CE or EC inferences. It appears that people sometimes believe that changing one independent cause can impact another.
Another aspect of causal inference questions was alternative source information. Participants had more difficulty answering explicitly unknown alternative source questions than the explicitly constant questions. The relatively high rates of "normative if constant" responses on explicitly unknown questions is consistent with other research showing that people often fail to integrate potentially varying alternative causes into their causal inferences, on both physics problems [9,10] and in other contexts [12][13][14]. Why might this be? It is possible that this statement is ignored or that people did not know what to do with this information. As a part of this study, we have also collected participants' open-response explanations of their reasoning and asked them to explicitly state their assumptions about the alternative source. In future work, we will use these responses to evaluate possible explanations.
Further, we found that ambiguous alternative sources were treated as if they were explicitly constant, consistent with the conversational pragmatics hypothesis. In future analyses we will also use the open-response explanations to differentiate whether participants make an explicit assumption that unmentioned alternative causes are held constant or whether they neglect those unmentioned alternative causes.
The overall premise of this program of research is that systematically conceptualizing physics questions according to causal inference type, and understanding the causal and pragmatic reasoning used on these questions, may provide a unified explanation for conceptual difficulties across physics content areas. Future work can investigate whether the pattern of findings here would be found for other physical topics, situations, and questions. If difficult causal inferences do underlie a subset of conceptual difficulties, developing physics instruction around causality, causal inference, and pragmatic assumptions may improve conceptual understanding across different content areas.