Supporting productive exploration in invention activities: are simulations too challenging?

Studies show that invention activities, where students invent a general rule from provided resources before receiving direct instruction on the target topic, are particularly beneﬁcial for learning outcomes. For most common implementations of invention activities, students are provided with instructor-designed contrasting cases with which to invent their rule. Alternatively, students could use an interactive simulation where they then have the agency to explore and collect observations on their own. While this provides a promising opportunity for developing more robust inquiry skills, it also introduces substantial challenges for the students that, in addition to learning about the domain, need to learn about expert ways of doing science. In this work, we compare different support structures that seek to mitigate these issues. Speciﬁcally, we incorporate a collaborative support structure and further provide students with either a short list of general rules to disprove or a list of important features that students are prompted to incorporate in their rule. We show that these support structures are not sufﬁcient to make the exploration of students in our simulation-based invention activities as productive as with using contrasting cases.


I. INTRODUCTION
An important goal of physics education is to provide students with opportunities to practice and eventually learn how to acquire new knowledge through scientific inquiry.This work demonstrates and evaluates different approaches to support the productive exploration of a target domain during two different types of "invention activity".
Invention activities are inquiry tasks where students invent a general rule for a certain physics concept or phenomenon by uncovering and interpreting patterns from observations.In the first type of invention activity we used, observations were provided directly to the students as a set of so-called "contrasting cases".These cases were deliberately designed to highlight the key features of the target domain when contrasted.Students constructed their general rule through a process of exploration and analysis characterized by identifying, constructing and interpreting contrasts within the provided cases.When these activities are followed by direct instruction, students have been shown to attain deeper conceptual understanding and achieve greater success in far-transfer task than with direct instruction alone [1].
In the second type of invention activity investigated, students generated their own cases from virtual experiments conducted in an interactive simulation of the target domain.The invention process was otherwise the same.This simulationbased approach has the benefit of giving students the agency to explore the domain and design experiments on their own, which has been shown to lead to the acquisition of productive inquiry skills as compared to traditional approaches based, for example, on a lecture or physical apparatus [2].
On the other hand, such an open-ended inquiry task combined with largely unconstrained interactive simulations, can also lead to unproductive exploration behaviours such as only attempting a limited range of possible experiments and confirmation bias [3].A recent study by Salehi et.al. has shown that college students working individually to invent a rule for buoyancy from an interactive simulation were more likely to miss crucial features in their exploration which affected their learning outcomes compared to students using pre-designed contrasting cases [4].These challenges are particularly concerning considering that the process of identifying the key features behind the target rule has been postulated to be a part of the core cognitive mechanisms responsible for the learning benefits of invention activities [5].
Overall, this leads to the much studied problem of designing effective guidance for inquiry learning tasks [6].Specifically, for simulation-based invention tasks, there is a need to better support productive exploration behaviours so that students can identify all key features of the domain.However, careful consideration must be made when doing so since adding too much guidance to simulations-based inquiry tasks has been shown to limit student engagement to shallow unproductive exploration behaviours [7].

II. SUPPORT STRUCTURES IN INVENTION ACTIVITIES
Approaches to design effective support structures for exploration in invention activities can be motivated by the seminal dual search inquiry reasoning model by Klahr and Dunbar [8].Their model represents the inquiry process as an iterative search of a hypothesis and experiment space.In the context of invention activities, the hypothesis space corresponds to the set of all possible rules of the target domain, while the experiment space corresponds to the set of all possible observations afforded by the activity.An invention activity with a well-designed set of contrasting cases effectively constrains the search of the experiment space to a small number of comparisons between the provided cases.These contrasting cases can also guide the search of the hypothesis space since they are designed to clearly highlight certain key features of the target domain.This suggests that support structures that constrain the hypothesis and experiment spaces without limiting a student's agency could be beneficial for simulation based invention activities.
Specifically, the cognitive burden put on a student's otherwise open-ended exploration of a simulation could potentially be reduced by: • providing a short list of possible general rules to disprove (supporting the hypothesis space search), or; • suggesting a limited set of features to investigate (supporting the experiment space search).A different but complementary approach is to support productive self-regulative behaviours such as monitoring and assessing progress, reflecting on observations and planning next steps.These metacognitive behaviours have been shown to have a positive impact on the learning outcomes of inquiry activities [9].Collaborative structures that support peer interactions on inquiry task (e.g. by working in pairs) have been shown to promote more regulative behaviours and yield more success in discovering general explanations [10].
This work seeks to determine whether using a collaborative structure combined with constraints to either the hypothesis or experiment space search is sufficient to support an equally productive exploration process in simulation-based invention activities as compared to contrasting cases based invention activities.We measure the productiveness of a student's exploration by whether or not it led to key features of the target rule being identified.

III. METHODS
To answer this question, we conducted a study with 181 students in a first-year calculus-based electromagnetism course in a North American public university.Students were randomly assigned to one of two conditions: (a) Simulation (SIM, N=91) consisting of two interactive simulation-based invention activities; or (b) Contrasting Case (CC, N=90) consisting of the same two invention activities, but with predesigned contrasting cases instead of a simulation.In both conditions, each student did two invention activities on two different topics: (a) electric potential and fields; and (b) parallel circuits.The invention activities were embedded in the course's weekly hour-long recitation and delivered the week before the target topic was introduced in lectures.This work is part of a larger study consisting of a longer sequence of invention activities.This paper focuses on the preliminary results obtained from the first two activities of this sequence.

A. Procedure and materials
The first 5 minutes of the recitation were allocated for a quick pre-test that consisted of a single question asking students for the target rule.Students then had 20-30 minutes to finish their invention activity.Afterwards, 5-10 minutes were allocated for a 3-4 questions post-test on the target topic (not discussed in this manuscript).Figure 1 provides an overview of the sequence of activities for both conditions.

CC and SIM invention activity design
Students in both conditions were required to complete a worksheet for each invention activity.The worksheet first included a short summary to familiarize students with essential background material on the domain (without revealing anything about the target rule itself).Students were then prompted to invent a general rule by comparing and contrasting as many pairs of cases as they deemed necessary to uncover the rule.As shown in Fig. 2, the worksheet also directed students to record their pairs of cases and explain how the contrasts informed the final rule.
In the CC condition, students were always provided with a set of 8 unpaired cases, each representing a different combination of important features of the domain.Students then had to create various pairings from this set to contrast and then record in their worksheet (see top of Fig. 2).
• In the fields and potentials activity, cases included the field magnitude at different locations of various charge configurations (positive and negative single charges, dipole and two nearby positive charges).All cases had several labeled equipotential lines drawn at constant potential increments.• In the parallel circuit activity, cases consisted of different simple parallel circuits with a range of key features varied such as the number of loops, resistance of resistors and battery voltages.All provided cases clearly listed or illustrated the values of the different important features and the numbers and visuals were chosen to match, as closely as possible, typical situations that could be created in the simulations used in the SIM condition.
In the SIM condition, students were asked to use an HTML5 PhET interactive simulation on the target topic (Charges and Fields PhET and DC Circuit Construction Kit PhET) [11].Students then had to create their own cases to contrast using the simulation.As in the CC condition, students were then directed to report their findings as pairs of cases in the space provided in the worksheet (see bottom of Fig. 2).

Supporting the hypothesis or experiment space search
In the first activity, students were prompted to invent a general rule for identifying the relative electric field strength at two locations based only on equipotential lines.The hypothesis space search was supported by modifying the task prompt in both CC and SIM conditions to include a short list of 6 possible general rules.Students then had to disprove the provided rules by contrasting different cases until only the correct one remained.For example, a student could disprove one of the listed general rules stating that "the electric field strength is larger where the potential is larger" by comparing the field magnitude and potential at two different location on the axis bisecting a simple dipole (the potential is fixed at zero along the bisecting axis but the field becomes weaker as one moves away from the dipole axis).
In the second activity, students invented a general rule to predict the current and voltage across a resistor in a basic parallel circuit.The experiment space search was supported in both CC and SIM conditions by modifying the task prompt to include a list of features that their rule had to take into account.These features included battery voltage, number of resistors and their resistances.While students were not explicitly told how to test for the impact of these features, this support provided obvious guidance towards exploring the key features of the domain.
All activities across topics and conditions shared the same collaborative support structure.Students worked in groups of two (and rarely three when necessary) until they converged on a final rule that everyone agreed on.Finally, two different groups would gather and share their findings.This final sharing phase provided an opportunity for students to discuss and reflect on their exploration by comparing their recorded contrasts with the ones uncovered by their peers.

IV. RESULTS
The pre-tests in each activity were scored on a binary scale: 0 (did not know or stated an incorrect general rule) and 1 (stated correct general rule).Overall exploration was quantified by the mean number of pairs of cases recorded on students' worksheets.To determine if a student's exploration was "productive", another binary measure was then established to assess whether or not the student had addressed certain key features of the target rule during their exploration.To determine this measure, the pairs of cases recorded in the students' worksheets were coded by the lead author (as explained in further detail below).
To compare the impact of each condition (controlled for success on the pre-test) on the probability of achieving a productive exploration or not (P (PE i )), the following logistic model was used for each activity: A. Incoming knowledge In both activities, there were no significant differences across conditions for the number of students that correctly identified the rule in the pre-test as demonstrated by a chisquare test of independence.Activity 1: χ 2 (1,181) = 0.29, p = 0.59; and Activity 2: χ 2 (1,181) = 0.66, p = 0.42.

B. Productive Exploration
A two-sample t-test indicates that there was a significant difference between conditions in both activities in terms of the mean number of recorded pairs of cases.In activity 1, m CC = 3.5, sd CC = 1.4 and m SIM = 2.5, sd CC = 1.1; t(179) For the first activity, it was only possible for students to disprove all of the incorrect final general rules if they explored charge configurations other than a single point charge.Specifically, with single point charges the relationship between electric field and potential is consistent with incorrect rules such as "larger field strength always occurs where the potential is greater".A successfully productive exploration was therefore attributed to a student if they recorded contrasts of cases with multi-charge distributions.
In the second activity, however, there is no such obvious criterion.Given that students were already prompted to investigate a list of key features, it made sense to define a successful exploration as one where a student recorded contrasts that addressed each of these features.
Figure 3 illustrates the proportion of students within each condition for each different pre-test outcome that succeeded in achieving a productive exploration.Notably, there is always a larger proportion of students in the CC condition that achieve a productive exploration than in the SIM condition (regardless of pre-test or activity).Fewer students in both CC and SIM conditions achieved a productive inquiry in the second activity.The data in Fig. 3 were fit using the logistic regression model in Eq. 1 and the results are listed below: Activity 1: Students with pre = 0 in the CC condition have 26 times higher odds (95% CI = (4.4,502), p < 0.005) to achieve a productive exploration than in the SIM condition.There was no effect for pre (p = 0.49), or for the interaction of condition and pre (p = 0.96).Activity 2: Similarly, in the second activity, students with pre = 0 in the CC condition have odds 4.4 times higher (95% CI = (1.5, 15), p = 0.01) to achieve a productive exploration than in the SIM condition.There was also no effect for pre (p = 0.94), or for the interaction of condition and pre (p = 0.83).

V. DISCUSSION
Identifying the key features of a general rule is postulated to be a part of the core mechanisms behind the success of in-vention activities [5].This requires designing invention activities to support a productive exploration of the domain, which contrasting cases are ideally suited for.Our work suggests various support structures to facilitate and guide this process in interactive simulations while preserving the unique benefit of practicing valuable inquiry skills in an authentic fashion.The main result of our study suggests that, regardless of prior knowledge or the support structure used, our pre-designed contrasting cases lead to a more productive exploration than our interactive simulation activities.This result is consistent with literature on the challenges of using interactive simulation that suggest that students often fail to explore the full range of experiments afforded by the simulation [3].
In the first activity on electric field and potentials, nearly all of the students in the CC condition had a productive exploration as compared to fewer than half for the SIM condition.Given the fact that observations related to single point charges were consistent with one of the incorrect general rules provided ("the electric field strength is larger where the absolute value of the potential is larger"), it is possible that SIM students halted their exploration after seemingly proving this rule (c.f.confirmation bias [3]).Since this was the first invention activity of the semester on a notoriously challenging electromagnetism topic for first-year students, it might have been beneficial to also guide the experiment space search (as was done in the second activity) by explicitly prompting students to examine multi-charge configurations.
In the second activity, the proportion of students that achieved a productive exploration dropped in both conditions (and slightly more so for the SIM condition).This could possibly be attributed to the more accessible domain of DC circuits, which is often covered in high school.Specifically, students might have recalled the correct rules after a brief exploration or from discussion with peers, and therefore might not have felt compelled to investigate all the key features.Notably, the gap between conditions was also significantly reduced as compared to the first activity.This reduction could be potentially explained by: • a task orientation effect (student in the SIM condition became more familiar with how to use simulations in invention activities); • learning of better exploration habits (students in the simulation condition now had practice exploring simulations and thus became relatively more efficient); or • the more accessible topic of DC circuits (students in the simulation had a relatively easier time exploring features that made more sense to them) Nonetheless, the fact that the CC condition still had a more productive exploration is noteworthy, especially considering the arguably heavy-handed experiment space support provided in the second activity.
This persistent and clear difference in the exploration outcomes between CC and SIM conditions might be explained by the fact that invention activities with simulations require a large cognitive investment from students [12,13].Providing contrasting cases reduces the extraneous cognitive load of an invention activity by guiding and constraining a student's search of the dual experiment and hypothesis space.Another explanation might simply be that more time is needed for our simulation-based invention activities.Students in the SIM condition had the additional work of familiarizing themselves with the simulation, generating their own cases from the simulation and a more lengthy and effortful process of recording these cases on the worksheet (using drawings or descriptions instead of just referring to the labels of provided contrasting cases).This might also explain why students in the SIM condition recorded fewer overall pairs of cases in their worksheets.If true, this could be problematic for instructors trying to implement such activities in time-constrained recitations, where invention activities already compete for time with traditional problem-solving tasks.
Another surprising result from this study is that there was no significant impact from prior knowledge on productive exploration.According to cognitive load theory, lower prior knowledge effectively increases the "intrinsic" load due to the complexity of the domain itself [13].For instance, one would expect that a low pre-test student in the SIM condition would be more overwhelmed by the task and have a correspondingly less productive exploration.A possible explanation is that the collaborative structure of the invention activities mitigated the impact of prior knowledge.Students worked in pairs and were encouraged to seek help from their peers.Students were thus able to consult each other when they were "stuck" by suggesting new experiments to try, contrasts to analyze and possible rules from identified patterns.Without support from their peers, low-pre students might not have otherwise been able to surmount these moments efficiently.
As aforementioned, this work only focused on the first two activities of a larger study that included several more invention activities.The next step for this work will be to investigate if the differences between conditions reduced further after students had more opportunities to practice inventing with simulations.In addition, since all students from both conditions later did an unsupported simulation-based activity, it will be possible to determine which of the conditions led to the more effective adoption of productive inquiry skills.Finally, to echo comments from the similar work of Salehi et.al., these results should not be taken as an argument against simulations but rather as an invitation for further careful investigation of support structures in simulation-based invention activities [4].The efficacy of different combinations of structures that support collaborative work and the dual search of the hypothesis and experiment space should be studied in different domains and with different interactive simulations.

FIG. 1 .
FIG. 1. Overall sequence of activities during the study (top) and steps within each activity (bottom).

FIG. 2 .
FIG.2.Snapshots of student work on the second activity illustrating the worksheet structure employed across all activities.Students were guided to contrast pairs of cases (called "example pairs" in the worksheets).The CC condition contrasted any two cases from a provided set (top), while the SIM condition contrasted cases that they generated themselves in the simulation (bottom).

FIG. 3 .
FIG. 3. Proportion of students that succeeded in achieving a productive exploration in Activity 1 (left) and Activity 2 (right), in each condition and for each pre-test outcome.