The “ revisiting ” strategy in physics tutorials

Physics tutorials are worksheets that guide students to answer conceptual questions. Two well-known sets of physics tutorials, the Tutorials in Introductory Physics (TIP) and Open Source Tutorials (OST), are reported to emerge from distinct conceptual change principles. For example, TIP is often associated with "elicit-confront-resolve", while OST is connected with "refining intuitions." Based on the analysis of interviews with tutorial authors and of the tutorials themselves, we looked for similarities between the tutorials. We identified a pattern that is common to both sets of tutorials, which we call the "revisiting" strategy: students are asked to return to a problem they have already examined. Elicit-confrontresolve and refining intuitions can both be examples of revisiting, suggesting that revisiting is an important and general technique that illuminates the similarity between the two sets of tutorials.


I. INTRODUCTION
Many Physics Education Research (PER) papers have studied the efficacy of instructional materials.For example, the Tutorials in Introductory Physics (TIP) [1] and the Open Source Tutorials (OST) [2] have repeatedly been shown to contribute to student learning [3].These results are valuable for those who are considering adopting TIP or OST into their classroom.They suggest that these specific materials are well-designed and beneficial.However, many physics instructors prefer to design their own materials.For example, studio physics methods such as SCALE-UP [4] often have no specific recommended activities.Published tutorials are intended to supply one hour of active learning per week, and cannot fill the five to six hours per week required by studio physics instruction.Our question is whether SCALE-UP instructors and other instructors who must design their own activities can learn anything from published tutorials.Is it possible to enhance student learning with an activity that is "like" TIP and OST even if it is not the same as TIP and OST?If so, the question remains: what does it mean to be "like" published tutorials?Instructors are given little research-based guidance about how to design activities.
We define two types of design techniques.Design "strategies" are observable techniques for designing activities.That means we can easily tell if the strategy is present in the activity just by reading the activity.Design "principles" on the other hand are closely tied to a theory of student thinking, and need not be easily observable.In this paper, we will focus more on the strategies.
In analyzing the tutorials for the benefit of future instructional designers, there are two paths we might take.First, we can try to understand the strategies that underlie the tutorials: how were they written?What do they have in common?Second, we could imagine testing which of the strategies are effective using controlled experiments that isolate one strategy at a time.This paper addresses the first approach, since we feel that the strategies structuring the tutorials require further clarification.Although some work has gone into differentiating TIP and OST [5], little has been written about strategies that are common to TIP and OST.Thus, some analysis must take place before we can think about experimental tests.Furthermore, if we can find a common theme that appears again and again in the tutorial design, it is likely that that strategy contributes to the tutorials' success.
This study is part of a larger cross-institutional effort to examine instructors' unpublished, personal course materials and identify the instructional strategies they are using.In this endeavor, we are not primarily interested in distinguishing TIP principles ("elicit-confront-resolve") from OST principles ("refining intuitions") [5].Instead, our interest lies in defining a broad category of strategies that are successful in a variety of physics activities.To this end, we also examined a number of non-tutorial activities to investigate whether our codes would appear in these activities and whether any new codes would emerge.

II. LITERATURE REVIEW
Several design principles have been described in the literature, such as elicit-confront-resolve (ECR) [6], as well as strategies such as think-pair-share [7], prediction [8], and student statements or debate problems [9].These strategies informed the development of our coding scheme.
Because SCALE-UP does not designate a particular curriculum, institutions and instructors must design their own instructional activities or adapt existing activities for SCALE-UP instruction [10].Research suggests that faculty members in fact prefer innovations that allow them freedom to make changes [11].A study of five secondary implementation SCALE-UP sites found that users mainly adapted and reinvented available curricula, such as the TIP [10].The present work is a first step in describing strategies that allow instructors to select and develop effective curricula for their courses, including, but not limited to, studio courses.

III. METHODS AND ANALYSIS
Our goal in this project was to identify and analyze instructional strategies in the tutorials.We took a constant comparative approach [12] to develop a list of strategies, with an eclectic data set that included a literature review, a set of interviews with four designers of the tutorials, a computer word count analysis of the tutorials, the tutorials themselves, and the OST design documents [2].We used an inter-rater reliability process to ensure that we agreed on the meaning of the strategies.Once this process had begun, we also coded axially, that is, finding relationships between different codes and trying to develop overarching categories.In the end, we arrived at a core category that contains all of our codes.This core category is described in the following section.
Interviews were one hour in length.We asked tutorial designers to characterize the design strategies and principles that they used to construct the tutorials.We also asked them about some specific episodes in the tutorials that we wanted to understand better.Interviews were videotaped and transcribed prior to coding.
We made a special effort to explain episodes involving ECR and refining intuitions, as these features are central to the tutorials.ECR has been described as a key feature of TIP in publications [6], and the TIP designers we interviewed largely confirmed this, although we had the sense that refining intuitions is more central to OST than ECR is to TIP.One participant said: "Well obviously elicit-confront-resolve is used pretty frequently [in TIP].Not as frequently I think as is commonly believed.But, … if you know or suspect that a learner is going to make a certain error because there's … some tendency, or some association they're gonna make, or some 'misconception' they're gonna have that's not being brought out by conventional teaching … if they're not brought out, if they're not elicited, then they'll justthe tendency will continue to be there, even if it may not be a mental model or a theory or a misconception."The importance of refining intuitions in OST was underscored by a tutorial designer who said that "The biggest design principle [for OST] was to help students frame their activity as building on rather than abandoning their intuitions," and that "refining intuitions and reconciliation … that's one of the core design principles." We performed a computerized word count of the Maryland and Washington tutorials, using the loglikelihood method [13] to find words that were characteristic of the tutorials as opposed to a set of cookbook labs [14].Words that appear significantly more frequently in the tutorials are highlighted by this method, such as the word "prediction."OST contains words related to working "individually" vs. in a "group," as well as "mistake," as in "mistake-catching." An open coding of the OST design documents, where the designers describe several strategies and design principles, generated codes such as prediction, mistakecatching, competing arguments, consensus, checking for consistency, intuition refinement, and debate problems.
Finally, we coded the tutorials themselves.To do this, we started with codes that emerged from the approaches described above.Researchers analyzed several tutorials independently using this list of codes, then compared their results to see if they were in agreement about the interpretation of the codes.Our initial codes were: (1) prediction, (2) checking for consistency, (3) statements by fictitious students (where students must agree or disagree with a statement attributed to a fictitious student), and ( 4) checking their answer with the instructor.One of our goals in developing codes was that when multiple coders analyzed a given document, even if we did not always agree, we should be able to easily come to agreement through discussion.With our initial codes, agreement did not always come easily.

IV. OUR CORE CATEGORY: REVISITING
After this process of analysis, we arrived at a core category [12] that summarized all of the codes in our final code set.This core category is "revisiting."Revisiting means that the tutorial asks an initial question, then addresses the same question a second timewhether by giving a hint, having students perform an experiment to check their answer, having students work individually and then answer the same question again as a group, having the students check their answer with the instructor, or by simply telling the student the answer.Thus, revisiting will involve an "initial question" followed by a "revisit."The "revisit" literally addresses the same question, not a different question on the same topic.(The latter would be an extension of the initial question, not a revisit.) Revisiting is related to ECR in that the "elicit" is the initial question, while the "confront" addresses that question a second time.It is related to refining intuitions because, in effect, intuitions must be elicited before they can be "refined."Thus, there is always a multi-step process of answering and re-answering a question.In fact, both TIP and OST authors agreed that the "elicit-confront" structure is important.An OST author said that students should "be acutely aware and maybe even emotionally invested in … this really stark apparent conflict between intuition and the formal result, but [they should] then interrogate what to do about that kind of conflict … as opposed to … discard[ing] the intuition.'… You could call it 'elicit-confrontreconcile.'" Thus, the authors disagreed about whether to focus on building an "intuition" or discarding a "tendency," but agreed on the basic "elicit-confront" structure.Elicitconfront is a common form of revisiting; the initial question is the "elicit" and the revisiting question is the "confront." Another function of revisiting may be that when students answer a question multiple times, their multiple viewpoints can motivate argumentation.Schwarz [15], citing Howe [16], writes about hypothesis testing, i.e. prediction: "When students worked in small groups and were asked to 'reach consensus' on a problem in physics, hypothesis testing led to conceptual change.A quantitative analysis of the dialogues showed that change was accompanied by productive argumentative activity." Third, revisiting is sometimes used in OST to show an agreement, rather than a conflict, between two approaches.For instance, OST tutorial 3 asks three times "does Newton's second law agree with your answer?"This is revisiting, since the student reconsiders the question in light of Newton's second law.In each case, the answer is meant to be "yes" because the student has (hopefully) been guided to the correct answer by this point in the tutorial.This might be characterized as "elicit-confirm." Once we developed the idea of revisiting, we also went beyond the published tutorials and briefly examined a few more activities, including ISLE [17], RealTime Physics (RTP) [18], Workshop Physics (WP) [19], and two cookbook labs.This gave us some assurance that the idea of revisiting is meaningful apart from the two published tutorials.

V. CODING SCHEME
In this section, we describe our final coding scheme involving different types of revisiting, which we developed as discussed in Section III.The codes are as follows: Prediction/Experiment (PREX): The student predicts the outcome of an experiment, then performs the experiment.Thus, the revisit is the experiment.
Statements to Agree or Disagree with (SAD): Involves one or more statements, often attributed to fictitious students.The student has to agree or disagree with them or find the error.The statements may be the initial question or they may be the revisit.
Revisit with Reasoning (RWR): Any revisiting pattern that requires student reasoning and doesn't fit the other strategies.A common wording for this strategy would be: "is your answer consistent with …" Procedure (PROC): Any revisiting that uses cookbookstyle procedures, such as plugging numbers into a formula.
Think-Pair-Share (TPS): The activity instructs students to work independently, then share or compare their answers.The sharing or comparing is the revisit.TABLE I: Relative use of each code in the two sets of tutorials.The number given is the ratio of the incidence of the given code to the incidence of all codes.The (*) means that the code was invented during the IRR process.Some codes did not appear in tutorials, only in labs.Checking With the Instructor (CWI): Checking an answer with the instructor is the revisit.

Code
Telling the Answer (TELL): In the revisit, the activity tells the answer to the question.
Printed Answer (PRIN): In the revisit, the solution is given as a separate printed document, such as a photograph.
Note that PROC and SYM did not appear in TIP and OST (PROC appeared in labs, and SYM in ISLE.) To obtain inter-rater reliability, we applied these codes to three OST tutorials, five TIP tutorials, and one lab each from ISLE, Workshop Physics, Realtime Physics, and two "cookbook" labs.The activities were selected at random.For each section of each tutorial, we marked whether each code was present or not, counting the code as being "present" if that item appeared one or more times.In this way, we coded 10% or more of each tutorial set and we had at least 50 code units, as recommended by Lombard [20].We computed Cohen's Kappa using the method of Smith [21] and obtained a kappa value of 0.79, which is on the high end of Landis and Koch's "substantial" category [22].A few codes (PROC, SYM, and TELL) were developed while checking inter-rater reliability (IRR).Thus, their IRR could not be measured.We have indicated with an asterisk (*) in Table 1 that we are less certain about them.
We then recoded all of the tutorials that we had practiced with to develop the coding scheme, for a total of ten OST and eleven TIP.Table I depicts the code counts for each tutorial in terms of the fraction of all instances that came from each strategy.Thus, the number 0.21 for RWR means that 21% of all TIP codes were RWR.On average, OST contained 6.9 revisiting codes per tutorial and TIP contained 3.0, echoing our participant's suggestion that ECR is not as central to TIP as people think.The participants suggested that other principles are important in TIP; an example would be "model building."We hope to discuss these principles in a future paper.We also tried counting the total number of each code rather than the number of sections in which each code appeared; but OST still contained twice as many codes as TIP.(12.2 vs. 6.1 codes per tutorial.)The table shows that the most common codes (PREX, RWR, and CWI) have similar relative incidence in the OST and TIP tutorials we examined.
In the data from the ISLE, RTP, and WP labs, codes were predominantly PREX and RWR, except for two SYM codes in ISLE and one TPS in WP.The two cookbook labs contained three PROC codes between them and a single instance of PREX.We examined only a few non-tutorial activities; further research would be needed to fully understand them.

VI. CONCLUSION AND FUTURE WORK
We found that "revisiting" is a common strategy in TIP and OST.The revisiting framework may be useful for cross-institutional comparisons of tutorials and activities for two reasons: first, it is straightforward; it focuses on features of documents -"revisits"that can be identified reliably.Second, it is adapted to both sets of tutorials and, tentatively, other activities as well.Furthermore, revisiting is a dynamic category: we are able to recognize new types of revisiting that aren't found in the tutorials, such as ISLE's "symposium." A limitation of the notion of revisiting is that revisiting may not be effective unless it corresponds to students' intuitive ideas or difficulties.The concept of revisiting also raises many questions: is argumentation a common learning mechanism for all revisiting strategies?How can we know which type of revisiting is suitable in any given case?Does revisiting have a different purpose in different activities?In tutorials without revisiting, what strategies are used instead, and how are students able to correct themselves if they don't understand the concept on the first try?To what extent is revisiting found in non-research-based materials?Future work can also investigate whether all revisiting strategies are equally effective.
In ISLE, students walk around to different tables and ask what the other tables found out.Talking to the other tables is the revisit.