Comparing student behavior in mastery and conventional style online physics homework

We compared students’ learning behavior when completing identical online calculus-based physics home-work assignments organized in two ways. One was designed for mastery learning where content is divided into smaller units, and students are required to attempt the assessment once before accessing the content. Students can proceed to the next unit after passing the assessment either before or after studying the content. The second is a conventional design in which students ﬁrst study a set of instructional materials equivalent to several mastery units then complete multiple assessment problems at once. Our major ﬁndings are: 1. in the mastery condition, students solved more problems correctly either immediately after studying the instructional content, or on attempts before accessing the instructional content; 2. for students who solved similar numbers of problems correctly, the mastery condition students spent signiﬁcantly less time studying compared to the traditional condition students; and 3. students who did not pass mastery units on their initial assessment attempts spent similar amounts of time studying as traditional condition students.


I. INTRODUCTION
Almost every conventional physics textbook is organized by chapters, with each chapter covering all of the concepts related to a general topic such as mechanical energy or linear momentum, and a considerable number of homework problems appended to the end of each chapter.In a traditional class, students' mastery of content knowledge is assessed by either a quiz or an exam after finishing one or several chapters.While recent development of online learning platforms enables instructional content to be easily disseminated through the Internet, most online courses are still organized in the traditional "instruction -homework -quiz" style [1,2].
An alternative to this conventional style of instruction, mastery learning, as proposed by Bloom [3] and Keller [4], has been adopted by instructors across multiple disciplines [5].In mastery learning, the conventional chapters are broken down into a sequence of smaller units, where each unit focuses on developing a single skill or concept.After initial instruction, students who could demonstrate mastery of the concept or skill on the assessment can advance to the next unit, while those who could not are allowed more time to study and are provided with additional scaffolding until mastery is achieved.By enabling students to spend different amounts of time and effort on learning, it is proposed that significantly more students in a mastery learning condition will achieve the desired level of mastery.There have been several recent attempts at implementing mastery style online homework in introductory physics courses with encouraging outcomes [4,6,7].
To what extent does student learning behavior differ when presented with the same instructional content organized either in a traditional textbook style or a mastery learning format?In particular, do introductory physics students allocate different amounts of time to achieve mastery in a mastery learning condition, as intended by Bloom and Keller?Advancements in online learning technology enable researchers to answer these questions by providing detailed information on student learning behavior via clickstream data.The current study explores the answer to these questions by organizing the exact same set of online instructional materials into two different versions, one according to the principles of mastery learning, and the other resembling a conventional textbook experience as much as possible.These instructional materials (text, practice problems, and assessment problems) were assigned as homework to two sections of the same introductory physics course taught in a traditional lecture format.By comparing the students' learning behavior under the mastery and traditional conditions, we test the following three hypotheses developed directly from the principles and hypotheses of mastery learning that more students will achieve mastery by spending different amounts of time on learning: H1 More students will answer more assessment problems correctly in the mastery condition than in the traditional condition.H2 For students who answered similar numbers of assessment problems correctly, the time spent on learning will be more uniform in the traditional condition, and more diverse in the mastery condition.In other words, some students in the mastery condition will spend significantly less time while others will spend significantly more time.H3 In the mastery condition, students who pass more units on their initial assessment attempts spend less time learning than those in the traditional condition, while students who fail more units on initial attempts spend more time.

II. STUDY DESIGN AND METHODS
We employed a flipped online AB experiment design similar to two earlier studies [8,9].Two versions of online homework were created for two chapters of a calculus-based introductory physics course, covering the topics of mechanical energy and linear momentum.The two versions of the mechanical energy homework were each assigned to one of two large sections of the course, both having about 220 students and being taught in a traditional lecture style.Each section was then assigned the other version of the homework for the linear momentum chapter.This design ensures that if similar differences between the two versions were observed for both chapters, they are not due to extraneous differences such as instructor and student population between the two sections.Students were instructed to complete each set of homework over a period of two weeks for course credit, during which the content was covered in lecture.No additional homework was assigned to the students during the four-week portion of the class involved in this study.

A. Creation of Mastery and Traditional Learning Conditions
The mastery style (labeled Mastery throughout the paper) online homework consists of sequences of Online Learning for Mastery (OLM) modules [10][11][12][13], implemented on the Obojobo platform developed by the Center for Distributed Learning at the University of Central Florida (UCF) [14].Each OLM module contains an assessment component (AC) and an instructional component (IC).In the Mastery condition, each module focused on one concept or skill, with the AC of the module containing 1 to 2 assessment problems focused entirely on the skills being assessed in the AC.The IC of each module contains 1 to 3 instructional pages with text, figures, and practice problems that provide immediate feedback and do not count toward the final grade.Each student was allowed 5 attempts on the AC but required 1 attempt before being allowed to access the IC.The justification for this requirement is that most students were introduced to the contents either during lecture or in high school courses, which serves as initial instruction in the original mastery learning design.To proceed to the next module, a student must either correctly answer all questions on the AC during any of the attempts or use up all 5 attempts.If a student passes the AC on the initial attempt before accessing the IC, they are encouraged to skip the IC and directly proceed to the next module.
The mechanical energy chapter contains 10 OLM modules, while the linear momentum chapter contains 9 modules [15].
The traditional style (labeled Traditional) homework was created by re-organizing the exact same contents of the OLM modules.Each sequence of Mastery modules was combined into three Traditional modules, with each Traditional module containing the same content of 3 -4 Mastery modules, including 3 -6 pages and 3 -7 AC problems.To minimize extraneous differences, the Traditional modules were also delivered via the Obojobo platform.The most significant difference was that students were first presented with the IC and not required to make an initial attempt on the AC.They were also given 5 attempts on the AC and must either answer all problems correctly or use up all the attempts, in which case the score of their highest attempt was recorded to the gradebook.In both conditions, a student received an isomorphic set of AC problems on their first 3 attempts, after which the 1st and 2nd set was presented for the 4th and 5th attempts.If a student fails the AC on the initial attempt and passes after studying the IC, we can attribute the knowledge gain solely to the IC.The symbolic solutions to the first set of problems were given as practice problems in the instruction but the final number for the answer was not given.

B. Measuring Learning Duration and Correct AC Responses
Under both conditions, most students visited the IC in a given module a single time.In 4% of the cases in the Mastery condition and 8% of cases in the Traditional condition, a student made a second visit to the IC of a module that was at least 50% as long as the longest visit and longer than 60 seconds.In those cases, the two visits were considered as a single visit and the attempt in between was neglected from the analysis.For the even rarer case of a student making 3 or more visits to the IC that are similar in duration, only the two longest visits were combined.We refer to either the single visit or the combined visit as a student's Major Study Session (MSS) for that module, and the duration of a MSS is used as the measure of students' learning time on that module.
In this study, we use the number of AC problems answered correctly either before or immediately after studying the IC as a proxy for a student's level of knowledge mastery.Specifically, we count the number of correctly answered problems on both the first attempt after the MSS and the first or second attempt before the MSS, if they exist.Students in the Traditional condition were grouped based on the number of correct AC responses on each module and were compared to students in the Mastery condition who correctly answered similar numbers of AC problems on the corresponding Mastery modules.

III. RESULTS
To examine H1 -H3, we compare the number of correctly answered assessment problems (N p ) before or immediately after a student studies the IC, and the duration of students' MSS (T s ), for each Traditional module.If a student was in the Mastery condition, their N p and T s is the sum of N p and T s from the modules that were combined into a corresponding Traditional module.
For H1, we compared the number of students with different amounts of N p on each Traditional module in both conditions.In Figure 1, we grouped students from low to high into three tiers of N p for each module, representing the size of each group using the area of a dot.The N p groupings were chosen such that each group on each module contains roughly 1/3 of the students.For all six modules, we consistently observe that more students in the Mastery condition have higher N p than their Traditional peers.Mann-Whitney U (MWU) tests on the population distributions of N p revealed significant differences between the two conditions in all but the final module, as shown in Table I.
For H2, we first plotted the distribution of T s on a log scale for each N p group on each module, an example of which is shown in Figure 2. In all 18 cases, we found that the T s distribution for students in the Mastery condition lies to the left (lower) of the distribution of students in the Traditional condition (Figure 2).In contrast to the stated H2, we did not find a separate group of students in the Mastery condition having a higher T s than the majority of students in the Traditional condition.We show the mean T s duration for each group on each module in Figure 1, represented by the y-coordinate of each dot.In most of the cases, the mean T s of the Traditional condition is significantly higher than that of the Mastery condition.The p-values from the MWU tests between the two conditions for each N p group in each module (adjusted using Benjamini and Hochberg's false discovery reduction for multiple comparisons [16]) shown in Table II confirm that most of the differences were significant, except for all N p groups on Module 6 and a few other cases.On Modules 2 and 6, the differences are greater for the low N p groups, whereas for Modules 3, 4, and 5, the difference is greater for high N p groups.In addition, we tested if students' T s is a linear function of the N p grouping on any module.ANOVA results [17] showed that there is a significant linear relation for the Mastery condition on Modules 1 (F = 9.8, p < 0.01) and 2 (F = 6.3, p < 0.01).
For H3, we identified a subgroup of students under the Mastery condition who did not pass any of the OLM modules corresponding to a given Traditional module on their first or second attempts before accessing the IC (labeled NoPassin2).If H3 were true (i.e.being able to pass the AC on attempts  FIG. 2. Major study session durations for students in the mastery and traditional conditions, with a subset of mastery students who did not pass before two attempts. before studying the IC is the major cause for some students in the Mastery condition to spend less time studying) then the NoPassin2 group would have similar or higher T s than the Traditional group.As shown in Figure 1 and confirmed by MWU tests in Table II, the NoPassin2 group has similar T s as the Traditional group (matching N p ) with few excep-tions.The most significant exception is on Module 2, where both low N p NoPassin2 groups have significantly lower T s , and the highest N p group has higher T s than the Traditional group.In some cases, such as on Module 5, too few students were in the NoPassin2 group to provide reliable statistics.

IV. DISCUSSION
In this study, we examined H1-H3 by analyzing students' interaction data and comparing N p and T s between the two conditions.We found strong evidence to support H1 by showing that more students in the Mastery condition answered more AC problems correctly on every module.We suggest several possible explanations for this observation.First, most students in the Traditional condition chose to make their first AC attempt after studying the IC, while students in the Mastery condition were forced to make an attempt before accessing the IC.This resulted in students in the Mastery condition taking one more attempt at each AC problem than the majority of students in the Traditional condition.Future analysis will explore whether H1 is still true when we allow problems answered on the 2nd attempt after study to be counted as correct on the Traditional modules.However, this advantage may be less significant than expected; a separate analysis involving similar OLM modules found that many students consistently submit an answer in a very short amount of time on their initial attempt [11].Second, the required initial attempt in the Mastery condition may have a Preparation for Future Learning (PFL) effect [18], allowing students to focus on information in the IC that is directly related to passing the AC, making learning from the IC an experience similar to Just-in-Time Teaching [19].Also, it may be more cognitively demanding and emotionally stressful for students in the Traditional condition to correctly solve multiple problems at once, compared to focusing on 1 -2 problems at a time on each Mastery module.
It must be emphasized that we cannot conclude that students achieved greater mastery of knowledge in the Mastery condition because the assessment problems were not administered under the same setting.A separate common post-test administered in a proctored setting is required to assess the learning outcomes from both conditions.
Our results did not support H2, as we found that students in the Traditional condition consistently spent more time on the IC.Remarkably, T s was not correlated with N p on any of the modules in the Traditional condition, whereas there was a significant positive correlation for Module 1 and Module 2 in the Mastery condition [20].
We believe that the main reason for the difference in study time, as discussed below in our analysis of H3, is that students in the Mastery condition can skip a module if they pass the AC on their initial attempt, which saves a significant amount of time.It may also be possible that the PFL effect mentioned above helped some students to be more focused and efficient in studying the IC.However, there may also be a technical reason for this difference.The IC of a Traditional module is several times longer than an OLM module, making it more likely for students to get distracted or take short breaks when studying the IC.Some of those breaks can be identified by our data processing scripts, but others remain indistinguishable from regular study behavior, contributing to longer T s for the Traditional condition.While it may seem that students in the Mastery group solved more problems with less study time, it is worth pointing out that, by studying the IC more thoroughly, students in the Traditional condition could have gained knowledge or ability that was not assessed in the AC of the module.Better designed assessments are needed in future studies to assess more implicit gains, such as the ability to transfer [12].
Finally, H3 is partially supported by the fact that the T s for the NoPassin2 group is similar or in some cases even higher than that of the Traditional group.This suggests that when students in the Mastery condition did not pass the OLM module on their initial attempt, they tended to spend similar amounts of study time as those in the Traditional condition.One noteworthy exception is on Module 2, where the low N p group still showed a significant difference between NoPassin2 and Traditional students.In addition, on Module 2, T s is highly correlated with N p for students in the Mastery condition, especially for the NoPassin2 group, but not for those in the Traditional condition.Understanding the implication of this exceptional observation requires a substantial amount of future analysis and even follow up study design.
As an exploratory effort in comparing student behavior in an online mastery learning design with a more traditional design, this study provided valuable insight, while leaving several caveats that must be addressed in future studies.First, we focused on a single aspect of student behavior: study time.While study time is at the core of mastery learning, there can be important differences in other behavioral aspects, including how students distribute work over time and how many practice problems they attempt (i.e. the "doer" effect [21]).Also, while most students in the Mastery condition directly skipped the IC of a module after passing on their initial attempt, some did access the IC afterwards.More careful data analysis is needed to account for those after-pass study times, and to examine their impact on the current results.
Alongside the common patterns observed across all six modules, for which we provided several possible explanations, we also reported several unique patterns that are observed only on one or two of the modules.For example, T s is highly correlated with N p for the Mastery and NoPassin2 groups only on Module 2. What unique features of Module 2 caused such a strong correlation?Answering this type of question will require conducting similar future studies on significantly more modules covering different topics in physics and other STEM disciplines.Those studies will likely provide valuable new insight into how different design choices for online instructional materials affect students' learning behavior and learning outcomes.

6 .
FIG.1.Students' major study session duration grouped by the number of correct responses in students' attempts after the major study session.Error bars show standard deviations in each distribution of study session durations.Marker size shows the number of students in each group; empty circles show the size of the group if we include students who passed the assessment before studying the IC (and therefore have no MSS duration).
pass in two attempts)

TABLE I .
Mann-Whitney U comparisons of Np distributions for Mastery and Traditional groups, including the mean Np for students in each condition.Asterisks indicate significance at α = 0.05 levels.

TABLE II .
Adjusted p-values from Mann-Whitney U comparisons of Ts for students in Mastery and Traditional conditions.