Mastery learning in the zone of proximal development

Mastery-style online homework is used at the University of Illinois at Urbana-Champaign in a freshmen preparatory physics course. Its implementation revealed that some content is difficult for students to master, including algebraic kinematics questions. By adding scaffolding for half the students on a level targeting these skills, we saw that these students were able to master the training more often and significantly more quickly than their counterparts with less help, while still outperforming their counterparts on more difficult follow-up levels with less time spent overall. The advantage from the single treatment became diluted as transfer and difficulty increased.


I. INTRODUCTION
The use of math in physics problem solving is both an integral tool as well as a notorious barrier for some physics students.In particular, student difficulty with algebraic kinematic problems has been documented by Torigoe, Gladding and others, who published two studies in 2007 showing vast performance differences (up to 50%) on identical problems when numbers were replaced with variables [1][2].The expanded follow-up study showed that the performance difference was most dramatic for lowperforming students [2].
Mastery learning, often traced to Bloom and Keller in 1968 [3][4], is a delivery method that requires students to correctly answer a set of questions on a topic before moving on to the next topic, while providing corrective feedback and opportunities to retest on similar problems if students do not mastered on that attempt.Because it allows variable time to be spent per students' preparation and ability, mastery learning boasts effectiveness for both low and high performing students, which motivated the implementation of mastery-style online homework in a physics preparatory course at the University of Illinois [5].The fraction of students who were able to master each level (mastery rate, in this paper) vary across the course's assignments, and a targeted training exercise for algebraic kinematics was very difficult for students to master in 2015.This level had been previously added to directly confront student difficulty with algebraic kinematics, shown by consistently low-scores on the type of problem on their midterm.
Because this level is particularly difficult, it is a natural place to check for limits of mastery-style learning.While mastery learning has been shown to be appropriate in many instances [6], our course content occasionally challenges students beyond what they are able to master.Vygotsky's concept of the zone of proximal development (ZPD) encourages learning to take place in the space between what students can do and what they cannot do, within tasks that students can do only with assistance [7].Low mastery rates imply that content is not in the range of skills most students can do, so introducing help may move student learning into the ZPD.In 2016, we split the algebraic kinematics level into two variations with different amount of scaffolding, one with more guidance, intended to be easier to master, and one that left more work to the students themselves.

II. METHODS
The experiment was implemented in Physics 100, a class intended for students who plan to take the calculus-based engineering physics sequence but enter college without thorough preparation.They are flagged for the course via a summer diagnostic test or self-identify as unprepared, but the course is optional.Enrollment is usually around 500 students, all of whom use the online mastery-style system.
A schematic of the delivery method for online masterystyle homework is shown in Fig. 1.A single assignment consists of several levels, each of which has four versions that target the same competencies with different surface features.Students are given one of the four versions, which they attempt and submit for grading.After submission, they are immediately told which questions were correctly answered and provided narrated animated solution videos to all questions.Students progress to the next level when they answer all questions on a version correctly (or if they have seen all four versions), otherwise they have the opportunity to retest on another version after studying solutions.Their final score is the best of all of their attempts.
The two treatments were given separately by splitting all students randomly between "two courses" online, both of which were identical up to Week 2's assignment.After four identical levels in Week 2, the fifth level differed for the two groups by providing "more" or "less" scaffolding for the same problems, followed by one more level, all delivered via the mastery system.The following level, Level 6, was not addressing the same competencies as the treatment in Level 5, but used the skills practiced in the treatment as a foundation to go further, so it was used as a proxy assessment to test how well the students could use the skills, albeit on a harder set of problems.The homework assignment also had a delayed feedback problem after the mastery levels.Students randomly received one of three versions assessing varying transfer from Level 6.The delayed feedback restricted students to one final answer for each question with feedback given only after the deadline, making the problem similar to an online quiz.The structure of these levels is shown in Fig. 2.
In creating the versions of the algebra treatment level, the problems were intentionally meant to mimic the student processes required in Torigoe et al's studies and our difficult midterm problem.All versions had two accelerating objects and required students to plug one unevaluated algebraic expression into another algebraic expression.This was often solving for time in terms of variables and evaluating other characteristics of the two objects at that time to create a simplified ratio.Students who received "more" scaffolding had a strategy map included in the question prompt, which included the specific equations needed and correct signs on the variables.Additionally, the extra scaffolding encouraged students to work in terms of a single object's variables; each step had multiple choice answers only in terms of object 2, for example.The "less" scaffolding versions gave the general kinematic equations in the prompt and provided small hints along the way, but did not provide specific equations or help students determine signs in the equations.The less scaffolded versions only reminded students to put everything in terms of a single object's variables at the end, which was necessary to simplify their ratio.

III. RESULTS
Just as in the previous year, levels within the masterystyle homework varied in difficulty and student mastery rates, shown in Fig. 3.The additional scaffolding in the split treatment level (Week 2, Level 5) successfully boosted the rate of mastery compared to students without extra training.The mastery rate doubled and the time spent on the level by those with extra scaffolding was significantly less, about 20 minutes compared to 30.
Performance on Level 6 (identical for both groups but more challenging than Level 5) was also higher for the students who previously had extra scaffolding, but both groups spent comparable amounts of time on Level 6. Mastery rates, score, and time spent on levels by both groups are shown in Table 1.Students who received more scaffolding were more likely to master (16% higher mastery rate, with p<0.0001 and effect size of 0.65 ± 0.1) and had a higher final score (8.5% score increase, with p<0.001 and effect size of 0.36 ± 0.1) on the same follow-up level.
Particular questions on the assessment were differently aligned with the treatment, so Fig. 4 shows question scores on students' first attempt, as well as categorization based on similarity to the skills in the treatment level.Questions that were "like training" corresponded to writing equations, "cumulative like training" was pulling the equations together to solve the system of algebraic equations, and "stretch" were conceptual questions that were not directly addressed in the treatment.Because Level 6 had five questions and four versions, there are 20 points on the scatter plot.Note that points above the line indicate better Using this grouping scheme, Table 2 shows the quantitative effects of the different treatments, showing up in each type of question in Level 6, cited as each type's average score.The scores of the delayed feedback levels following Level 6 are also given in Table 2.The delayed feedback questions are answerable up to a week late for 80% credit, so to remove those who were not taking their first attempt seriously, the values given are average scores on students' pre-deadline answer, after students spending fewer than 30 seconds were removed.Version 1 of the delayed feedback is most similar to Level 6, then versions are successively further transfer from the level.Within Level 6, the largest difference was seen in "like" problems, with an effect size of 0.28 ± 0.1, statistically significant to p<0.005.In the delayed feedback, the largest difference is in the most closely correlated version, with effects dropping off as the versions increase in transfer from Level 6.The strongest signal, for Version 1, had an effect size of 0.45 ± 0.2 but a p-value of p<0.05.Although splitting students into six sub-groups limited our data's significance, the results are consistent with the finding that near transfer problems were most effected, also seen in Level 6.

IV. DISCUSSION
When considering the right amount of help to offer students, there are competing reasons to provide more or less support.Students who receive more scaffolding may be unable to perform well when scaffolding is removed, but students without enough help may be unable to develop the skills at all.Vygotsky's concept of zone of proximal development recommends learning to happen in areas where students are unable to do a task alone, but are able to complete it with help, but it does not give much guidance in how to find those concepts or skills.In terms of our mastery homework system, this could be evaluated by how easy a level is to master; levels that are easy for all may not push students to think critically, while levels that have low mastery rates may indicate that more help is needed.This particular study showed that students who had higher mastery rates on their training level were able to perform significantly better on a more difficult level while spending significantly less time than those with the low mastery rate, suggesting that a mastery rate of 35% is below the ZPD.
Whether or not the 70% mastery rate of the group seeing extra scaffolding is ideal is not clear, but it is better than 35%.Within Level 6 itself, the greatest difference between groups was in the most similar problems, and achievement differences became less pronounced as transfer increased.This pattern was consistent in delayed feedback levels as well, though less significant.
This suggests that levels with low mastery rates are not necessarily creating spaces for students to self-motivate, but can possibly be limiting students' learning, particularly when following levels depend on the prior levels' skills.The effect of a single treatment was diluted as problems became more removed from the original training, but the basic skills still improved students' ability to perform the assessment at all, which was already a level of difficulty higher than the training level.The fact that students spent less time overall contradicts the idea that more time with material will necessarily result in better performance.
Looking back at Fig. 3, we have many levels which have low mastery rates.The results from this study encourage us to continue to investigate the effect of mastering on students' ability to demonstrate competence on a level's skills.In our next iteration of the class, specific low-scoring levels will be converted for half the class from masterystyle into immediate-feedback questions.This will reduce the amount of time spent by students on levels with heavy computation, and we can evaluate if students' performance is affected in later levels and on bi-weekly quizzes.Our results here suggest that reducing strain on students due to very difficult mastery levels allows them to perform better, and we will be able to test that further.

FIG 2 .
FIG 2. Structure of Week 2 assignment FIG 1. Schematic of online mastery-style homework delivery structure

FIG 3 .Average
FIG 3. Mastery rates for levels throughout the semester with rates for both treatments in Week 2 Level 5

TABLE 1 .
Mastery rate, scores, time spent on Level 5 (different treatments) and Level 6 (identical for both).

TABLE 2 .
Average student performance on types of questions, categorized by varying levels of transfer