The Learning Assistant model and DFW rates in introductory physics courses

Large introductory science, technology, engineering, and mathematics (STEM) courses historically have high failure rates, and failing such courses often leads students to change majors or even drop out of college. Institutional change models such as the Learning Assistant (LA) model can influence this trend by changing institutional norms. In collaboration with faculty who teach large-enrollment introductory courses, undergraduate learning assistants (LAs) use research-based instructional strategies designed to encourage active student engagement and elicit student thinking. In this study, we use logistic regression to investigate the relationship between exposure to LA support in these large introductory courses generally and failure rates in Physics I and II specifically at University of Colorado Boulder. We find that exposure to LA support is associated with lower failure rates in introductory physics courses and that the magnitude of the relationship is larger for female and first-generation college students.


I.INTRODUCTION
Science, technology, engineering, and mathematics (STEM) departments at institutes of higher education frequently offer introductory courses that can serve up to 1,000 students per semester. Courses of this size, often referred to as "gateway courses" are cost-effective due to the number of students able to receive instruction in each semester, but they often lend themselves to lecture as the primary method of instruction. Thus, there are few opportunities for substantive interaction between the instructor and students or among students [1]. Further, these courses typically have high failure rates [2] and lead many students who begin as STEM majors to either switch majors or drop out of college without a degree [3].
In response to failure rates and other educational issues, the Learning Assistant (LA) model was established at the University of Colorado Boulder (CU Boulder) in 2001. The LA model is a model of institutional change, which seeks to impact institutional values and practices through a lowstakes, bottom-up system of course assistance by awarding LAs to faculty and courses and by empowering faculty to institute change [4]. The LA program has a variety of instantiations in thirteen departments at CU Boulder. For example, in the physics department, LAs support recitation sessions that use the University of Washington Tutorials in Introductory Physics [5]. Alternatively, LAs use instructorcreated materials in the chemistry, applied math, and mathematics departments to guide discussion. These materials are designed to support learning in the respective courses, but are not necessarily from researched curricula.
During recitations in all departments, LAs facilitate discussion about conceptual problems and they focus on eliciting student thinking and helping students articulate and defend their ideas to others. In this, LAs guide students in learning specific content, but also in developing and defending ideas-important skills for higher learning in general. LAs meet weekly with course faculty to develop deeper understanding of the content, share insights about how students are learning, and prepare for future class meetings. LAs also take a weekly pedagogy course on research-based instructional strategies and educational theory. The model for training LAs and the design of the LA program at large are aimed at making a difference in the ways students think and learn in college overall and not just in specific courses. That is, we expect the program to influence student success in college, regardless of the department in which the LA exposure takes place.
Prior research in the physics department at CU Boulder indicates a positive relationship between exposure to LAs and course learning outcomes [6]. Other research suggests that modifying instruction to be more learner-centered helps to address high failure rates [2,7]. This study seeks to further understand the relationship between the LA program and probability of student success. Specifically, we answer the following research question: How do failure rates in Physics I and Physics II at CU Boulder compare for students who do and do not receive LA support in any STEM gateway courses?

II. DATA
Data for this study come from administrative records at CU Boulder. We focus on 10 cohorts of students, those who entered the university as full-time freshmen for the first time each fall semester from 2001 to 2010 and took traditional Physics I and/or Physics II courses from 2001 to 2014. The dataset includes information for 4,941 unique students, 3,696 of whom took at least one gateway STEM course in chemistry, physics, math, or applied math with LA support. Student-level data includes information such as race/ethnicity, gender, first generation status, and an indicator for whether a student ever received financial aid. Additional variables include number of credits upon enrollment, high school GPA, and admissions test scores. We translate SAT total scores to ACT Composite Scores using a concordance table provided by the College Board to have a common admissions test score for all students. We exclude students with no admissions test scores (about 6% of the sample).
We also have course-level data including instructor and grades. The sample is limited to five instructors who taught sections of introductory physics with and without LA support. We chose these instructors to ensure that each would have students with and without LA exposure. The outcome of interest in this study is if a student failed their physics course. We define failing as receiving either a D or an F as a final course grade or withdrawing from the course altogether after the university drop date (DFW).

III.METHODS
Ideally, we would use statistical methods to provide evidence about the causal effect of LA exposure on the probability of failing Physics I or II. To do this, we would need a second group of students like the first in all ways except for LA exposure. If the two groups were not identical except for this factor, we would need data to control for any differences between the students in each group that were also potentially related to course failure.
Unfortunately, this is not the case in our data. Since we are interested in the relationship between exposure to LAs in any STEM gateway course and failure in introductory physics, we have a limited number of students who potentially did not receive LA support. This historical issue threatens the validity of the outcomes of this study. Specifically, one interpretation of decreased course failure rates could be that LA support decreased course failure. However, we could not rule out the possibility that failure rates decreased due to other factors that also changed over time. It could be that the university implemented other student supports in addition to the LA model at the same time or that the types of students who enrolled in STEM courses changed. There is no way to determine conclusively which of these (or other) factors may have caused changes in failure rates. Thus, causal estimates of the effect of LA support on failure rates would be threatened by any historic changes that occurred. This limitation is why we include only students who began their college careers from 2001 to 2010. We study students who did and did not receive LA support in at least one gateway STEM course. Due to the nature of LA support in the physics department described above. Those students who did not receive LA support all took introductory physics between the years of 2001 and 2005. Since all physics students following 2005 received LA support, including data through 2017 would exacerbate potential historical confounds, so we only go as far as 2010.
Since we are limited in our ability to estimate the causal effect of exposure to LAs on Physics I and II failure rates, we conduct a descriptive study. We begin by providing raw counts of failure rates for the students who did and did not receive LA support in STEM gateway courses. Next, we describe the differences between those Physics I and II students who did and did not receive LA support with respect to our demographic and administrative variables. Finally, we use logistic regression to disentangle some of the relationship between LA-support and course failure.
The general specification for our logistic regression can be expressed as follows Here, is the probability of failing a course and X is a matrix of controls including all the administrative variables described in the Data section and dummy variables for instructor in order to control for instructor effects.
The results from this regression are an improvement over raw failure rate measures as we control for some differences in groups, but they do not allow for strong causal claims about the effect of LA support on failure rates. Despite this, it is valuable to understand more about patterns in course failure and the relationships between course failure and LA support to make more informed decisions about the usefulness of the intervention. Table I provides the raw failure rates for all students in the sample. Recall that this table includes physics students for only five instructors. For these instructors, the raw failure rates were 17% or less. This is slightly higher than the average failure rate (10%) over all instructors during this time period. For only the instructors included in our sample, the Physics I course failure rate is 9% lower for students who receive LA support in STEM gateway courses than those who did not and 2% lower in Physics II. The weighted average of the difference in failure rates across the two physics courses is 7%.

IV. RESULTS
Although the raw data indicates that students exposed to LA support have lower course failure rates, these differences are likely due, at least in part, to factors outside of LA support. Thus, we continue our analysis by identifying demographic differences between the two groups of students. The results in Table II indicate that students who took courses with and without LA support were similar with respect to race/ethnicity, first-generation status, and financial aid status. However, students who received LA support were more likely to be female and had more credits at entry, higher high school GPAs, and higher admissions test scores. We next use logistic regression to control for potentially confounding variables. We report the results in aggregate across the two courses for ease of interpretation. However, analysis separated by course yielded consistent results across Physics I and Physics II. Table III includes abbreviated regression results. These results are presented in odds ratios with confidence intervals. Odds ratios equal to 1.0 indicate that the variable is not associated with changes in the odds of course failure. Odds ratios less than 1.0 indicate that the variable is associated with a decrease in the chances of failing, while odds ratios greater than 1.0 indicate an increased chance of failing. Confidence intervals that include 1.0 suggest results that are not statistically significant 2 .
As shown in Table III, the odds ratio estimate is 0.48 for LA exposure with a confidence interval from (0.39-0.59). Since the odds ratio in this analysis is less than 1.0, LA exposure is associated with a lower probability of failing and the relationship is statistically significant since the confidence interval does not include 1.0.  IV. Difference in probability of failure for students exposed to LAs compared to those not exposed Key contrast % Difference with LA support White males -4 to -6 White females -7 to -11 Nonwhite males -4 to -6 Nonwhite females -7 to -11 First-generation males -6 to -9 First-generation females -10 to -14 Due to the use of dummy variables, only four (of five) instructor effects are reported in Table III. The last instructor serves as a reference category 3 . The reference category is chosen randomly. Dummy variables allow for comparing multiple instructors with one regression equation. Thus, the instructor effects for Instructors 1 to 4 are compared to the reference instructor. For example, students in Instructor 1's courses are 13% more likely to fail than the reference instructor. The purpose of including the dummy variables for instructor was to identify any potential instructor effects on failure rates. Since all the instructor confidence intervals include 1.0, instructor is not a statistically significant predictor of course failure.
To facilitate a comparison to Table I and a more accessible representation of the regression results, we next express the marginal difference in the probability of failing for specific groups of students who are and are not exposed to LA support. Table IV presents the range of difference in probability of failure for students who had LA support and those who did not for six groups of students. The results are presented in ranges because of variability due to instructor. The first row indicates the difference in probability of failing between those who received and did not receive LA support specifically for students who are male, white, not first-generation college students, do not receive financial aid, have the mean number of credits at entry, HS GPA, and admissions test scores.
These results indicate that the difference in probability of failing for these students who did and did not receive LA support ranges from -4 to -6%. In other words, these students who received LA support are 4-6% less likely to fail their introductory physics courses than those who did not receive LA support. The remaining rows in Table IV indicate other groups of students that differ from the first by only the indicated key contrast. We present these groups because they represent students from groups traditionally underrepresented in the sciences. In summary, the results in Table IV suggest that LA support in STEM gateway courses is associated with 4-15% lower probability of failing introductory physics depending on the instructor and group of students. The difference in failure rates is consistently larger for females as well as first-generation students. Further the relationship between LA support and lower probability of failing is strongest for first-generation females. This is consistent with prior research on studentcentered instruction in introductory science courses [8].
Additionally, these adjusted associations vary as a function of high school GPA. The results in Table IV only apply to students with average high school GPAs. The relationship between probability of failure and LA support is stronger (by a magnitude of 2-4%) for students with high school GPAs one standard deviation below the mean than those students with average high school GPAs. Additionally, the relationship is weaker (by a magnitude of 2-3%) for those students with high school GPAs one standard deviation above the mean than those with average high school GPAs. In other words, the relationship between LA support and course failure rates is stronger for those students with lower prior achievement than those with higher prior achievement.
Recall that the weighted average of the raw failure rates presented in Table I indicated that students who received LA support had 7% lower failure rates. The results in Table  IV suggest that after controlling for several confounding variables, the relationship between exposure to LA support in STEM gateway courses and failing introductory physics is even stronger for some. That is, for some groups of students, receiving LA support is associated with an even bigger decrease in the probability of failing than is suggested by raw failure rates. We found similar results in math and chemistry as well. That is, exposure to LA support in STEM gateway courses is associated with lower failure rates in introductory physics, math, and chemistry.

V. DISCUSSION
In this study, we attempt to disentangle the relationship between LA support and course failure at CU Boulder. Although we controlled for several student-level variables, we surely missed key variables that contribute to these relationships. Despite this limitation, the regression analysis represents an improvement over an unadjusted comparison of course failure rates.
Preliminary qualitative work suggests potential hypotheses for the relationship between LA support and student success. Observations of student-LA interactions indicate that LAs develop safe yet vulnerable environments necessary for learning. Undergraduates are more comfortable revealing their current thinking to LAs than to graduate TAs and instructors and are therefore better able to receive input and refine their ideas. LAs exhibit pedagogical skills and course experience that promote deep understanding of relevant content as well as critical thinking and questioning necessary in higher education. Also, through their interactions with LAs, faculty seem to be learning how to embrace the diversity of student identities and structure educational experiences accordingly. Finally, institutional norms are changing as more courses adopt new ways of teaching students. Although we see evidence of these ideas in our preliminary qualitative work, more research is needed to identify if these same themes are apparent at other institutions.
Further work is necessary to more fully understand the relationship between the LA program and student success. Additional research with data better suited to isolate potential causal effects is needed to bolster the results presented here. However, even though the present study is descriptive rather than causal, the findings are encouraging for further development and implementation of the LA program in STEM gateway courses. This is particularly valuable in courses in which failure traditionally stops students from persisting to graduation at institutes of higher education. Identifying relationships between models for change and course failure rates are helpful for informing future decisions regarding those models.