Evaluation of a Summer Bridge Program Using Multivariate Matching

Summer academic bridge programs often play a dual role of supporting students’ academic and social integration in the transition from high school to college. Assessment of the impact of such programs tends to rely on self-reported student surveys and institutional records and often does not control for self-selection effects. Using multivariate-matching methods, this study examines the effect of a summer bridge engineering program on subsequent academic performance such as cumulative GPA. Controlling for selection effects, the analysis accounts for pre-college academic and demographic covariates and creates a matched control and treatment group. Within this well-controlled framework, we find no impact on an academic outcome, which resonates with other research findings. Viewed through the lens of Nora’s student engagement model, we discuss social integration outcomes which can potentially assess indirect programmatic effects such as participation in a Learning Assistant Program and enrollment in reformed STEM classes. PACS: 01.40.Fk, 01.40.Di


I. INTRODUCTION
This paper presents an evaluation of the effectiveness of a summer bridge engineering program which controls for self-selection bias.We describe the role of the summer program within the contextual framework of student persistence models [1,2] and in connection with established reform efforts in STEM introductory classes at our university [3,4].Summer academic bridge programs often play two roles by supporting students' academic and social integration in the transition from high school to college.The design of such programs is grounded in Tinto's model of student persistence in college [1] and Nora's engagement model [2], and used as a common retention tool.However, literature on summer programs and first year seminars points to limitations in the assessments of these interventions due to selection bias [5].
Some findings connect programmatic impact to improved academic performance, while others show no impact, or even decreased academic performance.
Interestingly, Pike at al. [6] examined the effect of first year programs on students' academic success and found that participation in themed learning communities was positively associated with higher grade point average.However, introduction of variables accounting for self-selection eliminated this effect.While Pike's study does not look at the effect of bridge programs, it does raise the question of how students are selected to participate in a program.
In this study, we applied multivariate matching methods [7] to create matched treatment and control group so both groups have statistically equivalent precollege preparation (high school GPA and SAT Math score), gender, ethnicity and engineering discipline.We then examined the effect of participating in the program on students' cumulative GPA in their first or third semester.This approach allows us to control for preexisting differences between the compared groups to obtain an approximately unbiased estimate of the effect of the program.Last, we discuss alternative outcomes that can serve as suitable indicators of program success.

II. BACKGROUND
The integrated research and learning community of the Physics Education Research group at FIU has been successful in implementing multiple reforming efforts that have had an impact on student retention.Drawing from research on student persistence [1,2], FIU's summer engineering bridge program was designed to target the sensitive summer transition from high school to college by supporting student's integration into the academic and social systems of college experience and in particular into learning communities such as reformed STEM classes [3] and the existing Learning Assistant Program [4].
It has been hypothesized that a summer bridge program could affect persistence and degree attainment factors that are likely to influence minority, low-income and non-traditional student populations.Nora's model of engagement emphasizes academic and social factors such as formal and informal interactions with faculty, validating experiences (from peers and faculty), mentoring relationship (faculty, peers and advising) and campus climate (perceptions).According to Carlone and Johnson [8], underrepresented racial minorities (URM) students who self-identify as a "science person" and are recognized by meaningful others as such, are more confident about their academic abilities.Findings by Chang et al. [9] suggest that "institutions can improve STEM persistence of URM students by increasing the likelihood that those students will engage in key academic experiences: studying frequently with others, participating in undergraduate research, and involvement in academic clubs or organizations".

III. PROGRAM DESCRIPTION
With these factors in mind, the bridge program at FIU was designed to be a collaborative learning environment focused on peer learning and scientific exploration in a way that resembles a research group.Adapting Berkley's Compass Project [10], the program was designed around a central problem and engineering design and engaged students in researching solutions.Using a classroom setting similar to modeling instruction classes [3], an experienced modeling physics high school teacher guided the students through qualitative and quantitative model development and testing.
Full incorporation of Learning Assistants [4] in all program sessions created peer involvement and was expected to seed the formation of a persistent learning community.In addition, the program included informal and formal interactions with engineering faculty, meetings with engineering professional clubs on campus and introduction to resources on campus.
This study aims to evaluate the impact of the program by comparing an academic outcome in both, the observational program cohort and in a matched control group thus ensuring an adequate comparison of pre-college, demographic and academic variables.
Florida International University is a large urban research-intensive Hispanic-serving institution.As of fall 2014, enrollment was 54,099 students.The student body at FIU is 63% Hispanic and 14% Black, making FIU an important source of STEM graduates from underrepresented groups.
The program recruited enrolled engineering freshmen in the summer prior to their arrival to campus.Recruitment emails were sent to all freshmen with a mention of a stipend encouraging the students to attend the full program.Figure 1 provides the cohort demographic profile.The two cohorts described in this study had 40% female students and 77% Hispanic students.

IV. METHODS
The data for this study was derived from institutional records and program records for 2 cohorts (summer 2012, summer 2013) of first-time-in-college engineering majors (49 students).We drew the control group from the target pool of engineering freshman enrolled in the corresponding term.The complete sample included 1200 entries.
We chose covariates based on Nora's engagement model as well as based on other studies related to first year interventions [1,2,11].Using multivariate matching techniques [7], we were able to create a control group that matches the chosen covariates of program participants.By using the matching procedure described by Hazari et al. [12], we matched students in the treatment group and the control groups on a set of covariates so that after matching both groups have statistically equivalent pre-college preparation (high school GPA and SAT Math score), gender, ethnicity and engineering discipline, and then we examined the effect of participating in the program on their cumulative GPA in their first or third year.This approach allows us to control for preexisting differences between the compared groups to obtain approximately unbiased estimate of the effect of the program.

V. RESULTS
Table 1 summarizes the means of the covariates being matched between the control and treatment groups, both before and after matching.After matching, there was no significant difference between treatment and control groups on any covariate for any of the precollege and academic variables (p = 0.6 to p =1.0).
Comparing the matched groups allows us to find the effect of the program on students' college cumulative GPA at the end of the first semester (for 2013 cohort) and at the end of the third semester (for 2012 cohort) since the treatment (participation in the bridge program) and control (no participation in the program) groups were equalized on background characteristics that also influence an academic outcome such as cumulative GPA.Again, two cohorts were combined to form a bigger sample, thereby increasing the statistical power and generalizability.
As summarized in Table 2, the results yielded no significant effect and we accept the null hypothesis of no effect of the treatment on the outcome (The mean cumulative GPA was 3.07 with a non-significant difference of 0.0087, between program cohorts and the matched control group).

VI. DISCUSSION
Our results point to no significant effect of the program on certain student performance outcome which resonates with other studies on effectiveness of first year interventions.While we did not match students on every possible student variable, the study did consider the variables found to be most predictive of academic performance.This does not mean that the bridge program does not affect other important educational outcomes such as science identity, self-efficacy and more.
Cabrera [11] argues that limited programmatic assessment "mean there is little empirical foundation for the current research in terms of selecting constructs related to success stemming from summer bridge program participation".We suggest selecting concrete social integration outcomes that fit the context of the The results of this study may also offer programmatic implications.Chang et al [9] found that pre-college characteristics such as high SAT scores and higher academic self-concept contributed positively to URM undergraduates persistence.Their findings support a call for early interventions that "build and sustain science self-efficacy reinforced with activities in college that ensure student success (e.g, bridge programs, supplemental instruction and tutoring)".This may suggest adding specific academic elements to the program, such as math and physics curriculum (non-remedial) that will accompany the engineering design project.
In summary, summer bridge programs are designed to address key factors related to pre-college characteristics and college experiences that affect persistence.The effectiveness of these interventions can be evaluated by using rigorous quasi-experimental designs with multivariate matching techniques to overcome potential selection bias.

FIGURE 1 .
FIGURE 1. Demographic and academic profile of summer bridge cohorts (2012 and 2013, N=49).Engineering majors are described on the right.

Table 2 .
T-test difference between treatment and control groups after matching for each of the pre-college and academic variables.

Table 1 .
Covariate means before and after matching for each of the pre-college and academic variables