This study examined physics major retention to degree at two institutions with substantially different admissions selectivity. Two modes of leaving the physics major were examined: leaving college and changing to another major while staying in college. The risk of leaving college while still enrolled as a physics major was highest in the spring freshman semester. The changing major risk was substantially different between the two institutions. For the less selective institution, the students changed major at the highest rate in the fall sophomore semester. For the more selective institution, the risk of changing major was high through the first two years of college with highest risk in the fall freshman semester and the fall junior semester. Different features were important in predicting the two modes of leaving; these also differed between institutions. For the less selective institution, math readiness (being academically prepared to enroll in Calculus 1 in the fall freshman semester) was the most predictive feature for leaving the physics major while staying in college; high school GPA was the most important feature for predicting both leaving college and graduating with a physics degree. For the more selective institution, ACT composite scores were the only significant predictor of retention. The role of math readiness was dramatic at the less selective institution with 41% of students not math ready upon enrolling in college as physics majors; 59% of these students failed to enroll in the first required physics class.