Experiences of departmental supports by physics graduate students: results from 20 research-intensive institutions

Physics is one of the STEM fields with the highest attrition rates and a lack of diversity in graduate education. The results of retention rates of underrepresented racial minority (URM) students from the American Physical SocietyBridge Program (APS-BP) indicate that a potential correlation exists between experiencing a supportive departmental environment and retention. In this paper, we examine whether there are meaningful differences in physics graduate students’ self-reports of supportive components at their programs between URM and non-URM students. In addition, we examine whether differences exist between students’ self-reports of experiencing supportive departmental components between institutions that are and are not affiliated with the APS-BP. This study was based on physics graduate students’ responses to a survey regarding their experiences of departmental structures. In this paper, we use the 254 responses from U.S. citizen students enrolled at 20 research-intensive graduate programs. Performing multivariate analysis of variance and calculating effect sizes, we find that URM students report experiencing supportive components in their programs at higher levels than non-URM students, except for financial support that includes having covered tuition for the entire program, and being offered health benefits from college/department/program. Looking at other available data in the larger survey to better understand this outcome, we find that non-URM students receive graduate assistantships from their programs at a higher rate than URM students (89.7% vs. 62.5%, respectively), and that URM students are more likely to either obtain a loan or self-support (working outside of campus) their graduate education compared to non-URM students (25% vs. 12%, respectively). Overall, our results suggest that physics graduate programs have practices in place to better support URM students on aspects related to their progress in the program, but are not paying similar attention to their financial needs. Finally, we find that the APS-BP affiliated institutions provide more supports to their students than non-affiliated institutions.


I. INTRODUCTION AND MOTIVATION
Attrition and lack of diversity are two major and persistent issues faced in physics graduate education. Graduate attrition is at the high rate of 41% for the overall population of physics graduate students in the U.S., with African Americans and Hispanic Americans completing their Ph.D. degrees at rates of 56% and 59%, respectively [1]. Among the hard sciences, physics is the one with the greatest lack of diversity. The high attrition rates for students from underrepresented minority (URM) groups further increase this problem. The group populations that are underrepresented in physics are Black/African-, Hispanic-, Native-Americans, and female students. Only 20% of the annually awarded Ph.D.'s in physics are awarded to female students, and only 5% to students from URM racial/ethnic groups [2].
Diversity in education and equal opportunities is a matter of social justice and efforts for inclusion in the physics community have been made from several national organizations, such as the American Physical Society (APS). The APS developed a program, the APS Bridge Program (APS-BP), to help increase racial/ethnic diversity in physics graduate education. Students that participated in this program were placed in physics graduate programs while receiving several supportive departmental structures, known as key components.
The key components is a term first used by the APS-BP to refer to several departmental-level practices that are intended to create a supportive departmental environment. Those recommendations for action are broad and are directed towards faculty and administration, spanning from admission criteria, mentoring relationships, and to building relationships with university administration. At this point, the APS-BP has placed 64 URM students at 10 institutions across the U.S. 49 of these students are placed in three closely affiliated institutions with the APS-BP, known as bridge sites. The remaining 15 students are placed in less closely affiliated institutions to the APS-BP, known as partnership sites. The only difference between bridge and partnership sites, is that bridge sites receive direct funding from the APS-BP to prepare URM bridge students to apply and get accepted into physics graduate programs. Both types of sites aim to foster a supportive graduate environment with an increased emphasis given on students from historically underrepresented groups. URM students that enter graduate programs through the APS-BP have a 92% retention rate [3]. These records suggest of a potential correlation between experiencing a supportive departmental environment and persistence.
We developed a study to test this hypothesis, and in this paper, we examine whether meaningful differences exist between URM and non-URM physics graduate students' responses to the degree they experience supportive departmental practices. Those supportive departmental practices were measured by a set of 35-items that were developed for the purposes of this study and were inspired from some of the APS-BP recommendations that students could report on. Our study includes subjects from affiliated (bridge and partner-ship) and non-affiliated to the APS-BP institutions.
The research questions guiding this paper are: R.Q.1: Are there meaningful differences in the experiences of departmental supports between URM and non-URM students?
R.Q.2: Are there meaningful differences in the experiences of departmental supports between physics graduate students that are enrolled in institutions affiliated with and not affiliated with the APS-BP?
The information from R.Q.1 will help us identify whether racial minority status plays a role in students' experiences in graduate education. By identifying potential lacking supports to traditionally marginalized racial/ethnic groups, we can introduce recommendations to policymakers on how to improve the experiences of these students. R.Q.2 will help us examine whether affiliation with the APS-BP helps physics graduate programs facilitate practices that are meant to better support their students.

II. INSTRUMENTATION AND DATA COLLECTION
In this paper, we administered an online survey instrument (15-minute-long via Qualtrics) that, among other information and constructs, elicited participants' demographic information, and their perception of experiencing supportive departmental structures in their graduate program, which we named aspects of student experience. The aspects of student experiences consisted of 35-items that were developed by the researchers of this study based on some of the recommendations of the APS-BP for fostering a supportive departmental environment.
In the spring of 2019, we identified a list of researchintensive physics graduate programs, along with the email addresses of the department chairs and the graduate advisors of those programs. The first step was to invite the chairs of those departments to participate in our study by providing us with a list of all their graduate students' contact information, along with sending their students an email encouraging them to take the survey. However, some programs that agreed to participate would not provide the list of students. In these cases, the chair/graduate advisor sent the invitation to the survey (on our behalf) along with an anonymized link to the survey. The students were expected to click on the survey link to read the consent form and then decide whether they want to participate. The only incentive offered in the study was that we promised to prepare and share with each program (sent to the chair) a summary report with the mean values of the structures their students report experiencing, as well as comparison data from the other programs in the study. The graduate committees could use this information to improve their practices. We contacted 60 physics graduate programs across the U.S. (sending up to two reminder emails). 20 programs that agreed to participate are included in this study. Among these 20 institutions, 9 are affiliated with the APS-BP, and the remaining 11 are not affiliated.
A total of 376 physics graduate students fully responded to all survey items. Table I displays some of students' demographic information that were collected in this study. According to the national data on physics graduate education of the American Institute of Physics (AIP) [4][5][6], students that selfidentified as women/non-binary are slightly over-represented in our sample (26.7% vs. 20%), non-US citizens are significantly underrepresented (27.2% vs. 43%), and among the US citizens, URM students are over-represented in our sample (9.3% vs. 5.7%). The response rate of the participants at sites that provided us with a list of all their students is 27%. We realize that inter-sectional axes of oppression exist and we will try to take those into account in future work. Also, it is worth noting that only four of the 24 URM students in our sample are students that are enrolled at the graduate programs affiliated with the APS-BP. Thus, the mean values of the aspects of URM students' experiences mostly portray the views of non-bridge URM students.

A. Exploratory factor analysis
We performed an exploratory factor analysis for the 35items using the 376 student responses, attempting to capture the degree that students experience a supportive departmental environment. First, we computed correlations of all partial combinations of these 35 items. The Bartlett's test of sphericity, 2 =5184.519, df =595, p<0.001, and a Kaiser-Meyer-Olkin measure of sampling adequacy KMO=0.882, indicate that the resulting matrix of correlations was appropriate for factor analysis. In the next step, we conducted a principal component analysis on all the 35 items. The Kaiser-Guttman rule was used and we identified eight factors with eigenvalues greater than 1, which accounted for 60.15% of the total variance in the items. The scree plot supported the 8-factor solution. We also used Horn's parallel analysis finding similar results to the Kaiser-Guttman (KG) rule on retaining an eight-factor solution of eigenvalues. Using a Varimax rotation to turn the reference axes of eight factors about their origin we identified the items that fall into each of the eight factors by meeting the criterion of loading at least 0.4 on their respective factor. We used the Direct Oblimin rotations for double confirmation which produced similar results. The exploratory factor helped us determine the items that fall into each of the eight identified factors. The number of items, eigenvalue, and percent of variance by each factor are: mentoring (n=6, 8.37, 23.92%), professional development (n=6, 3.43, 9.79%), progress monitoring (n=5, 2.41, 6.87%), induction (n=6, 1.88, 5.37%), financial support (n=3, 1.39, 3.98%), research focus (n=3, 1.27, 3.62%), flexibility and research match (n=3, 1.18, 3.36%), and program fit (n=3, 1.13, 3.24%). The reliability measures (internal consistencies) of factors were assessed using Cronbach's alphas and the values for each aspect of student experience and an example item for each of these aspects are shown on table II. Coefficients of Cronbach's alpha > 0.80 are considered "very good," values between 0.80 and 0.70 are "respectable," 0.69 to 0.60 are "undesirable to minimally acceptable," and values below 0.60 are "unacceptable." B. Hedge's g effect size and multivariate analysis of variance The magnitude of difference in the mean ratings of the aspects of student experience between URM (Black/African-Americans, Hispanic-Americans, Native-Americans), and non-URM (White-and Asian-Americans) were assessed using Hedges' g effect size analysis and confidence intervals (CIs) [7]. We also performed the same analysis procedure to test the differences in means between students' responses based on the type of affiliation of their graduate program with the APS-BP. A value of 0.20 is considered a small effect, 0.50 is considered a moderate effect, and a value of 0.80 is a large effect [8]. We also performed a multivariate analysis of variance (MANOVA) to determine the statistical significance of the results. First, we performed several tests to examine whether assumptions of the absence of multivariate outliers, linearity, absence of multicollinearity, and homogeneity of variance are met. Only the component of financial support did not meet the assumption of linearity between all the other dependent variables. MANOVA is the most appropriate test that controls for the inflation of Type-II error when more than one dependent variable is used [9]. Statistical analyses were performed using the IBM SPSS version 25. We use both the effect size results and as well as the significance tests to confidently interpret the results [10]. While the statistical significance indicates the likelihood of the observed result if the null hypothesis is true [10,11], it does not assess practical importance [12]. On the other hand, the effect size refers to the magnitude of the difference between two means and it can also indicate the practical significance. Thus, using the CI, we will be 95% confident that the CI range captured the true effect size [14]. Thus, is it important to report the effect size along with the CIs. Therefore, we decide to use both the effect size with the CIs, and p-values as criteria for the interpretation of the results [10].

IV. RESULTS
We found that differences exist between URM and non-URM self-reports of aspects of student experiences at their programs, Pillai's Trace=0.079, F(7, 246)=3.007, ⌘ 2 =0.079, p=0.005. Four outcomes with small to medium effect sizes did not achieve p values less than 0 The students that self-identified with minority races are more than the 21 (Black and Hispanic combined), since in the analysis for R.Q.1, we also included three students that self-identified with more than one minority races only (e.g., Hispanic/Latino and American Indian/Alaska Native). Students who self-identified with both minority and non-minority races were not included in this analysis. Note: Students responded to those items on a five-point Likert scale ranging from "strongly disagree" to "strongly agree." The Cronbach's ↵ value of program fit was below the critical value 0.60, thus was excluded from the analysis.

V. DISCUSSION
The results of the effect sizes suggest that URM students report being better supported in their graduate programs compared to non-URM students in mentoring, progress monitor- ing, induction, and flexibility and research match. Physics graduate programs are paying increased attention to, for example, monitoring students' progress on coursework and facilitate interventions (e.g., tutoring) to help them overcome any faced challenges. On the other hand, the moderate to large effect size result (g=-0.71, p=0.001) for the component of financial support is highly concerning. URM students report experiencing less financial support compared to non-URM students. To better understand this finding, we examined more closely students' responses to this component. The component of financial support consists of three item questions: "My tuition is covered for my entire program;" "My college/department/program offers me health benefits;" and "I have no financial concerns about completing my degree." We found that meaningful differences exist between URM and non-URM students' responses to the first two items. Non-URM students' responses suggest that they are more confident about having covered tuition for their entire program (M=4.18, SD=1.23) than URM students (M=3.17, SD=1.74) (g=-0.79, [-1.21, -0.36], p<0.01). Moreover, non-URM students report of receiving health benefits at a higher degree by their college/department/program (M=4.08, SD=1.14) than URM students (M=3.21, SD=1.64), (g=-0.73, [-1.15,-0.30], p<0.01).
The large effect size (g=-0.79) between these demographic groups on having covered tuition for the entire program urged us to further inquire about students' responses to the survey item related to sources of their financial support. This question asks the respondents to indicate what funding sources have they used to support their graduate education, having the option to select all that apply (most of the responses included more than one option). We find that 89.7% of non-URM students hold graduate assistantships (either as primary or additional financial source, e.g., to fellowships), while only 62.5% of URM students are graduate assistants. 12.1% of non-URM students have either obtained a loan or self-support (working outside of campus) to support their graduate education. This rises to 25.1% for URM students. However, URM students obtain more fellowships/awards than non-URM (45.8% vs. 28.5%).
We can imagine that having a fellowship is a desirable financial resource, however, it might be perceived negatively in the sense that it lasts for only a year or two and therefore, the student needs to find alternative sources for financial support for the remaining of their education. Lovitts and Nelson [13] found that the type of financial support is linked to success at completing the Ph.D. They found that fellowship recipients are at risk of not completing the program because of the feeling of isolation from the other graduate students, concluding that fellowship recipients are more likely to dropout than teaching assistants. Moreover, we find that departments tend to offer graduate assistantship positions to non-URM students, and that URM students find alternative ways to self-support their education through external to the university sources. The results on the item of receiving health benefits further support the claim that URM students are financially under-supported at their programs compared to non-URM students.
Finally, the observed effect sizes in students' responses in experiencing supportive departmental components between affiliated and non-affiliated institutions with the APS-BP is an indication of the success of the APS-BP. We see that students enrolled at institutions that are affiliated with the APS-BP report experiencing aspects of mentoring, progress monitoring, induction, financial support, and flexibility and research match at a higher level than students at non-affiliated with the APS-BP institutions.

VI. CONCLUSIONS AND IMPLICATIONS
The development of the APS-BP is an indication that a call for attention has been signaled among educators and policymakers in higher education to better support students from historically underrepresented groups to decrease the existing lack of diversity. Results of these changes are found in this study, suggesting that URM students report experiencing supports at higher levels in their graduate programs compared to non-URM students. However, three are concerning differences with respect to financial support. To better understand students' experiences and any struggles they face in the pursuit of their graduate degree, we will conduct individual interviews and use additional data from the larger survey in the study to examine the effect of the aspects of student experience on their self-efficacy and intention to persist.