Tracking Graduate Admissions During COVID-19: Year One

COVID-19 has disrupted nearly all aspects of global society, with severe impacts on many people around the world. In 2020, supported by the APS Innovation Fund, we began a study to document and understand how COVID-19 has affected graduate admissions in physics, with a focus on the ways in which students and departments have been impacted. In this Work in Progress paper, we report on the first year of this study, which collected data from the 2019-2020 admissions cycle. We collected survey data from Directors of Graduate Study (DGS) and from students who applied to physics graduate school in this cycle (including students who did and did not begin graduate school in 2020), as well as conducted follow-up interviews with students and faculty to better understand the breadth of experiences of individuals affected. Four themes emerged from this first year of data: shifts in international/domestic representation, program resilience as a key distinguishing feature during COVID, the breadth of student experiences in 2020, and concerns about the expectations for the future.


I. INTRODUCTION
The COVID-19 pandemic has affected essentially every aspect of our society, ranging from the most severe outcomes such as the loss of life itself or of loved ones, to changes in requirements on physical presence in work and school, resulting in significant isolation for many people. It became evident early in the pandemic that, in the context of higher education, many of the effects would have ramifications for school admissions processes broadly and, specifically, for physics graduate programs. For example, GRE subject tests (including in physics) were cancelled in September through October 2020 [1] (a critical period for graduate applications). Many research experiences for undergraduates (REUs) were cancelled while others were heavily modified to facilitate remote or socially-distanced research experiences (see, e.g. example [2], [3]). Also, many institutions moved to pass/fail grade options in one or more semesters of 2020, so it became clear that many of the traditionally-used metrics for graduate admission might be unavailable in the admissions cycle.
These disruptions and resulting adaptations come at a time of increased national discourse around the challenges and problems in college admissions practices, with particular concern about the implications for equity and access in student populations (graduate and undergraduate). Several prominent university systems have wrestled with these concerns; for example, the University of California system [4] has recently dropped the use of SAT/ACT scores entirely in their undergraduate admissions, responding to ongoing legal cases related to equitable opportunity in admissions. MIT has suspended the use of SAT/ACT in 2021-22 owing to the COVID-19 disruptions in testing [5]. Harvard and other elite universities are reported to have accepted a more diverse cohort of students in 2021, and this has been attributed at least in part to their decreased reliance on standardized test scores [6]. In physics specifically, in the years preceding the COVID-19 pandemic, it has been noted that despite explicit guidance from ETS arguing against the practice [7], physics graduate programs often rely on the use of a GRE cutoff score in admissions [8][9][10] which has been clearly shown to disproportionately, negatively affect minoritized students [11,12].
The goal of the current Work in Progress is to investigate the effects of the COVID-19 pandemic on the physics graduate admissions process, and to identify prevalent themes in these effects (the four most prominent resulting from our ongoing analysis are discussed in the Results section). Of particular concern is the question of how the pandemic would affect racial, ethnic and gender diversity in graduate programs. Other concerns include: How will departments conduct admissions while missing information they may typically rely on? Will departments that have previously interrogated their admissions process (including developing holistic admissions practices) be better prepared to adapt? In this context, "holistic admissions" may refer to any process that seeks to evaluate an entire applicant, rather than reducing the applicant to a number or set of scores (although we note institutions may have their own working definition of this). What adaptations made during the COVID-19 pandemic will persist beyond its peak and what practices will return to the prior status quo? To explore these and other questions, it was critical to center student voices as well as DGSs to ensure that both institutional practices and their effects on students could be illuminated. DGSs further provided information on recent years' enrollments. Accounts of individuals' experiences are described further in Sec. II.
We note that departmental admissions decisions were byand-large complete for the 2019-2020 cycle before COVID-19 compelled widespread shutdown. Substantial additional work is required to explore the 2020-2021 admissions/enrollment cycle so a more complete picture can emerge. In the current work, we report on the first year of a planned two-year study that attempts to provide some accounting of departmental and student experiences. Another limitation of the current study is that admissions are only a small part of the process of graduate school and does not address the challenges that students face after beginning graduate school, which may be severely disrupted by COVID-19 for the foreseeable future. In the current study, we collected data to understand students' experiences after beginning graduate school but this was necessarily limited to the first semester (fall 2020). There are many topics worthy of further research including the impacts of remote/distance learning, challenges to community and cohort building, and, perhaps most critically, maximizing the health and safety of students and other members of the physics community. These important topics are outside the scope of the current work.

II. METHODS
In late summer 2020, a focus group of DGSs was held, consisting of 7 DGSs from physics departments across the United States. The purpose of the initial focus group was to identify programmatic concerns to probe in a broadly-distributed survey. In fall 2020, a survey was distributed through APS to 73 graduate physics departments, representing a stratified random sample of approximately one-third of all physics graduate programs in the U.S. The stratification accounted for undergraduate program size and diversity of recent graduates (binning institutions as being above or below the national median in both dimensions). 17 of these participated, providing numbers of enrolled graduate students (by year, gender, race and ethnicity, and country/region of origin for non-U.S.based students), information on departmental changes made in response to the COVID-19 pandemic, and plans for future changes. In January 2021, follow-up interviews were held with 7 DGSs to explore some of the survey findings. These semi-structured interviews were approximately 30 minutes long and used a protocol generated from survey responses, allowing for open-ended discussion.
From December 2020 to February 2021, graduate programs were asked to distribute a survey to any students who had been part of their application pool in 2019-2020 (whether or not these students had been accepted to their graduate program). The student survey probed students' admissions experiences, and their experiences in the first year of graduate school, if applicable. Survey questions related to admissions probed virtual visits/open houses, relocation concerns, and feelings of safety, among other topics. Items related to the first year experience also addressed perceived safety and physical presence requirements, and preferences in learning modalities. 136 students responded to the survey. Of these, 14 agreed to participate in one-on-one interviews in early spring 2021. These interviews were 30 minutes long and semi-structured to allow for open-ended discussion. Genderidentity data were not limited to binary categories, but low representation of non-binary genders makes identifiability a concern. Hence, we refer here to "female" (which includes any student who identified as female, regardless of any other identities, if any) and "non-female" (all others) students.
In the rest of this paper, we report on four major themes that arose in all four data sources (surveys and interviews with both DGSs and surveys). Each source was initially analyzed separately by the research team, and then emergent themes were triangulate across sources and methods, seeking to understand commonalities. The results were then organized into four major, non-exclusive themes, which appear below.

III. RESULTS
The preliminary first year results are organized into the four themes: (A) shifts in international/domestic representation, (B) program resilience as a key distinguishing feature, (C) breadth of student experiences in 2020, and (D) concerns about the expectations for the future.

A. Shifts in International/Domestic Representation
In the 17 programs that supplied enrollment data, total enrollment in 2020 is increased but is not statistically different from the 2018-19 average. U.S.-based ("domestic") enrollment increased by 25% (though this is not statistically significant using a repeated measures analysis, p = 0.17, Cohen s d = 0.45, 95% CI on difference: -27 to 137). Non-U.S.-based ("international") enrollment overall was not significantly different in 2020. This group was disaggregated by region of origin including: Africa, South & Central America (including Caribbean), Asia (excluding Indian subcontinent), Indian subcontinent, and Europe. For the region Asia (excluding Indian subcontinent) total enrollment was marginally decreased from 2018-19 to 2020 (p = 0.12, Cohen s d = 0.32, 95% CI on difference 5.1 to -40.1).
The representation of female-identified, African-American-identified, Hispanic/Latinx-identified, or Native American-identified students in this sample were not statistically different in 2020 from the 2018-2019 average. Several DGSs described an increase in international students asking to defer admission for a semester, year, or even longer. The closure of U.S. Embassies for routine services in most countries was often cited by DGSs as a reason for these requests, especially for students applying from China. Emerging travel restrictions were also regularly cited as another reason. During student interviews, multiple students described leaving their country of origin to visit a U.S. embassy which had reopened in a neighboring country.

B. Program Resilience as a Key Distinguishing Feature
It is well known that, at the individual level, various measures of resilience have been linked to the persistence and success of individuals pursuing long-term tasks (for example, see [13]). Emerging from the DGS survey responses, conversations with our DGS focus group, and our DGS interviews, there is evidence that institutional resilience may play a similar role at the programmatic level. While our results are preliminary, there are certain consistencies amongst departments that appear to be weathering COVID-19 more effec-tively. Namely, several of the institutions identified as having some form of resilience are members of the APS Bridge Program (indicating they may have already critically interrogated their admissions and student induction practices), routinely award degrees to large, more diverse cohorts of students (indicating they may have already instituted more humane admissions and student support programs), or have previously institutionalized programs designed to more effectively support their students.
One DGS representing a prominent department in a public research university in the inter-mountain west region described how their department worked hard pre-pandemic to connect incoming graduate students to each other in a paid summer research induction. Porting this experience to a remote mechanism was more straightforward than anyone expected and provided the "pandemic cohort" of 2020 an opportunity to build relationships student-to-student well before the semester began.
Two internationally prominent departments at major landgrant institutions in the Midwest had similar pre-pandemic commitments to holistic admissions practices and carefullydesigned student support for matriculated students. While national policies on residency requirements for employment and matriculation of students governed both (e.g. visa requirements imposed upon international students), different political pressures in different states produced a bifurcation. One of these departments had to create an ad hoc process for students not physically present to audit classes remotely without formal matriculation, with the understanding that when possible these informal audits would be converted to grades. Expending any resources on these students (e.g., tuition support or stipends) was not allowed. The other department confidently went ahead with enrolling remote, international students, covered tuition costs as normal, provided stipends for (remote) graduate TA work. As a result, these students had a more "typical" experience of the program. This difference in outcomes between these two programs in the first pandemic year is an interesting consideration for further inquiry. This also identifies one potential limiting factor on program resilience: the political and legal framework that may determine the ability of a department to allocate resources.
Two MS-granting departments in the same public university system in the Pacific Coast region had divergent experiences prompted by different institutional budgetary instabilities. Both were active in recruiting diverse students from the APS Bridge common application pool. Despite their departmental similarities, one was not able to fund budgeted TA positions while the other was able to leverage CARES ACT support to enhance TA compensation. Newly matriculated students thus experienced dramatically different scales of disruption. The dimension of program resilience relating to holistic admissions was not powerful enough to overcome a lack of institutional resilience.
Lastly, two nationally prominent public research universities in the same region but different states described essentially unchanged structures and working conditions for the 2020 incoming cohort, with the one caveat that the decline in international student enrollment had been made up with domestic students. Both programs had recently established policies of de-emphasizing GRE scores in admissions as well as developed "pipeline" relationships with other regional institutions to recruit potential students. This latter practice highlights a potential critical factor in program resilience: the ability to capitalize on local relationships to identify overlooked, diverse, and well-prepared students.

C. The Breadth of Student Experiences in 2020
The third major theme that emerged in this project is the sheer breadth of application, admission, and enrollment experiences of students in 2020. For example, one clear distinguishing feature amongst student participants (both survey respondents and interview participants) was the division between the experiences of international and domestic applicants. Many international students had to resort to delaying their admission or beginning (partial) studies remotely while waiting for restrictions to end. For several students, this imposed the dual burden of "partially" beginning graduate studies (or of having to put many hours into securing a visa interview at an American embassy, for example) while simultaneously having to continue to work in order to maintain a semblance of financial stability.
Domestic students reported a wide range of experiences during their transition to the first year of graduate studies, both positive and negative. Many students had been able to begin their graduate programs with relative "normalcy", including relocating as necessary and taking graduate classes in a remote/online modality. Relocation did not appear to have been a significant challenge, above the background level of COVID disruption. Several respondents had moved within the U.S. to start their programs, while some had already been located nearby (e.g. enrolled in a university in their home town). For the most part, the students who had begun studies in fall 2020 were taking courses in a remote/online modality: while some students felt that this experience was acceptable, others did not at all enjoy this particular disruption.
Some students reported that they accepted a graduate admission offer in spring 2020 because of COVID: their inability to secure a job at the onset of the pandemic led them to fall back onto graduate school. One interviewee said: "It sounds kind of bad but I realized I wasn't going to get a job anywhere; there [was] no sort of market and I had already applied to grad schools and been accepted and so I was, like, I guess this is what I'm doing. And if I don't like it, I haven't wasted any time so I . . . might as well go ahead and go to grad school." (Student 1) This student went on to indicate that, depending on the job market and their interest in their graduate program, they might choose to leave their PhD and take a job as originally intended if the opportunity arose. Noteworthy is that this student was progressing well in their graduate courses and was not facing any threat to their good standing in their program.
At least two participants reported that their personal/family life responsibilities had been a source of friction/difficulty with their program, in one case leading to significantly disrupted outcomes. A survey respondent noted "I did recently marry. This did require a substantial amount of time to put into the relationship that the graduate school wouldn't work well with." (Student 2). In another case, a student reported that their spouse's pregnancy had led them to ask their program if they could relocate during COVID (the program being primarily remote already) but were told that was not possible. This led the student to quit the program and re-enroll at another graduate program closer to home, a significant disruption to their academic progress (though their academic standing was apparently good at both institutions and the student's long-term plans were not changed).

D. Concerns About the Expectations for the Future
With all the changes during the COVID-19 pandemic, there is an overarching question of what will or should persist into the future. DGSs were asked the question, "To what extent do you expect your admissions practices to change in 2020-2021 due to COVID-19, international travel restrictions, or some other reason?" The average response was 4.1 on a scale from 0 (not at all) to 6 (almost completely). DGSs were also asked whether their 2020-2021 recruiting practices would change (or had already changed); half of the respondents said yes, citing changes such as partnering with a bridge program, increasing recruitment of domestic students, and recruiting more heavily through virtual conferences.
In some cases, changes seem likely to be sustained beyond the 2020-2021 cycle. When DGSs were asked "What do you plan to do differently in the 2020-2021 recruitment/admission cycle?" 7 out of 12 respondents who provided detailed responses specified dropping of GRE requirements (6 departments), or reducing the weighting of the GRE in their application rubric (1 department). Several DGSs confirmed this in the follow-up interviews, and described other changes to admissions practices related to this including: increased weighting of research experiences, core course GPA, and letters of recommendation, as well as asking students for personal statements and/or diversity statements. Asked about how to think about admissions when several traditional metrics may be unavailable, one DGS responded, "If they don't have anything, obviously it's detrimental to your ability to evaluate the applicant. But I've seen thousands of applications, and there are very few where all of those things are true." This was typical in that most DGSs indicated that admissions committees were able to work without GREs even when other information might be missing, in part or in whole.
Another example of a change that might persist is hold-ing virtual open houses for prospective students. Though not mentioned in the question stem, virtual open houses were identified as preferred practices for the 2021 cohort by 4 respondents. DGSs' justifications for this focused on lower costs and improved logistical capability to pair incoming and advanced students in conversations or other peer-mentoring. Student interviews on this point were mixed. Several students indicated that a virtual visit was 'better than nothing', but a few noted that through such visits they didn't learn enough about the city, neighborhoods, or departmental climate. Student and DGS opinions on the future of remote/distance learning were mixed. Many pointed out that distance learning had made the 2020-2021 academic year possible and much safer. Some even appreciated the convenience of having recorded lectures to refer back to. But, practically all students spoke of the isolation that was compounded by remote learning as a tremendous challenge, and many looked forward to resuming in-person instruction.

IV. CONCLUSIONS
Many changes implemented during and in response to the COVID-19 pandemic may persist beyond the height of the pandemic. Critically, however, the admissions process for the 2020 cohort was well under way by the time COVID shut down many facilities and programs. Data collected in fall 2020 do not inform, for example, the adaptation of admissions processes in the absence of GRE scores or other traditional metrics. The interviews conducted early in 2021 do give some limited insight on this but further investigation is critical. In the second planned year of this study, similar data collection is planned that will document the admissions process and initial experiences of the 2021 cohort. It would be of interest to compare programs that had previously developed more intentional admissions practices (including, possibly, bridge programs) to those that relied heavily on GRE scores and other traditional metrics. We hypothesize that these adaptations will be easier with the former group than the latter.
Lastly, we note that the shift in domestic/international enrollments we identified in our sample of institutions is largely attributable to events that happened after most departments had made their decisions about admissions offers (e.g. embassy closures and travel restrictions). Thus, the 2021 cohort will be the first one for which departments will make offers using applications made under pandemic conditions. The coming cycle will be a truer test of how equity and access to physics graduate school suffers under these conditions.

ACKNOWLEDGMENTS
Authors wish to thank all of the individuals who agreed to participate in this study for their time and wisdom. This project is funded by the American Physical Society (APS) Innovation Fund through Award IF-5.