In search of distinct graduate admission strategies in physics: An exploratory study using topological data analysis

Every year, graduate admissions processes determine which applicants are admitted to doctoral programs in physics around the country, and which are not. Despite their importance, relatively little is known about how admissions decisions are made and the normative practices surrounding them. In the current work, we adapt topological data methods (general techniques for cluster identification and relation) to look for the existence of distinct admissions strategies that institutions use in their admissions decisions. We analyze data drawn from a recent survey of faculty (graduate directors, etc) responsible for doctoral admissions, conducted in conjunction with the APS Bridge Program, which includes responses from individuals at over 85% of the active doctoral programs in the U.S. Our results suggest the existence of two distinct but similar groups of modest size, which differ in their strategies by their approach to the use of student grades and prior research experiences.


I. INTRODUCTION
Amongst STEM fields, physics continues to suffer from low representation of students who identify themselves as coming from traditionally-marginalized racial and ethnic groups.This underrepresentation is particularly marked at the graduate level, where barely 5% of PhDs awarded to U.S. citizens and permanent residents go to students who identify as African-American, Hispanic American, and American Indian/Alaskan Native [1].At the bachelor's degree level, these numbers are slightly higher, but still far below the representation of individuals from these groups in the general U.S. population.There is a relative drop-off in participation between the B.S. and the Ph.D. level, which calls into question all of the processes and mechanisms by which students may or may not transition between undergraduate and graduate school.One primary mechanism which could significantly impact graduate diversity is the graduate admissions process.Furthering our understanding of the criteria by which we select future generations of physicists is essential for the continuing growth and health of the field.Though many efforts have been made to increase the diversity and representation of physics at all levels, the situation persists.
It has been noted that commonly-used practices surrounding the GRE exam (e.g.hard cut-offs) may significantly limit diversity in new graduate pools [2].In earlier work, we have noted that certain motivational factors towards science learning are very important for scientists' career productivity [3,4], but these personal factors may be regularly overlooked in typical admissions decisions which focus primarily on "quantitative" measures of students' past performances.Others have advocated for the incorporation of grit [5] or other so-called "non-cognitive variables" [6,7] in admissions as a way to effectively identify a more diverse pool of students who will be successful.
To begin to address the representation problems of graduate physics in the U.S., the American Physical Soci-ety (APS) undertook the development of the APS Bridge Program, which began in 2012.It is designed to help increase the number of traditionally-underrepresented students who successfully complete PhDs in physics by building "Bridge Programs" for students who might not be accepted to or succeed in traditional direct-to-PhD programs.Further, the APS Bridge Program supports a range of initiatives to support new graduate students and prepare undergraduate students for the rigors of graduate applications and graduate studies more effectively.[8] In conjunction with the efforts of the APS Bridge Program, a survey of current admissions practices was created to investigate how physics graduate programs make their decisions on who to accept into their programs each year.Results from this survey have begun to be analyzed and reported elsewhere [9,10] but in this exploratory paper, we analyzed this data on admissions practices using a new methodology: topological data analysis (TDA).Specifically, we used an adaptation of the Mapper algorithm [11], which has been previously employed in other fields such as object recognition.The advantage of using Mapper over more traditional cluster analysis techniques is that it can help find patterns in data that the latter methods may fail to find.In this study we seek to quantitatively answer the question of whether or not there are multiple distinguishable patterns of responses used by admissions officials in their decisions of whether to admit new graduate students.

II. DATA COLLECTION AND METHODS
During 2012-2013, a survey consisting of 30 multi-part questions was developed to probe the admissions practices of active doctoral programs in the U.S., based on prior research [12] and existing literature [13,14].To help establish some aspects of the survey's validity, an draft version of the instrument was was circulated to participants of the 2 nd Graduate Education Conference, where focus groups with graduate student attendees and faculty were conducted separately.Once changes were incorporated, the final instrument contained several items on various admissions practices and, pertinent to the current paper, a 21-part question probing the importance of a wide variety of student criteria in determining admission, each on a 7-point anchored scale ranging from "Least important" to "Most important".These criteria included, among others, undergraduate grades, GRE scores, recommendations, and prior research activities, .
Using a contact list maintained by the APS of program directors and/or department chairs of all departments which manage active PhD programs in the U.S., these individuals were contacted directly and solicited to complete the final survey instrument, which was available online.They were also encouraged to distribute the survey link to faculty in their departments who were active in their graduate admissions processes.
In total, 170 individuals identifying themselves from 153 different institutions responded to the survey via email between August and November 2013.The response rate is estimated to be approximately 85% of all active PhD programs in the U.S. Note that the number of respondents is greater than the number of unique institutions: some departments did have representation from more than one of their faculty (up to three, in a few cases).As individuals are responsible for making decisions about admissions (though often by committee), po-tentially distinct strategies may exist between individuals even in the same department, so we analyzed responses at the level of the individual respondent rather than at the level of the department.
After maximum likelihood single imputation to account for missing data, we performed an exploratory factor analysis on the data, which identified 5 common factors underlying all of the different responses to the multi-part question.The factors were identified as "GRE Scores" (four items: Verbal, Written, Quantitative, and Subject), "Prior Research" (three items: experiences, publications, and conference presentations), "Fitting In" (four items: student interviews, research interests and opportunities, and departmental familiarity), "Recommendations" (three items: quality, reputation of writer, and rankings of student), and "Grades" (two items: GPA in physics/math and undergraduate courses taken) [15].A respondent's score on a given factor was the unweighted average of their responses to the questions which loaded onto that factor.Scores thus ranged from one to seven on all five factors [16].Respondents' scores on these factors formed orthogonal basis vectors for a new space in which we performed the topological data analysis.
The topological data analysis of this data begins by choosing an appropriate filter function [11].The choice of filter informs the eventual shape of the topological map; in our case, we chose to filter based on the sum of distances to the k nearest neighbors, which is a measure of density of respondents in the data space.Points with many close neighbors will have a lower sum than those whose neighbors are further away.Distance between points in this space was calculated using the familiar Euclidean metric.Clusters of similar responses appear as a series of linked nodes in a map, and dissimilarity between clusters can be seen from the distance in the network between the densest parts of the clusters.
All of the data cleaning and processing as well as the quantitative results reported below were conducted in R [17]; Mapper and associated helper functions were created by the authors in R.

III. RESULTS
To a rough approximation, the data is distributed in our factor space according to a multivariate normal distribution (i.e. an independent normal distribution in each of our five factor dimensions, each with their own mean).The length of the primary chain (see Figure 1), with no significant breaks or sub-chains, is evidence for a single multivariate normal distribution which increases in density as one approaches the center.The primary conclusion of this analysis is that there exists a wide variety of preferences in admissions, centered around certain, relatively popular means.
Digging deeper, we see evidence for a small difference between two relatively popular admissions strategies (the circled clusters in Fig. 1).Within the larger multivariate distribution, there are two related-but-separable strategies where the end of the chain forks into two groups, each with significant populations (N 1 = 31, N 2 = 14).For lack of better terms we will simply call them Cluster 1 and Cluster 2; Cluster 1 has more than double the population of Cluster 2. The membership of these two clusters accounts for roughly 25% of the total responses FIG. 2. A projection of the factor-space data into a plane of two factors.The red up-triangles form cluster 1 (N1 = 31), while the blue down-triangles are from cluster 2 (N2 = 14).Some data points are overlapping, due to the projection, and would be separated in the full five-dimensional space.
in our data set.[18] In looking at the average scores the respondents in each cluster had on the factors, we found a statistically significant difference in the both the means and distributions of the weight they assign to the Prior Research and Grades factors (see Table I).In particular, it appears that Cluster 1 respondents place significantly more importance on the Prior Research factor and significantly less importance on the Grades factor, compared to Cluster 2.
Projecting the factor scores of these two clusters onto the plane formed by the Prior Research and Grades factors (see Fig. 2) we can illustrate how the data scatters into two coarsely identifiable groups, whose centroids are well-separated.Attempting to visualize the data in two dimensions from the outset would have made it significantly more difficult to identify these clusters, especially when points are significantly separated in a dimension which is not visualized (such as the overlapping points at (4.0, 5.0) in Fig. 2).One surprising result was the lack of difference in the weights given to GRE scores; we previously hypothesized that there might be evidence of two schools of thought regarding the importance of using the GRE in admissions, but this was not the case.

IV. DISCUSSION
In this analysis, we found a wide variety of preferences in graduate admissions, as articulated by the relative importance placed on various factors.Looking at an individual responses, we see that some schools favor the use of GRE scores over the likes of, say, students' prior research, while others articulate the opposite preference; some place the most emphasis on the grades or courses.
Inside this natural variation, however, there is a modest split in preferences.While some variables, such as the weight given to GRE scores, seem to follow a unimodal normal distribution, others exhibit multi-modal behavior, suggesting that there are distinct strategies being used in graduate admissions.Cluster 1 was significantly larger than Cluster 2, suggesting that it constitutes somewhat of a normative strategy, while Cluster 2 is more non-normative while still being somewhat popular.The most defining feature separating the two groups, among the factors we analyzed, was a trade-off between the significance given to the prior research experiences of the potential graduate student as compared to the courses they took, and how well they did in math and physics as an undergraduate.For both Clusters, the Grades factor was weighted more heavily than the Prior Research factor, but Cluster 2 had a larger separation between these two factors than Cluster 1 did.
However, the natural variation between the respondents is a much more strongly pronounced effect overall, and most of the variation in this data is likely due to effects other than having a preferred "strategy", where the difference in cluster means was relatively small (at most one full point in the seven-point scale), and the variance p < 0.05, ** = p < 0.01, *** = p < 0.001, n/s = not significant of even the dense centers of the clusters was about the same size as the separation between them.
Further analysis of the results from Mapper should involve analyzing the non-numerical responses on the survey using other methods; we performed our clustering based on the numerical responses given to a single question, but that was only one of 30, many of which had open-ended answers.A qualitative analysis of the openended responses of the two Clusters could provide further insight as to why such a split in strategy might exist, and if there are certain attributes which characterize schools that belong to each cluster, or those which belong to no clusters at all and are distant from all other schools.This material is based upon work supported by the National Science Foundation under Grant No. 1143070 Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

FIG. 1 .
FIG. 1. Results of the Mapper algorithm as applied to our data.Nodes represent data points which were locally clustered.Links between nodes show that those nodes have overlapping membership.Darker colors correspond to areas of higher density of data points in the original data set.Node size corresponds to the number of data points in each node.Clusters (circled) were identified by finding strings of densely colored nodes with relatively high population (specifically, N1 = 31 and N2 = 14).Isolated data points of low density which did not connect to any others were removed for clarity in this representation.Created with Mapper, visualized in Gephi.Number of intervals = 30, overlapping by 50%.

TABLE I .
Difference in factors between clusters