Comparing the impact of informal physics program on undergraduate versus graduate student facilitators

Recent studies have begun to explore the connection between physics identity development and career skill development with the facilitation of informal physics programs by college and university students. Informal physics programs, or outreach programs, provide pathways for less structured, voluntary engagement with the field, beyond the formal spaces of a classroom or laboratory, where students can explore their passions within the field. This work narrows the focus of a prior study to explore the differences in experiences between graduate and undergraduate students who facilitated at least one of five outreach programs at a large, public, land grant university. Results from this work may help inform departmental efforts regarding retention, inclusion, and structures to promote skill development beyond the classroom.


I. INTRODUCTION
Undergraduate and graduate students have long served as integral parts of informal physics programs found in many departments and national labs around the United States. These programs, often referred to as "public outreach" [1], serve as a bridge between academics and the general public. This bridge between researchers and the public is facilitated by requirements from the National Science Foundation to incorporate broader impacts [2] as well as by substantial resources and materials provided by various professional organizations of physicists. Prior studies have demonstrated how outreach programs can provide unique opportunities for engaging children in STEM, and can serve as a particularly impactful avenue for underprivileged groups [3][4][5][6][7][8]. When focused towards the public impact of outreach, studies have shown a positive impact of outreach programs on developing an enthusiasm for science among children [7][8][9][10]. More recent work has begun to examine what impact facilitating informal physics programs can have on undergraduate and graduate physics students.
There is limited literature on the impact of informal physics programs on student facilitators. Nevertheless, we will briefly describe some of the early work on which our current work builds. Through studying the Partnerships for Informal Science Education in the Community (PISEC) initiative at the University of Colorado Boulder, researchers found that student facilitators of an after school mentoring program experienced positive shifts in their perspectives of teaching and learning, and improved their science communication skills [11,12]. Recent work from Fracchiolla et al. characterizing the experiences of two graduate physics students through a community of practice framework has suggested that facilitating informal physics programs may have a positive influence on the development of a student's physics identity [13]. Our recent study showed similar positive impacts to student physics identity for a larger sample of students who facilitated at least one of five different informal physics programs, and also showed positive influences on constructs including sense of belonging to the physics community and 21 st century career skill development [14]. It has been suggested that developing a student's physics identity may make them more likely to select physics as a career and persist in the discipline [15]. An area of study that, to the best knowledge of the authors, is missing from current literature is a comparison of the impact of facilitating informal physics outreach programs on undergraduate and graduate students.
This work seeks to address this comparison by building on our prior work [14], and exploring the differences in impacts of facilitating informal physics programs on undergraduate and graduate student facilitators using a mixed methods study. The informal physics programs which these students facilitated range from a significant once a year event involving hundreds of demonstrations and thousands of members of the public, to weekly or monthly events involving a small number of facilitators and a few dozen members of the pub-lic, as well as a program comprised of teams of undergraduate students led by a graduate student leader that develop, maintain, or improve experimental demonstrations. In the rest of this paper, we describe the data and framework used in this mixed methods study and then present and discuss results from a survey and interviews conducted with student facilitators.

II. METHODS AND FRAMEWORK
This work builds on data collected for our prior mixed methods study [14]. Data were collected through a survey consisting of questions targeting dimensions of student identity, sense of belonging to the physics community [16], and 21 st century career skills [17], as well as interviews conducted using didactic questions developed in collaboration with learning scientists. The survey was distributed via email in fall 2019 and responses were received from 117 undergraduate and graduate students. At the end of the survey students could elect to participate in a follow-up interview. Interviews were conducted with all 35 volunteers allowing researchers to probe deeper for student experiences related to facilitating informal physics programs.
Building on prior work, we developed a code book based on the framework from Hazari et al. [15] for physics identity. Additional codes were added based on elements of identity and learning described below. The framework used by Hazari et al. defined physics identity through the dimensions of belief in one's ability to understand physics content, recognition by self as others as being good at physics, and interest in the field demonstrated by the desire to understand physics [15]. Additional codes related to identity were incorporated through the lens of the Dynamic Systems Model of Role Identity, which characterizes identity as context-specific self-perceptions, beliefs, emotions, goals, and values [18]. Also included were elements of learning and how learning occurs through the frameworks of Situated Learning Theory and Transformative Learning Theory, which incorporate assumptions, beliefs, habits of mind, and perspectives [19][20][21].
Interviews were conducted by a researcher unfamiliar with each interviewee, and were coded based on the framework described briefly in the previous paragraph. A total of 64 codes were used to categorize statements [14]. The categories were: i) community (e.g. connecting with participants, accountability), ii) soft skills (e.g. creativity and innovation, teamwork and leadership, communication), iii) hard skills (disciplinary and non-disciplinary skills), iv) affect and experience (e.g. seeing new perspectives, transformational experiences, motivation and excitement, and v) identity (e.g. curiosity, positionality related to ethnicity and gender, worldview, becoming more confident). Categories i-iii were based on situated learning theory, category iv was based on transformational learning theory, and category v was based on DSMRI. The coding of interviews was done in stages, with six researchers initially coding three interviews. The team then met to dis-cuss the process and small improvements were made to the coding schema. Three of the original researchers then coded another block of five interviews. Intercoder agreement for this second group of interviews was > 0.8 between the first two researchers, and > 0.6 between the first two researchers and the third. The remaining interviews were then coded by at least two of the three researchers.
To compare the impacts and relationships among codes used for this study, the data were separated into two groups based on whether the responses were from an undergraduate or graduate student, and then we examined both the frequencies of individual codes as well as the relationships between them. To assist with exploring the relationships between codes, we employed a semantic network analysis for each group. This involves the creation of correlation matrices and determining the centrality of each idea via eigenvector centrality measures. The result of this analysis are maps, shown below, where larger blocks and a higher number of links correspond to the frequency and centrality of a code [22,23].

III. RESULTS
In this section we present results from a survey and interviews which were completed as part of our prior study focusing on the similarities and differences in the experiences of undergraduate and graduate students [14]. Our database of 117 completed surveys contained 67 responses from undergraduate students, 44 responses from graduate students, and 6 responses from students who were both at the institution being studied. The majority of responses were from physics majors, though a small number of responses came from outside of physics including engineering, atmospheric science, and computer science.
Self-reported responses from the survey for both undergraduate and graduate students were very positive regarding the impact of facilitating outreach programs on dimensions of networking, teamwork, sense of belonging, and their conceptual understanding of physics. Examining these dimensions with a Mann Whitney U test showed no statistically significant differences based on classification. Two other dimensions did show statistically significant differences, specifically impact on science communication skills and outreach being an important part of their education, both at the p < 0.05 level with graduate students rating these higher compared to undergraduates.
Some of the most common themes which were discussed by students during interviews are shown in Fig. 1. Approximately equal fractions of undergraduate and graduate students discussed how facilitating informal physics programs helped them to connect with their peers and participants, and helped them to become more confident. Graduate students were more likely to discuss the impact of outreach on the development of their speaking skills, as well as their leadership roles. Undergraduates were more likely to discuss an increased understanding of themselves after participating in outreach programs.
The semantic network map generated from interviews with undergraduate students, N = 12, at the p = 0.001 level is shown in Fig. 2. The semantic network map shows one large cluster with formal roles being the most central and connected theme. These formal roles include positions within the Society of Physics Students (SPS) chapter, which is heavily involved in supporting outreach programs, or as part of a team focused on the creation or maintenance of experimental demonstrations. This theme is connected to other significant themes including becoming an expert, increase in motivation, and development of communication skills. From one undergraduate's perspective, their formal role made them "feel like I'm contributing something not only to the physics department but to the society that I'm living in. I'm teaching people things I'm encouraging younger people to pursue science. And you feel like you're making a difference in the community." A smaller cluster of interconnected themes exhibits links between transformational experiences, seeing oneself as someone who can do physics, development of a research identity, and worldview. As one former undergraduate put it "associating fun activities with something that could also be a career based opportunity, was something that really shaped where I am now and why I wanted to go to graduate school." An additional noteworthy cluster shows a link between a sense of becoming a leader and becoming better at understanding others.
The semantic network map generated from interviews with graduate students, N = 23, at the p = 0.001 level is shown in Fig. 3. Similar to the undergraduate map, there is one large cluster and several smaller clusters. The most central theme for graduate students is a sense of overcoming imposter syndrome, which is seen to be linked to other important themes including researcher identity development, excitement and interest, as well as their individual roles. From one graduate student's experience there was "constant imposter syndrome... My PhD supervisor said that it never really goes away... but seeing people just ask questions, even like professors when I'm doing a physics demo... its a bit of a confidence boost." A smaller cluster shows a connection between becom- ing more friendly and a non-competitive worldview. As one graduate student put it when joining a program "I was a bit skeptical at the beginning because I was not sure if I could be a good mentor to someone because, most of the time, I don't feel like I have this kind of knowledge or this kind of way to outreach to other people. But it was so successful, both for the kids, but for me too...it was so nice to have this connection with my students." Another small but noteworthy cluster shows a connection between a sense of a collective role and becoming a role model. From one graduate student's perspective, they "acknowledge the importance of engaging the broader public in what I'm doing, I guess it helps me to think... of my tasks as a physicist as something that are not just for myself but, I guess, for others too. I see myself... less as pursuing an individual thing as something that belongs to everybody." Other notable small clusters include connections between speaking and presentation skills, teamwork skills and accountability to outreach leadership, as well as the ability to empower others and positionality of being a woman in science.

IV. DISCUSSION
The results from the survey and interviews presented in the previous section suggest that by facilitating informal physics programs both undergraduate and graduate students can experience positive impacts to their physics identity. Undergraduate and graduate students both indicated a strong positive link between facilitating informal physics programs and their sense of belonging to the physics community. Increasing this subjective sense of value and acceptance within physics may be an important part of a student's transition to seeing themselves as a physicist, and may be particularly vital for students from traditionally underrepresented groups [16]. The semantic network maps show that becoming an expert was also a central theme for both undergraduate and graduate students, being connected to excitement and interest, as well as confidence in research identity. From the work of Hazari et al., increase in interest is a major component of an individual's physics identity [15]. Through the informal physics programs included in this study, this increase in interest would primarily be the result of engaging with school-aged groups and families with a variety of hands-on demonstrations. By sharing a demonstration with someone new, physics students can potentially increase their confidence in their own knowledge and abilities, and grow their own excitement by sharing something new with a stranger.
For undergraduate students, holding a formal position related to outreach, either as part of a group within a program or holding an officer position in SPS, was seen to have significant ties to their feelings of becoming an expert and increasing their motivation. By taking on responsibilities in an informal program, undergraduate students get the opportunity to apply knowledge from classes, perhaps leading to more internal recognition as an expert, and experience feelings of accomplishment for a successful project or event associated with their role. Also, nearly twice the fraction of undergraduate students discussed understanding themselves as a physicist better after participating in outreach compared to graduate students. This may be due to the fact that undergraduate students are still early in their experiences as a physicist, while graduate students already have a better sense of their physics identity.
For graduate students, there seems to be a greater impact on the development of their communication and leadership skills. The latter of these would stem from the role that the majority of graduate students in the study had leading small teams of undergraduate students in year long development projects creating new experimental demonstrations. This role mimics that of a principal investigator conducting a research project with a team of support personnel in a low stakes environment under the mentorship of a faculty leader for the outreach program. To the authors' knowledge this may be a unique feature of the program which is described in detail in our recent paper [14]. Graduate students also exhibited a central theme of becoming an expert which was related to identity as a researcher. This seems a natural connection for graduate students as they are required to be engaged in research leading to a thesis for their degree.
While this work provides an exploratory glimpse into the comparative experiences that undergraduate and graduate students may have when facilitating informal physics programs, we must note some limitations. First, the data collected through the survey and interviews are self-reported data based on students' experiences and perspectives. Data were also collected at a single point in time from one large, land grant institution with a diverse enrollment. The informal physics programs that students engaged with also represent a subsection of the potential structures which exist in physics departments and laboratories across the country. Subsequent studies would benefit from sampling a larger collection of students from multiple institutions.

V. CONCLUSIONS
Our mixed methods study has explored the impact of facilitating informal physics programs on undergraduate and graduate students. Based on their self-reported data, both groups of students experienced positive impacts to their physics identity, seeing themselves as more expert, and their 21 st century career skills such as communication and teamwork. While both undergraduate and graduate students were impacted by their formal roles in informal physics programs, undergraduate students were more likely to better understand themselves as a physicist while graduate students experienced stronger development of their leadership and communication skills. These results suggest that undergraduate and graduate students both benefit from their facilitation of informal physics programs, but not necessarily in the same ways. We hope that our results may stimulate further study of the impact of informal physics programs on these two populations in the future.