Perspectives on informal programs: How site visits can help us learn more

We are continuing a nationwide effort to develop a systemic understanding of the landscape of informal physics using an organizational theory perspective. We have collected surveys and interviews with informal physics program facilitators, but this information is only from the perspective of the faculty or physics student leaders and does not tell us about the social dynamics within each program. Thus, to complement these data, we need to observe informal physics events as they occur. In this paper, we will discuss our strategy for visits to program sites to observe social interactions between program participants as well as programmatic details in action. We report on an initial site visit to a physics open house event, where we took field-notes and conducted interviews with participating personnel members. Here, we compare the types of data we are able to collect from site visits and interviews/surveys with lead program facilitators.


I. INTRODUCTION
Informal physics learning environments refer to learning spaces outside the traditional classroom setting. This may include programs such as a science cafe, a physics open house, or a demo show. Physics education research has demonstrated that the local context and implementation of curricular materials in classroom settings critically affect the impact of those materials on students. For example, a variety of factors such as faculty buy-in, the presence of LAs, backgrounds of students, and if the chairs and tables in a classroom are movable, all can affect the way students experience curricular transformations [1][2][3][4][5]. Our project seeks to characterize the even more diverse variables that are in play in the local context and implementation for informal physics environments. Without this information, we cannot diagnose ways to improve the learning experience or design different formats that more meaningfully engage audiences.
The landscape of informal physics learning environments is complex as programs can vary drastically in their structure, activities, contents, and engagement [6][7][8]. Due to the complexity and richness of the landscape, it is difficult to evaluate the functionality and effectiveness of the programs. While informal efforts are often the public face of their host institutions, there is limited understanding compared to the more studied formal learning counterparts. It is necessary to study informal learning environments more rigorously as these programs and activities not only impact the public, but also those participating in the programs. Without a clear understanding of the resources, structures, and environments of the programs, we are limited in our abilities to critique and evaluate the programs and their impacts.
This work is part of a broader project to study the landscape of informal physics in effort to better understand the key factors that play into the effectiveness of informal programs. The general goals of the project are to determine what aspects are important in an informal program and how those factors are connected with each other. Understanding the challenges that programs encounter, as well as the internal and external dynamics of the programs, can provide insight on the functionality of these programs. To study some of these factors, we have already implemented and analyzed surveys and interviews with program facilitators. This study builds off our previous work that collected interviews/surveys about program logistics from the lead program facilitators [6]. From that work, broad organizational themes emerged as program, personnel, audience, resources, institution, and assessment, as well as finer sub-codes for each category. These data, however, are limited to the perspective of the lead interviewee.
Here, we aim to address the limitations of the interview/survey data by expanding our methodology to include site visits to informal physics programs. We hypothesize that data collected from site visits will complement the interview/survey data by revealing emergent sub-themes that were otherwise filtered out. Thus, the analysis we present here will further guide the methodology of the broader landscape project. We seek to characterize the factors that are important to how an informal physics program functions by looking at a site visit from an organizational theory lens. We draw from the business literature on organizational theory, which breaks down the different operational aspects of organizations. Taking an ethnographic approach, we collected field-notes and artifacts, and recorded other interactions during a site visit to a large informal physics program in the midst of running its annual community-wide event. From these data, we looked to see if new emergent themes came up by using organizational theory as guideposts. Then, we examined the content of these emergent themes, focusing on how these new data complement the already collected interview/survey data. We find that site visits provided significant insight into how personnel, resources and audience were important during informal physics program events in ways that were not salient from the interviews and surveys. For the broader project, this research shows that site visits will be critical into establishing full profiles of how informal physics programs function.

II. CONTEXTUALIZING ORGANIZATIONAL THEORY FOR INFORMAL PHYSICS PROGRAMS
Programs may have multiple different goals and objectives they want to achieve, for example, wanting to focus on specific content or increasing engagement of certain age groups. The individuals participating in these programs may share the same goals, but their priorities or reasons for participating may also differ, for example, improving teaching abilities or experimental skills [9]. We take an organizational theory perspective to better understand the functionality and dynamics within these programs. Organizations operate under their own structure, and their performance is influenced by a multitude of components such as resources, the working environment, and the social relationships both internally and externally of the organization, and power dynamics within these relationships [9,10]. Organizational theory looks at the connectedness and weighted emphasis of these components' impact on the organization [10]. Informal physics programs have complex dimensions in terms of how the personnel and facilitators interact with each other and their audience.
As established in previous work, we believe that informal physics programs have many similarities to non-profit organizations [7]. Similarities include, but are not limited to, having the key components of personnel, stakeholders, program, and management; however, some of these terms need to be more appropriately reframed in the context of informal programs [10]. We have adapted these broad categories to an informal context. The six broad categories of factors that affect an organization's performance are: personnel (the people involved within the organization), programs (activities and logistical details), audience (the people the organization is reaching), institutional connections (the entity that the program is housed under, like a university or national lab), resources (financial and physical utilities), and assessment (self-evaluations by the organization) [10]. By contextualizing organizational theory in this space, we aim to create a multidimensional evaluation of informal physics programs.

III. CASE STUDY
In this paper, we focus on a case study of an open house style event held by a program we will refer to as "House Physics", which is housed in a large western R1 university. Site visits are more resource exhaustive for the researcher than interviews/surveys. We are taking a case study approach to guide our site visit methodology for future site visits. Before the site visit, the lead facilitator of House Physics, who is a physics faculty member, was interviewed by a member of the research team about program goals, basic logistics, activities, history, and recruitment and training of student volunteers. This interview was a little over an hour in length and had been previously analyzed for themes with the organizational theory constructs [7].
House Physics is a traveling physics outreach program that visits schools with hands-on experiments that undergraduate interns built. We chose this program as a case study for several reasons. First, two of the House Physics co-directors had participated in an earlier phase of the broader study and contributed interviews about House Physics. (In subsequent data collection, we split the single long-form interview into a combination of a survey and shorter interview) [6]. These interviews had been analyzed with a version of the organizational theory framework and a preliminary list of sub-codes were developed for each of the six main categories. From this analysis, it was clear that House Physics is a robust organization: it has a large number of university students helping with the program, it has long-term institutional support, and it undertakes many educational physics activities each year. However, its large size and scope as an informal physics organization made it difficult to determine a holistic picture of its organizational features from a few interviews alone. We determined that a site visit would allow us to explore the nuances of how, for example, volunteers interacted with audiences or with the program director, or the role of physical resources, like equipment.
House Physics holds a free annual open house event on the university campus. We decided to use this event and a subsequent planning meeting as the key events to collect data from on the site visit. The event consists of several ballrooms filled with hands-on science experiments, demo presentations, fundraising sales booths, and spaces for their science partners. The open house is advertised to the general public and has an attendance of about 8,000 to 10,000 people. House Physics has three co-directors who are physics and astronomy instructors at the university, two additional employees, and about 15 undergraduate interns. Some of the undergraduate interns are physics majors, but most of the volunteers are from a variety of other academic fields. For the open house event, there are about 50 volunteers who were part-ners or former interns, as well as approximately 125 undergraduate volunteers who were the current students of the directors' physics courses. Another important consideration is that the first author of this paper was recently a former House Physics undergraduate intern with the program. His close acquaintance to the House Physics co-directors and many of the House Physics volunteers allowed him to have access to the program events, behind the scenes activity, and have rich conversations with the program personnel.

IV. METHODS
For the site visit, the first author embedded with the House Physics group for two full days. The first day was the open house event and the second day was one of their weekly preparation days. He observed and took detailed field-notes of the open house events, including the opening meeting, volunteer training, and demo presentations. Similarly, he documented the prep day activities, including meetings, experiment building, and video production. Notes included observations of the social dynamics and interactions within the program, descriptions of informal interactions with personnel members, and documentation of programmatic and logistical details [11][12][13]. Photographs of the events and other artifacts such as flyers and program maps were also collected [11][12][13].
He also conducted semi-structured interviews with two of the co-directors using an interview protocol based on the broad organizational theory categories during the visit. One of the directors, who identifies as a white female, also serves in an astronomy teaching role at the university. The other director, who identifies as a white male, also serves in a physics teaching role at the university. The interview with the latter director is a follow-up to an interview originally conducted prior to the site visit.
Data analysis began with a single coder coding the fieldnotes and interview data in MAXQDA, using the thematic categories developed for the overarching landscape project: personnel, program, institution, resources, audience, and assessment [6]. In addition, the coder explored emergent subcodes and themes in just the field-note data, later discussing with the research team to consolidate into more concise labels. The developed sub-codes were compared to those developed for the pre-site visit interview data to provide insights on the additional dimensions that a site visit brings to understanding a program.

V. FINDINGS
From our analysis we have two main results: 1) we found over a dozen sub-codes describing key aspects of this program that were otherwise undetected or unverified via interview with the lead facilitator, largely in the category of personnel, 2) we observed a number of instances where the theme of 'personnel' overlapped substantially with other cat-egories, such as 'resources' and 'audience', demonstrating the complex interwovenness of these categories.
The sub-codes generated from the interview and survey data tended to be more programmatic, such as learning about a personnel member's responsibilities, the physics content topics, or funding. The site visit, however, revealed the ways in which informal physics programs function as a social enterprise with respect to the main themes. For example, subcodes emerged as to how personnel interacted with each other and how the program and personnel utilized available resources. We found that the majority of our emergent subcodes centered around 'personnel'. Here we describe some of the key personnel codes in more detail.

A. Emergent Themes in Personnel
One aspect we are interested in is the social dynamics within informal programs. In our interview with the House Physics lead facilitator, he talked about how the team dynamic plays a role in guiding the creative agenda and identity: "The other thing which we do is we definitely are a team. When we meet we just meet around a table, and mostly we don't have presentations, mostly we have just brainstorming, like okay, here's what we have in mind for this year, what kind of projects would people like to see? What kind of focus would we like to have?...So they're really very much part of the team, and I think they get that. I want them to really own it." In the existing codebook, that would be coded as a sociocultural aspect of the personnel, but there are limitations in what this tells us. From the site visit, we saw more specific details that could not be extracted from the interview. We were able to witness program facilitators repeatedly recognizing personnel members. For example, the facilitators mentioned several times in multiple group meetings to the undergraduate volunteers that they are the ones who make the program successful. Sometimes, recognition was given on a more personal level, such as when an intern made a new physics demonstration, and the group gave praise toward it, and the lead facilitator asked if they could use it for their event. There were repeated actions like this observed that inspired a potential new code regarding the recognition of personnel by the facilitators.
Another new type of code also emerged by observing personnel on site. By sitting in on the weekly prep meeting (as described above by facilitator) and by observing personnel interactions throughout the activities, we observed multiple group aspects such as socializing and working in groups as emergent sub-codes that were otherwise absent or minimized in the interview process. For example, there were moments where personnel members were sitting around a table building a demo together while also laughing and sharing casual conversations about non-work related topics.
When the researcher had direct interactions with the program personnel, the volunteers were willing to converse in informal conversations, often with personnel members initiating the interaction. These interactions provided different data points than what could be extracted with a long, formal interview. They were brief and more conversational, which brought in a broader range of perspectives from within the program and allowed for much of the discussion to be driven by the members' point of view. This led to additional emergent sub-codes such as reflecting on past experiences they had in the program. From both outside observation and oneon-one casual conversations, many members reflected on past events they participated in, how those experiences stuck with them, and how that influenced their future goals and aspirations. One former intern talked about how the skills they learned from building demos transferred over to their current job. Another volunteer talked about how they decided they wanted to pursue education after participating in the program.

B. Overlapping Codes with Personnel
We also observed multiple cases where emergent subcodes overlapped personnel and another of our themes. One of the most prominent that we saw was the relation between 'personnel' and 'resources'. With the current interview codebook, there is a broad label for the personnel's role in the program, which refers to the member's tasks and duties. However, this code does not give insight into how the personnel complete these tasks. During the site visit we found an abundance of personnel members interacting with physical resources and were able to observe how these interactions take place. These interactions manifested in many different forms. We observed the personnel members transporting, setting up, and taking down event projects. Repairing projects was an activity that volunteers engaged in during the open house and the general workday, as well as interns creating and building their own projects. From the interview with the director prior to the site visit, we extracted information on personnel roles and physical resources as separate categories.
Another crossover theme was between 'personnel' and 'audience'. From casual one-on-one conversations and sitting in on group meetings, we observed instances of personnel members recounting interactions that they had with the audience. For instance, in the weekly prep meeting that occurred after the open house, the personnel members reflected on their experiences at the event, with several members recalling interactions with families and other visiting groups. One volunteer talked about the positive conversations they had with a senior citizen group that attended the open house. From an interview, the personnel and the audience are looked at almost as separate contexts, but the site visit helps show the overlap.
The information we can extract from the interviews on the interactions between these two categories are broad and general, so the codebook is limited in identifying these overlaps. Such limitations make it more difficult to fully describe the program characteristics. Direct observation during the site visit allowed for codes across multiple themes to appear connected in a larger context. From the site visit, we observed how the personnel interacted with the audience, such as through casual one-on-one conversations. We also have an emergent sub-code of the personnel members reflecting on their interactions with the audience and sharing these stories with other personnel in both socializing and work meeting settings. We can see the interaction of the personnel and resources, but also the social interactions between members, where a lot of these emergent sub-codes seemed to blossom.

A. Benefits of a Site Visit
In this paper, one goal was to test a methodology for conducting site visits to informal physics programs. In our other work, we found that interviews with lead program facilitators can provide long, detailed information of the program in about an hour time frame [6]. However, since the operation of a program is complex and we are limited to the single perspective of the facilitator, these nuances are not readily apparent and may serve as confounding factors in the success and challenges of these programs. Taking an ethnographic approach for the site visit allowed for a large amount of rich data to be collected. The facilitator interviews present more of 'what' the program does. What are the key points that we know about the program? We can learn what the top-level programmatic details are, what the main roles of most of the personnel members are, who the majority of the audience is, and what the funding sources are. Through a site visit, we are able to see more of 'how' the program does all these things and how these themes connect with each other on a much finer scale. We are able to have brief, informal conversations with other personnel members to get their perspectives on the program, their role, and their experiences, in a more natural setting than with a longer, formal interview. In future research, we can more carefully analyze the importance of these codes and what role they might play in the success and sustainability of programs, even seeing if this occurs in other programs.
The key takeaway in this work is that we were not able to get these types of data from doing a singular, formal interview with the lead facilitator. Arranging longer, more formal interviews with as many personnel members as we had casual interactions with during a site visit is not easily feasible, especially for larger programs. The interview data allowed for a deep look through a single perspective at each of the overarching themes that we have established. The site visit allowed for many perspectives on just a few of the overarching themes, in this case, primarily 'personnel' with previously unseen overlaps with themes such as 'audience' and 'resources'. This finding suggests that neither formal facilitator interviews nor site visits should replace one another when studying informal physics programs, but rather these two ap-proaches are best suited to gather different types of data that complement each other when studying these programs.

B. Limitations with Data Collection
There are some challenges that arise with conducting a site visit, the most prominent being the actual data collection. Informal physics programs vary considerably, so the ease of information collection and the type of information can vary. For an all day event with over 100 personnel members and 10,000 visitors spread throughout an entire building, it is not feasible for one person to document every detail.
With a large scale event like this program, there appeared to be minimal interruption in the flow of the event as the research could sit in the background and blend in with the crowd to observe. From the lead researcher's prior experience with the program, the event appeared to run naturally like past events, giving little indication that the program or the personnel were acting more rigid while being observed. The invasiveness of the sight visit should be handled with care. Site visits to smaller programs could potentially have higher risk of disrupting the event. Another possible limitation could be with visiting programs that we are less familiar with, which could impact data collection.
There needs to be a plan beforehand on key aspects to attempt to observe. Having interview data before the visit helped to highlight key details to look out for that the facilitator deems as important aspects of the program, such as how people share in a group meeting. However, there also needs to be flexibility in the plan for the researcher to appropriately adjust to the active surrounding environment to collect crucial data as they appear. During the open house, there were several times that the researcher was moving from one location to another to observe, and was stopped by personnel members who wanted to discuss their experiences in the program. While the original goal in that moment was to observe audience interactions, it was difficult to pass up the opportunity for a brief conversation with a participant willing to reveal their rich perspectives on the program. There needs to be a balance between the researcher following their plan to observe specific components, as well as being flexible enough to react in the moment to what is happening around them.
In future work, we intend to conduct more site visits to expand on these themes and enhance the site visit methodology. We will also refine our emergent sub-codes to expand our codebook through rigorous analysis and discussion. The more finely tuned codebook will be used to compare more thoroughly with our interview data to aid in examining key themes and subtle nuances within these programs. In the broader project scope, we aim to find indicators that most impact the sustainability and success of informal physics programs. We hope this methodological process helps those interested in examining their own learning spaces consider how their data sources can work together for more holistic understanding. This work is funded by NSF AISL #1423496.