IPER programs: a narrative framework for program development

In this paper, we describe both propositional and narrative frameworks to design and evaluate IPER programs. While a propositional framework provides a way to report on outcomes and is of interest to funders, a narrative framework provides insight into the lives of students and how a program fits into their life story. We have found that listening to students and understanding their lives provides insight into how to design a program that keeps students engaged, growing, and returning to the program year after year. Understanding the student narrative is essential and precedes the design of effective STEM programs.


I. INTRODUCTION
For several years, we have been exploring strategies to design and evaluate a STEM based urban education program for middle school students called the Saturday Urban Scholars Program [1][2][3]. Narrative and propositional frameworks are important to design programs, describe their effects, and make sound judgments about them. Jerome Bruner differentiated narrative and propositional frameworks for reasoning as an effort to direct attention toward the lived experiences of individuals when examining their psychological development [4][5][6]. Tappan and Brown reviewed some of the disciplines that were making use of the distinction including philosophy, history, and several areas of psychology [7]. In a crossing of the two frameworks, narrative analysts (e.g. Daiute and Lightfoot (2004) and Josselson and A. Lieblich (1993) [8,9]) have demonstrated how "stories" can be used in scientific investigations to improve research and promote a researcher's understanding of different phenomena.
Though readers may be inclined to subsume a narrative framework into a type of scientific study (such as "ethnography," "critical ethnography," or simply "qualitative research"), like Bruner, we view them as separate and irreducible ways of thinking. The propositional framework is scientific and formal, leading to generalizations. The narrative framework is intuitive, inductive and experiential. Its function is to understand or explain the reality of how each unique participant lives and explains their lives. Narrative functions for all of us as a tool to make sense of our own lives, including our lives as they connect to others.
Program design and program evaluation are constructed differently when viewed from narrative and propositional frameworks. On the one hand, using a propositional framework, designers and evaluators employ scientific methods including instrument design, data collection, and analysis to plan and investigate their program. They strive to create programs based on a preconceived model that is likely to have generalizable effects, and which can be replicated by others who follow the same procedures. An important goal is to contribute to scientific knowledge by gathering evidence on important questions, establishing protocols for answering these, then objectively reporting on the outcomes. The program is considered a treatment that is expected to produce some effect on its participants.
On the other hand, through the lens of a narrative framework, a program is always unique, changing over time, and influenced by the context and culture in which it operates. In this sense, we can think of a program as a living entity with a history of its own, and many stories available to define it. We can also consider how a program is a shared space in which participants, mentors, designers, and evaluators tell their stories about their overlapping experiences. Using a narrative framework means that we try to understand the people in it, their perceptions, what the program means to them, how they fit the program into their stories, and more broadly, how it fits into their lives. This sometimes means that designers need to set aside research goals to prioritize participation, self-representation, and agency in the moment as the program unfolds day-to-day.
We have found the distinction between narrative and propositional thinking useful in thinking about the Saturday Urban Scholars Program, a STEM based out-of-school program we have designed and evaluated for inner city youth in New York State [2]. The school district partner is classified as a high-needs and low-resource-capacity district with 58% of K-12 students considered economically disadvantaged. The Saturday Urban Scholars Program (USP) is complex in its design. It is an informal STEM program that operates on the campus of Siena College for 14 Saturdays during the academic year. Students are bussed to campus, participate in two classes and eat lunch at the cafeteria. The program includes elementary and middle school students (grades 5-8) working with experienced high school students, who participated in the program for multiple years. College students from a variety of academic majors have multiple roles in the program. Some assist experienced STEM educators as mentors for middle school students. Some participate for several years leading them to develop and teach classes in the program. Though the majority of experiences are STEM-based, we include some experiences in arts and humanities. This helps to draw a diversity of students, including those who have an interest in the program, but also have some trepidation about the STEM focus.
In the next section, we distinguish the two frameworks in further detail with examples. We include the propositional framework for the purpose of drawing a contrast from what is typically done, to a different framework that we think is especially useful in program design and evaluation. In the remainder of the paper, we elaborate on the narrative framework, how we have applied it, and how this has led to important features of the program. We describe how a narrative framework involves paying attention to students, connecting to their selfrepresentations, and creating a climate where they feel safe to enact and share their stories.

II. PROPOSITIONAL FRAMEWORK
A propositional framework has helped us think about whether the program has any general effects, how it can be replicated, and whether its funding agents should continue to support it. We used scientific processes to gather data, analyze it, and generate conclusions in an objective manner such that others can either replicate the process, or replicate the investigations. As a result we have documented several surprising and encouraging effects [1]. After approximately 20 sessions, participants differ from nonparticipants in several ways: • middle school students that participated in Urban Scholars have greater pass rates in middle school Math (25%) and English language arts (32%) assessments than nonparticipants in Math (11%) and English language arts (16%) assessments, respectively, • Urban Scholars' students passed New York State High School Regents Exams at a higher rate (68%) than nonparticipants (51%), • those completing the program are more than twice as likely to not only continue in STEM-related high school courses, but also take advanced courses (i.e. AP courses), • participants have higher graduation rates (93%) than nonparticipants (62%) for the two years in which data are available (2016-2017).
Since we do not teach academic content to directly impact test scores, we were surprised when district data show possible impacts on student ELA and Math scores, and later on Regents exams and graduation rates [2]. We have searched for explanations to rule out the program as the mechanism -selection effects, systemic differences with control groups, etc. So far none of the analyses have sufficiently explained the differences between nonparticipants and participants during or after their involvement.

III. NARRATIVE FRAMEWORK
A narrative framework has helped us think about the lived experiences of participants and stakeholders. It provides different sorts of insights that are useful when we are trying to understand, explain, or make sense of what's going on. It promotes understanding of the participants and the program itself. It is operating when we are asking, what's happening here -what does it all mean.
We began to think about this framework as we reflected on our efforts, and decided that our success did not derive from our effort as investigators -in formulating a research-based model, in designing protocols, in creating valid and reliable surveys, in collecting and analyzing data, etc. Indeed, scientific studies of the program have emerged only recently [1][2][3]. We're only now designing research studies to delve into the reasons our program is successful. We think the success of the program has derived from several lucky, happenstance moments -times when we stopped, looked and listened to kids, and learned from them. We have been successful in part because we have somehow become a part of the stories kids tell about their lives and capitalized on this understanding to modify the program to be relevant to their interests in real time.
Calabrese Barton's experience is illustrative of how a narrative framework operates in program design [10]. She developed a connection with two girls living in a homeless shelter who also attended an out-of-school STEM experience at the shelter. She learned that the girls were intentionally silent in class while in school as a protest that the teacher didn't get to know them and to let their teacher know that they were bored. The girls were taken out of science class for ELA remediation, because the teacher mistakenly believed they were struggling. In contrast, Calabrese Barton got to know the girls and learned that they saw themselves as confident, intelligent, hard working and creative. Calabrese Barton allowed the girls to co-create her research design and data collection based on what she learned about them through their stories. This also led to unplanned changes in the afterschool program.

IV. UNDERSTANDING PARTICIPANTS
Attending to student lives can lead to important lessons about their views of the program. These lessons do not derive from formal processes in the way they do from a propositional framework -where researchers typically begin with predefined research questions, or interview protocols, and methodologies for the analysis of their data. Rather we are "catching the wind" so to speak. We are listening to their points of view as we build connections with them, and as we create experiences that fit with their ways of representing themselves. A point we hope to show is how informal, nonscientific processes -just ordinary life in the program -has led the program to be what it is.
We will use our experiences with three particular students as an example of their uniqueness and their variety in approaches to life in the program.
Sasha: Sasha entered the Saturday Scholars program as a 6th grader. She told us that her morning robotics class was too difficult and we often saw her with her head down on her desk, not too engaged with the activities. She seemed to enjoy going to lunch in the cafeteria with her friends, and she said that she loved her afternoon art class. Our records show that for the first two years, Sasha chose as many art and cooking classes as she could. However, in 8th and 9th grade, she chose mostly STEM classes. Later, she got involved in helping us create new experiences for other participants by testing out lessons using littleBits circuits and serving as a mentor in that class. She brought a younger cousin to the program. She said that she wanted her cousin to have the same experiences she had.
Malcolm: Malcolm entered the informal STEM program in 5th grade and participated for 5 years. He was a quiet student and always fully engaged in the classes and helpful to others. In a conversation, he mentioned that the program was a completely different environment compared to school. He described it as a place where he could be "my true self," and where he felt his "effort was valued." For example, we have a training video in which he explains, how the program creates a different space than what he experiences in school. He thought that project leaders, mentors and students held much higher expectations for each other in the program than he experienced at school. Malcolm told us how he learned things in the program about science and engineering that were never discussed in school. For example, he said in the video that "When I did a class here, it was called quantum physics and I did it and I wanted to do that for a little while. Physics can be fun, like it was in that class." He described the project as "really hard, but fun, too." He also really loved the history in Minecraft class and wished we would run it again so he could be a mentor because "I'm really good at history, too." Carlos: When Carlos first started in the program as a 5th grader, he signed up for Open Source Gaming. In the class, he seemed frustrated, picking fights, bored, and seemed to intentionally get on people's nerves. When the project leader talked to him at lunch, he had lots to say about video games, which was his reason for picking the class -they talked in detail about characters / characterization and organization of games he played, how music in a game can generate attention, etc. These were deep, complex discussions that suggested to one of our college mentors that he shouldn't be defined as trouble -as the problem-making kid he appeared to be in the class. In his next class, FLOSS Desktops for Kids, students dismantle and rebuild a recycled computer. Carlos worked really hard, helped fellow students with their computers and attended every class.
Looking back at our experiences learning about Sasha, Malcolm, and Carlos, initially we didn't know them, their goals, or their reasons for coming. In time, we learned that their goals were different from ours (e.g., they were not motivated to attend to sustain their interest in STEM). Sasha's goals as a younger student were important in getting her to come and keeping her engaged in the program (lunch and hanging-out with her friends, art classes). Over her 5 years of attendance, her confidence and interest in STEM grew. Logically, we wouldn't have included art classes in a program to influence attitudes toward STEM, but we offered them for students like Sasha who's interest in STEM shifted over time.
. Malcolm was happy to join into the science and history experiences, but he did this because he felt these classes recognized him as a smart kid. This was an important part of his own story. To fit into his story, and keep him coming, we had to have high expectations to match what he believed about himself. Carlos was not merely a trouble-maker. He was a young man with interests, skills, and motivations. When we connected with him as a person, listening to his interests, learning about his family, we were able to see the classes from his perspective.

V. APPLYING A NARRATIVE FRAMEWORK TO PROGRAM DESIGN
The narrative framework, or an interpretative process through which we design our program, provides opportunities for students to be co-creators in their learning. As explained by Basu and Calabrese Barton, incorporation of "funds of knowledge" in academic instruction provides "strategic knowledge and activities essential for achieving the goals a student has for his/her out-of-school life." [11] We implement this approach by understanding the interests of students outside of school and incorporating STEM topics into those interests. This allows our program to become part of their narratives -stories that they are excited to share with their friends and family.

Gaming
The addition of gaming to our program design came about gradually as we listened to students and added experiences, specifically Minecraft, to teach STEM. Gaming is an important interest in many young students' lives. Typically parents and teachers see gaming as a waste of time and maybe offer it as a reward once other work is completed. Kids were obsessively talking about Minecraft -on the bus, at lunch, in their classes. We listened, and decided to give it a try -maybe we could connect it somehow to science content, maybe we could connect it generally to "problem solving" or some other generic goal. As we learned about the game, we got better at connecting it to valuable content in science, while keeping it fun and engaging for students. In another paper, we describe how "Serious Gaming" is an important aspect of our program design [2]. However, we didn't start from "Serious Gaming" methodologies to applications in our day-to-day program. We started with adjustments in the program to connect meaningfully to students lives, and only later found a relevant scientific model to attach it to. Using Minecraft to introduce concepts in science, we were able to validate something important and valuable to our participants, and we've become a part of their lives and their story in a way that neither parents nor teachers in schools typically do.

Choice
Student choice has been a theme when participants talk about the program. They get to choose 2 classes out of 10 options for their morning and afternoon classes. They make their decisions by asking questions about the classes and they talk with their friends about their choices. When there's room, we allow students to switch classes if they change their mind. We also offer new classes every 4 to 6 weeks, so that students know that if a class is full or we're not able to move them, they'll have the opportunity in the near future. Because our program is not school and their attendance is voluntary, we know that students make decisions about whether to attend each Saturday because the program is fun and they get to be with their friends, but also because they have control. By giving them control and taking their interest and choices seriously, we are reinforcing the message that we work together in this informal STEM program, they're an equal partner, and we trust and value their participation and suggestions. From the feedback, we believe this makes the students feel valued.
Changes to the program happen regularly based on student interests and requests. A number of examples are provided in Table I. The table includes the name of the class and a short description, the request, concern, or suggestion of a student or students, and changes we made in response.
Using a narrative framework, we listen to students to create a flexible and dynamic program. We create a positive climate, similar to the idea of counterspaces for marginalized students in STEM, as described by King [12]. We provide student choice about the classes they take, recognizing each student as unique as described by Calabrese Barton [10]. We provide variety and novelty of classes to provide many opportunities to activate students' "funds of knowledge" as described by Basu and Calabrese Barton [11]. Finally, by incorporating gaming, we accept and recognize gaming as a legitimate activity and create a community that's important to their lives, similar to intersectionality in STEM education as described  [13]. Here are some things we have learned by applying a narrative framework in our program design and program evaluation efforts.
1. We need to understand them, and their lives, to promote their involvement; e.g. our goals differ from theirs, and their goals are very important in promoting their involvement; 2. Students prefer programs that bring novelty and variety; 3. Placements with choices, are preferred to placements without choice; 4. They expect to be seen and represented positively -as learners with great potential, as good people, as individuals who want their "true self" to be valued; 5. Usually, when a kid is creating "problems" -there is a problem with the program, not the kid; 6. Our experiences often differ from theirs, and our assumptions are not always valid for them.

VI. DISCUSSION
To put our program into context, our informal STEM program grew out of a humanities-based enrichment program administered by our education department. About 10 years ago, the first author joined the team, gradually added STEM content and over time the program shifted to mostly STEM activities.
Besides a survey on interest in STEM, we weren't conducting research on the program. Merely administering the program was a huge time commitment. In 2017, we were able to work with the school district to obtain aggregated school outcomes of our students as compared with students in the district. We were pleasantly surprised, to put it mildly, to find that we had significantly positive school outcomes. This is leading us to design qualitative research studies about students in the program to better understand how the program is impacting students, but we haven't conducted these studies yet.
So what can we offer to the PER community? This paper provides guidance about how to design an effective informal STEM program. The creation of a program has to happen before a research study can be performed. A program has to have some success as well, to find it valuable to study. Based on the positive outcomes of our quantitative study, we are just starting to design qualitative research studies. Therefore, we do not report here on a qualitative study that answers research questions. It may seem that way as we reflect back on how we designed our program and provide some anecdotes to show that each student enters and engages in the program uniquely. But this information was not gathered through a qualitative study. We are reporting on the program as we remember it and through conversations our staff have had with participants.
This paper contrasts the design of a program through a propositional framework and a narrative framework. We believe you need both for an effective program and we believe that once the STEM goals are decided through a propositional framework, a program is successful by implementing a narrative framework to understand the participants and make changes to keep the program relevant to their lives. For example, using only a propositional framework, one may model their program after a successful informal STEM program chosen because the curriculum is well developed and the program has reported positive outcomes. On the other hand, using a narrative framework, one may create a program that is completely student driven.
With only the propositional framework, it may be difficult to attract and retain students. By neglecting the narrative framework, you may never understand why students don't come to your program or why they don't stay. With only a narrative framework, lots of students may attend the program, but they may never have the opportunity for exposure to STEM content if students aren't self-motivated in STEM. With both frameworks activated, there is an opportunity to infuse STEM content into the lives of the students.
To summarize benefits and drawbacks of propositional and narrative frameworks: 1. Propositional -provides evidence of a working program but doesn't connect with the lives of the participants; 2. Narrative -connects to students' lives but may not meet STEM goals, i.e. students may request all arts or humanities classes 3. Both -staying true to both frameworks makes the STEM content relevant to student lives and stories

VII. CONCLUSIONS
In closing, we are not advocating for a happenstance approach to program design, which rejects scientific processes to inform what we do. Instead, we hope that program narratives, and a narrative framework for thinking about programs, can be discussed in greater detail, and given recognition as important influences on program design. Program narratives can be an informative area for scientific investigation. Acknowledging the narrative framework in the process of program design can bring scientists closer to the reality of the program design process.