The long-term effects of learning in an ISLE approach classroom

This study reports on the long-term effects of learning through the Investigative Science Learning Environment (ISLE) approach on high school physics students. Students learning physics through the ISLE approach engage in collaborative activities that mirror the practice of physicists to construct concepts and apply them to solving practical problems. Previous studies of learning in ISLE-based classrooms showed that the students develop abilities to design experiments, collect and represent data, communicate and many others. However, there were no follow up studies that demonstrate what the students remember and how they use those abilities in their future education and professional careers. We administered a survey to alumni who learned physics in an ISLE-based classroom to find out what they remember many years after taking the course as well as if they find their physics learning useful for future education and careers. We report on the findings from this study.


I. INTRODUCTION
Recommendations from the National Research Council's Framework for K-12 Science Education [1], the American Association of Physics Teachers (AAPT) [2], and the Next Generation Science Standards [3] changed the approach to teaching science to help prepare our students for success in the 21 st century. Instead of focusing only on the development of knowledge, they emphasize learning science by practicing it. One approach to learning and teaching that engages students in the appropriate science practices to be prepared for life beyond the classroom is the Investigative Science Learning Environment (ISLE) approach [4]. The ISLE approach encourages students to learn physics in ways similar to the work of practicing physicists. Simultaneously, ISLE helps students learn to take risks in their thinking, persevere when facing a challenge, and develop growth mindsets [5].
In an ISLE classroom, students consistently work in groups to construct every concept. First, they observe a series of carefully selected experiments (observational experiments). Then, students work together to identify patterns and come up with possible explanations (or hypotheses) for the observed phenomena. Next, students engage in hypothetico-deductive reasoning and design new experiments to test the explanations and make predictions of the expected outcome, based on the hypothesis under test, before performing the experiment. Students then run the testing experiments and make a judgement about their hypothesis. If there is a mismatch between the prediction and the outcome of the testing experiment, they need to check their assumptions and make revisions to their hypothesis before running more testing experiments. If the outcome agrees with the prediction, then students need to design more testing experiments and eventually apply this newly constructed knowledge to solve practical problems.
Students naturally participate in the processes that are at the heart of doing physics. They work with their classmates designing, conducting, and writing up their experiments, as well as to discuss their findings. They have numerous opportunities to revise and improve their work for full credit. Also, an important piece of equipment in ISLE classrooms are white boards: the students use them to develop concepts, collect data, and share their findings (see Fig. 1, Fig. 2).  Engagement in these reasoning processes provides learners natural opportunities to develop scientific abilities. Scientific abilities are processes, procedures, and methods scientists use when constructing knowledge and solving problems [6]. Among the abilities that students develop: 1. representing information in multiple ways, 2. designing and conducting experiments to investigate phenomena, 3. designing and conducting testing experiments, 4. designing and conducting application experiments, 5. communicating scientific ideas, 6. collecting and analyzing experimental data, 7. evaluating models, equations, solutions, and claims. To lay out the expectations for each scientific ability, the Rutgers physics education research group developed a set of rubrics [7].
The development of scientific abilities takes time. Students often view learning in this manner as unfamiliar and frustrating, as they are being held accountable for their own knowledge construction [8]. As a result, students need repeated engagement with the abilities they are expected to develop. While studies conducted at the high school and college level found that students are capable of developing these scientific abilities [9,10], as researchers we are unaware of the long-term effects of learning physics through this approach for students. We need to know if the ISLE approach helps students learn how to develop the mindset necessary to reason independently and solve open-ended problems before starting their careers. The goal of this study is to find out what former high school ISLE students remember many years after taking the course and whether they find their physics learning useful for future education and careers.

II. METHODOLOGY
The study used an explanatory sequential mixed-methods approach. First, we sent out a cross sectional survey to alumni who were enrolled in ISLE-based physics courses while in high school. Later, we conducted follow up interviews with nine participants and a focus group with four participants to validate the survey responses. Interviewed alumni were asked questions to make sure they understood the items on the survey. Focus group participants performed an experiment that required them to apply their knowledge of the scientific abilities.

A. Sampling
The study was conducted electronically with alumni from one Northeastern United States public high school where ISLE was implemented in the classroom for ten years. We sent e-mails with a cross-sectional survey to 627 students for whom we had valid email addresses. These students were enrolled in ISLE approach physics courses (conceptual, college prep, or honors) with the same teacher and graduated high school between 2010 and 2019.

B. Instrument
The ISLE alumni survey instrument consists of seventeen questions (open ended, rate scale, and selected response) [11]. We established face validity by discussing the survey questions with ISLE experts and educational researchers who found the questions appropriate. Section one of the survey asks participants' basic demographics information: course that they took, graduation date, college major, gender, and future plans and/or current profession. Section two is dedicated to their high school science experience. Section three relates to the scientific abilities; participants are asked to respond to questions about their current level of confidence with many of the scientific abilities they developed in high school as well as which abilities (if any) they still use today. Section four asks participants how they felt about collaborative work in their high school class and section five asks participants if there is anything they would like to add to their responses. Additionally, this final section asks participants if they are willing to be contacted for a follow up interview.

C. Data collection and analysis
The survey was administered through the Qualtrics system. Alumni had six weeks to respond. Out of 627 possible participants, 294 alumni opened the survey (47%) and 205 alumni completed the survey. This yields a response rate of 33% but a completion rate of 70% for participants who opened the survey. The average response rate for external email surveys with more than 100 participants is approximately 25% [12]. The average time for respondents to complete the survey was 12.9 ± 3.4 minutes. Table I contains a breakdown of the responses by year, course, gender, and major.
The response rates for those enrolled in college prep physics and honors physics was comparable to the original population of the classrooms. The conceptual physics response rate was slightly lower (possibly because many of those students took the course as freshman and graduated prior to 2016). While the male and female participants enrolled in the courses were split almost 50/50, more female students responded to the survey than male students. Additionally, more respondents were STEM majors.
We used categorical aggregation [13] to establish themes and patterns in student responses to the open-ended questions. After coding students' statements for patterns, another researcher familiar with the study went through the themes identified and agreed with almost 100% of the coded responses. Responses to the closed-response questions were collapsed into smaller thematic categories focused on elements of the scientific process. These thematic categories are consistent with both the process of ISLE as well as the scientific abilities [4,6].

III. FINDINGS
Themes that emerged from analysis of the open-ended questions that asked alumni what they remembered from their time in physics class include physics content, scientific abilities, motivation and mindset, the learning community, and transfer. Table II contains a breakdown of these primary and secondary themes, as well as the number of respondents and the frequency of occurrence in all student responses.

A. Physics content
The majority of respondents (57%) recalled some aspect of doing physics in class. Almost half of the respondents were able to recall specific aspects of the curriculum such as motion and forces, or energy and momentum. On average a participant recalled 1.86 ± 1.09 concepts. Responses from female (48%) and male (49%) alumni were fairly evenly split (p = 0.999). Honors physics students and declared STEM majors recalled the most content. Many of the participants also made connections between the class content and their lives beyond the classroom. One student recalled how they first learned about some of the experiences that they encounter in everyday life.
"I actually remember so much about the class and I talk about it all the time. I remember making the mobiles. I remember learning that when opening a door, the closer to the hinge you are the less distance you have to go but the more force you need. Little things like that really stuck me and I use them/think about them during everyday life. The way I think as a whole coincides with this."

B. Scientific abilities
Within the open-ended responses, almost 50% of participants mentioned some element of the scientific abilities, even if they did not use formal language. Fifty nine percent of the responses were from females and 38% were from males (p = 0.02). The majority of participants recalled elements of doing experiments and thinking like a scientist as part of the classroom culture. One alumnus recalled "having to take creative approaches to solve problems (manipulating equations, drawing diagrams, applying laws/rules to specific scenarios)." Another remembered learning "how to account for reaction times and other variables to ensure better trials during experiments." Several of the participants took away a newfound appreciation for the scientific process. For example, one participant reflected on her growth as a science student from lack of confidence to the belief that she is able to do science.
"…Through observation, inquiry, experimentation, [physics] class was the first time we delved into scientific processes pedagogically (even though I didn't know that's what was happening at the time)…I began to realize that I was not bad at science‚ the way other science classes were taught were perhaps not the best way for me to learn. I also began to realize that I liked the processes even if the content didn't immediately excite me." Alumni responses were consistent when comparing participant's ranking of individual confidence in each scientific ability (see Fig. 3) with the open-ended questions. The majority of favorable responses focused on elements of experimental design including making observations, posing a question, and looking for patterns in data. In addition to feeling confident with these scientific abilities, participants find themselves still engaging with these scientific abilities in their present schooling/career. When asked to choose three scientific abilities they still actively use today, the majority of participants selected abilities that encompass elements of experimental design.

Sub-Theme Student Quotes
Metacognition I learned the importance of understanding concepts, big concepts. If I try to fully grasp a concept and constantly apply it to a problem, I can solve any problem.

Persistence
Not doing well on something the first time around doesn't mean you won't ever do well. It's valuable to revise and learn from mistakes.
Growth I really learned how to approach problems from a logical mindset, how to admit to myself that I'm not good at something, how to reach out to people for help.
Learning how to fail I learned how to fail. Doing really bad in a class for the first time pushed me to ask for help and find new ways to learn things.

C. Motivation and mindset
Fifty five percent of participants (of those 59% were females and 40% were males, p = 0.05) made reference to some type of motivation and/or mindset they recall from their time in physics class. The majority of these responses came from declared STEM majors who took honors physics. Elements of student motivation and mindset broke down into four sub-categories: metacognition, persistence, growth, and learning how to fail (see Table III for sample responses).

D. Learning community
One of the key elements of the ISLE approach is the development of a learning community. Forty five percent of alumni mentioned some aspect of the community as something they remembered. Female students accounted for 60% of the coded responses (p = 0.03). Unlike the other three thematic categories, the students enrolled in the college prep physics course wrote the majority of the responses coded at this theme (47%). The quote below is from a student who took physics eight years ago: "Feeling at ease in the classroom where all students were accepted and treated well made my experience in class much more positive. I'm sure it also significantly improved my learning and retention, since I could relax and focus while in class."

E. Transfer
Thirty eight percent of participant responses to the openended questions mentioned aspects of learning that we classified as either Direct Application (DA) or Preparation for Future Learning (PFL) transfer [14].
DA themes included how physics helped prepare the participants for courses in college, productive study habits, and critical thinking abilities directly related to STEM fields. STEM majors who took honors physics in high school contributed the majority of responses related DA transfer (males and females evenly split). For example, one student spoke about how class content helped her explain physics to others. "We learned about non-inertial reference frames. This became a fact to share in all elevators and train cars." PFL themes included how physics class helped participants develop metacognitive, critical thinking, communications, and collaboration skills. The distribution of responses between the college prep physics and honors physics and STEM and non-STEM majors were closer for PFL transfer than they were for DA transfer. Additionally, over half of PFL-coded responses came from female participants. As one student said: "[Physics] allowed me to improve my abilities regarding critical thinking and logical problem solving which previously were some of my biggest weaknesses and have now turned into some of my strengths."

IV. DISCUSSION
In this study we explored how learning in an ISLE approach classroom helps students build the habits of mind necessary to productively contribute to their future careers. In their responses, participants recalled aspects of the course related to physics content, the scientific abilities, their motivation and mindset, and the learning community. With respect to the scientific abilities, alumni felt the most confident with the elements of experimental design that are prevalent in the goals of the Next Generation Science Standards and AAPT recommendations. Additionally, because the course provided opportunities for students to challenge themselves in a safe space, alumni shared how they were able to focus on growing their metacognitive abilities instead of worrying about how failure would impact their performance. Those participants showed evidence of embodying growth mindsets [15,16]. These findings are consistent with studies conducted at the secondary school level that found that supportive teachers and the classroom culture influence student engagement in science; particularly for female students and those who have found themselves in non-supportive science cultures [17,18].
Based on the analysis of the responses to the survey (with validation of the responses through interviews and a focus group), our findings are supporting evidence that the ISLE approach is successful at meeting the goals of helping students develop a broad range of abilities and skills that are applicable across many fields. However, this study is not without limitations. Due to the availability of contact information for former high school students, we were unable to administer the survey to non-ISLE approach physics alumni. Including a control group in the study would have provided insight into what non-ISLE approach alumni took away from their high school physics experience. Additionally, while a 33% response rate is higher than average for external e-mail surveys, we are curious how the findings might differ if we obtained feedback from additional alumni.