Physics students’ reactions to an abrupt shift in instruction during the COVID-19 pandemic

We present a preliminary analysis of the effect of the COVID-19 pandemic on students in the context of a large enrollment introductory physics course. During the spring of 2020, ours was one of thousands of courses forced to change abruptly from face-to-face instruction to online delivery. We report the effects of this change on students through the lenses initially available to us, including our online homework system, an online forum, and the course evaluations. While preliminary, our results suggest that students were generally unhappy with the transition, but this disappointment did not translate into significantly reduced effort or success. Their primary concerns were not technological but stemmed from behavioral considerations either internal to themselves or external, from the instructor. Although the event probed here was the pandemic, our observations and conclusions may be applicable to other situations in which instruction changes suddenly due to natural or human-caused disasters.


I. INTRODUCTION
During the spring of 2020, the COVID-19 pandemic forced extraordinary changes on educational systems at every level. As such, it presents researchers an opportunity to gain insight into how students and faculty cope with abrupt changes in educational practice. It also presents huge difficulties for researchers. We cannot create control groups, compare methods, or take a measured approach to creating and validating instruments. This paper presents an initial effort to extract meaning from the data available. It is the author's hope that other groups will perform similar analyses, and that the net result will be a clearer understanding of student and faculty responses. Although this pandemic is unique in the last century, it is not unprecedented [1], and similar pandemics may recur. Further, other crises such as flooding, wildfires, and war may cause similarly extreme changes in educational practice, over a variety of time and length scales.
As the pandemic unfolded, and schools at all levels shifted to online learning, there were many concerns that students would be demotivated, and that large numbers would drop classes, fail to complete work, or otherwise suffer academically [2][3][4]. Many of the concerns centered on students who were already disadvantaged financially or were members of minoritized groups. In both cases, such students are common at public, urban universities, including IUPUI, where this study was conducted.
The COVID-19 pandemic raises a host of interesting questions in this context. How effective are the technological choices made (or forced on) students and faculty? What aspects of a course can be mapped onto the new situation effectively? How will students' motivations be affected? Are the changes exacerbating racial, ethnic, and gender-related issues? In time, many of these will be addressed. In this work, however, we address three key preliminary questions: Issue 1: What, if any, changes were evident in students' efforts and success? Issue 2: What, if any, changes were evident in students' attitudes? Issue 3: What issues did students identify as causing the greatest difficulties?

II. CONTEXT
The setting for this work is a calculus-based introduction to mechanics at a public, urban university in the United States. The majority of the students are engineering majors, and the majority commute to campus. The class makeup is approximately 25% female, 6% African American, and 9% Latinx. The author was one of several recitation instructors, but was not the primary lecturer. The primary lecturer has had that role each spring semester since 2017, so we have three prior semesters to which we make direct comparisons. At the outset of the semester, 190 students were enrolled in a single large lecture, five labs sections and five recitation sections.
The course uses Peer Instruction [5] and Just-in-Time Teaching (JiTT) [6] in the lectures and group problem solving in the recitations. The primary instructional technologies used to support the course were MasteringPhysics [7] for homework and Tophat [8] for Peer Instruction and responses to group work in recitation. Tophat is a phone app that takes the place of traditional "clicker" technology. One advantage to this technology is that it can be used as long as the students have wireless or cellular signals. As a result, we were able to continue to use Peer Instruction online. The text by Young and Freedman [9] was linked as an electronic text through the course LMS, Canvas [10]. We also used an online discussion forum, CourseNetworking (CN) [11].
The LMS was used primarily for its gradebook function. CN was used for all day-to-day course communications, including making announcements, distributing handouts, etc. Students were encouraged to use CN to communicate with one another, to seek help from one another, to share tips for success in physics, and to exchange general thoughts and feelings on the course and student life.
Recitation sections were based on informal group work. Problems were projected on screen, and students worked in self-assembled groups of 3-4 on whiteboards mounted around the room. Each problem concluded with 2-3 multiple choice questions. Students answered these using the same system (Tophat) used for Peer Instruction in the lecture. The hour exams and the final were traditional paper-and-pencil tests. The exams were closed book, with a standard "formula sheet" provided. Labs met weekly for two hours and were graded on individually completed worksheets and reports.

A. Shifts in instruction due to pandemic
The semester was originally scheduled to include a one week break, March 16-22. The prior week, on March 10, university leadership announced that the break would be extended through March 29, and that subsequent instruction would be shifted online. During the extended break, faculty were encouraged to adopt a range of technological solutions to continue teaching; students were moved out of the dorms (with a few waivers available); and most other campus activities were placed in "hibernation." Wireless service was added to a parking garage to serve local students with unreliable internet service at home.
Changes to this course were at the discretion of the lecturer, in consultation with the recitation instructors (including the author). In most cases, the decisions favored efforts to maintain the synchronous, interactive components of the course to the greatest extent possible. Lectures were given at the originally scheduled days and times using Zoom. The lecturer used the Zoom whiteboard tool to present the material, and recorded the session so that students who could not "attend" or wished to review could view the lecture online. Tophat was still used to present several conceptual questions during the lecture. Students were encouraged to use the Zoom chat window to discuss the questions.
Recitations were also shifted to Zoom, again according to the original schedule, and with the same emphasis on group work/answers through Tophat. Students reported their preferred informal groups to the instructors during the extended break, and these were used to establish zoom "breakout rooms." Our recommendation was that within each breakout room, one student would share their screen and use the Zoom whiteboard. Most groups used this mode, however some groups found other solutions, e.g., sharing one screen and using a blank Powerpoint TM slide as a whiteboard. Faculty office hours and peer mentoring available through our departmental help center were also moved online using Zoom. Labs were shifted to an asynchronous mode. After the break, there were only two remaining labs scheduled. One of these was a computational physics exercise (using Excel TM to evaluate motion with air resistance using Euler integration). The second lab, scheduled to focus on simple harmonic motion, was replaced with a lab on the same topic using the PhET simulation "Masses and Springs" [12].
Both the third hour exam and the final exam were moved to an online format as well. In both cases, Tophat was used as a testing environment. This choice was made largely due to the tools in the Tophat platform meant to enhance academic integrity.

B. Data
Since the shift in instruction was not planned, we must rely almost entirely on the data available to use in the course of normal instruction. Happily, we have good background data: with the exception of a few pandemic-related questions added to the end-of-course evaluations, we have records extending three prior spring semesters taught by the same instructor, with most instructional choices steady.
We downloaded students' scores on each homework assignment from our online systems (MasteringPhysics, Webassign). Similarly, we used our Peer Instruction to determine lecture attendance. This data primarily reflects students' effort [13]. We also have instructors' records of the grade distributions, from which we extract DFW rates. This data primarily reflects the overall class' resiliency or grit [14][15][16]. We also have two sources of information about students attitudes [17][18][19]. We have the data from the CN forum (full transcripts of students' posts, with time and date stamps), and we were able to add 3 questions to the end-of-semester course evaluations addressing the shift to online instruction. We analyzed these by a rough coding of the evaluation responses as positive or negative, and by applying text-mining methods to the forum posts and written comments on the evaluations.

A. Efforts and success
During spring of 2017 and 2018, WebAssign [20] was used to deliver homework, whereas during 2019 and 2020, Mas-teringPhysics was used. There were also some changes in the numbers of assignments given each semester. Nevertheless, we can look at the trends in students' effort on the homework over time (measured by the number completing at least a portion of each assignment) and students' success (measured by the mean score of students who submitted at least a portion of the work). Figures 1 and 2 illustrate these comparisons. The assignments due after the pandemic closed campus in spring 2020 are assignments 16 and beyond. Assignment 15 was the last one due before spring break (2019 and 2020), and thus before the shift to online classes in spring 2020. In spring 2017 and 2018, Assignment 16 immediately preceded spring break. All four semesters show roughly equivalent trends: the percentage of assignments attempted declines somewhat in the second half of the semester, while the mean score on submitted assignments remains stable. Neither trend seems to have been significantly affected by the pandemic in spring 2020.
We estimate lecture attendance based on responses to the peer instruction questions. Students are awarded 50% credit for incorrect answers, and 100% for correct answers, so students have a strong incentive to respond as long as they are present. Attendance data for the spring semester of 2018 is unavailable, but the remaining three semesters are illustrated in Fig. 3. In this figure, lecture 14 was the last lecture held face-to-face in spring 2020. A coarse measure of success is the overall "DFW rate," The percentage of initially enrolled students who withdraw or earn a grade of D or F. The same primary instructor taught this course each spring since 2017; DFW rates for all four semesters are shown in Fig. 4. For context, March 15th was the last day to withdraw from the course, a few days after the announcement that classes would shift online. Students were also informed during spring break that they would be allowed to request that passing grades be converted to S (satisfactory: for grades C or better) or P (passing: for grades below C-but not including F).

B. Students' attitudes
We are able to gain insight to students' attitudes through their written responses to the course evaluations, and by analyzing their posts on the CN forum. We reviewed the free response comments in the course evaluation. The prompt is "In the following space, please provide any additional comments you have regarding your experience in this course." Of 101 students who responded to the evaluation, 54 submitted comments ranging from a few words to several paragraphs. Of these, thirteen comments (24%) directly addressed the shift to online classes. We coded these as positive or negative (none were neutral) based on the comments about change, not the overall tenor of the comment. As an example, we coded "I think he tried his best during covid, but was terrible for in person lectures" as positive based on the statement "I think he tried his best during covid." In contrast, we coded "[the professor] was excellent during lecture before we were required to move to an online learning environment. After the change, I felt that I understood [the professor's] notes less, and that [the professor] was less helpful." as negative. Using this scheme, the results were overwhelmingly negative. Eleven of the thirteen comments (85%) were negative, and only two were positive.
We also applied a text mining [21] approach to the full corpus of students' posts on the CN forum, and to all 54 comments in the course evaluations. This allows us to quantify students' sentiment (positive vs. negative) as a function of time, as shown in Fig. 5. We are also able to extract the emotional content of their comments as shown in Fig. 6 In these figures, week 9 was the last week of on-campus classes, weeks 10, 11 were spring break, and weeks 12-16 were weeks of online classes. Additional information on this methodology is available in [21].
We were able to add three questions to the end-of-semester course evaluation survey. The only format available to us was a six-segment Likert scale, with options "Strongly Agree, "Agree," "Slightly Agree," and the three equivalent of disagreement. The questions were: 1. My number 1 problem transitioning this course to on- There has been much discussion of whether or how to quantify and summarize Likert scale data [22,23]. Given the preliminary nature of this paper, we take the admittedly simplistic approach of simply stating the percentage of students who choose "Agree" or "Strongly Agree" for each question. Using this method, 25% of students agreed that technology was their primary problem, 48% agreed with internal human factors, and 42% agree with external human factors.
We can also gain insight into students' perceived problem sources by considering the 11 written comments described in the previous subsection, in which students expressed negative views of the transition. Six of these were "generic", simply saying that the course "got worse" or "was infuriating." Of the remaining comments, 4 were critical of the online testing process, 2 focused on the difficulty of collaborating using the Zoom whiteboard, and two criticized other faculty behaviors (less helpful, more likely to go over time in lecture).

V. DISCUSSION
It was surprising to note how little effect we observe due to the conversion from face-to-face to online classes or due to the pandemic. We found no significant evidence that students' efforts or success were affected. The measures of effort discussed here (homework completion, lecture attendance) show the same overall levels and trends as in prior years, with no discernible "artifacts" at the transition to online learning. The withdrawal rate was lower than average, but close to that observed in 2017. Similarly, students' scores on homework were not affected, nor was the DF rate increased.
Measure of students' attitudes were mixed. The attitudes expressed in the CN forum were not significantly affected, but students' attitudes expressed on the course evaluations were more negative. Our text mining approach shows that the overall sentiment and emotions expressed during the semester were stable, however, negative sentiment and emotions were more frequently expressed in the free response portion of the course evaluations. Eleven out of 13 comments on the evaluations that directly addressed the transition were negative. This could suggest that students are reticent to post overtly negative comments in the forum (which is not anonymous), or it could suggest that they take a "business" approach to the forum, focusing on the needs at the time rather than on generalities. When students were asked explicitly about what factors contributed to any problems they had, their emphasis was clearly on human factors such as their motivation and instructor characteristics, rather than technology.
We emphasize that our analysis reveals only results averaged over the entire class we are not able to evaluate differential impacts over subgroups based on racial, gender, or other factors that could complicate the shift to online classes. We hope to pursue these questions in future work.

VI. CONCLUSIONS
We observed that students' attendance and assignment completion rates were minimally affected, and the DFW rate for the class was in line with prior semesters. To the extent that these are proxy variables, we can draw the preliminary conclusions that students effort and success were minimally affected. Of course, we cannot conclude that our students' learning was equivalent. Similarly, we can draw a preliminary conclusion that students attitudes were negatively affected, but not dramatically so, and only as expressed in the course evaluations. We also find that students' felt their primary difficulties arose from human rather than technological factors.