Student perceptions of pre-assessments: “It’s basically just guessing anyways.”

Pre-assessments are a popular tool among researchers and instructors in higher education, but they may pose risks to students, instructors, and researchers that are largely unexplored. To investigate potential risks, we conducted ten semi-structured interviews with college students about their experiences taking pre-assessments. Students unanimously found pre-assessments to be non-threatening and their performance on them to be incon-sequential, because they did not interpret them as a predictor of success or indicator of their inherent ability. Instead, students attributed their pre-assessment performance to having previously taken a relevant course with similar assessment items. As a result, half of the students were not motivated at all to put effort into taking the pre-assessment, especially for difﬁcult items. Students believed that pre-assessments are intended to inform the instructor in making tailored course revisions to meet student needs. They valued pre-assessments more if they believed that the instructor uses the pre-assessment results to make course revisions, but half of the students doubted that their instructor would do so. Taken together, our ﬁndings suggest that pre-assessments do not pose a direct risk to students, but they may indirectly affect students’ learning experience or relationship with the instructor if the use of pre-assessments undermines the instructor’s credibility. Moreover, students’ lack of motivation to complete pre-assessments can render them an inaccurate measure for instructors and researchers. We therefore encourage instructors and researchers to clearly communicate the speciﬁc uses of pre-assessments to students before assigning them.


I. INTRODUCTION
Pre-assessments, such as the Force Concept Inventory [1], are common research tools in higher education, but they are rarely the topic of research themselves. The FCI and other similar assessments in physics have been used to assess the efficacy of educational reforms (e.g. [2,3]), assess the impacts of instruction by students' gender (e.g. [4,5]) and race (e.g. [6]), and how students' conceptual understanding relates to student attitudes (see [7] for a review). Many other STEM disciplines have followed suit by developing their own concept inventories and similar pre-assessments. Beyond their use for education research, pre-assessments are also commonly used by instructors as a pedagogical tool [8].
The prevalence of pre-assessments motivates this exploration into the risks that they may carry for the students who take them. Several scholars have provided recommendations for best practices for pre-assessments [9][10][11][12][13], but these recommendations are not grounded in research evidence and there has not been a systematic study of students' reactions to and experiences of pre-assessments. We initially consider the potential risk to students in terms of their affective state in response to taking a pre-assessment. Students have strong emotional and physiological responses to examinations [14], but it is unknown whether an ungraded preassessment causes similar reactions. Even during low-stakes classroom exercises, students can interpret constructive learning experiences as a threat [15]. If constructive classroom exercises can seem threatening to students, an ungraded assessment could produce similar emotional and physiological responses as a graded assessment. We further hypothesized that students' experience of pre-assessments may be related to students' understanding of their purpose.
As a first step in investigating this potential risks, we conducted ten semi-structured interviews with undergraduate students who had taken pre-assessments in at least one of their courses: seven had taken the FCI at the beginning of an introductory physics course and two had taken a preassessments at the start of an introductory biology course (one was excluded in the analysis). Our findings suggest that preassessments do not pose a direct risk to students, but they can (a) undermine the instructor's credibility if students question how the instructor will use it, and (b) provide inaccurate measurements if students lack the motivation to complete them.

II. METHODS
We completed ten semi-structured interviews via Zoom with undergraduate students. We used a semi-structured approach as this was an exploratory project designed to facilitate later design of a survey. We recruited undergraduate participants through the university message board and by emailing students enrolled in two introductory physics courses. Interested students took a screening survey to ensure that they had taken a pre-assessment in one of their STEM courses and to collect demographic data. One participant was removed from the analysis because the pre-assessment she had taken was explicitly designed to persuade under-prepared students to drop the course (an advanced computer science course). Interviews were conducted by the first author (Allen), who was not involved in the instruction of any course the subjects were enrolled in. Of the nine students included in the analysis, five identified as male, four as female, four as white, three as Asian, and two as Hispanic. Seven students were first-year students and two were in their second year. Each student had taken a pre-assessment at the beginning of the previous semester.
Each participant was asked the following questions: Clarifying questions carefully avoided anything that might prompt or prime students to mention any of the major themes found. Specifically, the interviewer never raised using pre-assessment results to make course revisions, or attributing performance to prior course work. Interviewees sometimes provided an answer to a later question in an earlier answer. For example, when asked about the purpose of a pre-assessment, most interviewees would elaborate on what they believed the results were used for and describe what they had been told about the purpose of the preassessment. In those cases, the later questions were skipped. The interviews lasted 5-8 minutes.
The automated transcripts from Zoom recordings were edited for correctness, punctuation was added for clarity, and names were replaced with pseudonyms. Short narrative descriptions of each of the interviews were completed. We used a grounded approach to identify themes among student responses, identifying common phrases and statements. Four themes were unmistakably consistent-or outright unanimous-among the responses. The quotes found herein were selected because they give concise summaries of the overall themes, but these specific statements were not given disproportionate weight when identifying themes during the analysis. De-identified transcripts are available online [16]. The nine interviews represent a spectrum of attitudes. Eight students described a relaxed or unconcerned state while taking the pre-assessment, but one student described a feeling of nervousness. Two students who described feeling unthreatened by the pre-assessment itself, but daunted by the quantity of upcoming course content, either stated that they found it motivating or that they forgot about it afterwards. Two students were strongly apathetic, describing the pre-assessment as an annoying waste of time.
On the positive side of the spectrum, four out of the nine students reported being motivated and encouraged by the preassessment. Christina describes an especially positive influence on her willingness to engage fully with the course: I was super chill going into the pre-test...I was unsure if I wanted to switch to a different bio course because I was, like, interested in a few. But when the professors were just kind of emphasizing that this is a pre-test, so that they can make the course as educational as possible and work as best as it could for us, I was like 'wow these professors really care about me'. Like this pre-test is to help me as a student and help me as a learner. So I really like that, and so I made the decision to like stay in the class, be more outgoing, have more friends. I definitely felt more comfortable, I guess, like reaching out to office hours and TAs and professors, because they just made an effort to tell me throughout that pre-test that, like, this is simply to make sure that we get the best education out of this.
-Christina, 2nd year, biology Despite having previously had a negative experience with a pre-assessment for math placement and confessing to an undesirable tendency to occasionally compare her intelligence to others, Christina, the only URM woman in our participant pool, reported the most positive experience here. She elaborated that the experience of the pre-assessment built her trust in the instructors, her commitment to stay in the course, and her willingness to take advantage of all the academic resources available to her. She attributed this strong positive influence on her behavior not to anything about the content of the pre-assessment, but to how well the instructors communicated with the class about the purpose of the pre-assessment.
Two participants felt neutral and they quickly forgot any negative or positive feelings they had during the preassessment: I didn't really feel, like, stressed or nervous. It was just a little bit like, woah this feels like a lot that I have to learn...Honestly, I didn't really think about the pretest after I took it.
-Jessica, 1st year, physics Jessica then clarified that the day-to-day demands of the course dominated her judgment about the workload of the course, and that she immediately forgot about the preassessment. Another student, Michael, said he found it "enjoyable" to test his knowledge, but also quickly forgot the experience.
A few participants expressed apathy and annoyance at having to take an assessment on content they had not yet been taught. As a result, they did not put any effort into answering the questions; especially the difficult ones. For example, Avril, a first-year student who took a pre-assessment in physics, stated, "[the pre-assessment is] just annoying... It's easy to just not put any effort into it at all" and Justin (firstyear, physics) said, "I think [the pre-assessment is] a waste of time... If you have a question require too much effort, I might just half-ass it." These students questioned the purpose of attempting problems they have not been trained to answer. They expressed a confident belief that other students were also not putting effort into the pre-assessment.
On the negative side of the spectrum, only one student reported any feelings of nervousness on the exam.
It's really nerve racking. I've done lot's of tests; lots of academic work. So when I get an academic test and the point isn't really to score highly, subconsciously I'm still wishing for a high grade-because of my past experience-even though it doesn't actually matter.
-Ricky, 1st year, physics Ricky, expressed nervousness about taking a test unprepared and going through the motions of answering test questions while "getting anything less than a hundred." He expressed an inability to suppress his emotional reaction to attempting questions he cannot answer, even though he knew that the pre-assessment was low-stakes.
B. Ungraded pre-assessments are seen as low-stakes Either when asked to describe their thoughts and feelings while taking the pre-assessment, or later in the interview when asked if it is important to do well on a pre-assessment, all of the participants described it as having no or low stakes. Participants characterized the stakes as the impact the preassessment has on their grade, but they did not describe it as an indicator of preparedness to succeed in the course. For example, Xiao described the differences between a preassessment and a regular exam: There's no repercussions of this. It's not going to affect my grade. It's pretty different because when I'm taking actual quizzes or exams there's actual, like, they're actually graded so there's a weight to it....It's a pre-assessment so you generally shouldn't know anything going into going into it, so how you do on it shouldn't really affect how you think going into the course.
-Xiao, 1st year, physics He articulated a belief that was common to nearly all participants: the stakes are proportional to the impact on grades. They attributed their carefree state of mind during the preassessment to its ungraded nature. He later stated that students at all levels of prior knowledge can succeed in introductory courses. Jessica, one of two students who reported being daunted by the quantity of upcoming course content, nonetheless stated that "it's okay if I don't do well." Like Jessica, all participants reported that pre-assessment performance is not an indicator of whether or not they would succeed in the course.
Ricky, like Xiao and others, focused on the ungraded aspect of the pre-assessment: I don't think it's important to score well, but just coming from a background where academics are always very important: it's always grades. It's just an ingrained thought even though I know consciously that it doesn't matter.
Ricky described that his state of mind during the exam directly conflicted with his beliefs about the stakes of the exam. He was nervous despite explicitly believing that it is safe to score poorly.
While the students' feelings about the purpose of the preassessment varied, the fact that they understood their performance on the pre-assessment would not impact their grade seemed to dominate how most participants engaged with the pre-assessment.

C. Students attributed performance to prior preparation
Students unanimously attributed pre-assessment performance to having taken prior courses on the subject. They either did not mention, or they directly refuted notions of intelligence or inherent ability impacting their pre-assessment performance.
Christina, who was once awed by students who could "miraculously" score well on the pre-assessment, later laughed off the notion that anyone could "invent" the answers during the exam. She reasoned that high-scorers were "normal people" who "came from a school that already taught them a little bit." Avril, who believes that you cannot do well on the preassessment unless you have studied the materials before, felt demotivated to attempt the unfamiliar assessment items: Well, I mean if they already took the class..like a lot of like the intro classes people already taken, like a class that's essentially the same....But like if you if you didn't already take it, then the motivation to like try on it,[pauses] because you know you're gonna get most of them wrong anyways you haven't learned any of it and it's basically just guessing anyways. -Avril, 1st year, physics Every interview participant confidently claimed that the only way to score well is to have previously taken a similar course that covered the relevant content.
D. Students are divided on whether pre-assessment results are used to adapt the course Eight out of nine participants stated that the purpose of a pre-assessment is to help instructors make well-informed course revisions to tailor the course to the needs of the students, but half of the participants did not believe that their instructors actually do this. Lance states that instructors could de-emphasise topics that students have already mastered in the upcoming curriculum.
If every single student gets one question right they can say "Okay, this is a pretty well-known topic, we can remove this from the syllabus because it seems as though everyone seems to know it." -Lance, 1st year, biology Overall, four of the nine participants expressed the same belief that their instructors were making targeted revisions to suit students' needs; another four believed the very opposite, and one participant had not considered either possibility. Ricky reasons that an introductory course cannot be revised based on pre-assessment results, because he believes that introductory courses must assume no prior knowledge. He questions the instructor's motives for giving the pre-assessment.
Um, they said {emphatically incredulous tone} it was to see where we were at, but I think it was for their own data to see how many people have already taken the class... because it doesn't really matter where you are at because they have to start a beginning anyway. -Ricky, 1st year, physics This attitude towards pre-assessments has implications for the instructors' credibility with their students. Although Ricky was the only student who implied his instructors were outright dishonest, four of the nine participants suspect that pre-assessments are not actually used for their intended purpose. These students also made unprompted statements that the instructor's "data", "stats", or "research" was not important to them.
We find that students with negative attitudes about preassessments do not believe that any course revisions are being made in response to the pre-assessment results, while students with positive attitudes do believe that such revisions are being made. The participants' statements about instructors using pre-assessment results to inform course revisions were entirely unprompted. Some participants shared their opinion about course revisions after being asked if it is important to do well on a pre-assessment, and some after being asked what the purpose of a pre-assessment was.

IV. DISCUSSION AND CONCLUSION
Though the students we interviewed experienced a variety of emotions about pre-assessments, they shared a common criterion for judging their value: whether or not they are used to make course revisions. None of the students we interviewed found their pre-assessments to be threatening, but half of them were annoyed by them; the same half who did not believe that the results were being used to make tailored course revisions. Those students put little-to-no effort into attempting pre-assessment items, and the pre-assessment caused them to question their instructor's motives.
This study was motivated by a concern that students might view pre-assessments as a measure of their preparedness to succeed and attribute pre-assessment performance to noncontrollable factors, such as their intelligence or inherent ability, which could raise psychological threats [15,17,18]. This would present a significant problem for educational practice and research as pre-assessments are recommended and widely used [1,7,8]. Fortunately, our interviews found minimal evidence to support these concerns. Instead, we found that students correctly judged the pre-assessments as lowstakes and not a measure of their intelligence. However, we uncovered threats that we had not previously considered. First, the perceived futility of the pre-assessment could corrupt the validity of pre-assessment data in a manner that cannot be solved by offering incentives to students. Second, the perceived futility of pre-assessments, and especially the belief that they are not being used for their intended purpose, presents a risk to instructors' credibility with their students.
We highlight several limitations for generalizing the findings of this exploratory work. Though some important themes were unanimous in our sample, we had only nine participants from a selective university who were reflecting on events from a few months prior. Results might differ for other student populations, or if interviews were conducted at different times relative to pre-assessments. We found no demographic patterns, but cannot conclude whether any might emerge in a larger sample. Further research, including surveys to study student experiences of pre-assessments, should be conducted to address these limitations.

A. Implications for instruction and research
Concept inventories that are used as pre-assessments like the FCI were designed to measure students' concept of Newtonian physics rather than to measure their training to complete physics problems. Yet students are not likely to put in sufficient effort to produce an accurate measurement if they believe that their efforts are futile. The half of our participants who were unwilling to attempt any pre-assessment items which they have never been taught to solve would not likely be swayed by incentives to participate. Research on student effort on pre-assessment and on incentives for preassessment participation have used methods which would not detect the patterns of low effort described herein [12,13,19]. Providing incentives does not address the core issue that students do not believe they can do better than mere guessing. Students may be less likely to "click through" the preassessment if they know that they are able to give a meaningful answer even if they cannot give a confident answer. Perhaps it would help to frame a pre-assessment in this manner: "You may find these items unfamiliar. That's fine. This is not part of your grade, so please do not think of the answer choices as one correct answer hidden among a few wrong answers. Rather, treat them as various representations of how people often think about physics. Your task is to pick the one that most closely represents how you think. Your time and effort on this set of questions helps us make the course the best it can be." One direction for future research is to test how framing messages like this one impact students' effort on pre-assessments.
At the outset of this study, we expected to find that preassessments pose certain psychological threats to students which instructors should attempt to mitigate. However, the low degree of threat expressed by our participants does not require mitigation strategies (e.g., framing the pre-assessment as a tool for evaluating the instructor [11]). The most likely risk implied by our evidence is a risk to the instructor's own credibility as they make a first impression with students.
Students' first impressions of their instructors have been found to influence students' behavior throughout the semester [20]. STEM instructors can make unintentional blunders even while actively attempting to establish a constructive classroom culture and build positive instructorstudent rapport on the first day of class. Classroom observations of STEM instructors' first-day messaging have revealed counterproductive mixed messages during moments of class time devoted to building relationships with students [21]. We propose that a pre-assessment could serve as a tacit mixed message if it is not carefully framed.
It is clearly undesirable for an instructor's first introduction to their students to be such that half of the students believe that the instructor's assignments are not worthwhile and that the instructor is not taking actions to meet their needs. A reasonable strategy would be to show the students where there are flexible elements in the syllabus that can be adjusted according to the results of the pre-assessment.
Since not all instructors can make immediate revisions to their introductory courses based on the pre-assessment alone, an appeal to "pay it forward" may be a reasonable alternative. Instructors could show students the beneficial revisions that have been made to the course based on previous students' efforts on pre-and post-assessments. Our participants did not express any awareness of course revisions on a timescale longer than one semester. An earnest request to pay it forward to the next cohort of students would still communicate the instructor's commitment to the students' success.