Detail Page

Using an LLM to Investigate Students' Explanations on Conceptual Physics Questions
written by Sean Savage and N. Sanjay Rebello
Analyzing students' written solutions to physics questions is a major area in PER. However, gauging student understanding in college courses is bottlenecked by large class sizes, which limits assessments to a multiple-choice (MC) format for ease of grading. Although sufficient in quantifying scientifically correct conceptions, MC assessments do not uncover students' deeper ways of understanding physics. Large language models (LLMs) offer a promising approach for assessing students' written responses at scale. Our study used an LLM, validated by human graders, to classify students' written explanations to three questions on the Energy and Momentum Conceptual Survey as correct or incorrect, and organized students' incorrect explanations into emergent categories. We found that the LLM (GPT-4o) can fairly assess students' explanations, comparable to human graders (0-3% discrepancy). Furthermore, the categories of incorrect explanations were different from corresponding MC distractors, allowing for different and deeper conceptions to become accessible to educators.
Physics Education Research Conference 2025
Part of the PER Conference series
Washington, DC: August 6-7, 2025
Pages 399-404
Subjects Levels Resource Types
Education Foundations
- Assessment
= Conceptual Assessment
= Instruments
= Methods
- Research Design & Methodology
= Evaluation
= Statistics
- Student Characteristics
= Skills
Education Practices
- Technology
= Computers
- Lower Undergraduate
- Reference Material
= Research study
Intended Users Formats Ratings
- Researchers
- application/pdf
  • Currently 0.0/5

Want to rate this material?
Login here!


Mirror:
https://doi.org/10.1119/perc.2025…
Access Rights:
Free access
License:
This material is released under a Creative Commons Attribution 4.0 license. Further distribution of this work must maintain attribution to the published article's author(s), title, proceedings citation, and DOI.
Rights Holder:
American Association of Physics Teachers
DOI:
10.1119/perc.2025.pr.Savage
NSF Number:
2300645
Keyword:
PERC 2025
Record Creator:
Metadata instance created October 20, 2025 by Lyle Barbato
Record Updated:
October 27, 2025 by Lyle Barbato
Last Update
when Cataloged:
October 28, 2025
Other Collections:

ComPADRE is beta testing Citation Styles!

Record Link
AIP Format
S. Savage and N. Rebello, , presented at the Physics Education Research Conference 2025, Washington, DC, 2025, WWW Document, (https://www.compadre.org/Repository/document/ServeFile.cfm?ID=17173&DocID=6101).
AJP/PRST-PER
S. Savage and N. Rebello, Using an LLM to Investigate Students' Explanations on Conceptual Physics Questions, presented at the Physics Education Research Conference 2025, Washington, DC, 2025, <https://www.compadre.org/Repository/document/ServeFile.cfm?ID=17173&DocID=6101>.
APA Format
Savage, S., & Rebello, N. (2025, August 6-7). Using an LLM to Investigate Students' Explanations on Conceptual Physics Questions. Paper presented at Physics Education Research Conference 2025, Washington, DC. Retrieved November 10, 2025, from https://www.compadre.org/Repository/document/ServeFile.cfm?ID=17173&DocID=6101
Chicago Format
Savage, Sean, and N. Sanjay Rebello. "Using an LLM to Investigate Students' Explanations on Conceptual Physics Questions." Paper presented at the Physics Education Research Conference 2025, Washington, DC, August 6-7, 2025. https://www.compadre.org/Repository/document/ServeFile.cfm?ID=17173&DocID=6101 (accessed 10 November 2025).
MLA Format
Savage, Sean, and N. Sanjay Rebello. "Using an LLM to Investigate Students' Explanations on Conceptual Physics Questions." Physics Education Research Conference 2025. Washington, DC: 2025. 399-404 of PER Conference. 10 Nov. 2025 <https://www.compadre.org/Repository/document/ServeFile.cfm?ID=17173&DocID=6101>.
BibTeX Export Format
@inproceedings{ Author = "Sean Savage and N. Sanjay Rebello", Title = {Using an LLM to Investigate Students' Explanations on Conceptual Physics Questions}, BookTitle = {Physics Education Research Conference 2025}, Pages = {399-404}, Address = {Washington, DC}, Series = {PER Conference}, Month = {August 6-7}, Year = {2025} }
Refer Export Format

%A Sean Savage %A N. Sanjay Rebello %T Using an LLM to Investigate Students' Explanations on Conceptual Physics Questions %S PER Conference %D August 6-7 2025 %P 399-404 %C Washington, DC %U https://www.compadre.org/Repository/document/ServeFile.cfm?ID=17173&DocID=6101 %O Physics Education Research Conference 2025 %O August 6-7 %O application/pdf

EndNote Export Format

%0 Conference Proceedings %A Savage, Sean %A Rebello, N. Sanjay %D August 6-7 2025 %T Using an LLM to Investigate Students' Explanations on Conceptual Physics Questions %B Physics Education Research Conference 2025 %C Washington, DC %P 399-404 %S PER Conference %8 August 6-7 %U https://www.compadre.org/Repository/document/ServeFile.cfm?ID=17173&DocID=6101


Disclaimer: ComPADRE offers citation styles as a guide only. We cannot offer interpretations about citations as this is an automated procedure. Please refer to the style manuals in the Citation Source Information area for clarifications.

Citation Source Information

The AIP Style presented is based on information from the AIP Style Manual.

The APA Style presented is based on information from APA Style.org: Electronic References.

The Chicago Style presented is based on information from Examples of Chicago-Style Documentation.

The MLA Style presented is based on information from the MLA FAQ.

Using an LLM to Investigate Students' Explanations on Conceptual Physics Questions:


Know of another related resource? Login to relate this resource to it.
Save to my folders

Contribute

Related Materials

Similar Materials