A Chain-of-Thought Prompting Approach with LLMs for Evaluating Students’ Formative Assessment Responses in Science
Clayton Cohn, Nicole Hutchins, Tuan Le, Gautam Biswas
Abstract
This paper explores the use of large language models (LLMs) to score and explain short-answer assessments in K-12 science. While existing methods can score more structured math and computer science assessments, they often do not provide explanations for the scores. Our study focuses on employing GPT-4 for automated assessment in middle school Earth Science, combining few-shot and active learning with chain-of-thought reasoning. Using a human-in-the-loop approach, we successfully score and provide meaningful explanations for formative assessment responses. A systematic analysis of our method's pros and cons sheds light on the potential for human-in-the-loop techniques to enhance automated grading for open-ended science assessments.
BibTeX
@article{Cohn_Hutchins_Le_Biswas_2024, title={A Chain-of-Thought Prompting Approach with LLMs for Evaluating Students’ Formative Assessment Responses in Science}, volume={38}, url={https://ojs.aaai.org/index.php/AAAI/article/view/30364}, DOI={10.1609/aaai.v38i21.30364}, abstractNote={This paper explores the use of large language models (LLMs) to score and explain short-answer assessments in K-12 science. While existing methods can score more structured math and computer science assessments, they often do not provide explanations for the scores. Our study focuses on employing GPT-4 for automated assessment in middle school Earth Science, combining few-shot and active learning with chain-of-thought reasoning. Using a human-in-the-loop approach, we successfully score and provide meaningful explanations for formative assessment responses. A systematic analysis of our method’s pros and cons sheds light on the potential for human-in-the-loop techniques to enhance automated grading for open-ended science assessments.}, number={21}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Cohn, Clayton and Hutchins, Nicole and Le, Tuan and Biswas, Gautam}, year={2024}, month={Mar.}, pages={23182-23190} }