Automated Essay Marking with Data-Efficient Augmentation and Score-Token Weighted Loss

Abstract

Automated Essay Scoring (AES) research has predominantly focused on holistic scoring, yet rubric-based trait scoring provides richer diagnostic feedback for both students and instructors. While encoder-based models such as BERT have achieved competitive trait scoring performance, they are limited to score prediction and cannot generate the explanatory feedback that makes rubric-based assessment educationally useful. Large Language Models (LLMs), by contrast, naturally support both scoring and feedback generation within a single unified framework. However, most existing work on fine-tuned LLM-based rubric scoring relies on either large annotated corpora or synthetic data augmentation to achieve competitive performance, limiting practical adoption in settings where neither is readily available. This paper investigates whether a small, carefully balanced dataset of real essays can enable a fine-tuned LLM to achieve competitive rubric-based trait scoring. We fine-tune Qwen2.5-7B-Instruct using QLoRA on progressively augmented subsets of DREsSNew (371, 478, and 736 essays), targeting under-represented score bands at each stage, and propose a score-token weighted loss function that concentrates the training signal on score-bearing tokens. Data augmentation drives the majority of the performance improvement: expanding from 371 to 736 essays yields gains of +0.042 QWK on Content, +0.021 on Organization, +0.041 on Language, 0.024 on Total. Score-token weighting provides a further modest but consistent improvement, most notably on Content (+0.025). Our best configuration achieves per-trait QWK of 0.631 (Content), 0.612 (Organization), 0.625 (Language), and 0.669 (Total) on the DREsSNew test set using only 736 real essays - remaining competitive with models trained on datasets over 65 times larger while preserving the generative capability needed for downstream feedback generation.