Developing a Framework for Using Large Language Models for Viva Assessments in Higher Education

Abstract

This paper presents a comprehensive framework for evaluating Large Language Models (LLMs) based on educational performance areas and established evaluation metrics. The study bridges the gap between traditional academic assessment criteria and modern AI evaluation techniques, aligning metrics such as coherence, relevance, completeness, and creativity with performance areas like problem definition, methodology, and product outcomes. Drawing insights from experimental results, the framework highlights the top 10 evaluation metrics frequently observed and emphasizes their significance in assessing AI -generated responses. A critical analysis identifies limitations in the initial framework proposed by ChatGPT, leading to refined strategies for more comprehensive evaluation. The refined framework addresses limitations of subjectivity, overlapping criteria, and weighting mechanisms, offering a dynamic evaluation model for both technical and educational contexts. The findings contribute to advancing interdisciplinary evaluation methodologies and offer valuable insights for educators, researchers, and developers in optimizing LLM applications for educational purposes.