Examination of the inter-rater agreement among faculty marking a research proposal on an undergraduate health course

Abstract

Purpose: This research looked at inter-rater agreement among faculty marking a research proposal on an undergraduate health course. The purpose of this study was to investigate if there were significant differences between marks as well as the comments given on research proposals that were double marked by two faculty members, where faculty were blind to the other marker’s marks and comments. Subjects: The subjects for this study consisted of 80 final year health students who submitted a research proposal and 13 full-time and 10 part-time faculty members who double marked the proposals. Methods: Faculty pairs marked the proposals independently and were required to meet with each other and discuss the marks given, reasons for the mark and the comments. Markers were asked to come to an agreement about the mark and the comments. Analyses: Agreement in marks was assessed using a Bland-Altman plot. Weighted Cohen's Kappa was used to estimate the agreement between the classifications given by markers. Comments were analyzed for differences using thematic analysis. Results: There was a wide discrepancy in the classification of students between markers. The weighted proportions found to agree on classification was 46%. Analysis of the comments indicated a wide discrepancy between markers. Conclusion: The outcomes of this study are similar to previous studies that have looked at inter-rater agreement when double marking was used to mark various types of written assignments. Further exploration of the inter-rater agreement in the marking process and other marking processes that results in a transparent system is needed.