Transforming University admissions with Adaptive Comparative Judgement

Dr. David Gill is currently an Associate Professor in the Faculty of Education at Memorial University of Newfoundland. Part of his role involves leading a small team of colleagues in the annual selection process of new course participants.

Current processes followed a traditional pattern where candidates submitted a lengthy letter of application for consideration. The applications were distributed between a small team of experts for marking against an established rubric. At the end of the marking period there was a moderation / standardisation process to ensure consistency between markers.

Dr. Gill and his team we interested in finding ways that can enhance the fairness, efficiency, and depth of its admission process, leading to a more rounded and insightful assessment of potential students.

The Assessment Challenge

The process was unsatisfactory for a number of reasons

• Subjectivity and Inconsistency: Even with a rubric, individual marker biases and interpretations can influence the scoring. This subjectivity can lead to inconsistencies in how applications are evaluated, especially if markers have different levels of experience or understanding of the criteria
• Time-Consuming: The process of marking each application in detail, followed by a moderation session to ensure consistency, can be very time-consuming. This can be inefficient, particularly when dealing with a large number of applications
• Limited Feedback: Traditional marking provides quantitative scores but often lacks qualitative feedback that could be more beneficial for understanding an applicant's strengths and weaknesses beyond the rubric's scope
• Overemphasis on Rubric Conformity: Focusing strictly on rubric criteria might overlook other valuable aspects of an applicant's profile, such as creativity, resilience, or unique experiences that do not neatly fit into predefined categories.

"Placing applicants successfully is of the utmost importance for the University and the students themselves. Our current process carries a number of risks and is burdensome on already over-worked staff."

The project

• Items: 28 applications were added to RM Compare. Each application was in text form. The longest ran to 8 pages, the shortest to 1.
• Judges: 6 judges considered the applications. Each Judge was a domain expert.
• Rounds: Each item was seen 16 times. Most judges they were asked to completed around 45 judgements.
• Workload: The time to judgement varied by averaged out at less than 2 minutes. This meant that the Judges spent around 90 minutes completing their judgements.

The results

In the rank (below) we can see that the team achieved very high levels of reliability over 16 rounds. The Misfit data showed high levels of consensus between the judges concerning the relative quality of submissions.

The Item Misfit analysis (below) identified a couple of Items which were harder for the Judges to agree on - these are shown by the block dots close to or over the solid red line. Interestingly one of the Items consisted of more pages than average (8), while the other had less (1). Despite the 'misfitting' we can still be confident that each item found itself on the right place in the rank.

Improvements Through Adaptive Comparative Judgement

Adaptive Comparative Judgement (ACJ) offers a holistic approach that could address these weaknesses effectively

• Reducing Subjectivity: ACJ involves comparing pairs of applications to decide which one is better, rather than scoring them against a fixed rubric. This method can help mitigate individual marker biases as decisions are based on direct comparisons, and the process inherently involves multiple judgements, leading to a more balanced view.
• Efficiency: ACJ can be more time-efficient compared to traditional grading. Since markers are only making comparative judgements rather than detailed rubric-based assessments, the process can move faster while still producing reliable results.
• Enhanced Feedback: Through the process of comparative judgement, markers often gain deeper insights into what distinguishes a strong application from a weaker one. This can provide more informative and actionable feedback for applicants.
• Holistic Evaluation: ACJ allows evaluators to consider the application as a whole rather than focusing narrowly on specific criteria. This holistic view can capture the essence of an applicant's potential more effectively, recognizing qualities that might not be explicitly listed in a rubric but are nonetheless important for success in the university environment.
• Scalability and Adaptability: ACJ uses algorithms to adapt the process based on the results of previous rounds of comparison, which helps in quickly establishing a reliable ranking of applications. This scalability makes it suitable for handling large volumes of applications.

Header Photo by Markus Winkler on Unsplash