Using Adaptive Comparative Judgement for assessing GCSE History - a research report from Qualification Wales (June 2022)

By Mark House

Focus on the user and all else follows

This important report takes a much needed 'deep dive' into the thoughts and opinion of teachers using RM Compare to trial the assessment of the teacher assessed extended writing element of GCSE History in Wales. There are important lesson here in both design and outcome from the research team. In particular, it highlights the benefits of a qualitative approach to 'get under the skin' of end users.

The full report is available HERE.

So what did the teachers think?

"Overall, participants would be happy to use CJ, providing it does not take more or much more time than conventional marking".

"Participants believed that CJ had several advantages over conventional marking.

  • The relative ease of making comparative judgements;
  • The mitigation of certain biases in assessments conducted by a learner's teacher;
  • The potential for professional development;
  • The potential to reduce the undesirable behaviours linked to targeting the mark scheme"

Thoughts on making CJ 'workable' in assessment at scale

The report picks out a few logistical challenges.

  • "The number of judgements per script (9 - 11) can create more work than traditional marking approaches"
  • "In order to reduce the time needed to process responses, learners should type their answers."
  • "The length of responses should be capped to a maximum of 1,500 words"
  • "Because judging responses on different topics was considered to be more challenging ....we may consider limiting the number of topic areas that one judge sees to two, if not even one."

The real challenge for a holistic approach to summative assessment?

Perhaps the real challenge remaining is not reliability, validity or 'workability', it is instead the perceived 'fairness' of the approach when compared to traditional marking. Whereas a marked script offers the candidate (and other stakeholders) transparency to where marks were won and lost, this is not the case in a CJ session where the final result comes from a judging pool. The report correctly observes that the attitude to the right of learners to appeal, is for example, an area that would need a different approach.

The compelling benefits of CJ for formative assessment

The report does a nice job highlighting the potential for using CJ as a formative assessment tool for extended writing. A number of teachers quickly identified the potential here of using the approach for peer learning. There was also a recognition how it could be used for standardisation and professional development. There was also a recognition that it would facilitate more divergent approaches to teaching and learning encouraged by new curriculum models.

What next?

We will be using this research to steer our work on RM Compare. There is a bunch of other research coming forward over the next few months which we hope will have a similar impact.

We continue to encourage and support new research projects. Get in touch for more information.