From Marking Errors to Life Earnings: How Comparative Judgement Can Reduce the Risk

If a single grade can change a student’s lifetime earnings, then we can’t afford a grading system that behaves like a lottery. Recent analysis shows that moving just one grade band at A‑level can translate into thousands of pounds more – or less – in annual income over a career. At the same time, long‑standing research on exam reliability suggests that a significant proportion of grades issued each year could plausibly have been different if a different examiner, a different day, or a different script sample were involved. Put bluntly: we attach very high, very real economic stakes to a process that isn’t nearly as precise as most people assume.

In this post, I want to explore that tension and ask a simple question: if grades matter this much, what can we do to improve the evidence behind them? And in particular, what role can Adaptive Comparative Judgement – and tools like RM Compare – play in making high‑stakes decisions more reliable and more fair?

When one grade means thousands

The TES article highlights a reality that will be familiar to many teachers, parents and students: A‑level results don’t just decide where a young person studies next; they shape the opportunities and earnings available to them for decades. A one‑grade difference can be the difference between getting onto a particular university course or missing out, accessing a certain profession or having to reroute entirely. Over time, those seemingly small shifts compound through different starting salaries, promotion pathways and networks.

When researchers link exam outcomes to earnings data, they find sizeable gaps between groups of students who left school with higher grades and those whose grades were lower. That doesn’t mean grades are the only driver of future prosperity, but it does confirm that they are a powerful sorting mechanism in the system. Once you accept that, it becomes hard to view a disputed or borderline grade as a minor technical issue; it’s a decision about someone’s future cashflows and life chances.

How confident are we in those grades?

Set against those stakes, the reliability story is sobering. If you work in assessment, you’ll already know that exam grades are estimates of underlying performance, not perfect measurements. Studies of marking consistency and grade reliability suggest that, especially around the grade boundaries, many scripts could legitimately have received a neighbouring grade. In other words, a candidate whose work is judged as a B on one day might be judged as an A or a C under slightly different conditions.

For the average candidate this “± one grade” fuzziness may not matter very much – it may not affect their progression, and the broad signal about their performance remains accurate. But for students sitting on key thresholds, that fuzziness matters enormously. These are the students for whom a single grade shift is the difference between an offer and a rejection, or between meeting a conditional place and missing it. The system treats these decisions as if they are precise, while the measurement science tells us they are not.

The human judgement problem

Part of the difficulty lies in how we currently capture human judgement. Traditional exam marking asks an examiner to read a script, apply a mark scheme, and assign marks that later roll up into grades. It’s a demanding cognitive task: the examiner has to hold in mind the standard of quality expected, interpret the student’s work, and navigate the rubric – often under time pressure and with limited opportunity for calibration.

We know that human judgement is context‑sensitive. The order in which scripts are seen, the examiner’s prior experiences, and even their level of fatigue can subtly influence how generous or strict they are. Moderation and standardisation processes try to control for this, but they can’t remove it completely. The result is a system where we rely heavily on individual marking events, even though we know that individuals are variable.

A different paradigm: comparative judgement

Adaptive Comparative Judgement starts from a different premise: that humans are generally better at saying “which of these two things is better?” than at assigning absolute scores. Instead of asking a marker to pin a script to a mark scheme right away, comparative judgement invites multiple judges to compare pairs of student work and decide which one shows higher quality. Across many such comparisons, a statistical model builds a rank order of all the work and can place scripts on a scale that reflects collective professional judgement.

Crucially, this approach doesn’t rely on a single marking event. Each script’s position is informed by many independent comparisons against a range of other scripts, often involving multiple judges. That makes it possible to gather stronger evidence about borderline cases and to check whether the emerging ordering feels coherent and fair. Where traditional marking asks “what mark does this script get?”, comparative judgement effectively asks “where does this script sit among all the others we’ve seen?”.

Where RM Compare fits in

RM Compare is a platform designed to make Adaptive Comparative Judgement practical at scale. It allows schools, awarding bodies and other organisations to:

• Upload collections of student work (for example, essays, coursework or project artefacts).
• Define comparison rounds in which judges see pairs of work and indicate which is better.
• Use adaptive algorithms to choose informative pairs, so that each judgement contributes meaningfully to the emerging rank order.
• Generate a robust ordering of scripts and, where appropriate, establish cut scores or performance standards aligned with that ordering.

From a user’s perspective, the workflow feels straightforward: you see two pieces of work, decide which one represents higher quality, and move on. Behind the scenes, the system is turning those simple decisions into sophisticated evidence about where each piece sits in the cohort and how clear the distinctions are between neighbouring scripts.

Reducing risk where stakes are highest

Connecting back to the Tes article’s theme, the most compelling use‑cases are precisely where the economic stakes are high and the traditional process is most fragile. For example:

• High‑stakes coursework components that are currently marked by one or two teachers could be run through comparative judgement to ensure that the top band, the pass/fail threshold, and key grade boundaries reflect broad professional consensus rather than the idiosyncrasies of individual markers.
• Scripts that sit near critical grade boundaries – the ones where a one‑grade shift would alter a student’s progression – could be flagged for additional comparative judgement. Instead of a single marker making a finely balanced decision, multiple judges could collectively determine whether the work really belongs above or below that boundary.

In both scenarios, the aim isn’t to replace grading; it’s to strengthen the evidence behind grades where they matter most. If a grade is going to play a significant role in somebody’s future earnings, then investing extra effort in the judgement process for that script is not a luxury; it’s a responsibility.

Fairness and the disadvantage gap

There’s also a fairness dimension. Analyses of exam outcomes often show persistent gaps between disadvantaged students and their peers. These gaps reflect a complex mix of factors – access to teaching, resources, prior attainment – but they are also shaped by how performance is judged and standardised. When expectations and standards vary between schools and regions, students in different contexts can face slightly different odds.

One of the strengths of comparative judgement, implemented through platforms like RM Compare, is that it can help make performance standards more transparent and shareable. When teachers across multiple schools are jointly judging work in a single comparative framework, they are literally aligning their sense of “what good looks like”. That alignment can reduce the risk that a student’s postcode or school context subtly influences the grade they receive, at least for the components assessed through comparative judgement.

From critique to action

It’s easy to read about marking errors and unreliable grades and come away feeling helpless. But the practical reality is that schools, trusts and awarding bodies do have options. They can choose to pilot alternative judgement methods, to focus extra scrutiny on the most consequential decisions, and to invest in tools that make professional judgement more robust.

RM Compare is one such tool. Starting small – perhaps with a single subject’s coursework, or with a moderation exercise across a group of schools – is enough to begin building experience and evidence. Over time, those experiences can feed into a broader conversation about how we want to balance efficiency, reliability and fairness in high‑stakes assessment. The Tes article provides a strong moral and economic case for that conversation; comparative judgement offers one concrete way to respond.

If your organisation is already grappling with concerns about grade reliability or fairness, where do you see the biggest opportunity to use comparative judgement to strengthen your decisions?