- Product
Comparative Judgement On Demand - a viable alternative to marking at scale?
The benefits of Comparative Judgement are well documented, however so are some of the challenges. A thought-provoking piece by AQA Lead Researcher Kate Kelly (2021) did a great job at highlighting some of these challenges when applied to assessing work at scale in a typical summer examination period. This blog post takes a closer look to see if a On-Demand approach to Comparative Judgement might have some of the answers.
A simple scenario and some basic assumptions
We will use the scenario and assumptions outlined in the original article.
Scenario
- AQA GSCE English Literature Examination
- 500,000 Students producing 1,000,000 scripts
- 4500 Examiners
- 5 week marking period.
- Markers working 15 hours a weeks (most are in-work teachers)
Assumptions
- Marking - Each script takes 15 minutes to mark (4 per hour)
- Comparative Judgement - Each script requires 20 judgements, each one taking 7.5 minutes (8 per hour)
Table 1 (below) gives a simple way to compare (no pun intended) the two approaches. As we can see, the CJ workload is significantly higher than it is for marking. As the AQi article rightly concludes - "Without some radical changes then, CJ is too time-consuming and expensive to use for marking".
| Total Scripts | Decisions per script | Total Decisions | Decisions per hour | Total hours | |
|---|---|---|---|---|---|
| Marking | 1,000,000 | 1 | 1,000,000 | 4 | 250,000 |
| CJ | 1,000,000 | 20 | 10,000,000 | 8 | 1,125,000 |
A 'radical change' - CJ On-Demand
We think an On-Demand approach gives us an opportunity to change some assumptions
CJ On-Demand Assumptions
- Each script will require 6 decisions per script
- Each judge will be able to make 12 decision per hour
As we can see below (Table 2) the radical change in assumptions produces some interesting results.
| Total Scripts | Decisions per script | Total Decisions | Decisions per hour | Total hours | |
|---|---|---|---|---|---|
| Marking | 1,000,000 | 1 | 1,000,000 | 4 | 250,000 |
| CJ | 1,000,000 | 20 | 10,000,000 | 8 | 1,125,000 |
| CJ OD | 1,000,000 | 6 | 3,000,000 | 12 | 250,000 |
That's it then! - job done!
Not quite! These are assumptions that require a lot more investigation BUT we are making good progress. We will be releasing 'manual' On-Demand functionality into BETA very soon which will help us to learn even more, even faster.
Our focus to this point has been on overcoming the efficiency issues as outlined above, however as the AQi article outlines this still leaves a number of other (big) challenges, for example the implicit move away from mark schemes and issues around transparency and accountability.
Get involved
We are working with a number of interested groups and continue to engage more. If you are interested, please get in touch.
We will be present at several events this summer - come and say hello if you are in attendance.