Ask anything
RM Compare has been optimised for use with your LLM.
See our Training Centre for more information.
- Product
Judge Workload Calculator
How much work am I asking my Judges to do?
A key consideration for any Creator is how much work and effort each session will create for Judges. There are plenty of things that could influence this for example
- You Items.
- How complex are they?
- How 'high stakes' are they?
- How different are they in value? If they are all roughly the same value this will make it more challenging and time consuming for the Judge to discriminate.
- Your Judges.
- Have they used RM Compare before?
- How expert are they in the assessment domain?
- How familiar and comfortable are they with the comparative approach?
- Your session design.
- Are you asking Judges to provide feedback?
As your get more experienced and perhaps investigate further (there has been plenty of research on this matter) you will get better at estimating judge workload.
Time to Judgement
A key variable when thinking about workload is 'time to judgement'. Helpfully your RM Compare reports provide valuable insights.
- Average Decision Time (Seconds)
- Average Median Decision Time (Seconds)
The Median time can be more reliable as this counters 'off-task' Judge behaviour where for example the Judge has left the task for a time but has not logged out of the system.
Judgement calculator
As you work through the session set up flow you will have the opportunity to use our Judgement Calculator. This allows you to define approximately how many Judgements you would like each Judge to make. Important here is the concept of Rounds.
Don't forget to check out the Blog
We regularly blog about workload
- Workload reduction and Adaptive Comparative Judgement
- Comparative Judgement On Demand - a viable alternative to marking at scale?
- Understanding time to judgement and workload.
Test your knowledge
Five quick questions to test your understanding