Managing Judge Workload (Basic Plan)
On the RM Compare Basic Plan, the number of rounds is set to 20 by default (A workload calculator is available on higher Plans). While this ensures a highly robust and reliable rank order, it can result in a high workload for your judges if you have a large number of items.
Because the RM Compare algorithm governs the rank order "second-by-second," you have full control over the actual workload. You do not necessarily have to complete all 20 rounds to achieve a valid, high-quality result.
1. Calculating the Maximum Workload
First, it is helpful to know the maximum number of judgements possible for your session. Use these formulas to plan your judge's time:
- Total Session Judgements:
Number of Items x 20 / 2 - Maximum Judgements per Judge:
Total Session Judgements / Number of Judges - Example: If you have 100 items and 10 judges, the maximum workload is 1,000 total judgements, or 100 judgements per judge.
2. How to Reduce Workload
If 20 rounds creates too much work for your team, you can manage the session in three ways:
A. The "Stop When Reliable" Method
You can monitor your session's progress in real-time. Most sessions reach high reliability long before the 20th round.
- Action: Monitor the Reliability Coefficient in your dashboard. Once it reaches your target, simply End the Session. The rank order at that moment is your final result.
B. Set a Judgement "Quota"
You can calculate how many judgements are needed to reach a specific "effective" round count (e.g., 10 rounds) and ask judges to stop once they hit that number.
- The Math: Items x Desired Rounds / 2 = Target Judgements
- Action: If you have 100 items and want 10 rounds of reliability, instruct your judges to stop once the session reaches XXX total judgements.
C. Increase the Judge Pool
Because the total pool of work is shared, adding more judges is the fastest way to lower individual effort. Doubling your judge count effectively halves the time each person needs to spend in the system
Summary Example
If you have 100 items, here is the total number of judgements needed (estimate) to reach different levels of reliability if you have 10 judges.
8 (Formative/Low Stakes) / 400 Total Judgements Needed / 40 judgements per judge
12 (Summative/Standard) / 600 Total Judgements Needed / 60 judgements per judge
20 (Maximum Depth) / 1,000 Total Judgements / 100 judgements per judge
Example of Email guidance for Judges
Hello Team,
Thank you for agreeing to participate as a judge in our upcoming RM Compare session.
Our goal is to create a reliable rank order of [Insert Number of Items] items using the "Adaptive Comparative Judgment" process. To ensure we get a high-quality result without over-burdening everyone, please follow the guidelines below:
Your Judging Goal While the system is configured for a maximum of 20 rounds, we only require [Insert Effective Rounds, e.g., 10] rounds of comparison to reach our target reliability.
- Your Target: Please complete [Insert Judgements per Judge] judgements.
- Time Estimate: Based on the complexity of the items, we expect this to take approximately [Insert Minutes] minutes.
How it Works
- Log in to RM Compare and select the session: [Insert Session Name].
- You will be presented with two items side-by-side. Simply choose the one that better meets the criteria.
- The system will keep track of your progress. Once you have hit your target of [Insert Judgements per Judge], you can stop and sign out.
Important Note You may notice a "rounds" counter in the system that suggests there is more work to be done. Please ignore this. We are monitoring the session reliability in the background and will close the session once we have the data we need.
Deadline: Please complete your judgements by [Insert Date/Time].
Thank you for your expertise and contribution to this process.
Best regards,