The rounds calculator
Learn how the concept of rounds work and how it can impact judge workload and the confidence in your results.
If we take a few of pieces of work (A, B, C & D) and a couple of teachers (1 & 2) we can see how the concept of rounds works.
| Teacher (judge) 1 | Teacher (judge) 2 | |
|---|---|---|
| Round 1 | Item A / Item B | Item C / Item D |
| Round 2 | Item A / Item C | Item B / Item D |
| Round 3 | Item A / Item D | Item B / Item C |
The key principle is that a round is completed when all work has been presented once.
The judging experience has no sense of rounds. Work is simply presented in pairs until an individual’s allocation is complete. Rounds are, however, useful and important to session administrators.
In simple terms we know that the more rounds we complete the more confident we are going to be in the relative merit of each piece of work. This does however have an obvious impact on the workload of judges, especially if there are not many of them and we have a lot of work to assess. The Rounds Calculator helps us to control this.
When setting up a session I can, if I choose, manually set the number of rounds - If I choose not to do this the default is 20. This facility is accessed through the link on the Judging section of the session admin dashboard.
In the Judgement Calculator, you can manually adjust the number of Judges, Items and Rounds. As you do this you will see the effect of each adjustment on the judge workload.
There is an obvious trade-off here. A lower judge workload simply means that work is seen less. This may mean that you may have less confidence in the final set of results. However, there are a number of things here that may influence how you finally set the Judgement Calculator, for example:
- How important is it that you have very high levels of confidence in the results?
- How experienced are the judges?
- What does the work look like? Is it easy and simple to compare or will each judgement take quite a bit of time?
- How easy will it be to discriminate the work? Is there a wide range of ability or is it tightly grouped?
In the image above we can see a session progressing. It has completed 16 rounds (that is each piece of work has been seen 16 times). Things are looking very good with very high levels of reliability (0.88). You will notice however that reliability has been pretty good for a while and we could, perhaps, have saved our judges some work by ending the session early.
Can't find what you’re looking for?
Get in touch, we are here to help.