Setting-up simplified pairs
The Simplified Pairs method of Comparative Judgement can be used to calibrate the scores (that is, the marks) from two assessments onto the same scale.
How is the Simplified Pairs method used in RM Compare?
Standard RM Compare sessions employ an adaptivity algorithm to intelligently surface pairs of items for judgement – this is called Adaptive Comparative Judgement (ACJ). This approach has a number of benefits including a dramatic improvement in efficiency. However, a Simplified Pairs session needs to remove the adaptivity and instead take control of both the pairing process and the judge allocation.
For Example
Let’s say we want to calibrate Test A (30 Items / Scripts) with Test B (30 Items / Scripts). In a standard ACJ session, all 60 scripts would be uploaded into RM Compare for judging. Pairs of items would be surfaced by the algorithm – in some cases, the pairs would consist of both items from the same Test, in others, there would be one item from each Test. This approach does not meet our calibration requirement. Instead, we want to force a comparison between two tests. We may also want to control which judges see which pairings and in which order they see them.
How do I set-up the item pairs and judges?
RM Compare has a specific pairs creation method. This is only made available on request – the majority of users do not currently have access to it. Those that do will see a change to the Create Session interface:
The initial Simplified Pairs set up is no different to the regular session creation experience. However, when you add Items and Judges each entry is given a unique code. This is shown in the first column of the image below.
A simple .csv upload experience allows us to use the codes created to describe how we want paired items to present. We can now choose the pairings, which one is Item A or B, and which Judge they are displayed to.
We can see our pairings before running the session.
What about the judge experience?
The judge experience is the same as it is for standard RM Compare sessions.
What about the results?
The removal of the ACJ algorithm from the session means that the standard reporting is unavailable. Instead, we can see outcome of each judgement including any Judgement Comments. This data is downloadable to a .csv for further analysis.
Further Reading
Evaluating the simplified pairs method of standard maintaining using comparative judgement. Benton, T. & Gill, T. (2021, November 3rd). Presentation at AEA Europe conference 2021
Comparative Judgement for Linking Two Existing Scales. Benton, T. (2021). Frontiers in Education. 6:775203.
Does comparative judgement of scripts provide an effective means of maintaining standards in mathematics? Cambridge Assessment Research Report. Benton, T., Leech, T. & Hughes, S. (2020).
Comparing the simplified pairs method of standard maintaining to statistical equating. Cambridge Assessment Research Report. Benton, T., Cunningham, E., Hughes, S. & Leech, T. (2020).
Improving awarding: 2018/2019 pilots. Ofqual report Ofqual/19/6575. Curcin, M., Howard, E., Sully, K., & Black, B. (2019).