Setting-up simplified pairs
The Simplified Pairs method of Comparative Judgement is used to calibrate the scores (that is, the marks) from two assessments onto the same scale.
How is the Simplified Pairs method used in RM Compare?
Standard RM Compare sessions employ an adaptivity algorithm to intelligently surface pairs of items for judgement – this is called Adaptive Comparative Judgement (ACJ). This approach has a number of benefits including a dramatic improvement in efficiency. However, a Simplified Pairs session needs to remove the adaptivity and instead take control of both the pairing process and the judge allocation.
Let’s say we want to calibrate Test A (30 Items / Scripts) with Test B (30 Items / Scripts). In a standard ACJ session, all 60 scripts would be uploaded into RM Compare for judging. Pairs of items would be surfaced by the algorithm – in some cases, the pairs would consist of both items from the same Test, in others, there would be one item from each Test. This approach does not meet our calibration requirement. Instead, we want to force a comparison between two tests. We may also want to control which judges see which pairings and in which order they see them.
How do I set-up the item pairs and judges?
RM Compare has a specific pairs creation method. This is only made available on request – the majority of users do not currently have access to it. Those that do will see a change to the Create Session interface:
The functionality allows the session administrator to have granulated control of both Items and Judges:
What about the judge experience?
The judge experience is the same as it is for standard RM Compare sessions.
What about the results?
The removal of the ACJ algorithm from the session means that the standard reporting is unavailable. Instead, all data is accessible through a .csv download for bespoke analysis.
Transform educational outcomes for your students by being comparatively brilliant. Try RM Compare free and join the global assessment revolution!
Evaluating the simplified pairs method of standard maintaining using comparative judgement. Benton, T. & Gill, T. (2021, November 3rd). Presentation at AEA Europe conference 2021
Comparative Judgement for Linking Two Existing Scales. Benton, T. (2021). Frontiers in Education. 6:775203.
Does comparative judgement of scripts provide an effective means of maintaining standards in mathematics? Cambridge Assessment Research Report. Benton, T., Leech, T. & Hughes, S. (2020).
Comparing the simplified pairs method of standard maintaining to statistical equating. Cambridge Assessment Research Report. Benton, T., Cunningham, E., Hughes, S. & Leech, T. (2020).
Improving awarding: 2018/2019 pilots. Ofqual report Ofqual/19/6575. Curcin, M., Howard, E., Sully, K., & Black, B. (2019).