Simplified Pairs Accelerator

Comparative Judgement without the adaptive algorithm.

The Simplified Pairs method can be used to calibrate the scores (that is, the marks) from two assessments on the same scale.

Demonstration Video

How is the Simplified Pairs method used in RM Compare?

Standard RM Compare sessions employ an adaptivity algorithm to intelligently surface pairs of items for judgement – this is called Adaptive Comparative Judgement (ACJ). This approach has a number of benefits including a dramatic improvement in efficiency. However, a Simplified Pairs session needs to remove the adaptivity and instead take control of both the pairing process and the judge allocation.

For Example

Let’s say we want to calibrate Test A (30 Items / Scripts) with Test B (30 Items / Scripts). In a standard ACJ session, all 60 scripts would be uploaded into RM Compare for judging. Pairs of items would be surfaced by the algorithm – in some cases, the pairs would consist of both items from the same Test, in others, there would be one item from each Test. This approach does not meet our calibration requirement. Instead, we want to force a comparison between two tests. We may also want to control which judges see which pairings and in which order they see them.

Read more about how to set up a Simplified Pairs session here.

Test your knowledge

Simplified Pairs Accelerator Quiz

Further reading

Evaluating the simplified pairs method of standard maintaining using comparative judgement. Benton, T. & Gill, T. (2021, November 3rd). Presentation at AEA Europe conference 2021

Comparative Judgement for Linking Two Existing Scales. Benton, T. (2021). Frontiers in Education. 6:775203.

Does comparative judgement of scripts provide an effective means of maintaining standards in mathematics? Cambridge Assessment Research Report. Benton, T., Leech, T. & Hughes, S. (2020).

Comparing the simplified pairs method of standard maintaining to statistical equating. Cambridge Assessment Research Report. Benton, T., Cunningham, E., Hughes, S. & Leech, T. (2020).

Improving awarding: 2018/2019 pilots. Ofqual report Ofqual/19/6575. Curcin, M., Howard, E., Sully, K., & Black, B. (2019).

Related Articles