The science (and art) of building successful ranks and rulers

Underpinning RM Compare on Demand is the principle of converting ranks to rulers. Finding the most effective and efficient way of getting this done is going to take some practice.

Perhaps the greatest attraction of using a comparative approach in assessment is its perceived simplicity. Certainly the judge experience is a simple one to understand where the task is to assess a series of paired items against a holistic statement. However, running a successful session (one that will produce a 'good' rank that can be used successfully as a ruler) can have a degree of complexity.

• Items: Volume? Type? Content? Quality? Contribution Method?
• Judges: Who? Experience? Expertise?
• Workload: Number of judgements? Decision statement?
• Validity Measure: Content-related? Criterion-related?
• Reliability Target: High? Medium? Low?
• Session length: Minutes? Hours? Days?

What could possibly go wrong?

Setting up and running a successful RM Compare session can look deceptively simple - beware the Dunning-Kruger effect!

How to become an expert

As Dunning-Kruger highlights a big risk early on is over confidence - take some time to build up your expertise. For example

• Spend a bit of time in the Help-Centre. It is fully searchable and is full of great advice, tips and tricks
• Read a few Blogs. These are updated regularly and feature examples of best practice.
• Take a look at some of the Research Papers available. ACJ is a relatively new area of assessment and there is an active (but small) research community.
• Get in touch with the RM Compare Team - we are here to help.
• Run a bunch of trial sessions on your own, perhaps using some demo materials.
• Start with some small scale, low stakes sessions.
• Practice!