Ranks, rulers and grades - a worked example

By Mark House

Hopefully you have had a chance to take a look at some of the new content in the Help Centre relating to the arrival of RM Compare Manual On-Demand.

In the following example we have taken a rank order, turned it into a ruler and applied some grade boundaries as a way to highlight some important concepts and understandings.

The Rank

We used the James Rank Order Session data for the purposes of this task. This is available as part of the Free Plan.

  • Candidates: 131 year 6 students from a federation of 13 schools all located in the same local authority. All schools had a non-selective intake.
  • Item Stimulus: "The local council have written to you as part of a consultation process. They are proposing to sell part of your school field to raise some money. All money raised would go toward education in the city - some of it will be allocated to your school to be spent on facility improvement. You have been asked to respond to the consultation by writing a formal letter to the council." (Note! additional information was provided to students to help them to consider the issues at hand. They were also taught the key writing skills needed for completing a task like this).
  • Items: Handwritten (Available here in the Demo Item Bank).
  • Judges: 36 teachers. All schools were represented in the judging pool.
  • Holistic Statement: "Which is the most effective piece of writing?" (Additional descriptors were provided to Judges).
  • Results: below (visit the Free Plan to see the full reporting data)

The Ruler

We have written before about the principle of transforming 'true scores' to 'scaled scores' as part of an approach toward grading.

Once we have items in a scaled score we can apply ruler boundaries, in this case at every 5% points. Within each ruler section we were then able to select a standardised item from the centre of the range (if your rank creation session is smaller you may just choose all the items within).

We now have a ruler with 20 standardised items on it. In the example below this presents itself as 5 items in each section of the ruler made available to Judges.

Grading

There are a number of approaches that could be considered for creating grade boundaries (shown in the example below)

  • Ranked: E.g. After Item 20 on the rank order
  • Scaled: E.g. After the top 10% of the scaled scores
  • Seeded: E.g. After the seeded item 17450.

A candidate experience - Bazza Timpson

Let's take a look at candidate Bazza Timpson. We can see (above) that his position on the original rank ('True Score') was 19. Depending on the grade boundary adopted here he may be awarded an A or a B grade.

On the scaled score we can see that he falls into the Ruler Item 3 Band - these bands are aligned to Scaled Grades. Items in Band 3 are graded B if we use the Scaled Grades.

Matching and Mis-Matching

In the following example 8 items were taken from the original rank. An attempt was then made by a user unfamiliar with the rank to complete a matching exercise using the ruler.

We can take these results and compare them with what we expected based on the original rank.

Name Expect Match Scaled Grade Actual Match Match Grade
L Timpson 1 A 1 A
B Timpson 3 B 4 B
T Flanders 6 B 6 B
R Flanders 8 C 8 C
N Munch 13 D 15 D
J Lovejoy 16 D 15 D
D Griffin 16 D 16 D
M Houten 20 E 19 E

We are able to observe a few important things here including

  • The Judge in this case made an exact match 50% of the time.
  • The error on mis-matches was small deviating by 1 ruler placement in 3 cases, and by 2 in another
  • The grade awarded was the same as expected

The crucial benefit of this process of course is the help it provides to assessors to become familiar with the standard, which takes us back to a core purpose of our work:

"How much simpler it would all be if teachers had – as a matter of normal practice – access to, and familiarity with, work from a national sample of schools, not just their own classroom?
Emeritus Professor Richard Kimbell

An Assessor familiar with the standard will have the confidence and competence to make judgements on new items as the come across them in their day-to-day work. Where they are uncertain they can use the ruler to assess any new items by repeating the matching exercise.

Always learning

A reminder that you can see our Roadmap here. You will see that this currently steers us toward building and consolidating on the ruler work completed so far while also looking toward a version that utilizes more fully Adaptive Comparative Judgement.

There is still a lot to learn and we very much welcome your feedback.

Over to you!

We can't wait to see what our users do with this new functionality. You will need an Advanced License to create a distribute rulers. Get in touch to find out more.