- Product
RM Compare .......... On-Demand?
Reimagining Assessment - Time for some 10X thinking?
The concepts and benefits of 10X thinking (sometimes called Moonshot thinking) have been around for a while now. Championed by the likes of Larry Page at Google in simple terms it challenges us to think about how we can improve what we are doing by a factor of 10 times, rather than 10% like everyone else. RM Compare On-Demand is our moonshot project.
So what's the challenge?
The collaborative approach at the heart of an Adaptive Comparative Judgement (ACJ) session (and one of the things that makes it so brilliant) demands that we bring together a bunch of items and judges so everyone can work together to produce a rank. At times this can be a bit like herding cats, and we know that things get more challenging as the size of the session increases. Our new Multi-Cohort design functionality helps a lot with this scaling challenge.
Each RM Compare session produces a unique rank. However, a specific ability score (parameter value) in one rank is not currently comparable to the same ability score in another rank. Ways to overcome this frequently involve transforming data into scaled scores and onwards to national standards (writing for example), however these workarounds have limitations, especially if we want to consider high stakes assessments.
To use ACJ properly at scale (for example in national assessments) we need to move beyond isolated ranks. Solving this takes us to the moon and beyond!
A dramatic leap into the future of schoolbased assessment
Let's imagine that we have solved the 'isolated ranks' challenge. So what?
Well, this allows us to introduce the concept of an ACJ Ruler - a core rank that can be used repeatedly. It is possible that we could use the ruler as a benchmark to merge separate ACJ ranks from schools across regions and even nations. One ruler to rule them all!
This is where the On-Demand concept comes from because we can assess against the ruler at any time.
So how might this work?
In a regular ACJ sessions all items are moving up and down the rank as judgements are made. In an On-Demand session ruler items would be fixed, and only new items would move until they reach their point of stability. This has the added benefit of dramatically reducing the number of judgements required.
We can add an unlimited number of schools for example to the same ruler. All it requires is that judges make their judgements of the new schools work against the work in the ruler. By doing this we can easily create not just school ranks but also regional, national and international ones all with minimum effort while still retaining all the benefits associated with the underlying principles of Comparative Judgement.
Is this more than just an idea?
Yes! We have been thinking about this for a while and have been working with our partners at the Technology University of the Shannon to test some of the underlying principles and assumptions. At the recent PATT39 conference the team presented their initial findings which you can read HERE (jump to Page 515 for the ACJ work). We are excited by the learnings and validations.
Great! When can we use it?
Going back to the Moonshot analogy we have looked through the telescope and feel confident in our landing area. We have also run some small tests in the lab and the rocket design looks great. We will be working with our partners around the world to further validate our initial findings so we can run some more tests as we move toward our first trial launch. This is still someway off but we will keep you updated.
You can see where we are right now here.
If you would like to join us on the mission, or would just like to know more, please get in touch