- Research
Research: Enhanced assessment and learning through Adaptive Comparative Judgement
The School of Engineering (University of Liverpool) has recently reported on a study they have completed to test the potential of RM Compare in 3 areas
- Peer assessment: Facilitate and inform peer discussion to help students reflect on their own attitudes to learning.
- Feed Forward: Reviewing and contextualising previously submitted work to help students to properly understand assignment expectations.
- Grading: Informing the final grade of a summative assessment and how the outcomes of completing an RM Compare activity compare between students and teachers.
Different deployment approaches were tested and evaluated. A total of 390 students were invited to take part. Participation rates varied and reasons for this are discussed in the report, with recommendations on how this could be improved.
The results suggested strong support for the potential of ACJ to have a positive impact throughout the learning journey for students and staff.
Note! The authors received no financial support for this work and have no affiliation with RM Compare or their employees.
| Table 1: General student perceptions | The RM Compare platform is easy to use | I would like to see RM Compare used in other activities |
|---|---|---|
| Strongly agreed | 65.2% | 25.9% |
| Agreed | 34.8% | 38.9% |
| Neutral | 0% | 31.5% |
| Dis-agreed | 0% | 1.9% |
| Strongly dis-agreed | 0% | 1.9% |
Key Results
Case Study 1: Peer Assessment
Students were asked to work individually to judge work submitted by their peers, offering feedback as part of the process through the RM Compare system.
The results concluded that "whilst somewhat effective in helping learners to improve their understanding of a topic more generally, it is clear that ACJ can help a student understand the quality of their own work".
The results also suggested RM Compare can overcome logistical and cost challenges to organising poster review exhibitions, especially for large cohorts.
| Table 2: Peer assessment student perceptions | I have a better understanding of the topic after judging | I have a better understanding about the quality of my own work | Viewing and judging with RM Compare is better than just viewing posters in an exhibitions | <
|---|---|---|---|
| Strongly agreed | 15.2% | 43.5% | 35.8% |
| Agreed | 45.7% | 45.7% | 34.8% |
| Neutral | 26.1% | 8.7% | 23.9% |
| Dis-agreed | 10.9% | 2.2% | 4.3% |
| Strongly dis-agreed | 2.2% | 0% | 2.2% |
Case Study 2: Feed forward
At the start of a poster design task students were invited to judge and compare posters from the previous year's assignment.
The results showed that "by completing the feed-forward activity, the groups work will be of a higher quality". Those students who completed the feed-forward activity out-performed those who didn't.
| Table 3: Feed-forward student perceptions | I have a better understanding of the assignment after judging last years submissions | I have a better understanding about the quality of my own work |
|---|---|---|
| Strongly agreed | 25% | 65.5% |
| Agreed | 50% | 12.5% |
| Dis-agreed | 25% | 12.5% |
| Strongly dis-agreed | 0% | 12.5% |
Case study 3: Summative Peer Assessment
This study compared work assessed in 3 different ways
- Teacher rubric graded
- Teacher ACJ rank
- Student ACJ rank
The results indicated that "there is a broad correlation between ACJ rank orders and rubric-based grades". This has the potential to significantly reduce the time taken to grade assignments - "for example, grading all posters took around 3 hours using a rubric but took each judge around 30 minutes to complete the ACJ exercise".
| Table 4: Summative peer assessment student perceptions | I am comfortable with RM Compare being used to inform final grades in future assignments | I trust my classmates to use RM Compare properly |
|---|---|---|
| Strongly agreed | 20.4% | 13% |
| Agreed | 31% | 25.9% |
| Neutral | 31.5% | 29.6% |
| Dis-agreed | 7.4% | 22.2% |
| Strongly dis-agreed | 9.3% | 9.3% |
Conclusions
- "....usability results, when taken with the results showing that ACJ can bring additional learner benefits, indicate that there would be value in continuing to develop and deploy ACJ facilities."
- "The results around using ACJ to grade work are also promising; as class sizes get bigger and staff time becomes more precious, anything that has the potential to save time would be welcomed".
- "Importantly, ACJ is scalable - it can be used in classes of 50 or 500". This scalability is not only useful when thinking about grading for large cohorts, but also when designing activities for large cohorts".
- "There is evidence to show that students have benefited somewhat from the deep learning that comes with critical thought, all without major facilitation from instructors."
- "ACJ could replicate a poster exhibition, all be it asynchronously, and at a fraction of the cost".
Reference
ENHANCED ASSESSMENT AND LEARNING THROUGH ADAPTIVE COMPARATIVE JUDGEMENT
T. Topping, M. Murphy, S. Saunders (2023). ENHANCED ASSESSMENT AND LEARNING THROUGH ADAPTIVE COMPARATIVE JUDGEMENT. 379-389. 2023 19th International CDIO Conference, NTNU, Norway