From Item Repositories to Standards Repositories: Making Shared Quality Reusable

For many education and professional-assessment organisations, the challenge is not a lack of evidence. It is a lack of shared, durable access to what good performance looks like.

Teachers, assessors and moderators build strong local understanding of quality through experience, discussion and review of real work. But when that understanding is not captured in a form that others can inspect and reuse, standards can diverge across classes, teams, institutions and assessment cycles.

The result is a familiar pattern: valuable moderation happens, shared insight develops, and then much of that insight disappears when the event ends.

Adaptive Comparative Judgement has helped address part of this problem. By asking judges to make structured comparisons between pieces of work, ACJ can generate highly reliable rank orders for complex performances such as writing, design work and portfolios. Studies have reported high levels of rank-order reliability, although reliable assessment still depends on factors including task design, the expertise of judges and the appropriate use of the method.

But a reliable rank order is not, on its own, a durable system standard.

The next opportunity is to preserve the evidence, context and calibration created through assessment so it can support the next cohort, the next moderation conversation and the next decision. That requires moving beyond an item repository towards a standards repository.

The limits of an item repository

The idea of a repository of student work is becoming increasingly important in education policy. Northern Ireland’s 2026 Independent Review of Statutory Assessment recommended a Writing Repository to gather examples of pupil writing from schools, enable anonymous comparison across the system and support improvements in writing standards with lower workload than traditional approaches (dera.ioe.ac).

This is an important development. A well-designed item repository can make authentic work searchable and useful by retaining context such as:

• Subject, curriculum area and age or phase.
• Task type, purpose and intended audience.
• Date, cohort and assessment conditions.
• Associated criteria, annotations or teacher rationale.
• Relevant permissions, anonymisation and retention information.

That turns individual pieces of work into meaningful exemplars rather than disconnected files.

Yet even an excellently catalogued repository remains primarily a library. It can help people find examples, but it does not necessarily explain how those examples relate to a consistent, defensible standard of performance.

What makes a standards repository different

A standards repository includes an item repository, but it goes further.

It connects selected examples of performance to documented calibration evidence: their position on an agreed reference scale, the task and context in which that position was established, the comparisons and professional judgements that informed it, and any associated standard-setting decisions.

In other words, it does not simply store work. It stores the evidence that makes work useful as a reference point for quality.

Layer	What it contains	What it enables
Item repository	Student work, metadata, annotations and permissions	Search, retrieval, inspection and sharing of authentic exemplars
Calibration layer	Comparative-judgement evidence, scale positions, confidence information and calibration history	Creation of trusted reference sets and benchmark items
Standards repository	Governed reference scales, benchmark collections, access controls, version history and deployment rules	Reusable standards that can support moderation, assessment and professional learning

The distinction matters because a standard is not merely an example of good work. It is an example whose meaning has been established, documented and maintained within a shared frame of reference.

From assessment event to institutional asset

Consider a network of schools using a curated set of writing samples as a reference scale.

Each sample is linked to the task it answers, the age group, relevant curriculum objectives and evidence from a calibration process. New work can be compared with those known reference points. Teachers can discuss the comparisons, understand the characteristics that distinguish stronger from weaker performance and use the same evidence in moderation or professional development.

The organisation has moved from a one-off judgement event to a reusable shared asset.

That is the practical promise of a standards repository:

• It preserves valuable calibration rather than requiring every assessment cycle to begin from zero.
• It makes standards visible through authentic examples, not solely through abstract descriptors.
• It supports consistency while retaining professional judgement.
• It enables carefully governed sharing across classes, schools, networks or jurisdictions.
• It creates an evidence base that can inform assessment, moderation, training and improvement activity.

This should not be understood as freezing standards permanently. Standards need review as curricula, tasks, populations and policy expectations change. But a governed repository makes that review explicit: organisations can see which reference sets are current, which have been superseded and what evidence supports their continued use.

A steady-state approach to standards

For many organisations, assessment remains cyclical and labour-intensive. Work is collected, moderation is organised, a result is reached and the process is repeated next time.

A steady-state model is different. It maintains a trusted, reviewed body of benchmark work that can be used repeatedly and updated deliberately. New work is considered against an established reference framework, while new evidence can be incorporated through controlled recalibration and versioning.

This does not eliminate the need for professional judgement. It gives that judgement a stronger shared foundation.

In this model, assessment data becomes more than an output of a particular event. It becomes a living organisational resource: evidence of what standards mean, how they were established and how they can be applied consistently.

The opportunity for assessment infrastructure

The move from item repositories to standards repositories changes the role of assessment technology.

Instead of treating assessment as a sequence of isolated events, organisations can build an infrastructure for storing, governing and deploying shared standards. Authentic work becomes searchable. Calibrated examples become reference points. Professional judgement becomes visible, reviewable and reusable.

For RM Compare, this is an opportunity to support more than the creation of a reliable rank order. It is an opportunity to help organisations retain the value created through comparative judgement, and turn it into a durable, governed standard that can be used wherever decisions about quality need to be made.