7.3.1: James Standard Session 1 vs James Standard Session 2
Progress over time - unlocking trapped value
So far, we have considered individual sessions, however in many cases what we are really interested in in progress over time to see if learner performance is improving and is being impacted by interventions.
Introduction and Context
In 7.2.1 we looked at a Standard Session run by James acting as a Department Head in a school. The session allowed him to look at performance across 7 classes and introduce some interventions.
In this scenario James has repeated the session and is keen to see if the interventions were effective.
Data sets
Both spreadsheets were downloaded and added into the LLM which was then able to look across and make the comparisons.
The investigation
The following prompts who how a series of questions helped to uncover if James had seen any progress over time
Prompt: Produce a visualisation that compares class progress between the 2 sessions. Use the true values ('Param as scores')
Prompt: Will the chart always net out at zero as it's a 'zero-sum' game?
No, the True Score chart will not always net out at exactly zero.
While the initial assumption that the underlying assessment model (RM Compare, based on the Rasch model) is a "zero-sum game" is largely correct for ranking at a single point in time, it does not apply to measuring progress over time.
Here is why the average change of all students across two sessions does not have to be zero:
1. Zero-Sum Applies to the Anchor, Not Progress
In a single assessment session, the raw Parameter Value scale (the logit scale) is mathematically centered—usually the mean difficulty of the items or the mean ability of the students is anchored at zero logits.1 This forces a relative, zero-sum ranking: if one student's logit score goes up, another's must go down relative to that center.
2. Progress is Measured Against a Fixed Standard
When you calculate the change between Session 1 and Session 2, you are comparing the scores of the entire group against a fixed, absolute standard (the anchored scale).
- If the entire cohort genuinely improves (i.e., every student produces better work in Session 2 than in Session 1, or their total score increases significantly), the mean Parameter Value for the whole cohort will increase, and the True Score change will be positive.
- If the cohort declines (as was the case for the small net change in your data), the mean Parameter Value will decrease, and the True Score change will be negative.
The chart is not comparing students against each other in Session 2, but comparing their Session 2 performance vs. their Session 1 performance against a stable, absolute ruler.
3. Evidence from Your Data
In the chart you reviewed, the weighted average of all class True Score changes actually nets out to approximately -0.03. This demonstrates that the cohort, as a whole, experienced a very slight average decline, confirming that the net change across the entire group is not mathematically constrained to zero.
Prompt: The decline is a bit concerning. Does the data tell us anything about why this might be the case?
Progress by Attendance
A student's attendance percentage might directly impact their progress. We can look at the overall correlation, but also see if the most significant decliners had poor attendance
Based on the small sample of top gainers and decliners, there is no clear direct link between low attendance and performance decline. For instance, the biggest gainer (Nancy Campbell) and one of the largest decliners (Jennifer Brown) both had 0.78 attendance, while three of the top five decliners had near-perfect 0.97 - 0.99 attendance.
Progress by SEND and EAL Status
Special Educational Needs or English as an Additional Language status often influence educational attainment. Let's examine the percentage of students with these statuses in the high-performing vs. low-performing classes (based on True Score change).
Inference:
- EAL: The top-performing class (Class2) had a higher-than-average EAL percentage, while the lowest-performing class (Class3) had none. This suggests EAL status is not the primary factor driving the decline.
- SEND: There is no significant difference in the percentage of students with SEND across the high- and low-progress groups (all around 17-23%).
Class-Specific Dynamics
Since the data does not point to a systemic issue (like all SEND students declining), the concern may be localized to the management or teaching environment within the declining classes: Class3, Class7, and Class6.
- Decline in Class3 (Worst Decline: -1.39): The large average decline is driven by significant drops from Nicholas Harris (-0.69), Mary Williams (-0.67), Robert Young (-0.67), and a huge drop from Charles Carter (-0.51) who had a high Parameter Value (3.18) in Session 1. The decline here is spread across the ability spectrum.
- Decline in Class7 (Second Worst Decline: -1.29): This class's decline is heavily influenced by large drops from Jonathan Lopez (-0.70) and Janet Scott (-0.66).
Given the polarization, the most likely reasons for the overall poor net result are:
- Differing Teaching Interventions: Class2, which excelled, may have received highly effective, targeted instruction between sessions that other classes lacked.
- Item Specificity: The writing task in Session 2 may have inadvertently favored the specific skills being taught in the improving classes (Class2, Class5, Class4).
- Teacher/Classroom Environment: The classroom environment, morale, or change in a teacher in Class3, Class7, or Class6 may have negatively impacted engagement and performance compared to the top classes.