Module 4: Analytical Questions
Diving Deep into Sessions and Data
- Duration: 45 minutes
- Focus: Move beyond basic reporting—use LLM-powered analytical questions to reveal patterns, diagnose problems, and lead real improvements
- Goal: Transform session data into strategic, actionable insight
Lessons
- Lesson 1: Core Analytical Question Types
- Lesson 2: Scenario and Sequence Questioning
- Lesson 3: Analytical Question Practice
- Lesson 4: Building Your Analytical Question Library
- Lesson 5: Validating Your Insights
- Lesson 6: Module 4 Wrap-Up
Lesson 1: Core Analytical Question Types
- Trend Analysis: “How has [metric] changed over time and what might this mean?”
- Pattern Recognition: “What similarities or clusters stand out in our judging data?”
- Outlier Detection: “Are there any items, judges, or cohorts that behave unusually?”
- Diagnostic Investigation: “What factors could explain a drop in reliability or engagement?”
- Action Planning: “Given what we’ve discovered, what next steps or improvements should we prioritize?”
Trend Analysis Questions
Template:
"Analyze changes in [session metric] over [time period] and explain possible causes for any significant shifts."
Example:
"How did reliability scores across session types change from Term 1 to Term 2, and what factors might explain this?"
Pattern Recognition Questions
Template:
"Identify and explain patterns or commonalities in [session results/judge decisions/item outcomes]."
Example:
"Do certain judging cohorts consistently agree or disagree, and what does that tell us?"
Outlier Detection Questions
Template:
"Spot any outliers in [session data], and suggest how to investigate or address them."
Example:
"Were there any judges or sessions with surprising disagreement on rankings, and what actions should follow?"
Diagnostic Investigation Questions
Template:
"Review our [session outcome/problem] and help me list possible contributing factors."
Example:
"Our last session had much lower reliability than usual; how can we use data to work out why?"
Action Planning Questions
Template:
"Based on this session analysis, what practical steps should we take to improve results next time?"
Example:
"Given higher reliability from collaborative judging, what process improvements should we try in future assessments?"
Lesson 2: Scenario and Sequence Questioning
Example Sequence:
- "What metrics showed the biggest change this term?"
- "Which judge decisions contributed most to that change?"
- "Are there any clusters that need special attention?"
- "What targeted interventions would address these findings?"
Practice:
Compose an analytical sequence targeting a challenge in your own RM Compare sessions.
Lesson 3: Analytical Question Practice
Exercise 1: Try a trend question for a metric in your recent session data
Exercise 2: Use a pattern recognition prompt to review item clusters
Exercise 3: Build an outlier investigation for a judge or item result
Exercise 4: Create a diagnostic question for low engagement
Exercise 5: Plan an action using your LLM’s recommendations
Lesson 4: Building Your Analytical Question Library
- Archive the questions and approaches that yield the best insight
- Add prompts for regular session audits, judge feedback, and reporting cycles
- Share high-impact questions and findings with colleagues
Lesson 5: Validating Your Insights
- Cross-check your analyses against RM Compare official reports
- Peer-review findings in team meetings
- Use dashboard features and Help Centre analytics guidance for accuracy
Lesson 6: Module 4 Wrap-Up
You now can:
- Interpret RM Compare session data with powerful analytical questions
- Spot trends, troubleshoot problems, and improve results
- Build and share an evidence-driven learning culture
Next Module: Collaborative Questions – Understanding Connections and DataShare
Assignment:
Run your LLM through an analytical review of recent session data using three frameworks, capture useful insights, and propose one change based on your findings.