Module 4: Analytical Questions

Diving Deep into Sessions and Data

Lessons

Lesson 1: Core Analytical Question Types

  1. Trend Analysis: “How has [metric] changed over time and what might this mean?”
  2. Pattern Recognition: “What similarities or clusters stand out in our judging data?”
  3. Outlier Detection: “Are there any items, judges, or cohorts that behave unusually?”
  4. Diagnostic Investigation: “What factors could explain a drop in reliability or engagement?”
  5. Action Planning: “Given what we’ve discovered, what next steps or improvements should we prioritize?”

Trend Analysis Questions

Template:
"Analyze changes in [session metric] over [time period] and explain possible causes for any significant shifts."

Example:
"How did reliability scores across session types change from Term 1 to Term 2, and what factors might explain this?"

Pattern Recognition Questions

Template:
"Identify and explain patterns or commonalities in [session results/judge decisions/item outcomes]."

Example:
"Do certain judging cohorts consistently agree or disagree, and what does that tell us?"

Outlier Detection Questions

Template:
"Spot any outliers in [session data], and suggest how to investigate or address them."

Example:
"Were there any judges or sessions with surprising disagreement on rankings, and what actions should follow?"

Diagnostic Investigation Questions

Template:
"Review our [session outcome/problem] and help me list possible contributing factors."

Example:
"Our last session had much lower reliability than usual; how can we use data to work out why?"

Action Planning Questions

Template:
"Based on this session analysis, what practical steps should we take to improve results next time?"

Example:
"Given higher reliability from collaborative judging, what process improvements should we try in future assessments?"

Lesson 2: Scenario and Sequence Questioning

Example Sequence:

  1. "What metrics showed the biggest change this term?"
  2. "Which judge decisions contributed most to that change?"
  3. "Are there any clusters that need special attention?"
  4. "What targeted interventions would address these findings?"

Practice:
Compose an analytical sequence targeting a challenge in your own RM Compare sessions.

Lesson 3: Analytical Question Practice

Exercise 1: Try a trend question for a metric in your recent session data
Exercise 2: Use a pattern recognition prompt to review item clusters
Exercise 3: Build an outlier investigation for a judge or item result
Exercise 4: Create a diagnostic question for low engagement
Exercise 5: Plan an action using your LLM’s recommendations

Lesson 4: Building Your Analytical Question Library

Lesson 5: Validating Your Insights

Lesson 6: Module 4 Wrap-Up

You now can:

Next Module: Collaborative Questions – Understanding Connections and DataShare

Assignment:
Run your LLM through an analytical review of recent session data using three frameworks, capture useful insights, and propose one change based on your findings.

Related Articles