Measuring impact without vanity metrics
A transparent approach to pilots, learning and defensible claims.
What this is really about
Downloads and clicks show usage, not better decisions. In pilots we separate reach, active engagement, career-management skills and longer-term outcomes.
Your next steps
Work through these steps at your own pace. Good career decisions come from concrete comparisons, not from finding one perfect answer.
• Define learning questions in advance.
• Combine quantitative signals with qualitative conversations.
• Report negative and ambiguous findings.
• Do not claim causality without a suitable design.
Questions for a useful conversation
Take these questions to a careers adviser, teacher, family member or apprenticeship employer.
• What change would matter to the audience?
• What can this design genuinely establish?
• Which unintended effects will we check?
From activity to impact
We separate reach, use, immediate learning, changed behaviour and longer-term outcomes. Opening a resource is use. Being able to compare two options using personal criteria may indicate learning. Arranging an employer visit is behaviour. A later transition may matter but is influenced by many factors and cannot automatically be attributed to the product.
Before a pilot we define a small set of learning questions, expected mechanisms and possible harms. We state which data is genuinely necessary, how qualitative voices are included and when a claim cannot be made. Subgroups are analysed only when safety and meaningful interpretation are possible.
An honest pilot scorecard example
A school might test whether, after two sessions, learners can name more concrete options, distinguish sources and set a next action. That is not evidence of better long-term outcomes. Interviews should also capture non-use, confusion and unintended pressure.
• Define the learning question and intended change in advance.
• Describe baseline and participation bias.
• Do not equate usage with impact.
• Report negative, neutral and ambiguous findings.
• Tie continuation decisions to criteria named in advance.