The core idea
A pilot explores feasibility; an impact evaluation asks what changed because of an intervention. An A/B test may use random allocation to compare approaches, but not every pilot is randomised or suitable for estimating impact. Define the outcome and analysis before seeing results where possible. The comparison should approximate what would have happened without the intervention, and practical or ethical constraints must shape the design.
Source and attribution [1]Using it in practice
Specify eligibility, allocation, outcome window and how people will be analysed. Consider whether assignment should occur by team or site to reduce spillovers. Estimate the sample needed for a meaningful effect and plan how missing outcomes will be handled. Do not withhold legal entitlements or essential support. Where randomisation is unsuitable, select a defensible alternative and report its assumptions.
An example, not a reported case
Worked example · illustrative
In an invented pilot, 90-day retention is 84% among 100 starters offered a new onboarding approach and 80% among 100 under the comparison approach. The observed difference is four percentage points. Under a simple independent-binomial approximation, its standard error is about 5.4 percentage points, so an approximate 95% interval is −6.6 to +14.6 points. The pilot is too imprecise to claim a clear retention improvement from these figures alone.
What to watch for
Before-and-after improvement can reflect seasonality or other changes. Repeatedly checking results and stopping at a favourable moment can distort inference. Small pilots may answer feasibility questions without resolving impact. Clustered allocation, non-compliance and spillovers require appropriate analysis beyond the simple example.
Define success in advance
Choose a primary outcome, meaningful effect size and review point. Include implementation measures and possible harms. A pilot can be useful if it reveals that delivery is impractical, even when it cannot estimate a precise effect.
Report the whole result
Show group counts, outcomes, uncertainty, deviations from the plan and contextual changes. Distinguish lack of evidence from evidence of no effect. Explain what the result supports doing next rather than reducing the decision to a p-value.
Specify the comparison before launch
Define the intervention, eligible population, outcome, follow-up period and comparison approach. Decide whether random allocation is feasible and appropriate, and at what level it should occur. Team-level delivery may create spillovers between individuals. Do not withhold legal entitlements or necessary support for an experiment. Obtain the relevant organisational and ethical review, and explain the process to participants as appropriate. A pilot should be designed to learn, not simply to demonstrate a preferred result.
Distinguish feasibility from impact
A small pilot can reveal whether delivery is practical and acceptable without being large enough to establish a reliable outcome difference. Measure uptake, implementation and problems as well as the intended result. If the intervention was not delivered consistently, a disappointing outcome may concern implementation rather than the underlying idea. Report that distinction. Conversely, enthusiastic feedback does not prove the intervention changed retention, performance or wellbeing.
Work through a simple illustrative comparison
Suppose two comparable groups each contain 100 eligible starters. Eighty in one group and 88 in the other remain at ninety days: an observed eight-percentage-point difference. That number alone does not establish statistical reliability or causation. Interpretation depends on allocation, follow-up, missing data, other differences and uncertainty. Do not present it as an 8% relative improvement; the relative increase from 80% to 88% is 10%.
Decide in advance how to respond
Set criteria for continuing, adapting or stopping, including costs and unintended harms. Analyse the outcome as planned and label exploratory findings separately. Avoid repeatedly checking for a favourable result and stopping at that moment. If the evidence remains uncertain, say so and consider a better-powered or better-designed follow-up. The decision may still use feasibility and practical judgement, but it should not claim stronger causal evidence than the design provides.
Define the comparison before seeing the result
State the outcome, observation period, assignment approach and decision rule in advance. Where random allocation is appropriate and feasible, it can strengthen the comparison; where it is not, explain how groups differ and what that means for interpretation. Consider whether managers or employees in one group will share the intervention with another. Include implementation evidence so a weak result can be distinguished from a programme that was never delivered as intended.
Report a useful conclusion even when the result is uncertain
A small pilot may establish feasibility and reveal barriers without providing a precise effect estimate. In an invented onboarding trial, employees may value the new support while retention data remain too limited for a strong conclusion. Report both rather than declaring success from satisfaction alone. Agree safeguards, appropriate access and fair treatment for participants. Decide whether to refine, extend or stop based on the quality of the evidence and the practical consequences, not only whether a statistical threshold was crossed.
Take it into your next conversation
Three useful questions.
- What question can this method answer, and what can it not establish?
- Are the comparison, observation period and assumptions defensible?
- What decision follows, and how will its consequences be reviewed?
Related terms
Go to the evidence
Sources & attribution
Further reading: Nadeem Khan and Dave Millner (2023), Introduction to People Analytics: A practical guide to data-driven HR, 2nd edition, PDF pp. 227, 244 and 294–295. Edition & reading notes ↗
[1] HM Treasury: Magenta Book Annex A analytical methods ↗
The core idea is an original summary of the cited work. Application notes, examples and sketches are our interpretations, not quotations or reproductions of the authors’ figures. Publisher records may require access to read the full original work.
Application guidance expanded on 26 September 2026. These practical suggestions and invented examples are editorial explanations; further-reading sources are not the original authors of the historical models unless expressly identified. Original overview review dates are retained.
Published 2026-09-20 · Reviewed 2026-09-22. Editorial approach