← People analytics

People analytics · technique

A/B tests and pilot evaluation in HR

A pilot explores feasibility; an impact evaluation asks what changed because of an intervention.

Statistical and evaluation practice; ADM HR worked application · 2026 teaching guide · Original editorial explanation
A/B tests and pilot evaluation in HR — simplified model sketchComparable groups · Intervention / comparison · Outcome & uncertainty. Editorial interpretation, after Statistical and evaluation practice; ADM HR worked application (2026 teaching guide).Eligible groupInterventionComparisonCompareoutcomesADM HR · ORIGINAL INTERPRETATIONStatistical and evaluation practice; ADM HR worked application · 2026 teaching guide
Original simplified sketch after Statistical and evaluation practice; ADM HR worked application (2026 teaching guide). Read the explanation for assumptions and limitations. Download SVG ↗

The core idea

A pilot explores feasibility; an impact evaluation asks what changed because of an intervention. An A/B test may use random allocation to compare approaches, but not every pilot is randomised or suitable for estimating impact. Define the outcome and analysis before seeing results where possible. The comparison should approximate what would have happened without the intervention, and practical or ethical constraints must shape the design.

Source and attribution [1]

Using it in practice

Specify eligibility, allocation, outcome window and how people will be analysed. Consider whether assignment should occur by team or site to reduce spillovers. Estimate the sample needed for a meaningful effect and plan how missing outcomes will be handled. Do not withhold legal entitlements or essential support. Where randomisation is unsuitable, select a defensible alternative and report its assumptions.

An example, not a reported case

Worked example · illustrative

In an invented pilot, 90-day retention is 84% among 100 starters offered a new onboarding approach and 80% among 100 under the comparison approach. The observed difference is four percentage points. Under a simple independent-binomial approximation, its standard error is about 5.4 percentage points, so an approximate 95% interval is −6.6 to +14.6 points. The pilot is too imprecise to claim a clear retention improvement from these figures alone.

What to watch for

Before-and-after improvement can reflect seasonality or other changes. Repeatedly checking results and stopping at a favourable moment can distort inference. Small pilots may answer feasibility questions without resolving impact. Clustered allocation, non-compliance and spillovers require appropriate analysis beyond the simple example.

Define success in advance

Choose a primary outcome, meaningful effect size and review point. Include implementation measures and possible harms. A pilot can be useful if it reveals that delivery is impractical, even when it cannot estimate a precise effect.

Report the whole result

Show group counts, outcomes, uncertainty, deviations from the plan and contextual changes. Distinguish lack of evidence from evidence of no effect. Explain what the result supports doing next rather than reducing the decision to a p-value.

Specify the comparison before launch

Define the intervention, eligible population, outcome, follow-up period and comparison approach. Decide whether random allocation is feasible and appropriate, and at what level it should occur. Team-level delivery may create spillovers between individuals. Do not withhold legal entitlements or necessary support for an experiment. Obtain the relevant organisational and ethical review, and explain the process to participants as appropriate. A pilot should be designed to learn, not simply to demonstrate a preferred result.

Distinguish feasibility from impact

A small pilot can reveal whether delivery is practical and acceptable without being large enough to establish a reliable outcome difference. Measure uptake, implementation and problems as well as the intended result. If the intervention was not delivered consistently, a disappointing outcome may concern implementation rather than the underlying idea. Report that distinction. Conversely, enthusiastic feedback does not prove the intervention changed retention, performance or wellbeing.

Work through a simple illustrative comparison

Suppose two comparable groups each contain 100 eligible starters. Eighty in one group and 88 in the other remain at ninety days: an observed eight-percentage-point difference. That number alone does not establish statistical reliability or causation. Interpretation depends on allocation, follow-up, missing data, other differences and uncertainty. Do not present it as an 8% relative improvement; the relative increase from 80% to 88% is 10%.

Decide in advance how to respond

Set criteria for continuing, adapting or stopping, including costs and unintended harms. Analyse the outcome as planned and label exploratory findings separately. Avoid repeatedly checking for a favourable result and stopping at that moment. If the evidence remains uncertain, say so and consider a better-powered or better-designed follow-up. The decision may still use feasibility and practical judgement, but it should not claim stronger causal evidence than the design provides.

Define the comparison before seeing the result

State the outcome, observation period, assignment approach and decision rule in advance. Where random allocation is appropriate and feasible, it can strengthen the comparison; where it is not, explain how groups differ and what that means for interpretation. Consider whether managers or employees in one group will share the intervention with another. Include implementation evidence so a weak result can be distinguished from a programme that was never delivered as intended.

Report a useful conclusion even when the result is uncertain

A small pilot may establish feasibility and reveal barriers without providing a precise effect estimate. In an invented onboarding trial, employees may value the new support while retention data remain too limited for a strong conclusion. Report both rather than declaring success from satisfaction alone. Agree safeguards, appropriate access and fair treatment for participants. Decide whether to refine, extend or stop based on the quality of the evidence and the practical consequences, not only whether a statistical threshold was crossed.

Take it into your next conversation

Three useful questions.

  1. What question can this method answer, and what can it not establish?
  2. Are the comparison, observation period and assumptions defensible?
  3. What decision follows, and how will its consequences be reviewed?

Related terms

Go to the evidence

Sources & attribution

Further reading: Nadeem Khan and Dave Millner (2023), Introduction to People Analytics: A practical guide to data-driven HR, 2nd edition, PDF pp. 227, 244 and 294–295. Edition & reading notes ↗

[1] HM Treasury: Magenta Book Annex A analytical methods ↗

The core idea is an original summary of the cited work. Application notes, examples and sketches are our interpretations, not quotations or reproductions of the authors’ figures. Publisher records may require access to read the full original work.

Explore People Analytics →

Application guidance expanded on 26 September 2026. These practical suggestions and invented examples are editorial explanations; further-reading sources are not the original authors of the historical models unless expressly identified. Original overview review dates are retained.

Published 2026-09-20 · Reviewed 2026-09-22. Editorial approach

Keep exploring

A little more context.

People analytics · practical playbook01

From a business question to a measured outcome

A complete worked route through casual-worker retention, from definitions to a fair pilot and an honest conclusion.

ADM HR editorial workflow6 min
People analytics · models & decisions02

Logic models for HR interventions

Separate resources, activities and deliverables from the changes an initiative is meant to produce.

Programme-evaluation tradition6 min
People analytics · technique03

Cohort analysis: compare like starting points

A cohort is a group defined by a shared starting event or period, such as joining in a particular quarter.

Statistical and evaluation practice5 min