6 Counterfactual Designs for Small Samples
An evaluator facing one clinic, one classroom, or one patient still needs to ask whether an observed change reflects the intervention rather than something else. Single-case experimental designs (SCEDs) and N-of-1 trials answer that question by constructing the counterfactual from the same unit rather than from a comparison group: they compare what happened to a unit under treatment with what happened to it during a controlled absence, withdrawal, or alternation of treatment. Chapter 4 separated this effect claim from mechanism and configurational claims and from the inferential operations used to assess them. This chapter asks how far that logic can be pushed: how repeated observation and controlled variation in treatment can produce credible causal evidence from as few as one unit, and where the approach reaches its limits.
Within-unit comparison does not require large-sample distributional assumptions. Randomisation inference, an inferential overlay discussed in the second section of this chapter, provides exact p-values by enumerating all possible phase assignments. The result is a family of designs that can produce internally valid causal evidence from as few as one unit, provided the evaluator can observe the outcome repeatedly across phases. Chapter 3 identified boundary methods (synthetic control, interrupted time series) that also invoke counterfactual logic but require larger temporal n. This chapter focuses on designs that operate at genuinely small n and that give each unit its own counterfactual.
Several sources anchor this chapter, and each does different work. Kazdin (2011) and Barlow et al. (2009) set out the logic and variants of single-case experimental designs. Guyatt et al. (1986) established randomised trials in individual patients as a clinical form of within-unit comparison. Kratochwill et al. (2013) and Tate et al. (2016) specify design and reporting standards. Later work on randomisation, replication, and analysis addresses particular inferential problems rather than redefining the design family (Kratochwill & Levin, 2010; Tanious et al., 2024; Wiley & Rapp, 2019).
The bibliometric map (Chapter 5) finds 1,356 title-and-abstract pairs naming single-case experimental design, 767 naming N-of-1 trials, and 186 naming randomisation inference, against 318,950 candidate records retrieved across the whole corpus. These are surface mentions of terminology, not a count of evaluations that used the design, and they should be read with that caution throughout this section. The literature that generates them sits mostly outside social-policy evaluation: SCED records concentrate in Developmental and Educational Psychology, Clinical Psychology, and Cognitive Neuroscience, and N-of-1 trial records concentrate in Statistics and Probability, Psychiatry and Mental Health, and Genetics, with the largest institutional-country shares from the United States, the United Kingdom, and Australia. Randomisation inference is narrower still: 114 of its 186 pairs (61 per cent) fall in Statistics and Probability, which reads as a methodological literature rather than an applied-evaluation one. Figure 5.2 shows all three terms rising from a low base since 2000; none of the three trends supports a claim about how often within-unit designs are used in development or social-policy evaluation specifically.
6.1 Single-case experimental designs and N-of-1 trials
SCEDs are a family of experimental designs in which each unit is observed across multiple phases, and the investigator controls when the treatment is introduced, withdrawn, or alternated (Epstein & Dallery, 2022; Kazdin, 2011). The counterfactual is constructed by comparing the same unit’s outcome between treatment and non-treatment phases. Because the unit serves as its own control, between-subject variability is eliminated. The family contains four principal design types, each suited to different assumptions about the outcome and the intervention. Table 6.1 summarises the key features.
| Design | Counterfactual construction | Reversibility required | Minimum phases | Replication type |
|---|---|---|---|---|
| Reversal (ABA/ABAB) | Baseline phase serves as the counterfactual for the treatment phase | Yes | 3 (ABA) or 4 (ABAB) | Within-case (phase repetition) |
| Multiple baseline | Untreated baselines of other units, settings, or behaviours serve as concurrent controls | No | 2 per tier; at least 3 tiers | Across-case or across-setting |
| Alternating treatments | Rapid alternation of two or more conditions within the same unit | No (comparison is concurrent) | Sufficient alternation points for separation | Within-case (condition comparison) |
| Changing criterion | Outcome is expected to track a sequence of pre-specified performance criteria | No | Baseline plus successive criterion phases | Within-case (criterion shifts) |
SCED phase notation. A = baseline (no treatment); B = treatment. ABA = baseline–treatment–return to baseline. ABAB adds a second treatment phase to strengthen within-case replication.
Reversal designs (ABA or ABAB) introduce the treatment in phase B and withdraw it to return to baseline in a subsequent A phase (Barlow et al., 2009). The counterfactual for the treatment phase is the observed baseline; the counterfactual for the return-to-baseline phase is the preceding treatment phase. The design assumes that the outcome is reversible: if the treatment effect persists after withdrawal, the return-to-baseline phase will not show a change, and the design cannot distinguish a lasting treatment effect from no effect. Reversal designs are strongest when the outcome fluctuates with the treatment, as in behavioural interventions or dosage studies.
Multiple baseline designs stagger the introduction of the treatment across at least three units, settings, or behaviours (Kratochwill et al., 2013). Each baseline that has not yet received the treatment serves as a concurrent control for the unit that has. The design does not require reversibility, which makes it suitable for interventions whose effects are expected to persist, such as training, education, or infrastructure improvements. The staggered introduction rules out maturation and history threats: if the outcome changes only when the treatment is introduced, and this pattern repeats across tiers, the inference that the treatment caused the change is strengthened.
Alternating treatments designs rapidly switch between two or more conditions within the same unit, typically within a single session or across consecutive sessions (Barlow et al., 2009). The design compares conditions concurrently rather than sequentially, making it useful for comparing the relative effectiveness of two treatments. Changing-criterion designs set a sequence of graduated, pre-specified performance criteria and examine whether the outcome changes in step with each criterion (Barlow et al., 2009; Kratochwill et al., 2013). They are suited to interventions that seek progressive behaviour change, such as increasing attendance or activity, without withdrawing treatment. Combined designs merge elements from the four principal families. Epstein & Dallery (2022) describes combined designs as flexible but complex to analyse, because the inferential logic depends on which components are present.
N-of-1 trials are a subtype of SCEDs rather than a separate family (Epstein & Dallery, 2022). They are personalised crossover trials in which a single participant receives treatment and control conditions in a randomised sequence, often with blinding (Guyatt et al., 1986). N-of-1 trials are most common in clinical medicine, where the goal is to determine whether a specific treatment works for a specific patient. The randomised phase order and blinding distinguish N-of-1 trials from standard reversal designs, but the underlying logic is the same: within-unit temporal comparison.
The choice among these designs depends on the intervention’s temporal properties. A reversal design is defensible only when both the effect and its removal can occur within the observation period. The evaluator must establish this before assigning a withdrawal phase. It is not enough that a programme can administratively be stopped: withdrawing a useful service may be unethical, and an effect that persists after withdrawal will not supply the return-to-baseline contrast the design requires. A washout period may help where an effect fades gradually, but its expected duration and the risks of withdrawal should be specified in advance (Barlow et al., 2009; Epstein & Dallery, 2022).
Multiple-baseline designs avoid withdrawal by using variation in start time. Their central requirement is not simply three units. The tiers must have outcomes that can be observed repeatedly, a credible reason why treatment can be introduced at different times, and limited interference between tiers. Staggering training for three clinics will not create independent evidence if staff move between clinics or if a national instruction changes their practice simultaneously. A common event that precedes changes in every tier remains a rival explanation. The design is strongest when outcome changes occur at different calendar times, closely after the planned introduction in each tier, and the untreated tiers show no corresponding change (Kratochwill et al., 2013; Tanious et al., 2024).
Alternating-treatments designs address a narrower question: which of two or more delivery conditions produces the better outcome for the same unit? They are useful only when the conditions can be applied distinctly and their effects can be measured before one condition carries into the next. They are poorly suited to interventions that accumulate learning, alter organisational routines, or require substantial set-up, because the observed outcome then reflects the sequence of conditions as well as the condition currently assigned. Randomising the order can reduce predictable sequencing, but cannot make carry-over disappear (Barlow et al., 2009).
N-of-1 trials make the same point in a clinical form. They are appropriate when the decision concerns an individual, treatment periods can be compared fairly, and the outcome is meaningful to that individual and responsive within the trial. Blinding is valuable when it can be maintained, but many behavioural, organisational, and policy interventions cannot plausibly be blinded. In those settings, describing a repeated before–after comparison as an N-of-1 trial risks implying protections that the design does not have. The appropriate label should follow the allocation, masking, and phase structure actually used (Epstein & Dallery, 2022; Guyatt et al., 1986).
No SCED variant substitutes for a design problem that it cannot address. A national reform implemented once, with annual outcome data and no plausible reversal, alternating condition, or staggered roll-out, does not become a single-case experiment because it concerns one unit. It may require an interrupted time-series design, synthetic control, process tracing, or a more limited descriptive question. The first design decision is therefore whether the intervention and outcome permit repeated, controlled contrasts at all. Naming a SCED after the fact cannot create them.
SCEDs build external validity through replication rather than through sampling. Within-case replication repeats phases within a single unit: an ABAB design replicates the treatment effect twice within the same case. Across-case replication repeats the entire design with new units. The two forms have different jobs. Within-case replication strengthens the causal conclusion for the studied unit; across-case replication examines whether the finding recurs beyond that unit. Without within-case replication, a single phase change could reflect coincidence. Without across-case replication, the finding may be idiosyncratic to one unit.
Randomisation within SCEDs strengthens the inferential basis. Tanious et al. (2024) describe two main strategies: randomising the onset of the treatment phase (common in multiple baseline designs) and randomising the sequence of conditions (common in alternating treatments designs). Blinding participants and assessors to phase assignment reduces expectancy and measurement bias. Randomisation also enables randomisation inference, discussed below, by providing the reference distribution against which the observed effect is tested.
Development evaluation has not yet adopted SCEDs widely, but several application domains are natural fits. Personalised behaviour-change interventions in health and nutrition programmes can use reversal or N-of-1 designs when the outcome is a repeated behaviour (adherence, dietary intake, hand-washing frequency). Adaptive management in fragile settings, where a single school, clinic, or district is the unit of interest, can use multiple baseline designs to stagger the introduction of a procedural change across sites. Treatment-development studies, in which a programme is being piloted before scaling, can use SCEDs to determine whether the intervention produces a detectable effect before investing in a large-sample trial.
Arrimada et al. (2018) followed 179 Spanish first-grade pupils with twice-weekly writing probes during 120 days of classroom instruction. The authors selected twelve pupils with low composition performance and poor handwriting for an additional 72 days of parent-supervised Tier 2 tasks. Ten showed improvement in composition quality, and eight performed within the normal range relative to peers at the end of the intervention. The study also used a comparison group of pupils whose baseline performance was just above the intervention threshold and a non-verbal measure to examine a maturational explanation.
This is a useful example of repeated measurement and an explicitly stated causal alternative, but not of a standard staggered multiple-baseline design. The Tier 2 phase began as a common change for the selected pupils. Its evidential value depends on the progress records, comparison group, and controls described by the authors; it should not be cited as though staggered treatment onset supplied the design’s main warrant.
Hampton et al. (2019) used a single-case multiple-baseline-across-participants design to examine therapist-implemented enhanced milieu teaching with three children with autism spectrum disorder in a South African special-school context. A random number generator determined the order in which participants began the intervention. Baseline and intervention sessions were video recorded; the primary outcomes were number of different words and spontaneous utterances. The authors report immediate changes in level, trend, or variability following intervention onset, while also noting variability, limited assessment of generalisation, and the need for further work on cultural and multilingual adaptation.
This is a stronger multiple-baseline illustration than the Spanish writing study because the onset order differed across participants. It also shows a limit often hidden in method descriptions: a design can support a functional relation in the sessions studied while leaving transfer to everyday classrooms, other communication partners, and other language settings unresolved.
Naylor & Bower (2005) report a single-case design with four-weekly assessments for young children with hemiplegic cerebral palsy: a four-week baseline, four weeks of modified constraint-induced movement therapy, and a four-week no-treatment follow-up, with each child serving as their own control. The reported gains in the use of the affected hand were greatest during the treatment phase and were partly maintained at follow-up. The design and reporting predate current SCED standards, and the pilot did not stagger onset across children or use a formal randomisation test.
The example is useful for a different reason: it shows what a baseline–treatment–withdrawal structure can and cannot establish when a skill, rather than a reversible symptom, is the outcome. Because motor gains from practice need not disappear once therapy stops, the follow-up phase cannot function as a clean return-to-baseline contrast; an improvement sustained after withdrawal is consistent with a lasting treatment effect but also with practice effects the design cannot separate from continued therapy.
Guyatt et al. (1986) describe double-blind, randomised treatment pairs within individual patients and report a trial that changed treatment for a patient with partially reversible airflow limitation. Active and placebo or alternative treatment periods were randomly ordered, while signs, symptoms, or laboratory measures served as treatment targets. The article is a foundational clinical demonstration of N-of-1 reasoning: the design was used to decide the optimal treatment for one patient, not to claim a population effect.
The case identifies the conditions that make this form of comparison credible. The condition was chronic and stable enough for repeated treatment periods, the outcome could respond within the study period, and active and comparison periods could be masked. These requirements explain why N-of-1 trials are a poor fit for many education, organisational, and policy interventions even though all use repeated observations.
Herrett et al. (2021) report the StatinWISE trial: 200 UK primary-care patients who had stopped or considered stopping a statin because of muscle symptoms were each randomised to a sequence of six double-blind two-month periods of atorvastatin or placebo, with symptom scores compared within each patient across periods. Aggregating the individual comparisons across participants found no overall difference in muscle symptoms between statin and placebo periods, and most participants who completed the trial intended to restart treatment.
The design illustrates a use of N-of-1 logic that goes beyond a single clinical decision: many replicated within-patient trials were combined to answer a population-level question about a common but contested side effect, while still returning an individual result to each participant. It depended on a chronic, stable condition, an outcome that could be rated within a short period, and blinding that could be maintained across repeated crossovers — conditions that rarely hold for programme, organisational, or policy interventions.
6.2 Designing a within-unit evaluation
The design decisions in a SCED are made before the first treatment phase begins. They concern the outcome, the unit, the phase structure, the treatment schedule, and the observations that will be regarded as evidence of a change. A graph cannot repair an outcome that was measured inconsistently, a treatment introduced while other changes occurred, or a phase schedule chosen after the result was known. SCED reporting standards therefore give considerable attention to procedural detail: what was measured, when measurement occurred, who observed it, when treatment changed, and whether the planned sequence was followed (Kratochwill et al., 2013; Tanious et al., 2024).
The outcome must be capable of repeated measurement at intervals that bear on the programme’s expected pathway. A daily adherence behaviour and an annual school-completion outcome do not provide the same basis for a within-unit comparison. Measurement also has to be stable across phases. If a district introduces coaching at the same time as it replaces a paper register with a digital system, a recorded increase in visits may reflect the new measurement process. The protocol should state the outcome definition, source, observation interval, person responsible for measurement, and any foreseeable changes to the measurement process. When assessors can be blinded to treatment phase, the protocol should say how this will be done and whether blinding can be checked.
Phase length is a substantive decision. The baseline must be long enough to show the level, trend, variability, and serial dependence of the outcome before treatment. The treatment phase must allow time for the intervention to operate and for the outcome to respond. No universal number of observations can establish either condition. A rapidly changing daily behaviour may permit a short phase; an outcome shaped by a school term, agricultural season, or administrative reporting cycle will usually require a longer one. The design should therefore justify phase length from the causal pathway and measurement process, not from a mechanical minimum (Kazdin, 2011; Kratochwill et al., 2013).
Multiple-baseline designs also require a defensible basis for staggering introduction. The tiers should be sufficiently independent that a change in one does not change the outcome in another, and the order of introduction should be determined before results are inspected. Where feasible, the evaluator can randomise the onset date within a pre-specified window. That decision both reduces discretion over the schedule and supplies the assignment mechanism required for a randomisation test. It does not remove threats created by a common shock affecting every tier at the same time, such as a national policy change, an epidemic, or a change in routine reporting.
The analysis plan should be specified alongside the treatment schedule. It should name the principal contrast, the graphical displays, the treatment of trend and autocorrelation, the effect-size measure where one is used, and the conditions under which a randomisation test will be reported. This avoids selecting a statistic only because it makes the observed pattern appear larger. It also makes clear that visual inspection and formal analysis answer related but different questions: the graph shows the observed trajectory, while an effect size or randomisation test summarises the contrast under stated assumptions.
Before data collection, the protocol should identify the causal claim; unit or tiers; outcome and observation schedule; expected timing and duration of any effect; phase sequence and rules for changing phase; treatment fidelity measures; concurrent events that could affect the outcome; planned handling of missing observations; analytic contrasts; and the scope of any generalisation. For a randomised design, it should also preserve the permissible assignment schedules and the procedure used to select the realised schedule.
6.3 Feasibility, ethics, and phase changes
The feasibility question is practical before it is statistical. A within-unit design requires a treatment contrast that can be created without exposing participants to unacceptable risk, an outcome that can be measured often enough to show the relevant change, and a delivery setting in which phase boundaries can be maintained. These conditions are more common in clinical, behavioural, and educational interventions than in policy reforms or services whose effects accumulate over years. The design should be rejected when its phase structure is incompatible with the intervention rather than retained because it offers a familiar label or an appealingly small sample size.
Withdrawal deserves particular scrutiny. Removing an effective treatment can be unacceptable where a participant faces a serious health, safeguarding, or educational risk. It may also be impossible: a person who has learned a skill, received a durable asset, or experienced a change in entitlement cannot be returned to the earlier condition. An ABAB structure is not ethically or substantively justified merely because a treatment can be administratively paused. Multiple baseline or alternating-treatment designs may offer alternatives, but they have their own requirements and should not be used to avoid an ethical discussion.
Phase changes should be governed by a rule fixed before outcome results are interpreted. A multiple-baseline study may specify an allowable start window and randomise the order within that window. A reversal design may define the duration of each phase and the conditions that would require stopping for safety. An N-of-1 trial may specify washout periods, masking arrangements, and clinical criteria for discontinuation. These decisions affect the causal contrast. Changing them after an apparent improvement or decline can create a design that follows the outcome rather than tests the intervention (Guyatt et al., 1986; Kratochwill et al., 2013; Tanious et al., 2024).
Treatment fidelity is equally important. A phase indicator in a dataset does not demonstrate that the intended intervention was delivered, received, or distinct from baseline practice. The protocol should identify the active components, how fidelity will be observed, and what departures will be recorded. If an intervention is implemented with different intensity across phases or tiers, the final claim must describe the contrast actually delivered. This is particularly relevant in service settings, where a new intervention may arrive with new supervision, reporting requirements, or management attention.
6.4 Claim boundaries and decision use
The output of a single-case design is a claim about a specified unit, treatment contrast, outcome, and period. The claim may be strong for that unit when the phase design, measurement, and treatment record rule out plausible alternatives. It says less about another unit with a different baseline condition, delivery context, or treatment intensity. This is not a reason to disregard single-case evidence. It is a reason to state the unit-level conclusion before drawing a programme-level implication.
The next decision should follow from the evidential gap. A positive and well-executed study may justify replication in a second setting, a systematic variation in delivery, or a larger trial. A null result may indicate that the treatment did not change the outcome under the conditions studied, that fidelity was weak, that the outcome was insensitive, or that the phase structure was too short to observe the expected pathway. The report should distinguish these possibilities using the implementation record and graph rather than treat every null contrast as evidence of no effect.
Commissioners should decide in advance what a within-unit result will be used for. The decision may concern an individual’s treatment, refinement of a school or clinic practice, selection between two delivery modes, or whether a pilot merits replication. It should not be a retrospective claim that a single successful case proved an intervention ready for scale. Replication across units and settings remains the route by which the external boundary of a within-unit finding is examined (Barlow et al., 2009; Tanious et al., 2024).
6.5 Analytical tools for within-unit designs
Visual inspection is the traditional analytical approach in SCED research. The analyst examines graphed data for changes in level, trend, and variability between phases. Visual inspection is transparent and requires no distributional assumptions. Its principal limitations are well documented: inter-rater agreement on whether a phase change is “meaningful” is often low; visual inspection is insensitive to small or gradual effects; and it provides no formal uncertainty quantification. Visual inspection remains a useful first step, but it is insufficient as the sole basis for causal claims.
Linear mixed-effects modelling (LMEM) offers a statistical alternative that handles the repeated-measures structure of SCED data. Wiley & Rapp (2019) demonstrate LMEM with data from a longitudinal training study of five individuals with acquired dysgraphia. The model specifies fixed effects for the treatment phase (capturing the average treatment effect) and random effects for individual variation (capturing how the effect differs across units). LMEM accommodates both binary outcomes (accuracy) and continuous outcomes (reaction time), handles unequal phase lengths, and accounts for autocorrelated errors. The key advantage is that LMEM produces standardised effect sizes and formal tests of statistical significance even with very few individuals. Wiley & Rapp (2019) argue that LMEM is preferable to visual inspection or simpler statistical techniques when the goal is to quantify the treatment effect.
LMEM for SCEDs. Wiley & Rapp (2019) demonstrate the approach with five individuals with acquired dysgraphia, showing that mixed-effects models yield standardised effect sizes and formal tests even at very small n.
Beyond LMEM, several effect-size metrics have been developed for SCED data. Non-overlap of all pairs (NAP) compares every baseline observation with every treatment observation and summarises the extent to which treatment-phase values are higher or lower, in the expected direction (Parker & Vannest, 2009). It is a non-parametric description of the observed phase contrast, not a correction for trend or a substitute for the design. Tau-U combines non-overlap with trend information; variants can account for a baseline trend when that trend is part of the analytic problem (Parker et al., 2011). The between-case standardised mean difference expresses a contrast in standard-deviation units and can support synthesis across studies when outcomes and designs are sufficiently comparable. These metrics communicate the size and direction of an observed change alongside, not instead of, the assumptions that make a causal interpretation plausible. Table 6.2 summarises the main analytical options.
| Tool | What it estimates | Data requirements | Output | Software |
|---|---|---|---|---|
| Visual inspection | Level, trend, and variability change | Graphed time series | Qualitative judgement | Any graphing tool |
| LMEM | Treatment effect (fixed) and individual variation (random) | Repeated measures; phase indicator | Effect size, p-value, confidence interval | R (lme4), SAS, Stata |
| NAP | Non-overlap between phases | At least 3 data points per phase | Effect size (0 to 1) | R (SingleCaseES), web calculators |
| Tau-U | Trend-corrected non-overlap | At least 5 data points per phase | Effect size, p-value | R (SingleCaseES), web calculators |
| Randomisation inference | Exact p-value under sharp null | Randomised phase assignment | p-value | R (SCRT), custom scripts |
The choice of analysis follows from the design and the question. Visual inspection is indispensable because it makes discontinuities, trends, outlying observations, and missing data visible. It is not enough to state that the graph ``looks convincing’’. The evaluator should specify the expected change and show whether the data display it: an immediate level shift, a change in slope, a reduction in variability, or a difference between alternating conditions. A model or non-overlap statistic can then describe the same contrast in a reproducible way. If the graph and the statistic point in different directions, that disagreement is a finding to investigate rather than a reason to report only the more favourable result.
Serial dependence deserves particular care. Repeated observations close in time are often related: a pupil’s writing score this week partly reflects last week’s score, and a clinic’s attendance this month partly reflects the previous month’s pattern. Treating each observation as independent can make a phase contrast appear more precise than the series warrants. The appropriate response depends on the design and data. It may involve modelling the dependence, using a randomisation test based on the actual assignment schedule, or reporting the uncertainty directly. It is not solved by collecting many observations that repeat the same measurement problem (Tanious et al., 2024; Wiley & Rapp, 2019).
Analyses should also distinguish an intervention effect from an implementation contrast. If the intervention phase includes a different observer, a change in eligibility, or additional monitoring, the estimate concerns that combined change unless the design separates its components. This is common in programme settings: introducing a new service can also alter supervision, data collection, and attention from managers. A credible report identifies those changes and states whether the design can distinguish them from the intervention of interest.
Randomisation inference provides a design-based inferential framework for any SCED in which phase assignments have been randomised (Edgington & Onghena, 2007; Kratochwill & Levin, 2010). The logic is straightforward. Under the sharp null hypothesis that the treatment has no effect on any observation, the observed sequence of outcomes would have occurred regardless of which time points were assigned to treatment and which to baseline. The analyst enumerates all phase assignments permitted by the randomisation scheme, computes a test statistic for each, and locates the observed test statistic in that reference distribution. The p-value is the proportion of assignments that yield a test statistic as extreme as, or more extreme than, the one observed. This produces an exact randomisation p-value without distributional assumptions. When exhaustive enumeration is infeasible, the analyst may draw a random sample of permissible assignments; the resulting Monte Carlo p-value approximates the exact value and should report its simulation uncertainty.
The sharp null hypothesis (no effect for any observation) differs from the weak null hypothesis (no average effect), and the choice between them has consequences. Under the sharp null, any individual observation is assumed unaffected; under the weak null, effects may cancel out across observations. The sharp null is the more stringent test and the one typically used in randomisation inference for SCEDs. Fisher proposed this form of exact inference; Neyman’s framework focuses instead on average effects and confidence intervals. For SCED applications, Fisher’s sharp null is the natural choice because the goal is usually to determine whether the treatment affected the specific unit under study.
Fisher vs Neyman. The sharp null (Fisher) posits no effect for any observation. The weak null (Neyman) posits no average effect. SCEDs typically test the sharp null, yielding exact p-values from the randomisation distribution alone.
Randomisation inference is not a standalone method. It is an inferential overlay that strengthens any randomised small-n design. The randomised phase-change schedule provides the reference distribution; the test statistic can be any summary of the treatment-baseline contrast, such as a mean difference, non-overlap statistic, or trend contrast. The approach combines naturally with LMEM: the model provides an effect estimate, while the randomisation distribution evaluates that estimate against the assignments the evaluator could actually have made. Bulté & Onghena (2009) describe randomisation tests for single-case designs and their implementation in the SCRT R package. The resulting p-value depends on the design’s permissible schedules. It is therefore an output of the assignment procedure, not a general measure of how convincing a before–after graph appears.
6.6 Measurement quality and data integrity
Within-unit designs place unusual weight on each observation. A small change in how an outcome is recorded, a missed measurement at a phase transition, or knowledge of the current treatment phase can affect the apparent contrast. The measurement protocol should therefore be treated as part of the causal design. It should define the outcome operationally, specify the observation schedule, record the person or instrument producing each measurement, and identify changes to routine data systems or assessors during the study.
Observer effects require particular attention when the outcome is assessed by programme staff or researchers. If assessors know that a treatment phase has begun, they may score behaviour differently even when the underlying behaviour has not changed. Independent assessment or masking can reduce this risk where it is feasible. When masking is impossible, the report should describe the risk and use records, outcome definitions, and inter-observer checks to show whether a change in observation practice could explain the result. The same issue arises when participants know the phase; expectancy and altered reporting can be part of the intervention contrast unless the design separates them.
Missing observations should be visible in the graph and the analysis record. Interpolation or deletion can create an apparently smooth phase pattern while concealing a practical failure in measurement or treatment delivery. The evaluator should record why observations are missing, whether missingness differs by phase, and whether the missing values occur at points that affect the planned contrast. A short series with several missing measurements around treatment onset may not support the conclusion the original design was intended to make.
Outcome reliability also affects replication. A measure that is responsive in one clinic, classroom, or household may be defined differently elsewhere. Before comparing effects across cases, the evaluation should establish that the outcome captures the same construct and that the observation schedule has comparable meaning. Standardising a metric after data collection does not resolve a difference in what observers counted or what an outcome represented in the two settings.
6.7 Reading graphs without replacing analysis
Graphs remain central because they disclose features that a single summary statistic cannot. They show the number and timing of observations, the baseline trend, the point of treatment introduction, variability within phases, missing measurements, and any apparent delay before an outcome changes. A report should reproduce the raw series or make it available, rather than display only modelled estimates or a bar chart of phase means. Readers need to see whether the claimed contrast is immediate, gradual, unstable, or driven by a small number of observations.
Visual inspection needs explicit questions. Did the outcome change at the planned phase boundary? Did a trend already exist in baseline? Was the change repeated across phases or tiers? Did variability increase after treatment began? Were there concurrent changes in measurement or delivery? These questions discipline graphical interpretation without turning it into an unstructured impression. They also identify where a model or randomisation test is unable to resolve the substantive issue.
Statistics should be selected because they represent the planned causal contrast. A mean difference may be unsuitable when the theory predicts a slope change. A non-overlap measure may be unhelpful when an outcome has a strong baseline trend. A randomisation test can assess an observed statistic under the actual assignment scheme, but it does not determine whether the outcome was measured well or whether the treatment contrast was credible. The report should connect each statistic to the feature of the graph and design that it is intended to summarise (Tanious et al., 2024; Tate et al., 2016).
6.8 Embedding within-unit designs in larger studies
Two strategies connect SCED evidence to broader evaluation frameworks. The first embeds N-of-1 trials within multi-site studies or randomised controlled trials. Each participant completes a personalised SCED, and the individual-level effect estimates feed into a group-level analysis. This design retains the within-unit causal logic while accumulating evidence across units. The second strategy meta-analyses across independently conducted SCEDs, synthesising effect sizes from separately published single-case studies using the between-case standardised mean difference or hierarchical models. Both strategies extend the reach of within-unit designs beyond the single case. Chapter 15 discusses integration protocols in detail.
Replication logic. SCEDs build external validity through replication, not sampling. Within-case replication repeats phases; across-case replication repeats the design with new units. See Chapter 15 for combination protocols.
6.9 Interpreting, reporting, and replicating a result
A phase contrast can support a narrow conclusion before it supports a policy recommendation. If the outcome changes when treatment changes, and the design has addressed plausible concurrent explanations, the result concerns the unit, outcome, treatment form, and period observed. It does not establish the effect for a different population, delivery organisation, or intervention intensity. The distinction is especially important where a small-n evaluation has been embedded in a pilot: a successful result may justify replication or adaptation, but does not by itself justify a claim about effects at scale.
The graph, the design record, and the statistical analysis should be read together. A statistically unusual contrast under a randomisation schedule can coexist with a small change of little practical consequence. Conversely, a clinically or operationally meaningful shift may be estimated imprecisely when there are few permissible assignments. Effect sizes, raw outcome values, and uncertainty information are therefore needed alongside a p-value. When the data display a gradual trend, a level change at phase transition, or a change in variability, the report should show which of those features the chosen statistic represents. A single summary measure cannot settle all three (Tate et al., 2016; Wiley & Rapp, 2019).
Reporting also needs to show what happened to the intervention and the measurements. A null result is difficult to interpret if the treatment was delivered inconsistently. A positive result is difficult to interpret if observers knew which phase was under way, observations were missing selectively after treatment began, or a new reporting system was introduced at the same time. SCED standards consequently require a description of treatment fidelity, procedural changes, inter-observer agreement where relevant, and the timing and extent of missing observations (Kratochwill et al., 2013; Tate et al., 2016). These are not administrative details. They indicate whether the observed series represents the intended treatment and outcome throughout the comparison.
Replication has two separate jobs. Within-case replication tests whether a pattern recurs when the treatment condition is reintroduced or contrasted again. Across-case replication tests whether the design produces a similar result in another unit, setting, or population. The second need not be identical to the first. A direct replication keeps the intervention, outcome, and phase structure as close as possible to the original study. A systematic replication deliberately changes a feature, such as the delivery setting or population, to examine the boundary of the earlier finding. The logic is cumulative: a series of adequately reported studies can show where an effect recurs and where it does not (Barlow et al., 2009; Tanious et al., 2024).
This does not make replication automatic. A multi-site programme may use the same label while differing in staff capability, treatment fidelity, baseline need, or the outcome that matters to decision-makers. Before pooling SCED estimates or describing a pattern as replicated, evaluators should establish that the intervention contrast and outcome definition are comparable. Hierarchical models and standardised effect sizes can help synthesise a set of studies, but they do not erase differences in the causal question each study addressed. The appropriate conclusion may be that an intervention worked under specified conditions, not that it has one stable effect across all settings.
6.10 Transportability and synthesis
Transporting a within-unit finding requires an argument about the treatment, the unit, the outcome, and the setting. A finding from a carefully delivered intervention does not automatically apply where staff have different training, participants have different baseline needs, or the outcome is measured differently. The relevant question is not whether the later setting is superficially similar. It is whether the features that made the original treatment contrast credible and the expected causal pathway plausible are present in the new setting.
Direct replication tests whether the same intervention and outcome produce a comparable pattern under closely similar conditions. Systematic replication changes one feature deliberately, such as delivery personnel, setting, or population, so that the boundary of the claim can be examined. Both should retain enough procedural detail to establish what was held constant and what changed. A series of studies that share only an intervention label cannot support a coherent synthesis if the active components or outcomes differ materially.
Evidence synthesis can combine standardised single-case effect sizes where designs and measures are sufficiently comparable. It should not begin with the assumption that every series can be pooled. The assessment needs to consider phase structures, baseline trend, outcome direction, treatment fidelity, randomisation, and the potential for selective reporting. Where these differ, a structured narrative comparison may be more informative than a pooled estimate. The aim is to establish where the counterfactual contrast has been replicated, not to manufacture one average effect from non-comparable studies.
For commissioners, the practical implication is straightforward. A single high-quality SCED can answer whether a treatment contrast changed a repeatedly measured outcome for a particular unit. Replication can examine whether that result recurs in a specified range of units and settings. Decisions about wider implementation should state which of those stages the evidence has reached. This preserves the value of a well-executed small-n design without asking it to establish a population claim it was not designed to make.
6.11 Strengths, limitations, and applicability
Table 6.3 summarises the conditions under which each SCED variant is applicable.
| Design | Minimum n | Key assumption | Causal claim strength | Main limitation | |
|---|---|---|---|---|---|
| Reversal (ABA/ABAB) | 1 | Outcome reverses on withdrawal | Potentially strong for the treated unit when phase changes recur | Cannot be used when effects persist | |
| Multiple baseline | 3 (tiers) | No spillover across tiers; staggered introduction feasible | Potentially strong when changes follow staggered introduction | Requires at least 3 comparable tiers; sensitive to co-occurring events | |
| Alternating treatments | 1 | Conditions do not carry over between alternations | Potentially strong for the condition contrast | Carryover effects can confound if conditions interact | |
| N-of-1 trial | 1 | Stable condition; outcome responds within treatment periods; carryover can be limited | Potentially strong for the individual treatment contrast | Repeated crossover and blinding are often impractical in behavioural or policy interventions |
The principal strengths of within-unit counterfactual designs are clear. They can produce causal evidence at n as low as 1 when their phase structure addresses plausible alternatives. Exhaustive randomisation inference provides exact p-values without large-sample assumptions. Each unit serves as its own control, removing stable between-subject differences from the comparison. Internal validity can therefore be high for the treated unit.
The limitations are equally clear. All SCED variants require repeated measurement: many time points per unit per phase. Reversal designs assume that the outcome returns to baseline after treatment withdrawal, an assumption violated for interventions whose effects persist (learning, vaccination, infrastructure). Multiple baseline designs require at least three units, settings, or behaviours, and assume no spillover between tiers. External validity depends entirely on replication; a single SCED demonstrates an effect for one unit and cannot, on its own, support generalisation. Development evaluation has adopted SCEDs sparingly to date, in part because many development outcomes are not repeatedly measurable in the short cycles that SCED phases require.
SCEDs have faced persistent scepticism, particularly from fields that treat group-randomised trials as the default standard of evidence. Three criticisms recur. First, findings from a single case cannot inform policy because they lack external validity. Second, the reversibility assumption restricts SCEDs to a narrow class of outcomes that return to baseline when the treatment is withdrawn, excluding most developmental, educational, or structural outcomes. Third, evidence hierarchies such as GRADE and the Campbell Collaboration have historically placed single-case evidence below group-level experimental evidence, limiting its influence on systematic reviews and policy guidance.
Proponents counter each objection. On external validity, Barlow et al. (2009) and Tanious et al. (2024) argue that SCEDs build generalisability through systematic replication across cases, settings, and behaviours rather than through sampling. A single SCED cannot generalise, but a series of replicated SCEDs can, and the replication logic is transparent: each new case either confirms or disconfirms the finding. On reversibility, multiple baseline and alternating treatments designs do not require the outcome to reverse, and the family as a whole is broader than the reversal design alone. On evidence standards, Kratochwill et al. (2013) established formal design standards for SCEDs within the What Works Clearinghouse, and meta-analytic techniques now allow SCED effect sizes to be synthesised alongside group-design effect sizes. Wiley & Rapp (2019) demonstrate that LMEM provides the same inferential outputs (effect sizes, confidence intervals, p-values) that reviewers expect from group trials. The debate is not settled, but the methodological infrastructure for treating SCED evidence on equal terms with group evidence has advanced considerably since the early critiques.
The counterfactual logic at small n trades breadth for depth. Within-unit designs produce strong internal validity for individual cases at the cost of generalisability. The integration strategies described above, and the multi-logic combinations discussed in Chapter 15, are how evaluators address this trade-off. The next chapter turns to a different causal logic, one that does not ask “what would have happened?” but rather “which combinations of conditions produced this outcome?”