3 Defining Small-n
There is no sample-size threshold at which an evaluation becomes small n. Ten observations may be adequate for one design and unusable for another. The relevant question is whether the available units, comparisons, or repeated observations can support the intended causal claim.
White & Phillips (2012) define small-n impact evaluation more narrowly as a setting with too few units of assignment to test differences between treatment and comparison groups. That definition captures a common problem in development evaluation, including national policy changes and capacity-building interventions in single organisations. It does not cover every constraint discussed in this book. An evaluation may have many observed individuals but few independently assigned districts, one treated unit but a substantial donor pool, or several cases that represent too few causal configurations for the proposed analysis.
This chapter uses small n to describe a binding inferential constraint, not a fixed number. The constraint must be stated in relation to the causal claim, the unit at which the intervention varies, and the evidence used for inference. This definition also separates sample size from evidence quality. One case may generate extensive records and observations, while a study with many participants may still have little independent variation relevant to the effect of the programme.
3.1 What does n count?
The unit counted by n follows from the claim and design. If a programme is assigned to districts, the number of surveyed residents does not increase the number of independently assigned units. The resident-level data may improve outcome measurement, but uncertainty about the district-level treatment effect still depends heavily on the number of districts and the assignment process. Treating participants as independent units would overstate the information supplied by the design.
The evaluation of biometric Smartcards in Andhra Pradesh illustrates the distinction. The programme operated across eight districts and affected approximately 19 million people, but rollout was randomised across 157 subdistricts (Muralidharan et al., 2016). Calling this an eight-unit or a 19-million-unit evaluation would misdescribe the experiment. For its main effect estimates, the relevant units were the subdistricts used in the randomised rollout.
The same issue arises when a programme reaches a bounded population. Observing all fifteen members of a training cohort removes sampling uncertainty about the measured outcomes of those participants. It does not create a comparison showing what would have happened without the training, nor does it establish that findings apply to later cohorts. Population coverage, causal identification, and generalisation are separate questions.
Several counts may therefore matter in one evaluation. These include the number of independently assigned or exposed units, the number of credible comparison units, repeated observations within units, and the number of cases representing different combinations of conditions. Reporting only the number of respondents conceals the constraint that governs inference.
3.2 Four binding constraints
Four constraints recur across small-n evaluations, and each limits a different form of causal claim. Few independent intervention units limit an effect estimate based on between-unit comparison. Weak support for a counterfactual comparison limits what any comparison, however constructed, can establish about an untreated alternative. Few or weakly informative repeated observations limit what a temporal design can support. Limited diversity across cases limits what a configurational analysis can identify. The constraints are described separately below and summarised together in Table 3.1.
3.2.1 Few independent intervention units
Some programmes vary across only one or a few units. A policy may be introduced nationally, or an institutional reform may affect one ministry. A clustered intervention may include many participants but only a few districts at the level where treatment varies. In these settings, conventional between-group estimation may be imprecise or impossible because there is little independent variation in exposure.
The number alone does not determine whether an effect can be estimated. Randomised timing, repeated introduction and withdrawal, or other design features may create informative contrasts within or across units. Their credibility depends on the assignment process, treatment stability, outcome measurement, spillovers, and the plausibility of the comparison.
3.2.2 Weak support for a counterfactual comparison
An evaluation can have many records and still lack a credible comparator. Untreated units may differ substantially from the treated unit, observations near an eligibility threshold may be sparse, or pre-intervention trends may not support a difference-in-differences design. The binding problem is then lack of overlap or relevant variation, not the raw sample count.
White & Phillips (2012) describe an evaluation of the Bangladesh Integrated Nutrition Project in which couples received nutritional counselling in only one of six project subdistricts. The effect of counselling couples could not be separated statistically from other features of that subdistrict. More observations within the same area would not resolve that confounding. The missing information concerned comparable variation in the form of counselling.
3.2.3 Few or weakly informative repeated observations
Within-unit designs require information over time. Three annual measurements and fifty daily measurements are not interchangeable, and the larger count is not necessarily better. Serial dependence, seasonality, outcome instability, changes in measurement, and the timing of programme introduction determine how much information the series contains.
Interrupted time-series analysis can examine one treated unit when a sufficiently stable pre-intervention series supports estimation of the expected trend (Bernal et al., 2017). No universal number of time points establishes that stability. Single-case experimental designs use repeated observations differently: controlled or randomised changes in treatment timing can provide design-based evidence even with one unit, provided the outcome is measured often enough across suitable phases (see Chapter 6).
3.2.4 Limited diversity across cases
Configurational methods require cases that display enough variation in outcomes and conditions to assess claims of necessity, sufficiency, or multiple causal pathways. The problem is not simply that the case count is low. Twenty nearly identical cases may contain less information about alternative configurations than ten deliberately selected cases with relevant variation.
The number of conditions matters alongside the number and diversity of cases. Adding conditions increases the number of logically possible configurations and can leave much of the truth table unobserved. Case selection, calibration, contradictory configurations, and limited diversity therefore determine what a configurational analysis can support (see Chapter 7).
| Binding constraint | What is limited | Consequence for inference | Information that should be reported |
|---|---|---|---|
| Few intervention units | Independent variation in assignment or exposure | Between-group effect estimates may be unavailable or imprecise | Unit of assignment; number of independent units; assignment process; spillovers |
| Weak counterfactual support | Comparable untreated observations or overlap | The missing outcome under another intervention state cannot be estimated credibly | Comparator selection; baseline differences; overlap; relevant concurrent changes |
| Few informative time points | Stable observations before and after intervention changes | Temporal patterns may not distinguish programme effects from trend or instability | Observation frequency; phase length; serial dependence; missingness; measurement changes |
| Limited case diversity | Observed combinations of conditions and outcomes | Necessity and sufficiency claims may be underdetermined | Case selection; number and calibration of conditions; observed configurations; contradictions |
3.3 Constraints can coexist
The four constraints are not mutually exclusive. A national reform may have one treated unit, no comparable untreated unit, a short and disrupted outcome series, and extensive process records. In that setting, the first two constraints limit an effect estimate, the third limits a temporal design, and the records may still be relevant to a mechanism inquiry. The evaluation should describe each condition separately. Calling the entire study small n does not say which conclusion is feasible.
The same point applies to a multi-site programme. Several sites may provide enough units for a limited comparison but too little variation in implementation to examine causal configurations. Repeated measurements may make changes over time visible while shared policy changes make it difficult to attribute those changes to the programme. An evaluation can use the information that is available, but it should not treat information relevant to one claim as a substitute for information required by another.
Constraints can also change during an evaluation. A comparison site may adopt a similar intervention, a planned outcome measure may become unavailable, a programme may be adapted differently across sites, or a data collection interruption may remove observations around the point of implementation. These events do not necessarily invalidate the evaluation. They require the team to revisit the causal claim, record the change, and explain how it affects the evidence. A design protocol that identifies the unit of inference, planned comparison, measurement schedule, and case universe makes this reassessment possible.
The interaction among constraints is one reason why simple thresholds are unhelpful. The same number of sites may be sufficient for an implementation study, inadequate for a cluster-level effect estimate, and useful for a deliberately bounded configurational analysis. The question is not whether a count has crossed a universal minimum. It is whether the design provides the variation, comparison, observations, or case diversity required for the stated claim.
3.4 Statistical consequences
Smaller samples commonly reduce precision and statistical power, but the consequences depend on the estimator and design. A null result from an underpowered study may be compatible with effects of policy relevance. Estimates can also be unstable because one observation or cluster has substantial influence. These problems should be shown through uncertainty intervals, design-specific power or sensitivity analysis, and examination of influential observations where appropriate.
Small samples do not make conventional p-values meaningless by definition. Exact randomisation inference may remain valid with few units when the assignment mechanism is known, although the set of possible assignments can limit attainable significance levels and power. Model-based analyses can also be used, but their results may depend strongly on distributional or functional-form assumptions that the data provide little scope to assess.
Three problems should not be collapsed into sample size. Identification concerns whether the design isolates the causal quantity of interest. Precision concerns how uncertain the estimate is. Generalisability concerns the cases, settings, or populations to which the result applies. A large dataset does not guarantee identification, and a carefully controlled small-n estimate may be precise for one unit while offering little evidence about other settings.
Small n also does not imply that the evidence is qualitative. Single-case experiments, randomisation inference, set-theoretic analysis, and formal simulation can all use quantitative data. Conversely, qualitative evidence may be collected in a large-n evaluation. The useful distinction is the role that each observation plays in the causal argument, not whether the dataset is labelled quantitative or qualitative.
3.5 Planning under a binding constraint
The relevant constraint should be identified before data collection whenever possible. A team planning an effect estimate needs to know the unit at which the intervention can vary, how many independent treatment changes or comparison units are feasible, and whether outcome measurement can detect the expected timing of change. A team planning a mechanism study needs to identify the proposed process, the records and observations that could distinguish it from alternatives, and the limits of access to those sources. A team planning a configurational analysis needs to define the case universe and determine whether the cases contain the required outcome and condition diversity.
These decisions also shape resource allocation. More respondent interviews may improve knowledge of implementation or experience, but they do not increase the number of independently assigned districts. More frequent measurement may strengthen a temporal design only if the outcome is measured consistently and the programme’s timing is known. Adding conditions to a configurational analysis can make a theory more detailed while leaving too few observed configurations to assess it. The value of additional data depends on the inferential problem it addresses.
Evaluation proposals should therefore state both the intended claim and the constraint that may limit it. This gives commissioners an opportunity to change the design while it is still possible: introduce a comparison, phase delivery, preserve a baseline, standardise an outcome measure, document implementation variation, or fund access to records. When these changes are not feasible, the proposal should state the narrower claim that will be pursued and what evidence would be needed for a stronger conclusion. This is more useful than promising impact evidence and redefining the claim after the data have been collected.
The same discipline applies at appraisal. A reviewer should ask whether the stated n corresponds to the unit of intervention and the claim, whether the design contains the comparison or observations it requires, and whether the conclusion extends beyond the cases and period examined. The reporting fields in Table 3.3 make these questions visible. They are not a checklist that certifies causal inference; they are the minimum information needed to assess it.
3.6 Data-rich boundary cases
Some methods study one treated unit while relying on substantial information elsewhere. They address a small number of treated units without necessarily operating under a generally data-poor design. The distinction prevents the unit of policy interest from being confused with the amount of evidence used for inference.
Synthetic control methods estimate a counterfactual for one or a few treated units from a weighted combination of untreated units (Abadie, 2021). Feasibility depends on the relevance of the donor pool, pre-intervention fit, the timing of treatment, and the absence of contamination. No fixed number of donor units is sufficient. The Basque Country study, for example, used other Spanish regions to construct a counterfactual trajectory for one treated region (Abadie & Gardeazabal, 2003).
Interrupted time-series designs may also examine a single unit, but they rely on repeated temporal observations and assumptions about the continuation of the pre-intervention process. Single-case experimental designs use repeated measurement with controlled treatment phases, while process tracing may examine one case using documents, interviews, and observed sequences. These designs share a small number of cases but draw their causal warrant from different sources.
| Design | Dimension that is small | Information used for inference | Implication for scope |
|---|---|---|---|
| Single-case experimental design | One or a few intervention units | Repeated outcomes and controlled or randomised treatment phases | Within the book’s core scope |
| Synthetic control | One or a few treated units | Untreated donor units and pre-intervention outcomes | Related, but often data-rich outside the treated unit |
| Interrupted time series | One treated unit | A sufficiently informative outcome series before and after intervention | Related; feasibility depends on temporal evidence |
| Configurational analysis | A modest set of cases | Diversity in calibrated conditions and outcomes | Within scope when the cases can assess the proposed set relations |
| Process tracing | One or a few cases | Observations bearing on a mechanism and rival explanations | Within scope; case count does not measure evidential depth |
3.7 Constraint, case selection, and intended use
Small n sometimes results from programme structure. The intervention already covers one ministry, four districts, or a bounded group of participants. The evaluator cannot add independent units after implementation and must decide which claims the available variation and evidence can support. The appropriate conclusion may be that an effect cannot be estimated credibly, even though the programme can still be described or its implementation examined.
In other studies, evaluators select a small number of cases because they need detailed evidence about a mechanism or want to compare particular causal configurations. This is a decision about case selection and evidence, not proof that large-n designs cannot address mechanisms or heterogeneous effects. A mixed design may use a larger sample to estimate variation in effects and select cases for closer investigation.
Evaluation questions do not determine methods on their own. Method choice also depends on the unit and timing of intervention, available comparisons, outcome measurement, case diversity, access to process evidence, and the standard of credibility required for the decision. Before selecting a method, an evaluation should report the elements in Table 3.3.
| Element | Question to answer |
|---|---|
| Causal claim | Is the evaluation estimating an effect, assessing a causal configuration, examining a mechanism, or combining these claims? |
| Unit of intervention | At what level is the programme assigned, introduced, or varied? |
| Independent variation | How many independent units or treatment changes bear on the claim? |
| Counterfactual support | Which observations represent the relevant alternative intervention state, and why are they comparable? |
| Temporal evidence | How many observations occur before and after intervention changes, at what interval, and with what missingness or dependence? |
| Case diversity | Which outcomes and configurations are observed, and which relevant combinations are absent? |
| Within-case evidence | Which records or observations bear on the proposed mechanism and its alternatives, and how independent are they? |
| Scope | Which units, period, population, and implementation form does the claim cover? |
The term small n is therefore incomplete unless the evaluator states what is being counted and why the count constrains the claim. Chapter 4 builds on this definition by separating claims about effects, configurations, and mechanisms from the inferential operations and evidence used to assess them.