2 Introduction
Some interventions provide only one or a few independent units for evaluating effects. National policy changes and capacity-building programmes in single organisations are common examples. White & Phillips (2012) describe these as small-n settings when there are too few units of assignment to test differences between treatment and comparison groups. The same difficulty can arise with several treated units if no comparison units represent a credible alternative to the intervention.
The number of respondents does not resolve this problem. A programme assigned across four districts may survey thousands of people, but the intervention still varies across four units. A national reform may generate many administrative records while providing no untreated version of the same country. Repeated observations, staggered implementation, or other forms of evidence may create useful comparisons, but their contribution depends on the causal claim and the design.
This book concerns causal inquiry under those constraints. It examines evaluations that seek to establish whether an intervention made a difference, which combinations of conditions contributed to an outcome, or how a causal process operated in a particular setting. Chapter 3 defines small n through the inferential constraint that is binding, including the number of independent intervention units, support for a counterfactual comparison, repeated temporal observations, and diversity across cases. No fixed sample-size threshold applies across these designs.
Small n does not justify a lower standard of evidence. It requires the evaluator to state what is being inferred, identify the variation or observations bearing on that claim, and explain the assumptions connecting the evidence to the conclusion. In some settings, the available evidence will not support an effect estimate. The evaluation may still examine implementation or a proposed mechanism, but those findings should not be reported as an answer to the unobserved effect question.
The appropriate response to this limit is not to abandon causal inquiry or to attach causal language to evidence that cannot bear it. It is to distinguish the questions that remain open from those the evaluation can address. An implementation record can establish whether activities occurred, who received them, and where delivery differed. Process evidence can examine whether the events expected under a proposed explanation took place and whether rival explanations fit the record better. Cross-case comparison can examine whether outcomes appear under particular configurations of conditions. These findings may be useful for management and policy. They do not become an estimate of an effect unless the evaluation also provides a defensible contrast with the relevant alternative state.
This distinction has practical consequences from the first discussion of an evaluation. A commissioner may ask whether a programme should be extended, while the available records allow only an account of implementation. An evaluation team may be able to introduce repeated outcome measurement but not randomise delivery. A programme may have enough variation across sites for a configurational analysis but not enough comparable units for a conventional regression. In each case, the design should state the claim it can support and the decision for which that claim will be used. The later chapters develop the relevant methods; the point here is that a method cannot repair an unspecified or infeasible causal question.
2.1 Impact evaluation and causal claims
The term impact evaluation is used in this book for evaluations that make causal claims about an intervention and its outcomes. These claims include attribution, where the object is an effect associated with a specified intervention contrast, and causal contribution, where the object is the intervention’s role within a configuration or process. Description of outcomes, implementation, or participant experience may be necessary to an impact evaluation, but description alone does not establish causation.
These two claims, together with the evidence and inferential operations that support them and the roles of participation and modelling, are organised in this book into a single framework, developed fully in Chapter 4.
This usage is narrower than the OECD-DAC impact criterion, which asks what difference an intervention makes and includes positive and negative, intended and unintended, higher-level effects (OECD-DAC Network on Development Evaluation, 2019). The OECD-DAC criteria are intended to be selected and applied in relation to an evaluation’s purpose, users, and context. An evaluation may therefore examine impact without estimating a treatment effect, but any causal conclusion still requires evidence appropriate to the claim.
This narrower usage also has a specific history. The OECD-DAC’s impact criterion, part of its evaluation framework since the 1991 Principles for Evaluation of Development Assistance, was never tied to a particular design; it asked whether an intervention produced higher-level effects, not how that determination should be reached. The equation of impact evaluation with counterfactual, and increasingly experimental, designs took shape later, in the mid-2000s. The Evaluation Gap Working Group, convened by the Center for Global Development, argued that a chronic shortage of rigorous counterfactual evidence was limiting what could be learned about development effectiveness, and called for an independent body to fund and synthesise such evidence (The Evaluation Gap Working Group, 2006); that recommendation informed the case for 3ie, founded in 2008. Around the same period, randomised evaluations conducted through the Abdul Latif Jameel Poverty Action Lab and Innovations for Poverty Action established randomisation as the reference design against which quasi-experimental alternatives were increasingly judged. White & Phillips (2012) and Department for International Development (2012) belong to a subsequent correction against that narrowing, arguing for a wider range of designs and methods under the impact evaluation label, while Deaton & Cartwright (2018) questions the definitional authority that randomisation had by then acquired. The usage adopted in this book sits inside that unsettled range: narrower than the OECD-DAC criterion, because it requires a causal claim, and broader than the usage associated with J-PAL and IPA, because it treats causal contribution, not only counterfactual attribution, as a legitimate route to that claim.
At small n, the inferential problem differs by design rather than admitting a single description. Few independently assigned clusters limit precision and narrow the range of feasible design-based tests, while weak overlap with untreated units undermines the credibility of whatever comparison those clusters could support. Short or unstable outcome series pose a different constraint, limiting what a temporal design can establish rather than what a cross-sectional comparison can, and configurational analysis has its own failure mode, becoming underdetermined when the cases at hand display too little diversity relative to the conditions under examination. None of this points to one alternative method; each constraint identifies a specific piece of information the evaluation lacks.
The distinction between attribution and contribution should therefore not be used to lower the standard for either claim. An attribution claim needs a contrast that represents what would have happened without, or under a different form of, the intervention. A contribution claim needs an account of how the intervention entered a process or configuration, evidence on the expected links, and serious attention to other plausible causes. The two claims may be pursued in the same evaluation, but they should retain their own warrants. Calling a mechanism account an attribution study does not create a counterfactual; reporting an effect estimate does not explain the process that generated it.
The availability of suitable methods affects whether interventions generate evidence that can later be appraised or used. Chapter 17 places this issue within an evidence-informed policymaking framework. The connection is limited but direct: commissioners and evaluation teams need the capability to identify feasible causal claims and designs before evidence can enter wider processes of interpretation and decision-making. The framework does not establish that adopting a particular vocabulary or method will increase policy use.
2.2 Existing guidance
Development evaluation already has substantial guidance on causal inference beyond conventional experiments. The NONIE guidance reviewed experimental, quasi-experimental, and non-experimental approaches without treating one design as suitable for every question (Leeuw & Vaessen, 2009). The DFID report Broadening the Range of Designs and Methods for Impact Evaluations distinguished experimental, statistical, theory-based, case-based, and participatory approaches and argued for designs fitted to the intervention and evaluation purpose (Department for International Development, 2012). The World Bank Independent Evaluation Group later provided a broader overview of approaches used to evaluate international development interventions (Vaessen et al., 2020).
Two guides address the small-n problem more directly. White & Phillips (2012) examine attribution when units of assignment are too few for conventional treatment–comparison tests. Their proposed common core centres on programme theory, alternative causal hypotheses, and systematic evidence about the links in the causal chain. The TASO guidance covers small cohorts in higher education and profiles a range of counterfactual, theory-based, case-based, participatory, and modelling approaches (Fox et al., 2022). It also makes clear that question-to-method tables require judgement and should not be read as categorical rules.
These guides establish that small-n evaluation is not synonymous with qualitative research and that causal claims can draw on several forms of evidence. They also show why a flat list of methods is insufficient. The approaches differ in the claim they address and in the work performed by comparison, intervention, process evidence, stakeholder knowledge, or modelling. A classification that places these elements at one conceptual level makes method selection appear simpler than it is.
The remaining problem is to connect the causal claim, available evidence, and inferential operation without assuming that each evaluation question belongs to one method family. A mechanism study and an effect study can examine the same intervention without producing interchangeable conclusions. Diagnostic updating can assess evidence about several kinds of claim. Participation can shape outcomes and evidence across designs, while simulation produces results conditional on a model. Chapter 4 develops these distinctions.
The book builds on this guidance rather than replacing it. Experimental, quasi-experimental, theory-based, case-based, participatory, and modelling approaches remain useful descriptions of bodies of practice. They do not, however, answer the question of what a particular piece of evidence is expected to establish. The same label can cover designs with different units, contrasts, assumptions, and claim boundaries. The framework used here retains the methods’ disciplinary histories while making the causal work of each design explicit.
2.3 From an evaluation question to an evidential warrant
An evaluation question becomes a causal question only when it specifies the intervention or condition, the outcome, the unit, and the relevant period. “Did the programme work?” is not yet a question that a design can answer. It leaves open what changed, for whom, compared with what, and when. The same uncertainty arises when an evaluation asks how a programme worked but does not state the proposed process, the evidence that would support it, or the alternatives to be considered.
The framework asks evaluators to make these choices explicit before selecting a method. For an effect claim, they must identify a credible alternative treatment state and the design features that make the observed contrast informative. For a configurational claim, they must define the cases, conditions, outcome, and the basis for calibration. For a mechanism claim, they must identify the proposed process, observable implications, sources of evidence, and plausible rivals. Diagnostic, participatory, and simulation components require equally specific accounts of their role. The resulting design may be simple, but its causal warrant should be visible.
This is also a way to decide when not to make a causal claim. Some programmes have no usable comparison, no process evidence that can discriminate among explanations, and too little variation for a cross-case analysis. The evaluation can still describe delivery, document outcomes, identify concerns, or improve measurement for later inquiry. It should not report those activities as causal evidence. A defensible limitation is more useful to a commissioner than a claim that cannot survive scrutiny.
2.4 Using the framework
The framework is intended for design, appraisal, and communication. At the design stage, it helps a team decide whether the available units, records, timing, and access to informants can support the intended claim. This may lead to a change in the evaluation question, additional measurement, a different sequence of methods, or a decision to postpone a causal study. These are design decisions, not failures of ambition.
During analysis, the framework provides a way to keep the conclusion aligned with the evidence. It asks whether a finding is an outcome description, an effect estimate, a configuration, a mechanism proposition, a diagnostic update, a participant-defined account, or a model result. Several may appear in the same evaluation. The report should state the role of each and avoid presenting their agreement as independent corroboration when they rely on the same sources or assumptions.
For commissioners and evidence users, the framework provides questions for appraisal. What causal claim was specified? What alternative explanation or intervention state was considered? Which observations bear on the claim? What was excluded or remained unobserved? What decision can the conclusion inform? These questions apply whether the evidence is numerical, documentary, participatory, or model-based. They do not require a reader to reproduce a technical analysis, but they make it possible to see where the causal warrant begins and ends.
The framework is not a decision tree that allocates a method automatically. A small number of sites does not decide whether process tracing, QCA, a single-case experiment, or another design is appropriate. The choice depends on the causal question, the nature of the intervention, the available variation, the evidence that can be collected, and the main risks to the conclusion. The decision framework in Chapter 14 develops this judgement in more detail.
2.5 Contribution and scope
The book makes three contributions to this body of guidance. First, it organises causal reasoning around three forms of claim: effects or difference-making, causal configurations, and mechanisms or causal processes. It then distinguishes those claims from the inferential operations used to assess them and from the roles of participation and computational modelling. The purpose is not to introduce another list of methods, but to state what each component contributes to a causal argument.
Second, Part II examines six established method families through eight chapters. Chapter 6 covers single-case experimental designs, N-of-1 trials, and randomisation inference. Chapter 7 addresses qualitative comparative analysis and coincidence analysis. The generative family is developed separately through process tracing (Chapter 8), contribution analysis, which uses theory of change as a precursor (Chapter 9), and realist evaluation (Chapter 10). Chapter 11 considers Bayesian and other diagnostic approaches to evidence. Chapter 12 discusses most significant change and outcome harvesting as approaches to producing and interpreting evidence. Chapter 13 treats agent-based modelling and system dynamics as model-based tools that can support several causal claims.
Third, the book develops practical guidance for choosing, combining, and reporting methods. Method selection begins with the claim, unit of intervention, available variation, evidence quality, and main threats to inference. Integration requires each component to have a defined role and does not become credible merely because several methods agree. The later chapters extend these principles to commissioning, communication, and evaluation in constrained settings.
The book does not propose that small-n methods substitute for large-n designs whenever sample size is limited. A randomised or quasi-experimental design may remain feasible with few units, and a large-n study can investigate mechanisms, heterogeneity, or contextual variation. Nor are the six method families interchangeable or equal for the same claim. Their credibility depends on design conditions, measurement, execution, evidence provenance, and the assumptions needed for inference.
The framework is not a sequence in which every evaluation must progress from a counterfactual estimate to a mechanism account and then to a configuration. Programmes and decisions differ. An evaluator may begin with a process trace because the question concerns a contested implementation account. A repeated outcome series may be feasible while case diversity is not. A participatory process may be needed to define outcomes before any comparison is meaningful. The task is to justify the combination and order of methods used, not to reproduce all of them.
The book is grounded in development evaluation, where programmes often operate through institutions, places, or bounded populations that cannot be multiplied for research purposes. These are also common conditions in domestic public policy, health, education, and organisational change. The causal principles and methods are not specific to development. Examples are included only where the cited evaluation documents the claimed use. The book does not use constructed cases or invented numerical illustrations.
2.6 Boundaries of the handbook
The book does not provide a universal rule for deciding whether an evaluation has a sufficiently small n. The relevant count is the one that constrains the claim: independent intervention units for an effect estimate, repeated observations for a temporal design, or diverse cases for a configurational analysis. Chapter 3 develops these distinctions. A study with many respondents may still face a small-n causal problem, while a study with few respondents may not.
Nor does the book treat methods as self-contained packages that can be applied once their terminology is learned. Good small-n evaluation depends on substantive knowledge of the programme, credible measurement, ethical access to people and records, and judgement about plausible alternatives. Statistical software, QCA software, evidence-test templates, participatory protocols, and simulation platforms can structure an analysis. None determines whether the intervention, outcome, cases, or sources have been defined appropriately.
The handbook therefore complements, but does not replace, detailed methodological guidance, domain expertise, and local knowledge. Its contribution is a common causal vocabulary for moving between them. It helps a team state whether it needs a contrast, a set relation, evidence about a process, an assessment of competing hypotheses, a participant-defined account, or a model-based scenario. It also gives reviewers and users a basis for asking whether the evidence meets the standard required by that choice.
Chapter 5 maps how publications mentioning selected method terms are distributed across disciplines, countries, and time. The search was used to map a literature and to identify records requiring full-text assessment. It did not establish method adoption, integration, evaluation quality, or a publication’s suitability as evidence for a causal claim. Its counts are therefore not used here to demonstrate the need for the framework or to estimate the prevalence of valid applications. Chapter 20 explains the further claim-level assessment required when literature is used for evidence synthesis.
2.7 Structure of the book
Part I establishes the foundations. Chapter 3 explains what may be small in an evaluation and why the relevant count depends on the design. Chapter 4 then separates forms of causal claim, inferential operations, and approaches to producing evidence. Chapter 5 maps how the corresponding method terms appear in the indexed literature, as a point of reference for the disciplinary and geographic detail reported in each method chapter.
Part II covers the six method families introduced above. Part III turns to method selection, integration, and quality standards. Part IV situates the methods within the policy system: Chapter 17 describes the evidence-informed policymaking context in which evaluation evidence is produced and used, and the chapters that follow examine commissioning, communication with policy audiences, and adaptations in fragile, conflict-affected, or otherwise constrained settings. Part V considers how small-n findings can be synthesised without treating unlike claims and warrants as interchangeable (Chapter 21), before Chapter 22 draws together the implications and identifies questions for further methodological and empirical work.