22  Small-n Evaluation as Causal Inquiry

Small-n evaluation begins with a constraint, not with a method. There may be one national reform, a small number of service sites, no credible comparison unit, too few observations before implementation, or too little diversity across cases. The constraint determines which causal questions can be examined and what evidence is needed to answer them. It does not make every non-experimental account causal, and it does not make a qualitative method the default.

This book has argued that the first task is to specify the claim. An effect claim asks whether an outcome would have differed under another intervention state. A configurational claim asks which conditions or combinations are necessary or sufficient within a defined set of cases. A mechanism claim asks how an intervention contributed to an outcome in a particular context. These claims can concern the same programme, but their evidence and limits differ. A single evaluation may address more than one of them, provided that it does not move from one claim to another without making the additional inferential work visible.

The distinction matters because small-n evaluations are often asked to carry more than their evidence permits. A change in an outcome after implementation is not, on its own, evidence of an effect. A coherent account of implementation does not establish an effect in the absence of a comparison. A set-theoretic association does not show what would happen if one condition were changed. A stakeholder account can establish that an outcome was experienced and recognised without establishing its population prevalence or attribution. A simulated result remains conditional on the model. These are not defects of the methods. They are boundaries on the conclusions that each can support.

22.1 What the framework establishes

The framework in Chapter 4 separates forms of causal claim from inferential operations and from ways of producing evidence. Regularity and manipulation can support several kinds of inquiry; they are not additional types of causal claim. Diagnostic updating assesses the probative value of evidence for competing hypotheses. Participation affects who defines outcomes, contributes evidence, and interprets findings. Simulation makes assumptions about processes explicit and explores their implications. The six method families in Part II draw on these elements in different ways: counterfactual designs, configurational approaches, generative approaches, diagnostic approaches, participatory approaches, and computational simulation.

This organisation is intended to prevent two recurrent errors. The first is choosing a method from a label such as qualitative, theory-based, or small-n before identifying the question and evidence. The second is treating different methods as interchangeable routes to a single generic conclusion. Method choice should start with the intervention or causal condition, outcome, unit, period, available variation, and plausible alternative explanations. Chapter 14 translates those questions into a decision framework. It does not prescribe a method from a numerical threshold, because the relevant constraint differs across designs.

No method family has general priority. A designed within-unit comparison can provide evidence about an effect when outcome measurement, phase structure, and threats to the comparison are adequately addressed. Configurational analysis can identify causal packages and deviant cases when condition definitions, calibration, case selection, and limited diversity have been examined. Process tracing, contribution analysis, and realist evaluation can assess proposed processes when their accounts are tested against alternatives and the evidence is not treated as independent merely because it comes from several documents. Diagnostic reasoning can clarify how observations change confidence in competing explanations. Participatory approaches can identify outcomes and evidence that would otherwise be absent, but agreement among participants is not causal validation. Simulation can investigate feedback and scenarios that cannot be observed directly, but it cannot compensate for missing evidence about a real intervention.

These boundaries also explain why mixed-method designs require more than a list of components. Chapter 15 treats integration as an inferential task. The role of each component should be specified before analysis: a within-unit contrast may address the outcome pattern; process evidence may test the proposed mechanism; a configurational analysis may identify the conditions under which implementation differed; participation may challenge the outcome definition; and simulation may examine a dynamic hypothesis. Agreement between components is more informative when they draw on distinct evidence and address distinct threats. Apparent agreement based on the same records, informants, or assumptions may add little.

The quality standards in Chapter 16 follow from the same argument. Transparency about the claim, case universe, source provenance, assumptions, excluded alternatives, and analytic decisions is not an administrative addition to causal inference. It permits a reader to determine what the evidence supports. The relevant quality check depends on the claim. A counterfactual design needs scrutiny of the comparison and outcome series. A configurational analysis needs scrutiny of calibration, consistency, coverage, and case knowledge. A mechanism account needs scrutiny of tests, sequencing, source dependence, and alternative explanations. Participatory work needs scrutiny of representation, substantiation, and power. Simulation needs scrutiny of the model’s structure, inputs, calibration, validation, and uncertainty.

22.2 From evaluation design to evidence use

The later chapters extend these considerations beyond analysis. Commissioners shape the eventual warrant when they define the evaluation question, allow time for design and data collection, specify access to records, and decide whether the evaluation must inform an immediate decision. A request for an effect estimate cannot be met by adding causal language to a descriptive monitoring exercise after the programme has begun. Where the available design cannot support the requested claim, the terms of reference should state the narrower question that can be answered (Chapter 18).

Translation also requires claim discipline. Policy audiences may need a short, decisive account, but brevity should not remove the intervention, outcome, population, conditions, or uncertainty that define the finding. A brief can say what was observed, what causal conclusion is warranted, what remains uncertain, and what decision follows. It should not turn a conditional configuration or a mechanism proposition into a general estimate of impact. This is particularly important when an evaluation is used to justify continuation, expansion, or withdrawal of a public programme.

Constraints do not remove these obligations. Conflict, displacement, political sensitivity, weak administrative records, and disrupted fieldwork can limit the claims that a design can sustain. They may also alter the ethical and practical costs of data collection. The appropriate response is to revise the claim and design openly, document what has changed, and distinguish missing evidence from evidence of no effect. Chapter 20 shows that a smaller or adapted design may still produce useful evidence, but only if its causal warrant is stated without inflation.

Commissioning, translation, and adaptation under constraint are also where the book engages most directly with the EIPM pathways introduced in Chapter 17: capabilities, structures and processes, and evidence culture. A clearer causal vocabulary and a more explicit terms of reference may change what an organisation asks for and how it appraises what it receives. Whether an organisation adopts that vocabulary, and whether adoption changes a policy decision, are separate questions that this book does not resolve; the framework identifies where such a change would have to occur, not that it will.

Chapter 21 adds a final implication. Evidence synthesis is not the mechanical accumulation of publications under a method label. Studies should be combined only after the reviewer has assessed whether their causal claims, units, outcomes, conditions, and evidential warrants are comparable. The result may be a pooled estimate, a pattern of replications, a refined mechanism proposition, a bounded configuration, a structured account of participant-identified outcomes, or a decision that the evidence should not be combined. Bibliometric mapping can help locate candidate studies and identify ambiguous terminology. It cannot establish that a publication contains an evaluative causal claim or that its result is suitable for synthesis.

22.3 A working discipline

The framework is intended for decisions made before, during, and after an evaluation. Before fieldwork, it asks teams to specify the intervention or causal condition, outcome, population, period, and claim at issue. They then need to identify what variation, comparison, process evidence, and sources of challenge are available. A design should be narrowed or changed where those conditions cannot support the requested claim. This is a substantive decision for commissioners and programme managers, not a technical adjustment to be made after data collection.

During the evaluation, the framework requires a record of analytical choices and changes to the design. New evidence can require a revised programme theory, an amended measure, a different case set, or the abandonment of an infeasible comparison. These changes need not invalidate the work. Their timing, rationale, and consequences should be visible, so that readers can distinguish a response to field conditions from a decision made after an apparent result.

At reporting, the same discipline separates what was observed from what was inferred. An evaluation can be useful when it cannot estimate an effect: it may document implementation, identify a mechanism that requires further testing, establish that outcomes differed across cases, or show that the available records cannot distinguish competing explanations. A qualified conclusion can inform an adaptation, a safeguard, a further data-collection decision, or a later evaluation design. It should not be presented as evidence for scale-up or withdrawal when the claim required for that decision has not been established.

22.4 What the framework does not provide

The framework does not remove substantive judgement from evaluation. It cannot determine whether a programme objective is desirable, whether the right outcome has been selected, or what level of uncertainty a commissioner should accept. It does not supply a comparison where the programme’s timing, data, or ethics make one unavailable. It does not turn several dependent accounts into independent evidence, or make findings from one setting transportable by calling them generalisable.

Nor does the framework prescribe a uniform sequence. Some evaluations begin with a prospective design and a clear decision point. Others begin after implementation, with incomplete records and a disputed account of what occurred. The questions in this book help teams identify what each situation permits. They may lead to a more demanding design, a narrower claim, a further phase of evidence collection, or a decision that causal attribution cannot be established. That is a result of the evaluation, not a failure to apply the framework.

22.5 Questions that remain

Several methodological and empirical questions remain open. First, evaluation needs clearer approaches to meta-inference across different forms of causal claim. Sequential and nested designs can specify how evidence moves between components, but they do not by themselves determine how a counterfactual result, a mechanism test, and a configurational finding should be weighed when they converge or conflict. This requires more than a generic appeal to triangulation.

Second, there is a need for empirical work on the practice of small-n evaluation. The bibliometric appendix maps a broad literature and the validated subset identifies work that appears relevant to evaluation. It does not establish how consistently evaluations specify their causal claims, document calibration choices, test alternative explanations, address source dependence, or substantiate participant-defined outcomes. Claim-level reviews, using the approach set out in Chapter 21, would provide a firmer basis for describing the field’s strengths and weaknesses.

Third, commissioning and translation require investigation in their own right. The chapters on commissioning and translation set out design principles, but they do not show how frequently commissioners specify causal questions in terms of reference, how evaluators negotiate claims that cannot be supported by the available design, or which forms of communication help decision-makers use qualified findings appropriately. These are empirical questions about evaluation practice, not matters that can be settled by methodological guidance alone.

Finally, the methods need to remain connected to the substantive settings in which they are used. Small-n evaluation is common in international development, but it also arises in education, health, environmental policy, public administration, and organisational change. The relevant causal question is not determined by the sector. It is determined by the intervention, outcome, units, evidence, and decision at hand. Cross-sector applications can therefore test the limits of the framework without treating one sector’s terminology or evidence conventions as universal.

The purpose of this book is modest. It does not offer a substitute for a credible comparison, a complete account of a mechanism, or a common metric for unlike evidence. It offers a way to decide what causal question is being asked, what kind of evidence can answer it, and what the resulting conclusion can support. In small-n settings, that discipline is often the difference between an evaluation that records a plausible story and one that provides a defensible causal account.