flowchart BT
subgraph L1["Form of causal claim"]
EFF["Effect<br>(counterfactual)"]
CFG["Configuration<br>(configurational)"]
MEC["Mechanism<br>(generative)"]
end
subgraph L2["Inferential operation"]
REG["Regularity"]
MAN["Manipulation"]
DIA["Diagnostic updating"]
end
subgraph L3["Evidence production and formalisation"]
PAR["Participation"]
SIM["Simulation"]
end
L2 --> L1
L3 --> L1
L3 --> L2
4 Causal Reasoning in Small-n Evaluation
Identifying a binding constraint on n, as the previous chapter did, does not by itself tell an evaluator what kind of causal claim can still be made or which method fits it. That is a separate question, and answering it requires a vocabulary that does not yet exist in most guidance: one that distinguishes what is being claimed from how the claim is being supported. This chapter sets out that vocabulary. Chapter 3 identified four ways in which an evaluation can have a small n: the relevant constraint may concern independent intervention units, support for a counterfactual comparison, repeated observations over time, or diversity across cases. Those constraints affect the evidence available for a causal claim, but sample size does not determine the form of the claim or make one method appropriate by default.
Evaluation methods are often grouped by disciplinary tradition or by a qualitative–quantitative divide. Neither classification explains what the evidence is expected to establish. A single-case experiment, a process trace, and a participatory account may all examine one programme, but they make different uses of comparison, intervention, observation, and judgement. The framework used in this book therefore begins with the causal claim and the inferential work required to support it.
Three levels need to be kept separate. The first is the form of causal claim: an effect, a causal configuration, or a mechanism. The second is the inferential operation used to assess that claim: regularity, manipulation, or diagnostic updating. The third concerns how evidence and assumptions are produced, including participation and computational modelling. Comparison and process observation are design practices through which claims and operations are made concrete. Methods often combine elements from more than one level. Treating all of them as equivalent causal logics obscures those differences.
The book retains the term eight logics as a practical map of the reasoning that recurs in small-n evaluation. The map does not imply that all eight items have the same status. Three identify the claim at issue; three describe operations that can support a claim; and two concern the production and formalisation of evidence. Table Table 4.1 places them at their respective levels. The rest of the book uses this distinction to describe what a method contributes and what additional evidence it needs.
| Component | Level in the framework | Question or contribution | Main limit when used alone |
|---|---|---|---|
| Counterfactual reasoning | Effect claim | Would the outcome have differed under another intervention state? | Does not establish the causal process or transferability without further evidence |
| Configurational reasoning | Configuration claim | Which combinations of conditions are necessary or sufficient in the cases studied? | Does not establish the effect of changing one condition |
| Generative reasoning | Mechanism claim | Through what process did the intervention contribute to the outcome? | Does not estimate an average effect or the prevalence of the mechanism |
| Regularity | Inferential operation | Identifies repeated patterns in outcomes, events, or conditions | A stable pattern can still be due to confounding or a competing process |
| Manipulation | Inferential operation | Examines what changes when a putative cause is deliberately varied | May be infeasible, incomplete, or insufficient to address all threats |
| Diagnostic updating | Inferential operation | Assesses how observations change confidence in competing explanations | Depends on the hypotheses, evidence assessments, and source dependence |
| Participation | Evidence production and interpretation | Shapes outcomes, evidence, and interpretation through affected actors | Agreement does not itself validate a causal conclusion |
| Simulation | Formalisation of evidence and assumptions | Explores the implications of specified processes and scenarios | Results are conditional on model structure, inputs, and validation |
Figure 4.1 shows the same eight components arranged by level. Inferential operations supply the warrant for a causal claim; evidence production and formalisation, in turn, shape what the operations have to work with. The arrows indicate support, not sequence: an evaluation need not proceed level by level, and a single design often draws on more than one component within a level.
4.1 Four traditions of causal reasoning
The vocabulary set out above did not appear from nowhere. It draws on a long philosophical argument about what it means for one thing to cause another, and four traditions in that argument shape how small-n evaluation reasons today. Each begins from a different question, and each privileges a different kind of evidence. Setting them side by side clarifies why the methods in Part II make the claims they do.
The first tradition is eliminative. John Stuart Mill’s methods of agreement, difference, and concomitant variation treat causal inference as the removal of candidate explanations by systematic comparison across observed cases (Mill, 1843). A condition present whenever the outcome occurs, and absent whenever it does not, becomes a candidate cause, while rival conditions are eliminated as the comparison widens. Mill’s logic requires little formal apparatus and applies naturally to a small number of closely examined cases, which is why it underlies both configurational analysis and the elimination of rivals in process tracing. Its weakness is that genuine causes can be missed when several conditions move together or when the relevant comparison is unavailable.
The second tradition treats causes as parts of packages rather than as isolated factors. John Mackie’s INUS account defines a cause as an insufficient but necessary part of a condition that is itself unnecessary but sufficient for the outcome (Mackie, 1965). An INUS cause is one ingredient in a combination that produces the effect, where other combinations could produce it too. The account captures two features that recur in development programmes: interventions rarely work alone, and the same outcome can be reached by more than one route. It provides the philosophical basis for configurational methods, which look for combinations of conditions rather than the average effect of a single one, and for the language of contributory causes used throughout this book (Cartwright, 2007).
The third tradition defines a cause through comparison with what would otherwise have happened. The potential-outcomes framework associated with Rubin treats the effect of an intervention on a unit as the difference between the outcome under treatment and the outcome under an alternative state, only one of which is ever observed (Holland, 1986; Rubin, 1974). Causation is a contrast between factual and counterfactual states, and the central problem is how the units came to be in one state rather than another. The framework makes the assumptions behind an effect estimate explicit, and it governs the counterfactual designs of Chapter 6. It says little on its own about the process connecting cause and outcome.
The fourth tradition represents causal systems as graphs. Judea Pearl’s structural causal models express assumptions as a directed acyclic graph and define an intervention as a change to that graph, using the do-calculus to determine when an effect can be identified from the available data (Pearl, 2009; Woodward, 2003). The formalism is the most explicit of the four about which assumptions permit a causal claim. At small n its value is usually representational rather than estimative: a graph can make the assumed structure and the threats to inference visible even when the data cannot support formal identification.
Table 4.2 compares the four traditions on how they define a cause, the place they give to mechanism, their degree of formalism, their data requirements, and their applicability at small n.
| Tradition | Definition of a cause | Role of mechanism | Formalism | Data requirement | Applicability at small n | Maps onto |
|---|---|---|---|---|---|---|
| Mill (1843) | A condition that survives systematic elimination across cases | Implicit, inferred from patterns of presence and absence | Low | A set of comparable cases with variation | High, where cases can be compared directly | Regularity; configurational and eliminative reasoning |
| Mackie (1965) | An INUS condition within a sufficient package | Central, as causes operate in combination | Low to moderate | Cases exhibiting different condition combinations | High, for a modest, diverse case set | Configurational reasoning |
| Rubin (1974) | The contrast between factual and counterfactual outcomes | Absent from the definition, studied separately | High | Units with credible assignment to states | Conditional on the available comparison | Counterfactual reasoning |
| Pearl (2009) | Invariance under intervention on a causal graph | Represented in the graph structure | Very high | Assumptions encoded as a graph, plus data for identification | Mainly a representation tool, not an estimator | Diagnostic and graphical reasoning |
The traditions are not rivals to be chosen between so much as different starting points, each privileging a distinct evidential warrant. Eliminative reasoning asks which candidate survives comparison; the INUS account asks which combination is sufficient; the potential-outcomes framework asks what would have happened otherwise; the graphical approach asks whether the assumed structure permits a claim at all. These warrants map onto the components introduced above. Mill’s eliminative logic informs regularity and configurational reasoning, Mackie’s account underlies configurational methods, Rubin’s framework governs counterfactual designs, and Pearl’s models discipline the representation of assumptions that diagnostic reasoning then assesses. The historical line runs from Hume through Mill, Mackie, Rubin, and Pearl to the applied treatments of Pawson and Cartwright on which later chapters draw. Keeping the four warrants in view explains why no single method is sufficient at small n, and why the chapters that follow match each form of causal claim to the operation that can support it.
4.2 Three forms of causal claim
4.2.1 Effects and difference-making
An effect claim states that an outcome would have differed if an intervention had been absent, introduced at another time, or implemented in another form. The potential-outcomes framework defines an effect through the contrast between outcomes under alternative treatment states (Holland, 1986; Neyman, 1990; Rubin, 1974). Structural causal models express a related intervention through changes to a specified causal system (Pearl, 2009). These frameworks do not require a large sample as a matter of definition. Sample size affects which comparisons can be estimated precisely and which assumptions can be examined with the available data.
At small n, an evaluator may estimate a difference through repeated observations within a unit, randomised intervention timing, or a comparison among a few units. The credibility of the estimate depends on the design. A change following programme introduction is not an effect estimate unless the design gives grounds for excluding maturation, secular change, measurement shifts, and other events at the same time.
Effect estimates do not by themselves explain how an intervention produced an outcome or whether the same result would occur elsewhere. Mediation, heterogeneity, and transportability can be studied within counterfactual frameworks, but they require additional data and assumptions. The limitation belongs to the particular design and evidence, not to counterfactual reasoning as a whole.
4.2.2 Causal configurations
Many programmes operate as one condition within a larger causal package. Mackie (1965) described an INUS condition as an insufficient but necessary part of a condition that is itself unnecessary but sufficient for the outcome. The formulation allows several combinations of conditions to produce the same result. It also allows a programme to contribute to an outcome without being sufficient on its own.
Configurational claims concern necessity, sufficiency, and equifinality within a defined set of cases and conditions. They may identify, for example, that effective local implementation occurs through more than one combination of staffing, authority, resources, and programme support. Such a result depends on how cases, outcomes, and conditions were selected and calibrated. It also depends on the observed diversity of configurations and on the treatment of contradictory or missing cases.
A set relation found across cases does not establish that changing one condition will produce the outcome. That intervention claim requires further assumptions about the causal interpretation of the conditions, their ordering, and the effect of changing one element while others remain in place. Configurational analysis can identify patterns consistent with causal packages, but its warrant is limited by the cases and evidence used to construct them.
4.2.3 Mechanisms and causal processes
A mechanism claim explains the process through which an intervention contributed to an outcome in a particular context. Generative accounts treat causal powers and mechanisms as features of the social world whose operation depends on surrounding conditions (Cartwright, 2007; Pawson & Tilley, 1997). Process-tracing approaches examine whether the observations expected under a proposed mechanism occur and whether competing explanations fit the evidence better (Beach & Pedersen, 2019).
Mechanism evidence can include the timing and sequence of events, administrative records, observed decisions, interviews, and traces left by implementation. The volume of evidence is less important than its relation to the hypotheses under consideration. Several interviews repeating the same account may add little if the informants rely on one source or share an interest in the conclusion.
A well-supported mechanism can explain an outcome without estimating an average treatment effect. It may also clarify why an estimated effect arose or why it varied across settings. Mechanism evidence remains vulnerable to incomplete records, post-hoc reconstruction, dependence among sources, and failure to consider plausible alternatives.
| Form of claim | Question addressed | Main basis for inference | Main boundary |
|---|---|---|---|
| Effect or difference-making | Would the outcome have differed under another intervention state? | A designed or justified comparison between observed and counterfactual outcomes | Does not explain the causal process without additional evidence |
| Causal configuration | Which conditions or combinations are necessary or sufficient in the cases studied? | Calibrated set relations and comparison across configurations | Does not by itself establish the effect of intervening on one condition |
| Mechanism or causal process | Through what process did the intervention contribute to the outcome in this context? | Evidence about sequences, activities, actors, and competing explanations | Does not by itself estimate the magnitude or prevalence of an effect |
The three forms overlap. An evaluation may estimate an effect, investigate the mechanism that produced it, and examine the configurations under which that mechanism operated. They are separated here because each claim requires different evidence and can fail for different reasons. They should not be treated as exclusive questions assigned to separate professional traditions.
4.3 Inferential operations and sources of evidence
Regularity and manipulation contribute to causal inference without forming two additional types of causal claim. Regularity is an observed pattern: an outcome repeatedly follows an intervention, or two calibrated conditions occur together across cases. Such patterns may support any of the three claims, but association does not become causal merely because it is stable or survives adjustment for observed covariates. The design must still justify why confounding, selection, measurement error, and competing processes do not account for the pattern.
Manipulation concerns what can be learned by changing a putative cause. Woodward (2003) defines causal relationships through their response to interventions under specified conditions. In evaluation, deliberate assignment or controlled changes in intervention timing can strengthen an effect claim. Manipulation is therefore a feature of some designs, including some single-case experiments, rather than a separate answer to a different causal question.
Diagnostic updating is also an inferential operation. Evidence changes confidence in a causal hypothesis according to how probable the evidence would be if that hypothesis, or a rival, were true. The updating may be expressed numerically through Bayes’ theorem or qualitatively through the relative probative value of observations. Diagnostic reasoning can be applied to effect, configuration, and mechanism claims. Its credibility depends on the hypotheses considered, the assessment of the evidence under each hypothesis, dependencies among observations, and sensitivity to contested judgements.
Participation concerns who defines the outcome, provides knowledge, interprets evidence, and challenges the resulting account. Stakeholders can identify unrecorded changes, explain implementation decisions, and expose assumptions that an external team would miss. Their agreement does not validate a causal claim on its own. Participation can strengthen an evaluation when different positions are represented and claims are checked against other evidence; it can weaken one when authority, selection, or programme incentives suppress disagreement.
Simulation examines whether specified rules and parameter values generate an observed or anticipated pattern. Agent-based models and system-dynamics models can clarify programme theory, explore feedback, and compare scenarios that cannot be implemented in practice. The results remain conditional on the model. A simulation supports a claim about the real programme only to the extent that its structure, parameters, and outputs are justified against relevant evidence.
4.4 How the method families relate to causal claims
Part II retains six method families because they correspond to recognisable bodies of practice. The families are not six self-contained theories of causation. Counterfactual, configurational, and generative methods are organised mainly around a form of causal claim. Diagnostic methods formalise the assessment of evidence. Participatory methods organise the production and interpretation of evidence. Simulation provides model-based analysis that can support several kinds of claim.
| Method family | Main contribution | Methods discussed in Part II | What the family does not establish on its own |
|---|---|---|---|
| Counterfactual (Chapter 6) | Estimates whether an outcome differed under alternative intervention states | Single-case experimental designs; N-of-1 trials; randomisation inference | The mechanism, transferability, or policy value of the estimated difference |
| Configurational (Chapter 7) | Examines necessary and sufficient conditions and multiple causal pathways | Qualitative comparative analysis; coincidence analysis | The effect of changing one condition while holding the causal package fixed |
| Generative (Chapter 8; Chapter 9; Chapter 10) | Tests explanations of how an intervention contributed to an outcome | Process tracing; contribution analysis; realist evaluation | An average effect or the prevalence of a mechanism beyond the cases studied |
| Diagnostic (Chapter 11) | Updates confidence in causal hypotheses using the probative value of evidence | Bayesian process tracing and related diagnostic approaches | Identification of an effect without a defensible design and hypothesis set |
| Participatory (Chapter 12) | Draws on affected actors to define outcomes, produce evidence, and interpret causal accounts | Most significant change; outcome harvesting | Causal validation through stakeholder agreement alone |
| Simulation (Chapter 13) | Explores the consequences of formal assumptions, feedback, and alternative scenarios | Agent-based modelling; system dynamics | A real-world causal effect without empirical calibration and validation |
External scrutiny of the causal warrant is not evenly distributed across the six families. Participatory and diagnostic methods attract the most direct challenge to whether a causal claim has been established at all: participatory accounts rely on stakeholder attribution without a designed comparison, and diagnostic updating rests on sensitivity and false-positive values that are usually elicited rather than measured, together with an independence assumption across observations that is often violated in practice (Befani, 2020). Configurational analysis draws a comparable challenge from a different direction, since its determinism, its sensitivity to calibration choices, and its treatment of equifinality are recurring targets in the methodological literature (Schneider & Wagemann, 2012). Counterfactual and generative methods are challenged less often on whether their warrant is causal, because they sit closer to established causal-identification traditions; the disputes they attract concern scope, such as transferability or the prevalence of a mechanism, rather than the warrant itself.
The mapping is not exclusive. A single-case experiment combines a counterfactual contrast with intervention and repeated temporal observation. Bayesian process tracing combines mechanism evidence with diagnostic updating. A participatory process may improve the definition and measurement of outcomes used in a counterfactual or configurational analysis. Simulation may examine a proposed mechanism or generate scenarios whose implications are then tested against observed evidence.
These combinations do not automatically strengthen an evaluation. Two methods may rely on the same incomplete records, accessible informants, or assumptions about programme implementation. Their agreement then provides less corroboration than the number of methods suggests. Integration is useful when each component has a defined role, addresses a distinct threat to inference, and produces findings that can be compared without changing the claim midway. Chapter 15 develops those conditions.
4.5 Avoiding category errors
The distinction among levels prevents several common errors. The first is to treat an inferential operation as if it determined the causal claim. Diagnostic updating can strengthen or weaken confidence in an effect, configuration, or mechanism proposition; it does not decide which of those propositions has been identified. Manipulation can strengthen an effect design, but a change in an intervention does not by itself establish that the outcome would have differed under a relevant alternative state. Regularity may describe an outcome pattern without determining whether the pattern is causal.
The second is to treat participation or simulation as substitutes for a causal warrant. Stakeholders may identify outcomes, correct an implementation account, and challenge the categories used by an external team. Those contributions can alter the question and evidence of an evaluation. They do not remove the need to assess alternative explanations for a causal conclusion. A simulation can make a causal theory explicit and reveal implications that are difficult to observe directly. It does not establish that the theory describes the programme unless its assumptions and outputs are tested against relevant evidence.
The third is to add methods without deciding how their findings relate. An evaluation may contain a repeated outcome series, interviews, a QCA, and a participatory workshop. The methods do not become a multi-logic design merely because they appear in the same report. The team needs to state which claim each component addresses, what threat it is intended to address, whether the evidence is independent, and how a disagreement will be interpreted. This is the basis for the integration designs developed in Chapter 15.
The framework therefore has two uses. It helps an evaluation team select a method family appropriate to a causal question. It also helps a reader audit a completed evaluation: identify the claim, locate the operation that supplies the warrant, and see where evidence production, assumptions, or unaddressed alternatives limit the conclusion.
4.6 Theory of change and causal reasoning
A theory of change is often the starting point for small-n evaluation. It can describe how programme activities are expected to lead to intermediate changes and outcomes, identify assumptions about implementation and context, and make a mechanism proposition available for testing. It can also help a team decide which outcome needs measurement, which actors and records may provide evidence, and where a rival explanation could enter the process. These are useful functions in counterfactual, configurational, generative, diagnostic, participatory, and simulation work.
A theory of change is not, however, an additional method family or a causal warrant in its own right. A diagram of activities and expected outcomes does not establish that the programme produced the outcome, that a configuration was sufficient, or that the proposed mechanism operated. Those conclusions require evidence appropriate to the claim. Contribution analysis uses a theory of change to organise an account of contribution and to identify assumptions and alternative explanations; the theory becomes more credible only when those elements are examined against evidence , Chapter 9.
This distinction is important for commissioning. Terms of reference frequently require a theory of change, but they should also specify the claim the evaluation is expected to investigate and the evidence that could support it. A theory of change may reveal that an effect estimate is infeasible, that a process-tracing question is more appropriate, or that the programme operates through different configurations across sites. It does not settle that decision in advance.
4.7 From a causal question to a design
The framework can be used in a short sequence. First, define the intervention or causal condition, outcome, unit, period, and decision to which the conclusion will apply. Second, identify the form of causal claim: an effect, a configuration, a mechanism, or a deliberately specified combination. Third, identify the operation that could make the claim credible, such as a within-unit contrast, a calibrated cross-case comparison, a test of competing process explanations, or diagnostic assessment of evidence. Fourth, decide how evidence will be produced, recorded, checked, and interpreted, including the role of participation or simulation where relevant. Finally, state the alternative explanations, assumptions, and scope conditions that will remain in the report.
The sequence is not a mechanical decision rule. Feasibility may require the team to return to the question after examining records, access, ethical constraints, or programme timing. A planned effect study may become a study of implementation and mechanism because a comparison cannot be established. A theory of change may reveal that cases need to be defined differently for a configurational analysis. A participatory process may show that the outcome initially selected by a commissioner omits a change that participants regard as consequential. Revisions are legitimate when they are documented and when the final claim reflects the evidence that was actually obtained.
Chapter 14 provides a fuller decision framework for these choices. The purpose of the present chapter is narrower: to prevent the method label from deciding the question before the causal claim, inferential operation, and available evidence have been identified.
4.8 Choosing among methods at small n
No method family has general priority independently of the claim, design, and evidence. Random allocation can provide a strong basis for an effect estimate, including at small n, but it does not guarantee adequate measurement, implementation fidelity, or external relevance (Deaton & Cartwright, 2018). A detailed process account may explain how an outcome occurred, but it can remain unconvincing if the evidence was selected after the conclusion or plausible alternatives were not examined. A QCA solution may fit the observed cases and still be sensitive to calibration or limited diversity.
Method choice therefore requires more than matching a broad question to a named family. The evaluator must specify the intervention or exposure, outcome, units and period covered by the claim, relevant alternative states or explanations, available variation, and intended use. The design can then be assessed against the main threats to that claim. Shared standards such as transparency about assumptions, evidence provenance, rival explanations, and uncertainty remain applicable across methodological traditions (Brady & Collier, 2010).
Purpose matters because an evaluation intended to decide whether to continue a programme may require different evidence from one intended to revise implementation. Patton (2008) places intended use and users at the centre of evaluation design. Intended use does not remove the need to assess whether the available design can support the requested conclusion. A useful answer to a different question is not evidence for the original claim.
Part II examines the six families in turn, beginning with counterfactual designs at small n. The chapters retain their methodological identities while using the distinctions developed here: what causal claim is being made, what inferential work the method performs, what evidence it requires, and what remains unresolved.