20 Small-n Evidence in Constrained Settings
The preceding chapters set out the conditions under which a causal claim can be designed, examined, and reported. In some settings, access to those conditions is limited. Fragile and conflict-affected settings (FCAS), institutional reform programmes in low-capacity environments, and emergency response evaluations may combine small-n constraints with political instability, resource scarcity, weak institutional capacity, limited data infrastructure, and restricted physical access (Sempé et al., 2025). The OECD has noted that evaluating peacebuilding and state-building interventions requires designs that accommodate instability, attribution ambiguity, and limited baselines (OECD, 2012).
This chapter examines how these moderating factors affect the six method families and the reasoning components set out in Chapter 4. It identifies threats to valid application, the adaptations that remain defensible, and the conditions under which the intended claim should be narrowed or the evaluation should not proceed.
20.1 The compounding gap
The FCAS evidence gap map reviewed 5,327 empirical studies and found specialised small-n methods in fewer than 2 per cent of records (Sempé, 2026). This is a literature-mapping finding, not an estimate of the prevalence of valid small-n evaluation. It nevertheless identifies a limited verified application base for methods that could be relevant where programme units, comparisons, records, or access are constrained. Reviews of impact evaluation and measurement in humanitarian settings also describe the difficulty of applying conventional designs under instability and limited data (Brück et al., 2016; Puri et al., 2017).
The FCAS evidence gap. Of 5,327 studies in fragile and conflict-affected settings, fewer than 2 per cent used specialised small-n methods (Sempé, 2026). The map identifies records; it does not establish the quality of their application.
The relevant question is whether the proposed method can be implemented without changing the claim beyond recognition. A method may be appropriate in principle but infeasible because the setting does not support its measurement, access, comparison, or safety requirements. Adaptation should identify what remains possible, what is lost, and whether the intended conclusion must be narrowed (Garbarino & Holland, 2009; White, 2014).
20.2 Constraints by logic
20.2.1 Counterfactual logic
SCEDs and randomisation inference require repeated measurement of the outcome across baseline and intervention phases. Three constraints can threaten this requirement in FCAS and other constrained settings. First, physical access may be intermittent. If data collectors cannot visit a programme site on a regular schedule, the temporal density required for a SCED is compromised. Gaps in the measurement series weaken the assessment of baseline stability and can limit randomisation tests. Second, an outcome measure may not capture the relevant construct in the setting where it is used. Third, phase-reversal designs may be ethically unacceptable if withdrawal exposes participants to harm.
Adaptations. Remote data collection, including mobile surveys, satellite imagery, or electronically transmitted records, may reduce gaps in a measurement series, but it does not establish that the remote measure captures the outcome validly. Multiple-baseline designs can avoid withdrawal where staggered introduction is administratively and ethically possible. Where gaps prevent the planned comparison, visual analysis can describe the observed pattern, but it does not restore the counterfactual warrant that the missing observations have weakened (Kratochwill et al., 2013).
Remote measurement. Remote sources may maintain an outcome series when physical access is restricted. Their validity, missingness, and relation to the causal claim still require assessment.
20.2.2 Configurational logic
QCA and CNA require defensible calibration of condition sets across the cases included. Reliable data on contextual conditions may be absent or based on proxies of uncertain validity; a change in one condition can alter the truth table and its solution. Very small case sets also limit the diversity available to distinguish configurations. There is no universal numerical threshold that resolves this problem. The evaluator must show whether the available cases, condition definitions, and outcome variation can support the proposed analysis.
Adaptations. Qualitative anchors and structured expert assessment can supplement uncertain indicators only when their rationale, disagreement, and relation to cases are reported. Neither QCA nor CNA is justified by a very small number of cases alone. Sensitivity analysis should show whether plausible calibration choices change the solution and identify conditions dependent on a single source.
20.2.3 Generative logic
Process tracing, contribution analysis, and realist evaluation can require access to people and records that describe how a programme operated, what changed, and why. Institutional memory may be disrupted, records may be inaccessible, and informants with direct knowledge may no longer be available. Respondents may also face security risks from providing candid accounts of programme operations, particularly in governance, justice, or security-sector reform programmes (Bush & Duggan, 2014). A theory of change may be incomplete when a programme was designed and adapted under emergency conditions.
Adaptations. Contribution analysis can begin from a theory of change reconstructed from programme documents and available interviews, provided the report is explicit about the evidence available for each step. Process tracing can draw on documentary evidence when informant access is limited, though the range of mechanism tests may narrow. Where respondent safety is a concern, data-collection protocols should be reviewed against “do no harm” principles before fieldwork begins. A realist inquiry may need to narrow its CMO propositions and report where evidence cannot distinguish the proposed mechanism from alternatives.
20.2.4 Diagnostic logic
Bayesian process tracing requires the evaluator to estimate the sensitivity and false-positive rate of each diagnostic test. These probability estimates are already challenging in data-rich settings, as Befani (2020) has documented. In constrained settings, two additional problems arise. First, the evidence base for estimating test probabilities is thinner. If the evaluator has fewer documents, fewer informants, and less contextual knowledge, the basis for judging whether a test result is a true positive or a false positive is weaker. Second, the prior probability itself may be harder to establish when the evaluation is operating in a context with little precedent and few comparable programmes.
Adaptations. Structured elicitation may make uncertainty in diagnostic assumptions explicit, but it does not supply an empirical basis for a probability estimate. Befani et al. (2021) is a proof of concept for simulation-assisted diagnostic reasoning, not an evaluation application. Sensitivity analysis should report how plausible assumptions affect the conclusion. Where evidence is too thin for formal updating, qualitative diagnostic tests can organise the assessment of alternatives, provided the report does not convert their labels into numerical confidence.
20.2.5 Participatory logic
Most Significant Change and Outcome Harvesting depend on the willingness and ability of programme participants and stakeholders to provide candid accounts of change. Power dynamics may silence some voices, and people may not feel safe discussing changes that implicate armed actors or government officials. Access constraints can limit the diversity of participants reached. Translation and cultural mediation add further layers of interpretation to the accounts collected.
Adaptations. Remote collection may extend reach beyond physically accessible areas, but digital access can itself be uneven. Local facilitators may improve communication where they have appropriate language skills, independence, and safeguarding arrangements. MSC selection panels can include affected participants alongside programme staff and external evaluators. Outcome Harvesting should document its substantiation process and the sources unavailable for it. These measures may reduce particular risks; they do not make accounts representative of groups that could not participate safely.
20.2.6 Simulation logic
Agent-based models and system dynamics models require parameter values derived from empirical data, expert judgement, or both. In constrained settings, the primary threat is that the models are calibrated to data of uncertain quality, producing trajectories that are precise but inaccurate. A secondary threat is that the complexity of the modelling environment creates a black box that commissioners and policy users cannot interrogate.
Adaptations. Participatory model-building can make local assumptions and disagreement visible when quantitative calibration data are scarce. It does not validate the resulting model. The model should be presented as a structured exploration of stated scenarios, with explicit documentation of which inputs are data-driven and which are assumption-driven. Sensitivity analysis should show how plausible changes in assumptions alter outputs.
Table 20.1 synthesises the key constraint, the principal adaptation strategy, and the feasibility assessment for each logic.
| Logic | Key constraint in FCAS | Defensible response | Consequence for claim |
|---|---|---|---|
| Counterfactual | Intermittent access disrupts measurement | Assess validity of remote measures; use staggered introduction only where feasible | Missing comparison evidence may reduce the study to descriptive pattern analysis |
| Configurational | Uncertain calibration and limited diversity | Use transparent anchors and sensitivity analysis; do not force a truth-table method | Configuration claims may be infeasible |
| Generative | Lost records and risks to informants | Protect sources; use available documents; narrow the mechanism | Mechanism account may remain partial |
| Diagnostic | Thin basis for diagnostic assumptions | Test sensitivity and use qualitative diagnostics where warranted | Do not report numerical confidence without a defensible basis |
| Participatory | Unequal voice and restricted reach | Safeguard participation and substantiation; document exclusions | Accounts may not represent inaccessible groups |
| Simulation | Uncertain inputs and opaque structure | Document scenarios, assumptions, and uncertainty | Results remain conditional model outputs |
Each row traces back to a Part II methods chapter: counterfactual (Chapter 6), configurational (Chapter 7), generative approaches through process tracing (Chapter 8), contribution analysis (Chapter 9), and realist evaluation (Chapter 10), diagnostic (Chapter 11), participatory (Chapter 12), and simulation (Chapter 13).
20.3 Cross-cutting adaptation principles
Three principles apply across the method families discussed in this chapter.
First, transparency about constraint adaptation should be built into the evaluation report. When a method is adapted for a constrained setting, the adaptation should be documented, the inferential consequences should be stated, and the reader should be able to judge whether the adaptation preserved the logic’s core warrant or compromised it. An evaluation that reports its constraints openly is more credible than one that applies a full-strength method claim to data that do not support it.
Second, integration is not a remedy for missing evidence. A second logic is useful when it has a distinct evidential job and access to a different source or unit. It cannot compensate for a gap shared by every component. A process trace using incomplete records and a configurational analysis calibrated from the same incomplete records may reproduce the same limitation. A counterfactual component with a damaged outcome series and a participatory component may answer different questions, but the latter does not repair the comparison. The integration protocol should state what each component adds and which uncertainties remain shared.
Third, the “do no harm” principle governs all adaptation decisions.
ALNAP (2016) provides the standard framework for evaluation ethics in humanitarian settings. The “do no harm” principle overrides all methodological considerations.
No methodological gain justifies exposing participants, informants, or data collectors to harm. When data collection in a conflict-affected setting would require access to areas controlled by armed groups, the evaluator must weigh the evidential value against the physical risk. When the publication of findings could endanger programme beneficiaries, the evaluation report must be designed to protect identities and sensitive information. These are not methodological constraints in the usual sense. They are ethical constraints that override methodological considerations and that any responsible commissioning protocol must address before the evaluation begins (ALNAP, 2016).
20.4 Triage, claim reduction, and stopping rules
Constrained settings require an explicit decision about whether the proposed evaluation is feasible before resources are committed. The decision should not be based on whether some data can be collected. It should be based on whether the available observations can support the claim requested without creating unacceptable risk. A short feasibility assessment can identify the intended outcome or mechanism, the accessible unit or units, the evidence that could distinguish the main alternative explanation, the security and ethical constraints, and the decision that would use the findings.
Where the assessment finds a gap, the claim should be narrowed before fieldwork. An evaluation may be able to document implementation when it cannot estimate an effect; it may be able to reconstruct selected decisions when it cannot test a complete mechanism; it may be able to record participant-identified outcomes when it cannot establish their distribution or attribution. These are useful products if they are described accurately. They become misleading when a report keeps the original impact language after the evidential basis has changed.
Stopping rules should be agreed in advance. They may include loss of safe access to a site, a security incident affecting participants or data collectors, loss of a critical outcome series, an inability to protect confidentiality, or a level of missingness that prevents the planned comparison. The rule should specify who decides whether to pause, revise, or stop the work, and how the decision will be recorded. It should not leave field teams to choose between safety and an unexamined contractual requirement.
20.4.1 Building evidence during delivery
Some constraints can be reduced through programme design. A programme can retain dated implementation records, define a small number of outcomes that can be measured safely, record changes in delivery and exposure, and agree procedures for secure data storage. These measures do not turn an emergency response into a trial. They preserve the chronology and observations needed for later contribution, mechanism, or descriptive inquiry.
Commissioners should distinguish this prospective evidence work from a retrospective evaluation commissioned after delivery. Where access is likely to be intermittent, a modest routine record may be more valuable than an ambitious endline survey that cannot be repeated. Where a programme changes rapidly, a dated record of decisions and adaptations can be necessary to interpret any later outcome pattern. The choice of what to preserve should follow the programme’s causal questions and risks, not a generic monitoring template.
20.5 Reporting constrained evidence
The report should state the constraints in its methods and interpretation, not only in a limitations section. It should identify which planned sources were available, which were not, how access or safety changed the design, and what conclusion follows from the revised evidence base. It should also distinguish evidence that could not be collected from evidence that was collected but is unreliable or contested. These situations have different implications for use.
A constrained setting does not justify lower standards of disclosure. It may justify a narrower claim, a delayed release, aggregation of sensitive information, or an internal technical annex with protected detail. The public product should still allow readers to understand the causal question, the sources used, the adaptations made, and the remaining uncertainty. This is the basis on which a commissioner can decide whether the evaluation informs immediate action, programme learning, or a later and more complete study.
20.5.2 A proportionate conclusion
The conclusion should match the design that remained feasible, not the design initially intended. It may state that an implementation record is consistent with a proposed pathway; that a mechanism claim is supported only for selected steps; that a model clarifies a scenario under stated assumptions; or that the available evidence cannot support the effect claim requested. Such conclusions can inform programme management and the design of subsequent evidence collection. They should not be presented as second-best versions of an impact evaluation.
The discipline required in constrained settings is therefore not methodological improvisation. It is clear specification of the claim, protection of people and data, careful preservation of the available evidence, and a report that does not conceal what the setting prevented the evaluation from knowing.
20.6 Recovering an evidence strategy after disruption
Disruption does not always require the evaluation to end. It may require the team to separate evidence that was collected before the disruption from evidence collected afterwards, revise the period covered by the causal claim, or postpone an outcome assessment until access and measurement are restored. The amendment should explain whether the disruption is itself part of the causal context. A conflict escalation, population movement, or sudden change in programme delivery can affect both the intervention and the outcome; treating it only as missing data may conceal a central alternative explanation.
Recovery should be staged. The first task is to establish safety and data protection. The second is to reconstruct a dated record of what changed in access, implementation, and measurement. The third is to decide whether the original claim remains testable, should be narrowed, or should be deferred. This sequence prevents pressure to resume data collection from determining the inference before the evidence structure has been reconsidered.
The final report can then distinguish an interrupted evaluation from a failed one. An interrupted study may still contribute a documented implementation chronology, evidence on conditions under which delivery became impossible, or a revised theory of change for future work. Its value lies in what those records establish, not in an attempt to recreate the original design through assertion.
The next chapter considers how small-n findings can be synthesised across studies without flattening their different warrants. Chapter 22 then draws together the book’s argument, the method repertoire, and the EIPM perspective developed in this part.