21 Evidence Synthesis of Small-n Studies
Evidence synthesis is often presented as the final step of an evaluation sequence: identify studies, assess quality, and combine findings. That sequence is insufficient for small-n evidence. Studies may use the same method label while addressing different causal questions, defining outcomes differently, or relying on different warrants. A pooled estimate, a realist proposition, a configuration, and a set of substantiated outcome accounts are not interchangeable summaries of the same evidence.
This chapter examines how small-n findings can be compared and combined without flattening those distinctions. It covers two related tasks. The first is synthesis across separate studies. The second is synthesis within a programme that has several sites, cases, or implementation episodes. The tasks share questions about comparability and claim boundaries, but they differ in their units, access to data, and opportunities for prospective design.
21.1 Start with the causal claim
The unit of synthesis is not the publication. It is a causal claim with an identifiable outcome, unit, intervention or condition, period, and evidential warrant. A study may contain more than one claim. A single-case design may support a within-unit contrast. Its implementation record may bear on fidelity. Interviews may provide evidence on a mechanism. Treating the article as one homogeneous item obscures those distinctions and can give weak evidence the appearance of support because it appears beside a stronger result under the same study label.
Before combining findings, the reviewer should therefore construct a claim record. The record identifies the causal question, the logic used, the unit and case population, the outcome definition, the intervention or causal condition, the evidence used, the assumptions, the conclusion, and the limits stated by the original authors. It should also record what the study did not establish. This permits later synthesis to distinguish a reported change from an estimated effect, a mechanism proposition from a verified mechanism, and an outcome account from an attribution claim.
| Field | Record for each claim |
|---|---|
| Causal question | The effect, contribution, mechanism, configuration, diagnostic, participatory, or simulation question addressed |
| Unit and population | The case, site, participant, organisation, programme episode, or study population to which the claim applies |
| Outcome and period | Definition, measurement, timing, and direction of the outcome |
| Causal condition | Intervention, exposure, configuration, mechanism proposition, or scenario considered |
| Logic and warrant | Comparison, evidence test, calibration, substantiation procedure, or model assumptions |
| Conclusion | The conclusion as reported, including uncertainty and conditions |
| Boundary | Alternative explanations, transport limits, missing evidence, and claims not supported |
The record is an analytical device, not a generic extraction form. It forces the synthesis to retain the relationship between a conclusion and the evidence that warranted it. It also makes it possible to compare studies from different fields without pretending that their methods supply the same type of causal knowledge.
21.2 Comparability before combination
Evidence can be combined only after the reviewer has determined what is comparable. Four questions determine that decision. Do the studies address the same causal question? Do they define the intervention, causal condition, and outcome in a sufficiently similar way? Do their units and settings permit a shared claim? Do the studies rely on warrants that can be assessed together?
Similarity of topic is not enough. Two programmes described as teacher training may differ in their active components, delivery, population, and outcome timing. Two process traces may examine the same policy but test different mechanisms. Two QCA studies may use the same condition label with different calibration anchors. These differences do not make comparison impossible. They determine whether the appropriate synthesis is a pooled estimate, a comparison of propositions, a configuration analysis, or a structured account of divergence.
The reviewer should also distinguish direct from systematic replication. Direct replication holds the intervention and outcome definition as close as possible to the original study. Systematic replication changes a setting, population, delivery feature, or condition in order to examine the boundary of the claim. The latter can be more informative for policy, but it does not justify treating all results as estimates of one common effect. The synthesis should report what was held constant, what varied, and what conclusion the variation permits (Barlow et al., 2009; Smith, 2012).
21.3 Synthesis across separate studies
21.3.1 Counterfactual claims
Single-case experimental studies can sometimes be synthesised through standardised effect sizes, hierarchical models, or structured visual comparison. The decision to pool should follow the design and outcome record. Phase structure, baseline trend, outcome direction, measurement scale, intervention fidelity, randomisation, and selective reporting affect whether effect estimates are comparable. Smith (2012) review the design, measurement, and analytic standards that make this assessment possible.
Pooling is not the default. A set of studies may support a conclusion about replicated within-unit contrasts while remaining too heterogeneous for one average effect. In that case, a synthesis can report the pattern of replications and the conditions under which the contrast appeared, did not appear, or could not be assessed. Chapter 6 provides the relevant effect-size and replication concepts.
21.3.2 Generative and diagnostic claims
Mechanism-focused synthesis begins with propositions, not effect sizes. Realist synthesis provides one established approach: it compares how a programme was expected to operate with evidence from different contexts, refining context–mechanism–outcome propositions as the review proceeds (Wong et al., 2013). Its appraisal is claim-specific. A trial may be relevant to an outcome contrast; a case study may be relevant to a decision process; an administrative record may bear on implementation. Their contribution depends on relevance to the proposition and the rigour of the particular evidence, not on an undifferentiated ranking of entire documents.
Process-tracing and contribution claims require an additional precaution. The same documentary source may be cited across several publications, and several reports may repeat a programme’s preferred explanation. A synthesis should identify such dependence rather than counting each publication as independent support. Contrary cases and alternative mechanisms are part of the synthesis, because they identify the conditions under which a proposition does not hold.
21.3.3 Configurational claims
Configurational findings can be compared when conditions and outcomes have compatible substantive definitions and calibration anchors. A synthesis should not combine solution terms merely because they use the same labels. The reviewer needs to inspect the case universe, the calibration rules, treatment of limited diversity, and the cases covered by each solution. Where these differ, the useful result may be a map of proposed configurations and their boundaries rather than a new truth table.
QCA can itself be used within a systematic review when primary studies can be treated as cases and their features coded consistently. That is a distinct research design, not a mechanical way to aggregate QCA publications. It requires an explicit theory of the conditions, transparent coding of the review evidence, and adequate diversity across the included studies. The resulting configuration explains a pattern among studies; it does not validate every causal conclusion reported in them.
Kahwati et al. (2016) provide a worked example. They applied QCA to studies from a completed review of interventions to improve medication adherence after heterogeneity had prevented quantitative pooling. The studies became cases; behaviour-change techniques and implementation features became conditions. Their experience identifies a practical limit for this chapter’s claim record: standard review abstractions did not contain enough intervention detail to support calibration. The team returned to the primary reports to clarify and supplement the extracted information. The QCA did not replace the completed review’s findings. It supplied a configurational question about combinations of features across the eligible studies. A synthesis protocol should make this additional coding and its source trail visible from the start.
21.3.4 Participatory and simulation-supported claims
Participatory evidence can be synthesised when the review preserves who identified an outcome, how it was substantiated, and whose accounts were not available. The aim is not to count stories as observations of one effect. It is to compare outcome descriptions, the values used in their selection, and the evidence offered for their accuracy or contribution. A synthesis may identify recurring changes, disagreements between groups, or outcomes that were not included in standard measures. It should not turn stakeholder agreement into causal attribution without further warrant.
Blundo-Canto et al. (2017) describe a learning system that adapted Outcome Harvesting across three collections of outcomes in agricultural research for development. For synthesis, its organisational implication is clear: repeated harvests can support comparison and learning when outcome statements, guiding questions, substantiation, and interpretation are documented consistently. They do not create a common effect measure. A cross-site participatory synthesis should report the range of outcomes, selection and substantiation procedures, and differences in who was able to contribute.
Simulation evidence requires a separate record of model purpose, structure, inputs, calibration, validation, and scenario definition. Two models may address the same policy issue while representing different causal processes. Their outputs should not be averaged unless their structure and uncertainty assessment justify that operation. More commonly, the synthesis compares assumptions, identifies where model results converge or diverge, and examines which empirical observations bear on the disputed parts of the models. Simulation can support a synthesis by clarifying a dynamic hypothesis; it cannot compensate for missing evidence on whether an intervention produced an observed outcome.
21.4 Synthesis within a programme
A programme may operate through several schools, districts, organisations, cohorts, or implementation episodes. Its evaluation can plan synthesis prospectively because the units, records, outcome definitions, and timing are partly under the programme’s control. This does not make the task equivalent to pooling separate studies. Sites may share staff, funding, policy shocks, reporting systems, or an intervention label that conceals meaningful differences in delivery.
The evaluation should begin by deciding what is to be compared across sites. A counterfactual component may examine repeated outcome series within each site. A configurational component may compare implementation and context across sites. Process tracing may investigate selected typical, deviant, or informative sites. Participatory work may identify outcomes or concerns not captured in the common measurement plan. Each component needs a specified role in the cross-site conclusion.
Alamos-Concha et al. (2026) illustrate a sequential design in which QCA identified configurations associated with training transfer in Flemish small and medium-sized enterprises and process tracing examined mechanisms in selected cases. The policy value of this arrangement was not a pooled estimate. QCA supplied a basis for cross-case comparison and case selection; process tracing supplied within-case evidence on how the proposed processes operated. The resulting conclusion remained conditional on the coded cases, the configuration definitions, and the selected process traces.
21.4.1 A protocol for within-programme synthesis
Before analysis, the evaluation team should record the common and site-specific elements of the programme, outcome measures, measurement schedule, available records, planned case selection, and likely sources of interference. A shared chronology is particularly important. It identifies whether sites received the same intervention at the same time, whether local adaptations occurred, and whether common events could explain an outcome pattern across all sites.
The protocol should then state how evidence moves between components. A configurational analysis may identify sites for process tracing. A process trace may identify a condition that needs to be measured across sites. A repeated outcome series may identify a change requiring an implementation review. The final synthesis should preserve the order of these decisions and distinguish a planned handover from an adaptation prompted by the data.
21.5 A protocol for synthesis
The protocol should be written before the reviewer begins to compare results. It need not require a conventional systematic-review question or a commitment to pooled analysis. It should state the causal question, eligible claim types, units and settings, sources to be searched, the evidence that will be extracted, and the conditions under which claims will be compared, combined, or retained separately. These choices can be amended when the literature reveals an unforeseen problem, but the amendment should state what changed and why.
The first stage is eligibility. A report is eligible only when it can contribute evidence to the stated claim. A paper that uses a method label without applying the method to a causal question is not an application. A study may be eligible for one claim and not another. A realist evaluation may contribute evidence on a mechanism while not providing an effect comparison. A SCED may be eligible for an effect synthesis but not for a review of system-level implementation. The protocol should record these decisions at the claim level.
The second stage is extraction and appraisal. Reviewers should work from the claim record and retain a link to the underlying passage, table, figure, or dataset. They should record whether evidence is independent, whether another report uses the same cases or records, and whether the conclusion was pre-specified or developed after results were known. A structured second review of a subset of claims can identify disagreements in interpretation, especially where calibration, mechanism tests, or outcome substantiation require judgement.
The third stage is a comparability decision. Claims may be placed in one of four categories:
- Directly comparable: the causal question, contrast or condition, outcome, and warrant are sufficiently aligned for formal synthesis.
- Comparable for patterned synthesis: the claims share a causal question but vary in context, implementation, or measurement; the synthesis examines replication, configurations, or proposition boundaries.
- Comparable for explanation: the claims address related mechanisms or outcomes but require a theory-led account of why they differ.
- Not comparable: the studies can be mapped together but should not be combined into one causal conclusion.
These categories are not a hierarchy. They are an account of what the available evidence permits. A review with no directly comparable claims can still be valuable if it identifies the mechanisms, conditions, and measurement differences that prevent a general conclusion. This result may be more useful for commissioning future work than an average constructed from non-equivalent studies.
The final stage is synthesis and challenge. The initial synthesis should be tested against contrary cases, alternative mechanisms, changes in calibration or inclusion decisions, and source dependence. The reviewer should ask what evidence would weaken the preferred conclusion and whether that evidence has been sought. A synthesis becomes less credible when every difference is interpreted as confirmation of a sufficiently flexible theory.
21.6 Reporting a synthesis
A synthesis report should make three decisions visible. First, it should show which claims were eligible for comparison and which were not, with reasons. Second, it should describe the evidence that supports each synthesis conclusion and any dependence among sources. Third, it should state the level of the final claim: a replicated within-unit contrast, a refined programme theory, a configuration applicable to a defined case universe, a set of substantiated outcomes, or a conditional model scenario.
| Reporting question | What the synthesis should show |
|---|---|
| What was compared? | The claim record, case or study universe, and exclusions |
| Why was combination appropriate? | Definitions, units, outcome timing, warrant, and relevant variation |
| How were sources appraised? | Claim-specific relevance and rigour, including source dependence |
| What did the synthesis find? | The conclusion, contradictory evidence, and the cases or studies to which it applies |
| What remains uncertain? | Alternative explanations, transport limits, measurement gaps, and claims not supported |
| What follows for action? | Whether the finding informs continuation, adaptation, monitoring, a further study, or no causal decision |
This structure prevents a familiar error: presenting a synthesis as stronger because it includes more studies while leaving the relationship among their claims unexplained. More documents can provide more evidence, but they can also repeat the same source, the same implementation account, or the same measurement problem. The quality of a synthesis depends on the comparison it makes, not on the length of its reference list.
21.6.1 Negative cases and quality appraisal
Contrary findings are not a residual category. They may identify a different configuration, a failed mechanism link, an outcome measured under different conditions, or a limit in the original claim. The synthesis should retain them as cases requiring explanation. Excluding them because their design or result differs from the majority can produce an account that describes only the conditions under which a programme appeared to work.
Appraisal should be tied to the purpose for which a source is used. A source may provide credible information on the date of a decision but weak evidence on the decision’s consequences. A single-case study may offer a strong within-unit contrast but limited information on transfer. An outcome account may establish that a participant-reported change was substantiated without establishing a population effect. Assigning a single quality score to an entire report can conceal these differences. The claim record should instead state what each source can and cannot contribute to the synthesis proposition.
The same discipline applies to grey literature and programme documents. Such material can be indispensable for reconstructing implementation, exposure, or a decision chronology. It may have been produced by an interested organisation and may omit contrary evidence. A synthesis should record provenance, access, and potential dependence before using the material as corroboration. Transparency does not require excluding grey literature. It requires explaining its evidential role.
21.7 The role of bibliometric mapping
The bibliometric appendix maps a literature. It identifies publications, method labels, co-mentions, and the extent to which records appear relevant to evaluation. That information can help a reviewer locate a field, identify ambiguous method names, and plan a search strategy. It does not establish that a method was applied correctly, that a publication contains a causal claim, or that the findings are comparable. The appendix should therefore be used as a discovery and scoping resource, not as an evidence synthesis.
The distinction is practical. A bibliometric record may name process tracing, contribution analysis, or simulation in a title or abstract while using the term in another technical sense or without applying it to an evaluation. A synthesis requires full-text eligibility assessment, extraction of the causal claim, method-specific appraisal, and a decision about comparability. The validated application subset can make this work more efficient. It cannot replace it.
Bibliometric mapping can also reveal a substantive absence. If few verified applications connect a method with a particular policy domain, the reviewer should not treat a large number of keyword matches as evidence of a mature synthesis base. The appropriate conclusion may be that the evidence base is too thin or too heterogeneous for synthesis. That finding can guide commissioning, primary research, and the next iteration of the literature map.
21.8 Planning for later synthesis
Commissioners can make later synthesis more useful by deciding early which elements of an intervention need to be comparable across sites or studies. This may include an outcome definition, a dated implementation record, core descriptors of the population and setting, and a way to record adaptations. It does not require uniform delivery in every context. It preserves enough information to distinguish a meaningful adaptation from an intervention that no longer has the same causal condition.
The decision should follow the intended claim. A programme seeking evidence on a repeated within-unit contrast needs measurement timing and phase information. A programme seeking to compare mechanisms needs a shared theory of change and a record of the observations used to assess its links. A programme seeking configurational learning needs consistent, substantively defensible condition definitions across its sites. A participatory programme needs a transparent approach to outcome selection and substantiation. These are commissioning choices as much as analytic ones.
Prospective planning also protects against selective synthesis. If evaluators specify the claims, core measures, and source records before results are known, later reviewers can see which sites or outcomes were not included and why. This does not prevent adaptation. It makes adaptation visible, and it allows a synthesis to distinguish a deliberate design change from a decision that followed a favourable or unfavourable result.
21.9 Limits and role in the handbook
Evidence synthesis does not create a common currency for all small-n methods. It makes their differences inspectable and identifies the narrower conclusions that a set of studies can support together. The appropriate result may be an effect estimate, a pattern of replications, a refined mechanism proposition, a bounded configuration, a map of participant-identified outcomes, or a decision that evidence should not yet be combined.
The chapter therefore connects the method chapters to commissioning and translation. Commissioners can preserve common measures, records, and decision points when a programme is expected to generate evidence across sites. Evaluators can specify how claims will be compared before results are known. Evidence users can see whether a synthesis supports a continuation decision, an adaptation, a further test, or only a more precise account of uncertainty. Chapter 21 returns to the wider argument: small-n causal inquiry requires a repertoire of logics, and synthesis must respect the warrant each logic provides.