19 Translating Small-n Evidence for Policy
An evaluation produces findings that require interpretation before they can inform a decision. Chapter 18 addressed how evaluations are specified and procured; this chapter turns to what happens after findings are produced. Translation is particularly demanding when an evaluation produces a set relation, a mechanism proposition, a diagnostic assessment, a participant-defined outcome account, or a model scenario rather than a conventional effect estimate (Greenhalgh & Wieringa, 2011; Lavis et al., 2003). The communication product must make the finding intelligible without changing its causal claim or evidential warrant.
This chapter addresses the translation challenge. It relates the evidential product of each method family to the EIPM outcome domains and proposes communication strategies that preserve the inferential integrity of each claim while making findings accessible to non-specialist audiences (Sempé et al., 2025).
19.1 Evidential products by logic
Each method family produces a distinctive type of claim. The differences are not merely stylistic: they concern the causal question, evidence, assumptions, and limits. Treating these products as interchangeable, or as substitutes for a statistical effect estimate, misrepresents what the evaluation has established. Policy evidence is interpreted in deliberative settings as well as through technical appraisal; the form of a finding therefore affects what readers can understand and challenge (Majone, 1989).
Six evidential products. (1) Within-unit effect estimate. (2) Set-theoretic statement. (3) Mechanism narrative. (4) Posterior probability. (5) Stakeholder-validated outcome. (6) Model-generated trajectory. Each rests on a different causal logic.
Counterfactual methods, including SCEDs and randomisation inference, can produce within-unit effect estimates: the measured difference between intervention and baseline phases for each unit. Where the design permits randomisation inference, a p-value can be derived from the assignment or timing mechanism rather than an asymptotic distribution. The claim is that the intervention changed the outcome for this unit, under the assumptions required by the design. The evidential product is often expressed as an effect estimate, but it describes the units and comparison examined rather than a population average by default.
Configurational methods (QCA, CNA) produce set-theoretic statements about conditions or combinations of conditions that are necessary or sufficient for an outcome within the analysed case set. They do not estimate a probability of success or the size of an effect. The policy product should retain the case universe, outcome definition, and calibration decisions on which the configuration depends.
Generative methods (process tracing, contribution analysis, realist evaluation) produce accounts of a proposed causal pathway. Their conclusions should identify which links are supported, which alternatives were examined, and which parts of the pathway remain uncertain. The claim concerns causal process, not the magnitude of an effect at each step.
Diagnostic methods, including Bayesian process tracing, assess how evidence changes confidence in a causal hypothesis. Where numerical updating is used, a posterior probability is conditional on the prior and on the diagnostic properties assigned to each test. A policy brief should state that condition and link to the underlying analysis.
Participatory methods (MSC and Outcome Harvesting) produce accounts of change identified by participants or programme actors and examined through stated substantiation procedures. The product should identify whose accounts were included, what corroboration was obtained, and the boundary between reported change and causal attribution.
Simulation methods (ABM and system dynamics) can produce model-based scenarios of how a system behaves under stated assumptions. The resulting claim is conditional on model structure, inputs, calibration where relevant, and the scenario definition. It is not an empirical estimate of programme effect without independent empirical support.
19.2 Mapping evidential products to EIPM outcomes
The EIPM outcome domains describe possible routes through which evidence enters policy; they do not permit a fixed ranking of method outputs. The relevant question is which decision a finding can inform and what the decision-maker would need to know to use it.
The four EIPM outcome domains — conceptual, attitudinal, procedural, content — are defined in Chapter 17. Each domain represents a different pathway through which evidence changes policy and practice.
An evidence product should state the finding, the causal claim it supports, the decision to which it is relevant, and the limits on use. A counterfactual finding may bear on whether to continue an intervention for the studied unit. A configurational finding may inform conditions to examine before adapting implementation. A mechanism account may identify an assumption requiring attention. These are possible uses, not a hierarchy of evidence products.
Communication can affect whether a reader understands an unfamiliar logic and can assess its relevance to a decision (Bowen & Zwi, 2005; Nutley et al., 2007; Weiss, 1979). It cannot make a weak claim strong. The translation strategies below therefore begin with the evidential boundary rather than the preferred policy message.
19.3 Translation strategies
Four strategies help evaluators and intermediaries communicate small-n findings without distorting their inferential content.
Strategy 1: Name the causal logic explicitly. Every evidence product should state which logic produced it and what kind of claim it supports. A diagnostic brief should identify the hypothesis, the observations assessed, and whether numerical updating was used. This alerts the reader that the finding is not a statistical effect estimate and provides a vocabulary for appraising it on its own terms. The eight-logics framework in Chapter 4 supplies this vocabulary.
Strategy 2: Translate the finding into a policy-relevant claim. A technical statement should be accompanied by a plain-language account of what it permits a decision-maker to consider. The translation must retain the type of claim: a sufficiency claim remains a sufficiency claim, and a conditional model result remains conditional.
Alamos-Concha et al. (2026) provide a documented example in an evaluation of training transfer in Flemish small and medium-sized enterprises. Their sequential QCA–process-tracing design separated two questions: the configurations associated with training transfer and the causal processes operating within selected cases. For a commissioner or policy user, the useful translation is therefore not a single statement that the training worked. It is an account of the configurations examined, the within-case mechanism evidence, and the settings to which that account may plausibly travel. The case also shows why a brief should link to its technical analysis: the basis for selecting cases for process tracing is part of the claim.
Translation rule. A QCA or CNA conclusion should retain its condition set, outcome definition, case universe, and calibration basis. A sufficiency claim must not be restated as a probability or effect size.
The translation must preserve the logical structure (Cartwright & Hardie, 2012). A sufficiency claim should not be restated as a probability or an effect size. A mechanism narrative should not be reduced to a single “bottom line.”
Strategy 3: Report the evidential boundaries. Every small-n finding has boundaries that the communication should make explicit. A SCED effect estimate applies to the units studied, not to a population. A QCA solution applies to the cases analysed and is conditional on the calibration decisions. A posterior probability is conditional on the prior and the probability estimates for each test. Reporting these boundaries is not a weakness disclosure; it is a specification of what the finding does and does not establish. Policymakers who understand the boundaries can use the finding appropriately rather than either over-generalising or dismissing it.
Strategy 4: Match the communication format to the audience. Different actors may need different levels of detail and different routes to the underlying analysis (Oliver et al., 2014; Patton, 2008). Intermediaries can help tailor evidence products for specific audiences (Lomas, 2007; Ward et al., 2009). A two-page evidence brief for a senior policymaker can lead with the policy-relevant claim and the logic that supports it, with technical detail in an annex. A technical appendix for evaluation specialists should report the full analytical procedure, including sensitivity analysis and robustness checks. A presentation to a programme management committee may use a phase-comparison graph, a display of cases covered by a configuration, or a diagram separating a proposed pathway from the evidence bearing on it.
19.3.1 The structure of a policy brief
A brief should begin with the decision to which the evidence is relevant, followed by the finding in plain language. The next sentence should name the causal logic and the evidence that warrants the claim. A short limitation statement belongs in the main text, not only in technical documentation. Readers need to know whether the conclusion concerns a particular unit, a defined case set, a proposed mechanism, participant-reported change, or a modelled scenario before they decide how much weight to give it.
The brief should distinguish a finding from a recommendation. An evaluation may establish that an outcome changed under a specified design, that a mechanism is supported by particular evidence, or that a configuration was associated with an outcome in the analysed cases. A recommendation additionally requires a judgment about costs, feasibility, equity, political authority, and acceptable risk. Those judgments may draw on the evaluation, but they are not produced by the causal method. Separating them prevents an evidence brief from presenting a policy preference as an empirical conclusion.
Technical material should remain accessible. A brief can link to an appendix that reports case selection, measurement, calibration, diagnostic tests, sensitivity analysis, source limitations, and amendments to the protocol. The link does not absolve the brief of explaining the central warrant. It allows the audience that needs to inspect the analysis to do so without turning the main product into a methods report.
19.3.2 Translation as a two-way process
Translation is not completed when a report is simplified. Programme managers, policy staff, intermediaries, and affected groups may identify a decision constraint, alternative explanation, or implementation detail that the evaluation did not address. These responses can improve the interpretation of findings if they are recorded and assessed rather than treated as confirmation. They can also identify a subsequent question for evaluation.
Engagement with users may clarify the decision context and identify missing evidence. It should not permit a preferred interpretation to displace the evaluation’s stated claim boundary. Where users disagree with a conclusion, the evidence brief should show the basis for the conclusion and the uncertainty that remains.
The report should identify the causal hypothesis, the diagnostic tests, the assumptions used to assess each test, and the resulting uncertainty. It should distinguish the evidence that supports implementation, the evidence that bears on a mechanism, and the evidence that rules out alternatives.
Technical report: State the prior, the diagnostic properties assigned to each test, the sequence of updating, and sensitivity analyses. Link the conclusion to the source material and explain any dependence among tests.
Evidence brief: State the conclusion in plain language, name the main evidence types, and identify the assumptions or alternatives that qualify the claim. Link to the technical report for the full assessment.
Verbal briefing: State the conclusion, the reason for confidence, and the main limit in one short sequence. Avoid converting a conditional diagnostic conclusion into an unconditional statement of fact.
19.4 Using findings in decisions
Small-n findings commonly inform three different decisions. A continuation decision asks whether a programme or component should be retained under conditions similar to those studied. An adaptation decision asks what needs to change in delivery, context, or monitoring. A learning decision asks what uncertainty should shape the next design or evaluation. The evidence brief should name the decision type. This stops a mechanism account from being used as if it were an effect estimate and stops an effect estimate from being used as if it explained how implementation should change.
The decision may require information beyond the evaluation. A conclusion that a pathway was supported does not establish that it is affordable to retain or scale. A configuration associated with success may identify conditions that are administratively or politically impossible to reproduce. An outcome account may identify a change valued by participants while leaving its distributional consequences unresolved. The brief should identify these additional judgments so that the evaluation is used as evidence within a decision, rather than as a substitute for the decision.
Translation products should locate uncertainty in terms that matter for action. They can identify whether uncertainty concerns the causal claim, transfer to another setting, the feasibility of an adaptation, or the values at stake in a recommendation. Each may require a different response: further evidence collection, a bounded pilot, an implementation review, or an explicit policy judgment. A generic recommendation to obtain more evidence is insufficient when the source of uncertainty has not been identified.
19.4.1 Recording use without overstating influence
Evaluation teams and commissioners may record whether findings were discussed, incorporated into a decision process, or used to change an information system, implementation plan, or subsequent question. These are observations about use. They do not demonstrate that the evaluation caused the decision, since policy choices are also shaped by authority, resources, timing, and competing evidence. A use record should therefore describe the event, the evidence product considered, and the role attributed to it by the relevant decision-makers.
This record can improve later commissioning. It shows whether a brief reached the relevant users in time, whether its claim and limitations were understood, and whether additional evidence would have changed the decision. It also prevents a familiar retrospective claim: that an evaluation was influential because a later policy moved in the same direction as its recommendation.
19.5 Numbers, visualisation, and uncertainty
Numbers can make a finding appear more settled than it is. A posterior probability, consistency score, effect-size estimate, or model projection is useful only with an explanation of what it measures and the assumptions required to interpret it. A policy brief should not place a number in a headline if the rest of the brief does not explain the unit, comparison, calibration, prior, or model scenario on which it depends. The numerical result may be retained in a chart or summary box, with a direct link to the technical account.
Visualisation can clarify the causal structure when it shows evidence rather than decoration. A phase graph can show the timing of an intervention and outcome observations. A theory-of-change diagram can distinguish a proposed pathway from links supported by evidence. A configuration display can show which cases are covered by a solution and which are contradictory or deviant. A model diagram can identify feedback assumptions. Each visual should have a caption that says what it represents and what it does not establish.
Uncertainty should be located, not merely acknowledged. An evidence brief can state whether uncertainty arises because an outcome was measured imprecisely, a rival explanation remains plausible, a mechanism link is poorly observed, a calibration choice changes the result, or the study covers a limited set of cases. These are different limitations. They call for different responses and should not be collapsed into a general caveat at the end of a document.
The form of the recommendation should follow this assessment. A finding with a credible within-unit effect and a known implementation context may support continued delivery while further evidence is collected on transfer. A finding that depends on a disputed mechanism may support a targeted implementation review. A simulation result may support testing a scenario or monitoring a feedback process. The brief should name the action that the evidence can justify, and the action it cannot yet justify.
19.7 What translation should not do
Translation should not remove the distinction between observation and inference. A brief may report that records show an activity occurred, that interviewees described a mechanism, or that a model generated a scenario. It should not silently convert any of these into proof that an intervention caused an outcome. The technical route from observation to conclusion may be shortened for a non-specialist reader, but it should remain visible.
Translation should not use uncertainty selectively. A brief that foregrounds favourable findings while moving limitations to a footnote changes the practical meaning of the evaluation. The same applies when a report gives extensive attention to uncertainty around a negative result but treats a positive result as self-explanatory. The relevant uncertainty should be reported for the claim that matters to the decision, regardless of whether the result is welcome.
Finally, translation should not create false agreement. An evaluation can be useful when it identifies a disputed mechanism, a configuration that does not travel, or an outcome measure that does not capture a change participants regard as material. Those findings may complicate a decision. They are still evidence. A policy-facing product earns trust by making that complication intelligible, not by removing it from view.
19.8 The “hierarchy of evidence” problem
A hierarchy that ranks all methods by design type is not sufficient for the range of causal questions addressed in this book (Guyatt et al., 2008; Petticrew & Roberts, 2006). Clinical evidence hierarchies were developed principally for questions about the average effect of a defined treatment, where randomisation addresses a specific threat of confounding. Development and public-policy evaluations may also ask about mechanisms, configurations, contribution, or emergent outcomes, often for interventions that are not standardised and populations that cannot be treated as interchangeable. A hierarchy used as a universal default can dismiss those questions before the evidence required to answer them has been assessed (Cairney, 2016; Parkhurst, 2017).
The eight-logics framework provides an alternative. Instead of ranking methods by design type, it asks whether the method is appropriate for the causal question being posed and whether it is implemented with the rigour that its own logic demands (Stern et al., 2012). A process trace may be the relevant source of evidence for an explanatory question, while a quasi-experiment may be required for an effect question. Neither conclusion follows from the method label alone. Rigour is a property of the match between question, logic, method, and implementation, not a property of the method alone.
This argument does not claim that all methods are equal or that rigour is relative. It claims that rigour is logic-specific. Chapter 16 sets out the corresponding quality checks, and Chapter 18 places them in the design process. The translation strategies in this chapter show how the resulting evidence can be communicated without changing the claim it supports.
19.9 From production to use
This chapter and the preceding one have addressed two stages in the evidence-to-policy pathway: commissioning (how evidence is specified and procured) and translation (how findings are communicated to policy audiences). The next chapter turns to fragile and conflict-affected states and other constrained environments, where the moderating factors identified by the EIPM framework interact with the methodological constraints of small-n evaluation.