flowchart LR A["Actors:<br>producers, intermediaries, users"] <--> P["Pathways of change:<br>capabilities; relationships and networks;<br>structures and processes; evidence culture"] P --> O["Outcome domains:<br>conceptual; attitudinal;<br>procedural; content"] M["Moderating factors:<br>political conditions, resources,<br>institutional capacity, norms"] -.-> A M -.-> P M -.-> O
17 Small-n Methods and Evidence-Informed Policymaking
Evaluation methods affect what evidence can be produced, but they are only one part of the evidence-to-policy system. The questions commissioners ask, the data and access available to evaluators, institutional incentives, relationships between researchers and policymakers, and the political setting all shape what is studied and whether the findings are used. A broader methodological repertoire cannot remove these constraints, although it can widen the range of programmes for which evaluators can make and assess causal claims.
This chapter locates the book within a conceptual framework for evidence-informed policymaking (EIPM) developed for the FCDO Research Commissioning Centre (Sempé et al., 2025). The framework distinguishes actors, pathways of change, outcome domains, and moderating factors. It helps clarify where a methods guide might contribute and what such a guide cannot be expected to change on its own.
17.1 The EIPM ecosystem
The framework identifies three broad categories of actors. Evidence producers generate research and evaluation findings. Evidence intermediaries synthesise, translate, or broker knowledge. Evidence users draw on evidence when making or implementing decisions (Court & Young, 2003; Nutley et al., 2007). These roles often overlap. A government evaluation office may commission a study, help produce it, and use its findings; an evaluation team may generate evidence and translate it for a policy audience.
The actors operate through four pathways of change. Capabilities concern the knowledge, skills, and resources needed to produce, appraise, and use evidence. Relationships and networks connect actors across professional and institutional boundaries. Structures and processes include the rules, incentives, routines, and arrangements through which evidence is commissioned and considered. Evidence culture concerns the norms and expectations that shape the place of evidence within an organisation or policy system.
The framework also distinguishes four outcome domains. Conceptual outcomes concern how actors understand a problem. Attitudinal outcomes concern their disposition towards evidence and its use. Procedural outcomes concern changes in how decisions are made. Content outcomes concern changes to the substance of a policy or programme. These domains are analytically distinct, but evidence need not move through them in a fixed order. A study may alter how a problem is understood without changing policy content, or it may influence a procedure only after being taken up through an established network.
Political conditions, resources, institutional capacity, socio-cultural norms, and the wider governance environment moderate these relationships (Cairney, 2016; Parkhurst, 2017). They affect which questions can be asked, which evidence can be produced, whose knowledge is considered credible, and whether findings enter a decision process. Figure 17.1 therefore presents the framework as an ecosystem rather than a sequence of steps.
The distinction between pathways and outcomes is particularly relevant to evaluation methods. A methodological guide may contribute to a capability by giving an evaluator or commissioner a clearer way to define a causal claim. It may contribute to a procedure if an organisation changes how it commissions or reviews evaluations. Neither change demonstrates that a programme has been altered or that a policy decision was improved. Those are content outcomes that depend on the evidence, the decision process, and the conditions in which the decision is made.
17.2 Where methodology enters
Methodological capability matters because an evaluation team must be able to identify a defensible claim, select a design suited to it, and explain the limits of the resulting inference, and small-n settings can make this harder when familiar large-sample designs are poorly matched to the programme. This does not make methods the principal constraint on evidence production: a suitable design may exist while access, data quality, time, ethics, or political conditions make it unusable, and commissioners may favour a familiar design even where another would serve the question better.
The possible contribution of small-n methods is clearest in the capabilities pathway. Producers may be better able to formulate and investigate causal claims when they can draw on a wider set of inferential operations; commissioners and intermediaries may be better able to appraise an evaluation when its assumptions and evidential requirements are stated explicitly. Structures and processes also matter: terms of reference, procurement rules, access agreements, reporting standards, and review procedures can enable or restrict the use of an appropriate method. Relationships and evidence culture are not downstream effects of better methods; they influence the questions that reach evaluators, the forms of knowledge admitted into an evaluation, and whether methodological capability is used at all.
17.3 How this book may contribute
The book’s most direct contribution is to methodological capability. The causal reasoning framework in Chapter 4 gives producers, commissioners, and intermediaries a vocabulary for distinguishing the claim being made, the inferential operation used to examine it, and the evidence on which the inference depends. The method chapters then show how different designs perform those operations under small-n conditions. This may improve design and appraisal, but whether it changes evaluation practice is an empirical question.
The guidance on method selection, integration, and reporting may also contribute to structures and processes. It could, for example, support more explicit terms of reference, make design choices easier to scrutinise, or provide a basis for reviewing claims that do not rely on conventional statistical metrics. These effects depend on commissioners and evaluation organisations adopting the guidance and adapting it to their procedures. They do not follow from publication of the book itself.
Any contribution to EIPM outcomes is less direct still. Clearer causal reasoning may change how an actor understands a programme, and a well-designed evaluation may eventually inform a policy decision, but between those points lie interpretation, contestation, institutional routines, and competing forms of knowledge that lie outside the book’s scope.
Table Table 17.1 summarises the mapping. The table is not a results chain. It identifies where the book can plausibly contribute and where additional conditions are required.
| EIPM pathway or outcome | Potential contribution of the handbook | Conditions outside the handbook |
|---|---|---|
| Capabilities | A shared vocabulary for claims, evidence, assumptions, and limits | Time, training, data access, and methodological support |
| Structures and processes | More explicit terms of reference, design review, and reporting requirements | Organisational adoption, procurement rules, and feasible timelines |
| Relationships and networks | Clearer discussion between commissioners, evaluators, intermediaries, and users about what an evaluation can establish | Trust, access, incentives, and opportunities for deliberation |
| Evidence culture | Greater scrutiny of causal language and evidential warrant | Leadership, norms, and tolerance for qualified findings |
| EIPM outcomes | Findings that can be interpreted and used with their limitations visible | Relevance to a decision, political and institutional conditions, and competing knowledge |
The final row is deliberately conditional. A method guide does not produce an EIPM outcome by itself. It may improve the quality and clarity of evidence entering a decision process. Whether that evidence changes understanding, procedures, or policy content is a separate empirical question.
17.4 Limits of the EIPM mapping
The mapping should not be read as a claim that more rigorous evaluation always produces more useful evidence. Rigour concerns the relationship between a claim and its warrant. Usefulness also depends on the question asked, the timing of the evidence, the decision under consideration, and the interests at stake. An impeccably designed study may arrive after a decision has been taken or address an outcome that matters little to those responsible for implementation. A less conclusive account may still identify a problem that requires attention. The two considerations should be discussed together, but they are not the same.
Nor can an EIPM framework decide whose objectives an evaluation should serve. Commissioners may define a question around a programme’s intended outcome, while participants identify a different consequence as more important. Government records may permit comparison on one outcome and omit another. The response is not to assume that one source is neutral or that all perspectives provide the same warrant for every claim. It is to state how outcomes were selected, whose evidence is used, and what each source can establish. Participatory approaches in Chapter 12 take up this issue directly; the point applies to all methods.
There is also a risk in treating policy use as the only indication that evaluation has mattered. Evidence may inform a decision without appearing in a formal policy document. It may prevent an unsupported claim from being used to justify expansion. It may clarify that an effect has not been established and prompt further data collection. Conversely, a finding may be cited in a policy decision while its limitations are ignored. The EIPM outcome domains help distinguish these possibilities, but they do not supply a simple measure of appropriate use.
For this reason, the book uses the EIPM framework as a contextual map, not as a causal theory of the handbook’s influence. The later chapters can identify how methods may be commissioned, conducted, translated, and synthesised more responsibly. They cannot show, by virtue of that guidance, that an organisation will adopt it or that policy will improve. Those claims would require evidence on organisational practice and decision-making.
The remaining chapters in this part turn from the methodological framework to practice. Chapter 18 examines how evaluations are commissioned, Chapter 19 how their findings are communicated to policy audiences, and Chapter 20 how designs are adapted in fragile, conflict-affected, or otherwise constrained settings.