flowchart TD
A["<b>Step 1</b><br/>Question specification<br/><i>Map questions to decision-matrix types;<br/>name expected evidential product</i>"] --> B["<b>Step 2</b><br/>Constraint mapping<br/><i>Document four dimensions of small n;<br/>record data availability</i>"]
B --> C["<b>Step 3</b><br/>Logic identification<br/><i>Identify feasible causal logics;<br/>determine integration design if needed</i>"]
C --> D["<b>Step 4</b><br/>Quality framework specification<br/><i>Set logic-specific quality criteria<br/>in the ToR</i>"]
D --> E["<b>Step 5</b><br/>Evaluator selection<br/><i>Assess methodological capability<br/>for specified logics</i>"]
B -.->|"Revise questions<br/>if infeasible"| A
C -.->|"Adjust constraints<br/>or questions"| B
style A fill:#e8f4e8,stroke:#2d7d2d,color:#1a1a1a
style B fill:#e8eef4,stroke:#2d5a7d,color:#1a1a1a
style C fill:#f4f0e8,stroke:#7d6a2d,color:#1a1a1a
style D fill:#f4e8e8,stroke:#7d2d2d,color:#1a1a1a
style E fill:#f0e8f4,stroke:#5a2d7d,color:#1a1a1a
18 Commissioning Small-n Evaluations
The preceding chapters addressed method selection and integration from the evaluator’s perspective. This chapter considers the decisions made by commissioners before data collection begins. Terms of reference, timing, access arrangements, and the definition of an acceptable conclusion all affect what the evaluation can establish.
The EIPM framework positions commissioning at the intersection of the structures and processes pathway and the capabilities pathway (Sempé et al., 2025). Commissioning processes determine which programmes receive evaluation, which causal questions are asked, and which methods are treated as acceptable. If they are calibrated only to large-n designs, small-scale programmes can be excluded or assessed through designs that cannot support the claims requested. The issue is institutional as well as procedural.
18.1 The commissioning problem
Terms of reference can make four choices before an evaluation team has assessed feasibility. They can frame the question as an effect question, prescribe a named design, define quality through statistical metrics alone, or require one primary numerical finding. Each choice can be appropriate for a programme with a clear intervention contrast, sufficient independent units, and outcome data suited to the design. They become a problem when the requested claim, unit structure, available comparison, and outcome timing have not been examined together.
Small-n settings make this problem visible. A programme may have too few independently treated units, no credible untreated comparator, limited temporal observations, or insufficient diversity across cases. An RCT or difference-in-differences estimator may then be infeasible, or it may answer a different question from the one the commissioner needs answered. Statistical precision is one form of rigour. It does not replace the requirements of configurational, diagnostic, generative, participatory, or simulation-supported inquiry. A commission should specify the claim required and the evidence available before it asks for a particular estimator (OECD-DAC Network on Development Evaluation, 2019; United Nations Evaluation Group, 2018).
18.2 From method prescription to logic specification
The decision matrix in Chapter 14 provides the foundation for a different approach to commissioning. Instead of prescribing a method, the commissioner specifies the causal question and the programme attributes, and the ToR identifies the appropriate causal logic or logics. The evaluator then proposes methods that implement those logics, given the available data.
This shift has three operational implications. First, the ToR should state the evaluation question in terms that map onto the causal tasks in the decision framework: a net effect, contribution, a mechanism, pathway diversity, emergent outcomes, feedback and dynamics, or diagnostic assessment of a disputed claim. A request for “impact” is insufficient until it specifies the outcome, population, time period, and causal claim.
Second, the ToR should characterise evaluability constraints rather than assume a design. The relevant constraints are the four dimensions of small n defined in Chapter 3: independent intervention units, support for a counterfactual comparison, repeated observations over time, and diversity across cases. It should also state access to records and sites, the timing of programme delivery and outcomes, known sources of interference, and ethical limits on the comparison or evidence strategy. This information permits evaluators to judge which logic is feasible.
Third, the ToR should define quality criteria that match the logic, not the tradition. For a counterfactual SCED, quality criteria include the number of phase replications, the stability of the baseline, and the results of randomisation inference. For a QCA, quality criteria include calibration transparency, truth table completeness, and robustness to case exclusion. For a contribution analysis, quality criteria include the specificity of the theory of change, the independence of evidence sources for each mechanism step, and the treatment of rival explanations. The reporting checklist in Chapter 16 provides a starting point for logic-specific quality criteria.
18.3 Commissioning risks
The following risks arise when a commission is designed before the causal question and available evidence have been assessed together.
The first is method displacement: a ToR specifies an estimator before the evaluation question, units, comparison structure, and timing have been assessed. The resulting study can be formally compliant while unable to support the claim requested. Reviews of impact-evaluation approaches argue for selecting designs in relation to the evaluation question and conditions rather than treating one design family as universally appropriate (Bamberger et al., 2012; Stern et al., 2012). Befani & O’Donnell (2016) provides a practical tool for matching questions to feasible methods.
Method displacement. The issue is a mismatch between a prescribed method and the claim, units, evidence, and timing available to the evaluation. Befani & O’Donnell (2016) provides a method-selection tool that begins with those features.
The second is evaluation exclusion. A programme may be treated as unsuitable for evaluation when it cannot support the design expected in an organisation’s guidance. Guidance on broadening impact-evaluation designs was developed partly to address the limits of treating experimental and quasi-experimental approaches as the only credible options (Department for International Development, 2012). A commissioning portfolio that applies this restriction will contain evidence about the programmes that meet it and less structured learning about those that do not. The absence of an effect estimate is not evidence that a smaller programme has no effects or mechanisms.
The third is logic mismatch. A commissioner may ask how a change occurred while specifying a counterfactual design, or ask for a net-effect conclusion while commissioning only a contribution account. The first design may assess whether an outcome changed without explaining the mechanism. The second may examine a plausible pathway without estimating an effect. The decision matrix provides a way to identify these mismatches before contracting.
18.4 A commissioning protocol
The following protocol translates the book’s framework into a commissioning workflow. It does not replace institutional procedures but provides a structured overlay for small-n evaluation design.
The first step is question specification. The commissioner, evaluation team, and relevant programme stakeholders formulate questions that correspond to the causal tasks in the decision framework. Each question should name its expected evidential product: for example, an effect estimate, a contribution account, a tested mechanism proposition, a configuration analysis, a substantiated outcome account, or a model used to examine a stated dynamic.
The second step is constraint mapping. The commissioner documents the four dimensions of small n for the programme: independent intervention units, support for a counterfactual comparison, repeated observations over time, and diversity across cases. The commissioner also records data availability and quality constraints that will affect method feasibility.
The third step is logic identification. Using the decision matrix, the commissioner and evaluation team identify which causal logics are feasible given the evaluation questions and programme constraints. Where multiple logics apply, the team determines whether a sequential, concurrent, or nested integration design is appropriate, drawing on the protocols in Chapter 15.
Step 3 draws directly on the decision matrix in Chapter 14. Step 4 draws on the reporting checklist in Chapter 16. The protocol links commissioning practice to the analytical tools in Parts II and III.
The fourth step is quality framework specification. For each selected logic, the ToR specifies quality criteria drawn from the reporting standards in Chapter 16. The quality criteria replace generic statistical metrics with logic-appropriate indicators of rigour.
The fifth step is evaluator selection. The commissioning organisation assesses whether candidate evaluators have the capabilities to implement the specified logics. Experience in RCTs does not demonstrate competence in Bayesian process tracing or fuzzy-set QCA. Capability assessment is part of the EIPM capabilities pathway: organisations that broaden the pool of evaluators with small-n expertise enlarge the evidence they can commission over time (Patton, 2008).
18.4.1 What the terms of reference should specify
The ToR should make the commissioning decisions available for scrutiny without pre-empting justified methodological judgement by the evaluation team. Table Table 18.1 provides a blank specification. It can be completed at inception and revised through the amendment process when access, implementation, or measurement differs from what was expected. The table does not require the commissioner to decide the final method before procurement. It requires the commissioner to state the causal task and the information on which feasibility will depend.
| ToR element | Information to specify |
|---|---|
| Decision and users | The decision, timing, and users for whom the evaluation is commissioned |
| Causal claim | The effect, configuration, mechanism, diagnostic, participatory, or simulation question to be addressed |
| Intervention, outcome, unit, and period | The condition or programme component, outcome definition, unit to which the claim applies, and period covered |
| Binding constraints | Independent intervention units, counterfactual support, repeated observations, case diversity, and any access or ethical limits |
| Evidence and access | Available records, outcome measures, relevant actors, permissions, source-provenance risks, and anticipated gaps |
| Method and integration role | The proposed method family or families and the inferential role of each component |
| Quality and review | Logic-specific quality criteria, review arrangements, and who may comment on factual accuracy or interpretation |
| Amendment and reporting | Conditions that trigger design revision, required documentation, claim boundaries, and planned decision products |
18.4.2 Adaptive commissioning
The ToR should allow a design to be revised without making revision costless or invisible. An inception phase can test whether the proposed comparison exists, whether outcome records cover the required period, whether implementation has varied as expected, and whether the sources needed for a mechanism inquiry can be accessed independently. The team should report the result of that assessment before moving to the main analysis. If the original claim is no longer feasible, the commissioner can decide whether to alter the claim, invest in additional data, defer the evaluation, or proceed with a narrower purpose.
A documented change is preferable to preserving the original label while changing the design underneath it. For example, a commission initially described as an impact evaluation may end with an implementation account and a tested mechanism proposition if no credible effect comparison can be established. That result may still be useful. The final report should describe the evidence that was obtained, explain why the planned claim changed, and state what the revised design does and does not establish.
Adaptive commissioning also requires a clear division of responsibilities. Commissioners are responsible for access, permissions, decision timing, and the conditions under which the evaluation is undertaken. Evaluators are responsible for explaining how those conditions affect the claim and for proposing defensible amendments. Participants and programme staff may need to be involved where outcome definitions, source access, or interpretation affect the design. The contract should make these roles explicit so that later disagreement about evidence or conclusions is not managed through informal pressure on the evaluation team.
18.5 Contracting for an evaluable design
The contract should protect the design decisions made at inception. It should state the expected access to programme records, sites, and relevant participants; the timetable for permissions and data transfer; the independence of the evaluation team; and the process for approving amendments. A method can be feasible in a proposal and become infeasible when access to the relevant unit, outcome record, or implementation chronology is withheld. That change should trigger a documented review of the claim, not an unreported substitution of a weaker design.
Commissioners should also distinguish between deliverables that document work and deliverables that make claims. An inception report should set out the question, theory of change where relevant, evidence plan, case or unit structure, and claim boundaries. An interim product may report implementation evidence or emerging descriptive patterns without treating them as final causal findings. The final report should explain what was collected, what changed from the inception plan, and which claims the evidence supports. This sequence gives commissioners opportunities to resolve access and scope problems while they remain actionable.
Independence requires more than a statement in the ToR. Commissioners may need access to confidential programme material and may also have a legitimate interest in the findings. The contract should specify who can comment on factual accuracy, who can request clarification, and who has responsibility for the evaluators’ conclusions. It should prevent a factual review from becoming an unrecorded negotiation over interpretation. Where independence cannot be assured, the report should say how that constraint affected the evidence and review process.
18.5.1 Budget and timing
Small-n evaluation is not necessarily inexpensive. Process tracing requires time to reconstruct sequence and examine alternatives. Participatory work requires engagement, safeguarding, and procedures for substantiation. QCA requires careful definition and calibration of conditions. A short commission can be appropriate for a focused question, but it should not promise a broad attribution claim when the time does not permit the required observations.
The commissioning timetable should be aligned with programme decisions and outcome timing. If findings are required before an outcome can reasonably change, the commission may focus on implementation, early mechanisms, or the evidence needed for a later outcome study. If a phased roll-out or an information system is still being designed, the commissioner may be able to preserve a comparison, repeated measurement, or a clear implementation record. These choices are most useful before delivery begins.
Figure 18.1 summarises the five steps as a sequential workflow with feedback loops. The arrows between steps are not strictly linear: constraint mapping may prompt revision of the evaluation questions, and logic identification may reveal that the original question framing is infeasible. The protocol is iterative by design.
TipCommissioning checklist for small-n evaluations
- Are the evaluation questions specified in terms that map onto the decision matrix question types?
- Are the programme’s small-n constraints documented across all four dimensions?
- Does the ToR specify causal logics rather than prescribing a method?
- Are quality criteria matched to the specified logic, not defaulted to statistical conventions?
- Has the evaluation team’s capability to implement the specified logics been assessed?
18.6 Institutional implications
Logic-based commissioning may require changes beyond individual terms of reference. Organisational guidance, review arrangements, and management systems can make it easier or harder to commission designs that do not fit a conventional treatment–comparison template (OECD-DAC Network on Development Evaluation, 2019; United Nations Evaluation Group, 2018). Review panels need expertise appropriate to the methods that an organisation expects to commission. Management systems can record the causal claim and method family used, alongside broad labels such as “qualitative” and “quantitative.” Leeuw & Furubo (2008) describe such arrangements as evaluation systems and emphasise that an organisation’s epistemological orientation shapes the evidence it commissions.
An organisation that recognises only counterfactual estimates as credible impact evidence will be unable to commission some mechanism, configurational, participatory, or simulation-supported questions as causal inquiry. Broadening the repertoire does not require treating all claims as equivalent. It requires matching the standard of appraisal to the claim and method. Commissioner–evaluator interaction during design is associated with later evaluation use in reviews of the utilisation literature (Johnson et al., 2009). That association supports early discussion of the causal question and feasible design; it does not show that a particular ToR format will produce policy use.
These institutional changes address the structures and processes pathway in the EIPM framework. They are slower to achieve than reforming individual ToR documents, but without them, reformed ToR remain anomalies within an institutional environment that defaults to large-n conventions. The next chapter turns from how evaluations are commissioned to how their findings are communicated: the translation challenge that arises when small-n evidence enters policy processes calibrated to statistical summaries.