flowchart TD Q["What is the primary<br>evaluation question?"] Q --> |"Effect: did it<br>cause the outcome?"| RE["Repeated measures<br>available?"] Q --> |"Contribution or mechanism:<br>through what process?"| CA["Theory of change<br>specified?"] Q --> |"Mechanism: how did<br>it produce the outcome?"| ME["Within-case depth<br>or cross-site breadth?"] Q --> |"Configuration: which<br>conditions produce<br>the outcome?"| PD["Sufficient variation in<br>conditions and outcomes?"] Q --> |"Accounts and unexpected<br>outcomes"| EO["Participatory methods<br>and verification"] Q --> |"Feedback or<br>non-linearity"| SD["Supporting simulation<br>alongside an<br>empirical method"] RE --> |Yes| SCED["Counterfactual designs<br>SCED, randomisation<br>inference"] RE --> |"No, but comparator<br>exists"| SC["Counterfactual designs<br>Synthetic control,<br>permutation tests"] RE --> |No| SWITCH["Consider contribution<br>or mechanism evidence"] CA --> |"Yes, with<br>rich data"| CAPT["Contribution analysis<br>+ process tracing"] CA --> |"Yes, limited<br>data"| CAMIN["Contribution analysis<br>with a minimal ToC"] CA --> |No| TOCF["Develop ToC first,<br>then re-enter"] ME --> |"Single case,<br>deep data"| PT["Process tracing with<br>diagnostic updating"] ME --> |"Multiple sites,<br>heterogeneous contexts"| REAL["Realist evaluation<br>(CMO propositions)"] PD --> |"Yes; transparent calibration<br>and case knowledge"| QCA["Configurational analysis<br>QCA or CNA"] PD --> |"No"| NOTFEAS["Do not use a truth-table<br>method; consider mechanism<br>evidence instead"] EO --> |"Known outcomes"| OH["Outcome Harvesting"] EO --> |"Unknown outcomes"| MSC["Most Significant<br>Change"]
14 A Decision Framework for Method Selection
Part II distinguishes three forms of causal claim–effects, configurations, and mechanisms–and the operations that can support them. The practical question is how to select and combine those operations for a particular evaluation. Without an explicit selection procedure, the evaluator’s training can determine the method before the causal question has been specified. A transparent procedure also lets commissioners examine the basis for a design choice rather than accept professional convention (Sempé et al., 2025).
14.1 Existing selection tools
Two existing frameworks offer partial guidance. Fox et al. (2022) organises methods into two families: “effects of causes” (counterfactual designs) and “causes of effects” (theory-based approaches). This distinction is useful as a first cut, but it groups different forms of causal claim under the same heading. Process tracing, contribution analysis, and QCA all fall under “causes of effects,” yet they use different forms of evidence and answer different questions. White & Phillips (2012) takes a different approach, distinguishing quantitative approaches adaptable to small samples from qualitative and theory-based approaches. The 3ie framework also maps sources of bias and proposes common steps for causal analysis. Both tools advance the field. Neither links the full range of evaluation questions to the distinctions used in this book.
We start from the evaluation question rather than from the method. The analyst specifies whether the claim concerns an effect, configurations, or a mechanism; identifies the unit or units in which it can be observed; and examines the variation, evidence, and plausible alternatives available. Those choices produce a shortlist of methods and evidential operations. They do not produce an automatic answer.
14.2 A claim-based decision matrix
Table 14.1 maps six types of evaluation question against four programme attributes. Each cell identifies methods and supporting operations that may be suitable. The matrix is not prescriptive. Case number is one consideration, but method choice also depends on the claim, the available variation, the quality and timing of evidence, the plausibility of alternatives, and the decision to be informed. Many evaluations involve more than one question type, and the next chapter discusses how to combine their evidential roles within a single design.
| Evaluation question | One or a few cases with rich evidence | One or a few cases with limited evidence | Several diverse cases | Repeated observations on the same unit(s) |
|---|---|---|---|---|
| Effect: did it cause the outcome? | Synthetic control only where a defensible donor pool exists; otherwise assess whether a counterfactual is possible | Do not infer an effect without a comparator, assignment rule, or repeated outcome evidence | Comparative designs or replicated SCEDs where their assumptions hold | SCED, including reversal or multiple-baseline designs, where phase structure and measurement support them |
| Contribution or mechanism: did it contribute, and through what process? | Contribution analysis; process tracing; diagnostic updating where tests are specified | Contribution analysis with a stated theory of change; targeted process evidence | Contribution analysis across sites; realist evaluation where CMO propositions can be examined | Process tracing or contribution analysis where a defensible chronology and process evidence exist |
| Configuration: which condition combinations produce the outcome? | Usually insufficient variation; trace mechanisms within cases instead | Usually insufficient variation | QCA or CNA only if the cases and condition measurement permit a truth-table analysis | Temporal QCA only where the time structure and independence assumptions are defensible |
| Accounts and unexpected outcomes: what changes are reported and which require further investigation? | Outcome harvesting or most significant change, with a verification plan | Most significant change |
Outcome harvesting across sites |
MSC with longitudinal story collection |
| Feedback or non-linearity: what does a formalised causal theory imply? | Model a bounded theory of change alongside an empirical method | Develop a qualitative causal-loop account before deciding whether simulation is warranted | Model heterogeneity only where agent rules and evidence justify it | Calibrate dynamic relations where suitable time-series evidence exist |
The matrix is not prescriptive. Most evaluations require more than one question and therefore more than one component. See Chapter 15 for protocols on combining components within a single design. A concise version for commissioning and inception work is in the method-selection aid.
Three features of the matrix need emphasis. First, counterfactual designs are not limited to large samples. SCEDs and randomisation inference can be useful with few units, but only when the design provides a credible contrast: repeated measurement, a comparator, or a known assignment process. Case number alone does not supply that contrast. Second, configurational analysis has an evidential lower bound. QCA and CNA require variation in both conditions and outcomes. A sparse truth table does not become informative because it has been coded carefully.
Configurational evidence. There is no universal minimum number of cases. The relevant question is whether the available cases can distinguish the proposed configurations, retain meaningful diversity after calibration, and support an analysis that can be challenged by contradictory cases. A small-n configurational design should make that judgment explicit.
Third, simulation appears in the bottom row as a supporting tool rather than a causal warrant. It enters the matrix when the question involves feedback or non-linearity, and it operates alongside an empirical method.
Figure 14.1 presents a simplified flowchart version of the decision matrix, guiding the evaluator from question type through programme attributes to candidate methods.
Each endpoint in this flowchart maps to a Part II methods chapter: Chapter 6 through Chapter 13. The flowchart simplifies; most real evaluations combine methods with distinct evidential roles.
14.3 Worked examples
Three published evaluations illustrate how the matrix maps real evaluation problems to logics and methods.
Example A: Gender-responsive budgeting in Kabale District, Uganda. Bamanyaki & Holvoet (2016) evaluated a civil society initiative that advocated for gender-responsive budgeting to improve maternal health service delivery in a single Ugandan district.
Bamanyaki & Holvoet (2016) combines contribution analysis with process tracing and Bayesian tests in a single-district governance evaluation.
There was one treated unit (the district), rich qualitative data (advocacy records, budget documents, key informant interviews with local officials and health workers), and a complex causal chain from civil society advocacy through budget allocation to service delivery. The evaluation question spanned contribution and mechanism. Entering the matrix at “contribution, few units, rich data” points to contribution analysis and process tracing. Entering at “mechanism, few units, rich data” points to process tracing with Bayesian updating. The evaluators used exactly this combination: they mapped the programme theory linking advocacy to maternal health outcomes, identified the causal mechanism, derived observable predictions, and tested the evidence using Bayesian logic (hoop and smoking-gun tests). The evaluation design combined three logics: generative (contribution analysis mapped the theory of change), diagnostic (Bayesian tests assessed the strength of evidence for each mechanism step), and the process-tracing method bridged both. Persson et al. (2013), analysing anti-corruption reforms as a collective action problem, illustrate the same structural challenge: complex governance interventions with a single treated unit that demand theory-based causal reasoning rather than cross-case comparison.
Example B: NHS modernisation in London. Greenhalgh et al. (2009) evaluated the NHS Modernisation Agency’s “whole-scale transformation” programme across four health care organisations and three clinical areas. The questions concerned mechanism and contextual variation: how did the programme operate, and why did its operation differ across settings? The evaluation developed initial context–mechanism–outcome propositions, examined them against evidence from each site, and refined them iteratively. This is a realist design for examining mechanisms across heterogeneous settings. It does not itself establish set-theoretic sufficiency; a configurational question would require a separate assessment of case and condition variation. Blackman et al. (2011) illustrates a related systems-oriented approach to public-health evaluation across English local authorities.
Example C: Writing intervention for struggling first-graders in Spain. Arrimada et al. (2018) followed 179 first-grade pupils with twice-weekly writing probes during 120 days of classroom instruction. The authors selected twelve pupils with low composition performance and poor handwriting for a further 72 days of parent-supervised Tier 2 tasks. Ten pupils improved in composition quality, and eight performed within the normal range relative to peers at the end of the intervention. The study illustrates the value of repeated measurement and of examining a maturational explanation through a comparison group and a non-verbal measure. It was not a standard staggered multiple-baseline design: the Tier 2 phase began as a common change for the selected pupils. The matrix therefore identifies the evidence that would be needed for a stronger counterfactual claim, rather than assigning the study to a design it did not use. Horner et al. (2005) set out the methodological standards that a multiple-baseline design would need to meet in an educational setting.
14.4 Making and recording the selection
The matrix is best used as a sequence of design decisions, not as a lookup table. A request for an impact evaluation does not establish that an effect can be estimated. The evaluator first states the proposed causal claim in a form that could be wrong: for example, that a programme changed a named outcome for a defined population over a stated period. The next question is what contrast could distinguish that claim from plausible alternatives. Where neither a comparison nor a repeated outcome series is available, the appropriate design may be contribution or mechanism-focused rather than a claim about net effect.
The unit of analysis also changes the feasible design. A programme may operate in one district but leave repeated observations of schools, facilities, households, or decisions. That structure can make a within-unit design possible. It does not automatically create independent cases. Conversely, a multi-site programme can have many sites yet little useful variation if implementation was uniform, outcome measures are not comparable, or the sites share the same shock. The design note should state which units provide the evidential leverage and why they can be compared.
Evidence is often available at different points in the causal chain. Administrative records may show delivery and timing; interviews and documents may illuminate decisions and mechanisms; outcome measurement may be sparse or delayed. Those differences should determine the questions that receive the strongest treatment. A contribution analysis can use implementation records and stakeholder accounts to examine whether a proposed pathway is credible. Process tracing can test a disputed step where records, interviews, and timing allow rival accounts to be separated. Neither procedure turns missing outcome evidence into an estimate of net effect.
The same discipline applies to simulation. A causal-loop diagram may help identify feedback, delays, or unintended consequences before a programme is implemented. It cannot establish that the programme produced an observed outcome unless its empirical assumptions are supported independently. When a model is included, the design should identify which empirical logic will test its assumptions or use its results, and what decision the model will inform (Hennessy et al., 2016; Sterman, 2000).
14.4.1 A short design record
Before fieldwork, the evaluator and commissioner should be able to agree a short written record of the selection. It does not replace a protocol. It records the decisions that otherwise tend to be made informally as data arrive.
| Design question | Record before data collection |
|---|---|
| Claim | The outcome, population, period, and causal claim that the study will and will not make |
| Decision | The decision, user, and time by which findings are needed |
| Unit and variation | The unit of analysis; treated and comparison units or phases; relevant heterogeneity; possible interference |
| Evidence | Outcome measures, records, documents, observations, and interviews; their timing, access, and likely quality |
| Alternatives | The main rival explanations and the observations that would weaken each one |
| Method and evidential role | The proposed method or combination of methods, with a reason tied to the preceding rows |
| Boundaries | Claims that the design cannot support, foreseeable ethical constraints, and conditions that would require redesign |
This record makes disagreement visible early. A commissioner may require an answer about effect when implementation is already complete and there is no outcome series or credible comparator. The appropriate response is to identify the resulting limit, not to relabel a theory-based inquiry as an impact estimate. Alternatively, the evaluation may be able to collect a series prospectively, introduce an intervention in phases, or retain a comparison group. Those are programme and commissioning decisions, not analytic adjustments made after the evidence has been collected.
14.4.2 Choosing between plausible approaches
Several approaches can be defensible for the same programme. The choice should follow the claim that carries the decision. A district-wide reform may justify process tracing if the issue is whether a particular administrative decision unlocked implementation. It may justify realist evaluation if the concern is how delivery differed across organisations. It may require both if the decision concerns a disputed pathway and the conditions under which that pathway operates. Combining methods is useful when each has a distinct evidential job; it is not useful when several methods repeat the same unexamined assumptions.
Method selection also has resource implications. Process tracing requires time to reconstruct chronology, identify alternatives, and obtain records that can discriminate between them. Realist evaluation requires sufficient access across contexts to refine rather than merely list context–mechanism–outcome propositions. Participatory approaches require time, safeguards, and a clear account of who has authority to interpret or contest findings. These requirements should be costed at commissioning. A design that cannot support its own evidence collection should be narrowed before fieldwork rather than repaired through a more elaborate analysis.
An independent review should test the selection against the initial claim. It should ask whether the proposed method has an appropriate unit of analysis; whether its contrast, process observations, or configurations can address the stated alternatives; and whether the intended conclusion exceeds the evidence. This is particularly useful when a familiar method has been specified before the evaluation question. It protects evaluators who need to say that the available evidence supports a narrower but still useful claim.
14.5 Feasibility is part of the design
Feasibility should be assessed before a method is chosen, not treated as a later operational problem. Access to sites, records, and participants determines what observations can be made and who can contest their interpretation. A plan for process tracing that depends on confidential correspondence may fail if access is not negotiated before key actors leave. A QCA that treats sites as comparable may fail if administrative indicators were assembled under different rules. A participatory approach may be inappropriate if participants cannot safely discuss the programme or if the organisation has not agreed how disagreement will be handled. These are design constraints because they alter the evidence available for a causal claim.
Timing is equally consequential. Some claims require evidence while a programme is being introduced: the sequence of decisions, implementation choices, and early adaptations may no longer be recoverable later. Other claims require time for an outcome to occur. A commission that asks for attribution immediately after delivery may be able to examine implementation and early mechanisms, but not a final outcome that has not had time to change. The selection record should distinguish the decision deadline from the time required for the proposed outcome and causal process to become observable.
Ethics and safeguarding also influence method choice. A withdrawal design may be unacceptable where a service is believed to reduce harm. Randomising an implementation sequence can be reasonable only when the sequence is administratively possible and does not unfairly delay access. Interviews about contested decisions can expose staff or community members to professional or political risk. The protocol should identify these conditions and specify how they will affect the evidence strategy. Ethical constraints do not make causal reasoning optional; they identify which contrasts and observations are unavailable.
14.5.1 Pre-specification and adaptation
The principal causal claim, unit of analysis, outcome definition, and basis for comparison should be recorded before results are examined. This matters in small-n work because there are often many reasonable analytical choices and little redundancy to reveal a discretionary change. Pre-specification is not a demand that fieldwork proceed mechanically. It distinguishes a justified response to new information from a change that follows an apparent result.
An adaptive design should therefore keep an amendment record. It should state what changed, when it changed, why the original plan was no longer suitable, and which conclusions are affected. Suppose interviews identify a mechanism that was absent from the initial theory of change. It may be appropriate to add evidence collection and to treat the mechanism as a revised proposition. It is not appropriate to present that proposition as though it had been the original test. The distinction gives readers a basis for judging what the evidence can bear.
The same applies to calibration and coding choices. A QCA needs transparent membership thresholds and a reason for the conditions included. Process tracing needs a stated account of the observations that would raise or lower confidence in a mechanism. Contribution analysis needs a theory of change that identifies rival explanations rather than merely restating programme activities. Changes to these choices should be explained in the report, with the consequences for interpretation. The aim is not to freeze inquiry; it is to make the development of the inquiry inspectable.
14.5.2 Sensitivity and negative findings
The selection framework should lead directly to a sensitivity plan. For counterfactual designs, the plan may specify alternative outcome windows, the treatment of missing observations, placebo or falsification checks, and the consequences of spillover. For process tracing, it may identify the evidence that would weaken the preferred mechanism and the sources most likely to produce it. For configurational work, it should test whether reasonable calibration choices or case thresholds change the identified configurations. These checks are part of the evidential warrant, not a technical appendix added after a positive finding.
Negative and inconclusive findings also need a place in the design. No observed outcome change may reflect an ineffective intervention, an outcome measured too early, an implementation failure, or a measure unable to register the expected change. A well-chosen logic helps separate these possibilities. Where the evidence cannot separate them, the report should say so. An evaluation is more useful when it identifies the unresolved alternative than when it converts uncertainty into a statement that nothing happened.
14.6 A review sequence for commissioners
Commissioners often receive a proposed method before they receive a fully specified claim. The following sequence can be used to review a brief or inception report.
- What decision will use the findings, and what needs to be known for that decision?
- What causal claim is proposed, for whom, over what period, and at which level of analysis?
- What observations could distinguish the claim from the most plausible alternatives?
- Which observations are genuinely available within the time, access, and ethical constraints?
- Which method addresses those observations, and what conclusion does it permit?
- What would make the proposed design infeasible or require it to be amended?
The sequence may result in a narrower question than the commissioning brief originally anticipated. That is often preferable to commissioning a method that cannot address the brief and reporting a stronger conclusion than the evidence supports. It may also reveal an opportunity to improve the programme’s information system or rollout plan so that a later evaluation can answer a question that is currently out of reach.
14.7 Common selection errors
The first error is to choose a method because it is familiar. A team trained in experimental methods may treat every question as a search for a comparator, even when the central uncertainty concerns implementation or a disputed mechanism. A team trained in theory-based evaluation may reconstruct a credible pathway without asking whether a feasible design could estimate an effect. Familiarity is not disqualifying, but it is not a design justification. The proposed method should be compared with the causal claim and the evidence it requires.
The second error is to treat a theory of change as evidence. A theory of change is necessary for contribution analysis, realist evaluation, and much process tracing because it makes the proposed causal sequence explicit. It does not show that the sequence occurred. The design must specify observations that would distinguish programme activity from the mechanism through which it was expected to matter, and from alternative explanations for the observed outcome.
The third is to use a label in place of an evidential strategy. Calling a study mixed methods does not explain how its sources will bear on the same claim. Calling it participatory does not establish how accounts will be substantiated or how unequal authority will be handled. Calling it a systems approach does not identify a feedback structure that can be examined. The selection record should name the observations, comparison, or test that makes the label consequential.
Finally, evaluators sometimes respond to a constrained commission by adding methods until the proposal appears comprehensive. A design with interviews, workshops, a survey, and a model can still leave the causal question unanswered if the sources are not connected to alternatives or if the work cannot be completed to an adequate standard. A smaller design with a stated claim, a credible source of leverage, and explicit limits is usually easier to scrutinise and more useful to the decision-maker.
14.8 When to combine methods
The worked examples illustrate a recurring pattern. Most realistic evaluations involve more than one question, and no single method answers all of them. The decision matrix helps identify the methods and operations required; it does not specify how to combine them. The integration plan should state the order in which each component will be used, which finding will be carried from one component to another, and how disagreement between components will be interpreted. A contribution analysis may identify the steps that process tracing will test; repeated outcome evidence may identify an unexpected change that requires a mechanism-focused inquiry; participatory accounts may identify outcomes that require independent substantiation. Sequential, concurrent, and nested designs have different coordination requirements. The next chapter addresses these protocols directly.