Method-selection aid for small-n evaluation

Use this aid before commissioning or beginning fieldwork. It does not choose a method mechanically. It records the claim the evaluation needs to make, the evidence available to support it, and the limits that should be stated in the terms of reference or protocol. The fuller discussion is in 14  A Decision Framework for Method Selection.

1. State the decision and the claim

Write the decision that will use the findings, who will make it, and when it must be made. Then specify the causal claim in a form that could be wrong.

Record	Write before selecting a method
Decision	What decision will use the findings, and by when?
Intervention or causal condition	What changed, for whom, and during what period?
Outcome	What outcome is expected to differ, and when could it be observed?
Primary claim	Effect; configuration; mechanism or contribution; stakeholder-informed account of change; or conditional implication of a formal model
Claim boundary	What conclusion is outside the scope of the proposed evaluation?

An effect claim asks whether the outcome would have differed under another intervention state. A configurational claim asks which combinations of conditions are necessary or sufficient in a defined case set. A mechanism claim asks how an intervention contributed to an outcome in a particular context. These claims need different evidence (Three forms of causal claim).

2. Identify the available leverage

The number of programme sites is not enough to determine a design. Record which source of leverage is actually available.

Question	Evidence needed	If it is absent
Is an effect claim required?	A comparator, known assignment process, or repeated outcome series with a defensible phase structure	Do not label a before-and-after description as an effect estimate
Are several diverse cases available?	Comparable outcomes and conditions, transparent calibration, and enough diversity to examine configurations	Do not use QCA or CNA merely because cases can be coded
Is a mechanism or contribution claim required?	A chronology, process evidence, a stated theory, and evidence capable of challenging alternatives	Narrow the claim to implementation or description where alternatives cannot be examined
Are participant accounts needed?	Safe participation, a record of who was not represented, and a plan for corroboration or substantiation	Do not treat agreement or selected stories as causal validation
Is a simulation proposed?	A bounded decision question, defensible model structure and inputs, sensitivity analysis, and an empirical role alongside the model	Do not use scenario output as an estimate of programme effect

3. Match the claim to a method role

Primary question	Candidate approach	Minimum claim boundary
Did the intervention cause an outcome difference?	Counterfactual design, including a SCED or a few-unit comparison where assumptions hold	The estimate depends on the stated comparison and does not establish mechanism or transferability
Which combinations of conditions are associated with the outcome?	QCA or CNA, where case and condition variation permit a truth-table analysis	The result applies to the defined cases and does not show the effect of changing one condition
Did the intervention contribute through a proposed process?	Contribution analysis, process tracing, realist evaluation, and diagnostic updating where evidential tests are specified	The account does not estimate an average effect without a counterfactual design
What changes do participants report, and which require further inquiry?	Most Significant Change or outcome harvesting, with selection and substantiation procedures	Counts and selected accounts do not estimate prevalence or attribution
What follows from a proposed feedback structure or set of rules?	System dynamics or agent-based modelling alongside an empirical method	Scenario results remain conditional on the model and its inputs

Methods can be combined only when each has a separate evidential job. The integration protocol should state what each component will establish, how information will pass between components, and how disagreement will be handled (15  Integrating Methods and Causal Claims).

4. Test feasibility before commitment

Record these points in the terms of reference or inception report.

Design condition	Check
Units and exposure	State the unit of analysis, intervention timing, interference or spillover, and which units supply comparison or variation.
Evidence access	List outcome measures, records, documents, observations, interviews, and their timing, quality, and access restrictions.
Alternatives	Name the main rival explanations and the observations that could weaken each one.
Ethics and safety	State which comparisons, questions, or forms of participation are unavailable or unsafe, and how this changes the claim.
Adaptation	Record the conditions that would require a change to the design and how that change will be documented.
Resources	Cost the evidence collection, checking, analysis, and independent challenge needed for the selected approach.

5. Record the conclusion the design can support

The final protocol should end with a plain statement in this form:

Given the available units, evidence, and constraints, this evaluation will assess [specified claim] for [population/cases] during [period]. It will use [method and evidence] to examine [stated alternatives or conditions]. It will not establish [claim outside the design].

This statement should be revised openly if the intervention, available evidence, or ethical conditions change. It is a safeguard against reporting a stronger conclusion than the design can support.