16 Bias, Transparency, and Quality Standards
Every evaluation method is vulnerable to bias. Chapter 15 considered how multiple logics can be combined; this chapter examines the safeguards that apply to each logic and to their integration. Larger samples can reduce random error. They do not correct a systematically biased measure, selective case access, or an unsupported causal assumption. In small-n evaluation, a limited number of cases and evidence sources often leaves less redundancy with which to detect those problems. The task is to identify the threat, show how it was investigated, and state what remains unresolved.
16.1 Sources of bias at small n
White & Phillips (2012) identified three families of bias that affect small-n impact evaluations. Each operates differently depending on the logic being used, but all three are present across methods.
Three bias families. Respondent bias concerns accounts provided by informants. Evaluator bias concerns analysis and judgement. Selection bias concerns the cases or observations included. Their consequences depend on the design and the available opportunities for corroboration.
Respondent biases arise when the people providing evidence distort it, whether deliberately or unconsciously. Social desirability bias can lead respondents to report what they think the evaluator wants to hear. Recall bias can affect accounts of past events, especially when participants are asked to reconstruct causal sequences after the fact. Stakeholders may also attribute observed changes to the programme because it is salient or because they have an interest in its continuation. These risks are particularly relevant where stakeholder testimony is a substantial part of the evidence base, including participatory approaches (Chapter 12) and process-tracing inquiries that depend on key informant interviews.
Evaluator biases arise from the analyst’s own reasoning processes. Confirmation bias can lead evaluators to seek and weight evidence that supports prior beliefs about a programme. In Bayesian process tracing (Chapter 11), prior probabilities and assessments of evidence can make those judgements visible, but they can also expose disagreement about them (Befani, 2020). Anchoring can give early evidence undue influence as new information is assessed. Vivid or recent evidence can receive more weight than systematic but less memorable data. These risks require particular attention whenever evaluative judgement is central to the route from evidence to conclusion.
Selection bias arises from the choice of cases, sites, or units. At small n, case selection is an analytical decision rather than a sampling procedure, and it can shape the finding. Selecting on the dependent variable, for example by choosing only cases where a programme appeared to succeed, can inflate the apparent effectiveness of the intervention. Convenience sampling can introduce systematic differences between the studied cases and the population of interest. In configurational methods (Chapter 7), the choice of cases determines the truth table and can affect which solutions the analysis identifies.
16.2 Mitigation strategies
Four strategies can reduce the risk of bias in small-n evaluations. Their value depends on the causal claim and on whether they change the evidence available to assess it.
Structured protocols constrain the evaluator’s discretion at each step. A pre-specified causal claim and theory of change, where relevant, identify the mechanisms to be investigated and the evidence that could support or weaken them. Coding rubrics specify how evidence is classified. Checklists require the evaluator to consider rival explanations systematically rather than selectively.
Pre-specification of causal claims applies beyond experimental designs.
Pre-specification. A time-stamped protocol or inception report can record causal claims, evidence plans, analytical decisions, and the conditions under which they may be amended. The point is an auditable record, not a particular platform.
Before collecting data, the evaluator states the causal claims to be examined and the observations that would support or weaken each one. Pre-specification does not prevent bias. It allows a reader to distinguish planned analysis from an amendment made after findings emerged.
Blinding can separate evidence collection from causal assessment where feasible. In Bayesian process tracing, for example, different team members may gather and code evidence and assess priors or evidence tests. This does not remove judgement, but it can make some decisions less dependent on one person’s preferred conclusion. Blinding is more difficult in participatory methods, where the evaluator is often directly involved in facilitating the evidence-generation process.
Sensitivity analysis varies the analyst’s assumptions and reports whether the conclusions change. The specific form of sensitivity analysis depends on the logic being used, and each method family has developed its own protocols.
In QCA, sensitivity analysis involves testing whether the solution holds when cases are added or removed, when calibration thresholds are shifted (particularly the crossover point), or when conditions are recoded.
QCA calibration and robustness procedures are introduced in Chapter 7. The sensitivity guidance here extends them to a formal quality assurance protocol.
Schneider & Wagemann (2012) stress that calibration choices require substantive justification and transparent robustness assessment. The evaluator should report plausible alternative thresholds, whether they change truth-table membership or solution terms, and the cases responsible for any change. Where a single calibration decision changes the solution, the conclusion depends on a choice that requires explicit defence.
In Bayesian process tracing, sensitivity analysis can vary the prior probability and the sensitivity and false-positive rate estimates for each test, then assess whether the posterior odds support the same conclusion. The analysis should make the alternative inputs visible and report whether reasonable changes alter the conclusion (Befani, 2020). If they do, the causal claim is sensitive to probability judgements that require explicit defence.
In contribution analysis, sensitivity analysis involves considering alternative contribution stories and assessing whether the evidence distinguishes between them. Each causal link in the theory of change should be tested against the most plausible rival explanation for that link, not just the overall alternative hypothesis. If the contribution story rests on a single causal link for which the evidence is ambiguous, the evaluation should report that link as a point of fragility.
In SCED analysis, sensitivity analysis can examine whether the conclusion changes when different appropriate analytical approaches are used, when influential observations are treated according to a stated rule, or when a justified alternative phase specification is considered. The report should state which analytical choices were planned, which were amended, and whether the main conclusion changes (Kratochwill et al., 2013).
If the conclusion is sensitive to reasonable alternative assumptions under any of these protocols, the evaluator should report that fragility rather than suppress it.
16.3 Evidence records and independent challenge
Small-n evaluations benefit from an evidence record that links claims to their sources. The record need not be elaborate. For each substantive causal claim, it should identify the source, when the observation was made, the relevant unit, the method through which it was obtained, and the alternative explanation it bears on. A document may establish the date of a decision but say little about how that decision was implemented. An interview may describe a mechanism but be affected by recall, position, or interest. Recording these distinctions prevents a source from carrying more weight than it can support.
An audit trail should also record coding decisions, excluded cases or observations, amendments to the protocol, and unsuccessful attempts to obtain evidence. These are not administrative details. A missing record can itself affect the causal account, particularly where the absence is related to organisational incentives or political sensitivity. The report need not disclose confidential material, but it should explain what evidence could not be inspected and the implication for the conclusion.
Independent challenge is useful when an evaluation depends heavily on expert judgement. A reviewer can inspect whether the initial theory of change contains genuine alternatives, whether process-tracing tests distinguish them, whether QCA calibration follows the stated substantive anchors, or whether phase changes in a SCED were fixed before outcomes were examined. The reviewer need not repeat every analysis. The purpose is to test the route from evidence to conclusion and to identify discretionary decisions that deserve fuller reporting.
16.3.1 Quality assurance during fieldwork
Quality assurance should occur while evidence can still be improved. Teams can review the shared chronology, check whether the planned sources remain accessible, compare coding across a small subset of material, and identify whether the implementation record contains the events needed to assess the theory of change. These checks should be documented separately from substantive interpretation. Otherwise a fieldwork review can become an opportunity to adapt the evidence plan to an emerging preferred conclusion.
The response to a shortfall depends on its nature. If an outcome measure is missing, an effect claim may need to be narrowed. If the original mechanism is not observable, the evaluation may still document implementation but should not claim to have tested that mechanism. If one case becomes inaccessible, the implications for comparison, configurational diversity, or transfer should be assessed before replacing it with a more convenient case. Explicit decisions at this stage are more credible than a later account that treats the final evidence base as though it had been planned from the outset.
16.4 Logic-specific checks within a common standard
The common safeguards do not remove the need for method-specific quality criteria. Counterfactual designs require an account of the comparison, the assignment or phase structure, outcome measurement, interference, and any concurrent events that could explain a change. A result should not be described as an effect merely because it follows an intervention in time. The report must show why the observed contrast warrants that interpretation.
Configurational analysis requires a transparent case universe, a defensible calibration strategy, and close attention to limited diversity and contradictory configurations. Researchers should report the cases retained in the analysis, the condition definitions, truth-table decisions, and the robustness of the solution. A minimally sufficient solution is not a general account of causation. It describes a relationship within the cases, measures, and calibration choices examined.
Generative approaches require clear propositions about the mechanism, conditions, and expected observations. Realist evaluation should distinguish proposed CMO configurations from configurations supported by the evidence. Contribution analysis should identify alternative explanations for the main links in the theory of change. Process tracing should state why a particular observation is expected under one explanation and not another. In each case, a detailed narrative is not enough: the reader needs to see how the evidence changes confidence in the claim.
Participatory approaches require additional attention to voice, authority, and risk. The report should identify who took part, who did not, how accounts were selected or substantiated, and how disagreement was handled. Participation can improve the relevance of questions and identify changes otherwise missed by administrative data. It does not give stakeholder accounts privileged access to causal attribution. That conclusion still depends on the evidence and reasoning set out in the design.
Simulation requires a record of model purpose, structure, parameterisation, calibration where relevant, validation exercises, and uncertainty. A model may be useful for examining the consequences of stated assumptions even when it cannot estimate an intervention effect. Reports should not use a plausible model output as confirmation that the programme caused an observed outcome. The empirical evidence for the causal links and the modelling evidence should be described separately before they are interpreted together.
16.5 Reporting standards
Several method-specific reporting standards exist. SCRIBE (Single-Case Reporting guideline In BEhavioural interventions) provides a structured checklist for SCED studies (Tate et al., 2016). The CONSORT extension for N-of-1 trials (CENT) adapts the familiar clinical trial reporting framework to single-case designs (Shamseer et al., 2015). RAMESES (Realist And Meta-narrative Evidence Syntheses: Evolving Standards) covers realist evaluation and realist synthesis (Wong et al., 2013). Transparency standards for QCA specify how to report calibration decisions, truth table analyses, and solution terms (Schneider & Wagemann, 2010).
These standards share common elements: they require the evaluator to report design decisions, describe data sources, justify analytical choices, and acknowledge limitations.
Method-specific standards. SCRIBE for SCEDs (Tate et al., 2016). CENT for N-of-1 trials (Shamseer et al., 2015). RAMESES for realist evaluation (Wong et al., 2013). Schneider & Wagemann transparency standards for QCA (Schneider & Wagemann, 2010).
They differ in what they emphasise, reflecting the different logics on which each method rests. A multi-logic evaluation therefore needs to show both how each component met its own standards and how their findings were related in the overall causal account.
Table 16.1 proposes a unified reporting checklist for small-n impact evaluations. The items are organised by evaluation phase and apply regardless of which logic or combination of logics the evaluation uses.
| Phase | Reporting item |
|---|---|
| Design | State the evaluation questions and the logic(s) selected to address each one |
| Justify the choice of logic(s) with reference to programme attributes and data availability | |
| Describe the case/unit selection procedure and the criteria used | |
| Report pre-specified causal claims and analytical protocols, together with any amendments | |
| Data | Describe each data source and its relationship to the causal claim |
| Report potential biases for each data source (respondent, evaluator, selection) | |
| Describe the steps taken to mitigate each identified bias | |
| Analysis | Report the analytical procedure for each logic used |
| Describe calibration, coding, or classification decisions and their justification | |
| Report sensitivity analyses and their results | |
| Interpretation | State the causal claim(s) and the evidential warrant for each |
| Report convergence or divergence across logics, if a multi-logic design was used | |
| Identify limitations that the mitigation strategies did not address | |
| Distinguish between findings that are well-supported and findings that are tentative |
The checklist is not a replacement for method-specific standards. Evaluators using SCEDs should follow SCRIBE in addition to this checklist. Evaluators using realist evaluation should follow RAMESES. The checklist provides a common framework for evaluations that combine methods and helps readers assess the quality of a small-n evaluation across traditions.
16.5.1 Using the checklist
The checklist is useful at inception, during fieldwork, and at reporting. At inception, it can expose a gap between the intended claim and the evidence that will be collected. During fieldwork, it can be used to record changes in access, measurement, or case selection before those changes become invisible in the final report. At reporting, it gives reviewers a way to distinguish an evidence limitation from a limitation that has not been considered.
Completing every item does not make a causal claim credible. The checklist cannot create a comparator where none exists, recover an unobserved mechanism, or resolve a politically constrained evidence base. Its role is narrower: it requires evaluators to show their design decisions, the sources that support their claims, and the threats that remain. A report that identifies a limit clearly can still be useful to a commissioner. A report that conceals the limit cannot be assessed.
The standard should also be proportionate. A short formative inquiry does not require the same documentation as a multi-year, multi-site evaluation. It should nevertheless state its question, evidence base, and claim boundary. Conversely, an evaluation making an effect or attribution claim should provide enough information for another researcher or reviewer to inspect the design, reproduce the main analytical steps where possible, and understand why competing explanations were discounted.
From an EIPM perspective, structured reporting standards can support the institutional conditions for appraising evidence (Sempé et al., 2025). Transparency about the claim, evidence, assumptions, and remaining limits gives users and intermediaries a basis for deciding how much weight to give a finding. The checklist above provides one instrument for reviewing these features across a multi-logic evaluation. It does not show that a reporting format will produce evidence use or policy change.
The final part of the book shifts from methodology to the policy ecosystem, examining how small-n evaluations are commissioned (Chapter 18), how their findings are translated for policy audiences (Chapter 19), and how methods adapt in constrained settings (Chapter 20).