11 Diagnostic Approaches
The previous chapter showed how generative methods identify the mechanism connecting intervention to outcome. Process tracing, in particular, collects causal-process observations and assesses how they bear on a hypothesised causal chain. Two evaluators may nevertheless assign different weight to the same observation. Diagnostic evaluation makes that judgement explicit. It treats an observation as evidence with a stated likelihood under a causal claim and under relevant alternatives, then uses Bayesian updating to show how those assumptions change confidence in the claim.
The diagnostic logic is not a rival to generative reasoning. Process tracing (Chapter 8) can identify what evidence would be expected if a mechanism operated and what alternatives need examination. The causal-process observations assessed here typically come from process tracing, realist case studies (Chapter 10), or process-oriented contribution analysis (Chapter 9). Bayesian updating provides one formal way to express the diagnostic value assigned to that evidence. The combination is commonly described as Bayesian process tracing. This chapter also considers the difficulty of estimating the probabilities that updating requires, and the limited role simulation can play in generating them.
The bibliometric map (Chapter 5) finds only 29 title-and-abstract pairs naming Bayesian process tracing, the smallest core-method corpus in the book alongside general elimination methodology. A corpus this size supports description, not a stable claim about disciplinary concentration; with that caution, the records skew towards Sociology and Political Science and Political Science and International Relations, and towards United States-based institutions, echoing process tracing’s own profile. Figure 5.5 (Chapter 5) shows a small, irregular year-by-year count consistent with a specialist literature rather than an established subfield. The corpus’s size is itself informative: it is consistent with the causal-legitimacy scrutiny diagnostic approaches attract, discussed in Chapter 4, and with the elicited rather than measured probabilities this chapter turns to next.
11.1 The diagnostic logic
The diagnostic logic evaluates causal claims through the probative value of evidence. Each piece of evidence is treated as a diagnostic test with two properties: sensitivity (the probability of observing the evidence if the causal claim is true) and the false-positive rate (the probability of observing the evidence if the causal claim is false). These two properties determine the likelihood ratio, which measures the evidential weight of the observation.
The Bayesian machinery proceeds in three steps. The evaluator begins with prior odds that the causal claim is true, based on available background knowledge. The evaluator then observes a piece of evidence and computes its likelihood ratio: sensitivity divided by the false-positive rate. The prior odds are multiplied by the likelihood ratio to produce posterior odds. If a second piece of evidence is observed, the posterior odds from the first update become the prior odds for the second, and the process repeats. The result is a transparent, sequential updating of causal confidence as evidence accumulates.
The updating rule can be stated formally. Let \(H\) denote the hypothesis that the intervention caused the outcome, and let \(\neg H\) denote the alternative (the outcome would have occurred without the intervention). The prior odds are \(O(H) = P(H) / P(\neg H)\). For a piece of evidence \(E\), the likelihood ratio is:
\[ LR = \frac{P(E \mid H)}{P(E \mid \neg H)} \tag{11.1}\]
Sensitivity = \(P(E \mid H)\): the probability of observing the evidence if the hypothesis is true. False-positive rate = \(P(E \mid \neg H)\): the probability of observing it if the hypothesis is false. Their ratio is the likelihood ratio.
where \(P(E \mid H)\) is the probability of observing the evidence if the hypothesis is true (sensitivity) and \(P(E \mid \neg H)\) is the probability of observing the evidence if the hypothesis is false (the false-positive rate). The posterior odds after observing \(E\) are:
\[ O(H \mid E) = O(H) \times LR \tag{11.2}\]
When multiple independent pieces of evidence \(E_1, E_2, \ldots, E_k\) are observed, the posterior odds are the product of the prior odds and all likelihood ratios: \(O(H \mid E_1, \ldots, E_k) = O(H) \times LR_1 \times LR_2 \times \cdots \times LR_k\). The posterior probability can be recovered from the posterior odds as \(P(H \mid E) = O(H \mid E) / (1 + O(H \mid E))\).
The relationship to process tracing is direct. The causal-process observations that process tracing collects can be treated as diagnostic tests. A programme document confirming a predicted decision sequence or an interview describing behavioural change may support a hypothesis to different degrees. Neither has a known sensitivity or false-positive rate merely by virtue of its source type. The evaluator must justify the likelihoods assigned, considering provenance, timing, source dependence, and the alternatives that could have produced the same observation.
11.2 Bayesian process tracing in practice
The political science literature formalised four classic process-tracing tests, each defined by an idealised combination of sensitivity and specificity (Collier, 2011; Van Evera, 1997). The straw-in-the-wind test has low sensitivity and low specificity: the evidence is consistent with the hypothesis but also consistent with alternatives. Passing the test slightly increases confidence; failing it slightly decreases confidence. The hoop test has high sensitivity but low specificity: the evidence is expected if the hypothesis is true, but also under many alternatives. Failing a hoop test can substantially weaken the hypothesis; passing it is only mildly confirmatory. The smoking-gun test has low sensitivity but high specificity: the evidence may be absent even when the hypothesis is true, but if found it is difficult to reconcile with alternatives. A doubly decisive test would have both high sensitivity and high specificity. Such evidence is uncommon in evaluation settings, and the label should not be assigned without a case-specific argument.
Table 11.1 maps these four tests to their sensitivity and specificity profiles.
| Test | Sensitivity | Specificity | Passing confirms | Failing disconfirms |
|---|---|---|---|---|
| Straw-in-the-wind | Low | Low | Weakly | Weakly |
| Hoop | High | Low | Weakly | Strongly |
| Smoking gun | Low | High | Strongly | Weakly |
| Doubly decisive | High | High | Strongly | Strongly |
Test mnemonics. Hoop: the hypothesis must jump through it to survive (failing is fatal). Smoking gun: rare but decisive if found. Doubly decisive: confirms if passed, eliminates if failed.
The chapter draws on three linked strands of work. Van Evera (1997) set out the classic distinction among tests with different confirmatory and disconfirmatory force. Collier (2011) clarified how those tests can discipline within-case inference. Befani (2020) and Befani et al. (2021) develop diagnostic evaluation for theory-based evaluation, concentrating on probability elicitation, evidence dependence, and the limited use of simulated probabilities. These sources are not interchangeable: the first two establish a vocabulary for diagnostic evidence; the latter address how evaluators can make that vocabulary explicit in applied work.
In practice, the evaluator maps each CPO to one of these test types, assigns approximate sensitivity and false-positive rate values, and computes the likelihood ratio. The updating is iterative: evidence accumulates across multiple tests, and the posterior odds after the final test represent the overall confidence in the causal claim. The sequence in which evidence is considered does not affect the final posterior odds, provided the tests are independent. Independence is a strong assumption, and its violations are a practical challenge discussed below.
The updating rule is useful only when its inputs are derived from a defined causal claim, a specified rival, and evidence whose diagnostic value has been argued. A real application should report the prior, likelihood judgements, treatment of dependence, and sensitivity analysis together. The calculation does not make a weakly specified hypothesis or a dependent set of documents more informative.
Brandão et al. (2023) applied Bayesian process tracing to six initiatives intended to reduce deforestation in São Félix do Xingu, Brazil, between 2006 and 2019. The study assessed 31 pieces of qualitative and quantitative evidence against nine components of a theory of change, covering protected areas, enforcement and monitoring, credit restrictions, commodity agreements, multi-stakeholder processes and value-chain projects. Its conclusion was differentiated: four theory components were assessed as practically certain, four as reasonably certain or highly confident, and one only cautiously confident. The analysis also pointed to different mechanisms for smallholders and larger landholders, and to complementarities between regulatory and market-based interventions (Brandão et al., 2023).
Befani & Stedman-Bryce (2017) use an Oxfam evaluation of a civil-society Universal Health Care campaign in Ghana to demonstrate contribution tracing and Bayesian updating. The campaign used the 2012 presidential election to advance health-policy priorities and involved more than 500 health-focused organisations. The worked analysis focuses on a narrower claim concerning the campaign’s challenge to reported coverage under the National Health Insurance Scheme; its broader goal of legislation for free universal health care had not been achieved at the time of the evaluation.
The contrast with the Brazilian application is useful. The Ghana case demonstrates how an evaluator can distinguish the probative value of observations within a contested advocacy process. It is not an independent estimate of the campaign’s effect on health policy. The Brazil study applies Bayesian process tracing to a multi-initiative environmental explanation over a longer period. Both cases require claims and alternatives to be specified before the evidence can be assigned diagnostic weight.
Raimondo (2020) reports the use of Bayesian process tracing to examine citizen-engagement interventions within the Dominican Republic’s Progresando con Solidaridad conditional-cash-transfer programme. The study is particularly useful because it addresses a real evaluation undertaken under time, budget, and political constraints. It used the method to unpack and assess a proposed causal mechanism, while discussing the practical difficulty of eliciting and applying probability judgements in an evaluation setting. The application is evidence about the method’s feasibility and use under constraints; it is not a general estimate of the programme’s effect (Raimondo, 2020).
Salazar et al. (2018) evaluated a Rare Pride social-marketing campaign intended to reduce the use of destructive fishing gear in Indonesia, using a qualitative impact-evaluation approach built around General Elimination Methodology (GEM). Rather than assigning a single probability to the campaign’s theory of change, the analysis specified rival explanations for observed behaviour change — including alternative livelihood shifts, enforcement activity, and prior trends — and used interview and documentary evidence to argue for or against each rival in turn.
GEM shares the diagnostic logic of Bayesian process tracing (evidence is assessed for its power to discriminate between the focal explanation and named rivals) without requiring the elicitation of numerical priors and likelihood ratios. The Rare Pride case shows this qualitative variant applied to a conservation-behaviour claim in a single site, complementing the São Félix do Xingu study’s numerically explicit and much larger multi-initiative application above.
Boissat et al. (2021) examine the so-called “Blackfish effect”, the claim that the 2013 documentary Blackfish changed public attitudes and corporate practice toward keeping orcas in captivity. Using a systematic mapping of media, policy, and organisational documents, and drawing on general-elimination reasoning, the authors trace the sequence of events following the documentary’s release and consider rival explanations for the reported changes, including pre-existing advocacy campaigns and prior regulatory pressure.
The example is a useful counterpoint to the Rare Pride case because the intervention here was not designed as part of an evaluated programme: the “treatment” is a media artefact with a diffuse and contested reach, and the rival-elimination task is correspondingly harder to bound. Both conservation examples show diagnostic, elimination-based reasoning applied outside the Bayesian process-tracing applications that dominate the rest of this chapter.
11.3 Reading a diagnostic application
The São Félix do Xingu study shows what a diagnostic application adds to a broad theory of change. The six initiatives were not treated as one undifferentiated intervention. The authors divided their explanation into nine components, assessed distinct evidence for each, and retained different confidence levels across components. This prevents a programme-level conclusion from concealing that some links are better supported than others.
The application also shows why the rival account must be defined at the level of the claim. The explanation of reduced deforestation could not rest on the fact that several conservation initiatives were present. The relevant questions concerned which measures operated at particular times, for which landholders, in an active frontier or a more consolidated area, and in combination with what other measures. The resulting interpretation distinguishes regulatory interventions, market-based approaches, and the populations to which they were directed. These distinctions came from the theory components and evidence assessment, not from a generic claim that the initiatives worked together.
The case should not be read as a formula for assigning a posterior probability to any multi-component programme. It was a substantial study covering a defined municipality, period, intervention set, and evidence base. Its numerical confidence categories depend on its authors’ hypotheses and likelihood assessments. What transfers to other evaluations is the discipline of separating claims, treating evidence as diagnostic of particular components, preserving uncertainty, and examining whether the explanation changes across actors and settings.
The application also provides a useful boundary for the method. Bayesian process tracing did not estimate the average effect of each initiative or calculate the proportion of reduced deforestation attributable to each one. The study assessed the role of interventions in a causal explanation of the observed deforestation trajectory. Where a commissioner instead needs an effect estimate for one intervention relative to a credible alternative, a counterfactual design is required alongside, or in place of, the diagnostic analysis.
11.4 Negative evidence and asymmetric tests
Diagnostic reasoning requires attention to what was not observed as well as what was observed. An expected implementation decision may be absent, a predicted behavioural response may not occur, or an outcome may improve in cases where the programme was not present. These observations can weaken a claim. Their meaning depends on whether the study had a reasonable expectation that the missing trace would exist and be observable. The absence of a confidential record may reveal little if records were never routinely retained.
Tests are often asymmetric. The absence of a smoking-gun observation may not matter much if the observation was unlikely even when the hypothesis was true. The failure of a hoop test can be more serious when the observation should almost always be present under the hypothesis. The evaluator should explain the asymmetry in ordinary language before it is represented numerically. Otherwise, a label such as ``smoking gun’’ can make a missing observation appear either more damaging or less relevant than the evidence warrants.
The treatment of negative evidence also protects against a cumulative-confirmation problem. An evaluation may record several activities, favourable accounts, and outcomes consistent with the programme theory, while failing to examine a missing decision record, a site with no change, or evidence that the outcome began before delivery. These observations may be more diagnostic than the positive material. A credible evidence register includes them and records whether they reduced confidence, led to a revised theory, or could not be interpreted because of a known evidence limitation.
11.5 Specifying hypotheses and diagnostic observations
Bayesian updating is meaningful only when the hypothesis and alternatives are stated clearly. ``The programme contributed to the outcome’’ is often too broad. The evaluator may instead examine whether a named programme activity changed a specified decision, whether that decision produced an intermediate outcome, or whether a particular mechanism operated during a bounded period. Each claim requires an alternative that is more informative than the bare negation of the hypothesis. A rival might be a concurrent reform, an existing organisational practice, a change in resources, selective participation, or altered outcome measurement.
The observation must be defined before its likelihood is assessed. ``Evidence of improved practice’’ combines several possible tests: an updated guideline, a staff account, direct observation, routine data, or an outcome trend. These observations differ in timing, provenance, and what they can distinguish. A dated decision record may bear on whether a manager changed a procedure. It may say little about whether staff implemented it. A later outcome indicator may be compatible with the mechanism but be equally compatible with another intervention. The evidence plan should state the specific observation, its expected timing, the sources that could produce it, and the relevant alternative explanation.
The diagnostic status of an observation is case-specific. A document explicitly linking a decision to an intervention can be highly informative if it is contemporaneous, authored by the decision-maker, and difficult to explain under the rival. It is less informative if it was prepared for a funder after the outcome occurred or repeats a public narrative. An interview account may be valuable if the respondent had direct knowledge and the account can be checked against records. It may add little where several respondents learned the account from the same manager. These distinctions should be made before numerical probabilities are assigned.
11.6 Eliciting and checking probability judgements
Probability elicitation should be structured around comparative questions. The evaluator asks: if the hypothesis were true, how likely is the defined observation? If the specified rival were true, how likely is the same observation? The answers may be expressed first as qualitative judgements and then mapped to numerical ranges under an agreed protocol. The important part is the explanation: why would the observation be expected under one account and unusual under the other?
Elicitation should involve people who understand the programme and the setting, but it should not be controlled by those responsible for the programme’s success. A small group can independently record initial estimates, discuss the evidence and alternatives, then document any revised values and the reason for revision. A designated reviewer can argue the strongest rival account. This process does not eliminate subjectivity. It makes disagreement and institutional incentives visible.
Sensitivity analysis is necessary when values are uncertain. The evaluator can recalculate the posterior using a range of plausible likelihood ratios, alternative priors, or a more conservative treatment of source dependence. If the conclusion remains similar across these choices, the result is less dependent on an arbitrary point estimate. If it changes substantially, the report should say that the evidence does not support a stable numerical conclusion. A sensitivity analysis cannot compensate for a badly specified hypothesis or for evidence that is not diagnostic; it shows the consequences of uncertainty in the stated assumptions.
11.7 Dependence, sequence, and evidence packages
Multiplying likelihood ratios assumes that observations add independent information. This is rarely true for a bundle of programme documents, several interviews within one organisation, or a report and the data source on which it relies. The evaluator should identify the original source of each observation and ask whether one item could reasonably have differed if another had remained the same. If not, the items should be treated as a package or their combined likelihood assessed directly.
Temporal sequence is also part of the assessment. A mechanism claim may predict that an intervention reaches an actor, changes a decision, alters practice, and then contributes to an outcome. Evidence for a later link does not make an earlier missing link irrelevant. Equally, several observations in the sequence may derive from the same implementation record. A diagnostic analysis should record the chronology and source dependence alongside the numerical calculation, not place them in a separate methodological appendix.
Evidence that contradicts the preferred hypothesis belongs in the update. A negative observation may have a likelihood ratio below one, reducing posterior odds. This is a feature of the method. Omitting it because the observation is inconvenient, or redefining the hypothesis after seeing it, produces a higher posterior but a weaker evaluation. Where the observation is ambiguous or of uncertain provenance, the report should explain why it received limited weight rather than ignore it.
11.8 Practical challenges and solutions
Befani (2020) identifies three practical challenges that arise when applying Bayesian updating to theory-based evaluation. The first is probability estimation. The Bayesian machinery requires numerical values for sensitivity and the false-positive rate, but evaluators rarely have empirical frequency data for the causal mechanisms under study. How often does a programme document confirming the predicted decision sequence appear when the hypothesis is true? How often does it appear when the hypothesis is false? These frequencies are seldom known. Befani (2020) proposes two solutions: using qualitative confidence descriptors (“very likely,” “unlikely”) anchored to numerical ranges, and using structured elicitation protocols in which the evaluation team collectively estimates the probabilities.
The second challenge is confirmation bias and conservatism. Evaluation teams that have invested in a theory of change may assign overly high sensitivity values to confirming evidence (they expect to find it because they believe the hypothesis) and overly low false-positive rates (they underestimate the chance of finding it under alternatives). Befani (2020) recommends explicit “red team” exercises in which a team member argues the rival hypothesis and re-estimates the probabilities from that perspective. Conservatism, the opposite problem, arises when evaluators are reluctant to assign extreme probability values, compressing likelihood ratios toward 1 and producing updates that are too small. The solution is to anchor probability estimates in concrete counterfactual reasoning: “if the programme had not been implemented, how likely is it that this document would exist?”
The third challenge is evidence packages. Multiplying separate likelihood ratios assumes that the observations add conditionally independent information. In practice, evidence often comes in bundles. Interview testimony from three programme staff members may all reflect the same institutional narrative rather than providing three independent observations. Befani (2020) proposes setting boundaries on independent observations by asking whether each piece of evidence could, in principle, turn out differently from the others. Evidence that cannot vary independently should be treated as a package and assessed with a combined or conditional likelihood, rather than entered as multiple independent tests.
11.9 Simulated probabilities for diagnostic evaluation
When empirical frequency data are unavailable and subjective elicitation is difficult, Befani et al. (2021) propose a third approach: using computational simulation to generate probability estimates for diagnostic evaluation. The model encodes assumptions about a causal process and is run under alternative parameter values or hypotheses. The distribution of simulated outcomes can then be used to estimate how likely an observation would be under each hypothesis. Those estimates are conditional on the model’s structure and inputs; simulation does not turn an uncertain mechanism into observed empirical evidence.
Befani et al. (2021) provide a proof of concept using a simplified agent-based model of protective behaviour in an influenza pandemic. They use simulated proportions infected under alternative vaccine-efficacy assumptions to derive likelihoods and update confidence in those assumptions. The authors present the model as a pedagogical demonstration. It is not a completed real-world impact evaluation. The approach is warranted only when the model is plausible for the question, assumptions and sensitivity analyses can be inspected, and simulation addresses a defined probability-estimation problem within a wider empirical evaluation. Computational simulation as a broader tool for small-n evaluation is discussed in Chapter 13.
Simulation–diagnostic bridge. When neither empirical frequencies nor expert elicitation can supply the probability estimates that Bayesian updating requires, agent-based models offer a third way. See Chapter 13 for the full treatment.
11.10 Reporting a diagnostic assessment
A diagnostic report should show the causal claim, the alternatives, the prior, each observation, the likelihood judgement, and the resulting update. It should also state whether observations were treated as independent, how source dependence was assessed, and what sensitivity analysis was conducted. Reporting only a final posterior probability prevents readers from identifying which judgement drove the conclusion.
| Component | What should be reported |
|---|---|
| Claim and alternatives | The bounded causal claim and the rival account against which each observation is assessed |
| Prior | The starting odds or probability, its basis, and whether a range was considered |
| Observation | The defined evidence item, date, source, provenance, and relation to the causal process |
| Likelihood judgement | Why the observation is expected under the claim and under the rival, including the numerical range used |
| Dependence and update | Whether observations are independent, how packages were handled, and the posterior calculation |
| Sensitivity | Whether alternative priors or likelihood assignments change the conclusion |
The conclusion should remain intelligible without the arithmetic. It may say that the available evidence favours one causal account over a specified rival, that the result depends on a small number of contested likelihood judgements, or that source dependence prevents a stable update. A posterior probability should never be presented as a measured proportion of causal responsibility. It is a conditional statement: given the claim, alternatives, priors, likelihoods, and treatment of dependence, this is how the evaluator’s confidence changed.
The report should preserve disconfirming evidence and evidence that cannot be scored. An observation may be relevant but too ambiguous to assign a likelihood ratio with confidence. It can be discussed qualitatively and kept separate from the numerical update. This is preferable to imposing a number simply because the method has a formal structure. The boundary between scored and unscored evidence should be stated clearly.
11.11 Commissioning and combining diagnostic evaluation
Diagnostic evaluation should be commissioned for a defined inferential problem, not as a general request to make an evaluation more rigorous. It is appropriate when a theory-based evaluation has a limited set of competing causal explanations, evidence whose diagnostic value can be argued, and a decision that would benefit from making those arguments explicit. It is poorly suited to a broad programme with an undefined theory, extensive missing evidence, or no credible way to distinguish the main alternatives.
The terms of reference should identify who will formulate the hypotheses, who will participate in probability elicitation, how rival explanations will be represented, and who will review the calculation. It should also allow time for evidence planning and sensitivity analysis. A team cannot repair an unstructured set of endline interviews by assigning likelihood ratios after the final report has been drafted.
Diagnostic updating can be combined with process tracing and contribution analysis. Process tracing can generate the causal-process observations and identify rivals at a particular link. Contribution analysis can organise a broader programme theory and show where a focused diagnostic assessment is needed. The methods make different contributions. A posterior update on one link does not establish every link in the programme theory, and a contribution story does not supply the likelihoods needed for a Bayesian calculation.
When the available evidence is weak, the appropriate output may be a plan for further inquiry rather than a numerical conclusion. The evaluation can identify what observation would discriminate between the claim and the rival, what records need to be retained, or which case requires a focused trace. Diagnostic logic then improves design by making the evidential gap precise.
11.12 Priors, thresholds, and decisions
The prior represents the evaluator’s confidence in a claim before considering the evidence selected for the update. It is not a neutral starting point simply because it is expressed as 0.5. An apparently agnostic prior may be inappropriate if existing research, implementation records, or the timing of an outcome make the claim more or less plausible before the new evidence is considered. Conversely, a highly favourable prior based mainly on programme expectations can predetermine the posterior. The report should explain the basis of the prior and distinguish background evidence used for the prior from evidence used as a diagnostic observation.
More than one prior may be defensible. An evaluation can present a range: one based on a cautious assessment of the programme’s plausibility, another based on the view of a sceptical but informed reviewer, and perhaps a third based on relevant prior evidence. If the final conclusion changes little across this range, the update is less dependent on the starting point. If it changes substantially, the decision should not rest on a single posterior number. The same principle applies where programme staff and external stakeholders have different justified views about the starting claim.
Posterior probabilities do not supply their own decision threshold. A commissioner may decide that evidence needs to reach a high standard before scaling an intervention with substantial cost or risk. A different decision, such as whether to investigate an implementation problem further, may be justified at a lower level of confidence. These are decision rules, not properties of Bayes’ theorem. They should take account of the consequences of acting or not acting, the reversibility of the decision, and what further evidence could realistically be collected.
The evaluator should resist converting posterior probabilities into labels that conceal their basis. A posterior of 0.85 can look precise and persuasive, but its meaning depends on the prior, likelihood assignments, hypotheses, and treatment of dependence. It is better to report the number with the conditions that generated it and to state whether reasonable alternatives move it materially. The number is an aid to disciplined comparison, not an institutional stamp of certainty.
11.13 Evidence quality before calculation
Bayesian updating cannot rescue evidence that is poorly defined, unavailable at the relevant time, or generated by a source with an unexamined interest in the conclusion. Before calculation, the evaluator should ask whether the observation actually bears on the causal link; whether it is contemporaneous or retrospective; whether the source is independent; and whether the main alternative could produce the same observation. If the answer is unclear, the first task is source criticism or further data collection, not numerical elicitation.
This point is particularly important for programme monitoring. Counts of activities delivered, participants reached, or reports produced are often necessary evidence about implementation. They are not normally diagnostic of a mechanism or outcome contribution without additional information. An evaluation that converts each favourable monitoring indicator into a positive likelihood ratio can create an appearance of cumulative confirmation while repeatedly scoring the same implementation account.
The method is most useful after a strong qualitative and documentary inquiry has made the causal claim specific. It can then clarify why one observation carries more weight than another and how several distinct observations change confidence. It is not a replacement for that inquiry. Where the case lacks a clear mechanism, credible alternatives, or independent evidence, the appropriate conclusion may be that a diagnostic update cannot be defended.
11.14 When not to use diagnostic updating
Diagnostic updating is not required whenever an evaluation uses qualitative evidence. A conventional process trace may be preferable when the causal claim is still being developed, the relevant observations cannot be defined in advance, or the main task is to establish a chronology and identify plausible alternatives. In those circumstances, numerical likelihood ratios can give an artificial appearance of closure before the evaluator has decided what the evidence means.
The method is also unsuitable when the alternatives are too broad or numerous to be represented in a defensible comparison. A national policy outcome may be influenced by many programmes, political decisions, economic changes, and measurement practices. A diagnostic analysis can focus on one bounded causal link, but it should not assign a posterior probability to the claim that the entire policy outcome was caused by a programme unless the alternatives and evidence have been specified at that level.
It should not be used to rank programmes on the basis of their posterior probabilities. Different programmes may have different priors, hypotheses, outcomes, evidence bases, and decision contexts. A posterior from one diagnostic assessment is not a common performance metric. The method helps an evaluator reason about a defined causal claim; it does not create a league table of interventions.
Finally, diagnostic updating cannot resolve a conflict caused by unequal authority over the evidence. If programme management controls the documents, selects interviewees, and determines the probability assignments, the resulting calculation may be transparent in form but not independent in substance. The response is to widen evidence access, create an explicit challenge process, or limit the claim. It is not to multiply additional likelihood ratios derived from the same account.
11.15 Auditability and revision
A diagnostic assessment should be reproducible at the level of its reasoning. The underlying evidence register should identify the observation, source, date, causal link, rival account, and any dependence on another item. A separate calculation record should show the prior, likelihood assignments, formulas, and sensitivity scenarios. Confidential source material may need to be protected, but the report should still describe enough of its provenance and role for a reader to understand why it was assigned a particular weight.
Revision is expected when new evidence becomes available or when a reader identifies an error in a likelihood judgement. The evaluator should retain earlier calculations rather than overwrite them. A revised posterior may then be compared with the earlier one, showing whether the conclusion changed because of new evidence, a different treatment of dependence, or a reassessment of the alternative explanation. This record is particularly useful in evaluations that unfold over several programme phases.
Reproducibility does not mean that independent evaluators will necessarily agree. They may use different priors or judge an observation to be more consistent with a rival. The method makes the disagreement inspectable. If the conclusion holds across defensible alternatives, confidence in the claim is less dependent on one team’s judgement. If it does not, that instability is itself a finding that should shape the decision.
11.16 Strengths, limitations, and applicability
Table 11.3 summarises the diagnostic methods covered in this chapter.
| Method | Input | Probability source | Output | Main limitation |
|---|---|---|---|---|
| Bayesian process tracing | Causal-process observations from within-case analysis | Subjective elicitation or qualitative descriptors | Posterior odds for the causal claim | Probability estimation remains subjective; independence assumption often violated |
| Simulated diagnostic evaluation | Agent-based model of the mechanism | Simulation-derived frequencies | Posterior odds with simulation-based probabilities | Requires a well-specified simulation model; computationally intensive |
The principal strengths of diagnostic evaluation are formality and transparency. Prior odds, likelihood ratios, and posterior odds can be stated, inspected, challenged, and revised. New evidence can be incorporated as it becomes available, leaving an audit trail of the judgements that led to the final assessment. Numerical outputs are conditional on those judgements. They should be read as a transparent expression of the evaluator’s assumptions about evidence, not as a mechanically objective probability of causation.
The limitations centre on probability estimation. Sensitivity and false-positive-rate values are rarely available from empirical data. Subjective elicitation can reflect the evaluator’s priors and institutional commitments as much as the diagnostic value of the observation. The independence assumption is frequently violated in practice, and violations inflate the apparent strength of the evidence. The method also requires a hypothesis and alternatives specified precisely enough for likelihood judgements to have a clear meaning. Befani (2020) discusses practical ways to manage these problems; they remain reasons to use the method selectively and to report sensitivity to alternative probability assignments.
Numerical likelihoods can convey false precision when they are based on limited evidence or a poorly structured elicitation. They should be accompanied by the rationale for each value, the alternative hypothesis against which the observation is assessed, and sensitivity analyses that show whether a conclusion changes under other plausible assignments. Where a likelihood cannot be defended, a qualitative process-tracing judgement may be more honest. In practice, when a team cannot agree on a value, or when independent reviewers propose ratios that would move the posterior in opposite directions, the defensible response is to report the evidence and the competing interpretations qualitatively rather than to record a contested number, and to state which link remains unresolved for that reason.
Bayesian updating can still improve evaluation reasoning when it exposes assumptions that would otherwise remain implicit. A claim that an observation is strongly confirmatory already assumes that the observation is much more likely if the causal hypothesis is true than if an alternative is true. Diagnostic evaluation requires the evaluator to state and challenge that assumption. It does not remove the need for case knowledge, source criticism, or competing explanations.
The next chapter turns from individual-case analysis to approaches that draw on the knowledge and experience of programme participants and stakeholders: participatory and agency-based methods for causal inference.