7 Configurational Approaches
The previous chapter asked “what would have happened without the intervention?” and constructed the counterfactual through within-unit temporal comparison. This chapter asks a different question: “which combinations of conditions are associated with the outcome in a defined population of cases?” Configurational comparative methods (CCMs) take seriously the possibility that an outcome depends on a conjunction of conditions and that more than one conjunction may be relevant. Their formal starting point is the INUS condition introduced by Mackie (1965): an insufficient but necessary part of an unnecessary but sufficient conjunction. CCMs search for combinations that are consistently associated with an outcome across a bounded set of cases.
Two principal methods dominate the CCM family. Qualitative comparative analysis (QCA), introduced by Ragin (1987), constructs truth tables from case-level data and applies logical minimisation to identify sufficient and necessary conditions for the outcome. Coincidence analysis (CNA), developed by Baumgartner (2009), takes a different approach: it searches the data for all minimally sufficient conditions without requiring truth-table construction or prior specification of which factor is the outcome. Both methods operationalise INUS logic, but they differ in analytical strategy, assumptions, and the role of the researcher. Mackie supplies the conceptual account of conjunctural causation; Ragin developed QCA as a set-theoretic comparative method; Schneider and Wagemann set out the main design and calibration conventions (Schneider & Wagemann, 2012); Baumgartner developed CNA; and Thiem’s work has pressed the field to demonstrate robustness rather than rely on a single solution (Thiem, 2022). CNA is distinct from event-coincidence analysis and detector-coincidence procedures used in other fields.
The bibliometric map (Chapter 5) finds 8,270 title-and-abstract pairs naming QCA, the largest core-method corpus in the book. It concentrates in Sociology and Political Science (3,608 pairs), Strategy and Management (875), and Political Science and International Relations (358), with a long tail into education, public health, and tourism research. The institutional-country pattern is unusual among the method families: China accounts for the largest fractional share (1,588 of 8,270 works), ahead of the United States (669) and the United Kingdom (400), reflecting a substantial QCA literature in Chinese management and information-systems research. Figure 5.3 shows sustained growth in QCA mentions since the mid-2000s. CNA cannot be read the same way: its 323 title-and-abstract pairs are the residue of a much larger 1,723-record discovery set, because the exact phrase also retrieves event-coincidence and detector-coincidence procedures in climate science and physics, and record-level screening of that set is not yet complete. The count is reported here as a discovery figure, not a comparative one.
7.1 Qualitative comparative analysis
QCA exists in three variants, each suited to different data structures. Crisp-set QCA (csQCA) codes all conditions as binary: present or absent. Fuzzy-set QCA (fsQCA) calibrates conditions to degrees of membership between 0 and 1, allowing graded distinctions (Ragin, 2008). Multi-value QCA (mvQCA) permits conditions to take more than two discrete values without requiring the continuous calibration that fsQCA demands. Table 7.1 summarises the three variants.
| Variant | Set type | Calibration requirement | Suitable case structure | Output |
|---|---|---|---|---|
| Crisp-set (csQCA) | Binary (0/1) | Dichotomisation thresholds | A bounded case population with substantively meaningful binary distinctions | Sufficient/necessary configurations |
| Fuzzy-set (fsQCA) | Continuous (0 to 1) | Three anchors: full membership, crossover, full non-membership | A bounded case population with defensible graded set membership | Sufficient/necessary configurations with consistency and coverage scores |
| Multi-value (mvQCA) | Discrete (more than two categories) | Category assignment | A bounded case population with stable, substantively distinct categories | Sufficient/necessary configurations |
The analytical steps of a QCA study follow a consistent sequence regardless of variant. The analyst first identifies the conditions (potential causal factors) and the outcome, drawing on theory and case knowledge. Raw data are then calibrated into set memberships. For csQCA this means applying a threshold to classify each case as in or out of the set; for fsQCA it means specifying three qualitative anchors (full membership, crossover point, full non-membership) and using a logistic function to transform continuous data into fuzzy-set scores. The analyst constructs a truth table that lists all logically possible combinations of conditions alongside the number of observed cases and the outcome value for each row. Logical minimisation, typically using the Quine-McCluskey algorithm, reduces the truth table to the simplest set of configurations that account for the outcome. The Quine-McCluskey algorithm works by comparing pairs of configurations that differ on only one condition: if two configurations both produce the outcome and differ only on whether condition A is present, then A is logically redundant for that pair, and the algorithm removes it. This pairwise comparison is repeated until no further simplification is possible. The result takes three forms: a complex solution (no simplifying assumptions about unobserved configurations), a parsimonious solution (all logically possible simplifying assumptions), and an intermediate solution (only theoretically plausible simplifying assumptions). The intermediate solution requires the analyst to state directional expectations before minimisation and to allow only those expectations to simplify the solution. Schneider & Wagemann (2012) recommend reporting all three solutions and focusing interpretation on the intermediate solution, because it makes the assumptions visible while avoiding the most permissive use of logical remainders.
Calibration deserves separate attention because it is the step at which substantive knowledge enters most directly. The three anchors in fsQCA are not determined by the data distribution; they are set by the analyst on the basis of theoretical or contextual knowledge about what constitutes “high” or “low” membership in a condition. A GDP-per-capita threshold for “high income” in a study of sub-Saharan African countries will differ from the same threshold in a study of OECD members. Poor calibration choices can shift cases across the crossover point and alter the truth table, changing the solution. This sensitivity is a feature of the method, not a defect: it forces the analyst to make substantive judgements explicit. But it also means that readers must be able to inspect and challenge the calibration decisions (Schneider & Wagemann, 2012).
7.2 Designing a configurational study
Configurational analysis begins with a bounded population of cases, not with a software search for combinations. The evaluator needs to state what counts as a case, why the cases belong to the same analytic population, the period represented by each case, and the outcome to be explained. A case can be a district, service site, organisation, policy episode, or programme implementation. These choices determine the meaning of a condition and the interpretation of a solution. A comparison of schools that received the same intervention is different from a comparison of national reforms, even if both studies code a condition called ``implementation capacity’’.
Condition selection requires the same discipline. Each condition should represent a causal factor that the evaluator can define, observe, and distinguish from the outcome and from the other conditions. Adding a plausible factor is not costless. With \(k\) binary conditions, the truth table contains \(2^k\) possible configurations. Six conditions produce 64 rows; eight produce 256. A modest case set will observe only a fraction of them. The analyst should therefore justify each condition from the causal argument and record which potentially relevant factors were omitted. A long list of weakly measured conditions creates an appearance of explanatory reach while making the truth table sparse (Ragin, 2008; Schneider & Wagemann, 2012).
Calibration must be recorded as a chain from source data to set membership. For each fuzzy-set condition, the report should identify the raw measure, the three anchors, the substantive reason for each anchor, the transformation used, and the resulting scores for every case. For crisp sets, it should identify the rule that places a case inside or outside the set. A reader should be able to see which cases lie close to a crossover point and whether a small change in a threshold would alter their membership. Reproducibility here means more than providing the final truth table. It means making the coding decisions inspectable before logical minimisation begins.
The truth table contains further decisions that should not be hidden in software defaults. The evaluator must specify the frequency threshold for treating a configuration as empirically relevant, the consistency threshold for treating it as sufficiently associated with the outcome, the treatment of contradictory rows, and the use of logical remainders. Each decision has a substantive consequence. Raising a frequency threshold can exclude rare but relevant pathways; lowering it can give a single case undue influence. Treating a contradictory row as noise may be reasonable when a measurement error is documented, but not when the cases differ on an omitted condition. Logical remainders can simplify a solution, but the assumptions used to do so must follow from theory and case knowledge rather than from a preference for a shorter formula (Schneider & Wagemann, 2012; Thiem, 2022).
Sensitivity analysis should test the decisions that bear on the conclusion. This can include varying calibration anchors within a defensible range, changing consistency or frequency thresholds, examining influential cases, and comparing complex, intermediate, and parsimonious solutions. The purpose is not to search until one solution survives every imaginable variation. It is to establish which relationships are stable under choices a reasonable analyst might have made, and which depend on one calibration, one case, or one counterfactual assumption. Findings that change materially belong in the result, not in an unreported robustness appendix.
Case knowledge remains necessary after the formal analysis. A solution formula identifies a set relation in the observed data. The evaluator still needs to examine whether the cases assigned to a pathway share the relevant causal sequence, whether an omitted factor explains the association, and whether the direction of the proposed relationship is plausible. QCA can discipline a cross-case comparison; it does not turn a calibration table into direct evidence that an intervention would cause the outcome if one condition were changed. That latter claim requires evidence about intervention, ordering, and mechanisms.
Two types of set-theoretic relationship are central to QCA. A condition is necessary for the outcome if it is present in every case that shows the outcome. A configuration of conditions is sufficient for the outcome if every case with that configuration shows the outcome. Consistency measures the degree to which the empirical evidence supports the set-theoretic claim (perfect consistency = 1.0). Coverage measures the proportion of outcome cases that the configuration explains. Most QCA studies focus on sufficiency analysis, but necessity analysis is a distinct and important preliminary step: if a condition is necessary, it must appear in every sufficient configuration.
QCA can be useful in an evaluation with several comparable sites, institutions, or policy episodes when the causal argument concerns combinations rather than an average effect. It is less useful when the cases do not form a defensible population, when conditions cannot be measured independently of the outcome, or when the evaluator needs an estimate of a treatment contrast. Software for QCA includes the standalone application fsQCA and the R packages QCA and SetMethods.
Kien et al. (2018) used fsQCA in the process evaluation of a cluster-randomised school health-promotion programme in Austria. The wider study included 24 intervention and 27 control classes. The configurational analysis focused on the intervention classes and examined combinations of five teacher and implementation conditions in relation to change in pupils’ emotional and social school experience at 20 months. It identified distinct pathways associated with improvement and with no improvement, including configurations involving implementation quality, teachers’ prior knowledge of the intervention, self-efficacy, and the delivery of physical-activity breaks. The analysis did not replace the trial’s intervention comparison. It examined why the intervention was implemented and experienced differently across classes, and the authors reported the pathways for improvement and non-improvement separately (Kien et al., 2018).
Kuruvilla et al. (2014) report a component of the Success Factors for Women’s and Children’s Health studies that applied Boolean, qualitative comparative analysis alongside statistical and econometric analysis of 144 low- and middle-income countries over 20 years, seeking to explain why some countries progressed faster than comparable others on maternal and child mortality. The configurational component distinguished faster- from slower-progress countries and identified combinations of health-system, governance, and multisectoral conditions associated with faster reduction in mortality, complementing the study’s separate statistical and case-based components.
This example sits at the large end of what evaluators usually mean by a “small-n” configurational study, and it illustrates a case-population choice that differs from the single-programme comparisons elsewhere in this book: the cases are countries rather than sites or organisations, and the outcome is a population health trend rather than a bounded programme’s implementation record. The multi-country QCA component was one part of a mixed-method architecture, not a stand-alone claim, which is the safer way to read a configurational result obtained on this scale.
Schneider & Wagemann (2006) apply QCA to the consolidation of democracy across post-communist states, identifying combinations of proximate and remote conditions, a widely cited illustration of configurational analysis of governance reforms across a moderate number of countries.
7.3 Coincidence analysis
CNA differs from QCA in three respects that have practical consequences. First, CNA does not require the analyst to specify which factor is the outcome before the analysis begins. It searches for sufficient and necessary conditions across all factors simultaneously, constructing causal models bottom-up from the data (Baumgartner, 2009). Second, CNA does not construct a truth table. It identifies redundancy-free Boolean dependency structures directly, using fit parameters (consistency and coverage, computed differently from QCA) to evaluate candidate solutions. Third, CNA can model multi-outcome structures in a single analysis: if condition A is sufficient for intermediate outcome B, and B combined with C is sufficient for final outcome D, CNA can recover the entire causal chain rather than treating each outcome separately.
The practical implication is that CNA is less reliant on the analyst’s prior specification of the causal structure. When the research question is exploratory (“what causes what in this set of cases?”) rather than confirmatory (“what combinations cause this specific outcome?”), CNA has an advantage. When the causal structure may involve mediation or causal chains, CNA can detect them whereas QCA would require separate analyses for each step. The R package cna implements the method and includes tools for robustness testing.
When to prefer one method over the other depends on the evaluation question and the analyst’s prior knowledge. QCA is the stronger choice when the outcome is known, the conditions are theoretically grounded, and the analyst wants to test specific configurational hypotheses. CNA is the stronger choice when the causal structure is unknown, multiple outcomes may be causally linked, or the analyst wants to minimise the influence of prior specification on the results.
7.4 Designing and interpreting a CNA analysis
CNA reduces one form of prior specification; it does not remove design choices. The analyst still selects the cases, the factors entered into the search, their coding, the population to which the analysis refers, and the thresholds used to assess candidate models. A wide search over loosely related factors can generate many formally adequate models without producing a credible causal account. The starting point should therefore be a bounded set of factors with a stated relationship to the programme and outcomes, even when the evaluator does not impose a single outcome variable at the outset (Baumgartner, 2009; Baumgartner & Ambühl, 2020).
Model ambiguity is a substantive result. More than one Boolean model may meet the selected consistency and coverage criteria. Selecting one solely because it is shorter or accords with the evaluator’s expectation conceals the uncertainty. The report should show the competing models, the threshold choices, and the case patterns that distinguish them. Where ambiguity remains after reasonable restrictions based on design and measurement, the appropriate conclusion is that the available cases do not identify one causal structure. A method that returns a formula is not necessarily a method that has resolved the causal question.
The coding of outcomes deserves the same attention as the coding of conditions. In the church-based cancer-education study discussed below, reach and time to workshop completion were converted into high and low categories using stated rules. Such decisions may be necessary for a configurational analysis, but the resulting claim concerns membership in those defined outcome sets. It does not automatically generalise to a continuous increase in reach, to cancer outcomes, or to an intervention effect. The distinction is particularly important in implementation studies, where a configuration associated with timely delivery can help diagnose programme conditions without estimating the consequences of delivery for participants.
CNA can also represent chains in which one factor is sufficient for an intermediate outcome and that outcome is part of a configuration for a later outcome. A chain is useful only when its ordering follows the intervention and measurement process. If all variables are measured at one point in time, a fitted chain may be compatible with several temporal accounts. Process information, dates, and knowledge of how the programme operated are needed to assess whether the proposed sequence is credible. CNA’s ability to search for multi-outcome structures does not establish temporal order from cross-sectional data alone.
Knott et al. (2022) evaluated implementation of cancer education workshops across 13 African American churches in Maryland. The study used CNA to identify combinations of organisational-capacity features associated with two implementation outcomes: intervention reach and the time required to complete the workshops. The analysis found more than one pathway to higher reach. Some churches succeeded with an established health ministry, while smaller congregations and mid-sized churches with recent health-promotion activity followed other pathways. The study also identified combinations associated with faster implementation, including recent health-promotion activity and particular staffing or leadership arrangements. CNA did not estimate the effect of the workshops on cancer outcomes. It identified configurations that distinguished stronger and weaker implementation across the participating churches.
Damschroder et al. (2022) used coincidence analysis in a cross-sectional study of 69 Veterans Health Administration facilities offering comprehensive lifestyle interventions, pharmacotherapy, and bariatric surgery for obesity in varying combinations. The analysis searched for combinations of treatment components, facility complexity, and context associated with higher population impact, defined by reach to eligible patients and weight outcomes. The final model identified five distinct pathways to higher-impact status, each depending on facility complexity level together with factors from one or more treatment domains, and it explained a large majority of the higher-impact facilities in the sample.
The study reads as a direct methodological counterpart to the church-based cancer-education example above: both used CNA to identify multiple routes to a defined implementation outcome across comparable sites, rather than to estimate a single treatment effect. The VHA analysis also illustrates a recurring feature of CNA work in health-systems research: the outcome (population impact) is itself a constructed threshold on continuous facility-level data, so the reported pathways describe conditions for crossing that threshold, not conditions for a continuous increase in impact.
Both published applications come from high-income health-systems research. Coincidence analysis has so far seen little use in international-development evaluation, so these cases stand as demonstrations of the method rather than as development exemplars. An evaluator applying CNA in a development setting would follow the same logic on a comparably sized, diverse set of programme sites, and would face the same requirement to justify the factor set and report every competing model that meets the fit thresholds.
7.5 The QCA-CNA debate
The relationship between QCA and CNA is not merely a matter of methodological preference; it has become a substantive dispute about what configurational methods can and cannot claim. Thiem (2022) has argued that QCA, as commonly practised, produces fragile results: solutions change substantially with small alterations in calibration thresholds, case selection, or the handling of logical remainders (unobserved configurations). Benchmarking studies using simulated data with known causal structures have shown that QCA’s truth-table approach can recover incorrect solutions when limited diversity is high, when the number of conditions is large relative to the number of cases, or when measurement error is present. These critiques target not the INUS ontology but the specific analytical procedures of truth-table construction and logical minimisation.
Baumgartner & Ambühl (2020) have argued that CNA avoids some of these problems because it does not rely on truth tables and does not require the analyst to make assumptions about unobserved configurations. They contend that CNA’s search algorithm is better suited to recovering causal structures from observational data because it tests all factors symmetrically. Proponents of QCA respond that CNA’s data-driven approach risks overfitting, that the method’s agnosticism about the outcome variable can produce causally implausible models, and that QCA’s requirement for theoretical grounding is a strength rather than a weakness (Schneider & Wagemann, 2012).
The debate remains unresolved. For the evaluator, the practical lesson is threefold. First, neither method should be applied mechanically; both require careful attention to case selection, condition specification, and robustness testing. Second, robustness checks (sensitivity analysis of calibration thresholds for QCA; variation of consistency and coverage thresholds for CNA; permutation tests for both) are not optional but essential. Third, convergence between QCA and CNA results on the same data strengthens confidence in the findings; divergence signals that the results are specification-dependent and should be interpreted cautiously.
7.6 Reading and reporting a configurational solution
A configurational formula is a compact statement about a defined set of cases. In QCA notation, an expression such as \(A * B \rightarrow Y\) states that membership in the conjunction of \(A\) and \(B\) is sufficient, to the degree shown by the consistency statistic, for membership in the outcome set \(Y\). It does not state that an evaluator can raise \(Y\) by changing \(A\) while holding everything else constant. It also does not state that \(A\) or \(B\) is necessary in every pathway. Readers need the formula, the direction of the relation, the outcome definition, and the cases represented by each pathway before they can assess what the result means.
Consistency and coverage answer different questions. Consistency asks how closely the observed cases conform to a claimed subset relation. Coverage asks how much of the outcome set is accounted for by a condition or configuration. A pathway can have high consistency and low coverage: the cases that display it usually have the outcome, but the pathway accounts for only a small part of the outcome. A solution with broad coverage may still leave important outcome cases unexplained. Reports should give both statistics for each pathway and identify unique coverage where several pathways overlap. Reporting only a final solution formula makes it impossible to see whether the apparent result depends on one narrow group of cases (Ragin, 2008; Schneider & Wagemann, 2012).
Deviant cases require analysis rather than disposal. A case that displays a proposed sufficient configuration without the outcome may indicate measurement error, a calibration problem, an omitted condition, temporal mismatch, or an incorrect causal expectation. An outcome case not covered by the solution may point to an additional pathway. The evaluator should list the cases in each row, identify the cases supporting each solution term, and discuss the cases that depart from the proposed relation. A solution that fits after the difficult cases have disappeared from the account is less informative than one that specifies what remains unexplained.
Configurational claims are usually asymmetric. The pathways associated with an outcome need not be the mirror image of the pathways associated with its absence. A study of successful implementation should therefore not infer the causes of failure by simply negating its successful configurations. Analysis of the negated outcome can be useful, but it is a separate question with its own calibration, cases, and solution. This matters for evaluation because a programme may reach an intended outcome through more than one combination of conditions, while non-achievement may result from different and more heterogeneous processes (Ragin, 2008).
The report should separate three levels of conclusion. The empirical conclusion describes a set relation in the cases analysed. The causal interpretation states why the cases and conditions can be understood as a causal configuration, drawing on timing, theory, and case evidence. The intervention implication concerns whether changing a condition is feasible, ethical, and likely to alter the configuration in another setting. These are connected but not interchangeable. A high-consistency solution may be an important finding at the first level while providing weak grounds for a policy prescription at the third.
A report should identify the case population and period; outcome and condition definitions; raw-data sources; calibration rules; truth table; frequency and consistency thresholds; contradictory and excluded cases; treatment of logical remainders; all relevant solution types; consistency, coverage, and unique coverage; robustness checks; and the case-level evidence used to interpret each pathway. For CNA, it should also report the factor set searched, fit thresholds, all competing models that meet those thresholds, and the grounds for resolving or retaining model ambiguity.
7.7 Using configurational findings in an evaluation
The most useful output from a configurational analysis is often a disciplined account of variation, rather than a ranked list of programme ingredients. A pathway can identify a group of sites in which a particular combination of conditions was associated with implementation or an outcome. Programme staff can then examine whether those conditions are changeable, whether the combination was present before the outcome, and whether the same operational relation appears in the records and accounts from those sites. A condition that is consistently present may be a practical constraint, a proxy for an unmeasured feature, or a feature produced by the same process as the outcome. The formula does not decide among these possibilities.
This distinction matters when findings are translated into adaptation decisions. Conditions sometimes describe aspects of the setting that a programme cannot alter within the evaluation period. Others describe delivery arrangements that can be changed, but only if the necessary resources or authority are available. Before adopting a pathway as a design rule, an evaluation team should specify the proposed change, the cases to which it is intended to apply, and the evidence that supports the expected sequence. This work may draw on implementation records, process tracing, observation, or subsequent testing. It is a separate step from logical minimisation.
Configurational results can also inform sampling. If a solution identifies two outcome pathways, a subsequent within-case study should not select cases from only the most common pathway. It should examine a case that represents each pathway and, where possible, a deviant case that the solution does not explain. That selection makes it possible to assess whether the labelled conditions correspond to a common causal process or merely share a coding pattern. It also prevents a high-coverage pathway from becoming the sole narrative of a heterogeneous programme.
The method is particularly useful alongside a counterfactual design when the two components answer different questions. A trial or quasi-experimental study may provide evidence about an intervention contrast. QCA can then examine heterogeneity in the implementation or contextual combinations observed among the sites. The QCA should not be used to relabel associations between implementation conditions and outcomes as causal effects of the intervention. Nor does an estimated average effect validate every configurational pathway. The designs need separate claims, measures, and assumptions before their findings can be brought together.
Commissioners can make this division explicit in the terms of reference. They should state whether the analysis is intended to describe implementation configurations, explain outcome variation within a defined programme, generate hypotheses for a later study, or contribute to a causal account with other evidence. They should also reserve time for case checking after the initial solution is produced. A request for a formula without access to case records, calibration decisions, and substantive discussion of deviant cases turns the method into a reporting exercise rather than an evaluative analysis.
7.8 Strengths, limitations, and applicability
Table 7.2 summarises the conditions under which each configurational method is applicable.
| Method | Case structure | Data type | Causal claim | Main limitation |
|---|---|---|---|---|
| csQCA | A modest, diverse case set relative to the number of binary conditions | Binary | Sufficient/necessary configurations | Dichotomisation loses information |
| fsQCA | A modest, diverse case set with substantively justified calibration | Continuous, calibrated to 0-1 | Sufficient/necessary configurations with consistency scores | Sensitive to calibration thresholds |
| mvQCA | A modest, diverse case set relative to the number of categories and conditions | Multi-category | Sufficient/necessary configurations | Limited software support; less developed theoretically |
| CNA | A bounded factor set and enough variation to distinguish candidate models | Binary or multi-value | Sufficient/necessary conditions and causal chains | Risk of overfitting with many factors; model ambiguity |
There is no defensible universal minimum number of cases for these methods. A study with ten cases may be informative when it examines a small number of well-measured conditions and contains relevant variation. A study with fifty cases may remain uninformative when the outcome and conditions are almost constant, when many conditions create an almost empty truth table, or when the coding lacks a substantive basis. Case count is therefore one diagnostic among several: the evaluator also needs to examine the number of configurations implied by the model, the observed distribution of cases, the number of outcome cases, and the degree to which the result depends on a few observations (Thiem, 2022).
Data provenance is part of that assessment. Conditions and outcomes may be drawn from administrative records, structured observation, document coding, interviews, or expert judgement. These sources can be appropriate, but the report should make their dependence visible. If a programme manager supplies both the rating of implementation quality and the judgement that a site succeeded, the apparent configuration may partly reproduce that person’s assessment. Triangulating measurements, recording coding rules, and testing plausible recodings can reduce this risk. They do not replace the need to show how the underlying information became a set membership score.
Configurational analysis can also contribute to a wider evaluation without carrying the whole causal claim. It can identify cases or pathways for closer within-case investigation, clarify which contextual features should be measured in a subsequent comparative study, or show where an average effect estimate may conceal divergent implementation patterns. The additional evidence must have a specified role. Process evidence can examine how a configuration operated; a counterfactual design can address an intervention contrast; stakeholder accounts can test whether the coded conditions represent the programme as implemented. Agreement among these components is useful only when they do not repeat the same source or assumption.
Software makes it easy to generate truth tables and minimised formulas. It cannot choose a defensible case population, decide what a condition means, or determine whether a counterfactual assumption is plausible. A reproducible study should therefore preserve the raw or shareable source data, calibration script or coding workbook, truth table, software version, parameters, solution output, and robustness analyses. This record allows another analyst to reproduce the calculation. It does not guarantee that the original causal interpretation is correct, but it makes the points of disagreement visible and open to revision.
The same material is useful to commissioners and readers who will not rerun the analysis. They need enough information to judge whether the cases were comparable, whether the conditions were measured before the outcome where that ordering matters, and whether the stated pathways rest on observed rows or on assumptions about rows that were never observed. A formula without this record is a conclusion detached from its evidential basis.
The principal strengths of configurational methods are well matched to development evaluation. They handle conjunctural causation: the recognition that outcomes arise from combinations of conditions rather than single causes. They formalise equifinality: different combinations can produce the same outcome, and the methods identify all such pathways rather than estimating a single average effect. The truth tables and solution formulas are transparent and auditable. The methods operate at case numbers (10 to 50) that are common in multi-site and cross-country evaluations.
The limitations are equally important. The limited diversity problem grows rapidly with the number of conditions: five binary conditions produce 32 possible configurations, and a study with 20 cases will observe only a fraction of them. Unobserved configurations require assumptions (simplifying assumptions in QCA, or exclusion in CNA), and these assumptions can drive the results. Calibration sensitivity means that small changes in thresholds can alter the solution, particularly when cases cluster near the crossover point. Configurational methods provide no temporal or mechanistic information: they identify cross-sectional patterns but do not explain how or why a configuration produces the outcome. For mechanistic explanation, evaluators need the generative approaches discussed in the next chapter.
Three criticisms recur in the methodological literature. First, QCA is deterministic in a probabilistic world: consistency thresholds below 1.0 are treated as acceptable, but the theoretical framework assumes that sufficient conditions always produce the outcome, creating a tension between the method’s logic and its practice. Second, results are fragile to specification changes: adding or removing a single condition, changing a calibration threshold, or including one additional case can alter the solution substantially. Thiem (2022) document this fragility across published QCA studies. Third, the limited diversity problem means that many solutions rely on assumptions about configurations that were never observed, and different assumptions yield different results.
Proponents respond on each point. On determinism, Schneider & Wagemann (2012) argue that consistency thresholds are an honest acknowledgement that real-world data contain noise, and that the alternative (ignoring deviations) would be worse. On fragility, robustness protocols now exist: analysts can systematically vary calibration thresholds, drop cases, and compare solutions to establish which findings are stable across specifications. On limited diversity, the distinction between complex, intermediate, and parsimonious solutions makes the role of assumptions transparent. The intermediate solution, which incorporates only theoretically plausible simplifying assumptions, offers a principled middle ground. The field has also developed formal robustness tests, including permutation-based significance tests, that were absent from early QCA practice.
Configurational methods identify which combinations of conditions produce the outcome, but they do not explain how those combinations work or why they succeed in some contexts and not others. The next chapter introduces process tracing, which examines causal processes within a case.