13 Computational Simulation as Complement to Small-n Designs
Agent-based modelling and system dynamics are not themselves small-n evaluation methods. Where Chapter 12 drew evidence from stakeholder accounts and verification, this chapter turns to computational tools that operate at any scale, from a handful of agents to millions. They can make a causal theory explicit, explore the implications of stated assumptions, and in limited circumstances supply probability inputs for diagnostic evaluation. They do not create an empirical counterfactual when a comparison is unavailable. The chapter concentrates on these supporting roles rather than on computational technique.
Three sources anchor the chapter without playing identical roles. Gilbert & Troitzsch (2005) provides a foundational account of agent-based social simulation, Sterman (2000) sets out system-dynamics modelling, including feedback structures and model testing, and Befani et al. (2021) shows a restricted diagnostic use of simulation: deriving likelihood inputs from a simplified model for a Bayesian update. These are foundations for building and interrogating models, not evidence that a modelled scenario estimates an intervention effect.
The bibliometric review provides a reason for restraint. Many records that name agent-based modelling or system dynamics describe a model, not an evaluation of an intervention. In the two-stage reviewed sample, 73 of 80 records used the intended modelling term and 24 were coded as applications; the purposive sample does not estimate the frequency of evaluation use in the wider literature. Supporting simulation is therefore kept separate from the book’s comparative method corpus (Chapter 23).
13.1 Agent-based modelling
ABM in brief. Autonomous agents follow local interaction rules; system behaviour emerges from the bottom up. Key platforms: NetLogo, Mesa (Python), Repast.
An agent-based model (ABM) represents a system as a collection of autonomous agents that follow behavioural rules. Each agent perceives its local environment, interacts with neighbouring agents, and updates its state accordingly. The aggregate behaviour of the system emerges from these micro-level interactions rather than from a set of top-down equations. This bottom-up architecture makes ABM well suited to modelling causal mechanisms, because the analyst must specify the rules that produce the outcome rather than merely estimating a relationship between variables.
Three features of ABM are relevant for small-n evaluation. First, ABM can generate probability inputs for diagnostic updating. Befani et al. (2021) set out how simulated evidence patterns can supply likelihood inputs where empirical frequencies are unavailable and expert elicitation is contested (Chapter 11). The estimates remain conditional on model structure and assumptions. Second, ABM can represent mechanism chains that process tracing investigates. A model can express a stated theory of change as agent rules and show what system patterns follow from them. If the model reproduces several observed patterns only under particular assumptions, it can narrow a set of plausible accounts; it cannot verify the assumptions that generated the fit. Third, an ABM can compare explicitly modelled scenarios, including a scenario without a programme rule. The difference is a model-based scenario contrast, not an empirical estimate of what would have happened without the intervention.
Simulation can inform policy design without supplying evidence that a policy caused an observed outcome. A model becomes part of an evaluation only when it is built around an identifiable intervention, is confronted with relevant empirical evidence, and is used to assess a stated causal claim or policy choice. These conditions need to be specified in the evaluation, not inferred from the use of simulation software or from the model’s policy relevance.
Hennessy et al. (2016) developed an agent-based model to aid the evaluation and refinement of ChildObesity180 interventions for school-aged children in the United States. The model drew on two obesity-prevention initiatives and public data, and was intended to explore intervention design, data gaps, and alternative implementation scenarios. The authors state that the modelling effort stopped short of testing explanatory or predictive power. This is a useful boundary case: it was developed within an intervention and evaluation context, but its output was model-based support for refinement, not an empirical estimate of programme impact (Hennessy et al., 2016).
Griffin et al. (2010) used a simulation model of Plasmodium falciparum transmission to evaluate strategies for reducing malaria across settings in sub-Saharan Africa. The model represented human and mosquito infection dynamics and projected how combinations of long-lasting insecticidal nets, indoor residual spraying, and treatment might reduce transmission at different baseline intensities, and where elimination was plausible. Its output is a set of model-based scenario contrasts, conditional on the assumed transmission structure and parameters, rather than an empirical estimate of any single programme’s effect. The case illustrates simulation in a development context: a way of comparing intervention scenarios and identifying where empirical evaluation and investment might focus, used alongside field data rather than in place of it.
Validation is the central challenge. An ABM is only as credible as its rules, and the analyst must justify those rules with empirical evidence or well-established theory. Pattern-oriented modelling offers one approach: the analyst specifies multiple empirical patterns that the model must reproduce simultaneously, which constrains the parameter space and reduces the risk of overfitting to a single outcome. Even so, simulation results should be treated as structured thought experiments rather than as empirical findings. They sharpen causal reasoning; they do not replace empirical evidence.
13.2 System dynamics
SD in brief. Stocks (accumulations) + flows (rates of change) + feedback loops = dynamic system behaviour over time. Key platforms: Vensim, Stella, AnyLogic.
System dynamics (SD) models represent a system as a set of stocks (accumulations), flows (rates of change), and feedback loops. A causal loop diagram maps the qualitative structure: which variables reinforce each other (positive feedback) and which counteract each other (negative feedback). A quantified SD model translates this diagram into differential equations that can be simulated over time.
SD supports small-n evaluation by providing a formal way to examine a theory of change in a feedback-rich system. An SD model makes proposed feedback structures explicit and allows the team to examine their implications under stated parameter values. It can identify assumptions that need empirical scrutiny; it cannot establish that a feedback loop operated as specified.
Second, SD enables scenario analysis that can inform contribution analysis (Chapter 9). A model can compare scenarios with and without a specified programme component. The difference between their trajectories is an implication of the model, conditional on the stock–flow structure, parameters, and assumptions about other influences. It can expose whether a contribution story depends on an implausible feedback structure or parameter value. It cannot demonstrate that the programme made the corresponding difference in the observed world. Sterman (2000) provides the foundational treatment of SD modelling, including calibration, sensitivity testing, and validation procedures.
A system-dynamics model can be used to test the internal coherence of a theory of change, identify feedback processes that monitoring may miss, and compare explicitly modelled scenarios. The result remains conditional on the stock–flow structure, parameter values and validation evidence. It is not an estimated programme effect. This is why system dynamics can sharpen a contribution analysis but cannot settle its causal claim on its own.
Merrill et al. (2013) used system-dynamics techniques within a case-study evaluation of electronic health-information exchange for public-health reporting at the New York State Department of Health. The team collected project documentation and qualitative evidence from seven experts over eight months, then developed causal-loop diagrams iteratively from reference modes and accounts of implementation. The model clarified interacting implementation factors and possible leverage points. It did not estimate the effect of health-information exchange on public-health outcomes; it supported an implementation evaluation of a complex technology programme (Merrill et al., 2013).
13.3 Integration with small-n methods
The value of simulation for small-n evaluation lies in its integration with the methods covered in earlier chapters. Table 13.1 summarises three concrete pairings.
| Pairing | Simulation role | Practical protocol | Cross-reference |
|---|---|---|---|
| ABM + Bayesian process tracing | Generate likelihood distributions under stated hypotheses | Build ABM of the causal mechanism; run Monte Carlo simulations under the focal and rival hypotheses; use the output distributions to inform likelihood inputs for updating | Chapter 11 |
| SD + contribution analysis | Construct and test the contribution story | Translate the theory of change into an SD model; simulate with-programme and without-programme scenarios; compare trajectories | Chapter 9 |
| ABM + QCA | Explore the implications of a configurational account | Translate a proposed QCA solution into agent rules; examine which model assumptions generate or disrupt it; compare the implications with observed cases | Chapter 7 |
The ABM + Bayesian pairing has the clearest methodological formulation of these three protocols. Befani et al. (2021) show how simulation-derived probabilities can be used where subjective elicitation would otherwise supply the diagnostic inputs.
The first pairing is the most developed methodologically. Befani et al. (2021) show how simulated probabilities can address the probability-estimation problem discussed in Chapter 11. The second is a way of testing the coherence of a contribution story, not a formal integration protocol. The third is more speculative still. An ABM can explore the implications of a configurational account, but synthetic configurations cannot test whether a QCA solution is robust in the empirical population. They can show which model assumptions would reproduce or disrupt a proposed solution.
13.4 Model construction, calibration, and validation
A simulation should begin with a bounded decision problem and an explicit account of what the model is intended to represent. The team needs to specify the system boundary, the actors or stocks included, the processes that connect them, the time horizon, and the outcome to be examined. Every omitted process is an assumption about irrelevance or about the model’s intended scope. These choices should be documented before scenario results are discussed.
Calibration aligns model parameters with available evidence; validation asks whether the model can reproduce relevant observed patterns without relying on the same information in every step. Neither task is a ceremonial check. A model can fit an observed pattern through several incompatible parameter combinations, and a good fit does not verify the causal mechanism embedded in the code. Evaluators should report the data sources, parameters estimated or fixed from outside evidence, calibration procedure, fit diagnostics, and the observed patterns used for comparison (Sterman, 2000).
Sensitivity analysis is the minimum safeguard for policy scenarios. It should show which conclusions persist when uncertain assumptions, parameters, initial conditions, or structural relations are varied within a defensible range. A result that changes materially under a plausible alteration is itself a finding about uncertainty. Scenario output should therefore be reported as conditional model behaviour, with the assumptions that drive it, rather than as a prediction of what an intervention will achieve.
13.5 Governance, participation, and transparency
Model building involves substantive judgement. Programme staff, participants, decision-makers, and technical modellers may each know different parts of the system. Their involvement can expose missing feedbacks, unrealistic behavioural rules, inaccessible data, and implementation constraints. It can also create pressure to include assumptions that favour an existing programme decision. The evaluation should record who contributed to the model, which assumptions were contested, and how disagreements were resolved or retained.
Transparency requires more than a diagram of the model. A report should identify the model purpose; entities, states, rules, and interactions; equations or code; input data and their provenance; parameter values; calibration procedure; validation tests; scenarios; random seeds where relevant; and all sensitivity analyses. Reproducible code and shareable data should be provided where confidentiality permits. Where they cannot be shared, the report should explain what cannot be inspected and how that limits independent review.
Participatory model building can improve a programme theory without validating it. Stakeholders may agree that a feedback loop is plausible because it reflects their experience. That agreement is useful for specifying the model and identifying evidence to collect. It does not establish that the feedback operated in the cases studied. The evaluation should keep the distinction between a jointly developed causal hypothesis and an empirically supported mechanism.
13.6 Scenario interpretation and decision use
A scenario asks what the model produces after specified assumptions are changed. Its result can support a decision when the decision concerns the consequences of those assumptions within the modelled system. It is most useful for comparing options, identifying threshold effects, locating assumptions that drive divergence, and deciding what empirical information would reduce uncertainty. It is not evidence that the corresponding policy will produce the modelled outcome in the world.
Commissioners should specify this role in advance. They should state whether the model is intended to clarify a theory of change, support implementation planning, identify information needs, generate diagnostic probability inputs, or explore conditional scenarios. They should not ask a simulation to settle an empirical effect question when no observational or experimental evidence is available. The relevant response may be to redesign the empirical evaluation, not to increase the complexity of the model.
Scenario results should report ranges and conditions, including cases in which the model gives a different answer. A single preferred trajectory hides the uncertainty that matters for use. Where a decision is sensitive to one unverified behavioural rule or parameter, the report should identify that dependency and set out what observation could test it. This turns simulation into a tool for disciplined learning rather than an authority that closes a disputed causal question.
13.7 When simulation is not appropriate
Simulation is a poor choice when the intervention, outcome, and system boundary cannot be specified with enough precision to construct defensible rules or stocks and flows. It is also unsuitable when available evidence cannot inform the assumptions that drive the decision, when the model cannot be reviewed by people with relevant technical and substantive expertise, or when the timetable allows only an untested model to be presented as a result.
The resource requirements are material. An adequate simulation requires modelling expertise, access to relevant evidence, time for iterative checking, computing resources where needed, and a process for version control and review. A visually persuasive model built quickly from uncertain inputs can be more damaging than a simple qualitative theory because it gives assumptions an appearance of numerical authority. The appropriate conclusion in such circumstances is often that the causal theory needs empirical investigation before it can be modelled for decision support.
13.8 Reporting, review, and reuse
The report should state the model’s role at the outset: theory development, implementation planning, conditional scenario exploration, or the estimation of probability inputs for diagnostic updating. It should identify the decision question, model boundary, causal assumptions, input data and their provenance, calibration and validation results, scenarios, sensitivity analysis, and the version of the code that generated each result. Graphs must identify whether a series is simulated or observed. Earlier specifications should be retained when the model changes during an evaluation, together with the evidence or judgement that prompted the change.
Independent review needs both modelling and substantive expertise. Technical review can assess computational reproducibility and the implementation of the stated model. Substantive review can assess whether the boundary, rules, data sources, and scenarios represent the intervention and setting. The handover package should include executable code, input data or instructions for obtaining them, a parameter register, scripts that reproduce reported outputs, scenario definitions, software versions, and a changelog. Before a model is reused, the team should assess whether the intervention, population, causal theory, and decision question remain within its original boundary.
13.9 Claim boundaries and combination with empirical designs
Simulation can sharpen an empirical evaluation by identifying observations that would distinguish competing model structures, estimating the implications of measurement error, or testing whether an interpretation is sensitive to a stated assumption. Process tracing can assess whether the modelled sequence occurred in an observed case. Configurational analysis can identify cross-case patterns that a model must be able to accommodate. Counterfactual designs can estimate an intervention contrast against which a model’s assumptions or longer-run scenarios can be assessed. These are complementary tasks, not interchangeable forms of evidence.
The contribution of a model is strongest when it is confronted with evidence capable of challenging it. A model that only repeats its own assumptions can be internally coherent and still irrelevant to the programme under evaluation. Conversely, disagreement between a model and empirical evidence need not mean that the data are wrong. It may reveal an omitted process, a mis-specified behavioural rule, a measurement problem, or a boundary that was drawn too narrowly. The evaluation should treat disagreement as an opportunity to revise the causal account.
Simulation supplies conditional consequences of a formalised theory. It can make assumptions inspectable, help select empirical tests, and clarify the decisions supported by available knowledge. It cannot establish that a programme caused an outcome without empirical evidence about the programme, its implementation, and relevant alternatives.
13.10 Strengths, limitations, and applicability
Simulation handles complexity, non-linearity, and emergence in ways that verbal theories of change cannot. An ABM can reveal unintended consequences that arise from agent interactions. An SD model can identify tipping points and feedback traps. Both force the evaluator to make causal assumptions explicit, which disciplines the reasoning process even when the model itself is imperfect.
The limitations are equally clear. Model validity is always contestable. Every simulation rests on assumptions about agent rules (ABM) or functional forms (SD), and different assumptions can produce different conclusions. The risk of confirmation bias applies: an analyst who believes in the programme’s effectiveness may unconsciously calibrate the model to confirm that belief. Computational expertise is a practical barrier. Building and validating a simulation model requires skills that most evaluation teams do not possess, and the cost of acquiring those skills may not be justified for every evaluation. Simulation results can be data-informed through calibration and validation, but they remain model-based: their implications depend on the model structure, parameters, and scenarios. They complement empirical evidence; they do not substitute for it.
| Criterion | Agent-based modelling | System dynamics |
|---|---|---|
| Primary role | Generate probabilities; model mechanisms; virtual counterfactuals | Map feedback structures; scenario analysis; dynamic contribution stories |
| Minimum data | Qualitative evidence on agent rules; quantitative calibration data for validation | Causal loop diagram; stock/flow estimates; time-series data for calibration |
| Expertise required | Programming (Python, NetLogo, or similar); simulation methodology | SD software (Vensim, Stella, or similar); differential equations |
| Causal claim | Supports claims from other logics; does not generate independent causal evidence | Supports claims from other logics; does not generate independent causal evidence |
| Main limitation | Model assumptions drive results; validation is difficult with few empirical cases | Assumes continuous variables and homogeneous populations; feedback structure must be known |
| Best combined with | Bayesian process tracing; QCA | Contribution analysis; process tracing |
Three criticisms recur. First, simulation models can produce any result by adjusting their assumptions. A model is only as good as its rules, and with enough degrees of freedom, the analyst can fit any narrative. Second, simulation adds a layer of abstraction that distances the evaluation from the empirical evidence. Stakeholders may distrust findings that rest on “what-if” scenarios rather than on observed data. Third, the expertise and time required to build credible simulations may not be proportionate to the evaluation budget, especially in resource-constrained settings.
The response is not that formal models remove these problems. Pattern-oriented modelling and sensitivity analysis can expose the implications of choices about parameters and structure; they do not validate those choices. A documented simulation may be more open to inspection than an implicit verbal theory, but its results remain conditional. The required expertise and time should be judged against the uncertainty the model is expected to reduce.
Simulation is a supporting modelling tradition, not an additional route to causal inference. The next chapters address the practical choices that follow: how to select and combine the book’s method families, and how to assess the resulting design.