12 Participatory Approaches to Evidence About Change
The methods covered so far place most authority for framing and interpreting evidence with the evaluation team. Chapter 11 showed how Bayesian updating formalises within-case evidential judgements; the present chapter shifts part of that authority to programme participants and stakeholders (Chambers, 1997). Participants can provide evidence about changes in their own practice, local conditions, unintended consequences, and alternatives that may not be visible in programme records. Their accounts remain evidence to be assessed, not privileged access to causation. Participatory approaches make those accounts central to the inquiry and specify procedures for reviewing them.
Participatory evidence. Participants may observe changes, constraints, and unintended effects that programme records do not capture. Their accounts require interpretation, corroboration, and clear claim limits.
Participatory methods do not establish cause and effect in the same sense as counterfactual, configurational, or diagnostic designs. They produce stakeholder-informed accounts of change and of the programme’s possible role; verification or substantiation assesses what those accounts can support (Copestake, 2014). This does not rule out confounding. The methods can identify changes, explanations, and unintended consequences that pre-specified indicators or programme records may not capture. A disputed mechanism or alternative explanation requires another evidential operation.
A second distinction matters. “Participatory methods” is a broad term covering everything from participatory rural appraisal to focus group discussions. This chapter is not about participatory methods in general. It covers two specific methods that use stakeholder knowledge to support contribution claims: Most Significant Change (MSC) and outcome harvesting. Both include structured verification procedures that go beyond recording what participants say.
The works cited in this chapter do different jobs. Chambers (1997) established a broader argument for participatory approaches that take local knowledge seriously. Davies & Dart (2005) and Dart & Davies (2003) developed Most Significant Change as a systematic story-based monitoring and evaluation method. Wilson-Grau & Britt (2012) set out outcome harvesting as a retrospective, substantiated approach to contribution claims, while Patton (2008) places these approaches within a wider developmental-evaluation tradition. The methods share an interest in stakeholder knowledge, but they have different collection, selection, and verification procedures.
The bibliometric map (Chapter 5) treats the two methods differently. Outcome harvesting yields 64 title-and-abstract pairs, a small but usable corpus: it concentrates in Management Science and Operations Research and General Agricultural and Biological Sciences, and its venues and institutions (Cirad’s Agritrop repository, the CGIAR’s CGSPace, and Evaluation and Program Planning) point to an agricultural- and international-development literature, consistent with the method’s origin in that practice. Most Significant Change cannot be read the same way: the phrase retrieves 47,766 candidate records and only 1,908 survive title-and-abstract filtering, because “most significant change” is common English and is used for unrelated results across many fields. Record-level screening to separate genuine MSC applications from this noise is not yet complete, so no subfield, venue, or trend detail is presented for it here; Figure 5.6 (Chapter 5) reports outcome harvesting’s trend only.
12.1 Most Significant Change
Most Significant Change is a story-based method that collects change narratives from programme participants and stakeholders, subjects them to structured selection, and verifies the selected narratives against independent evidence (Dart & Davies, 2003; Davies & Dart, 2005). The method proceeds in three steps.
The first step is story collection. Programme participants and field staff are asked an open-ended question: “what was the most significant change that occurred as a result of the programme?” The question is deliberately open. It does not specify which outcome domain to report on, and it invites respondents to describe changes that the evaluator may not have anticipated. Each story includes a description of the change and the respondent’s explanation of why the programme contributed to it.
The second step is systematic selection. Panels at multiple levels of the programme (field offices, regional offices, headquarters) review the collected stories and select those they judge most significant. Each panel records its selection rationale: why this story, and not others, represents the most important change. The selection process surfaces the values and priorities of different stakeholder groups, which is itself evaluative information.
The third step is verification. Selected stories are cross-checked against programme records, third-party observation, or follow-up interviews. A story remains testimony after it has been checked, but corroboration can support particular elements of the account: whether an event occurred, when it occurred, and who was involved. It does not by itself establish that the programme caused the change. The strength of any contribution claim depends on the evidence about timing, the proposed pathway, and relevant alternatives. Davies & Dart (2005) emphasise that verification is not a binary judgement but an assessment of plausibility, specificity, and corroboration.
MSC does not estimate an effect or rule out confounding. It produces a curated set of change narratives, selected according to stated criteria and checked to varying degrees against other sources. It can capture unexpected outcomes that pre-specified indicators miss, including at a single programme site. Selection shows what participants and panels regarded as significant; it does not show that the selected accounts are typical or that they establish programme success.
Wilder & Walpole (2008) report the use of MSC in Fauna & Flora International partnership projects, including a community conservation project in Cambodia. Field teams recorded their own observations and stories obtained through their regular contact with community members; the project then discussed, selected and, where possible, triangulated the stories. This widened monitoring beyond crop-damage indicators to social changes associated with an elephant-warning system, including reported changes in well-being and attitudes. The authors also document the practical limits: collection, translation and verification required sustained staff time, and community meetings could privilege more powerful voices (Wilder & Walpole, 2008).
Ho et al. (2015) used the most significant change technique alongside a community scorecard process to assess improvements to the local health system in a conflict-affected area of eastern Democratic Republic of Congo. Community members and health-facility staff identified and discussed changes they considered most significant following the scorecard intervention, and these accounts were used to complement quantitative indicators collected as part of the wider evaluation.
The example shows MSC used as one component of a mixed-method evaluation of a governance-oriented health intervention in a fragile setting, rather than as the sole evidence source. It illustrates a recurring practical advantage of the technique in such contexts: story collection did not require the security and access conditions that a structured household survey would have needed, while still surfacing community perceptions that the scorecard’s own indicators did not capture.
Holden et al. (2015) applied the most significant change technique to evaluate whether a community of practice equipped public-health nutritionists in regional Australia to work with remote retailers to improve the food supply. Participants in the community of practice contributed stories of change, which were discussed and selected to identify the outcomes participants themselves considered most significant from their engagement.
The case extends the MSC examples in this chapter from an international-development conservation project to a professional community of practice in a high-income country, showing the technique’s applicability to a capacity-building intervention where the “beneficiaries” are practitioners rather than community members. As with the other MSC applications here, the selected stories describe what participants valued; they do not estimate the community of practice’s effect on food-supply outcomes more broadly.
The limitations are equally clear. Not all participants submit stories, and those who do may be systematically different from those who do not: more engaged, more literate, or more favourably disposed toward the programme. The selection panels’ criteria may reflect organisational priorities rather than evidential weight. Aggregation across stories is difficult because each story is unique and context-specific. The method does not address confounding: if a concurrent policy change also affected the participants, MSC provides no formal way to separate the programme’s contribution from the policy’s contribution. In development evaluation, MSC is widely used in NGO and civil-society evaluations of complex programmes with diffuse or hard-to-predict outcomes, including community development, empowerment, and governance interventions.
12.2 Outcome harvesting
Outcome harvesting reverses the direction of evaluation inquiry (Wilson-Grau & Britt, 2012). Rather than starting from a theory of change and testing whether predicted outcomes occurred, outcome harvesting starts from observed outcomes and traces them back to the intervention’s contribution. The direction is harvest first, attribute second.
The six-step procedure proceeds as follows. First, design the harvest: define the scope, intended users, and available resources. Second, gather data and draft outcome descriptions: programme staff and partners identify changes they have observed and describe the intervention’s claimed contribution to each. Third, engage change agents: those who were directly involved review and refine the outcome descriptions, correcting errors and adding detail. Fourth, substantiate: independent reviewers verify each outcome against documentary and third-party evidence. Fifth, analyse and interpret: look for patterns across the harvested outcomes. Sixth, support use of findings: present results to intended users.
Substantiation is the essential evidential step. Someone with relevant knowledge, and sufficient independence from the programme, reviews each outcome statement and the claimed contribution. The review can check whether the outcome occurred, whether programme activity preceded it, whether the proposed pathway is plausible, and which alternatives remain. Wilson-Grau & Britt (2012) distinguish this procedure from unstructured testimony collection. Substantiation can strengthen an outcome statement and narrow an unsupported contribution claim. It does not convert a post-hoc account into an estimate of effect.
Limsuan & Ferrer (2021) used outcome harvesting to review a climate-smart village initiative in Guinayangan, Quezon, after its 2014–2020 implementation period. The evaluation engaged farmers, local leaders and other key informants to identify and examine changes in awareness, knowledge, practices and local strategies. It documented outcomes at several levels, from farmers’ learning groups to municipal action and the national Adaptation and Mitigation Initiative for Agriculture programme. The account is retrospective and contribution-oriented: it identifies how the village became a platform for learning and policy influence, without claiming that the initiative alone caused subsequent national implementation (Limsuan & Ferrer, 2021).
Railer et al. (2020) used outcome harvesting within the programme evaluation of Queen’s University’s transition to competency-based medical education across 29 postgraduate programmes. The evaluation sought intended and unintended implementation outcomes after the institutional transition. It used the harvested outcomes to examine how stakeholder behaviours and relationships had shaped the transition. The study provides evidence about implementation learning and stakeholder-reported change; it does not estimate the educational effect of competency-based education on trainees or patients (Railer et al., 2020).
Using harvested outcomes. Outcome harvesting can identify changes for further inquiry. Contribution analysis can then examine the programme theory and alternatives relevant to a selected outcome.
Outcome harvesting and contribution analysis both build contribution stories, but from opposite directions. Contribution analysis starts from the theory of change and gathers evidence to test each causal link. Outcome harvesting starts from what happened and works backward to identify which intervention activities contributed. The two methods can be combined: outcome harvesting identifies the changes, and contribution analysis tests the causal logic connecting intervention to change. In development evaluation, outcome harvesting is well suited to advocacy and policy-influence programmes where outcomes are diffuse and non-linear, to multi-stakeholder initiatives, and to complex systems interventions where the theory of change is emergent rather than pre-specified.
12.3 Designing participatory inquiry
The design decision is not simply whether to invite participation. It concerns who frames the question, who can submit an account of change, who selects or substantiates accounts, and who controls the final interpretation. These roles should be fixed before collection begins. A process that asks programme staff to recruit participants, record their stories, select those stories, and write the conclusions has limited independence even if it uses the vocabulary of participation.
The evaluation should define the population from which accounts will be sought and the routes through which people can participate. Open calls may reach people with time, literacy, confidence, or a favourable relationship with the programme. Facilitated small-group discussions may allow participation by people excluded from formal meetings, but can also make dissent visible to local authorities or programme staff. Individual interviews may protect confidentiality but remove the collective interpretation that some participatory approaches value. The appropriate route depends on the claim, the risks of participation, and the people whose experience is needed to test it.
Selection rules should be recorded. For MSC, this includes the domains of change, the period covered, who receives the stories, how stories move through selection panels, the criteria used at each level, and whether the reasons for selecting or rejecting a story are retained. For outcome harvesting, it includes the definition of an outcome, the sources used to identify candidate outcomes, the people who formulate the outcome statements, and the criteria used to decide which outcomes are substantiated. A transparent process can still contain judgement. It makes the judgement inspectable.
12.4 Evidence, substantiation, and disagreement
Participatory accounts can establish that an actor describes a change, identifies a causal process, or disputes the programme’s account. They do not by themselves establish the occurrence, timing, or cause of the change. The evidence plan should distinguish those tasks. It should identify which claims can be checked against records or observation, which require further accounts from people in different positions, and which cannot be independently verified. A substantiator should have relevant knowledge of the outcome and enough independence from those who benefit from a favourable conclusion.
Substantiation is not a search for endorsement. It should allow an external reviewer to reject an outcome statement, narrow the stated contribution, identify an alternative explanation, or record unresolved uncertainty. The report should retain outcomes that were not substantiated and explain why. Reporting only the final selected stories or confirmed outcomes produces an account of institutional preference, not a transparent assessment of change.
Disagreement has analytical value. Participants, implementers, local officials, and reviewers may describe the same event differently because they occupied different positions, had different interests, or experienced different consequences. The evaluation should document material disagreement, identify the evidence on which each account rests, and avoid resolving it by counting supporters. A minority account may identify exclusion, an unintended effect, or a rival causal process that the majority did not see. It deserves assessment on the basis of its evidential content and the risks attached to reporting it.
12.5 Power, ethics, and safeguarding
Participatory inquiry changes who is exposed by evaluation. A story can disclose a participant’s political position, conflict with an employer, experience of discrimination, or dissatisfaction with an implementing organisation. Consent must cover how the account may be recorded, selected, verified, quoted, and shared. An assurance of anonymity is not sufficient when a detailed outcome, place, or sequence makes the person identifiable to others in the setting.
The team should also assess whether participation is voluntary in practice. Invitations issued through programme staff can be experienced as requests from an authority. Group selection processes can create pressure to endorse a preferred narrative. Payment, transport, language, disability access, childcare, and meeting times affect who is able to take part. These are not administrative details. They influence whose evidence enters the causal account and whose experience remains outside it.
Safeguarding can limit public disclosure without removing the need for an audit trail. The evaluation can retain protected source notes, anonymise public accounts, aggregate sensitive outcome descriptions, and explain when a claim could not be independently checked. It should not report a confidential assertion as conclusive evidence merely because readers cannot inspect it. Where verification would expose a person to risk, the claim boundary should reflect that constraint.
12.6 Analysis, use, and integration with other methods
MSC and outcome harvesting generate material for analysis as well as reporting. The team can examine recurring types of change, differences across participant groups, the sequence between reported outcomes, and the assumptions participants make about causation. This analysis should retain the full body of accounts. A selection panel’s preferred story may be useful for organisational learning, but it is a poor substitute for examining accounts that were not selected or that contradict the programme theory.
The methods can contribute to a wider evaluation when their role is specified. Harvested outcomes can identify an unexpected decision, practice, or policy change for process tracing. MSC accounts can identify an outcome or group that routine monitoring missed, leading to targeted data collection. A configurational analysis can examine patterns across documented cases, while contribution analysis can place verified outcomes within a wider programme theory. None of these additions automatically validates a story or outcome statement. They address different parts of the causal claim.
Commissioners should decide whether the purpose is accountability, learning, adaptation, or the identification of outcomes for further inquiry. These purposes can be combined, but they create different incentives. A public account of achievement may discourage participants from reporting harm or failure. An internal learning process may generate candid accounts but have weak independence. The terms of reference should state who owns the raw accounts, who decides what is published, how adverse material will be handled, and when an external reviewer can challenge the programme’s interpretation.
12.7 Strengths, limitations, and applicability
Table 12.1 summarises the two methods.
| Method | n range | Evidence source | Causal claim | Main limitation |
|---|---|---|---|---|
| Most Significant Change | 1 site or more | Stakeholder narratives, verified against records | Verified contribution narrative | Selection bias in story submission; no confounding control |
| Outcome harvesting | 1 programme or more | Harvested outcomes, substantiated by independent reviewers | Substantiated contribution claim | Post-hoc attribution risk; dependent on substantiation quality |
Neither method controls for confounding on its own. Both can be combined with process evidence or another design where the evaluation requires a stronger causal claim. See Chapter 15 for combination protocols.
The principal strengths of participatory methods are their capacity to capture complexity, emergence, and unexpected outcomes. They centre participant voice and insider knowledge. They work at small n without requiring repeated measurement or pre-specified outcome indicators. They are less reliant on the evaluator’s prior theory about what the programme should produce.
The limitations are substantial. Both methods rely on participants’ recall, willingness to report, and relationship with the programme. Social desirability can lead people to over-attribute change. Neither method provides a formal way to rule out confounding. MSC selection reflects the values and power of the panels; outcome harvesting can cherry-pick favourable outcomes when substantiation is weak. Process tracing can investigate a mechanism behind a selected outcome, and configurational analysis can examine patterns across documented cases. These combinations do not automatically strengthen a claim: their roles and source dependence must be specified (Chapter 15).
12.8 Reporting and claim boundaries
The report should state the participation and verification process before presenting selected stories or outcomes. For MSC, it should identify the change domains, the number and source of stories, the composition of each selection panel, the criteria used, the selected and rejected stories, and the verification undertaken. For outcome harvesting, it should identify the outcome definition, the routes through which candidate outcomes were identified, who drafted and revised the outcome statements, the substantiators, the evidence reviewed, and outcomes that could not be substantiated. These details allow readers to assess the process that produced the findings.
The final claim should name the change, the actor or group concerned, the period, the evidence that supports occurrence, and the programme role that remains plausible after substantiation. It should also distinguish a verified outcome from a verified contribution. Evidence can establish that an organisation changed a practice without establishing why it did so. A contribution claim requires evidence about the programme’s activities, timing, proposed pathway, and relevant alternatives. A participatory evaluation should not turn a well-documented change into a causal conclusion merely because participants regard the programme as influential.
Quantification requires particular care. Counts of stories, selected stories, or harvested outcomes do not measure the size of a programme effect or the prevalence of a change. They may reflect the routes used to recruit participants, the number of people available to report, the ease of documenting certain outcomes, or a panel’s selection criteria. A report can give these counts as information about the process, but it should not treat them as outcome estimates without a design that supports such an interpretation.
12.9 Timing, resources, and reasons not to use the methods
MSC is most useful when the programme can collect and discuss stories over time. Repeated cycles allow selection criteria to be challenged, domains of change to be revised, and emerging concerns to inform implementation. A single retrospective story-collection exercise can still identify participant accounts of change, but it lacks the monitoring role and may depend heavily on recall. Outcome harvesting is often used retrospectively, but it also benefits from early planning: contemporaneous records, clear dates, and access to people outside the programme make later substantiation more credible.
Neither method is appropriate when participants face material risk from disclosure and the evaluation cannot protect them, when there is no feasible way to substantiate the central claims, or when the decision requires an estimated treatment effect. They are also a poor fit where participation is being used ceremonially: when the programme has already determined the questions, interpretation, and findings, while asking participants only to supply favourable stories. A descriptive consultation may still be ethically valuable, but it should not be presented as participatory causal evaluation.
The necessary resources include facilitation, translation where required, safe data management, time for verification, and independent review. The work is often underestimated because stories and outcomes can be collected quickly. The more demanding part is documenting how they were selected, checking whether the reported change occurred, tracing the claimed contribution, and retaining material that challenges the preferred account. A design that funds collection but not substantiation has chosen testimony over evaluation.
12.10 Use in a wider evaluation design
Participatory methods are particularly useful at the point where a programme theory is incomplete. They can reveal outcomes that were not anticipated by designers, identify groups for whom delivery differs, and expose assumptions that require later testing. This is a direct contribution to causal inquiry, even where the methods do not provide a counterfactual or a formal diagnostic update themselves. The next design step should follow from the uncertainty identified: a mechanism claim may require process tracing, an implementation pattern may require comparison across cases, and an outcome effect may require a counterfactual design.
The combination must preserve source independence. If participants, programme staff, and the substantiator all depend on the same programme records or organisational account, the appearance of triangulation can be misleading. The evaluation should identify where sources are connected and seek information from affected groups, independent records, or other actors when that is safe and relevant. Where it cannot, the conclusion should say that the contribution claim rests on dependent evidence.
12.11 Quality assurance during analysis
Quality assurance begins with the treatment of raw material. The team should preserve the original wording of narratives and outcome accounts, translations where used, the date and route of collection, and the changes made while an account is being condensed into a story or outcome statement. Editing is often necessary for clarity and confidentiality. It becomes a problem when it removes qualifications, alters agency, or makes the account fit the programme theory more closely than the source did. The record should show what was changed and why.
Interpretation should be separated from selection. A panel may select a story because it raises a difficult issue, is well substantiated, or reflects a valued change. These are different reasons, and they should not be collapsed into a claim that the selected story is typical or proves programme success. Outcome harvesters likewise need to distinguish an outcome that is important to stakeholders from an outcome for which the programme’s contribution is well supported. The process can report both judgements, provided it explains the basis for each.
Independent challenge is useful at three points: after the collection strategy is set, while candidate accounts are being substantiated, and before conclusions are approved. A reviewer can ask whose accounts are absent, whether an outcome statement includes more than the evidence supports, whether a named alternative has been investigated, and whether a sensitive account has been made safe to report. This is not a substitute for participant involvement. It protects participants and readers from a process in which involvement is used to shield a programme from scrutiny.
The final quality check should examine whether the chapter’s claim language matches the process followed. A story selected by a panel is a selected story. A story checked against one record is partly corroborated. An outcome reviewed by an informed but connected person has limited independent substantiation. A causal contribution may be plausible without being established. These distinctions should remain visible when evidence is summarised for decision-makers.
12.12 From findings to decisions
Participatory findings can support decisions when the decision matches the evidence. A verified account of an access barrier can justify investigation of a delivery problem. Repeated, substantiated reports of an unintended consequence can require an adaptation or safeguard. An outcome statement showing a change in organisational practice can identify a result worth tracing further. These uses do not require the evaluation to claim that the programme caused a population outcome.
The decision record should state the finding, the people affected, the evidence and its limitations, the action proposed, and the uncertainty that remains. This helps distinguish a response to credible participant evidence from a management decision made for other reasons. It also provides a basis for returning to the issue after implementation changes. Participatory evaluation has a learning function only if participants can see whether and how their accounts altered the programme’s response.
Feedback should be designed with the same care as collection. Returning selected stories or harvested outcomes to participants can correct factual errors, expose disagreement, and allow people to challenge an interpretation before it enters a formal report. It can also create risks if participants are asked to endorse a conclusion in front of those who control programme access or resources. The team should choose feedback routes that allow correction without making agreement a condition of participation.
The limits of the evidence should remain visible in public communication. A concise outcome story may be useful for a programme report, but it should not replace the record of selection, substantiation, dissent, and unresolved alternatives. Where a report uses a story to illustrate a broader finding, it should say that the story illustrates the finding rather than treating it as proof of prevalence. This is particularly important when stories are used to advocate for funding or scale-up.
12.13 Claim limits and transfer
Participatory evidence travels poorly when stripped from the conditions in which it was produced. An account of change may depend on the programme’s delivery model, the relationship between participants and implementers, local political conditions, or the participant group that was able to take part. Transfer to another setting requires an account of which of these features are relevant, and evidence that they are present or can be created elsewhere. A selected story or harvested outcome does not itself establish that recurrence.
The same restraint applies to negative findings. A report that participants did not identify a particular change may reflect the absence of change, the choice of participants, the way the question was framed, the risks of disclosure, or the timing of collection. The evaluation should investigate these possibilities before interpreting silence as a programme failure or success. This is one reason to retain a clear record of recruitment, non-participation, and the conditions of collection.
Participatory methods are a source of systematically collected, stakeholder-informed evidence about change and its possible explanation. They widen what an evaluation can observe and whose interpretation is considered. They do not remove the need to define claims, substantiate outcomes, investigate alternatives, or choose another design when the decision requires an estimate of effect.
The next chapter examines computational simulation, a supporting tool that formalises assumptions and explores their implications without collecting new empirical evidence.