Complexity Aware Approaches to Evaluations
Development interventions are based on defining problems in terms of deficits, highlighting a lack of skills, information, and understanding. It is problematic because they frequently lead to solutions which are not appropriate, implementable or sustainable within the specific context. Action inquiry-based methods do not rely on assumptions about ‘what isn’t’ but instead try to draw up ‘what is’ as a platform for generating solutions.
Action inquiry responses, therefore;
- Engage people on real issues that they see;
- Work to foster action on challenge assumptions which restrict the possibilities for action;
- Nurture many lines of response in a way that allows for complex dynamics to appear and inform solutions.
The approach is also based on a belief that not only is it right for local people to determine solutions to their problems but that unless they understand and own the process, any gains from capacity-building are likely to be very short-lived. Scofield Associates’ work has advanced from the complexities available in development work and the challenges associated with attribution and contribution in evaluation processes. Based on the knowledge gained while working in the Horn of Africa, Scofield Associates proposes utilization of two systemic approaches in monitoring and evaluation processes including; Participatory Systemic Inquiry and Outcome Harvesting.
PARTICIPATORY SYSTEMIC INQUIRY(PSI)
A systemic approach starts with the assumption that all issues and problems are held within a field of inter-relationships and that to create sustainable change, we need to understand these interrelationships. In development, this is the web of relations within which any of the issues that people might be concerned with, are embedded. PSI involves deep insights into the systems that are being analysed by reviewing causal relationships, systems of meaning, norms, power relationships, and social networks, among many others. The PSI process maps the different realities experienced by various stakeholders leading to an explanation for the convergence and divergence based on intervention. The method also leads to a robust creation and understanding of the theories of change, and a provision of action to be undertaken.
- Multiple Inquiry Brainstorming: While remaining cognizant of the stakeholders involved in a program as an example, the brainstorming session also opens more areas of inquiry and expansion of the stakeholder base. Additionally, the courses enrich questions developed in the thematic area of investigation. Sense-making is crucial for this to work. As data is generated from different parts of the system, the result allows for a testable outcome that incorporates triangulation to inform the further analysis.
- Starting Question Strands: This is not a process resulting in a comparison of the different groups using the same question. Distinct but overlapping inquiry questions are posed to them to identify the various issues emerging from the intervention. Different strands of questioning allow for diverse narratives to develop by ensuring the critical topics from the groups are articulated while providing a perspective to overlapping issues. The process is in stark contrast to the traditional research process that stressed on comparison using the same question to different groups.
- Seeding: Understanding the system dynamics is a critical component for the systemic approach. The perspectives from the groups can be shared with the teams to allow for cross-fertilisation of ideas during the research process. As an example, in a program involving law enforcement relationship building with the community, ideas and challenges raised by the community on specific issues may be opportunities for engagement and relationship building by law enforcement during the seeding session.
- Reporting: As a final stage, the research will collate the discussions into an analysed map depicting the relationships, impacts and linkages between variables that the intervention can or cannot control.
This method enables evaluators, grantmakers, and managers to identify, formulate, verify, and make sense of outcomes. An outcome is defined as a change in the behaviour, relationships, actions, activities, policies, or practices of an individual, group, community, organisation, or institution. Using Outcome Harvesting, the evaluator assembles information from reports, personal interviews, and other sources to document how a given program or initiative has contributed to outcomes. These outcomes can be positive or negative, intended or unintended, but the connection between the initiative and the outcomes should be verifiable. It is useful when the intervention being evaluated focuses on;
- Outcomes rather than the activities included
- The intervention implemented in a complex environment
- Real-time information about achievements is required
Outcome Harvesting Stage
- Design the Outcome Harvest: This process includes the development of a guide that includes the descriptions and the type of expected outcomes from the intervention. Additionally, the design also details the stakeholders to be involved, with the timeline and geographical areas where the search for change will occur. An example of an outcome statement:
“Development of communication platform for the community to report incidences to the police in Kamukunji sub-county in 2017.”
- Gather data for the outcome descriptions: The process depends on the defined outcomes from the implementing partners and information from an involved stakeholder. These are preliminaries that undergo a review by the implementing teams to capture all the areas required.
- Engage change agents to provide context and perspective: The implementing teams understand the goal, which is formulated as the overall theory of change for a specific program. However, each activity has a micro theory of change that provides a link to the goal of the project. Engagement with the implementing partner and the beneficiaries at this level will not focus on the activities but rather on the links the activities have to the overall goal and outcomes in the target area. These will enrich the outcome descriptions during the evaluation process. It also ensures the categorisation of the outcomes into thematic areas that reflect the needs of the client. These can include;
- level of uptake
- type of intervention and the resulting outcome
- level of intervention to the community
- short term vs long term sustainable outcome
- Substantiate: It allows for independent individuals with knowledge about the outcomes in the community to validate the data while providing perspectives on how some outcomes were achieved. It can be done through additional interviews based on the outcome descriptions developed to enhance the credibility of the findings.
- Analyse and interpret: Available data back analysis of the outcomes. This process employs the utilisation of a database that categorises the findings based on the needs of the client to provide evidence-based answers.
- Support findings: The evidence-based, actionable answers to the useful questions from the outcome harvesting process, the findings propose points for discussion to end-users through a discussion on the next steps with the clients.
BENEFITS OF THE PROCESSES
- The methods ensure a process that is driven by the community and provides insights into the dynamics and operations of the system.
- Both methods provide indications on the power relationships and their impact on interventions in the communities.
- Provides some insights into the questions surrounding attribution vs contribution as the definition and understanding of change from the community’s perspective can be supported by facts.
- Corrects the frequent failure to search for unintended results that include verifiable harvested outcomes.
- Both use a logical, accessible approach that makes it easy to engage informants.
- Both incorporate mixed methods in the collection of data.
SHORTCOMING FROM THE PROCESSES
- The process requires skill and time to identify and formulate high-quality outcome descriptions in Outcome Harvesting.
- Only those outcomes that the informant is aware of being captured. However, brainstorming session overcomes this shortcoming under the Participatory Systemic Inquiry method.
- Participation of those who influence(d) the outcomes to be harvested is crucial. Sampling, therefore, can rely on the purposeful selection of the beneficiaries.
- Starting with the outcomes and working backwards represents a new way of thinking about a change for some participants, especially in programs that do not have baselines that have a more longitudinal approach.
 United States Agency for International Development. “Defining Outcomes & Indicators for Monitoring, Evaluation, and Learning in USAID Biodiversity Programming: An USAID Biodiversity How-To Guide”. Environmental Incentives LLC. Washington DC, USA. (2016).
 Peter Clarke, Katy Oswald, “Reflecting Collectively on Capacities for Change”, IDS Bulletin 41.3, (2010).
 USAID, Complexity-Aware Monitoring Discussion Note (Brief), July 2018, https://usaidlearninglab.org/library/complexity-aware-monitoring-discussion-note-brief, accessed July 18, 2019.
 Ricardo Wilson-Grau Heather Britt, “Outcome Harvesting,” May 2012 (Revised November 2013),
 Peter Reason, Hilary Bradbury, ed. The Sage Handbook of Action Research: Participative Inquiry and Practice Burns Participatory Systemic Inquiry Reason, (London: Sage 2001).
 Stead, David, improving project success: managing projects in complex environments and project recovery. Paper presented at PMI® Global Congress 2010—Asia Pacific, Melbourne, Victoria, Australia. Newtown Square, PA: Project Management Institute.
 Danny Burns, Systemic Action Research: A Strategy for Whole System Change. (Bristol: Policy Press, 2007).
 Torbert, William. The Practice of Action Inquiry, https://www.researchgate.net/publication/247951312_The_Practice_of_Action_Inquiry. (2001)