"Stories can provide a powerful means of learning and gaining understanding by affording a context for insights into what one has personally not experienced." (Mattingly and Garro, 2000)
When talking about structural change or change at a systems level, development organizations often turn to the medium of advocacy which has, over the years, become an important piece of the complex puzzle. Various organizations and foundations are investing more resources in advocating for different programs and policies that may transform the political, social, and economic conditions that shape and are shaped by development, which itself is a complex process with no single and guaranteed to-do manual. These efforts, comprising of several interrelated actions and actors, could range from increasing community awareness at local levels to creating pressure to push for reform and influencing law and policies at the highest level- in no particular order. Though there is no single formula to effective advocacy, it is still necessary to understand 'what works, for whom and why' in order to foster an environment of cross-learning. To learn from, adjust to, share the knowledge and be accountable to institutions and organizations funding the advocacy activities as well as the communities supporting these activities, monitoring and evaluation of advocacy efforts becomes essential! But while achieving targets set for advocacy efforts is a difficult task, evaluating the different strategies is even harder.
An entangled landscape of non-linear and complex processes
Advocacy efforts operate in an ever-changing political environment which consist of diverse factors which may enable change or act as a barrier. As described by M van Wessel (2018), advocating for a change is therefore like shooting a moving target- here, the targets being policymakers, public, and private sector actors- all subject to numerous influences. Not only this, but these efforts are also multi-level, multi-actor and multi-sited with cross-linkages which only adds to the complexity of both, the strategies, and the process of change. Also, since change or outcome is difficult to achieve during a short period of time, the achievements which may be a result of advocacy efforts are often deemed too small in comparison to the outcomes expected. These achievements are often hard to trace and may be a result of indirect efforts which often go unnoticed (Arensman and van Wessel, 2018). Given that the advocacy efforts are operating in a chaotic environment of political agenda, it is a key requirement that the advocacy initiatives be inherently nimble, flexible, and opportunistic. Thus, the trajectory of change, unlike in case of service-delivery programs, is not linear and consists of an emergent nature of strategies (Teles and Schmitt, 2011). However, the same strategy could produce very different results in a different context and environment.
Given the tangled landscape of advocacy, a formalized and objective evaluation methodology, while useful, may not be able to map the ever-evolving trajectory of the advocacy efforts and the ecosystem in which they take place. All too often, monitoring and evaluation is based on pre-determined indicators that serve to measure attainment of and/or compliance with project goals, rather than being focused on learning about the workings of the system in a way that can facilitate understanding of the effects of the work on the system and inform adaptive management to improve outcomes. M&E specialists have, over time, realized the complex nature of the process of change and have thus adopted the Theory of Change approach which has been praised for being helpful in “articulating assumptions and pathways of change and in encouraging dialogue and adjustment over time, both as programs develop and in evaluation� (Stein and Valters, 2012). However, the feasibility of testing a theory, especially in case of advocacy becomes debatable as there is little to draw for purpose of evidence-based strategy development (van Wessel, 2018). To add to the contention against ToC, there is also a problem of establishing causality in case of advocacy efforts i.e., the difficulty of teasing out causal mechanisms that cause a change and assess their absolute result. Most work that has used Theory of Change as an approach focuses on gathering information about achievements rather than on the mechanisms of change (Jones, 2011). ToCs do not explicitly integrate strategies which is critical to advocacy because of their unidirectional nature which links an outcome to a certain strategy as specified.
Unentangling complexities- one story at a time.
Given the nimble and opportunistic nature of advocacy initiatives where wins might not be as envisaged, but may create ripples in the ecosystem, there is a need to document stories of 'quick and small wins' that are outcomes of advocacy strategies that occur along the path to primary envisaged outcomes. Also, while using advocacy evaluation frameworks which are output oriented, it becomes difficult to account for the networks built and the nature of the same. Funding agencies and evaluation frameworks often use a straightjacketed approach to assess the organizational capacity of implementors without taking a holistic account of their history, the salience of their past projects and their relation to the current work in terms of networks and strategies. The dynamic complexity of advocacy processes with envisaged outcomes being hard to operationalize & measure, calls for new and innovative methods that can be used to assess advocacy efforts- one of which is that of integrating stories into the theory of change, or rather 'theory of changing'.
Using stories to assess advocacy and chisel the learnings along the way, involves a 'systematic co-construction of the program's journey/ trajectory' between advocates and the evaluators, wherein the former tells the story while the latter assess the detail, evidence to co-develop, embeddedness in the context, and the plausibility of the story against the existing alternative explanations (van Wessel, 2018). Understanding the program as one existing within the organizational as well as the social and political ecosystem, requires asking questions which link it not only to the present but also past efforts of the advocacy organization and the various other actors in the ecosystem. Also, mapping the journey of the program through stories aids in moving away from a static theory of change and reflecting on a theory of chang(ing) which is adaptable. This adaptability allows for understanding meaning and relevance for programs and their stakeholders, as a journey over time. Some existing methods used to monitor and evaluate programs employ such stories such as 'Performance Story Reporting' (Roughly and Dart, 2009) and 'Most Significant Change' (Dart and Davies, 2005). However, these methods, while capturing the perspectives of beneficiaries, do not focus on capturing the insider stories i.e., stories of those running the programs, thus calling for methodologies that appreciate the value of knowledge of advocacy practitioners on advocacy processes.
Since no successful advocacy effort is the result of any one organization or initiative, capturing the stories helps evaluators apply a relational approach and complexity thinking through focusing on interconnections, patterns, and change pathways. Stories help revealing the different agents, competing factors, barriers as well as alternative explanations related to program achievements and setbacks. Even within the program, outcomes are often connected to each other i.e., different outcomes contribute to change over time, with outcomes building on previous outcomes (Teles and Schmitt, 2011; van Wessel, 2018). Stories can help capture these connections by drawing on the perspective of the different stakeholders rather than trying to objectively measure the progress. Tracking the development of the program through stories also helps in spotting leverage points in a system at which a small shift in one factor can produce widespread changes. This exercise also allows for understanding the development of the relationships which contribute to the success of the program or those that hinder the same i.e., if and how enablers become barriers or barriers become enablers during the journey of the program.
Through critically analyzing the assumptions underlying the theory of change, the changing ecosystem in which advocacy efforts take place and the capacities of different agents to adapt to the dynamic environment- stories contribute to a feed-forward mechanism wherein identifying the forces which inhibit or assist change/ win, can help in overcoming or reducing impact of forces that work against and leveraging forces that work in favor of achieving program objectives. Drawing on lessons from what worked during the course of the program, for whom did it work and why, can contribute to better and targeted strategizing. Also, recording stories of how advocacy efforts navigate and negotiate with different stakeholders in the ecosystem and how strategies are adapted to suit the zeitgeist, can build a repository of knowledge for other similar advocacy efforts to learn from. As stated by Wessel (2018), stories go beyond answering 'how something happened'- they also answer another very important question which is 'why it matters'- this answer, though based in theory and directed at the future, helps convey how a particular program sets the stage for further desired change. This helps in keeping stakeholders on board and attracting new stakeholders and investment.
While stories are helpful in assessing the progress of something as complicated as advocacy, it is important to keep in mind some limitations of using this medium for the purpose of evaluation. Connecting certain activities to outcome through stories may at times be driven by bias which clouds the judgement of the evaluator. Also, stories do not assert certainty- they only try to examine the plausibility of the story which makes it difficult for evaluators, advocates, and other stakeholders to come up with absolute conclusions which become a barrier to long-term funding.
Given the entangled landscape of advocacy evaluation, many evaluators have concluded that there is no one measure appropriate to evaluate the same. However, there is a felt need to evaluate advocacy efforts and learn from them and the medium to do the same must allow room for adaptability; one which is able to capture relationships, patterns of influence and the politics of advocacy. Narratives or stories are one such medium- an "explanation beyond the truth of events themselves; it 'is not simply knowledge about particular events, practices and ideas, but about the processes by which these come to appear meaningful, perhaps inevitable or mandatory, possibly contestable or even mad" (Hastrup, 2004 cited in Mosse, 2006).
References:-
Arensman, B. & van Wessel, M. (2018), Negotiating effectiveness in transnational advocacy evaluation. Evaluation 24(1): 51-68.
Davies, R. & Dart J. (2005), The Most Significant Change (MSC) Technique. A Guide to its Use.
Jones, H. (2011), A Guide to Monitoring and Evaluating Policy Influence. London: Overseas Development Institute.
Mattingly, C., & Garro, L. C. (2000). Narrative and the cultural construction of illness and healing. Univ of California Press.
Mosse, D. (2006). Anti�social anthropology? Objectivity, objection, and the ethnography of public policy and professional communities*. Journal of the Royal Anthropological Institute, 12(4), pp. 935-956.
Roughly, A & Dart, J. (2009), Developing a Performance Story Report: User Guide, Commonwealth of Australia.
Stein, D. & Valters, C. (2012), Understanding Theory of Change in International Development. London: LSE.
Teles, S. & Schmitt, M. (2011), The Elusive Craft of Evaluating Advocacy. Stanford Social Innovation Review. Available at: https://ssir.org/articles/entry/the_elusive_craft_of_evaluating_advocacy
Van Wessel, M. (2018). Narrative Assessment: A new approach to evaluation of advocacy for development. Evaluation, 24(4), 400-418.
