Using mixed methods to support planning, evaluation and learning

This post and inline presentation shine a light on methodologies that help us combine reflective practice and adaptive management. The post and presentation begin with some ideas and concepts that introduce adaptive management and social learning, and then move to outline how these approaches can be supported by a  number of planning and evaluative methods and methodologies.

Figure 1: Inline presentation – Adaptive management: Using mixed methods to support planning, monitoring, evaluation and learning (PMEL). This presentation is also available as a PDF download.

Over the past decade, the challenges facing those who have the responsibility for making sound social and environmental decisions have multiplied. Common to many reviews pointing to good governance and management practice are elements of equity and social justice, stakeholder inclusion, strategic thinking, accountability and fairness. However, this kind of development requires a more flexible and evolving process to planning for change, and poses new challenges for decision-makers, managers and evaluators alike. In response to these issues we are beginning to see the increased use of multi-stakeholder processes that facilitate the wide involvement of people in problem solving and decision making with respect to issues and plans that involve or impact on them. This multi-stakeholder approach anticipates that the complex challenges we are trying to address are increasingly characterised by apparently conflicting social perspectives, and highlight the need for participatory and collaborative processes to provide those involved with a better understanding of other points of view.

Adaptive management

Adaptive management is one such approach that sees planning, monitoring and evaluation as important and mutually reinforcing. Adaptive management is a dynamic approach which aims to more closely link science, management and policy  to help address complex social and environmental issues. One of the greatest challenges is to build workable mechanisms into this process to allow for learning, correction, and adjustment by all parties concerned. To do this will require the development of clear sets of objectives and indicators of success which promote accountability and participation, and which can be monitored and evaluated by the relevant decision-makers at the different levels involved. At base, these efforts  represent a search for a practice model which combines the features of:

• management-based experimentation and innovation
• complex system management, often on scales larger than individual enterprises
• methods for bringing about capacity for action among multiple agencies and actors (with typically divergent, not to say antagonistic points of view and interests)
• facilitation of the social processes and organisational capacity to accomplish these.

Adaptive management thus focuses on learning and adapting, through partnerships of policy-makers, managers, scientists, and other stakeholders who learn together how to create and maintain a sustainable and regenerative practice. As a model it helps managers maintain flexibility in their decisions, knowing that uncertainties exist and so provides the latitude to adjust direction to improve progress towards desired outcomes. One of the challenges for collaborative planning approaches is the need to start by developing some agreement on shared values (although shared values do not need to correspond to shared worldviews or even political and economic philosophies), a shared vision and defined goals. Without these the process can be quickly reduced to a few lowest common denominator decisions.

Adaptive management is both a process and a product. It involves practitioners and stakeholders in a facilitated process of analysis and reflection. At the same time it still develops narratives and plans to provide a guiding framework for the project/programme team and stakeholders. It is not a one-off exercise to be used in the design (or evaluation) phase of a research and development initiative, but implies that those involved are entering into an ongoing process of learning and adaptive management that continues throughout the life of the initiative. The process can usefully be seen to include a number of key steps. Common among these are the following:

• Clarify purpose, scope and scale.
• Describe the programme. Agree a Theory of Change and accompanying logic models (at appropriate scales).
• Agree evaluation priorities and key evaluation questions. Include room for the use of complexity aware approaches as well to take account of surprises and unintended effects (both good and bad).
• Develop evaluation criteria and performance standards (e.g. rubrics – encouraging the use of “sets of indicators”).
• Gather, assess, and reflect using (agreed) indicators and measures and appropriate evaluative methodologies.
• Implement plan-act-reflect cycles as adaptive management.

Mix and match across methods for reflection and learning

So we see that adaptive management refers to a dynamic management process. In this process goals are set, activities are undertaken to achieve those goals, results are measured and assessed –  and activities are adjusted based on what has been learnt along the way. However, when adaptive management initiatives fail it is often because the governing institutions applying adaptive management are hierarchical and prescriptive – rather than adaptive and flexible.  Prescriptive approaches often look for one or two methods that enable the process, rather than appreciating how a range of methods can be usefully mixed and matched to support and guide staff and stakeholders to learn their way as they go.

Theories of change (ToC) are vital to programme success for a number of reasons. Programmes need to be grounded in good theory. By developing a theory of change based on good theory, managers can be better assured that their programmes are delivering the right activities for the desired outcomes. And by creating a theory of change programmes are easier to sustain, bring to scale, and evaluate, since each step – from the ideas behind it, to the outcomes it hopes to provide, to the resources needed – are clearly defined within the theory.

Logic models are narrative or graphical depictions of processes in real life that communicate the underlying assumptions upon which an activity is expected to lead to a specific result. They generally illustrate a sequence of cause-and-effect relationships, i.e. a systems approach to communicate the path toward a desired result. The model describes logical linkages among programme resources, activities, outputs, and audiences, and highlights different orders of outcomes related to a specific problem or situation. Importantly, once a programme has been described in terms of the logic model, critical measures of performance can be identified. In this way logic models can be seen to support both planning and evaluation.

Rubrics are an evaluative tool for supporting learning, assessment and professional development. They offer a process for defining and describing the important components of complex tasks and behaviours. They can then help us assess these tasks or behaviors, and make evaluative judgments about about the quality of performance and effectiveness. Developing rubrics helps clarify the expectations that people have for different aspects of task or behavior performance by providing detailed descriptions of collectively agreed upon expectations.

Of course, in complex settings programme results can be difficult to predict. Complexity-aware monitoring is a type of complementary monitoring that is useful when results are difficult to predict due to dynamic contexts or unclear cause-and-effect relationships.  Common approaches include stakeholder feedback, process monitoring of impacts, most significant change, and outcome harvesting.

After Action Reviews are a form of group reflection; participants review what was intended (activity aims), what actually happened (intended and unintended outcomes), why it happened and what was learned in order to improve the process. The importance of reflecting on what you are doing, as part of the learning process, has been emphasized by many reviewers. Building the capacity to reflect on action so as to engage in a process of continuous learning is increasingly seen as an important aspect of behavior change, and it is beginning to be used in many models of changing professional practice. However, it is not a conscious behavior for many teams, and effort needs to be put in to provide teams with tools that can support reflection. These tools are usually known under names such as After Action Review (AAR) or Learning Debriefs, and are used by to capture the lessons learned from past successes and failures, with the goal of improving future performance.

Concluding thoughts

By creating useful feedback cycles between planning and action, the process of adaptive management can build team and organizational capacity to succeed in a variety of conditions. In a fast-changing environment, the capacity to learn lessons and adapt can be more valuable than any individual lesson learned. That capacity –  which can be used for adaptive management or for innovation –  is what can be gained by more closely linking outcome planning and learning-based reflective activities. Because we are dealing with complex issues that arise from settings with many stakeholders with differing views, responsibilities, and knowledge about the system (including science, management agencies and people making decisions on-the-ground), adaptive management has to also be about how to bring people together. Particularly it is about helping people work collaboratively – bridging disciplines, knowledge systems and cultures. By keeping these concepts in mind we can aim to manage more interactions within participatory and learning-based contexts to help those involved to engage in social learning and develop a shared understanding around goals, actions and indicators.

---

More information on related approaches, tools and methodologies can be found through the linked LfS pages  on systems thinking, design thinking and systems thinking tools. Other pages point to resources on  topics such as guides to help initiate and manage multi-stakeholder processes, managing participation – including marginalized voices,  facilitation tools  and reflective practice. Allied topics include  supporting constructive practice change, strategic planning and scenario development.