learningforsustainability.netPlanning & evaluation: scope, scale and intensity
Understanding the scope and scale of the activity which we intend to measure is an important consideration in deciding and evaluation. This page sets out some different frameworks that provide a starting point for thinking about how deeply we want to look for outcomes and results. The first of these, Targetting Outcomes of Programs (TOP), encourages us to look at how well we are doing activities that involve people, such as workshops and trainings. The three rings approach reminds us that our projects rarely achieve societal change in and of themselves, they only contribute within a wider network of projects and influences. Systems-based evaluations provides a framework for how information can be arranged and understanding developed. Rather than emphasising discrete elements and properties (e.g. roles and values), systems thinking emphasises relationships and context. These properties are ideally suited to multi-stakeholder situations where the aim is to help participants see how different activities and relations of cooperation between different parties fit together within a wider social programme. And finally the order of outcomes framework enables us to acknowledges that most sustainable development projects will only come to fruition in the long termand provides a pathway that shows how our progress towards increasingly sustainable forms of development will be monitored and evaluated. This framework offers a sequence of tangible outcomes which, if pursued successfully over long-term time periods, can indeed move us towards increasingly sustainable positions.
Targeting Outcomes of Programs (Bennett's hierarchy)
Targeting Outcomes of Programs (TOP) focuses on outcomes in planning, implementing, and evaluating programs. TOP is based on Bennetts hierarchy - a theoretically sound framework that has been tested, revised and refined, and widely used over the past 20 years to more closely integrate program evaluation within the program development process. TOP uses this simple framework to target specific outcomes in program development and then to assess the degree to which the outcome targets are reached. This framework stops us just evaluating activities, and reminds us that we must look for evidence that the activities (workshops, fieldays, etc.) are having an impact. This form of evaluation is summative, action orientated, and utilisation focused.
|
Level
|
Description
|
Summary
|
|
7
|
End results
|
Achieving and end result of economic, social, environmental/health value
|
|
6
|
Practice change
|
Behavioural changes in target group (direct/proxy evidence)
|
|
5
|
KASA
|
KASA changes: Knowledge - Attitudes - Skills - Aspirations (proxy indicators for behaviour change)
|
|
4
|
Reactions
|
Reaction to involvement (e.g. one-page survey)
|
|
3
|
Involvement
|
Involvement of different stakeholders (participation, monitoring, etc.)
|
|
2
|
Activities
|
Implementation of activities (activities monitoring)
|
|
1
|
Inputs
|
Programming resources to support program activities (inputs monitoring)
|
For more information: Targeting Outcomes of Programs 
Three rings of extension evaluation
The three rings approach (see below) is a framework presented in a paper by Jeff Coutts at the 2nd Australasian Pacific Extension Conference in November 1997: i) the inner ring is the project process, activities and outputs; ii) the middle ring comprises of the communities being directly affected by the extension intervention - with the recognition that other agencies and programs are also having an impact in this ring; iii) the outer ring represents the broader community/society where the impact can be measured at a macro level.
RIRDC Publication No. 98/136]
The inner ring - internal project level: According to this framework, the internal project level looks at the internal process and activities of the project over which the project team has some control. This level is similar to Bennett's levels 1-3 and includes activities, participation, inputs and outputs.
The middle ring - the direct project impact level: Coutts (1997) explains that as the second ring is entered the level of complexity and potential for confusion rises significantly. It becomes difficult to attribute the effects of the project intervention directly to the changes occurring, as the changes could be influenced by other programs of events. He suggests that a 'synergy matrix' be developed to highlight the relationship between the program being evaluated and other projects, agencies or groups impacting on the outcomes of the program.
The outer ring - overall industry, community or societal level: Coutts explains that most projects cannot lay full claim to observable outcomes at this level. The emphasis here is on measurable changes in tangible benchmarks, for example levels of adoption, levels of sustainable indicators, net farm incomes, and indicators of product quality. He states that due recognition should be given to the role played by other complementary and competing interventions, encouraging and inhibiting government policies, climatic, market and economic conditions. Describing these provides an overall context for the evaluation of a specific program or project. The role that the program is playing in contributing towards change is clarified, and the levels of changes in benchmarks are put into a realistic timeframe.
Coutts (1997) encourages the evaluator to assess the extent to which the project activities are complementary to, or adding value to, the knowledge processes provided by other parties, and how well these specific knowledge processes are being undertaken. This process can help to identify gaps in the provision of information or other support, and to reduce duplication of effort by different agencies.
Systems evaluation
Of course, as Allen (1997) points out many of the participatory processes that are in use in the development field are inherently evaluative. This is particularly true for all those processes that have built on principles of action research and learning - with its iterative cycle of planning, reflection and action. However, if we are serious about guiding different interest groups to work more cooperatively towards a common set of environmental goals, then we also need processes that are explicitly systems-based. Systems thinking provides a framework for how information can be arranged and understanding developed. Rather than emphasising discrete elements and properties (e.g. roles and values), systems thinking emphasises relationships and context. These properties are ideally suited to multi-stakeholder situations where the aim is to help participants see how different activities and relations of cooperation between different parties fit together within a wider social programme (Allen 1997).
The Snyder Evaluation Model is one such evaluation process which can be regarded as systems-based and participatory. This model lends itself to involving participants as co-evaluators through a three-stage process of evaluation: process, outcome, and short-cycle. Each of the three evaluation forms draw upon a systems model of how a project operates (see figure below). Resources are consumed by activities which produce both intended and unintended immediate effects in the pursuit of ideals and objectives, which in turn are intended to contribute to some vision of a better world.
As Dick (2006) points out, each process provides a different form of evaluation, and each builds on the understanding and insights gained during earlier stages. The activities involved in each of these processes has been summarised by Dick (1996) as follows.
The process evaluation enables the evaluator and the stakeholders to develop a better understanding of the functioning of the programme. In particular, it allows the stakeholders to understand the links between resource use, programme activities, the intended and unintended immediate effects of those activities, the predetermined objectives which are pursued, and the contribution of the programme to some overall and long-term vision.
The outcome evaluation enables the participants to apply the understanding which they develop in the process evaluation to assess which of their goals are being achieved, and how well this is being done. It also allows the development of performance indicators which can be used to set up ongoing feedback and monitoring.
The short cycle evaluation, in turn, uses the understanding and the performance indicators gained in the two earlier phases to set up the feedback loops which can be used to enable a programme to become self-improving.
Orders of outcomes
Sustainable development is the stated goal of many integrated (especially natural resource) management initiatives. Everyone acknowledges that this such projects will only come to fruition in the long term, yet typically very little is said about how progress towards this long-term objective is to be achieved. Equally little is said about how our progress towards increasingly sustainable forms of development will be monitored and evaluated. One approach which is making progress in this direction is the "order of outcomes" framework. This offers a sequence of tangible outcomes which, if pursued successfully over long-term time periods, can indeed move us towards increasingly sustainable positions. More detailed information on these frameworks can be obtained from Olsen (2003) Frameworks and indicators for assessing progress in integrated coastal management initiatives and Olsen et al. (2006)Ecosystem-based management: Markers for assessing progress. UNEP/GPA, The Hague.
These outcomes frameworks have adopted the term 'orders' to convey the selection of indicators that suggest a high degree of aggregation. First Order outcomes are the societal conditions that must be present when a program embarks upon a plan of action designed to modify the course of events in a coastal ecosystem. Together they form the "enabling conditions" that are required if coastal management policies, plans and actions are to be successfully implemented. First Order outcomes require building the constituencies and the institutional capacity to undertake integrated coastal planning and decisionmaking. First Order outcomes also require securing the authority, funding and other resources that make it feasible to implement policies and actions at the scale of large ecosystems. The setting of unambiguous goals is the final ingredient of the enabling conditions that together set the stage for the successful implementation of an ecosystem management policy and plan of action.
Second Order outcomes are evidence of the successful implementation of a coastal management program. They mark changes in the behavior of organizations and user groups. These include evidence of new forms of collaborative action among institutions, investments in infrastructure, and the behavioral changes of resource users in response to regulations and through their own voluntary actions.
Third Order outcomes are the socio-economic and environmental results that define the ultimate success or failure of the program. These need to be defined in unambiguous terms early on in any coastal management process, as vague or conflicting goals produce inefficiency and ineffectiveness. The sequence of these outcomes highlights the time needed to bring about socio-economic and environmental change - which can realistically be in the order of 5-10 year time frames, or longer.
Figure: The four orders of outcomes in integrated development initiatives
The ultimate goal of sustainable forms of coastal development (Fourth Order outcomes) is today an undefined ideal. It embodies the concepts of balance between competing needs and the moral imperative of not taking actions today that will reduce the ability of future generations to meet their needs.