TOWARDS IMPROVING THE ROLE OF EVALUATION WITHIN NATURAL RESOURCE MANAGEMENT R&D PROGRAMMES:THE CASE FOR 'LEARNING BY DOING'

ABSTRACT: The increasing use of participatory development approaches in recent years pose new challenges for decision-makers and evaluators. Because these programmes are designed to be responsive to changing community needs, one of the most pressing challenges is to develop participatory and systems-based evaluative processes to allow for ongoing learning, correction, and adjustment by all parties concerned. This paper outlines one such evaluation process, and uses a case study in New Zealand to illustrate its benefits in the light of current issues facing both evaluators and natural resource managers.

INTRODUCTION

The past decade has witnessed dramatic changes in the role that the public may play in decisions relating to natural resource management. There is now widespread recognition, in name at least, that participatory development is critical for achieving sound resource management. However, this kind of development requires a more flexible and evolving process to planning for change, and poses new challenges for decision-makers and evaluators alike. As Narayan (1993) notes, this requires major institutional reorientation at the policy level to ensure responsiveness to local demand, and to empower and enable communities to act. At the programme level, it means detailed outlines for action can no longer be drawn up at the outset since problem-solving is based on partnerships and cooperation, and not the quest to achieve some externally identified goal. Inevitably, the reality is that whatever aims are finally chosen, implementing the solutions to reach them will involve a long process of difficult dealings with a great variety of individuals, groups, and institutions who can make them fail or succeed (Mermet 1991).

One of the greatest challenges is to build mechanisms into this process to allow for learning, correction, and adjustment by all parties concerned. To do this will require the development of clear sets of objectives and indicators of success which promote accountability and participation, and which can be monitored and evaluated by the relevant decision-makers at all levels. This is the role of evaluation outlined by Cronbach et al. (1981), which holds that evaluation needs to improve the welfare of society by contributing to the political (decision-making) processes that shape social actions. In this regard an evaluation is no different to any other monitoring programme. It will pay off only to the extent that it collects the results of past actions, interprets them to provide ideas pertinent to decision-making, and enables people to think more clearly about their actions as a result (Bosch et al., 1996b). In particular, as Cronbach et al. (1981) point out, there is a need for evaluation processes that break away from concern with specific programmes, agencies, or communities to consider a social problem as a whole and the multiple lines of attack on it. This is especially true in the area of natural resource management, where resource managers must strive not only to align different perspectives, but also to engender an attitude which will support the social and institutional reform required to help communities identify and adopt sound management practices.

Against this background, this paper will use a case study to examine how the role of participatory and systems-based evaluation processes can be used to help society address the above concerns, and guide different interest groups to work more cooperatively to achieve a common set of resource goals. The Hieracium Management Programme (HMP) was initiated to address the problem of an invasive weed (Hieracium spp.) in the South Island mountain lands (high country) of New Zealand. However, as with many resource management issues, the apparent 'problem' is often most usefully viewed as a visible symptom of a 'problematic situation'. Accordingly, the programme objectives were not focussed on Hieracium as such, but rather Hieracium was treated as just one component of the wider, more complex difficulties facing those concerned with sustainable land management. The HMP set out to address this using the Integrated Systems for Knowledge Management (ISKM) participatory research approach, to facilitate the implementation of monitoring and adaptive management processes at different levels of decision-making throughout the region (Allen and Bosch, 1996; Bosch et al., 1996a&b; Allen et al., 1997). Over the past three years, the HMP has involved the active participation of a number of interest groups including farming groups, local government, scientists from different disciplines and institutes, and a range of central government funding bodies.

Because evaluation is basically an idealised problem-solving process that we use to learn about our world so we can take more informed actions (Shadish et al., 1991), this paper will begin by introducing the reader to the problem situation in which the HMP is sited. Then to provide the context for the current challenges facing evaluation, the changing ways in which society has structured its agricultural 'problems' over the past 50 years will be briefly reviewed. Within this context an example of a participatory and systems-based evaluation process which can help address these challenges, the Snyder Evaluation Model (Dick 1996), will be introduced. The paper will conclude by illustrating the use of this model in practice, and using the results to highlight a number of issues pertinent to the evaluation of natural resource management programmes today.

I. CHANGING PROBLEM CONTEXTS FOR EVALUATION

It is particularly appropriate to use an agricultural example to highlight issues in natural resource management, because as Dahlberg (1979) points out, agriculture represents the basic interface between people and their environment. From this perspective, the grasslands of the South Island high country present a number of advantages for those concerned with the improvement (or evaluation) of research and development (R&D) programmes. The high country comprises a microcosm of the major resource management issues surrounding extensively grazed ecosystems worldwide. Today, there is a pragmatic recognition of the worldwide trend towards a more holistic, multi-use, multi-value view of such extensively grazed grasslands. Grazing has increasingly become a variable component or even been abandoned in some areas, a change that highlights the diverse values that these grasslands are now expected to serve. In New Zealand these not only encompass traditional pastoral considerations but extend to national aspirations concerning issues such as indigenous Maori land rights, preservation of biodiversity and natural landscapes, sustainable management, tourism, and recreation.

Moreover, the economic and ecological sustainability of at least one-third of this region has been questioned by a recent governmental review. Concerns included land degradation, weeds (particularly Hieracium spp. - an introduced forb), pests (particularly rabbits) and the ability of farmers to manage for market and climatic variability (Martin et al., 1994). In terms of issues relating to achieving sustainable resource management, the South Island high country not only encompasses a wide range of contrasting situations, but also is increasingly characterised by conflicts over resource use between different interest groups. In addition, even as changing social and economic policies continue to shape resource development opportunities, the move away from centralised planning by government is increasingly requiring communities to deal with their own social, economic, and environmental needs on a regional basis.

At first glance, the aim of introducing an active adaptive management ethic into a rural region with the help of a programme such as the HMP may sound overly ambitious. However, while changing the value system which underlies land use practices and management may appear a daunting task, we should remember that it is something that happens quite regularly in response to different societal concerns and aspirations. Indeed, so marked are these changes in many rural areas that it seems reasonable to suggest that there have been a number of different eras of land management. Each is dominated by a different popular perception of land use, and thus the way people go about practising it (Bawden, 1991). These different eras are outlined in Table 1 as they relate to the South Island high country over the past 50 years. Of course, the way that we have generated the knowledge to address emerging agricultural problems has always been changing in a similar way, leading to new institutional approaches which are characterised by fields of enquiry which differ from those of earlier times (Rhoades 1989). However, as Bawden points out, these issues are more complicated than they appear because each emerging perspective (or world view) complements rather than replaces its predecessors, making for increased complexity. This is how learning, which embraces new assumptions about the way we know, as well as explicit new world views about how our environment could and should be treated, can lead to development being construed as proceeding in discontinuous 'spurts' or 'waves' (Bawden, 1991).

Table 1: Changing eras of land management and emerging fields of enquiry in agricultural research and development in the South Island high country (Ian Valentine, pers comm.)

Since early European settlement in the mid 1800s, extensive pastoralism has been the predominant land use in the high country. During the early years farming was in an establishment phase, with the farm focus very much on survival. However, following World War II a combination of good market conditions accompanied by the development and uptake of new technologies - aerial topdressing, trace elements, inoculated seed, etc. - ushered in a new era of production. In response to rising costs and declining terms of trade from the late 1960s, farmers increasingly looked to efficiency as well as production effectiveness. The success of land management efforts was judged almost solely in terms of simple production and economic measures, and not just by farmers. Indeed, during the late 1970s and early 1980s farmers were actively encouraged to take advantage of production technology by considerable public support in the form of incentives and subsidies.

In response to the production-oriented questions of the 1950s and 1960s, agricultural R&D efforts were based around the use of reductionist methodologies - particularly applied science - to 'fix' or improve components of farming and related resource management systems. However, as the base land resource became a limiting factor in the productivity era, researchers began to pay increasing attention to the 'whole farm' as an integrated production unit. This, in turn, led to the adoption of systems-based methodologies aimed at optimising the financial returns of each unit of production. Unfortunately, as Ison and Ampt (1992) observe, despite the growing recognition of the increasing complexity and social construction of agricultural problems in later years, there have been few recent innovations in research methodology other than the development of quantitative modelling and an increased focus on the development of expert systems.

However, in the emerging era of sustainability the questions are different. Beginning with the publication of books such as Silent Spring (Carson, 1962) we have become more concerned with the relationships between things. Public interest groups are no longer content to evaluate rural systems such as the high country merely in terms of economics and production, but are increasingly looking towards measures of ecological health, environmental ethics, and equity. Today there are a range of public pressure groups that increasingly voice their concerns about issues such as the effect that agricultural practices are having on the environment, or conflicting land uses. We also have farmers who publicly question whether they are farming 'sustainably' .... and challenge science to define the land management practices that need to be implemented to be 'sustainable'. However, one only has to consider simple questions - sustain what? how? for whom? over what time period? - to appreciate that sustainability can never be precisely defined. And as we grapple with those challenges and what they mean, we appears to need new ways of looking at the world and integrating management and research.

More recently attention has shifted towards the use of action learning and action research to go beyond what have been predominantly hard system approaches (Bawden et al., 1984; Scoones and Thompson, 1994). These soft system approaches explicitly recognise that natural resource management in the age of sustainability is not characterised so much by problems for which an answer must be found, but rather issues which need to be resolved and will inevitably require one or more of the parties to change their views (Bawden et al., 1984). However, in the main, the application of these learning-based participatory approaches within agriculture still fail to grasp the nature of the rapidly evolving social forces that are driving rural systems today. There are very few references in the agricultural R&D literature to participatory projects other than those which involve farmers and scientists dealing with agricultural management issues. Yet as communities and agriculture change, the juxtaposition of farming and other rural activities has become a battleground over water and related nutrient management issues, as well as other community impacts of changing land use (Abdalla and Kelsey, 1996). Only a decade ago, those working in the New Zealand high country were at least confident in the knowledge that they were dealing with what everyone knew was a largely extensive pastoral system. Today, whether the high country should be regarded as an agricultural, tourism, or conservation system, or some combination of all these, is increasingly problematic and contentious.

In response to these issues we are beginning to see the increased use of multi-stakeholder processes that facilitate the wide involvement of people in problem-solving and decision-making with respect to issues and plans that involve or impact on them. This multi-stakeholder approach recognises that natural resource management is increasingly characterised by apparently conflicting social perspectives, and emphasises processes to provide those involved with a better understanding of other points of view. It also appreciates that decisions related to sound land use will be dependent on the coordinated actions of many land managers and agencies, who in turn must act within the confines of a wider regulatory framework imposed by the community at large.

II. PARTICIPATORY AND SYSTEMS-BASED EVALUATION

Clearly the multi-stakeholder perspective challenges the common perception of what a 'programme' is. This perspective clearly recognises that each group of participants has its own viewpoint on the issue, and its own reasons for becoming involved in the project. As Schwedersky and Karkoschka (1994) point out, it is traditional to observe programmes within an operational cycle, from planning via implementation through to evaluation. However, to take into account the various perspectives and interests of the participants, it is necessary to look beyond this cycle. Inevitably, 'the programme' can be regarded as a number of sub-projects, each of which is 'steered' by a different group of participants in accordance with their values and aspirations. In the real world, 'cooperation' is a far more realistic goal than 'consensus' (Macadam, pers. com.). It is unlikely that groups with different interests, objectives, and values will work as members of a larger 'community' team. But with the help of appropriate participatory and systems-based processes it may be possible to help meet the different needs of those involved and develop 'win-win' strategies.

Of course, many of the participatory processes that are in use in the development field are inherently evaluative. This is particularly true for all those processes that have built on principles of action research and learning - with its iterative cycle of planning, reflection and action. However, if we are serious about guiding different interest groups to work more cooperatively towards a common set of environmental goals, then we also need processes that are explicitly systems-based. Systems thinking provides a framework for how information can be arranged and understanding developed. Rather than emphasising discrete elements and properties (e.g. roles and values), systems thinking emphasises relationships and context. These properties are ideally suited to multi-stakeholder situations where the aim is to help participants see how different activities and relations of cooperation between different parties fit together within a wider social programme.

THE SNYDER EVALUATION MODEL

The Snyder Evaluation Model is one such evaluation process which can be regarded as systems-based and participatory. This model lends itself to involving participants as co-evaluators through a three-stage process of evaluation: process, outcome, and short-cycle. Each of the three evaluation forms draw upon a systems model of how a project operates (Figure 1). Resources are consumed by activities which produce both intended and unintended immediate effects in the pursuit of ideals and objectives, which in turn are intended to contribute to some vision of a better world.

Figure 1: The Snyder Evaluation Model, showing how the different elements of a project can be seen within a general systems model.

As Dick (1996) points out, each process provides a different form of evaluation, and each builds on the understanding and insights gained during earlier stages. The activities involved in each of these processes has been summarised by Dick (1996) as follows.

The process evaluation enables the evaluator and the stakeholders to develop a better understanding of the functioning of the programme. In particular, it allows the stakeholders to understand the links between resource use, programme activities, the intended and unintended immediate effects of those activities, the predetermined objectives which are pursued, and the contribution of the programme to some overall and long-term vision.

The outcome evaluation enables the participants to apply the understanding which they develop in the process evaluation to assess which of their goals are being achieved, and how well this is being done. It also allows the development of performance indicators which can be used to set up ongoing feedback and monitoring.

The short cycle evaluation, in turn, uses the understanding and the performance indicators gained in the two earlier phases to set up the feedback loops which can be used to enable a programme to become self-improving.

PROCESS EVALUATION

The HMP provides a good example of how contemporary programmes in resource management are increasingly going to involve different interest groups and agencies working in cooperation. In order to achieve this they must have a common vision to work towards. Accordingly, the evaluation began by asking participants to generate the ideals that the programme could be expected to lead towards if it were 'spectacularly' successful. Through this exercise a list of ideals was developed and prioritised by the group. The most important of these were established community processes to help learning, accredited total quality management (TQM) farming systems, improved information flows to help decision-making, enabling legislation and, as a direct result of these, positive physical indicators of improved management. Although these ideals can be described as broad or vague - there was no mention, for example, of Hieracium - this in itself is the key to building political accommodation in such a way that different interest groups such as farmers, environmentalists, and local government can agree to work cooperatively. It also acts to ensure that the focus is not on a particular problem to be solved, rather it encourages participants to think about a problem situation to be improved, thereby broadening the range of solutions that may subsequently be suggested.

In a similar way the remaining programme elements were defined through the use of mind maps (e.g., the mind map developed for targets is shown in Figure 2), and the links between the elements are then compared and adjusted. Targets, activities, effects and resources are, of course, much more tangible than ideals, relating as they do to already defined actions. Accordingly, these elements mainly related to the shorter-term programme aim of helping address the specific issue of Hieracium - within its ecological setting in the tussock grasslands.

Figure 2: Edited version of HMP objectives mind map generated by participants during the use of the Snyder Evaluation Model.

One of the main strengths of the process lies in simply defining the different elements involved in the programme (targets, activities, and resources), providing all those present with a good overview of what all the different groups involved in the programme were doing, and how all these activities could be linked to work towards a common set of ideals. A further useful exercise, particularly when the evaluation is carried out with a diverse group, is to involve participants in describing related activities and targets that they are involved in even though these can often be regarded as separate from the programme under evaluation.

Developing the immediate effects is another particularly insightful activity, within this particular evaluation leading to useful discussion on a number of issues that relate to the use of participative processes in general. These included the positive benefits achieved by legitimising farmer knowledge, and the problems caused by the high expectations that surrounded the launch of the project within the community. It also highlighted issues that have arisen because some interest groups were not involved in the programme early enough, and the problems that 'soft' systems participatory processes pose to funders seeking to evaluate them from within the more traditional 'hard' systems perspective. Importantly, it provides an ideal opportunity for the facilitator to challenge participants with apparent inconsistencies between what people say they do, and their behaviour in practice. These activities, in turn, act to enhance the learning potential of the evaluation (Argyris, 1985).

The examination of the links between elements lies at the heart of this section of the evaluation. In this case the exercise led to a number of discussions to potentially improve the manner in which some activities are carried out, and identified new activities that need to be initiated. However, it also highlighted two major problems which face programmes looking to initiate and 'institutionalise' an adaptive management or learning culture within the community - funding and coordination. In general, there appears to be a trend among institutions to provide 'seeding' money for community-based projects, which is often justified by the belief that once the benefits become apparent, those in the community who benefit will ensure the funds to maintain the initiative. Unfortunately, in regard to natural resource management, the very issue of how costs and benefits are shared between individuals and the wider community remains contentious. In addition, there are few guidelines when it comes to creating an appropriate institutional framework that promotes the coordination of management activities undertaken by many loosely connected, but interdependent groupings and agencies (McLain and Lee, 1996). These issues are highlighted in Figure 2 where none of the ongoing HMP activities (those within boxes) have funding beyond the next twelve months, and the activities relating to the coordination and expansion of the process are conspicuously absent.

Nonetheless, as Dick (1996) points out, by the completion of the process evaluation, stakeholders should understand the links between adjoining elements. And in this regard the evaluation that participants did of the evaluation process itself bore this out. Replies to questions relating to the usefulness of the session specifically endorsed the process, particularly in relation to the understanding they had gained of how things fitted together, and surprise that everyone was in agreement with the same set of ideals. As one participant said of the day, it provided "the opportunity to all be reading from the same sheet of music". In contrast, questions relating to the least successful aspect of the day drew no criticisms of the evaluation process, but rather attracted responses to the various negative aspects of the programme mentioned above.

OUTCOME EVALUATION

If we are serious about community-based, adaptive management approaches we will require clear sets of indicators of success which promote accountability and cooperation, and which can be monitored and evaluated. As Narayan (1993) points out, the key questions managers face are what should be monitored and evaluated, and what processes should be utilised. This is particularly true in relation to participatory initiatives which are in danger of being seen in an adverse light by policy makers and funders as 'vague'. The prevalence of this perspective was both highlighted in the HMP evaluation discussed here, and is also being observed by other researchers (Anyanwu, 1988). There are a number of reasons for this, not the least of which is the open-ended approach to problem-solving inherent in participatory processes which does not sit well with conventional institutional planning processes.

During this evaluation the different parties involved appeared reluctant to develop relevant and rigorous measures of the outcomes of their own activities. In this regard, it must be acknowledged that given the current funding available to the HMP, it is hardly surprising that summative evaluation activities are not high on the agendas of those involved. Rather, given the committment from the parties involved for the continuation of the programme, much of the subsequent discussion built on the insights gained during the process evaluation phase to develop alternative funding options. Nonetheless, as other researchers have pointed out, this reluctance for rigorous summative evaluation is a common occurrence, and does pose a major challenge for evaluators. Indeed, Sechrest and Figueredo (1993) observe that it is probably not in the nature of organisations and systems to seek summative evaluations of their own activities. "The results of summative evaluation and even the rationale for doing it at all call into question the very reason for the existence of the organisations involved."

Yet we must remember that even within community-based approaches funders and policy-makers remain accountable for their actions. Moreover, communities and individuals tend to take on more responsibility within community development approaches for implementing their own solutions, and if they want to be taken seriously must also demonstrate their accountability. Clearly, this will happen only to the extent that the results of their actions can be measured and communicated to others.

Related to this issue of accountability is the controversy over the respective merits of quantitative and qualitative approaches to evaluation (Sechrest and Figueredo, 1993). Certainly, as these researchers observe, there has been a marked increase in the use of qualitative evaluation over recent years. However, as the participants in the HMP evaluation pointed out, in the end such processes must strive to demonstrate measureable improvements in the situation. Accordingly, this step of the evaluation emphasises the use of both qualitative and quantitative indicators to reinforce each other and increase both the rigour and the relevance of the evaluation. The key to this lies in developing packages of indicators to measure progress towards any particular ideal (Cronbach et al., 1981; Dick 1996). As Cronbach et al. (1981) point out, when just one indicator is used for an important outcome, the critic can plausably ask "would a different measure tell the same story?".

III. CONCLUSION: TOWARDS A SELF-IMPROVING PROGRAMME

As the participants in this evaluation are acutely aware, there can never be a final solution to natural resource problems. Evolving ecological and socio-economic systems will continue to require changes in action plans and long-term goals, requiring more than one-off evaluations. This can be well catered for by the use of the third, or short cycle, stage of the Snyder Evaluation Model which uses the indicators developed during the outcome evaluation, or some of them, as feedback on an ongoing basis. This builds ongoing evaluation into the programme's very operation, and ensures that feedback is provided to the people most able to make use of it for system improvement. Because sustainability issues need to be addressed simultaneously at a number of different levels of decision-making (Allen et al., 1997), the evaluation model also provides a means to ensure indicator packages are relevant to different system hierarchies from block/field goals through individual enterprise objectives to catchment/community goals.

This is particularly important in the area of natural resource management where the impact of policies depends in good part on the performance or reaction of people not under the direct control of any one policy maker. Timely communications are likely to be of more help to decision makers than 'final' ones (Cronbach et al., 1981). Proposals for actions are reshaped as experience is gained, and as more participants become concerned about a particular issue, cost, or benefit. In this sense, all social development activities must be seen primarily as experiments, and dealt with as complex and uncertain ventures in which the participation of those who are expected to benefit is essential (Rondinelli, 1983).

Because of this, participatory development takes time, and relies on the quantity and quality of the feedback and learning developed. As Sechrest and Figueredo (1993) point out, this requires the use of, and committment to, an iterative model of testing, feedback and revision. The example they cite required ten years of continuing commitment and support from the funding agency, and a matching commitment from the investigators.

The essence of Tharp and Gallimore's (1979) evaluation succession model is that one learns from one's mistakes. They started off their first year with what they thought was a reasonably good reading programme. However, when that programme proved unsuccessful they did not set about finding another programme to test. They asked themselves why what seemed like a good idea did not work at all, and they also asked other people (teachers, parents, even children). Then they built that feedback into the activities of the second year of the programme, etc. .... Few agencies, certainly not those of government, are likely to be in a position to make that kind of long term commitment, and few investigators would be able and willing to stick with the same project for 10 years - especially over the first several discouraging years (Sechrest and Figueredo, 1993).

Still, if we are going to develop programmes and approaches that are truly effective in our society, we must get beyond the notion of a quick fix, particularly when the expressed issue is likely to be merely a symptom of a larger underlying issue resulting from social change. This hidden nature of underlying social issues, and the myriad issues that arise as symptoms, makes it necessary for the would-be natural resource manager to help communities understand the interlinked nature of many apparent resource issues, and help them apply technical information in a larger context of shared understanding. In this context, sustainability becomes a measure of the relationship between the community as learners and their environment, rather than some externally designed goal to be achieved (Sriskandarajah and Digman, 1992). Participatory and systems-based evaluation models, such as the one described in this paper, can play a key role in this process by creating an effective learning environment for those involved.

This on-line paper is reproduced on this site with the kind permission of the Canadian Journal of Development Studies.

REFERENCES

Abdalla, C.W. and T.W. Kelsey, "Breaking the Impasse: Helping Communities Cope with Change at the Rural-Urban Interface," Journal of Soil and Water Conservation, November-December, 1996, pp. 462-466.

Allen, W.J. and O.J.H Bosch, "Shared Experiences: the Basis for a Cooperative Approach to Identifying and Implementing More Sustainable Land Management Practices," Proceedings of Symposium "Resource Management: Issues, Visions, Practice" Lincoln University, New Zealand, 5-8 July, 1996, pp. 1-10.

Allen, W.J., O.J.H. Bosch, R.G. Gibson and A.J. Jopp, "Co-learning Our Way to Sustainability: An Integrated and Community-based Research Approach to Support Natural Resource Management Decision-Making," In S.A. El-Swaify and D.S. Yakowitz, eds. Multiple Objective Decision Making for Land, Water and Environmental Management, Florida, St Lucie Press, 1998, Ch. 4. pp. 51-60.

Anyanwu, C.N., "The Technique of Participatory Research in Community Development", Community Development Journal 23, 1, 1988, pp. 11-15

Argyris, C., Strategy, Change and Defensive Routines,Boston, Pitman, 1985.

Bawden, R.J., R.D. Macadam, R.J. Packham, and I. Valentine, "Systems Thinking and Practices in the Education of Agriculturalists," Agricultural Systems, 13, 1984, pp. 205-225.

Bawden, R.J., "Towards Action Researching Systems," in O. Zuber-Skerritt, ed., Action Research for Change and Development, Brisbane, Centre for the Advancement of Learning and Teaching, Griffith University, 1991.

Bosch, O.J.H., W.J. Allen, J.M. Williams, and A. Ensor, "An Integrated System for Maximising Community Knowledge: Integrating Community-Based Monitoring Into the Adaptive Management Process in the New Zealand High Country," The Rangeland Journal 18, 1, 1996a, pp. 23-32.

Bosch, O.J.H., W.J. Allen, and R.S. Gibson, "Monitoring as an Integral Part of Management and Policy-Making," Proceedings of Symposium "Resource Management: Issues, Visions, Practice" Lincoln University, New Zealand, 5-8 July, 1996b, pp. 12-21.

Carson, R., Silent Spring, London, Hamish Hamilton, 1962.

Cronbach, L.J., S.R. Ambron, S.M. Dornbusch, R.D. Hess, R.C. Hornik, D.C. Philips, D.F. Walker and S.S. Weiner, Toward Reform of Program Evaluation, San Francisco, Jossey Bass, 1981.

Dahlberg, K.A., Beyond the Green Revolution, New York, Plenum Press, 1979.

Dick, B. Action Research and Evaluation on Line (AREOL), ftp://psy.uq.edu.au/lists/arlist/areol_intro04, Southern Cross University, Australia, 1996.

Ison, R.L. and P.R. Ampt, "Rapid Rural Appraisal: A Participatory Problem Formulation Method Relevant to Australian Agriculture," Agricultural Systems, 38, 1992, pp. 363-386.

McLain, R.J. and R.G. Lee, "Adaptive Management: Promises and Pitfalls," Environmental Management, 20, 4, 1996, pp. 437-448.

Martin, G., P. Garden, A. Meister, W. Penno, G. Sheath, G. Stephenson, R.Urquart, C. Mulcock and R. Lough, South Island High Country Review: Final Report of the Working Party on Sustainable Land Management, Wellington, South Island High Country Review Working Party, 1994.

Mermet, L., "Participation, Strategies and Ethics: Roles of People in Wetland Management," Landscape and Urban Planning, 20, 1991, pp. 231-237.

Narayan, D., Participatory Evaluation: Tools for Managing Change in Water and Sanitation, World Bank Technical Paper Number 207, 1993.

Rhoades, R.E., Evolution of Agricultural Research and Development Since 1950: Toward An Integrated Framework, IIED Sustainable Agriculture Programme, Gatekeeper Series No. SA12, 1989.

Rondinelli D.A., "Projects As Instruments of Development Administration: A Qualified Defence and Suggestions for Improvement." Public Administration and Development 3, 3, 1983, pp.307-327.

Schwedersky, T. and O. Karkoschka, Process Monitoring (ProM): Work document for project staff, Eschborn, Deutsche Gesellschaft fr Technische Zusammenarbeit (GTZ) GmbH, 1994.

Scoones, I. and J. Thompson, "Knowledge, Power and Agriculture - Towards a Theoretical Understanding," in I. Scoones and J. Thompson, eds., Beyond Farmer First: Rural People's Knowledge, Agricultural Research and Extension Practice, London, Intermediate Technology, 1994, pp. 16-32.

Sechrest, L. and A.J. Figueredo, "Program Evaluation," Annual Review of Psychology, 44, 1993, pp. 645-674.

Shadish, W.R.Jr., T.D. Cook, and L.C. Leviton, Foundations of Program Evaluation: Theories of Practice. Newbury Park, California, Sage, 1991.

Sriskandarajah, N. and D. Digman, "The Quest for Sustainable Agriculture: the Current Position in Australia," Agriculture, Ecosystems and Environment, 39, 1992, pp. 85-100.

Tharp, R.G. and R. Gallimore, "The Ecology of Program Research and Evaluation: A Model of Evaluation Succession," in L. Sechrest, S.G. West, M.A. Phillips, R. Edner, and W. Yeaton, eds. Evaluation Studies Review Annual, Beverly Hills, California, Sage, pp. 39-60. 1979.