Managing virtual teams and (increasingly) staff that work from home

The move to home-based working involves learning about different work practices, modes of communication and team dynamics.
(c) Can Stock Photo / tab62

Virtual teams have become common in recent years. Remote work allows businesses and startups to compete in an increasingly globalized society, and provides new freedoms for employees, freelancers and consultants to work with clients and colleagues – less constrained by geographic boundaries. Now, with travel restrictions and ‘social distancing’ strategies adopted by many organisations in the face of the coronavirus (COVID-19) disease, many more companies are encouraging staff to work from home – significantly fast-tracking this remote working trend.

Recent Gallup research shows there can be significant benefits from having staff working remotely (at least some of the time). For example, engagement climbs when employees spend some time working remotely and some time working in a location with their co-workers. Research also indicates that remote workers can be more productive than on-site workers. Indeed, it’s probable that, between the higher engagement and increased productivity of remote work, having off-site workers offers an untapped potential for maximizing business outcomes. We (the authors) have ourselves worked for over ten years as researchers, consultants and facilitators outside of mainstream office environments – and with clients and colleagues across the country and internationally.  In our experience, while it is good to mix this up with face-to-face interactions there are few areas of our work that can’t be successfully progressed using online networking and collaboration. However, for those new to this it does involve learning about different work practices, modes of communication and team dynamics.

A number of practitioners are sharing tips, guides and best practice in this area and we provide links to relevant posts and other on-line guides that we have come across through the related  Managing virtual teams, Managing virtual meetings and Facilitation tools and techniques site pages.  Among these, three strategic areas for improving online collaboration stand out: i) clarifying roles, goals and processes; ii) communication and engagement; and iii) online collaborative tools.

Clarify roles, goals and processes

Teams form to achieve specific goals which require collaboration and a mix of different skills and competencies. Importantly, teams can only function effectively if goals are clear and measurable. As Heidi Gardner and Ivan Matviak remind us, “the move to home-based working is a great opportunity for a team to revisit the basics in order to ensure everyone understands the team objectives, their individual roles, and how each person contributes to the outcome.” At the same time take the opportunity to clarify task design and the social and technical processes that can help complete them.

The disruptive nature of coronavirus will itself lead to new and sometimes competing tasks for teams and organisations. So teams need to continually re-clarify goals and outcomes at both the individual and team level. A trap in such a dynamic environment can be to develop an ever-expanding list of activities, so make sure that specific actions that emerge from the meetings are documented and clearly assigned to those involved. Talking about roles and skill-sets among your team will also help members to understand when they can turn to their colleagues instead of just you as the leader and helps build a sense of trust among teammates. Relying only on formal leadership to drive the direction and commitment of the team is less efficient than encouraging a shared involvement.

It is also worth remembering that team goals include both formal outcomes (the ones the team has most likely been set up to achieve) and informal outcomes (those that the individuals bring to the team or that the team itself has developed for its members). SMART is an acronym standing for Specific, Measurable, Attainable, Realistic and Timebound. Defining outcomes and their assessment by these criteria helps the team focus efforts, stay motivated, meet deadlines, have control and prioritize tasks.

Communication and engagement

While communication always underpins good team culture and performance, it is even more important for those working in virtual and distributed settings. Team leaders should be aware not only of the need to document decisions and related information, but also the importance of reaching out in a positive and personal way to team members. Some business literature has identified that workplace isolation can be real risk for employees who work in a different location from their manager or colleagues – which lowers productivity and engagement. On the other side of the fence (or the Internet), for managers unused to working through virtual teams there is a risk of becoming excessively task-focused. According to a study by Joseph Grenny and David Maxfield, the most successful managers are good listeners, communicate trust and respect, inquire about workload and progress without micromanaging, and err on the side of over-communicating, all while modeling the same behaviors for others on the team.

People interact more constructively when they are comfortable with each other. Take time at the beginning of each larger meeting to check-in with everybody – invite team members to take a minute and go around to talk about what’s going on in their lives  – professionally and personally. Go first to model how to do this in terms of tone and content. Use the opportunity to encourage team members to share something of their home work environment. This means that others will better understand the different situational factors that make each remote office setting unique. Maybe someone has demanding pets, young children coming home from school, or noisy trucks in the background.  Getting your team to understand each other’s context helps build relationships, and a team culture that will be more open to engagement and collaboration.

If your staff and colleagues see that you are modeling effective communication behavior they will follow your example. Setting a regular recurring time for the team to come together virtually via video conference provides a good base on which to build other communication initiatives. It’s always easier to cancel if a meeting isn’t needed, than it is to set up a team talk at the last-minute. And remember to pay attention to ensure that virtual meeting time is shared across the whole team – limiting those who like talking, and encouraging those who are often quieter.  If working virtually is new to your team take the time to collectively agree on some communication protocols. This can include guidelines about how people should engage in virtual meetings, and when to use different forms of communication (emails, phone, texts, video-conference, etc.). Be sure to not just rely on whole-team meetings, and take time to check in on people individually through phone or video calls. Encourage smaller work catch-ups between smaller team groups to progress shared tasks and activities.

On-line collaboration tools

Holding virtual calls and meetings may not provide the same rich experience that can be gained from  face-to-face communication. However, catching-up virtually can provide provide many enjoyable freedoms that more formal meetings in person can never provide – you choose time and place and pace of your work, when to have a break, and interruptions by your cat, dog, or partner won’t matter at all (if you don’t mind). It will take some time to “get this right” with your colleagues because there is no formal education about this kind of work yet, anywhere.

If this represents an early foray into communicating virtually then its probably not necessary to pilot a whole lot of new software. You probably have the tools for it already: your mobile phone, email, a shared file hosting portal (Google Drive, Dropbox , Microsoft OneDrive, etc.) and any video conferencing  software (Zoom, Skype for Business, Google Hangouts, etc.) will get you started. As you move deeper into distributed work practices then you may want to support asynchronous team communication (e.g. Microsoft teams, Slack), and collaborative whiteboarding and canvases (e.g. MURAL). When you want to explore a wider range of collaborative tools such as chats, online forums and project management software visit sites like Robin Good’s Online Collaboration Tools or Matt Stempeck’s It’s time for next-generation remote collaboration.

For many managers and teams the transition to working remotely may be challenging. But while the coronavirus, like other disruptions, will pass, the trend to remote working is unlikely to disappear. In the end, learning how to work more collaboratively and effectively online is likely to open up a world of new possibilities for the future.

[If anyone wants help getting started with virtual teams or online meeting management feel free to get in touch with us [Will Allen ( or Margaret Kilvington (]

This post provides an introduction to the wealth of links on the related pages – Managing virtual teams, Managing virtual meetings and Facilitation tools and techniques. Other closely related sections in the site include the Team building, communities of practice (COPs) and learning groups and Cross-sector partnerships and collaborations. The risk communication and engagement page also has links to a number of resources which are also aligned to responding to coronavirus.








An introduction to systems thinking and tools for systems thinking

This slide set provides an introduction to systems thinking and tools for systems thinking. It covers 5 main topic areas:

  • Introduction: Why decision-makers are moving towards a systems thinking approach to better deal with complex situations in health, environment, education, etc.
  • Basics of systems thinking
  • Systemic design – linking systems thinking and design
  • Introduction to range of tools/methods that support systems thinking and systemic design in practice
  • Nurturing & supporting systems thinking in your practice



Six ways that agencies can influence behaviour through public policy

Involving different groups and sectors in in policy design can build motivation and engagement in the “common enterprise” of generating positive outcomes’. (c) Can Stock Photo / stanciuc

Often agencies and policy makers become reliant on a handful of approaches that they use to influence behaviour. This post introduces the 6 E’s framework as a a framework for agencies wanting to utilise a broader suite of policy approaches to more effectively encourage different groups and sectors to be mobilised, supported and encouraged to participate in the “common enterprise” of generating positive outcomes’. It links policy development with systems thinking and adaptive management.

There is a growing range of complex policy areas, so-called ‘wicked’ problems, where it has become increasingly clear that government cannot simply ‘deliver’ policy outcomes to disengaged and passive sectors and publics. In these areas (e.g.  natural resource management, public health, poverty, education, the environment) it is clear that achieving significant progress requires the active involvement and cooperation of  a great number of groups and individuals.Yet all too often we approach these issues with linear and even siloed solutions that aren’t sufficient to address the problems at the scale at which they exist. Successful outcomes from addressing these ‘wicked problems’ increasingly depend on the coordinated actions of decision-makers at different levels – from individuals and communities to enterprise and sector organisations, to region and nation (policy agents). This is where concepts such as partnerships and co-design come into play. By sharing information, resources, activities, and capabilities we can achieve things together that we could never achieve alone.

Policy as a co-designed activity

Even where the issue is not necessarily seen as complex, it is often crucial that a range of stakeholders and publics get actively engaged to achieve desired outcomes . This recognition has led to a focus on co-design where the achievement of outcomes is seen as a joint responsibility of the government, key stakeholders and the wider community. While many policy drivers are still evolving from the mass production model (often regulation and legislation) of earlier times, there is growing interest in focusing beyond the internal workings and efficiencies of existing services, and looking more closely at how people engage with those services, and how different groups and sectors can be mobilised, supported and encouraged to participate in the “common enterprise” of generating positive outcomes’.

This growing realisation has led to a shift of interest towards models of ‘co-design’, which recognise that policy makers may often have more impact on key outcomes by using their limited resources to engage, involve and influence the behaviour of users and other parties, rather than concentrating on traditional modes of providing services. Influencing behaviour is not a linear process where policy makers enforce a particular policy with a distinctive and well-defined effect. Policy success depends on many factors and  particularly on the cooperation of different groups of society. Accordingly contemporary policy trends are recognising the need to creatively utilise the multiple mechanisms available in designing approaches to promote action on environmental and other complex issues.

The 6 E’s policy design framework

There are a range of policy choice tools that aim to link theory and practice, and in general these encourage policy makers to use a mix of options – taking the context and stakeholder characteristics into account. In this posting I look at DEFRA’s 4Es policy framework (see Figure 1 – centre circle) as one such tool which provides a framework to help policy makers develop a mix of interventions across four broad categories of policy tools – Enable, Engage, Encourage and Exemplify. More recently the UK Behavioural Insights Team through its Mindspace report has added two additional “E”s (see Figure 1 – outside circle) that round out a process of informed policy design and results measurement.  Using this tool as a checklist can help agencies to more effectively influence behaviour by combining approaches across these six broad categories.

Figure 1: No single solution – the 6Es model provides a framework to help design a mix of interventions within a broader systemic design and collaborative learning approach..

Collectively, the approach and the mix of measures drawn from across the 4Es need to address the core motivations and barriers of the broader range of stakeholders involved – including agencies themselves. For example, it’s likely including measures that engage and enable will lead to “change” through those most willing and able to act. It’s also feasible that some other groups may act if agencies demonstrate their own willingness to “walk the talk” and model the collaborative behaviours they would like to encourage. For others there is a role for regulation, or the development of minimum or voluntary standards before behaviour is influenced. Each element encompasses a suite of potential approaches and some of these are indicated below.

Enable – We need to start where people are.  This makes it easier for people to make the right choice. What is important is that people and sectors are provided with the support they need to make responsible choices (e.g. by putting the right infrastructure and services in place, building skills, removing barriers, training).

Engage – It is important to get people involved. Effective public engagement is necessary to provide legitimacy for interventions that may be introduced.  As David Pollack reminds us, “behaviour change can be controversial, involve difficult tradeoffs, and concern areas where government legitimacy is controversial. These questions are both tricky and of general concern to the public.” This may require new methods of engagement and co-design to explore what actions are socially acceptable or to gain legitimacy for a proposed practice change. In this areas agencies could leverage networks, work with trusted intermediaries and champions, and build a shared understanding through the use of dialogue and learning-based approaches.

Encourage – This covers the broad range of approaches that agencies have to encourage change (e.g. regulation, minimum standards, social marketing). It requires developing and putting in place incentives (in their broader sense) to encourage – and disincentives to discourage – a target audiences response. Providing feedback on progress is also important in this respect. One set of principles for encouraging change can be found in an earlier LfS post – Sustainability: six ways to influence behavior change.

Exemplify – In most behaviour change interventions, exemplifying desired changes is
important.  This requires those designing change to lead by example and to ensure that the actions of the policy agents are consistent with the change being sought. This puts the onus on agencies and policy partnerships to review their internal policies and take action to ‘exemplify’ the same behaviour they are asking for (e.g. demonstrate the ability to work in a partnership, collaboration, ensuring mesages are consistent across agencies).

Policy design and adaptive management

The final two E’s – explore and evaluate –  serve to embed the 4 E’s in a broader process of informed policy design and results measurement. This brings in concepts of systems thinking, systemic design, learning, reflective practice and adaptive management and policy. Together, exploring and evaluating create a broad learning environment through which the development and implementation of policy and management can adapt. A related post on this site – After action reviews and how they can be linked with ToCs to support strategic thinking – highlights how the 2 E’s can be used in tandem to create both the space and guidance for strategic learning, and subsequent adaptation and innovation.

Explore – Understanding the issue and the wider context in which it is sited. This needs to include inquiring into how the different stakeholders involved experience things and  to understand the behaviour that is being targeted. Broad approaches such as systems thinking and systemic design are particularly useful  in this setting encourages us to explore inter-relationships (context and connections), perspectives (each actor has their own unique perception of the situation) and boundaries (agreeing on scope, scale and what might constitute an improvement). Recognising these various influences is crucial, given the complex ways in which people make decisions.

Evaluate – If we are to improve the way we look to influence positive change then we need to understand the complex range of factors that affect stakeholder behaviour, and good evaluation provides a way of doing so. As David Pollock points out, “although there will always be a healthy tension between evidence-based policy and innovation-based policy, our collective mission should be evidence-based innovation”. Building the capacity to reflect on action so as to engage in a process of continuous learning is increasingly seen as an important aspect of behavior change, and it is beginning to be used in many models of changing professional practice. However, it is not a conscious behavior for many teams, and effort needs to be put in to provide teams with tools that can support reflection. These tools are usually known under names such as After Action Review (AAR) or Learning Debriefs, and are used by to capture the lessons learned from past successes and failures, with the goal of improving future performance.

The complexity and uncertainty of challenges such as welfare, health, crime, employment, education and the environment demand that our systems of governance and our policies be adaptive. Policy responses in these areas should be designed to both reflect local contexts and adapt as conditions change. In fact, instead of helping, governance and policy that are not adaptive could actually hinder the ability of households, communities and businesses to cope with and adapt to change. Frameworks such as the 6 E’s can be seen as a communication tool for the co-design of policies that offer a way to actively engage a range of stakeholders to find solutions for complex water management issues.


2017 in review – your favorite LfS content


Another year has passed, and this provides a good opportunity to reflect on what content resonated most with visitors on the Learning for Sustainability (LfS) site in 2017.  So – based on the site statistics* – here are the most visited pages in terms of topic areas, blog posts and downloads of hosted content.

The Learning for Sustainability (LfS) website operates as an international clearinghouse for on-line resources around collaboration, social learning and adaptation.  As you can see from the navigation bar above it provides pages of annotated links pointing to targeted resources on a range of relevant and interlinked topics. During 2017 the site averaged more than 900 visits* each day.  Managing collaborations, complex problems, Theory of Change, systems thinking, and reflective practice featured heavily in the most requested content.

Most visited resource pages

While the LfS home page remained the preferred choice of entry for most, the three resource pages that were most visited were:

  • Theory of change (ToC). This page provides links to guides for using ToC – a methodological approach for planning, participation, and evaluation.  It shows how its use can help orient diverse program stakeholders to work together and plan for outcomes by envisaging a ‘big picture’ view of how and why a desired change is expected to happen in a particular context.
  • Systems thinking.  This page points to sites providing toolkits and tools to support systems thinking. It encourages practitioners to understand and analyse the contexts within which they operate, as a precursor to designing programs/policies that can adapt as conditions on the ground change.
  • Selecting evaluation questions and types. This page provides guides to help program managers to develop appropriate evaluation questions that are driven by funders, project participants and other key stakeholders. Further links highlight how different evaluation types (and/or methods) are distinguished by the nature of the questions they attempt to answer.

Most read LfS posts

Some of our most popular and engaging content in 2017 was, not surprisingly, blog posts that provide introductory material that grounds key topic areas. Check out these posts that were most popular with readers in the last year.

Popular downloaded papers and reports

While the site primarily operates as a clearinghouse to on-line material hosted on sites all over the world, it does host a range of papers and reports. The three most downloaded media were as follows:

  •  How Decision Support Systems can benefit from a Theory of Change approach.  This 2017 research paper begins by describing a ToC and how it can be used in conjunction with DSS development. We then illustrate how to apply a ToC approach using a pest (rabbit) management example in Australia. We end with a discussion of potential benefits and challenges from using the approach.
  • Stakeholder analysis. This 2010 book chapter reminds us that a stakeholder analysis is just one (albeit usually the first) step in building the relationships needed for the success of a participatory project or policy.  It covers steps in conducting such an analysis, and then outlines some best practice guidelines.
  • Building resilience in rural communities.  This 2008 report aims to provide a toolkit outlining ideas and information  that could be included in new or existing social programs. It introduces and expands on 11 resilience concepts found to be pivotal in enhancing individual and community resilience.


* Using AW stats


Social learning – what it looks like

Social learning - 123RF- - 41266812
Social learning emphasizes the importance of taking time to pause and engage in constructive dialogue. [Copyright: rawpixel / 123RF Stock Photo]
Social learning is an approach to working on complex environmental problems, particularly those with high degrees of uncertainty, lots of interested parties and disagreement around causes, effects and even desired outcomes. There is no one definition of social learning, but the many descriptions of it emphasize the importance of dialogue between groups.  These negotiations help people to better understand different points of view, and develop processes for collective action and reflection over time. This post provides a brief introduction to the concept – more information and links to a wealth of online material about it can be found directly from the LfS social learning page. There are also a number of links through this post that will take you directly to pages that explore different aspects of social learning.

Just to begin, social learning is not what many people confuse it as – learning by people ‘out there’ about the important things we ‘in here’ think they should know! That is information dissemination, advice …. or even indoctrination. At times we may feel it necessary to directly tell people ‘how it is’ – but this more linear approach to communication should not be confused with social learning.

The concept of social (or collaborative) learning refers to learning processes among a group of people who seek to improve a common situation and take action collectively. This understanding effectively extends experiential learning into social learning. By broadening their perspectives and taking collective action – people can become empowered. Empowerment, in this sense, can be seen as enhancing the capacity of individuals or groups to make choices, and to transform those choices into desired actions and outcomes. Achieving such outcomes requires a long-term social process that evolves over time and is woven from a number of different activity strands. These strands can be thought of in different ways; e.g. this site highlights five key strands that underpin social learning:  i) systems thinking; ii) network building; iii) dialogue; iv) knowledge management; and v) reflective practice. Information on the way that these different strands that support social learning can be accessed through the ‘social learning’ index above.

Social Learning, as an approach to complex problem solving, has emerged in recent years alongside other approaches such as adaptive management and systems thinking. In fact it includes the core essentials of both of these. In a nutshell it is about creating situations where people can learn collectively to improve a situation. The aim of adaptive management is to enable groups and organizations to adapt their practices and learn in a systematic way, often referred to as ‘learning by doing’. The kind of thinking required is essentially systems thinking. This is about having an appreciation of the characteristics of systems, i.e., that each element will affect the operation of the whole, parts of the system are interdependent etc. The focus of systems thinking is therefore on interaction. Furthermore, systems thinking requires a shift of mind, a willingness to look at problems from different perspectives. It looks at underlying systemic structures; and encourages people to look beyond discrete events at underlying patterns of behavior and underpinning mental models. The aim of system thinking based inquiry is to seek leverage, seeing where actions and changes in structures can lead to significant and enduring improvements.

The point is that social learning would not be ‘social’ if it was not about people and their interactions. Because we are dealing with complex issues that arise from settings with many stakeholders with differing views, responsibilities, and knowledge about the system (including science, management agencies and people making decisions on-the-ground), social learning has to be about how to bring people together. Particularly it is about helping people work collaboratively – bridging disciplines, knowledge systems and cultures.  By keeping these concepts in mind we can aim to manage more interactions within participatory and learning-based contexts to help those involved to engage in social learning and develop a shared understanding around goals, actions and indicators.


After action reviews – and how they can be linked with ToCs to support strategic thinking

This post introduces After Action Reviews (AARs), and indicates how theories of change and AARs can be used in tandem to create both the space and guidance for strategic learning, and subsequent adaptation and innovation.

The importance of reflecting on what you are doing, as part of the learning process, has been emphasized by many reviewers. Building the capacity to reflect on action so as to engage in a process of continuous learning is increasingly seen as an important aspect of behavior change, and it is beginning to be used in many models of changing professional practice. However, it is not a conscious behavior for many teams, and effort needs to be put in to provide teams with tools that can support reflection. These tools are usually known under names such as After Action Review (AAR) or Learning Debriefs, and are used by to capture the lessons learned from past successes and failures, with the goal of improving future performance.

In terms of evaluation they are starting to be paired with design processes such as program theory-based approaches. Developing results-based framework diagrams can help with structuring and deepening participants thinking and planning (both in developing action plans and monitoring plans) for a project. Then after these plans have been implemented (action), an AAR can provide people with a platform and space to reflect on what happened and what they are learning about their project.

Four sets of questions that drive After Action Reviews (AARs)

AAR is a form of group reflection; participants review what was intended (activity aims), what actually happened (intended and unintended outcomes), why it happened and what was learned. AARs should contribute to information that not only helps assess an immediate activity, but also helps assess progress towards longer-term term progress set out in the underlying Theory of Change (TOC) and accompanying logic models. AARs can be used in short, frequent group process checks, or more extended, in-depth explorations to assess progress at the wider level. It can be undertaken by a group or an individual as they ask themselves four questions:

  • What was supposed to happen? By asking each participate about their expectations it can sometimes highlight problems in communication as individuals have different expectations.
  • What actually happened? By identifying what went on an accurate picture can be built up.
  • Why was there a difference? This is where participants need to concentrate on the what – and not the who – between expectations and what actually happened.
  • What can we learn from this? What learning points have been identified so that the organisation or individual continues to improve?

AARs can be conducted almost anywhere, and will vary in length. For example, an AAR can be conducted after a one-off event (in 15 minutes or so), or a much more focused meeting could be held to reflect on how planned intermediate and longer-term outcomes are being achieved at an organizational or program level. It is important that the underlying culture of those undertaking an AAR is one of openness and learning. It is not about allocating blame, but ensuring that those involved can move forward (and adapt where necessary). Lessons are not only shared by the individuals involved but can be documented and shared with a wider audience.

Every organization, every partnership or team, and every intervention will likely require different levels of preparation, execution, and review. However, a number of best practices do emerge across the literature:

  • Lessons must first and foremost benefit the team that develops them. The AAR process must start at the beginning of the activity (with clear aims set out in advance through a TOC or outcomes modeling process). AAR Lessons must link explicitly to future actions, that support desired outcomes. And leaders must hold everyone, especially themselves, accountable for learning.
  • Managers and facilitators should phase in an AAR culture. This can begin with facilitating simple AARs around team projects – keeping things simple at first and developing the process slowly—adding knowledge-sharing activities and systems, richer metrics, and other features dictated by the particular initiatives in question.
  • If there are issues with either openness or time, it may be worthwhile to gather individual ideas first – and then facilitate a group discussion.

By creating tight feedback cycles between planning and action, AARs build team and organizational capacity to succeed in a variety of conditions. AARs are not just a source of lessons, but provide a low-technology tool which teams and communities can use on how to draw new lessons from novel and evolving situations for which they did not train—situations they may not even have imagined. In a fast-changing environment, the capacity to learn lessons and adapt can be more valuable than any individual lesson learned. That capacity  -which can be used for adaptive management or for innovation –  is what can be gained by more closely linking outcome planning and learning-based reflective activities such as AARs.

A number of other Learning for Sustainability pages provide additional information on this topic. The page on selecting evaluation questions and types directly builds on this topic and includes links to additional external resources. A number of other pages are introduced through to the introductory page titled Planning, monitoring & evaluation – closing the loop.


Key evaluation questions (KEQs)

This post explores how evaluations benefits from being focused on a small set of key questions. These are often referred to as key evaluation questions (KEQs). They should be seen as high level questions that assess progress towards the main specified outcomes, and will be answered by combining data from several sources and methods. 

Evaluations provide an opportunity for your (or your clients’s) intervention’s overall progress to be considered, including focused consideration of specific aspects of the initiative. A well-developed theory of change (TOC) and accompanying logic models provide an outline that helps to develop measures of success that traces the intervention’s development and impact over time. These measures, in turn, need to be focused with appropriate KEQs that are driven by funders, project participants and other key stakeholders.

Criteria for developing Key Evaluation Questions (KEQs)The five criteria to evaluate interventions (relevance, effectiveness, efficiency, impact, and sustainability outlined in the OEDC/DAC evaluation guidelines provide a good starting framework for a range of initiatives in development areas (health, natural resource management, community resilience, etc.) . Evaluation questions also to be considered in a complex intervention such as this should address context, reasons for adaption and emergence of activities and outcomes, different perspectives and inter-relationships that impact project success, sustainability and transferability.

A useful starting set of key evaluation questions to guide initial analysis are:

  • Is the research delivering on outputs and outcomes as planned? (efficiency and effectiveness)
  • Have applied activities and their delivery methods been effective? Are there aspects that could have been done differently? (process effectiveness)
  • Is the wider project story being told? What range of outcomes (intended and unintended) has the research project contributed to – taking account of each of social, economic, environmental and cultural considerations (impact)
  • How has the project influenced the stakeholder community, and what capacities has it built? (impact)
  • Is the project being delivered on budget? What aspects of the participatory elements of the project could be done differently next time to cut costs while still delivering achievements? (efficiency)
  • Is the project impacting positively on key groups and issues that have been identified as important in project design? (impact)
  • Is there evidence that the initiative is likely to grow – scaling up and out – beyond the project life? (sustainability)
  • To what extent did the initiative deliver against the needs of key stakeholders? Were the size, scale and approach taken for each need appropriate? (impact & efficiency)

These questions need to be clarified by key project stakeholders. Some may be amended, others dropped, and new questions can be included. Developing these questions also provides an opportunity to revise the underlying theory of change and any accompanying logic or outcome models. In this way KEQs can be seen to help intervention planning and evaluation.

A number of other Learning for Sustainability pages provide additional information on this topic. The page on selecting evaluation questions and types directly builds on this topic and includes links to additional external resources on KEQs. A number of other related M&E themes are introduced through the introductory page – Planning, monitoring & evaluation – closing the loop.



More about outcomes – why they are important … and elusive!

This post looks more specifically at outcomes, and how they can be developed and written. It highlights the benefits of focusing on outcomes for project planning, implementation and evaluation. It also provides some tips and ideas for involving program staff and stakeholders in developing and working with outcome statements.

Until recently, the performance of many public sector programs has been judged largely on inputs, activities and outputs. Over recent years this approach has been increasingly questioned as being too concerned with efficiency considerations, without a corresponding focus on what benefits are actually arising from program funding and activities. Increasingly the trend is moving towards a focus on the specification and achievement of outcomes, revealing more about how effective programs are in achieving real development changes on-the-ground.

Outputs are the goods and services that result from activities. Outcomes are the constructive impacts on people or environments. In the past planning and evaluation has tended to focus on program outputs, or how we keep ourselves busy – the ‘what we do’ and ‘who we do it with’. This enables us to tell our partners, funders and stakeholders about what the program does, the services it provides, how it is unique, and who it serves. We can describe and count our activities and the different goods and services we produce. Now, however, we are being asked what difference it makes! This is a question about outcomes (see figure). Outcomes are the changes, benefits, learning or other effects that happen as a result of what the program offers or provides. Outcomes are usually specified in terms of either: i) social and organizational capacities (social outcomes – e.g. learning, understanding, perceptions, attitudes and behaviors), or ii) state conditions (the bio-physical, ecological, social or economic changes in a system).

outcomes model 2
Logic models highlight the need to consider both program efficiency and effectiveness


While most people intuitively appreciate this distinction between outputs and outcomes, experience in results-oriented training sessions suggests that for many program staff, turning that appreciation into practice takes time. As the Keystone (2009) guide points out it takes most people quite a lot of conscious practice before they start thinking in terms of outcomes, rather than outputs or needs or activities. An outcome statement describes a result – a change that has taken place. It is not a needs statement, or an activity that is still in progress. Outputs comprise the products and activities that you do, while outcomes are what we see as a result of our outputs.) One simple test is to ask two questions of each statement: i) is it written as an outcome? and ii) does it describe changes that we can plausibly enable or facilitate in people, groups, institutions or environments?

Outcomes may be specified in different ways. Often a distinction is made between short-term, intermediate and long-term, or just intermediate and long-term. Short-term outcomes can be seen as the immediate difference that your program makes in the wider environment. A long-term outcome often has a number of short-term and intermediate outcomes that together contribute to the ultimate achievement of the long-term outcome. Collectively these outcomes should contribute explicitly to the wider vision underpinning program development. An intermediate outcome is a specified intermediate state that contributes to the desired long-term outcome – a step along the way. Intermediate outcomes are especially useful when time lags in measurable state outcomes are significant or limit timely response.

The program outcomes and intermediate outcomes should be structured in a logical hierarchy reflecting how each leads to another and/or contributes to the long-term community outcome(s). A useful way of doing this is to take each outcome and ask the question, ‘If we achieve this, what will it lead to and how will it contribute to the long-term outcome?’ Look for gaps – starting from the highest level outcome and working down the outcomes model. A test is being able to read an outcome and say, ‘Yes, this will likely be achieved if all of these initial (contributing intermediate) outcomes (and corresponding outputs) are achieved.’ The answers to these questions will enable you to draft a succinct statement of each outcome.

Each outcome statement should therefore define what will change as a result of an intervention and by how much (or, at the very least, in what direction the change will occur). This then allows the means of performance measurement to be defined. The more clearly an outcome statement specifies a desired change, the easier it is to define an appropriate indicator or indicator set.

It is not always easy to identify outcomes, and harder still to clarify them, but there are a number of key questions that can help. For example, begin by asking what is/will be different as a result of the initiative? For whom? What will be changed/improved? What do/will beneficiaries and other stakeholders say is the value of the program? For an existing program, look at the major activities. For each activity, ask yourself, ‘Why are we doing that?’ Usually, the answer to the ‘Why?’ question is an outcome. Most importantly, seek ideas and input from others. Their perspectives will help provide a broader understanding of the program and its benefits. This activity will also help build consensus among key program stakeholders.

When writing outcomes be sure to describe the desired change. Keep your outcomes SMART: Specific, Measurable, Achievable, Relevant, Time-limited. Say ‘what’, not ‘how’ – Establishing the means and plausibility of the ‘how’ is a later step. Consider whether outcomes are likely to be achieved in the program time frame.

Table 1 Examples of outcome statement structure from a range of sectors

Who/whatChange/desired effectIn whatBy when
Agricultural productionIncrease$ valueOver x years
Biodiversity (species)IncreaseTrendOver x years
Public awareness of an issueIncreaseExtentOver x years

This post provides a short introduction to the language and concepts of outcomes. Links to a wealth of information, tips and guides from around the world can be found from the LfS Managing for outcomes: using logic modeling webpage.



Diagramming a theory of change

The previous post looked at the benefits of using a Theory of Change (TOC) to better understand your program or project. This post will look at how to use post-it notes and an expanded logic model framework to involve stakeholders in beginning to develop this bigger picture.

Involving participants in articulating their projects through a simple logic model using post-its provides a good starting point

Often rallying participants around the development of a visual logic model is a good place to begin the development of a theory of change. The use of key headings and post-it notes makes it easy to provide a structure to help people develop some early models that contribute directly to their program planning, and build confidence and capacity in the use of TOC outcomes-based approaches.

Logic models are narrative or graphical depictions of processes in real life that communicate the underlying assumptions upon which an activity is expected to lead to a specific result.  There are four components commonly included in logic models (Fig. 2). These are the four primary components of the project or program itself – inputs, activities, outputs and outcomes. There are also four supporting activities which encourage participants to think more carefully about the underlying theory of change that they are planning to use. These supporting activities are: i) an outline of the current situation and desired vision; ii) stakeholder analysis, to identify which stakeholders should be involved in model development; iii) the scoping and planning exercise that underpins any model development; ensuring that underpinning assumptions are documented; and iv) noting internal and external factors – including related activities – that may influence outcomes.

Fig. 1. How the eight essential components of a logic or outcomes model (colored boxes) fit together
Fig. 2. How the eight essential components of a TOC logic or outcomes model (colored boxes) fit together

There is no single or correct way to draw a logic model. It can be drawn horizontally (as in Fig. 1) vertically, or even in a more free-form fashion. Ideally, a logic model should be able to be displayed on a single page with sufficient detail that it can be explained fairly easily and understood by other people. Much of the value of a logic model is that it provides a visual expression of our underlying beliefs about why the program is likely to succeed through one step leading to another. Thus, each step between an activity and an output or between an output and an outcome can be thought of as an ‘if this happens … then that is likely to happen’ statement. For large or complex programs, the logic model may be divided into more detailed sections or sub-models. These may be summarized by a less detailed ‘overview’ model, often given on the first page, that shows how the component sub-models fit together into a whole.

As an example, Fig. 2 illustrates the main program logic elements set out in a horizontal fashion. The inputs are the resources used to resource the activities, produce the program outputs, and ultimately contribute towards desired outcomes. Inputs typically include such things as money, staff, and equipment/infrastructure. Inputs are usually measured as counts, such as hours of staff time, dollars spent, etc. Activities are the actual interventions and actions undertaken by program stakeholders, staff and agencies to achieve specified outputs. Activities can range from writing a memo, to holding workshops, to creating infrastructure. Activities are usually measured in terms of number of things done – e.g. x meetings held with communities. Outputs are the tangible results of the major activities in the program (the goods and services produced). They are usually measured by their number – e.g. reports produced, newsletters published, numbers of field days held. Collectively the inputs, activities and outputs define what the program does, and how efficient it is in managing those elements. Outcomes represent the effectiveness of the program – are the desired states of the community, biological system or production sector achieved by the program.  Outcomes may be specified in terms of short-term, intermediate and long-term, or just intermediate and long-term. A long-term outcome will usually have a number of intermediate outcomes that together contribute to its ultimate achievement.

The diagram above also shows the supporting information and activities that help the model (and the intended program) to be understood in its wider context. Starting out with outlining a planning and scoping phase helps participants to clearly define the problem or need, and the desired outcome. An ‘issue’ statement should explain briefly the current situation: what needs to change; why is there is a need for intervention; and, what problem/issue does my program aim to solve? This requires that ‘who, what, why, where, when, and how’ are all considered in relation to the problem/issue. Then, the overall purpose of the program needs to be defined. What are you trying to accomplish over the life of the program and beyond? The answer to this question is the solution to the issue statement, and will serve as the program’s vision. The program vision serves as a reference frame for all elements of the logic model that follow. Involving your key stakeholders (see the accompanying resources on stakeholder mapping and analysis) in the process of developing an outcomes model provides an opportunity to engage them in a discussion about the program and to get their input to the process.

The link between a program’s activities and outputs and its desired outcome is based on the assumptions that explicitly, or implicitly, are built into your program theory. Your program theory (or theory of change) sets out why you believe that the successful delivery of the program’s activities and outputs is expected to lead to the desired change (the predicted outcomes). It is important to document the program rationale – the beliefs about how change occurs in your field, based on research, experience, or best practice. This needs to be followed by identifying the corresponding assumptions that are built into the program rationale and to acknowledge and document where uncertainties exist.

A final discussion can help participants to take account of the risks and opportunities facing the program. These can derive from both internal and external factors. Programs that are operating in complex environments cannot control all the factors that will influence how, when or even if they reach their goals.  Therefore it is also important to be aware of similar or related external initiatives that will impact on the final outcomes. This is important in terms of attribution – how to ascertain how much impact can be attributed to your program. It also provides the opportunity to look for other initiatives to link and integrate with, to develop useful synergy and maximize the overall influence of the program.  Internal factors might relate, for example, to staff and stakeholder capacities.

Three key reasons for using logic models in program design are that they: i) helps you understand why and how something works; ii) provide a guide for implementing useful monitoring and evaluation systems; and iii) help you tell the story of your program quickly and visually. Logic models are most useful when they are developed at the beginning of a program.  In this way they can be used to plan how resources can be coordinated and even inspire particular  project strategies.  They can also at this stage help set  realistic expectations for outcomes, bearing in mind that the ultimate desired end-state outcomes of an initiative can often take many years to emerge.  Their initial development helps subsequent evaluation as once a program has been described in terms of a logic model, it is then possible to identify meaningful and easily measurable performance indicators. Finally, the simple, clear graphical representation that a logic model provides helps with program communication, and can serve as the basis for expanding the underlying TOC.

Finally ––Some tips for working with logic models

  • Start with ensuring a common understanding of the current situation and a shared vision:   It’s important to know where you are, and where you are trying to get to. These positions will have often been expressed in already published documents, mission statements, etc. The important thing is to ensure that there is some common understanding around the problem and the desired outcomes among all those that you are trying to work with on your journey.
  • Involve stakeholders:   A strong focus on the process of developing a logic or outcomes model (rather than seeing it as just a task to complete) can increase engagement in the program. Building a logic model provides an opportunity, often rare in the everyday provision of services, to involve stakeholders in a discussion on what it is about the planned initiative that is most meaningful to constituents.
  • Keep the model simple:  Concentrate on the most important activities and outcomes, and cut back on detail. Describe your activities and outcomes in language that is understood by a wide range of stakeholders.  This lets your logic model provide a common picture of your project that is easily understood. It’s important to get an  overview of the model on one page that can be used as a communication aid, and more detail can be added behind it if necessary.
  • Minimise the use of arrows:   In complex situations there are always many links and potential feedback loops between the boxes on the page. It is often enough to indicate the general movement of time and direction of the model.
  • Avoid siloed thinking:   Don’t just include steps and outcomes that are measurable or which you can absolutely prove you changed (attributable to you) – these may not end up being the most important part of the programme. Similarly don’t force lower steps to only contribute or influence a single higher-level step or outcome. Most elements influence a number of things in the real world.
  • Work constructively with disagreement:  Although it might be difficult, keep key stakeholders involved, including staff, program participants, collaborators, or funders. Take time to explore the reasons for disagreement about what should be captured in the logic model. Look for the assumptions, identify and resolve disagreements, and build consensus.

More information: Often people talk about logic models and theory of change processes interchangeably. Logic models typically connect programmatic activities to client or stakeholder outcomes. But a theory of change goes further, specifying how to create a range of conditions that help programmes deliver on the desired outcomes. These can include setting out the right kinds of partnerships, types of forums, particular kinds of technical assistance, and tools and processes that help people operate more collaboratively and be more results focused.