Prioritising Competing M&E Needs and Demands in an Adaptive Programme: 7 Takeaways

In this blog post, Gloria Sikustahili, Julie Adkins, Japhet Makongo & Simon Milligan discuss seven key lessons about effective monitoring and evaluation from their experiences of implementing a ‘iterative programming and adaptive management’ approach in the Institutions for Inclusive Development programme in Tanzania.

We’ve all been there. We’ve drowned in the weight of programme documentation; the need to capture everything, to report everything, to be seen to be held accountable for all our actions or inactions. Yet on other occasions we’ve all sighed with exasperation that the programme we’re tasked with supporting has very little to help us understand what’s happened along the way, and why decisions were made.

So how do we strike the right balance? Whose needs are we meeting? And how do we better negotiate and prioritise the needs of different documentation users?

The DFID and IrishAid-funded Institutions for Inclusive Development programme in Tanzania – a “test tube baby” on iterative programming and adaptive management – explores new ways of tackling wicked problems in a variety of areas, such as solid waste management, inclusive education, and menstrual health. As a programme designed to be agile and opportunistic, it needs to have fast feedback loops so that staff and partners can make informed decisions about if and how to adjust strategy and tactics during implementation. It must also be accountable for its performance and generate lessons about how change occurs.

So, as the programme enters its final months of implementation, what lessons have we learned about M&E in an adaptive programme in terms of what, when and how to document and for whom? Here are our seven takeaways:

  1. All parties benefit when genuine attempt is made to understand not only the programme theory of change, but also the basis for certain actions during the adaptive process. It pays to build rapport with donors and reviewers; to understand their interests, needs, concerns and motivations. While this might not necessarily lead to fewer requests, it can lead to more streamlined responses; responses that ‘tick the box’ first time around. Annual reviewers and mid-term evaluators also need to understand the interests of implementers and the programme managers rather than focusing only on donors’ interests. This means reviewers should take time to understand the programme’s journey to date and future trajectory, while balancing the interests of both parties is a necessity. All too often we, as human beings, find ourselves believing certain things of our partners, be they donors, reviewers, implementers, partner agencies or others. We can project our own insecurities, values, beliefs, assumptions and prejudices onto others, and can react negatively when faced with “another silly demand for yet more documentation”.
  1. Utilisation is key and this needs to shape decision-making about what and when to document, and for whom. Documentation efforts should be clear-headed and purposeful. In much the same way as Michael Patton champions Utilization-Focused Evaluation, programmes should adopt a similar principle, i.e. that the need to document should be judged on its likely usefulness to its intended users, including donors, reviewers, managers, implementers and partners. Without agreement and clarity about purpose, intended user and relative costs, programmes end up documenting for documentation’s sake. But with actions come consequences. Excessive time spent ‘feeding the beast’ is time spent away from working towards an intended outcome.
  1. Documentation needs and demands need to be negotiated and balanced: there is an ever-present risk of prioritising the documentation needs of funders and reviewers at the expense of managers and implementers, and doing so can be counterproductive. Don’t get us wrong: we aren’t saying there is a binary choice between the needs of funders and reviewers on the one hand, and implementers on the other. Yet, the reality is that those working hands on, at the front line, make decisions in real time and do so against a backdrop of uncertainty and incomplete or implicit understanding. By contrast, managers and implementers are often involved in slower, more structured processes. This meeting of two related, yet different realities creates an environment in which different needs, expectations and demands must be successfully navigated, even negotiated. Senior managers, such as team leaders, often find themselves acting as a buffer or intermediary with the donor. Inevitably, unclear, excessive or competing expectations about what must be documented creates uncertainty, bias and inefficiencies. All information users – donors, reviewers, implementers and partners – should think through what must be documented and reported, from what might be documented and reported and from what they would like to see rather than love to see documented and reported in an ideal world (thanks to outcome mapping for this prompt). As a general rule of thumb, key information should be synthesised and summarised as headlines. Brevity in documentation forces clarity of thought and aids the production of ‘formal’ reporting (annual & semi-annual reports, case studies, and even blogs!) when required.
  1. To aid programme performance, donors must make choices and recognise the consequences of their signals. We acknowledge that all parties (funders, managers and implementers) require a certain level of documentation for accountability and sense-making purposes. OK, let’s get the “Yeah, obviously” observation out of the way first. Programmes have many constituencies or stakeholders, not least the donor agencies, and each party have their own needs and interests in documentation. Of course, where programmes are under-performing, close scrutiny is expected and necessary. However, staff within donor agencies hold positions of power. General queries or requests for information can be construed as demands – demands that often require the time-consuming compilation of documentation without a clear rationale and can result in airbrushed content which overlooks the messy realities. Unfiltered lists of questions, requests and comments from a variety of donor staff to a report, for example, can tie up implementers as they seek to make sense of, justify and explain, resulting in a ping pong to and fro which might have better resolved out over a cup of coffee. A confident programme with confident, capable and experienced staff can push back but this takes time and trust, and neither come over night. 
  1. Reflection and documentation are two related, important, yet different things. I4ID, like many agile programmes, is built on experimentation. Staff value reflection but the process must be shaped by a desire to improve performance. To borrow from Graham Teskey’s recent blog post, purposeful reflection allows implementers to reach a decision about specific workstreams (i.e. this is what we are going to do from here); which  means that implementers will necessarily prioritise capture of what they need to know to move forward, building on tacit knowledge and shared understanding. And, that reflection-and-capture need not translate to extensive documentation. Yet, challenges arise when donors and reviewers seek to make sense of often messy realities, a number of months after the event. This can lead to a situation in which events and decisions are fully documented not because it is valuable to implementation but to cover off possible future need and demands. The answer? Understanding, keeping line of sight on what matters, and accepting ‘good enough’. The most successful staff capture-and-reflection platforms are those founded on real-time discussion and action. Early efforts in I4ID to document key events on a weekly basis using a Word-based template and again on a monthly basis using an Excel-based dashboard stagnated within 12 months. By contrast, those platforms that thrived at I4ID – the Monday morning meetings, staff WhatsApp groups, and the Quarterly Strategic Reviews by the programme team, donors and some invitees – were founded on real-time exchange of reflections and ideas. This suggests that staff respond more favourably to live, interactive platforms, not an impersonal capture and storage. These platforms should be well organised and managed to avoid biases and defensiveness.
  1. Synthesis occurs most readily where discussions are well structured. Discussion is great, yet it must lead to something. For that to occur, use should be made of three key questions in key events (e.g. weekly meetings, strategic reviews) and associated minutes: What? So what? Now what? For example, what has happened in the operating environment over the last month, what programme effects have we seen, and what lessons have we learned about how change occurs? So, what does that mean for us and specifically, if/how our programme can best support the reform agenda? Now what should be done and by whom, not least before we next meet? Objective discussions require good facilitation to avoid bias and defensive reactions. 
  1. Donors should consolidate and localise oversight functions wherever possible. The decision to have multiple levels of oversight – the donor agencies themselves, the annual reviewers appointed by the donors to verify the claims made by a programme, and the external evaluators appointed by the donors to capture lessons identified by the programme – creates a living organism that has many needs and expectations. Sometimes these are aligned, other times not. So, it is right to ask, at what point can oversight functions (e.g. an external Results and Challenge team that produces annual reviews, and a separate external Mid-Term and End-of-Programme-Evaluation team) be consolidated or streamlined? Donors should ensure that local institutions are included in the evaluation and reviewer teams to get local perspective on the process and results, build capacity to local stakeholders and promote adaptive programming locally.   

The Community Development Potential: Connecting Ourselves with Wider Change

Jim Cowan writes for the CDJ Blog on his latest book, The Britain Potential, and how community development thinking can adapt to incorporate wider concepts from social and political thinking.

In my book The Britain Potential, I draw on a number of analytical and practice frameworks from writers outside community development whose ideas are drawn from social psychology, political philosophy, and social constructionism. I have found such ideas can also help bring out, what I have called `the community development potential`. Community development, with its decades long track record of integrity, is all the stronger, in my opinion, for critically embodying frameworks that can connect it with much wider change. Let me explain.

For over 40 years I have worked with people living on housing estates, families seeking support, black and white people working together with shared anti-racist aims, and people with disabilities.

Doing community development every day, decade after decade, in these communities, I saw how the country works from the ground up, through the eyes of all the people I was working with. I knew the work I was doing with communities was always saying something much bigger. But how to find that bigger story? And if you find it, does it then shed any new light on community development? Does it help bring out `the community development potential`?

When I retired in 2012 I finally had the time to write the book that had been sitting inside me for years. People kept asking me what I was writing. I found myself replying, “Britain is a country with huge potential, but it is not realising that potential by a long way.” Not one person disagreed! Now that the book has been published, still no one has disagreed. Britain`s potential has definitely been held back. Community development is very much part of the process ofrealising this potential (along with many other interventions, initiatives and developments).

By exploring the idea of `potential`, I found a framework that does, I think, shed new light on community development. It comes from the work of Ken Wilber, a writer who has drawn from psychology, other academic disciples, as well as many strands of spiritual practice and writing outside academia. He has colour coded the stages of consciousness that he regards humanity as having been through and is moving into.  He is not talking about the actual mechanics whereby brain matter generates consciousness (the stuff of cognitive and neuro science). Rather he is talking about what colours and creates very specific states of awareness of oneself in relation to the world. In his words:

…there are indeed higher and lower (or more or less evolved and aware) structures of consciousness, and we, as individuals and societies, can grow to higher levels in progressive stages or waves of development.”

In my opinion, these colour coded form of consciousness offer those training community workers, as well as workers in the field, a valuable tool. They can become part of the deeply reflexive, ongoing, continual effort at the 360 degree personal development and autonomy necessary to do the work professionally.

Here is a diagram of Wilber`s colour coded consciousnesses

Wilber`s amber consciousness is about ethnocentric and controlling rules, norms, and forms of leadership like organised religion or institutions. But historically these kinds of structures enabled effort on a large scale, often stable and enduring. From its initial concept there was something of the amber about the welfare state.

Orange defines the shift from medieval to enlightenment thinking. This is leadership based on reason aiming for universals. It is critical of dogma and is responsive to change. But it is a materialistic kind of rationality. Inner life is disconnected. Today this is very much the world of science, technology, business and the market thinking of government.

What Wilber calls green consciousness is a pluralistic, non-directive leadership very able to spot problems but not at all good at coming up with solutions. The search for consensus can be endless! Green has powered civil rights, environmental movements and feminism.

Wilber characterises this heady mix of amber, orange and green as very much powering an increasingly polarised public culture. What amber, orange and green have in common is that they think they are the right way to view the world.  He rates amber and orange as being in the majority in the population (maybe 65%) and green as `the leading edge` at around 25%.

A quantum leap on from these forms of consciousnesses is teal which is present in between 5-10% of the population and destined, he believes, (because of the limitations of the other consciousnesses), to be a future leading edge. Teal is ultra-humanistic leadership with strong holistic problem-solving capabilities. It takes deliberate ongoing personal effort to develop and sustain teal. That teal is real and operating in the world is evidenced in, for example the ground breaking research on organisations by Frederic Laloux, as well as my book (Cowan 2019).

To flesh this out in ways that start to shed new light on community development, I say this about teal coloured consciousness in my book:

“The ego is not in the driving seat. There is a quest for wholeness, bringing together the ego and deeper parts of the self. A more expansive, embracing self is there. Teal is not fearful and needing to control. Problems become challenges. How can we grow from engaging with this problem? There is a healthy development of self and concern and interest in others. Teals develop themselves inwardly, spiritually. But they are able to connect this with complex realities. Life is always teaching us about ourselves and the world.  Teal has thinking and rationality, teal has doing, and it also has being the person I can be. Teal will tap into all kinds of knowing from analytics to the wisdom to be found through emotion and intuition”.

This gives us another take on community development`s objective of empowering others to be in control. Unlike the other forms of consciousness, which think they are right, teal understands these other forms of consciousness and can work with them. It is a particularly subtle form of leadership which holds the space, for example, for community activists to take those early steps, and then more steps with more people and so on.

Jim Cowan`s book The Britain Potential is available via thebritainpotential.co.uk. Via the website, subscribers can get the first chapter as a taster, which some university lecturers are beginning to use to run a project with students.

References

Wilber, K and others. (2008). Integral Life Practice. Integral Books. Pg 74

Laloux, F. (2014). Reinventing Organizations. Nelson Parker.

Cowan, J. (2019). The Britain Potential. Arena Books. 2019. Pg 27