Prioritising Competing M&E Needs and Demands in an Adaptive Programme: 7 Takeaways

In this blog post, Gloria Sikustahili, Julie Adkins, Japhet Makongo & Simon Milligan discuss seven key lessons about effective monitoring and evaluation from their experiences of implementing a ‘iterative programming and adaptive management’ approach in the Institutions for Inclusive Development programme in Tanzania.

We’ve all been there. We’ve drowned in the weight of programme documentation; the need to capture everything, to report everything, to be seen to be held accountable for all our actions or inactions. Yet on other occasions we’ve all sighed with exasperation that the programme we’re tasked with supporting has very little to help us understand what’s happened along the way, and why decisions were made.

So how do we strike the right balance? Whose needs are we meeting? And how do we better negotiate and prioritise the needs of different documentation users?

The DFID and IrishAid-funded Institutions for Inclusive Development programme in Tanzania – a “test tube baby” on iterative programming and adaptive management – explores new ways of tackling wicked problems in a variety of areas, such as solid waste management, inclusive education, and menstrual health. As a programme designed to be agile and opportunistic, it needs to have fast feedback loops so that staff and partners can make informed decisions about if and how to adjust strategy and tactics during implementation. It must also be accountable for its performance and generate lessons about how change occurs.

So, as the programme enters its final months of implementation, what lessons have we learned about M&E in an adaptive programme in terms of what, when and how to document and for whom? Here are our seven takeaways:

  1. All parties benefit when genuine attempt is made to understand not only the programme theory of change, but also the basis for certain actions during the adaptive process. It pays to build rapport with donors and reviewers; to understand their interests, needs, concerns and motivations. While this might not necessarily lead to fewer requests, it can lead to more streamlined responses; responses that ‘tick the box’ first time around. Annual reviewers and mid-term evaluators also need to understand the interests of implementers and the programme managers rather than focusing only on donors’ interests. This means reviewers should take time to understand the programme’s journey to date and future trajectory, while balancing the interests of both parties is a necessity. All too often we, as human beings, find ourselves believing certain things of our partners, be they donors, reviewers, implementers, partner agencies or others. We can project our own insecurities, values, beliefs, assumptions and prejudices onto others, and can react negatively when faced with “another silly demand for yet more documentation”.
  1. Utilisation is key and this needs to shape decision-making about what and when to document, and for whom. Documentation efforts should be clear-headed and purposeful. In much the same way as Michael Patton champions Utilization-Focused Evaluation, programmes should adopt a similar principle, i.e. that the need to document should be judged on its likely usefulness to its intended users, including donors, reviewers, managers, implementers and partners. Without agreement and clarity about purpose, intended user and relative costs, programmes end up documenting for documentation’s sake. But with actions come consequences. Excessive time spent ‘feeding the beast’ is time spent away from working towards an intended outcome.
  1. Documentation needs and demands need to be negotiated and balanced: there is an ever-present risk of prioritising the documentation needs of funders and reviewers at the expense of managers and implementers, and doing so can be counterproductive. Don’t get us wrong: we aren’t saying there is a binary choice between the needs of funders and reviewers on the one hand, and implementers on the other. Yet, the reality is that those working hands on, at the front line, make decisions in real time and do so against a backdrop of uncertainty and incomplete or implicit understanding. By contrast, managers and implementers are often involved in slower, more structured processes. This meeting of two related, yet different realities creates an environment in which different needs, expectations and demands must be successfully navigated, even negotiated. Senior managers, such as team leaders, often find themselves acting as a buffer or intermediary with the donor. Inevitably, unclear, excessive or competing expectations about what must be documented creates uncertainty, bias and inefficiencies. All information users – donors, reviewers, implementers and partners – should think through what must be documented and reported, from what might be documented and reported and from what they would like to see rather than love to see documented and reported in an ideal world (thanks to outcome mapping for this prompt). As a general rule of thumb, key information should be synthesised and summarised as headlines. Brevity in documentation forces clarity of thought and aids the production of ‘formal’ reporting (annual & semi-annual reports, case studies, and even blogs!) when required.
  1. To aid programme performance, donors must make choices and recognise the consequences of their signals. We acknowledge that all parties (funders, managers and implementers) require a certain level of documentation for accountability and sense-making purposes. OK, let’s get the “Yeah, obviously” observation out of the way first. Programmes have many constituencies or stakeholders, not least the donor agencies, and each party have their own needs and interests in documentation. Of course, where programmes are under-performing, close scrutiny is expected and necessary. However, staff within donor agencies hold positions of power. General queries or requests for information can be construed as demands – demands that often require the time-consuming compilation of documentation without a clear rationale and can result in airbrushed content which overlooks the messy realities. Unfiltered lists of questions, requests and comments from a variety of donor staff to a report, for example, can tie up implementers as they seek to make sense of, justify and explain, resulting in a ping pong to and fro which might have better resolved out over a cup of coffee. A confident programme with confident, capable and experienced staff can push back but this takes time and trust, and neither come over night. 
  1. Reflection and documentation are two related, important, yet different things. I4ID, like many agile programmes, is built on experimentation. Staff value reflection but the process must be shaped by a desire to improve performance. To borrow from Graham Teskey’s recent blog post, purposeful reflection allows implementers to reach a decision about specific workstreams (i.e. this is what we are going to do from here); which  means that implementers will necessarily prioritise capture of what they need to know to move forward, building on tacit knowledge and shared understanding. And, that reflection-and-capture need not translate to extensive documentation. Yet, challenges arise when donors and reviewers seek to make sense of often messy realities, a number of months after the event. This can lead to a situation in which events and decisions are fully documented not because it is valuable to implementation but to cover off possible future need and demands. The answer? Understanding, keeping line of sight on what matters, and accepting ‘good enough’. The most successful staff capture-and-reflection platforms are those founded on real-time discussion and action. Early efforts in I4ID to document key events on a weekly basis using a Word-based template and again on a monthly basis using an Excel-based dashboard stagnated within 12 months. By contrast, those platforms that thrived at I4ID – the Monday morning meetings, staff WhatsApp groups, and the Quarterly Strategic Reviews by the programme team, donors and some invitees – were founded on real-time exchange of reflections and ideas. This suggests that staff respond more favourably to live, interactive platforms, not an impersonal capture and storage. These platforms should be well organised and managed to avoid biases and defensiveness.
  1. Synthesis occurs most readily where discussions are well structured. Discussion is great, yet it must lead to something. For that to occur, use should be made of three key questions in key events (e.g. weekly meetings, strategic reviews) and associated minutes: What? So what? Now what? For example, what has happened in the operating environment over the last month, what programme effects have we seen, and what lessons have we learned about how change occurs? So, what does that mean for us and specifically, if/how our programme can best support the reform agenda? Now what should be done and by whom, not least before we next meet? Objective discussions require good facilitation to avoid bias and defensive reactions. 
  1. Donors should consolidate and localise oversight functions wherever possible. The decision to have multiple levels of oversight – the donor agencies themselves, the annual reviewers appointed by the donors to verify the claims made by a programme, and the external evaluators appointed by the donors to capture lessons identified by the programme – creates a living organism that has many needs and expectations. Sometimes these are aligned, other times not. So, it is right to ask, at what point can oversight functions (e.g. an external Results and Challenge team that produces annual reviews, and a separate external Mid-Term and End-of-Programme-Evaluation team) be consolidated or streamlined? Donors should ensure that local institutions are included in the evaluation and reviewer teams to get local perspective on the process and results, build capacity to local stakeholders and promote adaptive programming locally.   

RESOURCES: CDJ 50th Anniversary Conference

DSCF0334We were delighted to welcome delegates to Edinburgh at the beginning of July for the CDJ 50th Anniversary Conference: Why Community Development? Continuity and Innovation.

In the spirit of widening participation, we will be gradually uploading resources from this event – including presentation slides, video and critical reflections –  to CDJ Plus in the coming weeks.

We’ve already got some initial materials up on our new dedicated page:
50th Anniversary Conference.

Please do keep checking back to see what else we’ve added!

Community Development Journal – Open Call for Journal Editor

Since 1966 the Community Development Journal (CDJ) has been the leading international journal in its field, covering a wide range of topics, reviewing significant developments and providing a forum for cutting-edge debates about theory and practice. It adopts a broad definition of community development to include policy, planning and action as they impact on the life of communities. We particularly seek to publish critically focused articles which challenge received wisdom, report and discuss innovative practices, and relate issues of community development to questions of social justice, diversity and environmental sustainability.  The Journal is published four times a year and is circulated in over eighty countries.

The CDJ’s current Editor plans to stand down as of January 2016. To ensure continuity and handover, and following a selection process in spring 2015, new editorial arrangements will be set in place by July 2015.  Between July 2015 and January 2016, it is anticipated that the current and incoming Editor(s) will work in parallel in order to ease the transition and passing on of roles.

The CDJ is now seeking applications for the role of Editor(s).  It is expected that the incoming Editor(s) will share our commitment to the CDJ’s values and mission, and to the on-going development and enhancement of the journal itself.  It is likely that Editor(s) will be UK based though candidates from other locations may make a case as to why they feel it would be possible to meet the demands of the editorial role from another location.   The Editor(s) should have an outstanding knowledge of community development and a commitment to ensuring that the Journal retains its unique focus on providing a critically reflective and contextual account of the theory and practice of community development as it is practised and understood internationally.  The Editor(s) will work with an engaged and highly participatory Editorial Board and International Advisory Board.

The CDJ Board is open to different editorial models and invites applicants to state, in their application letters, their preferences in that regard.  For example, the following models will be considered:

  • Sole Editor with contracted administrative support
  • A model of co-editorship – with a maximum of two editors with contracted administrative support
  • Managing editor with Associate Editor.

Current remuneration for the Editor role is appropriate to the role, responsibilities and work undertaken. This will be discussed on application, and may be negotiable, within limits, according to the circumstances of the applicant.

The initial term of office will be for three years.

Contacts: A detailed Job Description is available on request. Prospective applicants are invited to send a written expression of interest, detailing their suitability for the position, along with a CV to: Ruth Pearce, Ruth.Pearce@warwick.ac.uk and /or Rosie Meade, r.meade@ucc.ie

Applications should be submitted by: Friday, February 27th 2015.