So What? Why Monitoring and Impact Measurement Matter in Humanitarian Contexts - Bangkok 2024

[Music] all right so while we're getting this set up uh good afternoon almost good afternoon everyone um my name is uh Elliot Seager uh and I've been delighted to meet so many Fascinating People here and have uh some really inspiring uh and interesting conversations which is probably why I'm on the verge of losing my voice at this point um but I have uh been in the monitoring evaluation learning impact space and humanitarian context for longer than I'd like to admit at this point um doing some work across Pakistan Afghanistan Yemen Sudan and a few of those great places to visit right now um and I'm very lucky now to be uh collaborating with mlin and the coip team um to think a little bit about how we look at some of these more traditional monitoring and evaluation approaches that have been used and implemented for the last 20 years and how we can strengthen those uh with new emerging technology um to make them more efficient more effective and ultimately um make humanitarian assistance more impactful uh in some of these complex and more volatile contexts there we go all right thank you all right so uh I've packed a fair amount into this so please forgive me but we're going to run through it and hopefully it'll be a good reference if you'd like to have access to it later um and so I'm going to focus a lot on what we've learned uh and a lot of our failures and a lot of our gaps in the systems that we have uh and how we're trying to address that uh going forward so what is uh monitoring and evaluation uh in humanitarian contexts and I think uh Anka gave a a brilliant presentation yesterday uh on some of the foundations of this so hopefully just we're going to build on that and talk about some of the more practical implementation um but a USA administrator once said to me why do you make it sound so complicated it's just to track what we do to test what we do and to talk about what we do uh and I think ultimately that Rings true in many ways and so if you talk about monitoring it's simply the ongoing collection analysis of data to track and verify that the outputs that we're trying to deliver are actually happening and there aren't ghost schools and we're not missing 10 meters of pipeline or all of these other things that have indeed happened evaluation is systematically testing the effect Effectiveness and efficiency of impact usually after humanitarian action has occurred and then learning is an ongoing practice that we try to integrate and build into all of our different systems um using more technology and feedback and listening to the communities and the people that we're working with to try and improve our systems in real time uh and as we go forward um so I'm sure many of you are familiar uh with this this is kind of un uh framework um and it's very relevant to Mel in the sector because um you know if you're working on say uh a health intervention um that's over five years and you're trying to reduce infant mortality by 25% in a specific District um that's fairly easy to develop uh a male framework for and measure impact but unfortunately more and more of the contexts that we're working in uh are either complex complicated or chaotic and what that means is the impact Frameworks that we develop and the monitoring systems that we use have to be tailored to those specific cont context and they have to allow for a bit of flexibility uh because we don't really know what's going to happen and things change very rapidly and we were having a debate yesterday I think about which of our sectors has more ridiculous acronyms um and I'm not sure who's winning yet but one of the ones we have is called camel uh which is complexity aware monitoring and evaluation and learning and I'm sorry that I brought that up but it does so uh we have different layers so the activity level this is are we effective in providing cash transfers or passing out bags of rice uh or delivering humanitarian Aid and then at the programmatic and strategic levels we have to use different methods to assess the context to determine whether that Aid is actually the Right Aid that we're providing uh even if it is verified and is there maybe the entire uh founding or theoretical rationale for that program uh was flawed from the get-go so it's important to have a multi-layered Mel approach whenever you're designing a program or designing your impact framework um this is the usaid Collaborative Learning adaptation model uh CLA which is all the rage right now but uh I think also still has quite a few problems but the idea is is that uh monitoring evaluation and impact measurement is not oneoff uh it's cyclical it's iterative and it has to continue to occur in a rapid process over the course of all the projects that we're working on so theories of changed and I think Anka uh really did a good job at explaining these yesterday but very simply it's if we do X then y will occur because of Z and I think this because of Zed uh aspect is the most important because that's our assumptions and we need to continuously test our assumptions to make sure that they're actually accurate and that requires different methodologies and different impact assessments depending on the type of humanitarian program that you're working on so I'm not going to go through all of this but I wanted to just give you some examples of what these theories of change look like in practice this is a project that I'm working on right now in the Border areas between Kenya Somalia and Ethiopia trying to improve peace building um and you see it's all nice and clean and fairly linear and our project is not going like this at all um but it looks good on paper so you know best intentions I think and down here also these are the assumptions that I was talking about and one of them here says governments maintain political will that hasn't been the case uh in our project and that can actually ruin the entire thing so um this is actually more of an Ask covering exercise in many any ways uh when you're not able to deliver your project and explain to your donor well you know we had this assumption and didn't really go that way so we've had to adapt accordingly then they can get more complicated these really annoy me and I think it's uh there's a lot of hubris among humanitarian actors that think that all of this is actually going to occur very cleanly and link up to each other so uh I tend to erir on the side of Simplicity but just when you run into these throw it in the bin I think is probably the best approach um I I think another aspect of our arrogance in the humanitarian sector is uh what we think that we can control um and so when we talk about impact uh a lot of projects will say oh yeah um you know we brought World Peace uh to South Sudan um and that was all because of our two-year $20 million program um and so I think we need to be very clear about uh contribution versus attribution uh is the jargon that they'd use uh in our sector and I think um if we understand that there are many many factors that contribute to the results or outcomes that we're seeing in certain places maybe ours was a fraction of that but we need to be clear about what the others are so as you're developing theories of change as you're coming up with indicators for those um this is really what we're being held accountable for and the farther we get out to the outcome and impact level um there's less and less accountability and so I think it's really important to be realistic and practical as you're working through some of that so here's some more jargon and another terrible acronym uh that we have but I'm sure many of you have seen this before but when you're developing indicators they should be specific measurable achievable relevant and time bound and you'd be surprised how many do not fit uh within this Matrix but I think coming back to Simplicity and practicability uh this is really important I'm also not going to take you through the detail of this but this is just an example of what it looks like in practice and this is a Lo logical framework we developed for humanitarian aid program for the UK government in Myanmar uh a couple of years ago also did not go to plan um but we capture it like this in certain ways and there are a variety of ways to do it and I think um some of the emergenc emerging technology we have now will make really boring spreadsheets like this less necessary and so that's something we're trying to deliver uh through koala as well um so when you're looking at indicator development these are the layers uh that we usually talk about and I think Anka would also add activities and inputs to this but this is uh over simp simplified but at the output level we're really looking at verification and I think when you talk about web 3 and you talk about blockchain that's really where it comes into its own uh and has a lot of value to add to the sector in terms of is it there did it get there is it being used what's the quality um all of those things which are very important but when you get to the outcome level you're looking beyond that so sure you've provided nutritional support to um people that are food insecure in Somalia uh was it the right food um was it culturally appropriate uh was it sufficient to actually meet their monthly quota or household needs um you know we talk a lot about uh plumpy doe I don't know how many of you are familiar with uh kind of nutritional programming but uh it's nutrition that's given to kids um in these humanitarian contexts that has is very high uh in certain nutrients and uh I remember going to a conference where everyone was freaking out because they'd run out of the right plumpy do for me uh and so someone came up and put up a PowerPoint and said I've got a new recipe don't worry we can do this and a Lebanese man in the back row just said oh so it's humus he's like this is my mother's recipe like why are we spending millions of dollars uh to look at this so I think that's really at the outcome level the kind of thing you want to be testing because I think we end up spending a lot of money on things that could be simplified and done much more efficiently and then at the impact level you're looking at are we actually adapting um to climate situations have we created livelihoods uh and then building resilience there's another buzzword that people love to use in our sector but I don't think anyone's really come up with what resilience actually means in our context but in a humanitarian context it often means building back better so uh recently there have been very severe floods in East Africa and we were responding um to those through cash transfers um to communities uh in these border areas uh where their entire livelihoods had been destroyed but we tried to use those cash transfers also paired with and coupled with Lively Hood's development initiatives so that in the future they would actually be more shock resistant to those floods going forward and that's where you need to do more rigorous and robust impact evaluation within those contexts and so then we have a few different methodologies that are relevant at those levels so it outputs direct observation just did someone see it um can we verify it's there can we use a geotagged photo uh also remote sensing which I think uh our colleagues were talking about earlier in terms of GIS satellite imagery um can we actually see if uh IDP camps are growing or if they're shrinking can we look at how big the flood area is in a certain area where disaster's occurred when you get to the outcome level we look at perception surveys uh we talk about most significant change and I think this is quite relevant in Conflict settings where you talk to communities about just generally over the last two years three years what's been the biggest change um that you've seen in your area and we take qualitative notes of these and we've often run AI analysis through there to actually see whether there's any link to the changes that they've perceived in their communities and the projects that we're running and if we see that there's no link between those two you've got a problem and you probably need to rethink your design and then at the impact level people love to talk about rcts and I've contributed to the design of three or four rcts in humanitarian context and not a single one of them went to plan uh one was for the Danish mining group in Somalia where they were like okay so we're going to provide um support to kind of conflict scenarios in these communities and then have a control group where we don't provide support and uh the donors were like well that's going to get in the news isn't it I don't think uh so a lot of them get halted but if you can do a randomized control trial and I think if you look at organizations like give directly and others they're doing a lot more in situations that are less volatile where communities are less mobile and you can actually get better data on some of that but other than that we do kind of theory of change contribution analysis uh and you can do some quasi experimental methods like difference and difference depending on the type of humanitarian response that you're looking at so why does this all matter uh in recent years about $21 million in Aid has been unaccounted for uh in Somalia uh I would say this is pretty problematic overall um and people within certain sectors don't seem to worry too much about this and this also comes back to the governance and political will issue we talked about before and understanding whether the government in a certain area where there is a humanitarian crisis is really interested in helping you succeed in terms of what you're doing uh more recently uh I'm sure many of you have been following the conflict in Ethiopia and Tigra um I've been working on this conflict for the last two years uh running a third party monitoring program for the World Bank uh and the reason that they asked me to come in and do this was uh they went on a monitoring visit they opened one of the warehouses That was supposed to have food for starving people in T and realized that it was completely empty and food for a thousand people had dis 100,000 had disappeared uh and they immediately launched uh an investigation into this which took six months and so the food Aid actually wasn't allowed to be delivered during that six-month period and hundreds of people starved in t as a result of that and the results of the investigation were that actually the local government in T and the somal uh sorry the Ethiopian national government had stolen uh that food Aid um and so what we've had to do is set up the third party monitoring systems that prevent that from happening in the future allow for accountability but also don't piss off the government enough to make them kick us out of the country uh and so there's a very kind of tight rope you have to walk in order to achieve that and part of what we've been doing is remote sensing we've been using blockchain uh as tracking and verification uh but at some distribution sites they wouldn't let us bring scales for instance to weigh the bag of wheat they were supposed to have 15 kgs of wheat for each household so I had to train enumerators to basically just be able to intuitively know what 15 kg felt like by lifting it up and that way we were actually be able to see where the aid diversion was occurring and report it to the local authorities to try and do something about it so uh the US government uh special investigation office has found that 40% of all Aid to Afghanistan was mismanaged or diverted which I think comes to around uh $31 billion um this photo was taken by one of my team members members and I like it because it's actually these men are sitting around What's called the Durand Line uh which is the border between Pakistan and Afghanistan and so we were providing Aid uh to communities who are living on the border who are some of the most vulnerable communities in the world and my colleague here will call him Bal for the purposes of this conversation I've always regretted what I did to him uh in this situation but we provided uh generators on Wheels uh to some of these remote communities so that because there was no electricity they could use the generators to power drinking water supply systems and guess what two weeks after we provide the generators they couldn't find them and I was like oh that's a great idea why did we put them on Wheels the idea was that they would rotate between the communities anyway so USA comes to me and says look if we don't have a geotagged photo of this bloody generator in the next week we're shutting your whole program down and back then and now I'm dating myself you couldn't just geotag a photo on your phone so I had to order 20 of these DSLR cameras had geotagging capability uh and give them to our staff to go and try and find these generators on the border of Afghanistan and Pakistan and turns out all of the cameras have massive satellite icons on the side and so if the government catches you with a camera in the tribal areas with a massive satellite icon on your camera you're probably going to get shot or kidnapped so I had to spend two days filing them off with a metal file and then my colleague Bal here hid it in his turban went on a 14-hour drive out to the community where we provided the generator and found out that sure enough the political agent or District administrator had moved it into his house to provide his family with electricity um so he goes into the house and he very kindly the district administrator goes to make him tea and he pulls the camera out and he's photographing the generator here and then he takes a 14-hour ride back and we realized that uh he wasn't in line of sight of the sky long enough for that geotag location to actually happen so we had to send him back out and sure enough he's abducted for two weeks and we have to start negotiation with local Elders to get him out and he was finally released but he still has PTSD from that whole experience and why am I telling you this story because we no longer have to do things like this we found technological solutions that will allow us to do remote monitoring and remote sensing in these areas and that actually relies on enabling and empowering communities to report on some of these issues themselves in remote ways using technology that's going to protect them from government eyes and and other types of spying so really encouraging so I think talking specifically about impact evaluation for cash um this is something that is becoming a growing evidence base and uh it's also a little bit controversial I think in the humanitarian sector um because the more and more that cash assistance is proven to be more effective than say passing out in-kind distribution or bags of rice um the less relevant uh a lot of these traditional humanitarian agencies are um so there's a lot of resistance to change uh in our sector um and I think what we need is to do more evaluation and more impact studies uh at this level to assess how we can actually distribute Aid or cash assistance in some of these scenarios um which will be less expensive more of the aid will actually end up in the communities that we want to help uh and it can be delivered much faster and in a rapid context than we could uh in other ways and what that requires doing uh is uh Market assessments so for instance uh in uh Somali at the moment um there's a drought and so there's a lot of food Aid that's being given um but some communities are saying actually we prefer to have cash uh rather than bags of wheat because our markets are functional and we're actually able to go out and use that cash and that also empowers uh communities to have more control uh and ownership over how they use assistance that's provided Ed uh and I think when we first talked about this uh 20 years ago uh eight agencies were saying no no no if we give people cash they're going to go spend it on Whiskey and cigarettes um and it's not going to have any material impact uh and we've seen that that absolutely is not the case um and so uh pivoting to this uh has been slow but it's something that I think is really taking up more traction and I think when you look at organizations like stellar and koip and others there's a really important use case uh for how we work with humanitarian organizations um to update their systems and to kind of demonstrate that the use case has value uh and it's also not going to put them out of business um and doing that I think requires a little bit of uh sensitivity and and negotiation around that um but it also requires integrating those systems with uh impact measurement Frameworks and so yeah cash and armed conflict is another issue but um we can talk about that a bit later so right now this is the proportion of humanitarian Aid uh that's delivered on cash transfers um and I think this is overall way too low uh and it also demonstrates that communities are much less empowered than they would be otherwise uh depending on the type of assistance that you're providing so what are the challenges in opportunities um so when you're doing impact assessment in Mel in complex contexts um access is probably the biggest issue so if you want to survey 3,000 people um in southern Iraq uh there's no way that's going to happen because enumerators walking around with clipboards or iPads are not going to have very much longevity uh in that scenario so how else do you have access so that comes back to well can you can you do push surveys um through some type of digital platform and can you do it in a way that's not going to put the respondents at risk uh and Safeguard those that you're actually collecting the data from but this is something that we're still struggling with in most humanitarian crises that we're working on uh cost so on average if you're trying to do a representative perception survey in some of these more challenging locations the cost is going to be upwards of $ 150 to $300,000 us to have something where you have 95% confidence uh in that data so it becomes very expensive and a lot of that is around the expensiveness of logistics required to actually get data collection infrastructure into some of the communities where you want to hear from people uh and because the roads are very poor because there's no communication uh infrastructure you have to spend more and more to actually be able to have uh trust in the findings that you are doing and to do it in a way that delivers uh high quality verification what you're looking at ownership and this is something that really does my head in um but if I'm being honest uh all of our humanitarian agencies are essentially doing research on the same topics with the same communities in the same areas over and over and over again um and these communities you know will provide you with information tell you about what their needs are talk about um you know what their key issues may be uh and then they never never hear anything back um so it becomes a very extractive relationship that we have um and also those organizations are so chuffed with themselves that they were managed to that they managed to get this data that they don't share it with anyone else um so the donors are like well what are you going to do they're like well you know this is our proprietary information um and so if you want to get information on it get someone else to go do exactly the same survey next week and so what that contributes to is what we call respondent fatigue and what we've seen in a lot of these human Arian context is that uh communities just don't want to talk to you anymore uh you keep taking all this data from them they're not owning it they're not seeing what you're doing with it and they're actually not seeing any material change in their situation as a result so if that was me I would just slam the door in their face um so that's something that we need to deal with as well so what are we thinking about uh in the koip context so I think you know we've got this amazing payment infrastructure and these collaborations with stellar and other organizations um and what would be really fantastic is if we could integrate some of these monitoring and evaluation and impact measurement processes into the payment mechanisms themselves to boost efficiency to decrease cost and to strengthen ownership of communities over their own data um so what we're building now and maybe some of you have come across this over the last couple of days is called the Billy app which is the name of our koala uh and uh I'm sure you've talked to chanoko and others um but it does everything from uh kind of AI driven uh llm conversations with communities uh that we call Community listening so instead of having um a standard push survey that you're pushing out they can just have more of a natural conversation with you about what the problems are in their locations and how they could potentially be fixed those llms can also provide them with guidance and insight and advice based on existing data about how they might be able to manage some of this themselves how they could contact relevant local authorities or how they could provide feedback if they think that the quality of assistance uh isn't good or it's being diverted or there's any other forms of mismanagement so that's great it also provides kind of semi-structured chat functionality um to help our clients and donors actually designed tailored and bespoke uh ways of gathering data from communities um that can be more nuanced in Tech textured rather than just a quantitative survey but it also does that too so being able to push out Quant surveys uh in these contexts uh is really important uh as a way of having representative sampling uh and also being able to do it quick enough to allow for things like early warning um and so I think that's another component uh of what the Billy app seeks to do is actually provide donors and humanitarian actors uh more timely information when a disaster or shock occurs URS because communities can tell you right away and it's managed also by the local Partners themselves so rather than spending $300,000 On A needs assessment they have immediate tangible actionable information at hand provided from a reliable source um that can be collected digitally without putting enumerators at risk or spending a bunch of money on infrastructure in the field to try and collect that data and then the goal is to try and flip the system on its head uh to give communities control access and decision-making over their own information um and part of this is looking at how blockchain uh can help facilitate that and how web 3 can help facilitate that um and we know that uh storing data on the blockchain can be a very burdensome time intensive exercise but new technology uh around data canisters uh and other systems actually allows us to do this in a much more efficient uh democratized way and there's going to push back in resistance to this because people like being proprietary over their own data they like spending lots of money on the same survey in the same communities all the time but we do think that over time making that information public where we can focusing on respondent data security anonymization will actually create a better relationship between humanitarian actors and the people that they're working with because they're allowed to give them instant real-time feedback in a meaningful way to improve some of their systems and actually make things more agile and responsive thank you very much all right thank you so much for joining [Music]