12. Impact Measurement for Digital Public Goods - Taipei 2023

[Music] this one will be about the impact measurement and today we have two very uh very different but also the same like they work in the same field but they have very different background one is stepen he's like kind of from the traditional more of the traditional impact measurement uh or or at least web to impact measurement uh scene and Carl Hill is from the open source or web free impact measurement so comparing these two different experiences I can't I I think we can actually get to a really interesting conclusion on how each of them can learn from each other uh first I would like to introduce them and welcome them to like also introduce yourself um Stephen would you like to go first and how you're related to impact masterman and uh who are you uh good afternoon everybody yeah I hope a traditional does not mean I'm old you're not okay uh I'm I'm Steven Yang uh founder of uh kopy impact investment and uh we focus on uh social business and uh and uh Bop as our investment targets and we also believe uh maybe in the future uh when the mainstream uh business uh become more serious about uh substainability transformation then we might start to look for uh search targets okay and when we are doing a impact investment uh we have to uh deal with how we uh majure those uh social impact so uh it become our uh daily opportun uh daily work to do the uh investment uh evaluation so that we can uh evaluate their social impact or intangible uh value uh to set up the investment terms okay uh we also provide uh consultant services for uh such thing as uh transformation of uh sustainability and one one Lin of our service is about uh helping company to do the annual sustainability report we use the approach of sroi social return and investment assessment as the basis but we skip the difficult part which is money monetization modeling so people can do the annual report and still a you know uh maintain the tracing of their stakeholders impacts and that's what we do I think what Stephen mentioned here is really important to fund in the commment since we have funding in our name and actually having a few funders here to share how they actually evaluate different project is very important for all of the open source project that are looking for funding like how exactly are the investors thinking about impact when it comes to like me like measurement and like how valuable your your your project is um Carl do you want to introduce yourself a bit sure um well first off thank you so much um to to Noah and to Vivien and the funing the commons organizers here uh this has been like the event that I've been looking forward to all year and so I'm really really happy to be here um one thing I'll add to the introduction that Vivian gave is I actually I also have a background in in traditional impact measurement so for the first part of my career I did work in like micr Finance International Development helping companies with sustainability compliance and ESG reporting so I've been able to see a lot about how that world works and then it was through that lens that I started viewing some of the innovations that were happening on Crypt networks and in funding open source and digital public goods that said ah this is like really fascinating and I saw an opportunity to hopefully help some of these networks speedrun a lot of the lessons that have been learned the way that public goods were traditionally funded and how impact measurement was traditionally done um so uh over the last year or so I worked um with protocol Labs on the hypers team more recently I've spun up something new um with uh with Ry who's somewhere in the crowd here um we both work together protocol Labs called open source Observer um we have a company we've set up called kba um which we doing that work under and the goal really is to get really good at measuring the impact of Open Source software because if we can do that then Network funders ecosystem funds foundations they can deploy Capital much more effectively towards driving open source impact um and I'll give some examples of that in a little bit but again thank you so much for being here I think one of my first questions would be uh for Carl like it's very interesting for you uh to see someone like actually jumping from a traditional impact uh scene to the web free one what what is the connection there and what really speaks to you that makes you want to learn about web free and its impact so there's a couple things the first one is that having having done work previously where you would need to go and interview people you would need to go onto farms and do surveys and gather data um like you know record keeping and writing that up and scanning it and so on I've seen how difficult it is to get good representative data to measure impact um and so one of the great benefits of working with digital public goods is that most of the work most of the artifacts are being produced are digitally native uh and so you can skip that step of needing to digitize what the work is um which is a real Advantage if you want to be able to get better at measuring impact being able to start with relatively clean easier to work with data is a big Advantage um My Hope of course is that we can learn things that are useful for measuring the impact of public goods and then extend that to the rest of the world I don't think it stops just getting really good at measuring open source software impact I think the goal would be to develop some new tools and some approaches that can then transform the way public goods are funded everywhere so what you're saying is basically to find an entry point to eventually evaluate all Impact that people are doing but open source is only one of the endle right and you choose open source and wrey is kind of Open Source itself so it's kind of a really good entry point uh I want to come back to Stephen uh to talk a little bit about like the traditional method like you mentioned sroi what is uh uh what have you been doing with s how do you like really measure impact can you walk us through the the process a little bit and also tell us like what is the biggest challenge of measuring real world impact okay when we do the um impact investment we pay attention to uh many aspects for example uh any uh action plan could influence uh the target uh stakeholders for example uh disables and their family but not only specific individual or organization but also the nonspecific or general impact or outcome for example uh our action might raise the social awareness it's not it's in general it's not for specific person or organization and awareness can be measured and for we also pay attention to the uh action taker action taker side because action taker might become more experienced more skilled and their service my St my become my turn from uh peace meal to more systematic service system right and all this progress can be majored okay um let me see uh I missed okay so uh canop impact investment actually maintain uh all kinds of impact or intangible value can be measured that means once they can be ma sure they can also be managed and that's uh our belief like for me uh I've been I've been through this like impact measurement process for some like really social project it's relatively not environmental because in like within ESG like environmental is relatively the one that's like easier to measure there's like carbon credit there's different system around it but for social like how people interact with each other and Stephen mentioned action taker each individual in the within the ecosystem are stakeholders and how you actually consider all these stakeholders within your measurement would be the key what what is the like challenge there for engaging these people and is it stopping people from measuring impact because it's relatively uh subjective okay um for both the user part and the uh then the action uh taker parts we face all kinds of uh issues for example for the action targets uh many people don't they seems to you know believe there must be some kind of value like they cannot major at all so they actually didn't take any action about how to Major those possible outcome and try to manage it and that's about the awareness right and even when they uh be aware of the possibility now we will uh encounter the uh Readiness of their information system because they need to start from zero to start build up the system so they can they can start to collect the relevant uh data or information and also for the action taker side you know what one issue is about the uh quantitative approach if it come to specific quantitative goal like monetization or the intangibles it become you know technically the threshold of Technic the technical threshold become quite High okay so that's one reason you heard of annual sustainability report annual CSR report annual ESG report but you never heard of any s annual report right and there's also uh one uh issues about uh the let me since I don't remember last time I speak English in the public uh meetings so I'm prepar for some uh talking points so let me check if I miss some point I want to make okay another challenges is about because this Market is still not big enough or profitable enough to to attract enough resources and talents okay so for service uh providers okay it's it's not many it's not uh I would say the capability and the capacity still need time to build yeah uh what what I hear that's really key is uh to start with at least something even though measuring impact can be very uh subjective it can be from your own perspective like a few perspective it can be really singular but at least it's starting to measure impact instead of just like leaving it there and don't believe that you can actually create some system around it so that's a really important mindset I think around it uh for that's for the like physical world uh I would like to jump into the digital world a little bit uh Carl can you tell us what exactly is digital public goods so that we can start diving into like measuring them sure so digital public goods that includes everything from open source software protocols programming languages open data sets public blockchains um digital comments place where we go and chat and share information I think increasingly as the boundaries between our physical life and our Digital Life get more and more blurry we're going to see that there's many more of these spaces that are creating a lot of value for society but don't have good governance built in and so part of the problem that we need to solve is not only how do we fund these digital public goods but also how do we effectively govern and decide who has a say and who who is able to effectively allocate resources determine what things are are needed need for these digital public goods and Commons to to flourish um we we we learn about how like uh physical impact is measured like what do open sour open source like Observer does like that help with that process for digital public goods so one of the things which is really powerful about crypto networks and and web 3 is that you get both a social network and a financial a set of financial rails out of the box effectively so you can see um you know how money is Flowing who contributing what where it's coming from and you can start to construct these graphs now in the real world doing that stuff it's possible but it's very difficult and it's often very localized um the real power that you have with um with digital public goods is that it's a lot easier to start observing some of these patterns and identifying you know layering data sets on top of each other so specifically what we're trying to do is take um kind of the financial layer so looking at public Goods which are receiving funding from different ethereum based um Grant ecosystems and then adding in the layer of what kind of um open source activity do they have so looking at code contributions looking at GitHub repos looking at contributors and then in addition to that we're also trying to layer in um the role the different kind of packages play because you you have um software which often relies on you know different dependencies and so you can actually build out a graph of how different software packages relate to each other um so just right now with those three data sets there's a lot of really interesting things that you can learn you can see um you know for instance you can see library that has been used by lots of applications and those applications are generating sequencer fees on a blockchain and then that blockchain might want to take some of those sequencer fees and turn that into grants that fund the libraries and so through these types of things you're able to build these really interesting regenerative financing systems and all the data to measure who's doing what and where it's coming from is there is visible um so instead of me needing to go and interview lots of people or build out a proprietary data set I can just start looking and start going where the data is and trying to surface Insight which I think is a really really powerful ability um because if you can do that you start to go beyond just looking at each individual contribution or each project and say how is this entire ecosystem doing how is this network of all of these different projects and contributors how healthy is it and what types of contributions or what types of Investments are leading to more growth or more positive outcomes for that for that Community got it so now we got the high level it's basically looking like so for open-source uh software there's already some data available uh that people can like that you guys can already use to like measure measure the impact I kind of want to ask you both of you to give us an example of how you guys measure impact for like one really specific example uh what is the process and how did you approach it uh that start with Carl um sure so one of the one of the big things that we're trying to work on right now is this indicator called Downstream impact so basically you have something which is very visible on on a blockchain which is how many users are here and how much how much fees are they generating for the network so anybody can go anybody can quickly observe that kind of activity the the more interesting question is say okay what are the different tools Upstream of that impact that those applications are using that enable them to deploy more quickly or provide security or provide other services that are necessary um and so what we're trying to do is Trace Upstream of that application layer what other things are happening and determine a way of measuring their respective contributions towards that Downstream activity because if you can do that then you could have someone who is creating a a software package has no Revenue models not charging users anything but everybody down stream of that who is using it and generating fees or you know extracting Revenue you can see their contribution and then you can actually fund that Upstream um so that's again the indicator is Downstream impact and we believe that that is a way of actually trying to take funding which comes from end users and Rec recycle that recirculate it back um to some of the contributions that happened Upstream of them so basically saying like there's a lot of uh like funding that is actually allocated into like the application layer but not very much to the fundamental like building blocks of each application and with like tracing need Upstream you can actually do that uh with a more traceable manner uh having everything on chain yeah got it what about stepen like uh okay for this year we are doing a project about uh the social housing the impact of social housing in taong city and when we are majoring the residents not only uh the financial benefits from living in that housing but also sentimental values interpersonal benefits or Community recognition or knowledge ability right all these kind of in uh intangible values uh is the most important is the most difficult issues we have to deal with the data accessibility data availability so one important approach is to do the stakeholder survey we cannot rely only on a let's say a digital tools we have to direct uh get we have to directly get the data from those stakeholders maybe one stakeholder you can say it's subjective but how about all the stakeholders sub subjective evaluation it become relatively objective so you can gather the data so I think uh that's why uh for S social return and investment approach their first principle is always get stakeholder involved that's very important approach for for us to address those uh data accessibility and data availability issues I think it's really interesting that uh Stephen here is pointing out how collectively with uh different like singular perspective things and action be become like plural it's like different voices different perspective and everyone contributing uh their own View and aggregating everything and make it the way that you impact you measure the impact I wonder how like cars think eventually the approach we are uh having for the like digital public goods or open source like uh impact measurement can like assist with these kind of measurement in a real world since initially you were actually from this side from this side right and you jump into web bre because it's like kind of an entry point for metion so what do you think is the tiping point of like actually having these tools like helping the real world um yeah I mean the the challenge with impact is that it's it's very subjective right and so something could exist and it could impact me in a different way than it would impact Stephen or would impact you and at the same time your relationship with that impact can change over time based on your context or how much experience you have um so I think one of the one of the mistakes that we've made is trying to search for some kind of a magic indicator which solves all your problems and you can say everybody needs to be optimizing for this one thing because that is the ultimate indicator um I think it's worth looking but I don't think we're probably going to find something like that I think much more of it is going to be able is going to require giving different types of communities and different types of users the ability to sense make and to surface or upvote things that are more important to them um it's very much a collective intelligence problem and and we should be embracing the fact that the more people bringing their views and their different you know ways of making sense of this and determining what is impactful to them is going to give us a richer understanding of of what things are are useful and what things are um you know most impactful uh in in my in my former life um one of the things I used to do was actually help companies measure their progress against sustainable development goals so you could have something like child labor or deforestation the problem with indicators they're very and there's that's Geneva or somewhere says here's what uh child labor is or here is what deforestation is and there's an expectation that the farmer or the family that is being measured against that standard has the same understanding and the same kind of you know ideology behind it and in most cases the people who are being measured have no idea where those standards came from they have no knowledge of what that actually means and why that is viewed as a good or bad thing um and so again in the traditional world that I worked in I feel we constructed a very unfair system um where people were being measured against the standard that they had no autonomy over and they had no input and in many cases had no understanding of why that was an important thing for them to be um trying to improve or upgrade against um so I think that ideally what I'd like to see is much more bottomup standard and sense making um communities figuring out what matters most of them and being able to upload that or Surface that to influence where where funding comes from and what things we're able to produce more of so what someone is clapping someone is applauding that yeah so so what web fre can the decentralized network can can bring to this uh this whole discussion is actually around how you can build up that relational layer of measuring impact like bringing people in instead of like having one centralized Authority telling you what the standard is and everyone just like blindly believe in the system they said and uh Stephen I I wonder like what you think is like this kind of decentralized networks value to uh the like uh the the network or the the impact work that you you've been doing for measurement for any kind of like blockchain any kind of this decentralized at work like what will the What is the value okay um probably there are two way uh two kind of approach one is uh top down the other is button up top down has uh been accumulated lots of uh domain knowledge already so we don't have to start from zero okay so if we want to make a more comprehensive or or more healthy framework for button up approach I think actually we can um kind of um leverage the previous accumulation of knowledge and the wisdom okay so for example uh when I when I'm talking about collecting data first how do you know what kind of data is relevant or useful you need to have some knowledge base right okay so for the development of a decentralized uh framework I think it's actually it's also uh very very effective and the very resourceful and very convenient in the future I believe so I sincerely believe so but it take times but now you know different part of uh this uh action taker They Don't Really Got A continual opportunity to kind of do the inter inter uh disciplinary uh communication and the dialogue right so uh maybe this is why uh viian invite me as a panelist to talk with car okay so I I would say all kinds of approach I welcome all kind of approach the most important things is how can you get reliable and useful data that's the most important things and data source can be from all kinds of source okay so of course it take time yeah I guess like one of my takeaways will be like interdisciplinary will be super important for it's like kind of the building blocks of uh of of bottom up because you want bottom up you want Collective wisdom on the thing and at the same time as opposed as opposed to like one authoritarian uh like data set or a standard that's like against it uh that would be something that I think the A decentralized system can actually help uh build uh forward uh any you guys want to add uh like what you learned or uh just like last word before we finish this uh this panel because it's like time is up nothing else to add if there any questions we can we can take some questions most excited about so hul ask what are you guys most about impact measurement in general what are you most exciting about okay are you exciting are you excited yeah I would say everybody might for those uh uh for those who want to make changes they all have their own theory of change if you don't have data to feedback you never know what you do is right or wrong you need something to make you make adjustment right so you need data to verify what you do is wrong or right or what you what kind of change you should make and that make me excited because that actually make me improve and grow as a change maker right the Improvement yeah growth yeah and also uh I heard I often there's a issue of being raised is about uh okay this um quantitative approach is not you know it's cannot solve all the problem sometime you have to rely on qualitative approach but I will say both quantitative and the qualitative approach actually is to provide different set of data it's all about different set different data right but for specific purpose some data could be useful and relevant some are not so it depends on what kind of purpose you want to you know realize learning about the data learning about everything and using the data to improve your own theory of change and that Improvement and learning about everything basically excites you Carl um sure I'll take the question so I'm excited about public goods funding I think that in general there's a perception among younger people that public sector work government funding things like that are really boring and a lot of young people say the last thing I would ever want to do is go work on something in the public sector or funding public goods right and I think this community in particular has done such an amazing job of changing that perception and making it so that many young people today and I meet all of them all the time like I'm one of the old guys right but like I I go to events and I see all of these new people that are coming in and saying yeah like I care about governance I care about bottomup data collection I care about digital autonomy I want to support these things I want to fund them and I want to get vocal about expressing the types of things the types of public goods that I care about and that's like part of your identity um so I think it's really really exciting now that this whole new generation is getting excited about governance excited about trying to fund things that matter to their communities and they now have all sorts of new tools to do that in a completely different way so I'm really excited about that that's a good ending I want to add one that's one thing that's not the ending I no actually I forgot what I he for okay okay I yeah now I remember okay I maintain here all kind of intangible data can be major to some degree if you don't believe what I say is true just go refund your ticket today oh no no no I'm just kidding just come to me and I will try to you know prove to prove it to you yes I think we had a really good panel today uh thank you all for joining us today and thank you Stephen thank you Carr it was really amazing to talk to you I look forward to talking to you more okay great thank [Music] you