6. Impact Evaluators: Existential Challenges and Opportunities - Bogota 2022

great so goals for the session um trying to speedrun through this but we're going to recap impact valuation retroactive funding and how that links to some of the broader public goods ecosystem Concepts we've talked about uh second we're also going to present a framework for impact evaluation design and operation we're going to talk a little bit about some of the kind of you know 100 ways to do an impact valuator the wrong way that we've learned in uh protocol labs and then I'd like to open this up I think there's some existential problems that we could really use the community and a bunch of diverse opinions to help help solve so what is an impact evaluator it's retroactive publ good funding it's a impact evaluator essentially it is a mechanism that funds projects based off outcomes that have been achieved versus um expected outcomes so this is typically attributed to optimism who pioneered it with a retr pgf experiment um a lot of others have done this in different formats um right now I'd say where we're at is like we have a good starting point in terms of theory uh base experiments but really I think it's there's a lot of problems space to cover when it comes to how this actually manifests the design implementation and iteration of impact value raters which is what we've been trying to do at protocol Labs so there's some great reference material out there by Carl from optimism um our very own Juan and Evan um have given some good um general reviews here so encourage everybody to check that out so we'll start with what problems do impact evaluators actually solve and I think there's this you know this common Mantra of well we're we want to make sure that people get rewarded for the work they do I'd say I would put two flavors on it one is it represents a missing component of what let's call the you know the market dynamics of a public goods ecosystem so it's value signaling um right now we're seeing a lot of kind of proactive funding through various types of Grants um bounties Etc uh Foundation uh funding but what we're missing I would assert is a very good feedback signal that allows these funding systems to self adjust and identify if the projects they're funding and the way that they are funding them are actually creating the outcomes that we are Desiring and that there's a lot of ways to pick that apart about what is a good outcome um how do we actually evaluate that how do we measure that what we think is that impact feedback and impact evaluators they add this missing component that allows us to back propagate signal about what is valuable back into a market so people can actually be more efficient in allocating the capital in the projects that they choose the second part is when we think of it from like an incentive design perspective um there's a bit of a gap right now in public goods uh that see you know I don't want to over financialized but you know the the analog would be something like VC and this is idea that you the funding system profile the risk and the reward and the uncertainty needs to match the projects that they're trying to fund and in order for us to see those really kind of high uncertainty High upside problems emerge in public goods we need large scale High uncertainty type funding mechanisms to support them we think things like impact valuation are very good you know opportunities to do that and impact evaluators linked to this broader you know strategy that we've talked about um impact valuation which hula and um Kevin talked about earlier um is a component of this hypers ecosystem and it's another kind of transaction within that ecosystem that we need to enable to actually support this kind of broader impact Market um or market dynamics within public goods and it's an important part of this kind of composable refi funding ecosystem where you have projects different types of funders different fights of me types of mechanisms all working together to solve the best and most effective public goods projects work towards the outcomes that we want but impact evaluation is really hard and a lot of people have tried it's we've tried and like we've you know failed to various degrees um but we know that we need to make it more efficient and scalable I would draw the analog to we've seen kind of valuation methodology in capital markets Finance um evolve over hundreds of years so over that time you know new methodologies new Frameworks exist you know discounted cash flow comparables analysis that allowed people to independently and more effic efficiently create an impact evaluation um about you know what how valuable something was um which really shorted the kind of transaction cost that it took for people to understand what is valuable and allocate their funding towards it what if we had this concerted effort and we accelerated it so we build the methodology the tooling the theory to actually lower the transaction cost of an impact evaluator and what would be the impacts on the public hoods ecosystem I would assert that by lowering the transaction cost of an impact evaluator we can see a lot more retroactive funding through and we could see a lot more efficient um work done within the publics ecosystem so when we think of this I think one of the things that we've um tried to explore is how do we create like a base framework for how you would set up an impact evaluator um I think it's like it's a broad concept but like what are the things that you actually need to consider when you build one out so I would say there's four key things from a design perspective one you need a good Reserve asset pool um you need an incentive function you need contribution tracking you need value measurement so Reserve asset pool basically means where is the capital that you are going to or the capital or the reward system whether that's social credit that's um nfts whatever you want where where does that exist and how do you actually make sure that that is maintained over the life cycle of impact evaluators especially if an impact evaluator is retroactively planned for two to three years in the future how do you maintain maintain that asset pool so you can actually allocate it done in the future incentive functions I think this is super interesting because you really have to think of it from what are the outcomes and the signal and the behavior that I want to incentivize through an impact evaluator and how do I shape the reward curve to do so so I can do kind of a reward curve that rewards everybody equally and you'll kind of get equal outcomes across all the different projects or I can do a kind of power law Distribution on rewards and then we start to see the emergence of projects that are going after very high upside uh you know type prizes think the X prize think um some of the the Fiat connect work that's am was talking about earlier it's also important that it's positive sum so that people competing in any of these impact Valu ators are actually working together um and an example of that is you know the more work that is done against this this area um the more reward is released so think of that as you know as we kind of like as more people join an open source software project more rewards are released so the addition of of one more person to this impact evaluation project actually creates more funding so there's an incentive for people to bring in and work against this um also you have to think about in terms of repeated application so how do we keep on doing this because it's going to take time to propagate the signal and cor and create like a proper um control system Tye Dynamic there um contributions are important figuring out what the unit of contribution is that actually you know coordinates or indicates this impact evaluator is very important and value measurement is very hard because it's how do we kind of create a function that translates what we think is value in you know the terms of the impact evaluator to kind of the Fiat and the reserve currency and I think the Nuance part here is how do we Design This with an acceptable loss function or a loss coefficient and then adjust and what I mean by that is any sort of impact evaluator is going to have some people or some entities that will um work against you know the the Ben or the idea and it's just kind of like the good Arts law if we create any sort of metric people will game against it and it's up to us to understand what is the acceptable loss coefficient of you know Bad actors in the system so that we can actually still generate good work so I'll skip over this really quickly I think one important thing in the framework that we're looking at is like if you're setting up an impact value it's really what you want to keep free and like what do you want to actually like have the community or have a system create so thinking about you know what are these different degrees of freedom um you can think about like what is the scope of the evaluation that we're creating you know what are the contributions what constitutes good work against the outcomes we're looking to create um what is kind of like the reward and allocation system how do we size the amount of reward that impact evaluator should give out and then how do we allocate that among the different contributions to an impact evaluator in a way that actually um you know properly rewards incentivize to people now what we've seen is that these are really hard to do so you have to constrain some of these and set these centrally so that you can actually operationalize it um you need to focus where there are expert inputs um you need to focus focus this on very specific steps um and constrain the rest and then finally there's kind of this concept of like you can almost do a top down where you're setting we've seen like a retro pgf for example set a you know prize pool and then that prize pool is allocated the you know the judges vote to actually allocate prize pool um that's one way of doing it or you can think of something that's more can do a block reward where you set up the reward function at the basis and then it just adds up so every additional unit of work um increases the reward so this is a way that we've thought about um depending on the situation um actually constraining the system and making it more operational so protocol Labs has been running a lot of experiments on this um to quickly explain kind of like what this looks like these are various different experiments that we've run um kind of more broadly and then focused over the last 12 months so how I would read this is on the bottom as you go more to the right you're making a trade-off of the operational cost is higher so it's it takes more from the people who are actually interacting with the system who are voting um it's higher lift on them but the system setup cost is actually lower so the example I'll give is like filecoin filecoin is an impact valuator however the the system setup costs are extremely high you have to basically build design and deploy a protocol which takes a lot of thinking whereas you can do something that is more on subjective quantification which is an experiment we ran where we get a group of experts who basically determine everything they determine what projects were valuable they determine within the scope of PL how much value was created and they go and vote about how much value should be allocated against a bunch of those the benefit of that system is it's very flexible you can literally take a bunch of experts and say evaluate anything but you don't get any scale efficiencies and it is really really hard to do so example is filecoin massively powerful system um very hard to design and takes a lot of setup time then we look over to things like slingshot which is actually building on top of the core file coin protocol and actually is an incentive system designed to onboard storage providers so so think of that as like an Impulse function that actually had to um we had to build infrastructure around that and that's been kind of accelerating storage provider adoption um over the last kind of you know 12 to 18 months I believe then we have something you know we did a kpi impact evaluator where we tried to um quantify ecosystem activity um and Design systems to measure that um and you know that was that was a mixture of system design where we actually created quantifiable metrics but also had um various experts feeding us like feedback on you know what was important during a time period and that so I think the key takeaway here is that we've actually experimented a bunch of these different things and what we found is through these experiments there's different ways of kind of formulating this impact evaluator problem and you're really making a tradeoff of how much time do you want to spend creating this like very intricate system that um accounts for all of like the edge cases and Bad actors versus how much do you want to quickly deploy a you know high lift operational system um that can be applied against anything so we could do an impact evaluator about like who was the most valuable speaker today and we would just get a voting system with a bunch of people who are you know evaluators the the audience and they would evaluate based off kind of like a value system that they choose and we could use that to you know give out clout um or maybe give out more fun in the Comm teach what whatever whatever your value function is so I think what this has led to us to kind of think of is there's a little there's a few different archetypes and how you can design them and like where they can be applied that is I think Super useful so I think starting on the left you've got this like Clos quantitative protocol level impact evaluator this is something where all the rules are set all the impacts and or the inputs and the outputs are measured um systematic developed this is the most powerful but it's also the most rigid so you can think of like you know there's the Galaxy brain um I've created the ultimate it protocol and like block reward design um which creates massive value like Bitcoin um it could also turn into a paperclip uh making AI robot that just creates Pap paperclips because it's been told to optimize on paper clips and reproducing as many paper clips as possible as you move over I think you're seeing different paradigms where it's how much of a system do you want to set up to create this impact evaluator and then how much do you want to let experts actually provide input and experts are a little bit tricky I've used the kind of analogy of herd and cats if you experts do whatever they want um you're going to get a lot of like really really wide varying and not necessarily super useful um feedback whereas you kind of have to cage the cat a little bit and give them very specific directions on what they are giving input on um so you see some things like you know retro pgf or some of the work that we did with like um voting about like what was valuable contributions to open source software um you give them a constrained set of output or constraint set of inputs and then feed that into your broader system so this is all playing back and forth between like where do you actually apply these so I would say close quantitative protocol that's your layer one incentive system for impact evaluators then you move over I think a quantitative system that you can layer on something like slingshot that we did was more of an Impulse function but you have to be careful because it can warp the existing cryptoeconomics in a way that you don't necessarily want by introducing kind of like weird incentives um that Cascade over time then we see like restricted expert systems and I would call that like a retro pgf um type where you have you know a specific set of projects in a specific scope and you use that to um kind of like aggregate expert input when you Scope it down so that they're only giving input on very specific things that's the most scalable um but it's also the most restricted and rigid so really it depends on like what is your what is the shape of your problem that you want to apply an impact value word against and then understanding the design considerations because there's no bad and I stole this from somebody else but like there's no such thing as a bad answer there's just trade-offs um so it's like really kind of like the degree in which you want to make that that tradeoff so I'll go into some quick takeaways and then I'll open it up because I want to get some some feedback on some interesting problems we've run into but takeaway so far has been that you know quantitative systems within web 3 particularly there's just this massive infrastructure gap for measuring ecosystems so how do you measure an ecosystems activity in a decentralized objective way because I think we're all so used to the idea like oh we should have metrics about everybody who's using the network but that implies like a central measurement system um which kind of goes against some of the ethos I think there's there's good work being done about how we can kind of like aggregate opinion across like decentralized networks but that's beyond my technical knowledge for this talk um so I think that's a really important thing so really you see quantitative systems emerging around um existing quantitative infrastructure so how do you kind of use onchain um which are like verifiable metrics to inform your impact evaluator and build on top of that for for expert systems um the rate limiting factor has always been expert subjective input um I see a dangerous hole that we ourselves fell into which was whenever we didn't have a good quantitative system we'd say oh well just get experts to vote on it um but the chances are that you're experts you have to be very careful about selecting them um so that they have the right domain expertise and the right amount of bias and mixing that group and that pool but also understanding that we are such a small field and a lot of the things that we're kind of you know impact value impact against um have such Niche knowledge that the people who know this are the most time constrained and you can get 20 minutes of their time every six months um I think there's you need whenever you're introducing new incentive systems um you need to account for existing forces at play um that's something we learned um because we were going through the process of launching like some larger scale impact evaluators and we had to talk to the cryptoeconomics team that designed the filecoin ecosystem and they had some strong feelings that we would warp their existing system by introducing any additional incentives um I think right now we found the most success when you have impact evaluators on a very tight scope and as a component of a larger program so we've uh we've talked about work that we're doing on Blue funds which are decentralized um kind of like public goods funding efforts and when you link uh grants funding uh bounties and impact values together under like a very defined value capture umbrella um it actually works pretty well because you can kind of measure the kind of impact a lot better as an isolated and you're seeing this also with uh things like you know I think optimism has done a lot of work great work uh working with the ethereum ecosystem and um being able to kind of you know link that together and just have like a very specific scope among other existing things like gitcoin so I think one thing to is that transaction cost limit impact valuations significantly it's really really freaking expensive to run an impact valuator in terms of Machinery in terms of timing all this sort of stuff so right now we can't run it on everything it's like you can't do small tail reward systems at least in my opinion um so we need to be very careful about the experiments we run and it almost necessitates a large volume of capital um to kind of like Supply the overhead and this is something that like it's a new it's not just like a a mechanism design it's so not just like a technical problem it's also a it's a behavior change and people need to start going through the expectation of reward and this is going to happen over time because it only works if people expect there to be a retroactive um evaluation of what they did as valuable so what we end up there is like there's kind of a you know a cold start problem of the initial funders are going to have to understand that they're going to have a big loss at the beginning or like more loss than the steady state because they're going to have to build behaviors and build expectations as they go so I'll skip this really quick I think the there's a lot of experiments going on about ecosystem scale um open source tool Ling needs to to improve to increase the efficiency of all this expert input um you need we're working on composability with other funding mechanisms to link them together in a better way I would propose a few future directions and this is just coming from the perspective protocol Labs but I think it's helpful to start this conversation is I think we need to develop cor more like quantitative impact evaluations um with measurement systems I think a lot is being done on Expert systems where you have people voting on a specific scope I think for us to have scalable mechanisms we need to build out more kind of quantitative systems uh we need to expand the tooling system suite and Link for op operability there's a lot of different approaches out there the real thing I want to focus on is like crowdsourcing experiments and coordinating within the ecosystem because as we heard like during the hypers search Talk there's a lot of people who've been working on this problem in the past in various different shapes and sizes and there's a lot of people working on this in parallel and I think there's a lot we could learn about each other hopefully this is a first step about us talking very you know openly about the things that are working and aren't um but I think we'll try and launch something to get you know if you're doing an experiment on impact valuator come let us know and like we're happy to provide feedback supports um tell us where you know ours has worked in the past and like where we might be helpful um but I think getting the community more oriented so we're not working in silos is uh super helpful and I think there's just a lot of like need for more scalable uh expert systems like quadratic voting or quadratic funding so things like liquid democracy uh just making more scalable use of expert time so this is the last part and would like to kind of open up to the crowd here is there's a lot of kind of what I'd call existential problems which is the stuff that keeps us up at night um that we need people smarter than ourselves to figure out and I put some kind of directional Solutions down here but you know these These are open problems and like we don't have it all figured out yet but um I would start off with kind of you know at the top Roi of retroactive funding and impact evaluators um I think the key here is like how do we prove that somebody should spend a mil ion doll on on an impact evaluator instead of a million dollars on a grant program um and until we see you know that actually happening in practice um I think like that that is and how we solve that I would say is like I think we need a little bit of development about the theory of like how do we establish a counterfactual for impact evaluators like you know if we spent the million dollars somewhere else in this example um what would be the impact um would that be a better use of our time um I think some of the stuff that we can do there is like there's a lot of like good modeling that seems like BL block science have done and just like building ethero that would really help us push forward but the deliverable there would be is like if you have a system that can tell me that I should spend my $5 million across you know grants bounties and impact evaluators in this optimal mix that would be huge um particularly you know the the dollar spent on impact evaluators is net positive Roi I think that would go a lot way to to the scalability um second thing is like creating an appropriate Reserve pool to like reward and like just the capital efficiency issue of impact evaluators especially when it's future you need to have a pool of funding that isn't going to disappear and especially when you start thinking of like if we want to reward impact exponentially which there's a discussion about whether or not that um is actually needed versus like rewarding to the level of incentivization you need to make sure that that capital is available to be retroactively rewarded or else people won't believe enough to actually go after these these problems so you know and also when it's just sitting there it's not very Capital efficient especially if it's a depreciating asset um it's really hard to justify oh we're going to have a bunch of money sitting here and we're not going to do anything it for year just so that um it's available for this eventual retroactive reward because we think that there's benefit there so I think there's a couple ways we could do this um you know creating Aqueduct from ecosystem token pools has been effective in the past you've seen that with like Foundation models where people are creating pools of capital that think about public goods in the long term uh staking so taking the tokens that are meant to be or tokens in this case being the reward pool you know putting that in a way that it actually generates some returns but actually has like ility to you know be retroactively funded and then things like you know ecosystem taxation redirecting gas fees towards impact evaluators um I think you know you could draw an analog to some of this interesting stuff that optimism is doing with that so um that that's one potential area um this is a really big one is incentive systems for impact Valu entities we talk about this idea that impact evaluation is great it's going to kind of back propagate signal it's going to tell us what's valuable but why do I as a person who is smart and as an expert um why do I spend my time evaluating impact and like going through the process of like issuing this for third parties um so there's a couple different ways we can think about that is like you know maybe there's a reputational staking system um where that reputation of like you have actually contributed actively um similar to kind of like contributing to an open source ecosystem um there's potentially like a way of aligning incentives with like sticking upside um whether that's like you know locally in ecosystems um that's a big problem that I would love help with um and then the last is just like it's really costly to run any of these experiments right now um Almost Doesn't justify the amount of money that we're able to send out because um impact valuators haven't been massively scalable yet so any tooling methodology and I think this is more solvable and tractable the more we put effort into running these experiments building the right infrastructure the F like the better our transaction cost will be um this is like over time this has been proven in pretty much any Market uh we just need to continually build and have the incentives there to do so so we need help here um we're also hiring so if anybody wants to join the network funding team and work on these kind of problems uh we need startup operators PMs and research scientists and do we have time do we have time for one question got one minute one minute um all right well we'll open it up any any questions criticisms comments hi so with retroactive evaluation of a contribution and Awards being allocated at that point how do you avoid turning off people who are contributing in good faith but then you realize that maybe that's not the most valuable contribution like some I'm kind of listening through this I'm worried about like some inherent BS from people who are trying to do something positive and you're like yeah you don't get any money for the last three months so it's a it's a great question and we actually ran into this in one of the experiments we did on an open source repo I think it requires careful calibration um and this is where it's like the fine-tuning of the reward function is so important I think like and it's a mixture of like there's monetary reward and then there's kind of like reputational and it's kind of you know I don't I don't say Good Vibes reward but like there's like a social component too and the truth is like it's it's always going to be a balance like you're going to throw some people off but the idea of like can we actually incentivize like net positive impact by you know introducing these incentives and like showing and signaling that we are like as a community valuing the inputs um it's it's really it ends up to being about the design about like okay if we're doing it for a three-month period like we're acknowledging things beforehand um and like it's about how you set your scope and it's just like understand the implications of every kind of design decision you make is important and there's always going to be a factor of like you might piss some people off but is you know is that net positive for the ecosystem or the project you're looking at um that's for you to decide and like in our case we overc calibrated rewards and I think we in retrospect we actually went back and um adjusted it but I think we undervalued contributions from open source members um earlier on and had to kind of like a retrospectively say like okay we we messed up um we didn't calibrate our rewards properly we need input from maintainers from the open source contribution about like what is the right way to do this and we adjusted our reward function not to be just like monetarily or just not focused on like for example like a PR and a repo as like an indicator of like overall contributions to um like a repo but more broadly about like you know reviewing codes um you know contributing to the community Through comments and it just required us to recalibrate our um kind of reward function um to account for that