Impact Evaluation With Counterfactually | Sejal Rekhan | Lau NaMu | Devansh Mehta | David Dao |

[Music] so I see the fun up we can start so welcome um today I have the honor to give you a speedrun on data D impact analysis when it comes to web free especially in governance and impact funding I think one of the most exciting spaces that we as a community can have especially considering that it's the year of the regenerates but also considering that we as a web fre Community have an unique opportunity because we have so much data on chain to leverage that to make better decisions and show to case to the world that this is a really really good system and we want to make it right I'm honored today to be uh joined with two amazing panelists you will hear a lot about them um in a just a couple minutes we're going to introduce uh Devan and laamu who are both pioneers and legends in that space and I want to give it to Devan first to introduce yourself quickly so um hey guys my name is is Devan and I work in the ethereum foundation leading their Ai and public goods track um so I've been looking mostly into how we can um create like Ai and ml competitions and how we can use that to make the space of funding capital and like identifying Cil and other things uh more accessible to machine learning model Builders perfect hi everyone my name is lamu I'm the founder of metrics Garden Labs we are an imp studio and we focus on several different things one of those is Consulting uh we've worked with optimism with the ethereum foundation and with some of the other ecosystems in mapping out what impact looks like in their ecosystem so that they can appropriately fund the work that is being generated and contributing to their growth let's jump straight in so public goods funding in the ethereum ecosystem one of the most important things that we can talk about because of our software dabs built on top of repository that are open source and also the community that organizes events how is it currently funded LA and what do you see are the benefits also challenges of how the system is currently doing it yeah so I think as we've seen across ecosystems one of the things that's being funded the most is infrastructure the reason for it is because it's one of the easiest uh easiest contributions to attribute to the owners of the work and to clearly map out how they are linked to each of the particular ecosystems which makes it a lot easier to quantify how much funding they should be receiving uh but at the same time it leaves for such a big room for improvement in terms of funding all of the other different needs that an ecosystem has in order to grow and in order to continue to attract new use cases new Builders into them and so what we've seen here I'd say particularly this year in 2023 there was an attempt to fund more categories across public goods but this year in particular we've seen a very very narrow scope of what's being funded and again focusing on infrastructure and ensuring that the technology layer of the different ecosystems is well served but it does leave a lot of room for improvement for several of the other activities and contribut such as education Community contributions for towards user onboarding and the development of localized use cases to emerge so I'd say that's one of the largest opportunities one of the reasons that we see this Gap is the gap of data to measure this contributions what is the impact that they've had for each of the ecosystems and therefore being able to correctly attribute how much funding they should be receiving yeah I think that's it's actually a great follow on about how we can expand the number of categories and I think one of the most promising projects to emerge uh this year is Kito AI K AI t o Ai and basically what they do is they have the industrial great Twitter API and they get all of the tweets for a project so who is the number one yapper for arbitri and they create a leaderboard and that actually allows ecosystems to also fund not just infrastructure but also fund like marketing and actually get to know who who is doing the most marketing so I think like one of the most exciting things is how we can expand from just infrastructure which is easily measurable to even other categories like marketing and others and other sectors that are equally important for an ecosystem to grow I think that puts us straight into the question of impact which I think is at the center of funding impact is such a fussy term could you help me explain a little bit just make sense with what does impact mean and why should we fund it mhm so impact I and I think this is a really good question because one of the be big misconceptions in the space has been uh identifying impact as the outputs so a lot of the projects and a lot of the ecosystems have been focusing on like did you complete your deliverables and identifying these specific outputs that projects had committed to deliver as the impact that they've been generating in the ecosystem which if we're thinking of a theory of change which is a really good framework to think about what is that impact that we want to have that wouldn't fit within the what impact actually is so impact should be the outcomes of the actions that a project has had or like or the work that they've been doing which is very different from the output so output the and I think the reason why some of the projects have hyperfocused on the outputs is because those are really tangible and really easy to identify if the people that have requested a grant or the projects that have requested a grant have actually completed these deliverables but one of the big misconceptions is confusing outputs for impact where impact is actually the long taale of what comes after the actions have happened and this is something that takes time to be observed and in our industry which doesn't really take time to consider what's happened and look back at what the results have been of what has been funded is something that is usually a big problem yep so just building on that quickly um I think like outputs are what you deliver and outcome is the effect of what you of your output so if your output is a journalism article and then the outcome might be uh if that article gets cited in some lawsuit or like it gets cited by a congressman or something like that so that's the output and outcome and impact I Define which is kind of unique definition is outcome relative to cost so I mean educating a child is great it's an outcome but if you do so at a million dollars it's a negative impact because you spend so much to do it for a single child so I think accounting for the intuition that cost has to be baked into impact is another important aspect yeah and I think like one of the pieces that has not even been explored in the in the space uh which I understand it's also because it's challenging and we can't even get to impact evaluation in terms of a positive sense like we are not there yet but one of the items that's still greatly unexplored is negative impact right like how do we take into account when some of the outcomes from the different outputs that uh players are generating in the ecosystems is actually impacting the ecosystem in a negative way and how that should be handled and whether that should be penalized or considered in a different way wonderful I think in many ways impact is what you the action you've done minus if you wouldn't have done this action which is considered a counterfactual so counterfactuals are really important in measuring impact and you both have been involved in a project called counterfactually could you talk a little bit about this as well I mean you're the expert on counterfactual analysis and how it works um and maybe I'll do a story and then you can kind of talk about it in the concept wise so the story which David himself told me was that uh the carbon credits Market historically used to use historical baselines so the like the rate of deforestation was maybe like 5% and then it reduced to 3% over the next 2 years so you mint a carbon credit for that 2% decrease but this is obviously a flawed methodology because the entire world deforestation rates might be decreasing so that's where the idea of counterfactual analysis came in where these economists went and calculated that um you have to look at the neighboring areas where an intervention was not staged and then minus the uh deforestation rate from your area minus the neighboring area where you are not active and then they actually found that the value of like a lot of these carbon credits was worthless and it led to like this massive Scandal and the Vera CE you had to resign and it was like really fun uh thing to Stick it to the Man so I think like we are like we aren't seeing this analysis happen in the blockchain world so much it's like hey I did this app and it's got so many transactions but this idea that if you didn't exist what what would have happened and which other app would have got those transactions I think that's a kind of a black hole that no one is really addressing yeah and I think pulling it into the like into operation in the web 3 space one of the reasons why counterfactuals are really interesting is we have a limited amount of funds that we're directed towards public goods so we want to identify how can we use these resources in the most efficient way to get the most out of this funding and through that one of the things that we need to do is identify which are the teams that are having the most impact towards the specific goals that we're trying to achieve in a particular ecosystem so by using counterfactuals we can look at which projects were not funded that are very similar to other projects that we're actually funded and what does this intervention which is the fact the action of funding the projects uh what type of effect did it had on the growth of the project and then on the impact that they were able to deliver for the ecosystem so we ran a counteraction analysis on onchain data on arbitrum and that gave some really interesting insights Dean you used to be a delegate in arbitrum could you tell us a little bit more about it um yeah so like arbitrium had run a massive program called the short-term incentives program which was meant to boost the growth of protocols on arbitri so if you use use any protocol they'll give you like money for using it um so the protocols have to pass on the money to the users and it was something around 70 or $80 million that was allocated for this purpose but after David and all of us ran the analysis using counterfactual analysis and basically what we did was we found a synthetic control and um the idea of a synthetic control is um what which chain is most similar to arbitri before a particular time period period so in our analysis we found that arbitrium was 0.6 optimism 0.3 ethereum and 0.1 polygon so because polygon and optimism did not run the short-term incentives program we look at the graph of how they grew and we look at the graph of how arbitrium grew after the program came into effect and we found like a negligible impact of this incentive program um on on on the growth of arbit RIT like it grew at almost the same rate as the synthetic control so that's kind of in a brief how that case study looked uh when we actually applied this analysis to web 3 protocols awesome so looking into there's a there's a famous saying in our space that says you cannot value but you cannot measure right and in many ways now you have been developing this really wonderful framework for impact for optimism retro pgf um but many of these metrics might not be measurable should we not care about them no we should um but I think what it actually gives us is a good opportunity to identify what are the data sets that we can use to um to develop proxies for these particular metrics so I think one of the there's a lot of nuance that we can explore when we're thinking of the data that's used and the metrics that we're using to understand what impact means within each one of the ecosystems and one of the things that's still to be explored is what are some proxies that we could use from existing data sets that can correlate well with the impact that has been generated by some of these projects I I definitely don't think that it's something that just because we can't quantify it in the same way in which code contributions can be Quantified which are still very flawed like if we look at the first type of metrics that we were leveraging at the end of last year at the end of 2023 beginning of 20 and even throughout 2024 when we were looking at how many stars does a project have how many Forks does a project have which doesn't really tell you much about how much value that project is being generated the fork might be empty maybe no one is building on top of that fork um so there's still a lot of muans to look into how can we find proxies for some of these um for some of these contributions that are a little bit that are more fuzzy than code and then how can we either generate new data sets or pull from different members in the community that are more in tune with the type of data that these projects are generating so that we can leverage this information yeah just building on that I think one approach is proxy data um uh I think it still might fall prey to goodhart's law so goodhart's law basically says when a measure becomes a Target it seizes to be a good measure so um so I mean like what are the ways out of good heart's law and I'll say one and then you can respond um is the idea of like non-deterministic models so something like the arbitr chelate incentives program has it's a very deterministic model that if you have uh these many comments on the Forum if you have these many votes you're going to get paid this much money and it's actually resulted in a lot in a lot of lowquality comments because people are trying to game it and like it's falling prey to good heart's law so one of the initiatives that we are working on at the ethereum foundation is using a non-deterministic model so that's basically what an AI or an llm is where they predict how much money a project should get and we use a jury to serve as ground truth data and the ground truth data is used to see which llm model is most closely aligning with the human preferences so that's one way out of good heart's law but I'm curious to know what other ways yeah so another way is to design the metrics I think right now what we've been exploring is just metrics that are that already exist that metrics that are already out there and we're not really looking at complex metrics which then enable you to choose what are the type of metrics that even if they are gamed they are actually generating a net positive value for your ecosystem so you can even sort of like play the game with those that are trying to extract from your system and identify what is what are the type of metrics that I can include in this design so that even if someone is trying to game them it's aligned with the outcome that I want to see for my ecosystem and the gains that they are having are still under the value that they've generated for my ecosystem but I think we're still in very early stages in terms of identifying what are existing metrics and we haven't taken a step towards how can we build more complex metrics I want to use the last 10 minutes to first ask you about deep funding and then throw in a spicy question in the end but so you mentioned already a little bit about the AI agents what the E Foundation has done is it true that vitalic himself came up with this mechanism why do you think he cares so much about this and can you talk a little bit more about deep funding yeah sure um I'll start with a story I'm a journalist so we like stories uh after jeevon um I just made a tweet saying that these are the things I'm interested in working on in 2025 and one of them was using AI to quantify impact um and then vitalik just saw the tweet and he messaged me and said hey I've been thinking about something similar and he laid out this very sophisticated mechanism that he's clearly been thinking about for years and that eventually became deep funding and the broad idea is a dependency graph so ethereum is underpinned by around around 5,000 open source repos and each of these repos is used multiple times so when we create the dependency graph of ethereum there are 15,000 edges in that graph um and basically vitalic is interested in trying to give a weight to these lines in the graph because then he can put money into one public key and it gets sent to all of the Reapers based on how much value they've provided so obviously the question then becomes how do we decide the weights that should be assigned to like to these repos and the lines and that's where vitalic felt that AI can be used that we have a competition where anyone even any of you are model Builders or no AI here just like submit your predictions of how important these different edges are there are around 15,000 of them and um kind of as a juror vitalic said that when I like when someone asks me to rate something I lose attention and then I end up giving random numbers and I don't like that so then I have to take a break so he's saying how do we make it that I only have to give a few judgments and then those few judgments go into selecting all of the projects and kind of and and that's what deep funding is that a judge gives a few judgments those few judgments go into selecting the winning AI model and the winning AI model gives weights to all of the kind of repos that are there in that so that's kind of the quick overview of it can you learn more about deep funding with deep fund.org it's super interesting one of the definitely more interesting experiments that is happening right on the space so Dean La you both are caring a lot about impact measurement and you both care about that this space of public goods gets funded from a journalism side from a design and principle perspective but you both also approach this in a very different way right while Devan as far as understood and also observe you you're moving in the space in incredible speed you're trying out out things you're breaking things you're trying out again and you're seeing what comes out well now you're more the conscious designer you're sitting there sometimes thinking much much very long on how to get these metrics right not even thinking if there's data on that What Not Who of you is right who of you is right that's a very good question um I think these are I think both approaches have their benefits to it one of the reasons why I I tend to be more reluctant in terms of moving really fast is for the because of the long-term impact that this actually has on the communities and the different projects that are contributing to ecosystem growth um for example one of my main push backs when optimism shifted to slash several of the categories that they were funding in Richard pgf starting 2024 was because this ecosystem had already created this positive flywheel in terms of signaling that if certain type of work is getting done we'll find a way to reward it which means that you're already creating expectations in different types of contributors for what type of work you want to see and identifying that it and not only identifying but committing to fund that type of work and if suddenly you breach the trust that you have developed with this particular groups it becomes a lot more difficult to get them back into your ecosystem and this is something that is easily observed in how humans very and a are very reluctant to change their patterns not only in terms of Technology but if we want to look at an example of technology is most companies need to ensure that when they are pushing out a product it has has to be this minimum lovable experience like you need to like the experience that you're having with that product cuz if it doesn't work the deflection rate is so high and the cost of trying to get those users back into your product is twice as expensive or even more than acquiring a new user so if we're creating this terrible funding experiences for contributors because they don't fit into one of the particular categories it to me and this is why I care about it so much and I try to move a little bit slower and thinking more about what is the long-term effect that this is going to have in the ecosystem is it means that we're pushing people away that we don't know if we're going to be able to get them back right um so this reminds me of my favorite quote from the philosopher Emanuel Kant who said that Concepts without percepts are empty while percepts without concepts are blind and kind of what that mean like a concept is a unit of an idea and a percept is a unit of experience so if you have ideas but no uh physical instantiation of it that's an empty set it's just an idea without an actual physical manifestation and if you just have experiments but without Concepts then it's blind so ideally you want both the concepts and the conceptual thinking and you want like moving fast and breaking things as well um and ideally both of them go well hand in hand together um so that's kind of I think overall what I feel but I feel like the idea stage has to convert to pilots and something I'm excited about in optimism is their futari experiment where uh they have given like you can sign up for it today like you can go and get play money from optimism and you can bet which protocol is going to generate the most tvl because tvl is one of those metrics which you just want like it's okay if you're gaming it we still want you to lock your assets on our chain um and like you have to bet which protocol is going to do the most tvl in the next 6 months and if you're right then you basically get real money from your play money so I think that's kind of one of the cool experiments of converting the concepts of having UNG gameable metrics into actual physical Pilots that you have which I think again it comes again with this Nuance of the design because for example for this particular pilot something that we're going to test is if people actually care cuz this is play money right so it doesn't really really follow even the theory of future key in which you're incentivized to do this extra work cuz you're getting a monetary reward that justifies putting time into this thing but if we're doing it with play money so we actually don't know what's the actual benefit that I'm getting from investing x amount of time into producing and like really trying to assess which of these protocols has the best odds like how have they performed previously like this is a lot of work that if I don't know what is the type of reward that I'm going to getting why am I going to be wasting my time to do this which I think is part of the thinking through the design of these experiments really well so that we understand if it's actually going to lead to the outcomes that we want to see we're still going to gather data but I think another really important piece to keep in mind when we're experimenting is if this data is clean and useful CU if we're just producing data for the sake of producing data and this is trashy data like there's this concept as well in data science that trash comes in into your model trash comes out if your data is bad the outcomes are going to be bad I can definitely feel some heat which is pretty good in Denver you know it's sometimes too cold you know you want to be at a place where it's hot so in the last 3 minutes that we have just quickly because you both have been so influential in that space how would someone who just get started in public goods funding impact measurement INB 3 how would you recommend them to go through I think I've actually written a lot of Sops here I you in 2023 I was selected as the ethereum next billion fellow so I actually had to figure out how we talk to people who want to enter the space and what actions to give them so if I have to give you points number one you should get an ens an ens is an identifiable name to your wallet and to your public key number two you should lock some glm into octant octant is is a staking protocol where they have 100,000 e and they distribute around 300 e every 3 months so it's a lot of money that they're giving and you get a right to say which project should actually get that money so you should do that and you should tweet about it you know hey I've voted in octant and these are the projects I supported the third is a gitcoin and giveth rounds so make sure that you actually vote in the gitcoin and give it rounds because your supporting projects in those rounds results in money to them uh from the matching pool um so that happens every 3 months around um so that's gitcoin give it and the and the last is taking part in Dow governance basically take like lock up the token of a DA that you're interested in and start voting on their proposals and in all of this you have to be tweeting like that's like the number one rule that's how I got to be working with the EF and on deep funding and all of that so just like whatever you're doing just tweet it out so that other people can learn and build on it yeah so I think there's it really depends on why you are interested in getting into public goods if one of if you're cuz the pathways are very different if you're a project and the reason why you're getting into public goods is because you want to get funded if you want to work on designing public goods or if you want to Leverage The mechanisms offered by public goods to distribute funding so if you want to be like if you're a project looking for funding through public goods I think participating in the Octon rounds is really good participating in gitcoin is also pretty good um there's giveth as well recently round one of the largest I think it's qf rounds precise specifically around public goods so there's multiple uh there's multiple projects building for the distribution of funding for public goods so definitely check out gitcoin octant gith then if you're interested in coming in as a designer of public goods and stewarding where precisely these thought uh the thoughts of and the type of experiments that are being carrying out in public goods are then definitely join a DA identify one that it's a dow that funds public goods I would suggest like optimism octant is also really good like they're bringing in a lot of new people that contribute to how the rounds how the the epoch are being designed um then get involved as well with gitcoin and if you you are what was it an nouns an nouns nouns is also really really good for that um but yeah I'd say those are them wonderful I think we all have like a really long list of homework to do now in case you guys want to join this I think this was a really wonderful comprehensive list and so thank you so much for your time Lao Devan and a big Applause for our panelists and you are a really good moderator I wish that we heard more from you as well yeah that is was Beau