Retro Funding: Experts vs Data, reflections from two large scale experiments - Bangkok 2024

[Music] hi I'm Jonas I uh lead the retr funding effort at the optimism Foundation more higher higher better okay I see some some familiar faces that's nice um I'm here to to tell you about two experiments that we ran within the retr funding world uh one was focused on datadriven decision- making and the other one was looking into uh the role in uh expertise on decision-making outcomes uh the kind of goal here is if you're funding if you're building your own like grants funding mechanism uh these insights learnings that we're taking away from these experiments might help you in doing your own thing cool who here has heard of optimism retr funding before nice who here knows what it is great okay because I thought I don't need to do give you guys an intro on um what this program is uh here's my two-minute summary we're trying to reward the contributors to optimism for the impact that they provide so for the value that they create for optimism where we want to go with this is that everything from core protocol contributions down to De tooling um and other projects are rewarded for their impact on optimism um um we do this retroactively we're kind of creating the future by looking into the past by continuously rewarding what has provided value we create this forward-looking incentive much much more in-depth thing to go into here why retr funding makes sense um some rough numbers we've had over 700 projects um in five retro Rounds Around 58 million op there's actually one going on right now which is rewarding governance contributions here are some of the top project that we've seen across rounds just a random sample of the top 10 to get a sense of this um all projects that you hopefully know and love okay skipping onto the the uh interesting topic so two large scale experiments um one was uh what we framed as as metrics driven evaluation so the idea here was um instead of having voters review each project can we uh have voters decide on measurement EG metrics uh that are measuring some form of impact and the way that the voters govern is just they decide what metrics are a valuable signal and which ones are not the second one was on uh expert based evaluation so we had um a round which was quite Technical and we tried to see what is the effect of expertise uh on the voting outcomes uh I'm presenting here but this is like a lot of teams that are working on this um made with Love by teams like open source Observer Agora gitcoin charme open rank optimism foundation and more and this is governed with Care by optimism citizens this is a group of around 130 people that deeply care about making optimism work um metrics based evaluation this was the first experiment that we ran so the scope of this retround was rewarding onchain builders for driving the adoption and usage of optimism so like what you may think of as a more classical growth campaign um this round had around like 250 Builders 10 million op to be allocated and uh in this round each voter selected some set of metrics and applied weights to them so this is like a snippet of what this voting experience looked like uh you went in as a voter you saw around 16 metrics that represented different measurements of onchain activity of the onchain builders uh and you were able to select these metrics add them to your ballot and then add weights um what you see on the right side is we showed each voter um the project allocations that are the outcome of their of their metrics and weights so here as an example this person chose like uh gas fees uh as an important measurement to understand how important how much impact a project has um generated and on the right they see the outcome of all their uh all their metric choices these were the top five metrics oh yeah yeah I tried to I tried to skip that over we also had an open source reward multiplier so we gave voters the choice to say I want to give additional rewards to onchain projects um whose code is licensed under um an open source license as defined by the open source initiative this is a whole rabbit hole because it actually turns out it's pretty hard to like reliably um reli classify which re like which repos are actually open source uh but thank you for the like Interruption question love that more of that please um these were the top five metrics nothing out of the ordinary gas fees trusted users on boarded um trusted recurring users there's a whole side here in how we um how we structured these metrics uh with some trusted user models more interruptive questions go for it you know yes we're aggregating a very informed question exactly another Rabbit Hole here is among the 16 uh metrics we gave some of them on a logarithmic scale essentially allowing you to express the preference of saying I believe a project that has like 10,000 transactions only has like double the importance of a project with a thousand uh transactions cool some some observations from this round um most people used like five to six metrics in their ballot and the the alocation which is like a bit too far down for you guys to see um the the the results of the round had like a very strong power law distribution so the top 20% of projects got around 70% of the rewards and you see really that the the concentration is quite high and then there's this like very wide spread with the median project only getting 13 kop and this is not really surprising because this is kind of what you expect an economy to look like to a certain extent like you expect to find Paro distributions um more observations from this much more efficiency for voters which which they liked for a change in the previous round we had voters spend 15 hours on average reviewing projects um now this was two hours uh the scale per project is nice bill because we could add infinitely more onchain projects and the time spent by voters wouldn't change um there was like medium to high satisfaction with the results we run surveys on this stuff so people rated this as like five out of seven um 75% of like survey respondents said they felt this round measured the impact more accurately than the previous round um and all of the survey respondents I think we had like 35 people that participated in the survey said we should do do more experiments with metric based evaluation like we should lean more into that model for different stuff the biggest feedback from this is that the human touch is missing um so humans are really good at like qualitative assessment and we can't really get this qualitative assessment by purely looking at the data so what was really valuable in this round is having humans sense check that these metrics actually have the desired outputs and just giving very like qualitative feedback like this project seems very off I've never heard of this before but they perform incredibly well in this metric uh and this was the biggest feedback from voters that it felt like uh purely leaning in on Data Without Really sense checking it uh seems dangerous and that there's actually value to the human input in this process so quick quick takeaways from this um data is good at objective measurements if you know what you want to measure and you have the data why not do it thank you um data is bad at like data doesn't give you qualitative Nuance so humans are are a better fit for that but overall we were able to like validate that this thing makes the process more efficient um and has like a higher accuracy than pure asking people for their purely subjective input on what projects they believe to be valuable um next thing we shipped was expert-based evaluation so this round was about the op stack which is um essentially the technical stack making up optimism uh and it included things it had a category for ethereum core contributions so ethereum core development work this was around 8 million op 90 projects and what we try to understand there is um do experts do a better job at judging the impact of different projects than non-experts how this round worked was a kind of going back to a classical classical model so um each voter went in they had a number of projects to you around 30 uh and we asked them to like rate the impact of the project and in the end wait the impact so um this is like a example of a ballot where somebody orders the project and then assigns weights saying like this project should receive 25% of the rewards so on and so forth how we assigned expertise was pretty interesting so we did this based on the GitHub uh the GitHub contributions of Voters um we have had around 68 experts in this round uh we used a web of trust type algorithm from open rank shout out to them which we seeded with the optimism um the op stack repos to kind of get a sense check of like what is the pro proximity of different voters to these repos uh then we had non-experts which were just existing citizens with no GitHub activity relating to the op stack here's here's a brief overview of what we found um black points are experts white points are non-experts and you will see the diff um you see that like in some areas like the the votes of experts versus non-experts heavily varied so these are the results of saying what if we only took expert votes and calculated results and what if we only took non-expert votes uh a really interesting one is like as fear rollup as a service which is like a Ras provider that experts voted um voted for it to receive zero and non-experts voted for it to receive 130k op um the distribution of this whole round was like overall very flat which is interesting so uh compare this to like the round four uh the round four uh um distribution and it's like different worlds right this doesn't have a power law at all the top 10% the top 20% of projects get around like 25% of the rewards La go for it yes question so there's um question was can you can we talk about the second category we had three different categories of projects uh and the second one was very low I don't know if I'm going to get into that L but we we can chat about it afterwards um overall what you see super even distribution and um if you compare this on a project by project basis uh on a on a voter basis you see that uh this is just a result of people voting very evenly so uh each of these nice little visuals is um represents the distribution curve of each ballot that was submitted red ballots are experts gray ballots are non-experts so what you see is experts have much higher variance in how they vote specifically like the highest highest variance BS uh and experts have much lower variance we don't have conclusive answers why that is but one hypothesis is that experts just feel much more strongly when they vote and exp and non-experts probably have like less conviction in their voting the result of like low variance um this is a graphic from from trend from protocol Guild and his recent article is uh that you don't represent impact well right if if something is is such an even spread um that is pretty much a red flag that you're not actually um representing value well so here the example was like protocol Guild which represents core developers got pretty much like one kop per core developer an independent ethereum contributor got like 70 37k op right so it doesn't it doesn't quite add up uh and yeah this is again experts versus non-experts you see non-experts were even which was uh an even more even spread across the board so some some takeaways um an interesting observation here was people had very high conviction that this design of the round went well we asked them going into the round do you feel good about this round and the results you you expect to see they said yes we believe it's the best round ever um much more than the previous one we ran on metrics but people were not very satisfied with the results in the end which shows some interesting like interesting input bias um the expert allocation was preferred across the board so we showed people the expert allocation the non-expert allocation without labels ask them which one do they prefer 80% said they prefer the expert allocation cool so so some takeaways from this um experts seem to be better than non-experts at like ranking projects based on their impact that's one takeaway we have from this um it seems like cognitively outright comparison is a very hard task right accurately reflecting of saying this project seems like it has provided five times as much value as the next project seems like a very cognitively difficult task and um you we see a lot of like simplification of this task go for it would uh some of these results make more sense if rather than saying expert not expert you said coder M or like coder out cuz like would would some of these results sort of make more sense if you said coder non- coder versus expert nonexpert cuz like maybe they're discriminating based on like solving her technical problems versus a basket of different sure I think I think expert is like we just looking at like category specific knowledge might capture right like we're not saying these need to be like super op stack experts but it just shows relevant knowledge plays a big role but it's it's it's in terms of sort of what are like why are they better at discriminating like they're discriminating based on like solving her technical problems pres of course I think the rollup the RAS provider example is great right because it's it's like probably somebody looked into it found out that the RAS service wasn't actually live yet and like guest voters probably didn't feel as confident in making such a judgment um but yeah so so there is some value in how people rank stuff uh in the like values they assigned to it uh these are much more problematic all right um where are we going next with this uh we're going to lean into a model which is trying to combine The Best of Both Worlds can we find a world where we make datadriven decisions and we have humans in the loop to give us good subjective qualitative judgment uh that's kind of the next chapter we want to go into I'm I'm I'm not talking about it here though it's kind of the Cliffhanger that's that's pretty much it any two minutes any questions go for it hul yeah so the L is can you actually have Ral funding that then creates kind of the follow looking um Behavior people actually start projects with expectations about reliable future do you have like more data on that or you also think that actually get that bigger experiment you got to run so many small experiments that make difficult to um totally different really important topic how do we even measure like the success of the existence of this program how much economic activity does it create right um I think the blocker actually is iteration Cycles if you run rounds once per year on a topic you right like there's so much noise in there with data we might be able to like run continuous rounds that update like every other day right and then we could have much more like direct imp applications of the incentives that we said but totally totally important topic just not covering it here I I heard there was a workshop about it though yeah um so how do you measure the marginal contribution of a project it seems like it seems like something like solidity is very valuable but let's imagine that there are no further uh contributions on GitHub to solidity at some point um clearly still the community would say this is a valuable contribution to the ecosystem and but but no more work is being done so how do you how do you measure like especially as there's multiple rounds um not not the like once you've already gotten an allocation how do you prevent them from getting a second allocation for work already done um I I think in this model uh it actually the solidity team can like if if it works fine right they could stop working today and we would fund them forever because we care about is just like the ongoing value that they provide rather than like are they updating their code um they're probably like good examples of libraries that exist since forever and everybody's still using them they're just like minimally maintained um and I think we should still value the library existing on an ongoing basis I I don't I don't know if that answer the question I don't know I don't know do I have more time [Music]