30. Measuring the Impact of Open Source Efforts with Open Source Observer - Taipei 2023

[Music] all right hello good afternoon everybody welcome welcome thanks for uh uh staying this afternoon my name is Raymond Chang and I'm the co-founder at Kuba Labs where we are building a tool called open- Source Observer open source Observer aims to measure the impact of Open Source software where we believe that's going to be the key to unlocking value in Impact markets for open source software so as part of the founding team for uh hypers search you might have seen this diagram before in previous talks right now today we live in a world where open source software pretty much just focuses on the right- hand side of things we have incredible developers building open- Source software that basically the entire economy in the world is built on top of generating incredible impact today but with historic ly infamously chronically underfunded public good what we want to do with open source Observer is help flesh out this leftand side of the picture illuminate what is the type of impact Downstream impact that we're having from open source software to the real applications that exist today to the users that are using them so that we can help imbue value to hypers Sears so that prospective retro retroactive funders can come in by hypers certs and support this work I think this is the vision that we're trying to build towards a world where we can retroactively fund public goods that's delivering value to the world at the end of the day we firmly believe that impact should equal profit today with respect to Grants when you receive a grant the amount of reward that you get is pretty much fixed no matter how much impact you generate and we believe that it should be the case that people are incentivized to generate more impact and the more impact that they generate should lead to more financial Rewards so with open source Observer we're trying to build a tool here that helps internet economies measure the impact of Open Source contributions to the growth and Adoption of their platform as of today we're predominantly focusing in the short term on crypto networks predominantly because it's much easier to measure we can take a look at open source software contributions on GitHub we can see where these open source libraries exist in the software supply chain basically the dependency tree and in the crypto world we can actually also see all user analytics because transactions are entirely on chain so we can measure high value users we can measure retention we can measure funnels and life cycle of users and we can measure for any particular Library like ethers the types of Downstream impact all the applications that would be affected if such a library were to go away and the number of users that would be affected if such a library were to go away we want to bring all these things together into an easyto use platform we think of Open Source Observer as basically Moneyball for open source uh this is you know we equate or we liken open source funding today kind of like how baseball was in the 1900s using outdated methods without a real clear understanding of what led to success that's kind of the world that we live in today and we want to transform that this is a fantastic movie fantastic book if you guys haven't uh read or watched it before uh it's on most air pran flights if you guys are flying somewhere you can probably you can probably see this um uh on your flight home but in the movie based on a on a true story Billy Bean here played by uh Brad Pit uh is trying to disrupt this old school stagy world where you know recruiters mostly hired players based on their gut feeling when they when they went to places and they didn't really clearly understand what aspects of a team led to the most success and what Billy Bean did which was incredible is he took he brought in data science he brought in statistical analysis he brought in correlative studies studies and they really dissected what are the aspects and what are the core metrics of a team that lead to success what are the best high Ry Investments that I can make to be able to have a winning team and in 2003 what they showed with the Oakland A's was that with a budget that was about a third of the size of the New York Yankees the okan a were able to compete competitively at the same level that's incredible the idea that we can achieve the same amount of impact with a third of the budget this is exactly what open source needs today it's important because uh crypto networks today we have a lot of capital that we plan on deploying we estimate that there's about $30 billion dollar today sitting in ecosystem treasuries and it's predominantly going to go towards things like open source software and the better that we can allocate this Capital the more impact the higher Roi that we can achieve per unit dollar that's going to be critical to any the success of any crypto Network for the most part every single crypto network is funding towards this graph right here on one axis high quality highly engaging useful developers building engaging applications that bring on sticky users on the y axis here everybody wants this to go up and to the right and we don't actually have a clear understanding of when we make grants when we make funding allocations what are the things that actually lead to this what are the fun programs that are leading to real continued sticky users as opposed to just short-term blips in traffic historically as an industry we've been doing pretty terribly um we are investing more and more money every year now on the order of tens of billions of dollars and by some estimates they're still only on the order of tens of thousands of developers in this entire space which is kind of mind-boggling right these numbers are estimates so you may not actually agree with the exact number figure but I think there's one aspect of this graph which is I think uncontestable is that we have to be doing significantly better like an order of magnitude better in allocating capital and keeping sticky users around and sticky developers around than we are today what we want to do with open source Observer is create a data warehouse where we can bring on the visibility we can make a public playground so that anybody can do these types of impact analyses similar to how tools like Bloomberg data science tools business intelligence tools have completely transformed traditional trafi Capital markets for the impact markets we are not at that level of visibility yet and we want to build this as a public good for the community so that we can do this together which is why we're exceptionally excited to be doing this with one of our first Flagship Partners it's optimism retr pgf this is one of the partner ecosystem partners that are deploying a significant amount of capital today um this quarter they plan on Distributing about 30 million op that's over $60 million to predominantly open source projects over 300 open source projects today and we're really proud that this small community of about a 100 expert voters um we're helping them build out better lists better impact analyses to better inform these votes because at the end of the day their thesis is they need to be rewarding the projects with the highest impact in order to generate you know the most growth into their Network we just finished voting you know literally last week uh the we're excited to see what results come out in January but uh we want to continue doing programs like this to help different ecosystems analyze the impact of their funding programs now um I just want to share a little bit about what are the types of things that we're looking at um we mostly just started on this journey um we want to look at things like how developers are bringing coming onto the platform um which of these projects are leading to more developer growth than others we also want to take a look at how collaborative these developers are whether they're forking each other's projects whether they're submitting fixes to each other's projects that is one sign of healthy ecosystem growth um we also want to look at how they're engaging their open source Community are they doing an effective job of growing that open source uh ecosystem or are they staying you know isolated in their in their own little pod we're also looking at more traditional Notions of metrics um um we're also looking at things like simple shallow metrics like you know stars and downloads but I think one of the things that we're really excited to think about is Downstream impact in a world like optimism r pgf one of the things that's most interesting is that these things directly lead to revenue for the platform right if you think about a library um libraries you know help build out these daps daps are bringing users onto the platform these users actually lead and translate to very visible Revenue metrics in the form of sequencer fees for their Network and we want to understand the margin of value of any particular project and how these are affecting the the global revenue of the network um part of that and one of the areas where we're diving really deep is now we can start to do interesting things like user segmentation right we can take a look at what types of users are the most high value which ones are transacting large you know dollar amounts which ones are transacting frequently versus infrequently and basically do the types of user Analytics an ecosystem level that historically we've mostly been doing on a project level it's very common for individual daps to use user analytics we want to look at this from an ecosystem wide um level this is basically the thesis of the optimism retro bgf uh uh experiment um this is lifted directly from their website they believe that if they take a portion of their sequenc or fees in perpetuity and and put them into a program like retr pgf and allocate that Capital well to bring the most high value developers to bring the most high value users onto their platform that's going to bring the most engaging applications and the highest amount of demand for their block space which effectively translates into more Revenue that they can again deploy into retr pgf in this virtuous cycle and the fundamental question that we want to ask is is this working like are they actually doing something that's yielding you know a net positive change in the r system and we don't have a conclusive answer to that yet but we have some early signals and some indications that we're not just you know bringing on traffic that's temporary there's actual sustainable growth here that's that's kind of interesting so this is a graph of developers that are in the optimism R pgf ecosystem over over the years and it's not just the case that we have new projects that are attracted to optimism because of ret pgf existing um that's you know this darker red graph here it is also the case that if you are a project that's generating high impact Russia pgf now gives you the ability to have additional Revenue additional Capital to hire on more developers so we're also seeing that people that have been participating and round after round now have the resources to uh continue making more impact if they're historically the projects that have been uh doing well in in Prior retrop pgf rounds we're just at the very early days of this form of impact data science as you can see a lot of the analyses that we're doing are still pretty primitive but this is one of the areas that I think is most exciting because we're we're just at the early days of as a community understanding how to measure impact like what is impact in any particular Community what is the best way to measure that what is the way to do this in a way that's less gameable and how do we figure out what good looks like and that's something that ultimately I think we have to do as a community together in a crowdsourced way so we think of Open Source Observer as the tool to do that we want to bring onchain data offchain data in a single uh Data Warehouse in a way that's we're committed to being open source open data and open infrastructure as a tool we want to build this in a way such that it makes the data as widely available to the public as possible so that we can all collaborate on doing these types of community data science over impact for public goods it's kind of crazy what kind of a year we've had um you know hok has just spoke yesterday about the work that's been done on hyper boards and hypers Marketplace it's kind of mindboggling to think that we just launched the hypers protocol like ear earlier this year um but it's been a really interesting year and we're excited to be growing out this ecosystem uh we've been building out the hypers SDK so that other people uh can incorporate hypers into their ecosystem and in particular at Kuba Labs we're very excited to bring on and expand the world of impact measurement and impact evaluation we think that's going to be critical to unlocking value in the hypar ecosystem bringing back the original vision of what we wanted to build you can tell here we're just in the early days of what we need to be doing you can think of Open Source Observer as just you know a small part of this picture measuring just for one impact domain which is open source software and retro pgf which is just one fun a single fun with a single set of value systems for a particular crypto ecosystem but there's so much work that we still need to do together in the world of impact evaluations um we've only been indexing projects in the optimism ecosystem and we're now starting to build out what we call the kba data Collective we want to build out this open-source uh data warehouse so that anybody can be rewarded for contributing valuable data when measuring the impact of of public goods we want to be able to reward the types of data scientists that are generating the most insight insights or the most useful dashboards towards measuring impact and we want to do this in a way such that we're building out effectively a data D where with the community we can do this together so impact analysis and experimentation I think of this as like exceptionally early days um we're excited to work with folks in the community uh to build out this data warehouse together and to build the most useful analyses so if you're interested please conf find me after this talk um one of the things that we are very aware of and need start thinking about now is what happens when we do this regularly retr pgf we only on our third round but they have a commitment to do this every year multiple times every year and what I think we're only in the early days of understanding is what are the incentives what happens to this from a game theory mechanisms design perspective when you actually do this regularly because if all you're doing is funding downloads you're going to get perverse bit incentives and you're going to get bad behavior and so how do we design metrics that are less gameable how do we design systems that analyze impact in a way that incentivizes the good stuff and not the gaming and spam that's a really difficult question I don't have a an answer for that yet but I think we need to figure this out together and the answer is not don't measure right that seems to be kind of silly from a funding perspective retrop pgf is obviously just one ecosystem every single ecosystem has a different thesis and a different set of value systems and a different set of goals and we're excited to work with different ecosystems and funders to understand what are the unique properties of this community and how do we build it in a way um how do we do this in a way that's general purpose that can satisfy the unique needs of any particular uh Network we're particularly excited about potential shared goals between ecosystems so that if you know two ecosystems have a goal that they want to fund together um we can potentially create collaborative funding pools around them and lastly of course this is just in the space of Open Source software and as people have mentioned in this conference multiple times that is just a small space in the world of public goods Goods we want to do this in a way that explores other spaces as well but we think because it's so measurable here we can make some early impact and hopefully extract these learnings and apply them to other spaces what I'm really excited about here is we think retrop pgf hypers certs retroactive funding these types of Novel mechanisms have the opportunity to massively transform how we think about public goods funding and we're just at the early days of partnering with a handful of ecosystems but this is a much bigger problem than just in crypto and what we're particularly excited about is this growing recognition from governments this growing recognition from nonprofits and other corporations that digital public infrastructure is infrastructure that we all need to invest in and we're hoping that what we learn from here can be something that's appli well beyond what we do in crypto so if you're interested please join us at our telegram Community you can find our QR code here help us and join us in our mission to fund public goods thanks amazing thank you Raymond for [Music] that