Logos | What AI Submissions Taught Us About Funding Open Infrastructure - Martin Strobl

[music] Okay, thanks for sticking around. Uh, good. So, a bit later, so 15 minutes, it's fine. Uh, okay. So, I work at Logos.

We are trying to do an incentive campaign for our builders. uh we are trying to uh have open source contributors rewarded and uh basically contribute to the ecosystem, stick around uh be active in the community. But we launched this program in Q1 this year and we have actually learned a lot from it. So this talk is about that. Uh let me start by motivating it by with this with this tweet by by the tiny corporation which is George H's company uh where they are saying basically that open source contributions that they receive on their repos uh are great and previously they would you know consider them and look into them because they know that people have contributed time in in creating those.

uh but today everything can be oneshotted and it's a very uh demanding work that they need to put in just review that amount of things so they just close those PRs so that's the the reality of the world we are living in um so back to the story that I was outlining so uh we were trying to introduce a program for the logos ecosystem which is a blockchain a storage solution and a decentralized me messaging solution uh all packed together in in an app called base camp and we we are looking for contributions we've identified things that we would like to have in the system uh and built for the ecosystem uh and it's all privacy focused right uh so so we thought we would come up with this concept of lambda prices uh lambda prices are modeled uh after the x price uh So it means that we just say basically what we would like to have in terms of functionality and some other requirements and then we are waiting for submissions and we put a price tag on that right so it's essentially like a bounty uh and we are waiting for submissions and and builders to show up and actually like say hey like I created this uh give me the money right so then we need to review and assess uh what they did uh and it's first comes first served because they they basically get served those reviews as they land. So we we've done that with like 16 or and more prizes you know those are like uh primitives uh sometimes know whistleblower app is an app in itself and we have had 50 and more now uh submissions over over the course that has this has been run. Uh but we we basically thought this is a very clean incentive you know you you put the code in you get the money right but the fact is that well we thought it's like you know we put out the price we we they they put in the effort what we get out of it is the code and the builder learn some skills which we hoped like would be very nice for the ecosystem if they stick around and you know become a steward in a sense that they they they would uh contribute to the ecosystem further or they would just guide other uh other builders and be a motivator uh like creating that building uh building uh community. But the reality is that you know price uh of course that's still there but builders don't put their effort they put their tokens in uh and what we get out of out of it is still the code but it's the agent skills that are improving over time. It's not the builder's investment in in his own skill or commitment.

It is uh it is the agents. Uh so it's a very superficial delivery if you if you would like uh and we we saw that uh because we started to receive those AI generated submissions uh they were not too bad right uh they passed some of the main criteria but uh when they were tested and scrutinized we realized that sometimes they are sloppy under the hood uh they assume things that are consequential and and have bad impact on whatever uh downstream usage uh and sometimes they just straightforward light about what they do and what they don't do. Uh so so the criteria stopped working uh the the old world was where you know writing code was the hard scarcy part uh and passing test was was me meant that there is oversight and understanding of the builder. Uh but we live in a cloud world where code is cheap suddenly uh and judgment is not. Um and we actually had an issue where uh we provided like a specific feedback.

It was just a next prompt for them to provide to feed to their agent that built like yet another version of what we had to review, right? Which also had some other deficiencies still uh because it was not like perfect AI work. Uh we even had one guy who uh he basically like introduced a a PR with with a completely new price. Uh in the next PR he put in the solution to that price and he was just asking to put a price tag on that. So that was kind of bold.

So so we closed that one. Yeah. Uh and and yeah. So, so and these builders they don't really have the insights into actually how the thing works under the hood because they are just interested whether the checklist is satisfied right whether the functionality uh that is uh required for the price lands uh in in their submission. So we decided that we need to filter and screen.

Uh we we cannot just tighten the the checklist because the checklist is the exploit like if you have a checklist uh that's a guidance for the agents to to actually uh follow and um ignore anything else. So what we started to do is we started to look at things that AI is still bad at which was the logos stack integration of as I was talking about base camp you. So AI is not really good at doing base camp modules. We asked them for a video walk through which means that they we wanted to see them explain their own code which was some uh level of effort that uh they needed to put in and that was a kind of a very good indicator of the quality and and in that they would you know justify some of their design which means that they would also uh try to explain some some trade-offs that they had to navigate for example and then we also started uh an agent ourselves which was just uh checking those submissions and on on a very like superficial level checking whether they are coherent and make sense so that we don't spend too much money on each of them too much time on each of them reviewing each other uh and we started to have rate limits because if you enforce rate limits uh it leads to a better quality of whatever is submitted uh so I don't I want to stress that we are you know not against AI when we are doing this. We actually use AI, you know, all the time, right?

It's just that we want to um have a builder community that is healthy and valuable. Uh so when we ask for local stack integration, we want them to have like good knowledge of what we are actually doing, right? uh and and video walk through is just putting a phase uh behind behind the code to just creating some identity folder builder uh and design judgment just really signals uh they have clearly understood the the trade-offs and the things that they need to navigate. Uh so this worked for a bit uh but actually so it works in a in a way where we were able to get good code delivered but what we didn't get was that still the builders were not invested in the success of their work. Uh they just delivered whatever was asked uh and and that was it basically they moved to the next thing.

So we we changed it again. Uh and before I dive into that, I just want to basically uh cover why we need that builder community. Uh and that's because we need success stories on the builder end. Uh we need personal experience that attracts the next builder. So if you if you tell someone, okay, I built something for Logos, it was a good success, and you know, you should do it too because it's fun and you can get some money for that.

So that that's just the way things could spread. uh naturally I think and in a healthy way uh and we need stewards basically people who onboard others answer hard questions in the forums uh and also create standards what is good uh we need the reusing code for u identifying like building blocks that that are essential for the system and we need you know as an organization we need feedback to actually get a better developer experience. So back to what we changed and what we are changing this week actually. Uh so if we cannot measure effort really well uh and we need to rely on those things that are kind of associated with it like signals like you know I was saying like the video or or other things why don't we just measure the adoption like who is using this like you know why should we pay for this if no one is using it right that's probably the question we should be asking so so we thought well that probably makes sense uh we you know effort doesn't really lead to commitment on their end. So that link is broken.

Uh and if even if we are trying to catch up, we are only only going to be catching up all the time to you know because AI is progressing so far uh that so at a fast rate that we just cannot probably uh keep catching up. So we thought well if this was supposed to be like hard to build we don't care like we are interested whether it is actually being used. So, so now we also have we still have the functionality criteria on what we want to have delivered but we also have the adoption criteria which is uh which is uh basically number of integrations and I'll dive into that later. So you know usage and traction and testimonials. Uh so and maybe this this approach is future future proof.

We don't know but we are feeling that perhaps uh we are not catching up to AI because we don't care who delivers the solution that uh that fills those adoption criteria. Right? So we we are not paying for authorship and we don't care whether authorship is is an AI or if if it's a a real builder. So uh we are just aiming for that quality and impact. So and the adoption is a signal that comes from the market comes from partners and other builders uh and and that that is really what matters for us.

So uh specifically we are looking at thirdparty integrations uh observable usage uh sustained activity and and human validation through testimonials and and like acknowledgement in the community. uh and we are still you know it's an experiment still so we are retaining control to fight account farming and purchase a agreements and and engagement so just want to give an example uh which is a program registry so program registry is uh programs are smart contracts and logos uh and and they are represented by a program ID and that's it so so we are trying to get a program registry through this lambda price program it is not live uh but it's coming uh and we are trying to you know attach a name author version ideal and other other information such that any programmable in the registry can be called by anyone. So uh and and and we are looking strictly at adoption criteria in a way where we require here like 50 registered programs uh with 20 distinct authors uh and then some integrations in apps and along along this we we will have other prizes for programs that that are actually need to be delivered in in the lumber prize program. So and for them we will require them to register in a program registry as a criteria for for the price. So if you are building a program so there's going to be like 10 submissions for the for the program registry and if you are building other price you need to pick a program registry that you actually register your submission in.

So that already feeds the adoption criteria for the program registry if that makes sense. So it it is uh dependent on that one. And today we are launching the first prize under this new scheme which is an open street map integration which is uh worth 1,500 bucks. Uh it is a decentralized map data distribution on the logo stack. It it works through logo storage and then also on chain for for the register and and you know you can uh you can start building this today but probably the adoption criteria will uh land over time right so we can have more submissions but the one which actually has the most hosts and the the most data map data supported will ultimately be claiming the prize.

So, so there there's a there's a uh there's a gap between the submission of the price and and between finishing the code and actually hitting the hitting the adoption criteria so that we know that uh we are funding the the category winner basically. So let me conclude here. So writing code has become cheap. uh authorship and effort are not the signal anymore for commitment and quality of code. So we are trying to fund differently.

We are funding for adoption and impact uh real integrations usage uh and we are looking for understanding and genuine engagement across the board and generating that impact whether it's uh by usage or actual social usage. So, thank you very much. Uh, this lambda price, the new lambda price is out. You can check it out. Uh, you can go to build.

logos and co. Uh, where you can also find our RFPs if you are a dev studio. Thank you very much.