Building Bridges Over the Pitfalls of AI-Driven Web3 Development | Rob Green - Patchwork

hi welcome back today we have Rob green with Patchwork please take it away all right thank you very much so oh and the remote that I have is not working so all right we're gonna go like this hi everyone I'm Rob um this uh this image that you see here uh was generated by Claude uh 3.5 Sonet I believe uh 37 was not out by when I made these slides two weeks ago but I think uh a lot of the things are going to apply almost exactly the same um I asked uh I asked Claude to generate a cartoon-like image of a bald man with a black shirt smiling and waving like that and I think really it did a pretty good job it made an SVG but can you see anything that's wrong with the picture yeah one hand right this does a pretty good job sort of summarizing what you get out of naive llm usage it's close but there's something a little bit wrong so llms are not always great at designing perfectly cohesive Solutions meaning that it'll accomplish a few tasks but when you look at the whole end to endend it might some of the parts from the beginning might not actually match up with the end uh it can't always think through the consequences of a design um meaning that it'll give you working code but sometimes there's some really hidden troubles in it and um one particular problem is that sometimes even in a single file it will generate code that uses different versions of the same Library so that code couldn't possibly compile and run the way it is and it also doesn't always fully understand your intentions now llms are actually very good at talking to you in your natural language because they're trained in your natural language um they're pretty good at just making up stuff like making up some data making up a function you might need um they're great at mimicking examples so if I say hey here's an example of a thing can you now apply some uh a different style to it and do the same thing it's pretty good at that and they're excellent at copying things they've been trained on and that's really key I think most people know all this by now anyone who's been using them so when trying to use an llm to write a smart contract specifically I rather than list the exact problems I'd rather show you an experience I had going into it as a naive developer with my prompt so I wrote this exact prompt I said write me a smart contract that is a game item system make weapons with modifiers where the modifiers can be purchased and assigned to the weapon make armor sets where the armor pieces again are purchasable individually and then get assigned to the armor set but these can be unassigned and traded separately again I got a little wordy but I figure that's how people talk to them right and so it says yeah great I'll make you a smart contract system and so what I'm expecting to see is something like this I mean I think I said I want to be able to buy the modifi separately right so we would think they should maybe be 721s that I could maybe buy on open C and I want to be able to assign them to the weapon and same with the armor I want to be able to buy pieces and put them in on an armor set so I think this is reasonable expectation of what my my prompt would have described but it didn't do that at all and I'm not going to show all the code it generated because it generated a couple hundred lines to try to create this and it made it very bespoken non-standard and I said no no no make it so that everything is a 721 and that there are different contracts for each type of thing because it tried to make one mono contract that had all the things and had a purchase function that something would know how to call and all types of stuff even though I explained that and it said okay I'll split it up good we're getting somewhere and then I got a contract from that and I noticed that when I had an armor set that it didn't consider what happens if I sell the armor set it didn't consider that those armor pieces maybe should also belong to the set that means when I sell the set it should belong to that own the new owner and I said all right I want an armor set that when sold applies the new ownership to all the assigned pieces of armor right and it goes okay I'll I'll I'll Cascade the the ownership right and it did it wrote that now we started trying to compile because this is just me getting the code back and looking at it and I noticed that uh I was was trying to compile it with open Zeppelin 5 and the latest which has been out for a while and uh some of the code didn't work and it said it said oh you're right I used open Zeppelin V4 I'll update it for V5 and I said okay it gave me code I tried to compile it still failed so I pasted the errors in it said right right yeah in open Zeppelin V5 things change so I'll update that for you and I was like okay good finally we're going to get past this so it gave me new code I pasted it in I'm like what can you just fix this please and it goes sorry I guess in open zepp1 V5 something changed again and I'm like we talked about this right I thought we're using V5 you can't get the okay let's go more problems and it goes I see the issue is that I'm still not using open Zeppelin V5 this I mean this is really my interaction with my I kept taking the code try to build it again same thing this just continues here again it's like oh okay well now in open Zeppelin V5 something changed it's now virtual I got to change it I'm like okay well let's let's keep going still can't compile the code it gave me and it said okay we're getting somewhere we just need to make sure that we have the right Imports that we have the the we're now using the hooks and finally it generated code that compiled except for the test and I looked at the test code it generated and it literally concatenated an old version of the test that generated and a new one if you can see it so I just hand copied that off and said okay we're we're not perfect here and after 28 iterations with 3.5 I finally had working code right so we're all good right it was a little little bit more work it would have been nice if that had been an agent you would have just watched it loop 28 times and do the thing instead of me telling it um but but at least we have some code right so we're good well the problem is that once I got the code from it I looked through it and I realized that what it generat for me is largely unusable um it never asked me how I want the mints to work on these 721s it just sort of made an assumption and ran with it and what I wanted and what I sort of would expect a system to be able to do is to offer some form of self-service meaning a user can show up and they could go to mint. fund and create something or maybe they could like um maybe we we just have something where there's some payable mint function or something and they can buy the item they want but it sort of assumed that I was just going to Mint all the items myself and put them for sale myself and that was you know an assumption that I didn't want self-service um one huge problem is that it had zero logic to check again it generated working contracts it generated working tests everything passed but it had double assignment errors where if I had that armor like I was talking about there's an armor set if I assign those pieces to one armor set I could mint another armor set and assign the same pieces to it and then the cascading ownership would break when I transfer the second one so it was really a broken contract um there was no nothing to no data driven setup for the mint strategy so I couldn't say I want to I want it so that you know this is a helmet and it costs you know 00333 e or something I couldn't do anything and a ton of minor issues but after all those 28 iterations I still have to start over I have to make new contracts because these aren't going to work for me the biggest issue is that the llm itself does not understand the greater architecture requ ired to build safe efficient and interoperable contracts and some of this is pretty specific to Smart contracts because there's a lot of great examples out there but you have to get offly specific if you want it to actually work in the way I was just saying like if I if I want there to be you know data that that that gets um push to it to to configure minting and I want users to self service so a person running into these problems might assume that well the llm just didn't have enough context right The Logical next step is let's the llm usage let's tell it the logic in detail that we want to have let's give it examples let's let's give it more context and let's refine the core prompt surely I can prompt my way out of this okay that's logical now if you do that the good news is that you're going to significantly improve your output it does work better to refine your prompts but the bad news is that it's still just grabbing and generating contract code out of nowhere you still have to manually audit every single line not just audit you have to think through logical Minds that are going to be in there like that double assignment problem a lot of people wouldn't notice that and and I'm sure it had other problems I didn't notice as well and you're only going to find out after you deployed this code right and it's questionable if this is better that whole process is really better than just manual development for oneoff contracts that you're going to make it might be safer to just do it yourself so you saw the prompt that I had in the first place right I I posted that this is my refined prompt I'm not going to read it to you but you can see it is full of technical details I'm explaining exactly which function I need to have I'm telling it what modifiers to put on it I'm I'm telling it data types to use for things I'm giving it rules I'm saying only use open Zeppelin V5 I'm saying make sure that you can only assign a weapon modifier to one weapon at a time I'm saying make sure that only the contract owner can actually add weapon configurations I'm giving it everything and I'm giving an example contract this is what I mean by refining your prompt you have you you literally have to go this far in order to onot something or even get uh a few shots of of response to get something that is what you want and you have to think through this ahead of time so it's kind of a pain right so I'm going to propose today that people consider a hybrid approach all right I'm going to call this the 8020 approach I'm going to say that you should either use an existing Library or existing software or develop if it's really custom develop procedural code generation that does about 80% of what you need done let's say you're an app and you're going to need 40 different smart contracts it's an asset system or something and you've got all these classes of assets I think that you're better off generating 80% of the code for those and then only customizing the little bits that are really different unique or different logic with each contract either by hand or with an llm to do that right and if you do this the llms requirements go down substantially um it no longer has to account for your overall design you're doing it it you can give it really detailed context on what it needs to have you can give it the exact logic that you need and um you can give it an example and your prompt will be smaller and it will do a much much better job for you because it doesn't have to do all of it at once it can just focus on the custom logic that you need this approach is applicable for projects that have lots of contracts or you know um where lots of different but sort of similar contracts or or groupings classes of contracts right now if you want to develop code generation tools the general process is first you need either manually or using an llm you need the correct code so it's just fine I mean prompt away get the code that you need audit it make sure this is right then templatized take that code that it made refactor a bit make it so that you have just just those reusable sections with with um within a template and uh and then now you can just generate the contracts and it's always that audited code you're getting every time right add some tests and you're guaranteed it's deterministic and you're going to have success so you might say no no that's that's not what I'm interested in I I I think that we just need llms to get better and better at doing this they're going to they will right uh this sounds like a very old school approach because it is but the reason that it's not a bad approach is because it's highly effective the the output you get is deterministic and not doing this the alternative is typically I made one a couple of contracts and I copy and pasted those to these others and I change the logic and I copy and pasted those we've done that at our company and I can tell you that we almost always forget something or another and we have funny little bugs here and there sometimes not severe but copy and paste is only so good the bad news is it's upfront uh The Upfront cost is pretty high and if you're just generating if you're trying to make one or two contracts this probably isn't the best approach for you right so if you're interested in this you could say well what what things are really good to generate procedurally uh things that are going to be conventions across a group of contracts let's say you're going to have many contracts that are minable well the mint functions and that you might want to have um the relations that that are going to be so if my contract is going to relate to this other contract do or um generating the code that enforces the relationship is great uh anything involving the mint strategy so I can upload the same way to tons of different contracts I have just anything you see repeated across contracts is good candidate for procedural so if you're wondering well how do I go there though how do I actually do that well this this is this is a project configuration that we use what we're doing here is if you can kind of read this code it's this is just some some uh Json that is is showing it's not Json this is typescript and it's showing that I have some contracts I have character have a game item they have these Mo like here's a model I've got these fields they need to have this one I we have a thing called fragments it relates to this game item here so my generator can look at this and it can say I know there's a relationship between game item and character I will generate the code to securely manage that relationship for you I know that you want these fields of these types I will generate code that can that can handle the the bits of security that you want for access to those things right so the way a workflow works like this is first you need to make that app configuration something like I just showed then you pipe it through a procedural contract J there generator I could speak um there are actually many out there that you can use um you will get some generated contracts but those generated contracts are going to be missing some of your custom logic so you take that generated contract you give it a refined prompt you put the two together through the llm and now you have a final contract where you only need to audit the diff between the generated contracts and the llm generated Parts it's significantly easier to do and you get far more security and and uh much higher chance of success with your with your code gen so an example post generation LM prompt like I showed you my my one before paragraph lots and lots of logic rules in it this is an actual one you could do I could say now that I have a generated contract I could say modify the contract given to add the following functionality a weapon modifier increases durability or damage of the weapon is attached to so it'll add some code when you assign that weapon modifier it'll it'll modify the weapon and then I want to say only one of each weapon modifier type may be attached to weapon a time and here's my generated contract your llm now has a very high chance of success it can see where it needs to go it can see how everything is already laid out it doesn't have to do any guess work and it can probably generate this code just fine and you can probably audit it just fine so we have a full implementation of tools that do code generation we're not the only ones that do that but uh if you want to take a look at any of the stuff uh we have a system called Patchwork that takes that configuration and does all of that uh if you're curious or have questions please join our Discord and uh we'll help you out with anything that you're you're trying to do I'll leave this up here for just one second in case anyone's interested also I'll provide a link at the end too and the future of Coen ultimately we all know we're heading toward no code we know that in the future it's going to be plain English that you're going to use to write your software things like smart contracts may stay special case for quite a while because of the general permanence and and because of um how big a blast area is when you have a bug and how sometimes you just can't fix it it's it's on it's on chain you're done but we know we're going toward no code and I think the real question though uh for us Engineers is what does that look like under the hood we know the user experience but under the hood is it just a a sea of of Agents or is there is there a bunch of hybrid stuff under there where we're making specialized Frameworks for agents to use as tools to give us secure generated code and I think it might be something like that there's a link to the slide some resources and whatever you need and I'm a little bit early but uh that's all I've got so thank you very much no thank you appreciate your conversation out here so well done