Building Successful Codegen Agents - Berkeley 2024

it's great to be here I'm actually here from Sweden and er told me there's going to be an open source event I I love open source so what's better than Ai and open source it's clear that AI is going to transform pretty much everything so I wanted to do something even more ambitious and I'm on a mission now as I see to unlock human creativity at a a very early company which is my my second one I'm building um and that actually came out of this project how many people have heard about something called GPT engineer here some people cool so I uh built this tool internally to automate myself so I could retire as a software engineer and as CTO and then I put it out as a as a tweet and at the world's most active and popular code generation project so what uh GPT engineered allows you to do is to play around with asking AI to create software so I'm going going to ask um GPT engineer to create a community website well and then what it will do under the hood is zoom in here let me know if if I should slow down zoom in whatever what it will do if I have Wi-Fi which it might might have uh is to call a large language model and to wrap that into a lot of code maybe retry if the code doesn't work work and so on um and create a community and while this is running so I mean this is maybe not so new to most people but when this is running I'll just explain what what we found uh in the community building this it's that this AI Sy these llm based agent systems they don't work 100% um so what you want to do is to tackle problems where you can instantly see if they are behaving as you want and and this I think is a good example building a simple website um and then once this is done you you're usually you're not happy with the first version of anything you do as a human either so you want to do small changes to it and I I can't even promise that this is going to work but uh it if you apply this in the right way we are uh taking way or with the right user interface we are replacing this the simple things that Engineers do today software Engineers do today and uh we're moving towards a world which I think is very inspiring and it's a world where anyone can build software and um the reason I build software is because I want to create express some creativity and I I think that's what we're going to see um much much much more as this technology comes into play what what's the conclusion of out of out of this well um since we we saw that okay there's a lot of people building software engineering with AI um but there is an unique perspective here on focusing on the simple tasks and then building it out we started uh me and previous uh co-workers we started a very very ambitious company in all of this at the same time I was like okay this we going to do do this open source if we're building a company that is all about letting anyone build so uh not just building devits then then the case for doing it open source is not so strong so the way we approach this and enjoying the passion of building open source as well is that we're have a startup that is commercial and that funds an open- Source community that we have a Synergy with so the the open source Community mission is to support coent Builders and it it embodies the the ethos of actually just creating value for for everyone and doing everything open and free so you don't have this trade of should you make everything open source or free or not uh while as what the the startup does is to build a separate technology that is not yet open source uh and with it looks looks like this question how do you how do you do both open source and the commercial upside um and I'll go from this into a bit about more about the community but first what what product are we building so as I mentioned the first place to start when you're creating a fully autonomous software engineer that does everything as human software Engineers can do which I know will be possible is to start with the the UI we have a close most weight list product that people have tried and if you go in and you look at what people think it's um generally very favor favorable uh people using it it lets you ask for anything you want to build connect it to how developers like to work which is through GitHub and a git repository so that you both non-technical people can ask for changes to your application and Technical people can develop it as they are used to uh and and this enables any found or CEO that wants to do quick quick changes or product managers designers to do to do changes uh while the developers can maintain it just like they want to I'll see show a quick video on on the product that that we are building so here someone is asking for a quick interactive application and this is a toy example a game I'll I can show you some examples that are more interesting interesting for real software products but uh again what's interesting here is that the code is available just like it is for uh as it was built with a only humans um and I think here is a fun example where someone has built a cryptocurrency um coin market cap clone connecting to the actual apis of the uh code and going through a lot of iterations by just asking our product in uh natural language so you can actually use this here so that that's about the the the product that we are building and this is uh quickly evolving we we have a very strong team over in Europe um what is the community doing so I I as I mentioned this is uh how we are doing um both the open source and the commercial product and the mission is to support Coen Builders and US Coen Builders ourselves and every everyone else both from Academia and in the industry have these problems that you are solving a very hard technical problem and you have a lot of sub problems and there are many ways to evaluate and H you need to build your software you need to build your product and evaluating takes a lot lot of time unless someone has built out very simple INF infrastructure for this so that you can compare how does different how do different uh code generation agents AI um systems compare on different tasks and which what should you use where and if you have very strong technology we all everyone wants to show that you have strong strong very good technology in a way that can be verified uh objectively so this these are the problems that a community which I mean I'm on the board of the community I'm Board of the startup but they are two separate entities uh are addressing and the focus is for the community uh right now is to make it easy to valuate agents I'll show you quickly what that looks like so um here we have an abstraction that makes it very easy for a developer that's building an system for writing code to um expose their capabilities by just wrapping it into an abstraction this is too much for non- technical people but it's uh showing how how simple that uh is and then once you have done so you run one command and then in here we are comparing an external code generation agent um on evaluation on a Benchmark to see how does how does this specific agent perform on this specific benchmark and um this is this community effort is very much in the early stages um as I mentioned we have a a t ton of active contributions um and specifically on benchmarking we have um done a lot of early work on on get making it possible to run different benchmarks for any agent um but we are looking to actually have someone with who is both Technical and the leader to take uh take lead as a um lead maintainer in the open source Community which is again separate from the startup so if you know someone who you know is passionate about this this field and um competent and the leader please connect with me and let me know so I think that's um high level what I wanted to talk about um I put together a link for you to be able to reach out and you can add me on LinkedIn I think with this QR code and if you want early access to the tool I mentioned you you can see the link to signing up on my profiles and uh I don't think creating a slide like this is the most easiest thing so actually I made this slide with our tools I just brought what I wanted and I got this slide here yeah nice that's H it for me so if you have any questions happy to answer and if you want to see a live demo during 2 minutes I can do