Building AI Agents with XMTP - ETHGlobal Bangkok 2024

hey everyone I'm fabri from the XMP team the parent company is called epimera and we are the developers of the XMP protocol called uh XMP of course the app that is called comerse and now presenting message kit so we are owning all the layers of the stack everything is open source every everything is going to be soon decentralized so you can think of building on top of the protocol and inbox or you can think of building on top of the app agents Bots mini apps uh everything that is fun for developers today so we are going to go deep into how to build agents using message kit and so for that we are introducing message kit which is a framework for building Bots and AI agents on top of the xmtv protocol uh everything on on top of the xntp protocol is is encrypted end to end open source and soon to be de Central so we are going to dive deep into what kind of things we can build and for that we defined what we call AI skills and AI skills is a five-step approach to reasoning so to build cool agents that are smart and can perform actions so we are going to go step by step uh like an overview and this is all of course documented in the docs but this is what I'm trying to so we are going to start with the demo this is what the reasoning agent looks like and this agent is going to be interacting with ens is going to be interacting with domains registering renewing uh doing fun stuff with ens domains so let's let's put play this should take like a minute this is the converse app I'm opening I'm starting a conversation with an agent this is called ens agent recognize that I'm fabri that's my Converse username is fetching that from the from an API also recognize is register fabric. e is register so now he offering me cool Alternatives that I can use to registers domain so fabri lfg actually is available so points me to the to the website easy enough now what else can we do so yeah so I was curious who's the owner of fabric because my name is fabri so I wanted to know more about this owner apparently we can get the domain information it's also fetching from the ens apis and he's asking me if I want to tip him for the for getting there first sure let's tip him and this is just to show that I'm able to tip from an agent I'm able to perform a transaction this is going to send a frame so this Frame triggers a base transaction by default is one usdc that's what we think of when we say tipping maybe $1 um and as you can see the agent understands everything that I say I don't need to perform like very strict commands uh can I renew it no you can't it's not yours so in the back end that this is how reasoning looks like it knows that I'm not the owner of fabri and then performs action based on that can I register pallic no you can't it's it's register so understand the context understands my name my my domain and you can just have a conversation with the agent and the agent is going to be as smart as you can wish um what else is in this demo okay so that's reasoning that's the that's the end of the demo so now we are going to go backwards how how can we achieve this type of interaction with an agent using message kit so as I was saying there is a f step approach so we're going to start with reasoning now we're going back to fine-tuning which fine-tuning may be the most important thing so if you see this piece of code this is how I train this agent this is all that is needed in the prompt besides of course the code that goes and checks the the apis but this is why the bot is a smart this is called fine tuning it's not that hard when you when you hear that you need to train a bot or you need to fine-tune a bot this is actually what training and fine tuning looks like uh today so is basically giving example interactions and how to respond based on those interactions and you will see that I'm using uh new lines as separators and maybe after a new line you see a command so that command is going to be performed internally by the bot so that's uh well fine tuning is a very important step to make the B smart before fine tuning we are going to so we are going to embed that fine tuning into the the whole system prompt so a system prompt is the thing that you're going to send to the agent and then you're going to also send the the ongoing messages so it has memory so in this example for creating a system prompt we are using some prompt rules which is uh you can see them in on our dogs it's like don't be rude be helpful you are a a funny agent on a web free messaging app don't guess don't make assumtions you know like guidelines and things to the vot to behave then we are giving context of the user who is speaking to the agent that's fetching information about your ens or Converse username also recognize if you have NES domain and has all your information your wallet address so the B is aware in every interaction of your context as a user then is going to parse all the skills we call them skills which are the the actual skills like checking an domain with an API and fine-tuning and then we in that fine tuning maybe we want to replace some values like you can see values between the slashes like in domain or prefer name so we can do like um a refresh based on the yeah customized to the to each user so after we generate the system prompt this go is I can show an example later but it's a it's not that long uh we get into processing so processing is hey I want to check if vitalic e is available now it goes into parsing the agent is going to based on our fine-tuning this interaction maybe uh somewhere here so the agent is going to split the response into two lines and the second line is check balic because it start with a slash the agent recognizes that is a skill that needs to be executed So Below you are going to see that how it iterates through the messages of the array and if it starts with a slash execute the skill if not assumes it's a simple message I just send you a response now going deeper this is how parsing looks like so we have now we need to execute the skill the check balic doe so what happens here is that internally this command is sent to message kit and message kit returns a a shason this Jason says the skill is called check because it's already defined we are going to see how to define skills and these are the parameters of this particular skill and you can see the domain and it's available in in a Shon format so you can you you're now you get to the code and you're able to play play around with the parameters and the command if the if instead of check there was a typo this wouldn't work so this command needs to be very strict and the the agent knows that these commands are strict so they always get reliable responses so what happens after we have the values now we handle them so in this example this is how to handle the check skill I'm receiving the domain as a parameter which was balic doe then I'm using one of our our functions that is get user info is going to give me information about balic doe like address uh domain description profile picture and so on and is checking okay if there is an address hey it looks like sorry if there is not an address means that the the domain is free so that's the response that you see in the demo and if it found an address then then is registered so it's already register let's propose some cool domains and now I'm executing the cool skill internally which is going to return five examples five cool examples like Fab lfg like we saw um finally this is how we create the check skill so it's basically like a shason structure very similar to how you define in discore or t bots so you will say this bot which has a tag ens at ens you can use that tag inside a group chat for example and it's going to recognize that you are tagging that bot and it has more skills in the demo this I'm just showing the check skill so the check skill receives one parameter you can see that domain between brackets triggers just like a double check to see if the the skill is trigger examples of how this command should be used these are all these are all parts into the system prompt automatically so this is what gives uh the smartness to the bot is very important this part is also very important to be defined very clearly and then we Define the the type of parameters that we get so in this type is a domain is a string but we could check if it's the e or if it's a URL or maybe a yeah evbm address so now that we have seen how to get to reasoning uh again all of this was demoed on Converse this is our open source app it's available on the App Store and it's going to allow you to interact with both creating a message kit each bot creating a message kit is just as any other user in commerse you will need a private key and it creates an account and then you can talk to that account so we are going to see one last demo with different skills these these all these examples are open source are available in the docs and this is called base agent and I want to show how bation looks like so okay let's see the Demo First you know you can see like comers doxyz dobas agent because every profile has its own like landing page on Converse so people can chat to you like a deep link into messaging you and we're going to also see something more interesting that is real transactions so this B can help me with swapping with minting with uh requesting test n funds so let's say I want to swap okay this is going super fast let's say I want to swap so it has a frame this Frame receives the swap parameters that were passed by the agent and creates a ready to go button that is going to trigger swap in one uh maybe in coinbase wallet so one e for usdc again this is using all this the skills that we saw uh defined and fine-tuned and then pared now let's say we want to Mint in this example I'm going to use uh we have many ways of minting I just want to send a SORA URL and it will return me an open frame that is available to Mint on any minting um yeah frame so that's one way of mainting also triggering a frame now let's see let's pay let's try to pay in this example this is my manager called soul and I'm going to try to pay one usdc to SOL SOL is his username in Converse and this is how it looks tip Sol triggers Co wallet you just send one again these frames are customizable you can send any receiver address any amount any token any chain and they will just be triggered in the wallet uh finally if you need test funds you can go to base agent it's asking me sepolia or gly uh sepolia is fine now it's is connecting to the apis of learn web free they provide facet DEET uh in a very convenient way and as you can see gives me a receipt with the transaction that we just saw so that was base agent now how how much time do we have left we have okay any questions so far yes I wanted to understand if this is a conversation app so in case I tell that I want to swap one usdc and then in the uh it it responds back and then my following question is okay let's make that five so does it understand the context of the previous chats yes so uh there is one thing that it's called memory yeah and right now I'm using only what I call shortterm memory every time you restart the bot fresh yeah got it got it but if you keep keep speaking it's like uh the memory is there yeah it's it's in memory yeah okay my second question is is this kind of scalable because from what I understood is that it splits your question into bunch of uh you know instructions and commands yeah the passing and then based on the commands it's executing and then after that execution it's kind of a Handler IFL statement kind of thing so you know if it's SL check vital. e then execute this skill blah blah blah but then uh is it kind of skill because if you build five skills okay if you I mean you know just add five skills then it would only work for five skills but it would not know how to combine those five skills in work right to combine to to combine in the sense if I if I ask let's say can you check U if vitalic do e is available and also send one uh one usdc to solve yeah is it possible yes I only show an example that is like this one was only uh I want to check if vitalic is available right and it answer in two lines your your your question will return more lines yeah so how would it understand which uh okay so you just picking up the command from there and executing it one by one yes okay got it yeah and you can even like let's check out the docs real quick maybe you are in in the middle of multiple common processing like you need like to ask five questions so you will use something like this uh by the way this is message key these are the dogs you can start with a very simple CLI and you will have three examples the ens agent a group bot with uh use cases like tipping payments and so on uh middle wordss like how to connect with external apis for example I using this for the deset funds and to your question maybe you you need more Co you maybe you need your the bot to answer a specific one between commands so you canot wait for a response okay got it this will allow you to execute multiple commands but in between you know let the user interact with them got it thank you so much and this is this is this is very new so uh I hope it works but let me know yeah yeah it's okay um okay I can any other questions I can demo the CLI so so we we have more time if anybody has questions don't hesitate I had uh should I go I had question so how on how like how difficult is it to integrate an external API in case I want to build something that is a bit new let's say not a base agent not a crypto related stuff yeah but something else so if I want to integrate some external apis a bunch of external apis add a bunch of skills how difficult is that to do it yeah so they are um we call them middle wordss okay so for example you will just in this case this uh I use the Chrome middle War to send daily updates if you subscribe to a bot so you just have this file for example and then you would integrate that into your hand your main Handler so this is the this is the full ens bot okay with all the skills so as you can see I have a reset Skill I have a register how to register enss domain and each one of these code blocks can talk to any external API and returnal response and you can put those files inside like um I like this structure so this is a structure of a bot so I would what I would do is to um so index skills and prompt are like basic stuff and then I will put into handlers or maybe a lip folder all the external apis and middle words um yeah so these examples don't use that many because are the basic ones but if you yeah like for example if I go to friends day so we were hosting FRS day a couple of days ago and we were managing five different Bots each bot would perform a different action so you had Earl the boat ER would on board you to the event and add you to the group chat then you have Vu who was delivering popes then you had Cusco which was um the ens agent that we just saw just copy pasting the code and calling calling it Cusco and then p will give you advice about de consite events and panut will give you advice no we we allow you to play uh to play World in a frame so in this example maybe it's more you can see how the file structure it is but I have um so I'm fetching the event API from the ens team I'm also using a local database lowb I'm using Chron because the bot send the updates like hey vitalic is about to speak so this is this is more like a complex B I also have local data like this was a structure of uh of the popes and the subscribers you could also see more handlers and in this case you will start the five Bots you will start like five runners each Runner of uh for five Bots inside one bot and each Hunter will have its own like bot name and so you you you will be See Clearly who who is talking to whom uh yeah this is just like a cool example thank you so much uh sorry is there any like um ni Vector database that decentralized or weet three Vector database that you might advise us to use uh the database in web free yeah Vector database Vector database no um no I haven't tried any um no that's interesting why are you thinking about Vector database I don't know I I don't know maybe rag or something like talking no I'm using shason databases simple stuff because for for more complex stuff I recommend you using an SDK or agent kit from coinbase so the the cool thing is that a message kit plays well with all of them so message kit you can think of message kit as more on the ux side of agents and then if you going to perform like more deep more deep skills on on chain like agents autonomous autonomously performing swaps then you would need more like hardcore infrastructure like Brian SDK or agent kit from coinbase but you can you can combine both like making it fun and easy to use but also complex and even integrating Lang chain for example and using Vector databases with Lang chain uh this is a note back end so there is no limits to what you can build honestly um yeah just trying to keep it simple in the examples but hopefully you manage to build cool stuff okay and that's how you start with message kit that's the CLI I put any name you're going to see that on the right I have Converse on the left I have the repo this is the ens agent demo because I don't have my private key set it's going to generate a random one for me which I can just copy paste so each time that I start the bot it's going to keep the same address again we have we have the skills we have the prompt the index and now it's is even de blinking me to converse so I can say hi uh I don't have the openi API key yeah no open I key found so if you if you input now your open I key this is going to be like just the example that I showed before and if you want to see how a system prompt looks like in full you go again these are the five AI skills reasoning fine- tuning prompting processing par and definition and in prompting you will find the full prompt this is the full prompt so initial rules it was very hard to to get to these ones like every word counts so um yeah if you change any any detail this could like work very bad user context this goes and fetches information from enss or Converse apis then then passes the skills automatically so now the bot knows what are the skills and how to use them with examples very important and then we will go into the fine-tuning example responses and that's it that's a full system prom for the B shouldn't be more complex than that for six skills the more skills of course the more complex uh yeah then you then you are going to find more information about xntp like how you receive a text message how you receive a reply how you receive an attachment like an image how to use sht vision for interpreting image and splitting the build there is an example so lots of examples everything open source you can go here um happy to help you building along the weekend and if you have any questions we are going to be at the booth So yeah thank you I'm um let me introduce myself where is the because I was called fabri but I'm not fabri anymore I'm human agent. now this is my new identity and I'm very happy about my new domain so yeah product and at epimera uh happy to talk to you after this talk thank you so much