Emergent Behavior in DeFi: Agent Based Simulations and Reinforcement Learning with Tom McLean

uh what we call parameter optimization and emergent behavior in defy so this is a trimmed down version of a two-hour Workshop that we gave in Devcon last year so I'm going to try and cover all the bits that we covered there I've updated it a bit with some new information um but there's also various additional work and stuff if anyone wants to dig in later that I've got links to share and all this stuff uh so yeah I'll cover I'm going to go over what Vega protocol is uh to give you some context behind what I'm discussing here uh why we think that large-scale agent-based simulations are a really good solution to the issues that we see becoming more prevalent in defy as it grows and something that we think should be tackled now but can also be a really good benefit to the the ecosystem and like a selling point in future if they're tackled properly I'll then dive into the code a bit more on what we call the vegan null chain and the vegan market simulator this is how we kind of answer the questions that we raised and since the workshop I'll try and do some actual coding um see how well it goes and okay and yeah show some of our runs of scenarios and stuff so first in quick introduction I'm Tom McLean I've worked in traditional Finance for seven or eight years price joining vehicle protocol and I've been here at Vega for about a year working in the research team sort of answering questions like this and building out tooling to do so protocol itself is we're a layer one well we're building a layer one blockchain uh it's proof of stake it uses tender men for its consensus layer but beyond that it's totally sort of custom processing and the idea is that it's really optimized for trading derivatives and generally margin products it's got a full order book system so it's not an amm Style placing orders people can come along they trade later you've got a whole shape of prices one of the key distinguishers of vegan protocol is that the market creation is totally permissionless so people can come along and propose their own markets based on their own oracles the community decide what goes in and once it's voted people can trade on it in future instruments should also be totally commissionless and can be created by the community whilst this is the strong selling point it also obviously introduces a whole load more complexity because you've got to ensure that if someone can create a market it's going to be it works and it's not going to block the entire system there's also various liquidity mechanisms to encourage early adoption and early trading so the people who join early and state more liquidity get more of the revenue later when the market grows a lot of people ask why why run your own L1 why not just Branch off ethereum or other evm type things one of the crucial things is that we think a limit order book style Market you really can't have fees on placing Trend placing orders because that's a lot of a lot of your information comes from that and you don't want to be sort of blocking that information flow that really makes the market work so we have no transaction fees on any of that people pay a fee when they make a trade but before that no fees additionally we can have Atomic Closeouts which means when someone is distressed when someone runs out of money you can immediately close them out properly close them out ensure that the market stays robust without waiting for another person to come along and close you out we can also Implement risk calculations really sort of bare metal level to make sure they're fast and actually proper risk calculations in future there's also plans to have what we call Wendy it's a custom order of block ordering which ensures fairness as much as possible in terms of first in first day of trading so what do we see as the problem and how are we going to fix it this is a problem which really isn't just isn't just an issue in defy it's an issue in traditional Finance as well if you imagine markets I mean a lot of the blow-ups in markets in the past have been because people just can't picture the whole Market you can't test it any given bank will have sort of risk models and tests for certain scenarios that they see if this person collapses what do we do if the market moves Five Points this way or that way what do we do but there's very little at a sort of systemic whole level and it's really impractical to do it like Regulators might a couple of times a year do a whole practice with multiple Banks but that's hugely time consuming hugely effort consuming and you don't actually have a guarantee that what you do there is actually what would happen when the market does collapse because there's so much human intervention moving to a D5 World though so much of it is in code that we can really we can assure that once we've tested something if the inputs are the same that's what it does we can also set up scenarios to run as often as we want and with whatever parameters we want and really sort of stress test the whole system this is something that we see becoming more and more of a problem now as things grow as D5 protocols even at simple levels the interaction between them will inevitably create sort of much more complexity than you can really reason through a priori and as they become more complex as their interactions become more complex the difficulty just explodes so it's something which you which can really be an advantage of D5 but it's something we have to tackle Bill manageable before it becomes just too big like a traditional Finance system awesome um yeah so there's also a lot of parameters especially in Vega protocol because of the complexity but also in other markets too that uh it's hard to immediately go from what you want to set the parameter to to how you want the market to behave so unless you can really test out how the market will behave under this setting or that setting you don't know and you can't really make safe well well reasoned decisions so we want people to be able to trust markets to be able to trust that they know what's going to happen if things go wrong or in a weird edge cases so that's why we think that this is a really good solution when we say agent-based modeling it's ultimately it's a pretty pretty simple concept you build out what we call agents what people call agents who act as an individual person in the market might they might buy and sell randomly they might trade they might try and exploit the market but they can you can rationally reason out that this person does this that person is that and then ultimately you put them together in a market and you run it and you see what happens but through doing that you can hopefully simulate really good models of a real world when we're going to talk about types of Agents we can start at a really base level with zero intelligence agents these will be hard-coded based on sort of the state at the time but with no real optimization no sort of learning as time goes on examples of the kind of things we have here will be we'll have some some traders who just buy and sell randomly each time there's I know 50 chance they buy a 50 chance they sell so they're not doing anything particularly smart but crucially they're not changing their behavior as time goes on at the other end of the scale you might have an optimizing agent they have sort of a full knowledge of how how the market Works how other agents work and for them the whole world is a an optimization problem so once they step away they can figure out exactly what they think should be doing but once they come to the market they're not changing their behavior they they know what they think is the perfect thing to do every time and they'll always do that in the middle ground and what can be really helpful in our case a reinforcement learning agents generally for anyone doesn't know their their General cycle is that they will look at them world they will take an action based on their beliefs they will look up how that action affects the world how that benefits them and then they will change their behavior in the future often people might try and use these to build like good trading agents but interestingly the main thing that we care about here is that they try to explore the whole state space of the market which is what we really want and be able to have confidence about Its Behavior in certain situations so now that we've got a bit of an overview I'll dive into how we're how we're implementing this and how we're testing this as I mentioned we're using something we call the Vega Market simulator which ultimately is a full Vega code base with the tender mint layer which is a consensus layer and generally does the that does the communication between nodes between validators and builds blocks we get rid of that and we call it a null chain instead that means it just accepts transactions blindly and allows you to forward time or stop time or do whatever you want really and gives you real control over the market you can fast forward time hours days or stop it dig into it see how it's behaving from there we can then build an API layer that translates your blockchain level Communications you're caring about block time about state that kind of thing into a market level language that you talk about trades buying selling placing orders and that lets you really Express agents much more it's much more how they should be thinking rather than worrying about being on a blockchain from there we build out what we call scenarios so these are ultimately sort of aggregations of composable agents that will will have selected from some library and then a state of the world so we might have a price movement and that's where we'll set up the Vega setup itself like block size time a block represents things that you might care about but don't necessarily agents don't we split a scenario into agents who take their actions each step and the environment which is sort of the number of steps some config some logging that kind of thing and then it's just a diagram to picture out how we split it out we've got the environment and agents which are just each one will initialize print itself some money generally and then for however many steps we go they just get a they get a turn they get a run turn they do their stuff and we'll move on to the next one an additional benefit of the market simulator is that we can in addition to this null chain version we have a network version that we can use for trading with other people so if we want a test net that the general Community can come and try out or just um load load test the network we can spin these up we can connect to a remote Network and because we've just stripped out that layer all the Bots and the Agents can do their thing and they're just Trading the same way and then a bit just a bit about Vega itself if you're spinning it up locally you'll have a core node that is your your network a data node that is like your historic layer so that's just a big SQL database basically that ingests events and uh gives you a nice historic View and then a wallet and a console because ultimately we find a console that's built for people's trading it's also it's a really good debugging tool because it's designed to make the market clear and usable for people uh so from here I'm going to jump away from the presentation for a bit and show building a really basic agent that I don't know how zoomed in is that visible one more I wonder where the zoom is soon cool uh so this is I'll I'll send a link to this repository there it'll be at the end of the thing but ultimately this is a really simple repository that lets us wrap one of our scenarios and has a simple agent in it and a run step so the Run is we'll run it with a this simple python button agents there's dots for it and what it does is it sets up this scenario that is curve Market maker it's a market maker it builds a nice curve for you you can I don't trust the internet here but it's got a function to pull historic prices and then it makes a makes a price press curve for you over however long you want and you can once you've got this this builds your agents and we're going to plug in our own agent here plug it in and then we just ultimately call run and it'll run through and tests the agent sorry so the question was how how are the interactions between the agents modeled ultimately the way we see it is Agents don't interact directly with each other they all they see the market so each one gets a turn it can query from the market whatever it wants it can send stuff back to the market and then they interact through the with the market as the intermediary so each each sort of Step will have a block of agents and generally we'll sort of Shuffle the order so they don't it's not always the same one going first each time but then they they interact by the the market changing and state changing so uh yeah so the we've got a simple agent here it basically has these two steps and initialize which we're not going to change too much here it's it gets a market and it loads information about the market about the asset and then it mints prints itself some money because we're in a null chain world we can we can just take money for free and then the meat of it is this step function and that's what's going to be called every every time around and so let's put together a simple one if you're if you load this repository we've got a few sort of useful helper functions at the top so I think today we're going to build an agent who let's say they pick the mid price at each time point and if it's higher than the last one we'll buy because it's obviously going to go up and if it's lower it's crashing and we'll sell it so it's a very kind of describing the market as you want it I think so we'll we'll just pick our mid night we'll need a last minute let's say it's greater than our last mid but then we're gonna buy and we've got another function up here to buy place a market order a market order for for people who aren't too versed in these things just means that it's buying an amount of whatever price it can get so that just saves us having to think about prices and then we'll say if the minutes less then we will instead sell and then at the end we're gonna have to update on it and that's really all you need to make a a bot with some highly complex logic we can run this now and it'll it'll run it'll trade on the market and if you wanted to it outputs a whole massive log at the end that you can dig through but for now all we really care about is how much money we're making so we've got another helper function up here you can throw in and it's going to load our positions and then print out a position you know that is let's try running it and see if we see if I coded that right so we're spinning up here we log out some some output about we've got end points for rest for grpc for graphql and then we are trading and losing a large amount of money but if you want something that maybe trades a bit better this is an easy place to start off and hopefully don't lose six thousand dollars so that'll run up and run it all way off now jumping back the presentation so we've got a few agents in our library that if you want to spin up and test they're all sitting there waiting for you we've got an Ideal Market maker and a curved Market maker these are optimizing agents as I discussed before so they use mathematically perfect the values for best bid and ask based on their model of the market if their model of the market is correct then they're good prices if it's wrong then it's wrong they also sort of place a nice shape of orders behind it and we've then got some other agents to simulate sort of activity on the market a liquidity taker who blindly trades up and down an informed Trader who has some estimate of the future and momentum traders who use a technical analysis type trades to not make money so well we've got some time left how are we actually using these to inform the development of Vega protocol and improve it and make it more robust one of our key things is in parameter setting this is choosing good values for the inputs that I mentioned earlier to do this we just set up a market that is simulating some real world conditions we throw some agents in and like a good experiment we run the same initial conditions with our one value changed and we see how it changes and hopefully we've got some Metric of how we want that to affect the world and we'll choose the one that does the best there's a few difficulties here still obviously one of our main ones is that we've got to choose our initial conditions so really there's a few ways to go about it obviously one we just test the range we test either simulated prices that conform to some real world fitting or we just run actual historic prices the choice here depends on what you're wanting to test some of our things will be testing uh what happens when the market crashes and there you you can take there's a few examples obviously if crypto markets crashing though we can take and run through and we've got to assemble our agents as well so here this is probably going to be too small to see but the brass at the end the interesting bit so this is running here one of our parameter tests that I don't trust the internet but there's loads data from historic beef prices we run 20 different price windows and for each parameter we choose reduce asset and run them so here we're just running through I'm sure it's entirely invisible different uh agents that we've got plugged in so we run with sort of a curved Market maker a whole batch of random Traders and then here we're jumping through time to show it sort of simulating stuff if we jump to the end we've then got a nice Jupiter notebook to visualize how this impacts the world so you can see for most things these parameters are pretty the same Market State means that we're in a nice trading thing it occasionally drops into an auction but they're pretty similar where we get a difference here is we can see at the end here if I stop it uh I don't know if that's visible but it says received liquidity fees so what we were tweaking here was a parameter that affects how well Market makes it paid which obviously for a market is a big factor you've got to decide and we have a payout based on how competitively you're placing prices so we chose here basically affecting how much the payoff drops off as you move away from best bid or ask and we can once we've got the simulation quite easily look at these we can say how much does a market maker make on varying conditions we can look at how much they make on a centralized exchange and how much they expect to be making and then we can reasonably make some choices about what our parameters should be to ensure that prices are good and market makers still make enough money to stay with us so quickly welcome to time the RL agents I mentioned earlier how do they help us the main way we found so far as I say is testing weird edge cases so when you've got your your zero intelligence or your optimizing agents you can really run a lot of tests but there's still there's a limit to how much Market State they cover because they're still doing sort of a small set of actions they're doing reasonable things but often we want to put Market in totally unreasonable States we want to go to weird worlds that you can't you can't reason out you can't set up a agent to go in each way so what we do for that is we're running long-term RL simulation runs where it's just constantly trying to learn better and better trading and will re reinstantiate the agent get them to try different paths and the great thing about this is that it's constantly trying to go new ways learn new things and through doing this we've found a number of sort of weird edge cases that you might only hit one day a year running it normal trading but because we're running 24 hours a day we can hit them in a day or two and fix these before they become issues on the real Market so closing out well it took us time uh just a plug for mainnet Sims that we've got running so we're coming up to hopefully releasing a code version that we believe is is suitable for production trading so we're running a few what we're calling main net Sims which is a week of trading on a test net with Tesla assets but with a reward pool at the fat to about 69 000 fake tokens which is our governance token uh it's worth a dollar or two currently uh split between the top Traders so if you want to come along and have a go we're trying to get as realistic a marketing conditions as possible we've got two more to go I think the next one starts in about a week so you've got just about the right time to to get things set up WTF and then we've got yeah so thanks very much for your time this QR code takes you to this presentation if you want to take a photo of it it'll get you back here it's a few interesting repos we've got vegan market Sim which is the whole complex thing agents which was a simple one that I showed just there that's got the one thing to run and then a basic Market maker that's a separate sort of agent that runs on a live Network and does some semi-smart listening to a binance price and quoting on Vega so you can you can maybe just spin one of those up and run it on the simulation and see if you win yeah thanks very much for your time uh don't anyone has questions [Applause] uh yeah you mentioned uh earlier in your presentation about your API that you built that API was that yeah sorry that I guess helps you avoid some of the difficulties of actually interacting with a actual chain Yeah so basically uh I'm just curious if you could maybe explore that a little more on kind of how that helped you or how you built it or anything you have on that would be great and so I can I can show you the the repo later but broadly it's uh I mean it's a big class that's sort of got methods that just have Place limit order place market order and then the values on that will be things that a Trader cares about when they're placing limit order so the price the market the how much you're placing that kind of thing and then it converts it to a being a blockchain most of the numbers are sort of ins so it does all this like conversion from floats to ins and it does the communication blockchain and stuff so the Asian just calls a thing called Place limit order and doesn't care that it's connecting to a local node that's connected to a remote node that it's waiting you can have functions to wait for the block to commit before it gives you back the thing but the agent itself is just trading on a market for what it cares okay cool and so were you using that API in the examples you were showing us was it being implemented there yes yeah so that was all used in the FI and it's um it's uh without going too much into python stuff it's like a context that you go into and then it sets up all your connection and stuff when you do that so you can and is that exclusive just to Vega like that it wouldn't work on like for like a uh yeah it's uh it interacts with the sort of Baker chains yes okay it's using uses uh our rest endpoints our grpc endpoints to to do the communication so okay cool thank you last question thank you um my question is when you're training does like the smart agents do you train them in uh train them against an environment that includes other smart agents so are they all kind of like antagonistically learning from each other or is it kind of a fixed environment that they are trained against and then you kind of set them loose against each other in the multi-agent simulation uh we've kind of done both really um our original was obviously just like one in a thing we've tried out adding adding a couple uh one of I guess we've kind of got an advantage in that we don't really care whether they actually make money in the end so our main thing is that we want them to cause a bit of chaos and send the market in weird directions so yeah we kind of we can throw a few in and see what they do and observe that just as well as doing one or two so got it thank you cool thank you um I think we've got to go but anyone else who wants questions I'm happy to answer