Why DeFi Agents Who Learn Will Outperform Those Who Don’t | Sam Green - Cambrian Network

hi welcome back to the bidle uh stage I would like to present and uh Sam green founder of Cambrian hi everyone uh so the outline for today's talk is I'm going to introduce myself and then I'm going to talk about reinforcement learning and the types of problems that it solves I'll then talk about reinforcement learning applications in Defi and I will give some tips on getting RL working if you're a builder and uh then I'll give a quick introduction to Cambrian Network which is my current focus and just for some context setting this is a research oriented talk and uh I'm trying to distill a lot of the knowledge that I've gained over the years on this topic uh for builders in this talk so um I started my career hacking cryptography Hardware I did that for 5 years and then I went and worked on a PhD in reinforcement learning specifically High high-speed RL or low power RL um at UC Santa Barbara I finished that in 2019 and then I co-founded semiotic Labs uh and uh semiotic Labs is a cordev team in the graph we are focused on or they are focused on AI and verifiability for the graph semiotic also is the parent company of odos which is a leading Dex aggregator uh they they processed about $350 million uh in Trans in transactions yesterday um and uh semiotic is also the parent company of my new project and also if you're taking pictures I have a QR code at the end of the uh deck and you can uh you can get all the slides if if you'd prefer to do that then so reinforcement learning is a branch of AI that is focused on learning through trial and error it's um the um the inspiration for RL comes from uh behavioral psychology and it is the basically it's the way that all animals including humans learn everything and so what we've done in RL the computer science uh implementation of RL is take models of animal behavior and basically uh animal learning and map it to implement it in algorithms and with RL you you can pretty you can use RL in basically two two environments you can either use simulation where your agent can learn things in simulation and then you can map that simulation or what you learned in simulation to the real world or you can actually learn online in the real world or your agent can and as a note uh you've heard of RL lhf when uh GPT came out rhf is not reinforcement learning it is a type of supervised learning where a human gives feedback to the uh output of the of uh the llm when whenever you start getting into RL you'll see a diagram like this where basically what's happening is you have an agent in this case this little robot spider uh that is making actions based on what's what we call a policy so a policy just means a decision-making function uh so an agent has a policy you can think of it as its brain and that policy is what determines what actions the agent is going to take in its environment and so you have an agent that is interacting in some sort of environment taking actions and what happens is the environment will change whenever the agent makes actions and those changes get output as what we call States and most importantly uh the environment will also uh put rewards and the goal of the agent is to learn to take actions that can maximize its collection of rewards so this is an example of uh a recent example of using RL um to train this robot that's riding a bicycle this came out about two weeks ago this the policy for this uh robot was trained in simulation using reinforcement learning and then then after it learned how to perform these tricks in simulation they just copied the neural network or the policy that was learned into the into the actual physical robot and the interesting thing about RL is that it can learn behaviors that are at expert human level or Beyond human level and that's really the excite for me that's the the reason that reinforcement learning is really exciting is because it can go beyond just imitation this is a yeah so you see you saw those sorts of Tricks those are very really difficult for a human to do and really difficult for an engineer to hardcode uh into a robot so um I like to when I think about what reinforcement learning is doing at more of a like a technical level I think about things in terms of trajectories so if you think of uh of uh what What's Happening for decision-making tasks you often have to do thing many different things over a long time Horizon to get to a goal for example uh let's say where you are in life right now is the result of many decisions some good and some bad um and the idea with reinforcement learning is to basically pick decisions over a long time Horizon that can lead to an optimal outcome um and uh for maybe for some intuition if you think about what's happening with an llm as it's gener generating its its words there are many directions that llm can go there are many different types of sentences you've seen or code that you've seen in llm generate and you can think of those as trajectories each token that the llm outputs you could think of it as a decision working towards some sort of long-term goal and uh in general RL is uh used to try to find the best sequences of of actions to get to the best long-term result and in finance you could think of uh of what's Happening Finance it's not just a single decision that you're making in in any sort of financial transaction really you're working towards some long-term goal even if you buy some buy an asset eventually you have to sell it um and so it's a sequence of actions even just with a simple transaction it's it's not just one one thing and of course with Finance you're looking at maximizing long-term gains which often could involve hundreds or thousands of actions over many many years and so I'm interested in RL to basically maximize uh those sorts of actions um one more RL example uh and this goes back to kind this goes back to the fact that RL can lead to things Beyond imitation learning remember when deep seek came out the big thing about deep seek was that it was the first public use of reinforcement learning uh for training a llm and the interesting thing is that previously when whenever these llms were initially created they were just trained on copies of human text they were just trained to mimic all of the internet all everything that people said on the internet but what happened during the training of of deep seek and this is actually a screenshot from the deep seek paper is that Midway through training with reinforcement learning the agent actually learned to identify the errors that it was making in its uh in its thinking and then readjust itself on the fly during training and if you think about what you what you would find out on the internet you don't really see uh you don't really see people thinking through their thoughts correcting themselves and then continuing because normally what people post is their finished edited product and so you're seeing this sort of uh emerging Behavior emergent behavior that uh happened because of of RL and it's that red text right there where that that was the first time the model started actually like doing something that wasn't just in the training data that was uh used to warm start it so why does RL map well to finance well like I said Finance is a inherently a sequence of decision-making uh yeah it's a sequence of decisions that are made you are optimizing over long-term rewards and with RL you have what's called this exploration exploitation problem and in general uh sorry in finance you have this exploration exploitation problem and in general RL is really good at at optimizing problems that have where you have to make a decision or your agent has to make a decision between exploring and exploiting and so in finance an example would be exploit an example of exploitation would be saying that you you know treasury bills offer a certain amount of yield so you could exploit that fact to get that little guaranteed amount of yield over some you know time Horizon whereas exploration might be saying well I know there are other assets that might have higher volatility but they could have higher yield so maybe an agent could decide instead of just exploiting this low guaranteed yield option it could go and explore and try to find other types of yield also RLS is uh good for working under shifting Dynamics so as you know markets constantly shifting regulations are changing new tokens are popping up are disappearing uh companies are doing better or doing worse and so with in finance you don't just want you wouldn't just use some fixed strategy usually you would want your strategy to adapt over time so for the next few minutes I'm going to go into some examples of where uh RL is being applied in in finance applications and at a at a high level um we can see I'm just going to use three example uh classes of problems where uh people especially academics have been talking about using RL uh one is for algorithmic and high trading frequency uh sorry high frequency trading Bots the other one is for robo advisors and automated portfolio management and the the third class that I'll talk about is agent-based modeling uh but before I go into those um to those example domains for a moment just like to talk about like who's using RL in finance today well at at a high level uh financial institutions aren't eager to share the details of their strategies because when they share those details they're basically giving away Alpha they're giving away their gains the more people that are using a strategy the less valuable the strategy is um so but we can we can assume that at least hedge funds have at least experimented with these techniques these guys are hiring lots of computer scientists and mathem policians who understand these techniques really well um and we know that uh but hedge funds aren't really talking about they're not going to talk about this stuff because they're not going to give away their Alpha uh Banks and asset man managers are publicly exploring RL and they're publishing about it and they're talking about their results um they're not really trying to protect Alpha they're trying to do things like um protect basically get better orders executed and uh manage their risk so they don't really mind sharing the details as much because often times M these sorts of Institutions they're public about their strategy anyways and then finally I've been talking to some crypto projects that are starting to use RL for defi so let's look at some examples of the algorithm and high frequency trading uh problem space so these are General problems that exist today people have solutions for this today but are L RL is an interesting approach to consider for these problem domains uh let's we'll start by looking at Market making the problem Market making is basically deciding to a market maker basically buys assets at a a slightly uh lower price than the current spot price and they sell those assets at just a little bit above the spot price and they make a they make a profit on What's called the bid ask spread and it is basically risky for them to hold these assets as the asset as the spot price is Shifting around and so they have to decide when to buy when to sell how much to buy how much to sell um and at what prices and so this Maps nicely to RL because they're not just looking for a single successful Buy sell decision they're trying to maximize their profits while engaging in these uh in these pools over a long time Horizon also Arbitrage Bots Arbitrage Bots are looking for assets that can be purchased cheaper in some uh in some market and sold at a profit in a higher market and there are there's uh execution risk and there is competition for doing Arbitrage because it's free profit and it's highly competitive and it's something that you would want to be automated and uh finally this uh Trend Trend following and mean reversion idea is you have assets that follow other assets for example Bitcoin follows a stock market and ethereum and all other assets often follow Bitcoin so taking advantage of these statistical properties is something that an RL agent could do so the second the second class of problems is liquidity provision and yield yield farming uh automated liquidity so you could build you could use R RL to build automated liquidity providers uh this maps to um dexes so with dexes like Unis swap you basically have liquidity providers that put in uh pairs of assets let's say eth and some other uh some other token and the risk here is that when one of the tokens is pumping with respect to the other token then what happens is everybody sells the token that is everybody's buying the token that's pumping and it Le leaves the liquidity providers with the less valuable token but liquidity providers also get trading fees uh for providing liquidity and so there's this uh complex decision process on how much liquidity to provide what assets to provide it in when to remove the liquidity which would be based on you know the market and your belief about how the Market's going to move and this is yeah so RL could be used here and then we have yield aggregation so you know you've seen things like um th% yield well then how do you you off yield is a common uh instrument in crypto how do you determine which tokens that you would try to pursue for yield how do you anal Anze the risk of those tokens some of that might be sentiment based some of might be team based who's behind the project and using Ai and uh automated decisions you could you could basically collect a lot of the information you would need to manage you manage your risk and go after yield opportunities and finally simulation is a is a big unlock for reinforcement learning if you can simulate your environment accurately then that this is like the the holy grail for for RL is ability to to train agents in a simulated environment and as a note here um and and why would you want to do that one you could train uh agents to participate for profit in protocols two if you're a protocol Builder you could test your cryptoeconomics by having both Good and Bad actors trying to you know behave uh honestly and behave with extractive behavior in your protocol to see if you're see how your protocol holds up and um yeah so these are the these are some three example uh applications of RL and I think right now they there're only some people are starting to apply RL in defi but it's a very it's very early uh and I think there are a lot of opportunities for Builders uh with a financial background to start experimenting in the space and uh yeah so I have yeah I have three more three more minutes so with um now I'm going to go over some challenges with RL we have especially in in finance and defi we have Behavior sentiment and uh lots of risk that's constantly shifting around and so your agent an agent needs to be able to deal with all of these these changing conditions you also have delayed rewards so the longer the longer your reward is Away In Time the more difficult it is to to train an agent or a policy that can that can go after that reward that's way up potentially way out in time we have a lot of times these live markets are are difficult to simulate and so yeah it can be hard to it can be hard to actually capture the the the real world in a simulated environment and so you your agent might have to actually learn in the real world which in a financial setting could have a lot of risk it's also difficult to answer what if scenarios with RL or in general when you're doing simulations so for example if let's say you're you're dealing with some uh a low market cap pool then if your agent did something it it could be difficult to test in simulation how the actual Market would respond and reality so these sort of what F questions can be hard to test also explainability is challenging with RL especially with neural networks it can be difficult to tell what a neural network is doing and why it did something and this by the way is probably one of the reasons that RL hasn't been widely adopted in finances because of explainability and finally we have regulatory risk for active management of users funds so some tips for solving problems with RL if you have human training examples you should use that you should start with supervised learning uh to get your hands on on the data and get experience with the data you should um try classical approaches first to become become familiar with like how you would solve the problem generally and then I recommend checking out some classic RL algorithms and getting experience with that and then you can get into more then you can approach the the RL solution finally with my 30 seconds left Cambrian Network what we're doing is basically we're providing high-speed financial data to uh defi agents and so if you're building anything that uh anything agentic with Finance i' I'd love to talk with you and you can uh scan this QR code for these slides and you can connect with me on Twitter and I'm running a study group on applications of Finance to reinforcement learning if you're interested in joining thank [Music] you thank you so much um if there are any questions please let the chat know and I can definitely moderate it to him and thank you so much for speaking uh I might have one or two questions um out of all the projects you've seen what has been like your favorite Challenge and solution any any interesting use cases that you really see to enjoy well um so applying RL to finance I can't answer in the finance context because it's it's so new um there's not a lot of other than the academic results there's there's not a lot that is exciting to me but uh Alpha go was interesting the game of when ARL was used a game to to beat the game of Go why I thought that was so cool is go is a 5,000 year old game and RL and many strategies have been studied for 5,000 you know 5,000 years and RL found new strategies that experts had never discovered before and now experts study RL and incorporate or and study the The Go Solution for their for their own use okay great some sounds great thank you so much okay