Mempool Primitives: A New Class of Money Legos - Matt Cutler

so we're going to move on to uh block native matt cutler so matt are you here let's see i am here he is matt good to see you hey john great to see you as well good to be back yeah good to be back good to have you back so we're going to talk about some new legos it sounds indeed rock and roll here we go so uh happy eath denver 2021 everyone i'm matt cutler i'm founder and ceo from block native we've been building in the ethereum space since 2018 but we actually launched the company uh back at east denver 2019 and so east denver holds a special place in our heart and it's funny all the stuff that we're doing right now you could look at and see it back in 2019 if you squinted and imagined but we're doing it at just a much greater scale and on with with the best companies the best projects out there so it's super fun i'm going to spend the next 30 minutes or so talking about mempool primitives um mempool is sort of front and center right now with uh gas prices being what they are and everything the hacks that happened this week um and just there's a lot going on there but mempool data is exceptionally hard to work with and so we're going to talk about uh how to work with it why it matters and what you can do and i'm going to go hands-on and show a bunch of stuff for all the builders out there who want to build on top of memple data so um you know we're gonna talk about mempool and primitives but just to level set like let's talk about them each individually so what's a primitive um they're basically just building blocks and the whole secret here is uh ethereum is basically built on this notion of composability you can take the various lego bricks put them together create a new flavor sensation um and and they enable your developer and builder expressiveness your creativity um now what we sort of really focus on is mempool data making it easy and accessible to work on we're going to talk a bunch about that so lego bricks here so so the idea here is early dapps were like a lego house they were pretty crude but as the ecosystem has evolved and more and more primitive types i.e more and more lego bricks enable much more sophisticated much more polished much more approachable experiences for every level of the ecosystem so we believe that focusing on mempool primitives basically allow all builders and all traders in the space to to work with mempool data equally and um and directly so let's talk a little bit about mempool and and what's going on here now while a lot of folks are aware of mempool as a concept i find myself in conversations where folks sort of need to think more deeply about it so we talk about the mempool as the pre-consensus layer it's the place where transactions go before consensus can be reached which has all sorts of very profound implications so first off the mempool or the pre-consensus layer impacts everything right in fact impacts every transactions every protocol every dap every builder every trader every user everybody deals with it but it is largely uh uh uh opaque and and invisible right um what's going on in the mempool determines the transaction fee or gas price the order of transactions the timing of transactions when they get confirmed any slippage that may be a part of it and the terms of settlement so overall if we look at this like it really matters to the participants to the protocols to the transactions to the financial results of them and yet it remains a relatively um obscure or or not very common uh piece of infrastructure to work on and our objective here is to fix all of that so uh but the problem is it's kind of messy right so this whole world is super opaque uh because it's pre-consensus there's no truth like there's no what is the canonical version of the mempool what's actually going on uh you don't really know turns out that there's no such thing as one mempool that every node in the network has its own unique copy of the mempool and those nodes and their contents can vary wildly particularly under periods like we're going through right now of high congestion so you could have a node or even a set of nodes and you could look in their mempools and not actually have good visibility into what's going on in the network or what's going on with transactions that you care about uh blockchains and and ethereum of course in particular are these uh discrete systems they click forward block by block but the mempool is a sub second streaming system literally transactions stream in in real time at every second so the mechanics of the data are quite a bit different than traditional on-chain data hey guess what because it's pre-consensus it's mutable so you can change transactions while they're in the mempool that has all sorts of implications for how transactions behave and user experience it's probabilistic so there's nothing guaranteed in this world it's all a matter of probabilities about what's going to happen and with as many transactions are happening and enough time all sorts of low probability events will happen in the mempool which we've seen and the data that's there is really quite sparse so if you look at the actual contents of impulse doesn't tell you very much and so you got to do a bunch of work as a builder to try to extract any useful meaning out of it and do anything with that now it's also inhabited and this has again come to the four in recent months but the mempool is not this benign place there because it's mutable and it's programmable it's a highly adversarial environment so while you're doing your transactions there are all sorts of automated systems that are operating in the mempool looking at mempool data to try to figure out if they can extract value from your transactions if they can extract value because they know something a little bit before everybody else does and so uh this is of course this notion of ethereum as a dark forest this is sort of the the paper the the blog post that coined it all by dan robinson and and georgios i totally recommend reading this if you haven't it's a fascinating read and this this meme of ethereum as a dark forest with apex predators uh occupying it has now carried through and there's many many blog posts out there now we at block native exist to fix this we exist to build global infrastructure to make the mempool understandable kind of like what we're doing right now to make it transparent so everybody knows what's going on it to make it accessible so people have equal access to the data so it's not this weird asymmetry that's happening today to normalize the data so there actually is a consensus via the quote unquote but like a common view of what's actually happening in the mempool what matters to make it programmable and easy to work with so you as a builder don't need to reinvent the wheel every time you want to do something to make these programmable elements composable so you can use lego bricks you can use these mempool lego bricks with all the other stuff that you're doing to deliver next generation experience for users to deliver more secure protocols to deliver new and more innovative trading strategies to basically take advantage of uh this massive opaque weird data set in the experiences that we deliver to um to the ethereum users and that we leverage as as uh as participants in the ecosystem so at the end of the day we here at block native are all about making the mempool you know sort of a buildable component of the ecosystem just like everything else on equal footing and unfortunately it's not there right now but we're making pretty rapid progress and we're pretty excited to sort of share that so let's talk about building with mempool data and let's sort of give some examples of of some of these primitives that we're pioneering and how you as a builder can incorporate them into your hacks now at eat denver but also into everything that you're going to be building moving forward as you participate in the ecosystem so what i'm going to go through today are sort of four primitives and the idea here is it's kind of crawl walk run where we're going to start with some simple ones to start working with just to get a feel for it and then we're going to build up to more sophisticated concepts now to do so i'm going to use uh tool sets that we offer at block native uh one of the things that we've really focused on is to to make this stuff much more um tangible so you can look at mempool data you can work with mempledata in a very um visceral way and not such an abstract way we have found you know earlier in our in our life cycle we had build our data platform we had apis we would talk to people about it and they'd sort of knob their head but it wasn't until we we showed folks how to go hands-on with the data before the the light bulb started to pop and so um we're working very hard on these capabilities and there's more coming out uh pretty soon but that's what we'll be using just to sort of illustrate a lot of these concepts now the four primitives i'm going to go through is hot wallet monitoring which is a one-on-one thing any user out there can do this monitor your own hot wallet and then two we're going to generalize that into a smart contract transaction streaming we're going to monitor a ethereum smart contract live on mainnet right here it's going to be this little teeny tiny smart contract called uniswap v2 router which of course everyone knows and is the most popular and widely deployed widely used smart contract in the ecosystem then we're going to show how to generalize that into providing user transaction feedback for anybody who's using adapt believe it or not a user transaction feedback in our ecosystem is uh well behind what it should be and and many of the top many dap experiences but many of the top decks in the world actually lie to their users about transactions because they're unable to handle replacement transactions cleanly and because we're working in the mempool we make short work of that we'll show a little bit of that and then finally the most sophisticated one rather a new take on how to do gas estimation which is what we call block prediction so trying to predict what's likely to go into the next block and then based on the gas prices of those transactions going into the next block to inform you as a builder to form you as a trader um how to then appropriately set the gas fees on your own transactions and we'll talk through all of that so okay with that what i'm going to do is is jump right in so the first thing we're going to go through is hot wallet monitoring right so the basic idea here is use the mempool to detect inbound or outbound activity to your hot wallet um and and it's pretty interesting because of course we're going to do this when we're doing our own transactions but this technique will monitor and tell you whenever tokens are airdropped to you so right now airdrops are happening quite commonly people are asking have i received it you're sort of hunting around these techniques just by hot wallet monitoring in the mempool will just give you a push alert and say oh hey i just got some um uh some some airdrop tokens awesome right and then also it's it's kind of peace of mind so uh when you're out there yield farming or or doing swaps right you know exactly what's happening with every transaction and you know you have peace of mind that someone's not doing something unauthorized because if they are you'll get notifications of so and so the whole idea of this is this notion of reducing transaction anxiety we all know that feeling of you submit a transaction and you get that uneasy feeling in your stomach about what's going to happen is it going to go through am i going to get a stuck wallet and and we thought at the beginning of all of this transaction anxiety was something that was for noobs right what we've learned is transaction anxiety is universal but but actually some of the most experienced traders i know the most experienced builders who have the most value at play have the greatest anxiety because they know all the ways that things can go wrong and this one little technique hot wallet monitoring basically just just lowers your blood pressure lowers your anxiety anytime you're transacting so so let's get into this now i'm just going to go to our home page it's sort of the easy place to start um blocknandof.com and there's this big button right here called start for free and if you click that you'll pop over to what we call mempool explorer so mempool explorer is this visual front end for working with mempool data and the basic idea is pretty simple uh you create things you're subscribing to over here and then you get real-time results off of the mempool over here um we're gonna create a new subscription called um matt's hot wallet oops right and then i gotta give it an address now it could be any smart contract address or wallet address but i'm just going to go over here to my meta mask which is by the way on rinkeby i'm going to copy that and i'm going to paste it here i'm going to hit create and hocus pocus just like that we are now live monitoring oops i gotta switch this over to rinkeby the rink could be test net for any transactions involving my hot wallet now of course there's no transactions but we can go ahead and we can create one so i'm gonna pop over here to metamask and i'm going to send myself uh 0.1 eth and now when i hit confirm a whole series of interesting things are going to happen so this is stock metamask it's broadcasting transactions via infuria and and a whole sequence of events happens quite rapidly so what happens is i'm going to sign my transaction i'm going to broadcast it to an infuria it's going to hit a an inferior gateway it's going to inferior is going to inspect that transaction make sure it's valid if it's valid and there's room in that gateways mempool it will insert it into them into its mempool it will then broadcast that transaction to its peers over the peer to peer protocol or gossip protocol and that transaction will circulate through the rinkeby test net now we at block native operate this global infrastructure including rinkeby test net nodes and so we're listening right now for any transactions involving my hot wallet address and so we're going to pick that up we're going to see that there's a new transaction we're going to match it to this subscription and then we're going to broadcast an event so it shows up here and all of that happens this quickly so three two one confirm and there's the pending transaction well and there's the confirmed transaction rinkerby is quite fast today and so uh you saw two events come in so the first event is the pending transaction my intent to send and then the confirmed transaction the test that has confirmed it it's now on chain of course in the test net and each one of these fields as properties of that transaction now the mempool itself only contains a few of these the rest are expanded so these are all standard things that you can derive from the contents of a mempool entry but aren't there by default you gotta actually do work to stretch all this stuff out and again one of the things that sort of makes this easier is we do all that on your behalf so it's easier and more addressable to work with now these are json payloads so they're easy to work with and you can then of course um uh basically uh upload these to our to an api and make them composable so what i actually do is um i do that with an api key and then write it out to a web hook that goes to a slack channel and so anytime my hot wallet is involved in a transaction i get messages in slack pushed to my phone on my desktop everywhere i am and i can tell you it's it's pretty um it's pretty nice to have that experience just to know what's going on and just the other day i did a big swap it was 123 in gas fees and just seeing what was going on with that sort of put me at ease while that was happening so anyways that's uh a mempool primitive one how to monitor your own hot wallet the other thing i would recommend is is do what i did like do some test transactions on rinkaby to get a feel for this and um and it will start to make you know start to make sense okay so uh that's mempool primitive one for reducing transaction anxiety now let's generalize it so what we're going to do now is instead of looking at a wallet address we're going to look at a smart contract address so what we're going to get is a real-time data stream of mainnet transactions and and this notion of smart contract transaction streaming is you can have multiple contracts so you could be looking at multiple things at once and then you can filter those down to only get specific events that you care about and the whole idea here is hey you're seeing transactions at their earliest point of origination in the in the network and because you know first you can therefore act first on it so i'm going to go back to my mempool explorer and what i'm going to do is reset it okay so that just brings it back and we're going to go ahead and watch a uniswap whoops v2 router and what i'm going to do i happen to have the uniswap v2 router smart contract address right here and add that and now when i hit create we're going to monitor mainnet for uniswap transactions happening live this is you know live without a net and what do you think is going to happen um we're going to get fire hose so these are actually all of the pending mempool events that are being broadcast and we're detecting right now uh against uh uniswap v2 now you can see this sort of shooting by right um it'll get to 50 and then it will stop so this is what we would call a fire hose it can be pretty hard to work with because as a developer as a builder now i gotta ingest this rapid data stream and i gotta pick through for the things that i want now something worth noting here uh we actually decode the abi on the fly using memplexplorer you can actually upload custom abis so again that's something that helps work with this data we'll talk about that in a second but uniswap is one of the ones that we support out of the box now this this fire hose can be overwhelming and so we want to do is we want to filter it down and so what we allow um is the ability to filter on pretty much any aspect of the payload so for instance let's only look at exception transactions on uniswap v2 so i might go down and say i want to look at the status of each transaction where it is not confirmed right let me save that we can see that right there i'm going to add another filter to the subscription and i'm going to have the status be not pending because that's the typical transactions pending and confirmed and so now what we'll do is just press play over here and now you'll see there's it's a lot less activity we're filtering out all the pieces of it oh here's a stuck transaction um and you'll see these events come in now by the way these there's a lot of data here so we can actually filter out the payload so let's just look at uh status i'm gonna add that and i'm gonna look at hash here as well um there's hash whoops add that and now we can see drop transactions cancel transactions speed up transactions um i'm just going to stop the auto scroll here now uh what's interesting is on major smart contracts like unistop v2 you have a lot of replacement transactions so these are transactions that get broadcast and then a second or more than a second transaction gets rebroadcast with the same knots but with a higher gas price and they get characterized as things like speed up and cancels and because we're monitoring these transactions in the mempool and keeping track of it we actually can identify which transactions are associated with prior transactions and then label them as such this is super powerful because as a builder you generally are not aware of a replacement transaction because you're not party to it the replacement transaction is a function of the user and their wallet not involving the underlying dapper protocol or an automated system as well and so being able to keep track of uh speedups and cancels and other forms of replacement transactions is a very powerful consequence of leveraging mempool primitives while you're building now just uh in this regard all of these hashes are actually hot links into our friends at etherscan so if you want to see the status of a particular transaction we can click on that it'll open it up sometimes that happens it was probably replaced by something else let's see about this one so there we go sometimes these transactions get sped up and replaced so quickly that they don't even show up in ether scan which is pretty interesting but here's one where we just said it was a cancel and it sort of shows the state of that so you can you can see that going on there now um you can do all sorts of other really useful things when you're do using this memphis primitive so what i'm gonna do is i'll uh clear both of this right and now what we can do because we're decoding the abi what i'm going to do is i'm going to filter the subscription on actually specific contract calls so i want to look at the contract call method name matches and then look here's all of the contents of the uniswap v2 router abi so if i only want to know when someone is swapping exact eath for tokens i can put that in there and then i can isolate this out let's remove both of these let's start over here and now here's a pending transaction for and here's more coming in for swapping for exact token you can add filters here for above a certain value or a certain criteria and now you're beginning to of this massive sea of transactions flowing through the mempool able to pick out the ones that are most relevant to your project well it's relevant to your users and do useful things with it through composability so that's a real quick example of uh the second mempool primitive which is a smart contract transaction streaming now the third primitive i want to go into is to sort of generalize this which is user transaction feedback so now that you know you know what's going on with these transactions in the mempool uh you can extend that capability directly out to end users right so you can inform your end users when there's pending transactions you can inform your end users when their transactions are confirmed but you can also inform your users if they get stuck if they get dropped if they get speed up or cancelled and the whole idea here is uh this is kind of a big headache to build on your own and leveraging um this sort of infrastructure it's like short work this is minutes to maybe a couple of hours to do um the whole idea of doing so why would you do this if you provide increased transparency behind transactions to your end users they will do more transaction with you they'll do them more frequently and more recently and at higher values this is all about boosting confidence in the ecosystem and helping people feel comfortable particularly as all these new users are coming on board to the ethereum ecosystem we as builders have an obligation to make that experience as smooth and easy as possible now we offer a library called notify this is block native notify.js is our documentation you can read through here it's actually pretty straightforward it's fully documented obviously but at the top of our documentation we have this little react demo that's sort of just a test harness for what we do here here you can connect your wallet i've already done so but what i'm going to do just to show these transaction notifications is i'm going to send myself a transaction you see these little pop-ups down here so what we're doing is we're sharing with the end user information about what's going on so if i reject this oh i get feedback around that now i can try that again let's just send a transaction you see my meta mask fires and when i hit confirm your transaction has started a timer is going by right now by the way my slack is firing off because this is my own personal hot wallet and then as soon as there could be test net confirms that transaction you'll see that right there and so these little tools down here which by the way you can move around the screen pretty easily it's pretty configurable and you can change the look and feel of them um uh basically just provide information back to the user about the state of their transactions and so you can integrate this stuff directly in your front end and and users benefit and you don't have to worry about it so uh every transaction that gets initiated on your dap your um uh your your your systems can provide a status update this of course is also available via api so you can integrate and leverage this data in all sorts of interesting ways and we encourage you to do so what we hear consistently from uh our customers who deploy these sorts of capabilities is uh there they it increases uh uh positive behavior it increases user satisfaction and it results in and just a lot fewer support requests to them as well so that was uh transaction feedback now the fourth one is gas estimation via block prediction so i'm going to spend just a couple of minutes on this and then wrap up um but basically the expectation of the mempool is kind of like a flash mob like we're all going to bring our nodes together you're going to uh put your transaction we're all going to coordinate together and guess what we're all going to have a good time but in reality uh it's more like a mosh pit that it's a pretty violent place and it can surge in terms of its activities so you ask your friend hey i need to go to the bar and get a drink how's the mosh pit you say well two minutes ago it was mellow and then you go through there and a new band has shown up and you get your nose broken on your way to the bar right and and this is very much the same thing that happens with uh the mempool is it's subject to surge behavior right now by the way with eath at all-time highs the level of activity is quite high and so basically what we're offering here what we're trying to do is look at what what's the likely contents of the next block based on the top of the mempool we're going to then inspect those transaction gas prices and then we're going to set confidence intervals to say hey how confident are you that at a minimum gas price you're going to get into the next block right so the whole idea here is tooling to enable developers and traders to target getting into the next block without grossly overpaying so um to do this is actually pretty easy i'm just going to go back to mempool explorer and reset and you see this little widget right here maybe if i stretch this out a little bit yeah there it is 367 way right now if this is our our gas platform explorer but what i'm going to do is hit this gear and i'm going to select both show detail and show payload and then we're going to actually see uh api let's see here why is that not happening we should be seeing events right here oh i know why i have to press play um so now we're connected see the green dot there uh okay there we go so here are these um gas platform payloads uh by default this updates every five seconds then i'm gonna stop the autoscroll it's gonna keep filling in and what we see here is uh the current block number is four four one um the next block number is four four two uh the our ecosystem the nodes are basically telling us there are 264 transactions projected to be in the next block and at the 99 confidence interval so 99 sure to get into the next block the minimum gas price is 339.

so if you're a builder or trader and you've got to make sure you're transacting to be the next block you need to be at 339 or above to have a 99 confidence interval um of this and if we look at it uh oh in other versions of this it'll tell us the count of those transactions right so here's the 95 the 19 95th confidence interval the 90th confidence interval the 80th the 70th and the 50th 50th so if you want a 50 50 chance of being in the next blog next block minimum gas price at the moment of this payload was 282 so you can see the gas prices are constantly changing as new blocks get created as new transactions come in and these conf the prices associated with the confidence intervals change dynamically and so again this is an api so you can incorporate this capability into your um your your your project and therefore help users have predictable transaction performance without overpaying and this is all powered by real-time mempool data not back test models or anything like that and so there's a whole bunch of really interesting things you can do when you're using real-time mempool data for this purpose so with that i just want to wrap us up you know if we think about on-chain data or blockchain data as the truth the mempool contains all possible future truth it is the future and by looking at mempool data you literally can see seconds or sometimes minutes into the future and this is super powerful so here's a quick me put together obviously from marvel right i looked forward in time and saw 14 million possible futures and how many of them did i win the trade one right so knowing the conditions you need to win the trade or to avoid a hack or to secure your wallet or to protect your protocol that's all data in the mempool and we're trying to make this stuff as easy to use as possible look it's a dark forest out there it's an adversarial environment and like let's bring light to this and let's uh really uh illuminate this stuff so that everybody in the ecosystem is working with the same data everybody has access to the same tooling and that users builders traders alike can take advantage of these capabilities so we very much encourage you to join a lot of the top projects in the ecosystem that are using our libraries and and if you're ready to start building you can start for free at block native look at it yourself there's a ton of content there's a ton of writing out there and we just really want to encourage everybody in the ecosystem to to pay attention to mental data and to incorporate it in thoughtful and meaningful ways and that's all for me i'm matt cutler from block native it's great to be back at east denver and i'll pass it back to john