Mastering the Mempool: Wander Through the Weird & Wonderful World of In-Flight Tx's – Matt Cutler

hey good morning welcome this is mastering the Memel we're gonna spend 25 minutes going deep into a relatively unexplored part of the blockchain and there's a bunch of new data a bunch of new stuff we've never published before that we're gonna be happy to share and I'll be hopefully have some time to take questions at the very end so we like to make this comparison that traditional transactions whether you're depositing a check or swiping a credit card or doing an equities trade or shipping a package run like clockwork they're pretty confidence-building you know what's going on and like you could take your Swiss watch and turn it over and you can see the movement inside and while there's a lot of complexity there it's it's pretty well organized and it gives you confidence that everything was gonna work according to plan but but how many of you have ever done a blockchain transaction before okay yeah we're a teeth Denver it's kind of more like checking your luggage at the airport you you hand the transaction over and you have that sort of uneasy feeling like I hope this is gonna go okay and then the person who gets it just sort of throws it over their shoulder into some maze of complexity and then you wait on the other end refresh refresh refresh refresh refresh and most of the time your bag comes out or the transaction in your life I'm happy that happens smoothly but when it doesn't come out you're like you know I got bit by this and then you go and you talk to someone like hey where's my bag and they go we don't know your bag is it's in the system somewhere right and you know if you are if you're on vacation and you got to go get a new swimsuit it's inconvenient but if you're going for a business meeting or something else it could be catastrophic right and therefore theorem transaction anxiety is a real thing and this is not something which is just for new users in fact in our research what we find is the most experienced users with the most data but the most at stake have the biggest issues do you guys need something on the the video feed here is there an issue I'm gonna do something here I'm gonna unplug and replug I spent so much time on these gorgeous slides let's hope that they show up there we go um transaction anxiety is real and you know what it's okay for us to talk about it this is this is a safe space okay we're at ease Denver it's a safe space and we're gonna come back to this theme a lot because the notion of trains at transaction anxiety in our ecosystem does not get anywhere near enough airtime and as I talk to people as I talk to investors this is a very real and deeply problematic aspect of our ecosystem people are not going to use things that induce anxiety of them particularly when there's value in motion now how big of a problem is this well interestingly we did some research in combination with coin metrics and you can see all this stuff you're not going to read it all but believe it or not since the beginning of time since the advent of the Genesis block on Bitcoin there have been 3.1 billion transactions in aggregate of which 1 point what are just over 1.1 billion occurred in 2019 alone or fully 37% of all blockchain transactions across all public chains have occurred about in the last 12 months we're growing really fast and an aggregate this represents four point six trillion dollars of value in motion we spend so much time talking about stuff on chain on shame by definition is value at rest it's settled it's confirmed it's locked right it's not changing the most interesting part of this we think are the most interesting opportunities are in the value in motion and the value in motion is contained by definition in the memoir okay and this notion of we have this cache of transactions called the mempool but it works in this very specific and somewhat unusual way not very well-documented way not very well explored way sort of changes everything and that's what we're here to talk about right now mastering the mempool now before we get started what the hell is the mempool right now believe it or not there's shocking amounts of controversy on this if you do a Wikipedia search on the mem goal this is what you get 404 not found there is no mempool entry in Wikipedia we have submitted but it hasn't been approved right and if you google a theory of mempool you will find effectively not very satisfying results now we just have this definition shared staging area in front of a blockchain that enables transaction or during transaction fee prioritization and general block construction right except if you go to guess it's not called the mempool at all it's called the transaction pool and if you've got a parity it's not called the mem pool at all it's called the transaction queue so we can't even agree about the vocabulary with which to articulate this critical area of our infrastructure now we go across the industry Bitcoin calls it the men pool and Libra calls it the men pool and tees those calls it the men pool and pretty much everybody else calls it the men pool so for our purposes and for the purposes of people understanding what the hell this stuff is we're gonna call it the mempool okay we're going to acknowledge that different implementations have different terms for it but we're not here to endorse specific implementations we're here to talk about important concepts to the ecosystem and remember this is a safe space okay we're here at eath Denver to save space okay okay so now mastering the mempool there's another word that requires definition and it's not the word that you expect it's the because believe it or not there's no such thing as the mem cool there is no canonical mempool there's no master reference mempool instead there are many in fact every single node has its own unique instance of the mempool and if you could read this it would show that there is about eight 8650 etherion ohms nodes online as of last night I think is what I took the screenshot and every single one has its own unique set of transactions in that which is determined by a variety of factors including its peering relationships its network topology the the local peering relationships interesting factors like the Great Firewall of China the distance between nodes and also the individual settings of each node itself so guess this is the default configuration of a guest node contains 4096 mempool slots and 4 gigabytes of memory what that means actually the 4 gigabyte of memory is is shared so you actually get less than 4 gigabytes of memory is that if you have 4096 transactions in your pending mempool and your node receives a 4097 you drop transactions off the back okay and there's all sorts of interesting little settings you can do in here not so little and that's a little that determine which transactions your local copy of the mempool happens to accept at any given time believe it or not parity has doubled the nodes but double the slots by default the same amount of memory and different rules for what governs its queue so again the tools that you're using to look at a mental matter quite a bit okay different nodes different settings different data and as a result of this mem pulls often miss transactions now what's a miss transaction it's a transaction that gets written to chain that's confirmed in a block that you're a local couple copy of the mempool never saw it literally pops into existence from your perspective and it's confirmed and then it went into no intermediate state now that's not what actually happened with the the chain with the transaction what actually happened is your copy that mempool was looking at a different set of transactions now this data is from our own internal network this is a node this is a cluster of three different nodes running together at the same time and we basically do this analysis as part of what we as part of our core operations and what we find is for a single node in a single geography and a cluster configuration you're gonna miss anywhere from four to eight percent of all transactions okay and so transactions miss the men pool is the percentage over here it can fluctuate and vary right a volume of ten stars actually sorry I said that backwards the orange is the volume the blue is it's less than that but at times you'll see these spikes where you're gonna miss stuff now again what does it matter it only matters if the transaction you care about is the one that's missed there's a whole bunch of work that goes into ensuring that you can get full access to all transactions furthermore while transactions are on chain are immutable everything in the men pool can be overwritten so you can overwrite a pending transaction with a replacement transaction commonly in our space called the speed-up or a cancel and and this chart right here again you can't read it shows the relative percentage of speed-up and canceled transactions in the global mental editing time and so most of the time the oranges cancels it's like a tenth of a percent but at certain times it spikes up to like eight or ten percent same thing with speed ups and we watch this behavior we watch BOTS go to battle with each other to get confirmed on a block and they start ratcheting it up each others the the gas price associated or the gas associated with each transaction and to do a speed upper cancel you have to replace a transaction that's currently pending with an identical nonce from the same wallet with at least 10% more gas by default okay and so you have this ratcheting behavior where the gas price starts to escalate quite quickly until one of the algorithms nopes out and someone gets that transaction put it and we see these battles rage they spike in existence that they spike down and this creates all sorts of interesting congestion factors in the mempool we'll talk about that in a second okay so remember there is no canonical mempool transaction anxiety is real our belief our assertion is the route of transaction anxiety is the complexity of this underlying technology and because this underlying data is not well characterized not well understood not well leveraged by builders it results in that end-user and enterprise experience which is like sort of uneasy we cost transactions in its value in motion and we cross our fingers and this is problematic okay remember we're in a safe space okay so let's get into the meat of it mastering the mempool and this is a real simple day in the life of a transaction it gets signed it gets submitted you wait for a while hopefully it gets confirmed well this is what a transaction in the mempool actually looks like this is a mental entry so you can see at the top the transaction hash you can see a from address the gas the gas price again the hash the input which is just a call in the nonce which is the number 8 up there this is the hex of the nonce to value and that's sort of it it's quite sparse and it's quite compact by design this is actually derived from the architecture of the Bitcoin mempool and they were very concerned about keeping everything as small and compact as possible now what actually happens in our world in the world of aetherium is there's all this innovation happening to make transactions faster to make transactions cheaper to make transactions easier to make transactions we're private and what they do is they introduce massive amounts of complexity into the nature of transactions and how they're constructed so now you have all of these factors both happening at layer 1 and at layer 2 with smart contract based wallets and real layers you have layer two state channels and payment channels they can get thrown into a mempool that the other wait and you can replace replace them with a speed-up or cancel and they can have any one of a number of different states or outcomes some of which are not very well characterized and you got a lot of transactions pending at any given time so this is a screenshot I took from ether skin they have a running graph of this is mempool pending transactions by minute this is from last night and this scale is in tens of thousands so 10,000 20,000 30,000 40,000 50,000 and you can see these massive jumps and then these drops right as transactions surge in and out now keep in mind that at any given moment there are tens of thousands to hundreds of thousands of pending transactions oh by the way each transaction can result in multiple internal transactions again you can't read this but the light blue line is main that transactions the dark above it is resulting internal transactions and at many moments the number of internal transactions can be two to five X the volume of maintenance actions so when we think about what transactions are actually happening what's going on here you have to pay attention to both the external transactions and the internal transactions but for perspective this is from a blocked chair this is the average number of transactions in a block its measured in hundreds okay so I have tens of thousands of pending transactions competing with each other and this mempool where you have the surge activity sloshing in and out to get into one of anywhere from 50 to 150 transactions maybe a few hundred more transactions per block right and by the way these get written out on a probabilistic basis about every 15 seconds so in the mempool you have continuous flow data sloshing in and out coming constantly and every 15 seconds give or take you get the top of the men pool skimmed off theoretically and this results in all sorts of interesting behavior this is a transaction which we noticed last night it the gas estimate for this as this should complete about 1 hour and 44 minutes it's been pending for almost five days this is probably still pending right now there's a six thousand and six hundred and fifty percent discrepancy between the predicted performance of this transaction and the actual performance of this transaction okay now does it matter at all depends are you buying a house is this part of a whole chain of transactions which is expected to complete or is it some sort of test even there's no way to know and as we drive mainstream adoption of these technologies as we encourage more and more users to do more and more stuff these sorts of incidents become a bigger and bigger headache for the ecosystem okay real-time gas estimation I won't talk too much about this is an open loop we hear about this all the time it is an incredibly hard problem for the protocols and the daps out there to accurately give guidance on what a given transaction gas price should be given current real time mental considerations this is anxiety is real again we're in a safe place but now what's nice about this is this mempool data can be enriched you can actually stretch all of the information out that's contained and you can get all sorts of useful things that's useful for you as a depth developer it's useful for the protocol manufacturers to mint to figure out what's going on with their transactions it's useful for the end-users themselves so the sparse amount of data which is associated with a medical transaction can be expanded to include not just from an - but counterparties can include the ABI for the individual function calls can include the internal transactions themselves these can then be organized into live data feeds this is an example of the 0x protocol data feed and this transaction visibility can begin to yield transaction confidence transaction confidence for you the Builder transaction confidence for the people leveraging the tools that you're building in particular we're seeing a lot of people using real-time transaction feeds during their development process do you have much faster feedback loops to debug their smart contracts to debug their doubts right this is an example again you can't read it of a transaction that failed after eighty-four seconds right why did it fail what's going on and we do a lot of work with our customers to help them understand why the resulting transaction behavior occurred and what changes they can do to improve things okay we can take this data you can use this data and you can improve the user experience so rather than expecting the user to pull and pull and search things and understand what's going on in ether scan you can just push the information to them and let them know what's going on in their transactions and we're powering push notifications for while it's like pillar and I do to tell you as an end user not just when you're sending a value but when value is coming in and being received interestingly we're helping defy protocols benchmark real world transaction performance so you said your defy protocol in the wild people are transacting with it how long do those transactions take the reality is they don't know they have no way to monitor this is synthetic data that we have you know made up but this is reflective of real-world work we had one of the major d5 protocols approached us and say we have a target of 30 seconds or less to confirm a transaction but we don't know how long that actually takes we help them understand that fully 26% of all transactions against their smart contract we're falling below their benchmark so what this was more than ten million dollars of in-flight transactions that were performing below what they expected super i opening for them as a protocol and now they have visibility and can measure if they can work on improving this they can work on improving this by making their their protocol more efficient they can work on their gas price estimation and they can monitor this over time in the early days of the internet everyone was so focused on making websites that work and then people realized faster websites perform better than slower websites and everyone's harder to focus on performance so this data can be used in this regard as well so in closing just key takeaways and recommendations look in flight transactions represent value in motion which is among the most interesting aspects of our ecosystem but this anxiety is holding us back holding us back and and the secret is in the mempool you can leverage live mental data today to build faster improve your user experience benchmark your protocol performance and we believe trigger real-time events so based on things your getting off of these data feeds you can trigger live dynamic interactive experiences for your end-users and for your infrastructure and it doesn't need to be difficult now I share all of this of course because this is what we do here at block native you can get going with our API in literally less than five minutes you can be receiving live data and the way that we described in less than five minutes what should happen here there's supposed to be a QR code that I put on the slide that supposed to be part of a game but we never got it so there's a game come find me later if you want to scan and hopefully we will get it but that was mastering the mempool I'm not cut there from blocking in it but I don't have any time are we good I think we're good okay we'll ask some questions awesome thank you guys any questions sure yes yeah so yeah what we do is we provide we solve a lot of the challenges that you just saw here it requires a lot of infrastructure and a lot of operational expense so you can work with us and get it for free or by the SIP as opposed to you know having to build all this infrastructure yourself yeah yes oh it's a very good question so the bidding wars that happened in between BOTS basically the miners are incentive to prioritize transactions of higher gas price right and so the bots are basically competing to get more attention of the miners by increasing the fees associated with their transaction they're not competing necessarily for the miners attention they're competing with each other right because they're trying to be the first one into the next block and there's certain economic situations that they're looking at by the way powered by a lot of this similar data to try to get economic advantage for doing so there's a bunch of information asymmetries that are happening in the space part of what we're trying to do is eliminate that we're trying to democratize access to this information because we think it's super valuable for the whole ecosystem have we seen any evidence of miners prioritizing differently it's all probabilistic so this is one of the big challenges it's not a rule set so you're gonna put it in and you're gonna hope that the right thing happens and all I'll say is we see weird behaviors weird transactions unexpected results things that have occurred but we can't explain why they occurred all the time right so as you really get into this data set as you really start looking there's a lot of anomalous behavior that's happening and we're trying to surface this as rapidly as we can we blog about this stuff quite a bit so if you're curious to learn more about speed ups or cancels here is to learn more about various aspects of the men pole by all means check out a blog blog native calm we're just beginning to start publishing some of this data so I always say we have a vast amount of data that we ourselves are just beginning to scratch the surface of in fact what we find is we don't have all the answers it's by working with members of the ecosystem that the interesting questions start to surface up so we're trying to make it as easy as possible to get into and start manipulating this data we don't publish it yet at a macro archive letter level but that's something that we plan to do as a mental Explorer like functionality but again we're early in the cycle and we want to invite everybody to come in and check it out okay awesome thanks so much guys have a gradient Enver it's a safe space [Music] you