Real Time Observability Reveals Ethereum's Unexpected Inner Workings | Matt Cutler - Blocknative

uh good afternoon everyone thanks for coming to the Neptune stage I'm Matt Cutler I'm founder and CEO of block native and we're going to spend about the next 20 minutes or so talking about realtime observability in the ethereum ecosystem uh it's pretty interesting the past few sessions that have been on the stage are all about sort of the regulatory environment and this stuff has a lot of implications for the regulatory environment um and this is going to be a session where I'm not just going to show you some stuff I'm going to encourage you to start to experiment on your own so uh uh this is not necessarily going to be a happy tale okay uh there's a graphic novel uh from 1986 which is almost 40 years ago called The Watchman and the Watchman is about it's a dystopian view of a society where they're superheroes who are basically like really powerful but kind of regular people with flaws and they're not just always good all the time they're kind of self-interested and they kind of get out of favor and there's this whole saying about who watches the Watchmen because they kind of have Godlike superpowers but they're not necessarily always used for good and this is a little bit of what's known as foreshadowing for the current state of affairs on the ethereum network because there's a number of superher like entities that have sort of Godlike Powers it's not entirely clear if they're always being used used to everyone's benefit and this is why we need observability so uh we're going to talk about observability some new capabilities new data new Pathways actually do some live demos and then do a little bit on looking ahead uh so what is observability observability is something which actually comes out of the cloud apps world and it's this idea that you can look inside a live system know all the individual states and try to understand what's going on and why they're doing what they're doing so imagine you have a cloud application right you have your own code it's interfacing with other people's code there's some open apis you're using you're using various Cloud uh providers like Google and Amazon and then it slows down and you're like why the hell is my cloud application slow right now you need observability to be able to see inside to see why it's behaving the way that it is and so oh you can't see that there's a cloud there um but like this is the classic uh observability challenge you know why why why oh that's why right so so now we have this new class of uh of application it's called the web three public blockchain it's composed of an incredibly complex array of uh individual Technologies uh abstractions modular systems uh infrastructure Network peer-to-peer layer and observability here is the ability to sort of see inside of that and understand not just what it's doing but why everything is happening the way that it is and this is what we specialize in our our our uh our project block native and what we tend to find is the theory of how this stuff works can be actually quite a bit different from the reality of how it works and that's why observability is so important so we're trying to see like what's happening inside of ethereum and like the nice way the happy version is like it's a watch look at this fancy watch face isn't that clever and we turn it over and like there's this complex CER working but it's it's really neat to see it all work together it's jeweled it's so perfect perfect but the reality is uh more like this that there are uh users paid more than 60 million in fees to settle down to the base layer last year from l2s malicious Ms costing users billions uh ofac violations which we have to deal with each violation carries up to 30 million in fines and up to 30 years in jail and and these are all things you need strong observability to to deal with and manage um and and under the covers there's this whole system called Mev boost which is basically how blocks are produced and this is a observability dashboard called Mev Mev pick sorry Mev boost picks by Tony from the ethereum foundation and it says Mev boost blocks over the past 14 days 90.9% of the of all blocks produced are outsourced to this thing called Mev boost now many of you may not be familiar with Mev boost but it's like where blocks come from and the interesting question is why do nine out of 10 blocks come not from the validators but come from this third- party system this very complicated third party system and the reason there it is is that the the ecosystem of meev boost produces blocks that are five times as valuable as internally generated block so there's a very clear economic incentive if you're a validator or a staking pool to Outsource block production to this U Mev boost ecosystem because you basically make 5x the rewards and then can offer greater py and attract more stake right great economic incentives except 86% of all of those blocks produced in the past uh seven days were built by three Builders three Builders produced 86% of the 90% of blocks so call it 80% of all blocks by the way this is called rated Network what we're what we're looking at here another observability dashboard so that's that's not so decent centralized that 86% of the time you got to talk to one of these three Builders and who are those Builders well now we're looking at relays scan. and we can see right here they're called Beaver build Titan and um rsync Builder also known as wintermute and and over the past seven days they have uh together uh uh uh built more than 86% of the blocks do you know that for your transactions to get on chain you got to work with them what happens if one of these entities doesn't like you or or more likely one of these entities is running a strategy that competes with what you're trying to do not great right and significantly why do they do this why are Builders doing what they're doing this is the Builder profitability of those three Builders over the past seven days and and I know the ma the the it's sort of hard to read here but it's 1,483 eth in profit to the builders to these top three Builders over the past 7 days and the math here is about 1,500 eth about 3350 USD eth about $5 million per week or $260 million per year is being generated in profit from the Builder Network that undergirds how your transactions get onchain okay and so the economics here are pretty interesting there's a lot of money flowing around and the question is what are the builders doing how does this stuff work are is your transaction being treated fairly and again this is why we need observability so when we say who watches the Watchman it's really who watches the settlement okay so we have these Builders who talk to relays who deal with stuff for the public mempool there's private order flow Searchers and validators but there's also like now in tents and bundlers and pay masters and pretty soon we're going to have blob pools and bid pools and swaps and l2s and l1s and censorship and bridges and delays and ofac and these are all realities of the network today all stuff that involves ephemeral data layers there's no permanent record of what's going on all of which contribute and and combine to determine settlement which transactions get included on the chain and in what order okay and so this is a pretty interesting set of challenges if we're trying to build a incredibly neutral foundation for the next economy if we're trying to build something where it's not a two-lane system full of Have and Have Nots dominated by information asymmetries like we have in the traditional Financial system but is basically a Level Playing Field for all the participants and so this is why we have really worked hard on this notion of observability to say hey we need to be able to see what's going on not just really high-end skilled people or really uh experienced people but pretty much anyone in the network should have access to these systems so that you can decide for yourself you know are you being treated fairly and what's going on and so uh We've recently introduced this capability called etherow doxyz and I encourage you to go check that out um it's a new capability made possible by a grant from the ethereum foundation and it's basically a starting point framework that we're building out now and we're going to be inviting the community to build out as well um and there's some terms down here and uh it basically tries to do three things answer the question what's happening on ethereum right now okay which is an interesting problem because if you were to want to know what's going on right now where would you go what is the thing or system or service you look at to see what's going on in real time at this moment and one of the things we discovered was there's not really good resources for the general population or the researcher population and so we're making these available then a real-time transaction Explorer to basically allow anyone to be able to view the public mol um sort of an ether scan like interface but for the real time ephemeral data layers underneath and then all of that written down to a historical data archive measured in terabytes so that researchers can replay historic events and do net new research and so we're going to dive into that so uh uh I'm going to switch over here and this is what's happening on ethereum right now this is etherow XYZ again made possible by a grant from the ethereum foundation and and we can see the current block number or the last block number the amount of eth burned in the last block um the current ethusd exchange rate and the total value staked and then over here of course it's always a live demo see I might need to refresh this or maybe all of y'all are going on the network right now and it's it's slow well there should be a countdown timer over here that's giving the countdown to the next block the gas price uh of the current base fee the number of marketable transactions in the mempool the percent private uh uh in the me pool and then basically these things like the T current value of the eth transfer value in flight and the current uh stable coin value in Flight um the current uh defi swap value and the number of of um optimistic L2 transactions I'm going to try this oh let's see is it going to go I'm on the Wi-Fi come on baby this is always the the way of things whenever you do a live demo you always have a network issue well it's going to be a little less fun then um okay so if we look into this it's more fun if we do it all live luckily I have some stuff uh put up here we can then click through and we can look into see I'm hoping this is going to live it's it's not coming through um we can look into things like the transaction Journey so this is is uh now we're looking into the public mempool and we can look at the the story of individual transactions when it entered the m poool and then it's journey through so sometimes usually they're pending and they're replaced but but often or sorry they're pending and they're confirmed but then they go through and they actually get really interesting when people start to swap them out for new transactions and you're going have these really interesting and complex Journeys that go all the way down to being confirmed um we can look inside uh the batches of of uh optimistic L2 rollups and we can see not just when something like optimism writes down to the L2 to the L1 but we can look inside and see the transactions themselves that make it up so if you're a user of optimism or a developer for optimism you can then programmatically uh be able to show to your users uh basically uh every stage of the transaction Journey from what's happening on optimism itself to its ultimate settlement on ethereum L1 underneath I'm going to go back here I'm trying one more time let's see here it's not going to go okay uh uh but what starts to happen is interesting so we have this notion of public transactions that they get transmitted to the public mempool you can also have private transactions private transactions are transmitted directly to builders and get included in blocks without um ever being broadcast to the peer-to-peer Network and the reason reason why I'd want to do that is you're say a a Searcher who's trying to uh affect ordering so basically you want a certain set of transactions to be included in a certain way or you have a large uh trade that you don't want to have front run or you're participating in an nft drop and you want to make sure you get included there's basically sort of back doors into the network where you can transmit transactions privately to these Builders and they can then insert them and they just appear on chain and so uh this utility is is the only public utility for watching private transactions because basically what happens is these uh private transactions show up on chain out of the Clear Blue Sky they they never appear as pending but we start to see weird things as well this is a block this is a a block number just so I can read it out uh 18721 1577 uh most blocks have 100 to 200 transactions this this block has five they're all private and basically a block has a gas gas limit you're trying to fill it up to about 50% this block it's used 2.55% so a builder basically decided hey there's five transactions that are really important that I need to get in now and I'm just going to put them in and not anything else I just want to make sure that there's no other interactions between transactions going on chain and uh what I'm doing here and so basically this block is massive underutilization of block space the only way to get included in this block were to have been transmit privately to this specific Builder and the interesting question here is why did this Builder do that right and and I say this because um the interesting thing here is these tools this observability you can watch these blocks as they go by now typically a block is any from 10 to 15% private these days but every now and again you'll see one that's a lot higher percentage or has a weird construction the ordering is strange and then the question is why is that being is that happening that way what is the incentives why why what's the reason behind all of this and this is what observability is all about it's all about this notion that um we can uh get into this I'm going to try this one more time um see it's just a Wi-Fi connection let's see uh and and and really get into uh you know what's going on and the whole point is is you don't need to be a PhD to do this stuff now with these this generation of tools uh anyone can get in and and make this stuff happen so it's partially it's partially behaving it's partially not um uh oh we have um we have support coming in all of this for um uh EIP 4844 as part of denune so you can see here we're looking at type three transactions uh on the sapoia test net and we'll be doing that on Main net hopefully right at uh the launch as well of the sorry the hardfork and then uh we have this historic mempool data archive which enables researchers to go back download uh terabytes of data potentially and go back through and investigate anomalies and and this has been used for some pretty interesting purposes I'm going to switch back my Wi-Fi is not cooperating to my deck and um uh here's an example of some research that we did using our archive which basically helped figure out that one of these superheroes Builders was not providing me rebates that were promised to end users okay and so there's a process by which uh you submit transactions and if there's me you do a rebate and there's a whole bunch of agreements that are made associated with that but there's no observability inside of that meaning you're just kind of trusting these actors to do the work that they say they're going to do and there's a bunch of assumptions in the stack underneath the covers and it turned out in this case that there was a builder that had a a block a a a a back running bot that was misconfigured and was basically accidentally stealing refunds from transactions that flowed through them and the only way that we were able to detect that this was going on to notify the wallet to work with the builder to identify the bot and to fix everything was because of the observability infrastructure that we have and because of of the the use of research and archive there and so today as we look at the complexity of the network as we look at the number of actors in the network and we think about okay what if one of these uh is misconfigured what if one of these the trust assumptions associated with it combines with a different set of trust assumptions does that create an edge does that create a profit opportunity does that create a data leak you know most of the time what we have seen is where there's problems it's not malicious action it is just something not behaving the way that it was expected to do but as the amount of value flowing through the network increases the economic incentives to play games here goes way up okay and so this is why this notion of observability is so important and as Regulators start to look into our e our our world and say hey are you guys is this fair or is there Market manipulation happening the presence of observability and the number of people who are participating in observing and being being witness to the network those of you who are watching The Watchmen I think will really help say oh yeah this is a very transparent Network oh yeah this is very well understood oh yeah this is self-correcting and so this is why we care very deeply about this topic and why I encourage you to participate in this process as well there's been a whole bunch of research that's been published based on our data both publicly accessible uh dashboards a lot of tweet activity out there uh research papers from folks like uniswap and others and more so if you're an academic researcher and you're looking for uh something new to discover uh we know that there's a ton of new things to be found in this data set uh we're a relatively small team we're good at collecting data but our data science capability is limited so this is why we're opening it up as as part of what we're doing and there's new stuff coming online all the time new tactics new Bots new strategies new protocols new capabilities of the network like 4844 that are going to be whole new research Vistas moving forward which we're super excited about so this is why we're building this stuff out and why uh we think observability is a really critical topic for the ecosystem right now and we want to encourage as much participation as possible if you want to start your observability journey uh go to ethern now. XYZ uh join our Discord uh we'd love to hear from you we're trying to make this stuff better all the time and uh really appreciate everybody being here today so I'm Matt from block native uh talking all about real-time observability and the unexpected inner workings of ethereum thanks very much [Applause] cheers