Mantle 🛠 EVM Data Engineering

[Music] and welcome to um evm data engineering DIY Edition so yeah let's get started so gmgm gumgum so this is my Twitter handle at lipm I'm currently an engineer at mantle so I get to play around with a couple things one of them being data from generated from the e and so you know for a rid of contact so I attended the rust exium Conference held by Paradigm in San Francisco and so one talk that really caught my attention was the data end game so keep in mind this was mostly a rust conference but the tools introduced were mostly rust based but they didn't need it to be Russ it was just that it was incredibly insight ful to me to see how modern data engineering tools are being done and so I just thought I would share my experience in the last two to three months and how I use the processes of using tool and my experience using it and how to actually use it how I found it to be used most effectively but anyway paradigm's vision for this is that the data end game should be fast cheap and effortless on any any data any format on any hardware so in the in the conference three tools were introduced ref also known as rust e um which is inspired from go ethereum so rust ethereum is just a a execution layer node that you can connect with your your consensus layer something like priser and then they also introduce a data extraction tool called Cairo which is super interesting and we can kind of dive a little bit deep that later um but and then paus um so Paradigm didn't actually make paus paulus is actually a third party Library made by uh an open source third party Library so but it is a tool that will be heavily used in this in this talk and throughout the examples so a very interesting thing that came out of Paradigm talk in my in my point of view was that um a lot of the full archival nodes um used to be 14 terabytes or above but with the introduction of ref and Aragon um this this database size has Strunk down to two terabytes which is almost an order of magnitude and and it will be an order of magnitude uh oh sorry it is an auto of magnitude which is kind of insane um so even the um RPC throughput latency and success rate looks really promising now I've had some issues with it in the past but that was but you have to keep in mind that ref kind is in beta version Oh yes um so having the database size go from 14.5 terabytes down to something less than two terabytes makes things very interesting because now um running a full archive and when I say a full archival I mean the history of the entire blockchain is accessible under two terabytes is really a big game changer because now it being under two terabytes means you can post it in the cloud for if I'm not mistaken um an order of magnitude cheaper and you can also host it on higher end laptops on your own self um so yeah and so a little bit disclaimer is that ref is actually heavily inspired by Aragon and you can actually read about this in the acknowledgements in the ref section so for example Aragon introduced M mdbx as the database of choice which surprisingly is maintained by run one Russian guy I think but anyway it made the like a lot of these incremental improvements kind of turned that database size uh sorry reduced the database science down to less than three terabytes which is insane but yeah um the key thing I get to take away from that chart is that it is now possible to run an archival ethereum mode like a full on higher and consumer laptops and relatively cheap um as a business on most on most uh clout provids and I think if you guys are on the ref this ref telegram excuse me right now there is a discount code for for one of the cloud provider companies that uh that is being shared around so if you like to just play around and sync up your ref node um I think yeah there's a there's a coupon code lying around the telegram but I don't have links to the telegram so you're going to have to use your Google skills to figure that out but yeah um going back to the original Point um you know we are living in the information age data is the new oil you know but like oil data data is valuable but useless if unrefined in my opinion archival nodes are the equivalent of modern day oil reserves think of all those unisoft trades all those unisoft V3 strategies all those um uh all those positions on are long short delta neutral those Z farming you've done those are all information and with enough information you can get a sense of idea of how of of what's happening on D line but that's impossible to do if you don't have the raw data and this is how you can refine the raw data so you have some a you have some not and again this is all taken from the rust and E theorem confence I just happen to be playing around with it and I really liked it I found it quite confusing especially because there's so many options here like you have your ref node and then you can you can go from ref to like query results through like something called alloy what is alloy you have ref DBP you have ref indexa you have Kyo you have eus you have eus Rh or you can go to something called a uh saniti well it's not San something uh you can go into like pocket or Delta through Cairo and Bre index so uh something I would like to focus about today is just a small section of this this would be these local files um Cairo as well as paus and you know the first time I looked at this slide it was quite confusing to me like what is pocket and what is Delta but luckily uh oh sorry anyway yeah pocket and Delta don't worry we'll kind of go through um what this what these new terminology means together so we won't be navigating in a doc Force so let's say hypothetically um you decided to try out ref you downloaded a ref and now you're running ref cool so you have basically achieved the very first step um in running in running your own data refinement all your data engineering DIY Edition so we'll give bre roughly about two days two to three days to sync depending on your location and then now that it sync hypothetically we would like to extract it out we would like to use the tool called Cairo into something to extract it out into pocket plus Delta files now I know that there's a lot of terminology here like why we what is a pocket file what is it Delta those sounds like homemade jogon but good thing for us is that we have a friend called chat GPT and you know this is uh essentially what what it's returning me so what's if you're wondering what pocket is a pocket pocket is simply another data format in data engineering and so in these days you would use pocket over like most sry most uh most companies that deal with scale will use pocket over CSV simply because it's it's more efficient to start them on as as a raw format for example in let's say S3 where you're being charged for by the gigabyte um and it's more efficient to query then and you don't have to worry about escaping CNAs or any kind of weird things um the only issue with pocket is that it's not human readable and you have to use a tool to consume it so yeah but this is like the comparison between pocket and most file formats for example you don't we can ignore col compressable splitable will only focus on readable and complex data Evolution now schema evolution is is sorry complex data structure such as like arrays Json data if you really wish to um like uh obviously unpassed um and then you could have more complex data like like strings which is usually quite a pain in CSV especially you have commas um BYOD paret is not readable but it fulfills a lot of the other criterias yeah again just to emphasize pocket is just a a data format something I would like to also emphasize is that a lot of you guys might be a lot of the data scientists out there might be or data uist might be quite hesitant to switch to the these new tools because you know it's it's rust and you know I'm familiar with python I don't want to leave my comfortable pythonic land so something I would like to emphasize is even though tooling is mostly rust based they are pipon R available so if you are a python scrub like me and you have not been hardened by life with dur the rasb track do not fret because most of the tools apart from Cairo has a python python implementation to it especially the data engineering bit the data extraction bit requires a bit of rust but um since the hard part of Russ which is dealing with mutexes and asynchronic has been handled by the Paradigm team it's quite trivial to extend on on CYO anyway U if you haven't been Harden by the Bor Checker and you like player on you know python man that's also fine so say let's say hypothetically again you got the ref note up and you know you've decided to to run chyo to extract the data set you got a couple data sets available for you such as logs blocks transactions traces Storage St and so on so these are inbuilt data sets um that Paradigm has gracefully provided for us to play around with so for example the each data set correlates to different kind of raw data that you would like to extract and organize but not but not sanitize or transform in any way so these are raw data from from you know for example the logs only extracts up the logs um and it gives you a bunch of fields such as block number transaction hash um and the topics and even the transaction index which is very useful later on um if you would like to extract out block data for example it just gives you like block number block cach the builder of the block so sorry the extra data which usually correlates to the Builder um the I'm not sure if it gives you fees but it gives you the base fees so each data set correlates to a distinctly different kind of data you would like to deal with storage difference and gives you the the the changer slots um internal slots um on the evm and yeah anyway so let's say hypothetically we managed to run Cairo and extract it out into a pocket file on a local local directory while staring this in local files because it were too cheap to pay for aw ss3 or even any kind of blob starch for from the cloud service providers so say hypothetically go to bind for balance Deltas blocks logs prices now this is my own take on this Stars difference and transactions how can you read them because it's not human readable right well it's very trivial actually you just use Pauls which is the which is actually a rust based tooling but they have a pipe in wrapper for you um to play around with so you you can um import poers as usual in Python and then read the pocket file and this is the kind of data you're looking at now it's being stored in binary data but the but it's fairly trivial to transform them in Python and um just looking at this we got about 100 blocks from this and from the 100 blocks we can see about 12,000 transections and and Cairo pretty much just extracted this in real time there was barely any weight time but keep in mind I'm running my own rest node if you guys would like to run this against public nodes you have to be a little bit careful because you will get rate limited but I've seen this is this is fair new which is why I haven't added to the slide but if I remember correctly I think someone else wrote a load balance for this also in Rust so you can effectively point to multiple RPC providers um while extracting your data but personally I think that I would prefer to extract data from my own node because I know that it's more reliable and I know that it's full walkable node uh okay so the problem with this as you guys might have pointed out is that exploring the data is actually quite cumbersome like I can't just read the first 10 lines of the of the file for me to have an idea of what's going on I have to I have to run a python interpreter I have to import paers and then I have to type out this long ass name ethereum uncore transactions uncore uncore block number exactly you want block number 2core 2core the end number not pocket which is very very tedious like I don't know if you guys um if you guys like typing and you want to bump up your your wh him your words for a minute that's great but for for me for scrub for a boomer like me I don't think like my my fingers are not as agile these days but luckily um not not storm who has graciously provided us with this tour has uh introduce this Alias PPQ python pocket I presume where all you have to do is put this Alias into your bash RC file or your profile config file or something um and then you can now instantaneously um explore your data now the goal of this is to make data exploration as frictionless as possible because you don't want that slight mental hurdle to to to prevent you from uh from from exploring your data it's like going to the gym you don't want to think about it you just want to do it you want to make data exploration as easy as ABC so you there are certain ways to kind of streamline your workflow and that's with to help with aliases um or whatever you guys have been finding useful so polas has a very simple API um it composes really really well with it it's slightly different from pandas so for you guys who are coming from pandas this API might be a bit different but I highly recommend that you check it out because things are very intuitive very simple that kept especially simple so that things compos really really well and they've made a really big effort to try and name things correctly which is always a big plus because you know what it's doing behind the scenes so for example in this case if you would like to do if for your sequel gu guys out there if you like to group everything into a chunk and aggregate things um this is how you do it so in this example I'm grouping this whole transaction 12,43 transactions and I'm grouping them by recipients and then I'm aggregating the transaction hashes into a list of transaction hashes and then I'm counting the transaction hashes and sing them that by the setting order so as you can see in this example the first two address seems to be the most popular transaction in the last 100 blocks um between block um 18 million 255,000 to 18, 255,000 and 99 so that from that two address unfortunately I didn't make this uh human readable yet but it's fairly tral to do so but you can see that the two address um is had had about 1,000 transactions to it um and then so and so on so it this API is incredibly easy to compose and like trust me if you just spend like like half a day looking at it you'll be so productive with it it is honestly crazy how productive I've been with PA as opposed to pandas even though you could both use Arrow like you could both read pocket files behind the scenes but polus is just so much cleaner that I don't have to Google on things like I just type it out and auto complete helps me out anyway um and in this example you can even do intersections between lists over for example I'm grouping the transactions by by the block number and then I'm calculating the unique addresses within the block and then I'm trying to find the uh the intersection between the addresses between the block numbers uh which is kind of cool um so I can see that some addresses have more cits than other others which is probably like sex toex sorry sex all tax transactions um and yeah Cairo is actually extremely easy to extend for anyone who plans to extend on Cairo the downside is that well you have to write some Rust but the positive side is that you can also just write solidity um Aster PM so notice how I showed you guys a custom field custom pocket file called prices just then well that's actually a a custom implementation that I added onto Cairo and I wrote very very minimal rust in fact I just copy pasted most of the code but because Cairo extracts data on the blockchain what you could do um what you could do for anyone who's seen my other talks is that you could use something called arbitrary code execution for for data retrieval to and implement it onto Cairo whereby you write some solidity code the TLD is that you know um in the in the Constructor of your solidity oh sorry in the Constructor of your contract you return data but because normally you don't return data in your Constructor but you can hijack The Constructor to return data using assembly and then you can write some kind of arbitrary your arbitrary business logic or kind of data engineering pipeline to extract out data from the blockchain and then return it to Cairo off which could be processed without writing any additional stuff um so the idea is that you write solity code compile it down to BU code and then you just send the buy code through Cairo to your node which returns you a bunch of data remember it's just an e call at the end of the day because you're actually fake deploying the contract to return the data you don't actually deploy the data then excuse me from there you can kind of you you have the raw data which you can then insert into a pocket file which parad again like CYO has done the heavy lifting for the asynchronicity the chunking the uh mutexes the has all been done all you have to do is is consume data and return data so extremely easy to extend anyway yeah so tldr the the custom data retriever works by returning data in the Constructor and so you can extract data by deploying quote and quote put open quotations on deploying because you're just deploying contracts in E call so you just you're not actually deploying anything it's a readon contract deployment but this this roughly looks subject like this you have a bunch of prices and then you get the prices and then you kind of encode the data um in some bytes and then you return ROM bites in assembly uh I'm not sure how you can do this in bite butle but this is how you could do it Solly and you can you 100% can do it but yeah with e call AB ususer you can basically extract and organize data in an ad hoc manner I call this jit D just in time data um so if anyone wants to once the use this St please call jit D thanks anyway to summarize I know we're running a bit late on time we want to get like a few minutes for questions we can use ref to have raw data chyo to extract them and we use polish to refine them um that's like the simplest method I found and probably the most effective I know there's a lot of tooling out there and you guys might get very overwhelmed by the choice of tooling but if if you would like to dive into Data engineering these are the only Tre three tools I've been using and I've been fairly successful so far well um but yeah thank you everyone for listening to me ramble all