Mantle 🛠 Modern Tools for Web3 Data Analytics

[Music] so I am at lip evm I'm currently an engineer at mantle so if you guys want to build any bounties on mantel let me know AB is just outside um so yeah today I want to talk about uh data analytics tools for web 3 there's a lot lot of flavors out there but for more context about this talk was the Paradigm uh Paradigm had a rust X ethereum day in San Francisco a couple months ago and they talked something about the data end game where everything is fast everything is cheap everything is effortless and you can get any kind of data in any format for any hardware so I thought that was very intriguing and so I kind of went on this little rabbit hole on how to better extract data um I know the standard today is using the graph or for example extracting data through Json RPC calls um and those are fine those are good um it's just an issue when you want to do analysis on data that is that is more more than a thousand blocks old um it takes time it takes effort and it is not eort effortless what we want is to be able to transform uh extract and transform data effortlessly um uh over sorry we want to be able to extract and transform large chunks of data effortlessly and so today after going down the rabbit ho I'll be telling you what I've learned and hopefully you will learn something new as well so tools introduced so uh you know we got ref a rust based evm node uh we got Cairo which is another tool from Paradigm which is a rust based data extraction tool and also we got polers which actually isn't to a tool but it's actually just a data frame Library which uh implements the apachi Arrow standard and we'll get a little bit more onto that later so uh uh to extract data we first need a node so because that's where our data comes from so here's something some interesting statistics um to sync an archival node on ref it would take around 50 hours and take you roughly 1.92 terabytes so this is a few months out but so I would say they're around 2 terabytes right now on Aragon it would take you around 5 days or 2.2 terabytes on GE it will take you one plus months um and 14.5 terabytes and netam mine it take you one plus months and also around 14.5 terabytes so I guess you might ask like why the the discrepancies like why does it take such a big Delta to sync from to sync archival when I say archival I mean everything from block zero all the way to the latest block uh why does it take 50 hours in fact there are snapshots for ref right now which reduces that down to six so if you spin up a fresh machine um you can actually get an archival note in less than 6 hours and you might ask like why is there so much discrepancy so the reason is because ref actually uses a new kind of database technology M mdbx or something um developed by a Russian guy single Russian guy by the way um that is supposingly a lot more performance than levels DB which is what gef and what I think nine uses which is why we can now store a archival note under two terabytes which is really amazing um so a little bit of disclaimer here ref is actually heavily inspired by Aragon and and if you look through the source code you can actually see a lot of its Inspirations and where it comes from um and here are some acknowledgements so ref blah blah blah blah like has a lot of Inspirations from geff Aragon akula especially mdbx um but yeah anyway what the the point I'm trying to make is that it is possible to run an archival ethereum note on consumer laptops now so back then if you wanted to run a archival node of geff or Aragon that would take you 15 terabytes but now with ref and Aragon it takes you maybe 3 four terabytes which is pushing it but it's still very doable you know and data is the new oil right uh we live in the information age and like oil data is valuable but it's useless if it's unrefined so we'll explore now that we've explored how to extract the data and how to get your oil we'll explore how to refine the oil yeah anyway archival notes are basically the equivalent of modern day o oil reserves and what's cool about um data analysis is that everyone's on a Level Playing Field cuz you know no one's controlling some kind of secret data the data is open out there for everyone to to analyze and extrapolate so um here's what a standard refined process could look like you have your archival note ref and then you would extract that out into local files or into your Cloud buckets um via some kind of format historically this would be like a CSV um and you would maybe use something like uh e to CSV which is a tool um to extract the data from your node into your local files and upload it to the cloud and then you would kind of do some kind of transformation um is the mouse working yep do some kind of transformation here via maybe something custom or maybe um uh something standard and then you would present the query results here so there's ref there's pocket and Delta if you don't know what that means don't worry we'll we'll go through it later and then that's your query results so what I want to talk about today is these three things uh Cairo the local files and polus um so let's say hypothetically you got your ref note set up um and you're ready to extract data so what do you do uh I would turn to Cairo you know cairo's uh again a tool from Paradigm which is used to extract data into the pocket format um and you know if you were like me three months ago and I didn't know what pocket was I went to chat GPT and you know I kind of asked it to ex explain what it is in the context that I care about but basically pocket is just a data format uh it can be like CSV it can be a Json pocket is just a more efficient way to store data and a more efficient way to store read and transform data um and it's also open source has a standup behind it uh I think it's like Apache arrows if I'm not mistaken and a lot of real-time time series data especially financial data uses the um Apache Arrow standard so it's battle tested and scalable um yeah so this is just like a brief overview of the differences between uh CSV Json pocket and a AO which I've never heard before until I put this picture in um but yeah so pocket fulfills a lot of criteria it's compressible it's splitable it supports complex data structures for example um binary or even bytes it supports strings it supports list of lists of strings um but the only downside is that it's not human readable so uh if you don't have any tooling to help you with or if you're not familiar with computers um you know your typical abbys or your typical data data scientists who can only read stuff from Excel um pocket might be a bit lome or tedious but since we're all computer experts here this shouldn't be an issue so let's uh continue um and you know I notice uh you might have noticed that I've mentioned rust many many many times uh you know it sounds like I'm a fanboy uh even though I am more of a hasc fanboy that's un related uh even though these tools are rust related um if you are more inclined to use it in another library for example python um don't worry there are python libraries and rappers available you know uh you know this is my favorite image you know the python baby boy gets uh you know is gets his aid from the uh rust Soldier who's returned from the battlefield anyway so this is how we can use Cairo so this is the API for Cairo and I think this is like a month or two outdated and Cairo might have updated since but using Cairo you can extract a bunch of data sets a predefined data sets I will add from your theorin node so you could do things like you could hypothetically extract out logs from a certain range of blocks so you would type CYO logs into your terminal dasb and you would put in like 16 M mil to 17 mil blocks so you extract out all these logs into like a pocket file or um if you're more inclined to extract out the block data for example the AL of the block the base fee of the block the uh the block number the block hash and so and so on you can also do that through the block data set um and there are actually a lot more other data sets a predefined data sets that I would add such as transactions storage differences where you can find out the slots and Slot differences where you can find out the slot Deltas for each contract events which is just logs and Bounds differences which tells you the ethereum bounds Delta um for each block but anyway this is how you would use Cairo Cairo data set and um some other parameters so let's say hypothetically that we've um used Cairo and we've extracted out these data sets to our file like how can we make sense of it how can we can we use it well how I would use it is I would read it in a in Python because I find it a lot more easier to perform complex data Transformations with so how you would read it how you could read it is you could import polers and read it um using the library um but this is very very tedious right you need to open up your python interpreter so that's like one one command you need to import the library which is another command and then you also need to type out PL which is polar's read pocket and enter the file name which is also very tedious if you're not in the correct directory so that's like three to four commands just to read a file so tedious so uh it's just not very uh I would say not very reasonable not very practical for your average data data scientist Joe but you know uh luckily um we're all Linux experts or to some extent computer experts and you know you realize hey I could just Alias this into some kind of command and uh luckily for us not not storm at notot Storm from Twitter uh mentioned that hey why don't you just Alias this uh command PPQ and then you know now instead of typing out fall commands all you do is type out PPQ and your file name and then you can read it read the data frame like that in a jiffy um so yeah this is like little things that actually make exploring the data a lot more effortless which um when exploring data is made more effortless you will explore it more frequently and then you will find a lot more uh a lot more relations or correlations that's what I've personally found reduce the barrier of Entry reduce the friction required and you will find yourself um doing a lot more of these tasks um but anyway so we have the data in a pocket format and now we want to do some kind of transformations to it for those of you who are familiar with pandas um this syntax might be a little bit different um because I think uh like pandas Panda's API has kind of evolved over the years but paas is a pretty new one and it has actually learned a lot from um um what pandas would cons would consider good A lot of the API is very clean very composable uh and very uh uh consumer friendly so this is an example of how let's say I wanted to find um which addresses would send out the most transactions um in a given list of given chunk of block so I would group everything by the addresses the two address and I would aggregate things um by the transaction hash so pl.

C transaction hash do explode just means I want to convert when I group everything by this address I want to group everything and accumulate it into a list and after accumulating it into a list uh as you can see here in transaction hash is a list of binary data I also want to count the number of times this transaction has been sent uh has has receiv received this sorry this address has received a transaction and then we can just do a DOT count and we can Alias it so this is like three lines of code and you've managed to figure out um the most frequently sent addresses um in etherum from a certain block range which is pretty cool in my opinion cuz if you were to do this in for Json RPC where you have to manually query each block data like doing a thousand blocks would take you maybe 10 to 15 minutes this took me less than like 10 milliseconds so when you're doing like chunks of historical data this is where this method really shines um you can do do more complex analysis um with uh with with its with polers via its apis but you have to go in and kind of Discover it for yourself the API library is quite verose and quite Rich so it might seem a bit intimidating but I highly encourage you to start off with like a project to kind of check things out for example here we're checking out the intersection of addresses like which addresses would send to other addresses which is quite interesting if you're looking at from a forensics perspective because you might want to say okay what addresses uh might send transactions at a similar time interval you could definitely do this with a one liner in Pocket which is not this by the way but you could definitely do that um through like some kind of uh between range between time interval thing um and this is done in in in chunks chunks of data and you uh polus has been shown to be able to transform over like a million million rows in less than like a second uh if you were to do that via Json RPC or if you do it by CSV it could take a lot longer but yeah anyway tldd uh Cairo pretty easy to use uh very very open source and extendable um in fact I've also open source sorry I've also extended it a little bit by writing out a custom data set that extracts out the pricing data um so in this case for example if I wanted to get the price of e BTC or if usdc or E die I can write out like a Custom Contract uh to do so uh via the E call uh data set yep so uh we'll speed through this a little bit but tldr is that you can write arbitrary code for data retrieval without the contract deployment and how you do that is by abusing the E call up code sorry the E call Json RPC code um so this works by returning data in the Constructor because uh usually you don't return data in the Constructor in solidity you can't do that because you know why do you want to return data in the Constructor it's useless but for our use case um where you just want to opt optimistically deploy a contract and get its return data um and extract the data into polus you can definitely do so uh so we could create a contract like so like prices uh which is a un 256 a list of list of numbers we get the price for say the index and then we encode it into this ABI encoding scheme and return the data uh you have to go into some assembly around but uh that's the price you pay for uh wanting things to be clean I suppose but yeah so this is how you would like abuse use the E call code and I actually have a whole repository dedicated for that if you're interested but yeah you can just easily extract and organize data in an ad hoc manner you know and you know maybe I will call this J JD just in time data um but yeah uh tldr uh running bit short on time uh with ref we can now extract we now have the raw data and with Cairo we can extract the raw data very efficiently and store them uh locally or in the cloud and then we can use polish to refine them and so that's the structure that I found to be working really well for me um yeah uh thank you very much