All Your Data Belongs To Us - ETHGlobal Bangkom 2024

to all your data belongs to us I'm at uh lip evm uh on the Twitter's yeah Google gagag gumgum Google lip evm on Twitter at uh currently I work at mantle and on the side I do uh I work at cryptic Woods where we have a me dashboard and in the process of making the meev dashboard lip m.com I've run into a couple challenges uh and issues regarding the data processing Pipeline and I thought I would just share my struggles with you guys today on how to better transform extract sorry extract transform and load your data so the focus of this talk today would loading part for now and just be focusing on the extraction and the transformation of things um the goal of this is to create an application where data extraction is highly optimized for uh pre-processing and all Pro processing uh including any kinds like lip m.com we are a me dashboard where we do a lot of bespoken Transformations for transactions for example what does this bundle how much e did this bundle extract so and so on so so there a lot of accounting a lot of data that you have to account for and I'll be going through what I use on the back end to achieve these kinds of Transformations we have a couple goals with the transformation side of things we want to make it easy we want to make it back fillable and we want to make it fast um by easy I mean that uh you don't need a PhD in computer plus 5,000 at any arbitrary point in time and this is easily achievable with an archival note and we also want it to be fast so yes we'll be focusing on the extract side of the pipeline for the time being pitched as a binary to extract blockchain data uh into a variety of formats there's pocket there CSV there Jon your analysis uh it's just like a pandas data frame for any one of you guys whove who's done any sort of data analysis Python and this brings up a lot of possibilities because the main back end of Cairo is written in Rust so if it's a float if it's a unsigned integer of eight bytes of 16 bytes or is it an uh just an integer of eight bites I want to perform some analysis on it so oh and also Cairo also extracts data in parallel or asynchronously by default so a lot of the bottlenecks that you get uh with python you can easily overcome it with Cairo there's a couple data sets in Cairo where you can play around with there's um balances there's blocks where you get the the data such as the block number the hex string the the the block value the minor and so on uh like you can also query erc20 balances such as like how much how many usdc was in this contract at this given time or how many usdt was in it I'm focusing on the E calls in particular because this allows you to create bespoken data sets uh when I say bespoken data sets I mean data sets that are not not specific to the chain but rather are stored in arbitrary state for example the price of e against usdc on on a unisoft V2 pool for example and for those of you who are not familiar with e call uh how it works behind the scenes is that if you would like to query custom data back so if you go to unisoft do org or any other any other exchange and you put in one e to usdc generally what happens is that it would perform an e call to the blockchain to see to query how much usdc you might get out and that's the whole point of e call is to query data from the blockchain and we can now use if call inside of Cairo to it's very simple you import the Cairo package in Python and you call do collect and you provide it with e calls uh in this specific and usdc and I'm querying it for the latest blocks uh and the five previous blocks so I would like to uh so this is the code the code getting the reserves and if I run this file this is main.py I should be getting five blocks of data of the ethereum price so [Music] python hopefully it works hopefully the latency isn't too bad here cool so the price of ethereum right now is 3,37 usdc I hope everyone can see it I can zoom in a bit more 3,00 and 37 usdc on block 21981 23123 to the latest 10 all right we can also query the time but this is how simple it is for you to extract data at any arbitrary point in time um using Cairo all right so um the problem with Cairo is that if you would like an additional data you now have to query in a separate call so for example if I wanted the price of e against usdc on that particular block I have to do one e call if I would like the price of e against wbtc I would have to do it in a separate call this is extremely lame if you're trying to build a data set that is that is very expressive because that would mean that your the amount of queries that you're performing will blow up one network requests for each query isn't great we have an O of N and we want an O of one can we achieve that the answer is yes of creating a bunch of separate queries to the blockchain in you would query the multi call contract which lives on the blockchain which would then help you query all your queries against uh that chain in one query so the very small distinction now is that instead of using Cairo to call that particular contract You're Now using Cairo to call multi call 3 to call a variety of contracts all in one query and if you're a front-end developer trying to save on uh trying to trying to reduce the number of optimize your front-end load speed multi call 3 is actually a really good tool because you can now query up to like a variety of information like an arbitrary length of information in one quer reducing like uh reducing a bunch of uh variable load speed times into into one query very handy highly recommend it the main problem with multi call 3 is that you now have to encode and decode your data in a more complicated format you can't just call uh get reserves you have to encode your data into a relevant format after first uh this is just the decoding bit um previously previously this was all you needed to do you just call Two address and get reserves and you get the numbers out but now um the output data is and out the output data from Cairo you now have to decode it into a array of query query within MTI call 3 to get your data out um I messed up the side too was this one univ V2 w usdc we have encode it to be multi call three compatible AKA a a array of address and bites and a bull this the bull basically just means do not stop if it fails and only then um can we encode it for multi call 3 um and and perform the necessary code two all right so this is our initial file let me zoom out a bit so I have some space let's go to the multi call 3 file notice how much more it's slightly more previously we could just directly call get reserves to the to the univ V2 pool but now and if we run the if we run the multi 3 file it should produces a similar results uh M3 main all right so this is is the current e price but let's say I would like to get another call data maybe a different pool a different uni V2 pool what I can do now uh e scan univ V2 we'll try and get the price of BTC against usdc God damn un V2 copy and paste this wbtc usdc so this would encode the data into it but we haven't decoded it so we still we we still only have the ethereum price against usdc here what we need to do now is copy paste this function change the return data from 0 to one because it's now the second query um and I if I remember correctly we is always bigger than BTC I think depending on the address address 0 x22 yep so it's wbtc reserve and usdc Reserve and we'll just copy copy paste wbtc is 18 decimals usdc is six decimal so this is wbtc [Music] usdc hopefully this is right okay great it worked um so we got the price of Bitcoin against usdc on all these blocks and this is all done within one query per block um very handy and in fact you can change um you can change it to some arbitrary point in time for example if you want want it un Block 17 million uh ethereum on Block 17 million and the price of Bitcoin at Block 17 million all done in one cre per block TM um yeah so yeah that's it that's that's my live coding I really appreciate you guys hearing me ramble uh that's my Twitter at lip BM if you guys want to check out the examples for the code above you can check out my repository at github.com lip evm cy- python example uh happy to take any questions if there are any no questions great everyone is speechless thank you for your time and I appreciate it for