Building a Modular Storage Layer for Ethereum on EIP-4844 | Qi Zhou - EthStorage

much so uh it's my great pleasure here to present our work on building a modular uh storage layer especially solve some particular issues on top of ether the latest update that is EIP 4844 so first of all a quick introduction of myself um I have been working in this space uh for since 2018 I was a software engineer at Facebook Google working on Distributing file systems and high performance systems and very active in ethereum ecosystem so have published tons of Articles ideas in terms of in in the etherum research and magician and also author a couple of eips and so my primary interest is in uh ethereum data availability Technologies and layer to and so I also received multiple supports from etherum Foundation Foundation including the uh personal Grant on Cross layer to Bridge and did availability sampling research and lastly uh is a story itself so first of all why we need uh ethereum uh Storage Solutions as we already have a couple of like fcoin rvy that are re running pretty well these days and one thing is is be motivated by the future ethereum upgrades where the data available technology will bring a large amount of data on chain so one key upgrade is EIP 4844 that is part of the Cancun uh which is the first upgrade to support data availability and so what is data availability is a way we are able to scale a to scale the eeran data publication in using a lot of advanced cryptographic Primitives so here is the road map that I captur from rolic Tre at the end of last year where there's a plan for the surge which is going to support 10,000 uh 100,000 transaction per second on top of ethereum and where the core technology here is data availability including EIP 4844 and full da that is planned in the next couple of years so what is EIP 444 it is introduced a new type of transaction of ethereum which the transaction can include new object called binary large objects which has 128 kilobyte size and it is not be able to access by the evm itself but it's hash the commitment is available on chain so that with some Advanced cryptography for example dat availabil exampling this consensus can prove that the data corresponding to this commitment is available on chain uh in their data availability Network rather than access each bite of data that is available in code data so in this way ether is able to scale dramatically in the future through da which can create about 50 terab of data per year on top of ether Network that is more than 10 times more data that can be generated by current ether network but one feature that we found in EIP 444 spec that was uh that was drafted about two years ago is that because there's a large amount of data that now can be published on chain so where is to store this data especially in the blockchain from our for Bitcoin ethereum is expected to store all the data from Genesis but this creates significant storage burden for all the validators Imagine 50 terabytes per year for each validators which means that we have to run a validator with maybe tens or hundred of terabytes which is definitely exceeding a lot of capacity of those validated nodes in order to solve this problem the efference solution in the protocol is we're going to this disc or Pro those data once is published on chain after 18 days and this is the first time the if particle to volunteer disc discover those data once it has been published in a couple of days so this is the specification that is merged in the ethereum uh spefic specific specification repo and actually this data expiration issue applies to all the data availability networks including sesia which maintains data for one a couple months and also da which they would like to pre the data after 14 days and so this pluming actually create a new issue or a new fundamental issue on future blockchain especially for ler TOS is that if they would like to derive the latest states of security from ethereum only on ethereum then it has to rely some additional assumption or additional trust because unlike current this days is those data will be discarded in 18 days and such plan is already SK scheduled in next months on March 133 so when we observed such problem about two years ago one idea come up with my mind is that it's possible to extend those data availability so that we are able to incentivized large amount of storage know that natively running on top of E network as a super set and store those data with much much longer time potentially forever with this feature we are able to offer a lot of unique things to a first it can offer much better L to security for all those ZK R UPS including open stack po and others so we have already integrate our system into OB stack and second is that with much longer Storage Solutions we are able to offer much much cheaper Storage Solutions with large large amount of onchain data including NFD images autonomous worlds and AI model itself with these Technologies we are able to upload the AI model model which can be a couple gigabytes and verify its result with the promotes as input and tell the corresponding output is correct and which enable a lot of applications that was not possible before so that's the basic motivation of is storage is a modular longterm data Avail data available layer building on top of ethereum current called short-term data availability layer which is built on top of E p444 Etc and first of all we are a modular ones so that it can be easily pluging into the current ethereum mro machine and L to and the networks so that so that we do not require any protocol upgrad except the da itself and second with some enhancement of the token Matrix we are able to achieve a very high and permanent availability of those data that was originally published on top ethereum and discard in 18 days and lastly we are able to scale the capacity a storage capacity of AUM from a few terabytes now is can really easily reach to pabit of data which means that we are able to upload a large amount of data that was not possible before how we are able to achieve those features there are two major innovations that we invented during the development of is storage the first thing is to build a unchanged streaming storage fee distribution why because there's a huge misalignment of the storage incentive by system on existing ethereum Network all the storage fee for example for the for the as a store all the store fee is distributed to the proposal in one time while all other nodes validators for noes there's no any incentive or penalty to store the data so there there's a huge storage misalignment between the cost and incentive we at the first one fix the problem on top AUM by receive Up Storage payment from user and then distributed those fees to those storage providers over time as long as they can prove that they really store this data over the time using our SN based storage proof algorithms and this algorithms actually is the our next Innovations which is to be able to verify those we call data availability sampling over time uh with a very very efficient verifier and for this research is exactly the one that receed the support from eim foundation so here is an overview of is story Network it has two major components one is is a storage uh storage contract which allows any applications to call a put Meer and say Hey I want to store this data and this blob in their layer to network and and also it will pay a very small amount of storage fee in terms of eer so that it will be distributed to the storage providers over time the second thing is is a story L to network which is a permission list network with a bunch of notes that store the corresponding bloed data once it's receive the data and upload from and and download the data from from the if one it will perform the data availability sampling over time so that they're able to prove the local storage that they store over CH even though the data will be PR after 18 days when there's a proof that is successfully created a story provider will some corresponding proof to the onchain efference SM contract that is deployed by UTS and once such verification is successful then the corresponding story fee will be distributed to the corresponding story provider to reward the time and number of the data that is stored by this node here is an uh the role map of is storage so in the last year we have a lot of major achievement one is that we are the first place at the ethereum Aton super demo and we raise 7 million and 100 Mil and for this year we have a a lot of uh planned uh launched the first one is we have fully integrated with EIP 444 and we are going to launch our public test net shortly so we have a application of our public test net at our booth so for you to come and fill in your information and we will invite you guys to participate our tesna and experiment all these Innovations and so this is the integration part with EIP 444 we have worked closely with eum Foundation core developers and potentially Crush one or two de internal Dev net during the stress test phase and we have be able to run the full Network when the if launch is internal dep net is 12 right now is public testet that's poia and hoski and also we have some third party Community member that are able to come up with no and prove their local storage um by themselves an a storage can be more than a long-term data available layer of eum thanks to Smart contract and programability is a story can do more than just a static file story system it's essentially a database with key value interfaces this enables a lot of dynamic website applications rather than static ones that we can found these days uh like filecoin RV another interesting differences of is storage is that it's fully in integrated with efferent development stack as a developer I only need to have eer know how to use iser wallet and smart contract and then I can easily store all the data I would like to store and organize by smart contract on on on aerum so this will be significantly reduce the learning curve of all the developers and users to use its storage we also propos a web3 access protocol which is a decentralized HTTP Pro protocol to access those data that is on chain and be hosted by the smart contract so the smart contract can Define the corresponding logic to display those Dynamic data that is uploaded by the users rather than a static ones and with this we are able to especially together with e scality road map on computation that's currently at Ed we are able to deploy a lot of applications for example decentralized social network even YouTube or decentralized gaming everything now is able to onchain including the front end so there are a couple of interesting applications that can be enabled by is storage and the new access protocol here I just give a highlight of fully onchain AI so if it is a storage now we are able to upload a large amount of AI model for examp potentially three or four gigabytes of data that is on chain we can further be able to verify the inference on chain given the AI model that we have published in this is storage and so that we can tell given a promote and the corresponding outputs we can tell that this is the correct answer by the AI model which is impossible before and also this is powered by OPM L optic Mystic machine learning uh developed by Aura and this is have some new applications with AI and this is two of some I'm pretty H uh I'm pretty play glad to play with them one is uh using large language model as the game engine so that now we are able to use large Lang uh model and compose all those elements so that it can come up with more complicated F and all these composition the sequence of composition are available and verified by Chain there's a more advanced things now we are able to deploy a tax based DND on chain and so that we are able to build a fully onchain auton autonomous world uh on top of a fair Network so this is pretty much all I have and so many thanks for attending my today's talk so if you have any interest both in running a no of the network or would like to deploy a application on top of is a storage or web stre access protocol feel free to reach me at qcore Qi hou and I'm looking forward to see the new web3 world there thank you very much and yeah I have I can also answer a couple questions any questions hi thank you for your presentation uh can you talk a little bit little bit more about the pricing like uh the cost of running like a YouTube video or a streaming using the service that's a great question so the question is up about the price to uh for example store a video or some like large amount data right so the price actually is not limited by it's a store itself it's most limited by etherum gas because the transaction itself need to carry the base gas price gas gas Base gas that is 21,000 and a couple of gas to qu a contract to basic index the data that I would like to store on chain so for that part I think um in general um you can imagine we just take two as a store to store all this data that is on chain which takes about one1 $1 $2 according to current gas price and there's a further Improvement of this storage price is to be able to deploy is storage on Layer Two so that is storage is layer three as a storage layer and in that cases because the gas price is significantly reduced the the price of storage will be reduced to one to the about 100 that's our our current estimation especially according to the for example stack or up chance gas price uh metering and so this is something that now we can achieve and so compared to layer one gas prices layer the deploying on layer one that will about one to the 1,000 and Deploy on Layer Two that be one to the uh 100,000 versus experence storage native storage so that is the single maybe single sentence description of the price any other questions one more question hi uh so for people who want to retrieve their dataor so for people who want to retrieve their data do they have to pay the gas fee for that as well uh so how they retrieve data for retrieving the data do they have to pay the gas fee uh do they need to pay gas feed to retrieve the data okay that's another great question so you can imag imagine is storage is a superet of ethereum which means that all the retrieval Market of is storage can derive existing retrievable Market of ethereum in general ethereum there couple of retrial Market one way is that I run alone by myself so that I can retrieve everything I need or I can use inferior Alchemy which will provide some tears of the retrieval ability uh in terms of how many requests I would like to make all these things similar things can be applied to is search even though is a bit centralized way there's another interesting thing that we are working is to enable fully decentralized access protocol that work together with ethereum portal Network protocol uh portal Network so ethereum portal network is uh ethereum sub project by the uh ether Foundation sub projects that is able to enable decentralized station RPC by building uh DHT Network similar to ipfs but only the data that is verified on chain so so not any garbage data on on ipfs it has to be verified on chain so with that we are able to also offer this kind decentral Access Network together with the blobs that is introduced by EIP 444 and access by the application so there couple of ways but fans to ethereum itself the network itself we can just borrow the best practice to of all this retrial markets is instead of inventing something new and need to be that better tasting over time all right thank you so much CH that was a great talk can we give it up for CH from eat storage um thank you very much