How to Create Pools Using KYVE to Efficiently Store and Validate Data Streams - Fabian Riewe

vivian hello everyone great to see some familiar faces here great to see some new faces here my name is fabian i am the co-founder and ceo of caive we started the journey i would say a year ago from now roughly and um yeah this talks about how to efficiently store validate data streams using hive so one of the oh sorry the talk is going to be divided into three parts first we're going to do a quick overview about what is chive how it can be used next up is a quick demo 10 minutes after we jump into a small q a session if you guys have any questions around what we do so the fundamental problem in the blockchain ecosystem right now is there is no good way right now to store and access data because all the nodes are storing the sorry are storing the data very condensed at their own thing and this makes indexing and retrieving large amounts of data very very tricky because actually we don't know where the data is you are looking for as a developer it could be some state it could be some transaction it could be some value transfer and of course this leads to the rise of indexing services which then of course need to take the data from a node which hasn't stored decentralized move it over to a more centralized solution and then is able to kind of retrieve it from that front same problem goes in with backs with backups or snapshots is if you're a foundation and you're running a protocol it's kind of tricky for you to take a snapshot of the current state of your blockchain storing it as i said somewhere locally somewhere centralized and then bring it out to your node operators to be able to bootstrap the nodes from there because why should the node operators trust you why should they know that you might not fit might enough have um i'm sorry modified some state or something like that so it's really kind of a trust problem you're creating there by having a decentralized solution but then putting the data access over to a centralized solution again there's also no nice way right now to cache state transitions we see this happening with all those big blockchains really kind of growing up to infinity i think solana is a great example there where there only actually a couple of endpoints user developer can use to actually get access to the old kind of block zero data and it's really kind of a a threat that we will start losing the data um if it's happening we also see this on eip 444 and ethereum right where we need to start this enter of the the purge just because basically the nodes have to deal like with all this data and they really don't know where to store them and then it's also economic problems because if the mining flow rises and basically you don't know then if my node is going to be economic anymore if i need to maintain this big data center just for all the data then what's going to happen is i need to shut down my node right in the end leading from a big variety of decentralization down to just a few centralized nodes from node operators which are able to basically work with this piece of data and this also goes in well with the fourth one here the cost of distribution is expensive right we do want this data storage of redundancy we do want to see the data being stored on as many nodes as possible also on the other side it's quite expensive stormy redundancy so this is uh the reason basically why it's becoming a growing challenge for indexers and for blockchains in general right um to scale to be able to hand out the data to developers and basically then just leads to scaling issues not only in on the web three node side of things but also on the web 3 data access side of things i'm sorry there we go and this is the reason why we have developed chive and what kive does it it takes blockchain data routes it through a decentralized network and then stores it on decentralized storage in our use case we're using re for that and basically the way it works is if you're a developer you're going to set up one of those storage pools and you can connect it to any blockchain any data source doesn't need to be really web three datas which could also be an api you want to archive for example and then the way it works is you have those storage pools which connect to the api which have a set of uploaders and validators where the uploader basically relays the data onto permanent storage and then the valid datas they take down the data run some logic against it check that the actual data you have been stored you want to store is actually correct and then is also able to retrieve the data out of it so it's kind of this full-blown cycle where you as a protocol as a developer really don't need to worry anymore like where do i how do i need to worry about the storage for all those nodes because basically you get it out of the box from here um kaif started as i said a year ago we were john and i my co-founder sitting all the way behind there working on a bounty for bridging the data from polka dot over to rv because they set up a bounty for that and we really thought that's a that's a cool thing to get started and then we showed it to other protocols got some great feedback to it and then suddenly the script turned into more scripts and we we had a project we had first integrations out um then we did a pre-seed round did a small test and release i would say in like march this time grew the community improved the ui and ux around it we are right now rolling out a big update um where the demo will be actually about where we introduce the protocol governance because right now what we've been doing is we as a team needed to maintain some some debugging purposes and other things going on so now with the new governance we're introducing i think they're going to be highly uh decentralized then we're rolling out the incentivized test net later down this year and then we also of course focus around the main at launch by the end of this year we have integrated with i would say any major l1 and l2 out there kind of giving them the offer to store their data making it possible for them to also retrieve it without the worry about which indexing service they're going to introduce they don't need to build any in-house indexing service or any in-house data solutions all right let's jump into the demo i can show you an overview about what it looks like working with chive there you go so we set up a demo environment if anyone wants to check it along on their local device you can go to kivinterface.demo.heroquip.com i'm logged in here with one of our addresses and you can see it starts off with a very nice dashboard where you can see the activity you as a user would have done you can see all the notes you can see your delegation chive is a delegated proof of stake model to make sure that basically the data which denotes uploaded are also economically secure and then those are the storage pools you can see here we just for the demo purposes we rolled out the evm runtime so with moonbeam here avalanche moonriver aurora and evmos the way it works is if you would be a developer you would write one of those runtimes which basically just specify how do i connect to an api how do i get the data we will look into this later as well and then you would just deploy the pool here through the governance and then the indexing process process would start you see it here on the moon beam for very well you can see how many percent of moonbeam has been indexed right now we had 45.1 percent we are usually um takes original about one to two days to um archive a whole chain you can see how much the storage size on how we face we're storing there you see the costs of course yet set to zero because we're in our demo environment you see the pool funders and the pool validators the pool funders are very interesting a concept we have introduced so the way why a pool requires funding is let's say you are for example the moonbeam foundation would be actually interested in storing your data then you would for example fund this pool to make sure and start the archiving process then what happened over time is this funding would get paid out to all the nodes as a reward for uploading the data if a pool would ever run out of funding you would have the problem that it pauses and so everyone who builds indexing solutions and is kind of dependent on the data we are storing have then trouble accessing it so everyone is highly incentivized to keep the proof funding up and running in a note section you can see all the notes right now we have three active notes running you see the self stake you see the voting power i think that's pretty self-explanatory coming from a proof-of-stake model over here we could now delegate to a node you get some rewards for it as an end user if you would not run your own node and basically vary overview things like that in the activity you then actually see the stored data so you would see the rv transaction id here where it has been stored you see which node was the uploader you would see the votes you would see the bundle sizes you would see the time and if it's valid or invalid and actually the bundle size is quite an interesting thing so what happens on our site to make the data storage more efficient we take the data we are storing bundling all the blocks up into one big data bundle encoding it and compressing it to make sure it's stored very efficiently and then you can see the bundle size as well and you can see actually how it starts to shrink the more you get towards the life tail of the chain so what happens if if the nodes would be running here you could see that we would basically always be up to like 99.

9 percent of the chain and then we would store the latest data routed through the system and then it would be indexable um for an indexing service so and then of course in our current app if you look in here you see looks exactly the same they're just some different runtimes out here so we have a cosmos runtime as well we have the solana snapshot runtimes and some other exciting stuff uh happening here as well but it's exactly the same concept if you would now want to access the data we rolled a site product called warp which allows you to access the data kiva storing through graphql for developers so here i'm connected to the moonbeam interface for that and as you can see here it's rather straightforward if you're a developer to understand what's happening here so we are querying a moonbeam block where we filter for the field where the number is one and then if we would execute the query here the result is already there you can see you now get the first block of moonbeam and this all comes basically out of the box by using chive there's no need to to use other indexing services another cool feature we have introduced with warp is that it's actually possible to query multiple data source endpoint from a single query so you can see here this query would probably the first blocks on avalanche moon river and moon beam where the number from the block is less than 10 and greater than or greater than 5 i'm sorry and if you execute it here you can see of course we get one two three four five and so on up to nine from avalanche it goes on with moon river and then it goes on with moonbeam at latest so you can see here that this is very revolutionary because you can finally now use one api to create multiple sources and this really comes in handy in the cross-chain world if you want to build for example a cross-chain token bridge can now have two smart contracts which then just request to both both apis from the two end points to really make sure that's fully decentralized there's no custody involved anymore there's no centralized server sorry there's no centralized server required anymore to make this happening this is uh kind of what we try to do with kaiv is really being able to connect the whole web 3 space through this one api making sure that all the traffic is routed through us and stored so anyone can really recover in case there are any problems or in case there's anything else going on if you would now be a developer and you would want to know well how can i how can i build my own chive implementation it's actually very straightforward so what's happened is we as a team have built just a core package here where you would just uh use the chive class form and then you would just extend it with the functionality you would need so in this case we're looking at the evm integration and there are only really three functions you would need to think of the first function would be how do i request my api so in this case what we are doing here is we are requesting the api uh by the by the height right so we just say well from this side to this side we want to do the request store them in the notes local database next function is creating a bundle so how you're going to bundle all the data together how you're going to do the the encoding of it and we have a bunch of helper functions to also make that possible and then what happens is the node automatically submits it to the caf network deals with all the voting submits the proposal and so on so there's really nothing from a developer perspective to worry about it all comes out of the box and then the last function you would need to worry about here is the load bundle function it's like how does my node interpret a bundle how does it work with it how does it deal with it it's also pretty straightforward happy to share some more details with it and this was basically the main overview about what chive is how you can use it in your application how you can use it to store data um it works with any web to web three data source and um yeah we have a bunch of use cases starting from um as i said like multi-chain block explorers is a cool thing you can build with kive going over to the cross chain going over to the data storage things we really try to cover the whole hello web 3 space i would now like to hop over into a quick open q a question q a session