IPFS & Filecoin 🛠️ Storage & retrieval with FVM - Matt Hamilton

foreign [Music] my name's Matt Hamilton I'm a developer Advocate with protocol Labs working on filecoin and specifically the filecoin virtual machine Fem so what I'm going to talk about today is storage and retrieval using fem and a little bit about filecoin set the scene for people that maybe you're not quite aware about falcoin and about ipfs for example and the differences between them so let's just talk a little bit about what filecoin's Grand division is the falcoin master plan so Step One is build the world's largest decentralized storage Network well we've done that and we're continuing to build the current storage network is about 14 exabytes of data so that's 14 million terabytes of data so it's about one percent of the total data center storage capacity worldwide at the moment and continuing to grow step two onboard and Safeguard Humanities data so we're taking data on uh currently from a number of different data sources so things like scientific data and anything from scientific data to nfts everything in between uh that people want to store on a decentralized storage Network to keep long term and then step three is bringing compute to the data and enable web scale apps and that's where really gets interesting uh at places like eth Global and talking about fem the filecoin virtual machine because fem is the element that allows you to do programmable storage on falcoin so you can kind of think about it as various sort of layers so right at the very bottom we have uh a peer-to-peer network based on lib P2P which is also another technology from filecoin and you've got things like ipfs ipfs is another decentralized Network by protocol Labs you might be familiar with it especially if you've done anything recently with nfts it's used very much a lot for that but ipfs is kind of a non blockchain solution it's just a decentralized storage Network um but it doesn't have the crypto economic model that filecoin has so with ipfs when you store data um well I'll go into this in a little bit more about the kind of the approaches you have with regards to storing data and and why we now have things like filecoin but you can think of layer zero file Point as being the storage layer you think of layer one as being fem the compute over state that we we now have the fem enables and then Layer Two will be things like what hopefully people within this room and here at this event might be building on top of all coin so decentralized applications and solutions built on top of this you might notice a little Corgi there um called biscuit biscuit is fall coins mascot and you'll see a little bit more about biscuit as we go along in this talk so this is kind of the evolution of storage kind of starting from things like ipfs which was now I think about eight nine years old now um through up filecoin is now about three years old the fem was launched earlier this year in March um and so these are kind of like the evolution of decentralized storage so I talked about the different uh kind of approaches and if we look at for example ipfs so ipfs gives you various ways in which you can store your data so the first one is that the client can have their own node or nodes they can run their own nodes and store their own data so you can spin up your own ipfs node um run it locally on your your laptop or or whatever and pin data to that and it can be then replicated on the network the problem being is that if you turn your laptop off or whatever then that data is no longer available another approach is you can run nodes with a community so you can get a whole bunch of people within your community that are all running nodes as well so collectively between you you are storing that data and there is no one single point of failure but again this relies upon your community uh to all be storing that data and then another one is just the client storing it on their own in which they use a sort of like a pinning service so that's to prevent the situation like I said where you turn your laptop off whatever and suddenly your data is now inaccessible so services like infuria pinata you might have heard of are offer ipfs pinning services so you pay them a fee and they will pin your data on the network pinning is the process by which a node says I want to keep this data on the network it might be stored on multiple other nodes effectively cached but pinning is is keeping it live on the on that node so it won't ever expire from that node now one of the issues with this being is that if you're using something like one of these centralized services like pinata and suddenly your credit card expires or they go offline or whatever it might be then suddenly there's nobody pinning your data potentially your data May cease to exist valcoin is an attempt to solve that problem so farcoin is actually a separate Network to ipfs it uses a lot of the same technology as ipfs so you might have seen with ipfs what are called content IDs content identifiers if you see an ipfs address it normally starts ipfs colon slash and then a long string of what might seem like random letters uh in in that address now that address is actually a Content identifier it is a hash of the data that you are storing and actually my colleague dawn has a really good analogy for this now imagine you are going the difference between content addressing and location addressing with location addressing which is what HTTP uses would be the equivalent of say saying to somebody look I need you to go and get some my favorite cereal from the grocery store they say okay well well what is it you say well it's on the third shelf on the left uh second up and uh it's the the fourth box along you go along to the supermarket and maybe that box has moved they've done a restock or they've moved stuff around or you know some kid has been messing around with stuff then uh Suddenly It's it's not in the right place and you don't necessarily get back what you expected whereas with content addressing you're actually describing the content itself so you might say okay my favorite cereal has got a a picture of a a Red Bowl on the front of it it's UPC number is this it's 750 grams um and uh you know you could describe the contents of it and it doesn't matter where it is somebody can find it within the shop right they can go in there and find the the box of cereal that matches that description and the great thing is it might not be in just one place they might have say a special offer on or a sale or something and it might be up near the cash register as you walk in and you don't have to necessarily Traverse the whole shop looking for it it might be right there in front of you they might have it in multiple locations and a Content identifier is a bit like that a Content identifier is a hash of the data it describes the data itself not where it is but what it is and one of the advantages with that is when you get that data down you can check the address against the content you have received so you can be sure without needing to trust any other service that the data you have been served is the data that you were expecting that's used on ipfs it's also used on filecoin as well for story data or Falcon they both use these content identifiers cids so with filecoin the data is stored off chain you negotiate a storage deal with what's called a storage provider to store some data for a certain amount of time at a certain rate right so I want to store four gigabytes of data for two years and I'm willing to pay this amount for it now that data is transferred effectively well it can be transferred off it's transferred off chain uh through various mechanisms it can be downloaded by HTTP could be using one of the ipfs um uh protocols like bit swap the ipfs users it could literally be a station wagon full of hard drives turning up with with the data on remember the the data Network Falcon is designed to store you know petabyte ranges of data so it's very flexible in how you actually get that data to a storage provider once the storage provider has that data they then seal the data which is a process cryptographic process at which the data is transformed into a into a form in which they can run uh proof of space-time um uh validations over that data so what happens is the blockchain every 24 hours uh queries the or sends a challenge to the storage provider to say have you still got the data right and it uses a there's two um uh proofs proof of replication and proof of space-time that are used by the client to prove or buy the storage providers to prove that they still have the client's data and they haven't lost it now the storage provider and the client both put up some funds for this storage so the storage provider has to put up some collateral and the storage client puts up their fee so if you were to store the data for say two years then the fear which you put up gets put in a escaro and then paid out to the storage provider over the space of that two years again the storage provider also puts up some collateral and if for whatever reason they lose the data then that collateral is slashed so there's a financial incentive for the storage provider to keep the data online this is the bit that gives you the decentralized uh equivalent of pinning on ipfs right so the filecoin blockchain contains a whole series of proofs that the data is stored and information about the storage deals which storage providers are storing the data for for how long now based on top of that we now have the filecoin virtual machine so the farpoint virtual machine is a polyglot virtual machine it uses webassembly it has a webassembly runtime in which you can build additional runtimes on top so the first runtime we built which is why we are here is the Falcon ethereum runtime the evm the F evm allows you to run ethereum bytecode directly on the filecoin layer 1 Network so anything that you write in solidity that you can compile down and run on ethereum or polygon or whatever you can run on filecoin as well that allows you to then build on top of that decentralized apps daps data dowels tooling like Bridges all that kind of stuff because you're using the same technology that is being used on other evm blockchains so it makes that a lot easier and that enables things like decentralized compute as well so either compute over the state that we have on the filecoin network or off-chain compute over the data itself so filecoin the fem doesn't allow you to run to access the actual data and if you think about that the reason why is if you're storing gigabytes of data you don't want to be trying to load gigabytes of data into a smart contract especially not when you've got you know a blockchain that's ticking along 30 seconds per block by the time you've loaded all that data in you'll be out of execution time so any heavy lifting and compute you would do off-chain you could then record the results on chain for example through something like an oracle you can also build some interesting defy use cases so I mentioned before about how the storage provider has to put up some collateral and the storage and the client puts up a fee that is paid out over time so you can build things like loans that are collateralized on that future Revenue that the storage provider will be receiving so this is equivalent of invoice factoring in the in a traditional business world right a storage provider can prove on the blockchain that they have this future Revenue due to them from the storage deals that they have and they could potentially lend or borrow against that Revenue so you get some very interesting default use cases of which you actually have you know bare metal ultimately at the bottom of this um you know defy system and you actually have storage providers collateral there so I mentioned the fevm is fully evm compatible and so it's built like I said as a um uh a a polyglot kind of wasm runtime that you can create additional runtimes on top of so the one that we're interested in is the one in the the top right here uh the evm runtime that we have right so that allows you to use hard hat remix metamask all the tools you'd usually use for ethereum development you can just put in the filecoin RPC endpoint and deploy an execute directly against the file coin in the same way that it would do something like ethereum so in terms of runtime we're also working on a thing called in interplanetary consensus IPC so this is the next layer of scalability for filecoin and you can think of as like a layer two uh execution layer for falcoin but also gives uh subnets so it's like sharding the network into horizontal scalability with subnets um that you can use as well we're also working on allowing um what we call native use defined wasm actors so in the filecoin world uh we we refer to Smart contracts as actors within the within the network and soon you'll be able to actually compile directly down to wasm and run wasm actors so you might be able to write your smart contracts for example in rust or tiny go or assembly script and compile down to wasm rather than in solidity and things like new foreign blockchain runtime so we can run additional runtimes on top um so you know we might have say a Solana runtime for example uh on top of there for example so what can the fem solve um here's an example of things like data dowels so this is a effectively a two layers meant to look like a cake there but on the bottom layer we've got some Primitives that fem enables things like repair workers so I mentioned about how a storage provider has to prove that they have the data still you could have a smart contract that actually checks that and if one of the storage providers is not able to uh provide your data that the smart contracts could negotiate a new storage deal to replace that failed one from a known good one and uh automatically effectively heal the data you can have renew workers so you could have Perpetual storage when the data comes to the end of its uh storage deal or when the storage deal comes comes to an end rather that a smart contract could renew that storage deal and continue on further you can create new storage deals as well so you could have something like a data Dow so building on the next layer now this Green Layer on top you have things like governance tokenomics uh things like endowment management so you could create things like data dowels which you have a decentralized autonomous organization in which the participants are voting on uh the the funding of storage imagine something like Commons good data for example uh somebody had the example of a film festival in which a film festival the participants within the film festival could effectively vote on the films that were showed in the film festival but also on the effectively what would become the long-term storage of that data as well do things like endowment management so a smart contract could actually actively invest uh funds that it has been supplied with and used those funds in something like one of like a D5 lending pool and use the interest earned from that to fund the long-term storage of the data so you could have some Perpetual storage in that way so I mentioned biscuit our little Corgi mascot from filecoin so I'm going to hit give you an example of a data down called the biscuit data dial this is an example dated out we come up with mainly my colleague Sarah who has a very active imagination and loves corgis and so we're going with this the Corgi biscuit biscuit dated out so basically day to now is a community of Corgi dog owners that want to better understand their corgis so as an example you might have a bunch of Corgi owners that submit data to a data dial about their dogs it might be behavioral data it might be video data um I don't know Biometrics or corgis maybe Corgi fitbits um that's not a crazy idea as it sounds actually work for a startup that was going to Pivot into uh dog uh dog uh Fitbit type tracking um it was a crazy idea but anyway the idea big is that you could take this data from these individual um participants of this dowel store the data within the Dow and then the smart contract could do things like wait until the data reaches a certain level maybe they have to upload data in some periodic time fashion like maybe they have to upload the data uh every Monday to Friday and get like a streak for example they maybe have to get a five-day streak and if they do that then they get some kind of reward so a small contract could mint some sort of token it might be something like an nft it might be a fungible token that could be used to reward the Corgi owner for supplying their data to the network right and what could be used what could that data be used for that data might be used for things like training AI right so imagine you've got all this Corgi data now you could actually use that to train an AI model maybe that AI model is used to create almost like a robovet for example that allows you to ask questions about your call his behavior maybe this AI could then refer to that data and the training of that data is paid for by the doubt and the Corgi owners maybe they get a governance token they can vote on the um actions that happen whether that's some training data that happens um they might get some rewards that could be used for services within the Dow so again things like veterinary services for example um you could you know you could get some discountable reward for supplying the data that could then be used against you know real world uh Services as well so the Dow treasury um went once the Dell takes These funds in the Dow treasury could then potentially actively invest those funds through things like liquidity pools dexes um uh you know lending lending pools for example uh that then support the Dow going onwards so it could be effectively self-sustaining by generating Revenue that way that then funds the purpose the persistent storage of that data the renew the repair the replicate that I mentioned before to keep this data active and the end result healthier happier corgis achieved right so that's an example might seem a bit of a trivial example but it hopefully gives a bit of a flavor of the kind of thing that you can build up using fem using filecoin using things like data dowels um on top of the network and you can access things like off-chain compute so protocol Labs have a project called bakayow that is a decentralized compute platform and there's a bridge called lily pad that allows you to from a smart contract pass a for example like a prompt to stable diffusion running off chain that could then generate an image that could then be stored in idfs that could then be returned back to Fem to Mint an nft the point start image that was generated and that is stored on ipfs so you can kind of Go full circle there between on-chain and off chain with data there's a bunch more FM compatible tools um so there's things like there's a a file coin local net Docker image you can run you can spin up on your laptop by a local uh effectively file coin Network and run locally you can then uh play both sides of the equation by being a storage provider yourself right so you can actually not only create storage deals but accept those storage deals reject them uh whatever um there's cross chain Bridges sealer and axler for example both on the token side and on the general message passing side uh oracles things like teller um Access Control lit um uh Lighthouse which is a storage aggregator um a number of like D5 protocols as well that are being created so there's a project called glyph um that's that top icon there you might not be familiar with in the top right uh glyph has created a big lending pool um that allows you to do staking of filecoin token and that can be lent out to storage providers number of network analytics apis so zondax have a network Explorer called Barracks that allows you to interrogate the network uh there's one by protocol Labs called starboard as well decentralized compute back here I've mentioned that allows you to run uh decentralized compute jobs so there's a whole bunch of stuff within within fem and more stuff kind of coming as we go along and if you want to find out more there's a QR code there if you want to scan that QR code or go to that link tree address at the bottom there that will take you to a list of resources including if you're doing a hackathon you'll see a link there to the hackathon what we call hackathon cheat sheet um that is a document we have on GitHub that gives you a sort of full uh list of various different resources around fem as well so there we go I think uh I think that's the end uh yeah so thanks very much um here's a bunch of the fem team there's actually a lot more than that this is actually a bit of an old photo now um but uh yes you know it's a big project it's been built by filecoin over a number of years now but finally live and allows you to Brand small contracts directly on the file Network so thanks very much [Applause] if anyone has any questions or if you want to grab us actually probably even easier there's uh we're just in the booth just literally on the left as you go out the door if you want to come and have a chat to us at the uh the filecoin booth uh we can chat there thanks