Buckets of Data: Strategies for Leveraging Off-chain Data Storage - Carsen Farmer

all right well next up we're gonna just move into it we've got uh textile and um carson uh carson farmer so is carson here i'm sure he is so carson is with textile we're behind on time so i'm just gonna kind of skip my talkie talk and let you do your walkie walkie so you ready [Music] can carson hear me hear me looks like we're having an audio issue carson okay i'm back on all right so while carson's getting going do we have are you back carson i don't know can you hear me oh that's much better oh good okay you are ready to go take it away uh i will attempt to do so okay thanks everybody um sorry about the technical difficulties there uh it's going to be tough to follow up with the energy of the last talk but i will do my best um so i'm carson farmer from textile i'm talking today a little bit about ways that hackers can leverage um some off-off-chain data storage for developing their adapts and uh hacks and fun experiments um and it'll be a little bit of shilling and talking about some products and things that i hope folks can take advantage of and i'll try to keep it on time to keep us running smoothly so if you're not familiar with textile already we're a company focused on pretty much enabling developers to leverage web technologies as easily and productively as possible so we're a bunch of tool builders like probably a lot of folks watching right now and we like to build tools that help developers write deploy scale and build apps with a particular focus on ipfs and filecoin and maybe smoothing over some of the rough edges that folks might encounter when trying to deploy data off chain so we've been around the eth community for over three years now we've been coming back um pretty regularly uh at this event and other eth events around them around the globe or at least in the before times we did that um and we've learned a lot uh from developers in this space about basically how they do or or don't use ipfs and other decentralized uh storage technologies in their adapts why what are the pain points why did they end up going you know screw this i'm just going to dump this thing this nft on aws instead of trying to embrace some of the um decentralized storage technologies so uh that's kind of where we we slot in we spend a lot of time trying to smooth smooth all the smooth over those edges and make it easy to slide decentralized technologies into a developer's existing development stack so i want to talk today about a couple of the open source tools that we've built that developers can use um that hackers can use to leverage ipfs and filecoin a little bit easier in their in their hacks and i'd really love to see a bunch of hacks that leverage filecoin this is the first time um at an east event where filecoin is fully live and ready to go and there's a bunch of hackers uh that can take advantage of that so i really hope we can help you uh get there and uh we do have quite a few projects that have been building on textiles so if you haven't heard from us before maybe some of these other teams you have heard of um this is our first time as a direct sponsor of denver but you know you might have heard of some of these projects like fleek or any type food fish using some some of the textile stack vault 74 is a pretty cool looking uh chat app um we've got things like wrapped fill uh sort of bridging the gap between ethereum and file coin and a whole bunch of other really really cool projects so uh if if if you get a chance check them out many of them are here um at the conference so ask them um we're pretty nice folks to work with i think um anyway each of these companies have used textile as part of their stack to you know leverage ipfs and filecoin in particular and that's what i want to kind of talk about today so uh i'll do a little shilling at the beginning and then i'll get into some more technical stuff after um but the shilling i promise is uh really to show you how it's pretty easy to integrate some of these tools into your hack so this is what our uh stack includes for the most part um there's a lot of products and apis that developers can use uh from things like uh object storage on ipfs so like object or blob storage so that would be replacing your dumping of nfts on you know aws to dumping it in something that feels a lot like that but is deployed over ipfs and potentially even archived to filecoin um so we've got tools that automate deal making um to file coins so that you can kind of set and forget and flush you know chat archives or nft assets or user data or you know logs of interactions things like this there's a pretty straightforward way to get those on filecoin and ipfs we also uh provide a fully decentralized database called threaddb which runs over uh lib p2p and a bunch of ci tools and hosting solutions to kind of wrap all this stuff together um and you know uh for the for the hackathon we actually have a couple of bounties that are focused on some of these specific tools and apis which are all open source and all easy to kind of play around with locally or to use our hosted offerings um so we have some uh uh ways that developers can interact with that and if they wanna if you wanna pop over to our uh booth and ask you know how how am i gonna win this thing um i'm happy to provide some some feedback there but one thing i wanted to talk about as a useful tool specifically for the hackers of this conference is um we've deployed a bunch of these open source libraries all together on something we call the hub and so i wanted to to focus on a few of these specific uh open source apis so these ones here in blue are ones that if you um they're pretty easy to integrate into like a javascript based um dap or if you're running on uh you're running a server-based app we've got go clients and even some python clients for certain um apis they're pretty easy to integrate they're fully open source you can run them locally and test you can use the hub hosted one so that you don't have to um but there's there's a lot of nice ways to integrate them and i'm going to go through an example of how you do that in a little bit here um for uh you know for some of the more industrious uh hackers out there you may want to um uh take a look at some of the hosted options that we have so that you could um have your own uh file coin infrastructure hosted by us you don't have to run a lotus node you don't have to think about keeping it um up to date on the file coin chain you know this is the same sort of thing you expect when you're using hosted ethereum development tools and services as well but this is a file coin one so this is something that if you're interested uh you probably want to come and talk to us specifically the rest of the talk i'm going to talk about some things that you can just npm install and get going with right away um but the main thing i really want to get across here is i'm here to support the full range of ethereum hackers at the event and so that means we're here to support like established uh projects that want to integrate filecoin uh into their existing offerings or toolbox or however they want to do that but also just single developers who've never even heard of ipfs or filecoin at all and our booth is right next to the ipfs booth so you can pop over there ask them some questions pop over to our booth ask us some questions and i really hope uh folks can find a nice easy entry point there so if you do have an established project you can get in touch um we're we're sort of beta testing some infrastructure hosted infrastructure if you want to do that but if you aren't established an established project you're just here to hack and explore then i got a little story for you so in addition to being a company that builds tools textile is also what we like to think of ourselves as a research lab and so we we started a research project about four years ago which we called textile photos some of you may have you know fiddled around with that and and experimented with us that was really an experiment to see if we could build an engaging consumer app in this case it was a photo sharing app on pure dweb technology so pure decentralized peer-to-peer on a mobile device so we set ourselves up you know for a pretty tough go and what we learned was that honestly the technology wasn't there at the time yet the networks and platforms weren't designed for it the user experience had to suffer just a little bit too much and it wasn't uh it wasn't fun for the user to experience that so we took a step back we started building and releasing all the tools that we would wish we had had in the first place and that was like okay you need offline data persistence got it you need to be able to share data between users decentralized database you need encryption okay and we turn these into a bunch of open source products which i showed in a previous slide and now a bunch of other projects are using those to build new apps a lot faster and i'm pretty sure you could build textile photos in like a hackathon now um the time that's the time uh that we've reduced to you know building a lot of these things and that is a pretty key metric uh for our team when it comes to hackathons and why we like coming to eat and other other events like this um we had to build a bunch of really lower level stuff before we could even get to the point of deciding that the photos app just wasn't going to cut it it wasn't going to be that user experience we wanted so a key metric for us and actually i think a lot of other teams in the space right now is something that our our ceo andrew calls time to failure i don't know if this is like a common term or not maybe it is but the idea here is essentially we want to make it as easy and fast as possible for someone to enter the ether general deweb developer community test out a new idea or concept and then fail as fast as possible and the reason for this is you know the web we've got right now didn't really magically just start with a bunch of successful companies it started with tons and tons of failures one after another until something kind of clicked and everybody kind of piled onto that and then that ended up you know maybe turning out to be not a good idea or a great idea or so on and so forth then we kept building and i think we're really at that stage right now with web 3 technologies we're ready to you know have a bunch more failures and hopefully uh see some stuff that uh that sticks so you know our goal really with the the hub is to give developers more time to try and fail stuff so that uh try and fail so that you don't have to you know spin up an ipfs node um and then uh try to you know figure out how to sync data between peers and then archive that to falcon only to find out that like your idea was actually not that great in the first place or oh you know what there's just no way that this is gonna scale to more than a local host dev environment or so on and so forth so the idea here is um you know that's what the hub is about and that's why this is is a lot about just developers who are just trying stuff out it's a it's a direct manifestation of our effort to reduce time to fail um so it's a cloud tool it's a centralized uh service that you can use to access decentralized uh web technologies and so you know at the most basic level we're talking about remote ipfs and filecoin nodes they're all uh hooked up together to allow you to experiment and uh you know for each event you know we have a free pretty we have a pretty generous free tier and then we'll just give you coupons and things like that so that um you know your technology and your cost isn't the thing that ends up holding you back right instead you can explore and try things and you know hopefully it's just that your idea wasn't that great in the first place uh no i'm just kidding i'm sure everybody's got great ideas and this will allow you to experiment a little easier so that's enough of that chilling preamble stuff out of the way how do you actually use some of this stuff so there's a couple ways that a developer can get started with textile stuff for the hackathon so i'm going to assume you want to leverage the hosted apis in this particular instance just because it's easier but if that's not you and you'd rather run all this stuff locally that's great too um if that's the case you probably don't need a bunch of slides to help you with that um but you might need some help along the way so if and when you do just stop by our our booth in shillzone uh fork one spork sport chill zone where we'll be there or you can join our slack channel or the discord channel or however you want to communicate with people we can help you get set up there but i'll assume you're gonna use the hub stuff for now um and uh there's kind of two ways that you're going to enter the the textile network through this one is um basically through team collaboration so you've got your hacker your hacking team you've got a cool idea and you want to share assets and code and even deploy the maybe it's a client-side dap or something like that together so you can go to hub.textile.io or just to go to the textile website and you'll get find all the links there to sign up you can log in you can create an organization you'll download the command line tool you can invite collaborators and then you can start leveraging some of the tools so you can create a new bucket for the project and i'll show you an example of this in a second you can create a new bucket for the project you can push data say it's web website static site assets or nft assets or something like that and then you can sync those um assets between collaborators in real time um you can make edits you can push you can sync um and then you can even archive that to filecoin along the way and so this would be great for a flow for a dow or something like that where you've got multiple collaborators and you want uh you know a record on ipfs of the of some sort of collaboration that's happening and then you want to essentially stamp for longer term storage that record on filecoin so that's one you know mode of collaboration and i'll show you how to do that in a second the other would be for developers who actually want to get in and start building an app and so here you can again download the command line tool or log on to the hub create an api key and that will give you the ability to use the textile hub as your ipfs peer and as your file coin um node and and all of the apis buckets apis and things like that that come with it and um you can embed those keys in your app and then what happens is is something kind of cool where effectively you the developer are allocating space on behalf of your users on the hub now that space allocated for your users is their space their they own it they'll always have access to they'll they'll always have the keys to get their data out they'll always have access to the data on ipfs or over ipns or over pub sub or over anything and i'm going to come back to that kind of concept of multi-protocol access in a second but anytime your users log in you can allocate new spaces for them using something as simple as a public key and we have demos for deriving public keys from um you know any sort of browser-based wallet or you can create a new random key if uh you know if you can assume that your users are onboarding fresh there's lots of ways to generate that public key identity for them to use and then you can actually start doing things like allocating buckets for your users which they own uh creating mailboxes that allow them to share updates um and even uh uh potentially flushing data to file coin so here's a here's a quick demo of the that that first flow where you're a team of developers and you're creating an organization this one may be called hack week um and you're going to uh actually push some data to a bucket so in this case we're initializing a bucket uh that was a personal one then you can initialize an organization bucket you can have it encrypted so that everybody needs uh it's encrypted for the org personnel only or you can have public ones if you're deploying a website and you can push data and sync that data between peers and all sorts of cool stuff like that all by running the hub buck daemon locally but one thing i wanted to highlight here and i don't know if you can see my pointer or not is um you can see i'm kind of uh moving the mouse over these links so these are all links that every time you update a bucket are live links and one of them you'll note is an ips link so this means that this data is available over ipfs and ipns giving you a static reference to dynamic data on ipfs why that's useful for adapt developers is that this gives you a static address that you can use um in things like um on-chain references to dynamic data so if you've got um you know some sort of nft assets or something like that that are going to be changing over time that are dynamic you can have a static reference to that data on ipfs or sorry on on chain somewhere or within your your adap ecosystem that is a reference to dynamic data and only those that have permission to write to that particular bucket we'll be able to update it so you can imagine all sorts of scenarios where you have access controlled access to nft assets or you've got permissioned access for subscription services and all sorts of other things like that and then if you start to marry some of these concepts with things like new cipher we have a bounty to kind of explore that and other you know access control scenarios you can start to see a lot of really interesting applications where you have data control over a decentralized network and a lot of that is driven through the default encryption that uh textile and threads provides so explore that take take a look at that and have a little bit of fun on the other end of the spectrum for those who want to build apps here is a complete uh example of integrating buckets into a simple javascript app now this is an example using node.js but it works equally well in the browser and you have access to the file like a files api um so that you can drag and drop you know nft assets or uh you know json files or any anything that you might want to conceivably dump into a blob or object store type api so it's very simple to get started basically you go and download the command line tools create a user group key so this is your your um your api keys and there'll be a secret part and there'll be a not so secret part put the not so secret part in your app don't put the secret part in your app standard uh web 2 api best practices so do as i say not as i do in this example here and then you can get going um in this case i've created a random identity or random public private key pair for my user but you might have that derived from a ethereum address or a ceramic account or however you want to handle that and then you simply uh authenticate with the remote thread daemon or sorry the remote bucket stamen create a new bucket as easy as client.get or create get a reference to a file stream of some kind and just start push pushing paths of data to a remote and you can specify the very fine grained path so you know the path to some file.

jpg or just uh generic names you could do it all top level it's very easy to structure how things get pushed um uh to the bucket you get very sort of idiomatic apis for working with files in javascript and we have go clients that feel like go clients and then you can of course get a reference to the links so you can see where is the store what's the ips address for this thing what's the gateway address that i might want to use if i'm pushing images and i want to embed those images in my app maybe i'm deploying a website can i c name this to the textile textiles default gateway can i access this over my own gateway there's lots of ways to access data and then finally that last line uh client.archive that is archiving that bucket to filecoin and that's using the embedded powergate api that's available on uh the textile hub and if you have questions about that please do reach out um it's doing a bunch of default uh deal making and deal orchestration and it's literally with one line of code archiving an entire bucket to file coin and so you can just sit back and wait for the hub to make a deal on your behalf and then access the deal metrics and all sorts of things like that afterwards i'm not going to go into details there but all this is to sort of drive home the idea that everything that textile builds may have one or two or a few ways in but it's always going to have many ways out and we we always take this multi-multi-protocol approach to all the apis that we build so there may be one or you know one or two ways in but there's always got to be a lot out and so you know you'll always be able to push data to the network from one place say it's textile or some other powergate instance and then retrieve it from any number of other places so every textile project is designed like this which and what that means is there's always one one more way to retrieve your data you can get it direct over peer-to-peer lib p2p you can get it over ipns you can get it from textile directly over our gateways you can get it over pub sub you can get it over good old http rest style apis um and in the case of things like buckets and filecoin you can even use the filecoin network to um to retrieve that data so you know uh having forbidden textiles ceases to exist you'll still be able to get all those filecoin deals by running uh your own um uh filecoin node or lotus node um and uh you can in the really great thing about this is you can also provide that functionality to your users so if because this is all content addressable data on ipfs um if you don't like the service that we're providing you know here's the filecoin deal for your data you can take it and run with it um and that's a big piece of the puzzle that's always there we hope you like you like it and stick around and use it because it works but no lock-in and the same applies to all of your users so that's a pretty powerful thing to be able to do so just a quick reminder here's a couple of things that you can start to explore we have bounties for buckets powergate threaddb our mailbox api and we have a bunch of joint bounties with other teams so for sure come build with us um we're always going to be available over our slack channel we've got a special one for east denver our boot our booth in the oh i think i spelled that wrong maybe sports chill zone um we're over on the discord channels as well and um we've got uh some bounties and things like that if you're interested in playing around and i think i managed to keep that pretty much on time or even short of time um so please do get in touch check out our bounty slide hopefully you saw some of that yesterday i'm not gonna go into the bounties again um today but uh that's me i'm carson with textile uh come say hi i'll be spending the rest of the day at our booth trying to learn how to dance so uh hopefully i'll see you all there