🎤 Overview of Estuary, the decentralized storage service trusted by 150+ TiB of data clients

foreign [Applause] [Music] we're gonna jump into our next chat here uh which is going to be reviewing another ecosystem uh if I get the slides to work here another product in the kind of protocol Labs ecosystem so I'm going to bring up uh Karine and uh Alvin from the Estuary team um there we go um so yeah so they're gonna chat about estatory and uh and how it's being used uh entrusted by 150 I believe this is pronounced petabytes of data clients um so uh yeah I mean I'll just jump it over to them um another awesome chat with another ecosystem or another product within the protocol Labs ecosystem so um I'll let you kind of yeah uh show your cameras and then I'll get my screen share out of the way um and then feel free to uh to jump into it nice to have you here with us yeah thank you so much let me just share my screen a second all right perfect all right so hey everyone I'm Karine I'm a product manager here at protocol labs and I'm working on Estuary and astero is a reliable way to upload public data onto filecoin and pin it to ipfs and you can find it at story.tech so the outline for today's presentation is as follows the problem of centralized storage why Estuary how Estuary Works a thousand foot view how do developers use Estuary estuary's performance and the future of estuary so firstly the problem of centralized storage so first centralized storage is storage that is owned by a singular entity or organization and this causes many problems because if you store data with this one entity um and that energy decides to go down for the day then you're not able to access it and this also has other problems such as accessibility worldwide there's a limited amount of data centers so that also also causes problems and there's also Monopoly over storage because there's only a few different storage and Cloud companies so this allows them to dictate prices allowing them to put them Sky High determine who can access this data and where it can be accessed worldwide now a solution to this is actually the decentralized storage model that filecoin is based off of um so this allows for an open market for storage and a peer-to-peer network of storage providers and this peer-to-peer network is between a client and a storage provider so a client is an individual who wants to upload their data um and a storage provider is an individual or an organization that's able to facilitate this now the great thing about filecoin again is that storage providers are able to set their own prices hence it's an open market so in comparison to the centralized storage systems where you only have a few different Cloud companies to choose from and a decentralized model you actually have multiple Cloud companies um yeah so they're able to set their own prices and the client themselves are able to shop at through and see a whole different um a whole different array of storage providers out there and how this works is that um storage providers actually um to prove that they're having the storage uh they have to provide cryptographic proofs and this cryptographic proof means that they're historian the data of applying correctly and that proof that's put on a block to the network and the deal between the client and the storage provider is actually replicated six times to ensure that the data is available worldwide so why Estuary now I started actually solves a lot of the issues with file claim because it's actually a little bit difficult to use it it requires you to spin up your own ipfs node and to do this you need to know how to do so first of all and the computational power necessary so asteroid itself is its own specialized ipfs node that protocol Labs runs and basically what it is it's a brokerage between the client and the storage provider so let's say you yourself are the client and you want to store data on filed mine all you have to do is just ask a story hey can you make storage deals for me and Esther will automatically start to do this um so how this works is that you will have a piece of data and this data has an Associated CID and the CID um once the deal is made is put onto the filecoin network so that's really important for retrieval later on and again um this this uh deal the client between the deal between the client the storage fire is replicated six times also while the deal is in progress um your data is still retrievable off of the Estuary node because it is pinned in there for hot retrieval and once the deal is made then it's available for culture people and here's a little bit more of an in-depth visual of the architecture here so I'm a lot of the clients and then in the middle you can see the Estuary main node and then we also shuttle so you can send the deal request through the shuttles which eventually goes to the main node and then makes deal with the storage providers and these storage providers later on can apply to become Miners and that's just an additional incentive layer for them so as you can see the X-ray node is actually quite complex and it's using two protocols filecoin and ipfs um so yeah many things that go into it and if you want you can look at the code base yourself but if you're just a developer looking for something a little bit easier you probably just need this Mrs usray.tech this is our web UI and it's super easy to use all you have to do is drag and drop your files and Astoria automatically starts making deals on your behalf and you can even check out your deals at s3.tech on the deals tab so here you can see your Miner ID The Proposal receipt the filecoin dealer receipt and the status of the deal and the moderator ID usually starts with an f and that's important because it tells you who exactly is storing your data and in the fellow Queens you'll receive you can see this minor ID along with your CID now as I mentioned before the CID is super important for retrieving your data so let's say this was a traditional pending service um if you have your PIN you can achieve your data no problem but let's say that pin is somehow lost well that's a big issue because now you can't retrieve your data anymore but what what's so great about us array and filecoin is that even if you lose your PIN you're still able to get your data back because of that CID so since file coins also appear to peer Network you're able to ask any storage provider hey I need the CID can you go ask your peers of other storage providers to retrieve it for me so it will find someone that eventually has that CID and it will come back to you even though this this is like slower you will still have your data back for sure and Estuary makes uploading data to filecoin and replicating that data around the world a five minute problem so as I said before the deal between the client and storage provider is replicated six times what's so great about this is because if the storage provider decides hey I don't really feel like being on the network anymore well you still have five other storage providers that have your data and as well you're able to access it all around the world and here's a good representation A visual representation of That So This Is Us right now this is the world here are the fall coins storage providers here is the Estuary nodes are able to talk to these storage providers and here are the deals and as we get more and more deals this the data that we have keeps replicating across the world more and more and more so now Alvin's going to talk about how do developers use Estuary all right okay thanks thanks for that Tony uh thanks for that uh I'm not sure if I can share my screen uh maybe you need to understand first and then I'll share my yeah thanks for that yeah okay hold on uh there we go right slideshow right yeah okay all right so so yeah I mean um so first yeah my name is Alvin so I'm the software engineer uh here at BL and I'm I'm I'm one of the software Engineers uh maintaining and supporting Estuary and uh yeah I'm gonna here to give you an overview on how to how developers can use Estuary like for their development for building their apps so I think uh Korean has mentioned earlier about the benefits of using Estuary uh being able to upload data and sort of like replicate that data six times over on the FICO network network and then make it sort of reliable right and achievable so there's a lot of mission critical applications that anyone can use or can can build on top of asteroid right so it's easy to say that yeah you can build sort of a uh an app that relies that an app that has data Centric data center cap data the the titles are out of data and SRI is a perfect it's one of the perfect like you know platforms to do that right so but before we even go to the examples and how developers can use it or how UK developers could use it it's important to understand the architecture so SRI has three main components the main node uh the shuttle and the front-end application so the main node is sort of like the main broker node that sort of uh um that's already acts broker between the clients and the file coin Network so when you upload a file on front end it depends on the domain main node and then it facilitates and manages the creation of those six deals six file coin deals six replications meaning and then it presses that under the file coin Network to the storage providers and the shuttles is basically the way for this main node to scale so if you need more like storage if you need more processing power if you need to process more data you can actually scale the main node and add more shuttles to it right so actually currently the main Estuary production environment currently has six shuttle nodes so you know we have managed to scale to manage to support a lot of files by scaling our main Network main node um and the lastly is the front end which is basically the user interface for any user survey or for any consumers to upload their files onto SRI so I think this is one of the demonstrations that Korean um showed earlier wherein you can any users can easily drag and drop files onto the front end and then every all of these replications all of these facilitational files to the file coin so the file core network is done on the back end so uh so the what's good about you know Estuary is that when this was built it was built with an api-centric mindset meaning all of the core features or or like functions of it are are are accessible via API endpoints so meaning if you upload if you upload a file a car file or even if you're retrieving a list of cibs all you need to do is all you need to do as developers is that you just need to call a specific endpoint uh passing a token an API key and then you know you can retrieve that from your Edge from to your applications so technically any platforms and platform that understands HTTP and can also sort of parse that HTTP response can literally use estuary right so here are some some of the examples right so example one file upload so all you need to do is really just uh call the endpoint content add um and then pass the better token authorization so this is a key that you can actually retrieve and they can request the invite for it for and then you just need to pass the data and then you're good to go right so just file upload just you know just pass that data and then your your overall set now of course you wanted to have you wanted to wait you wanted to to have a way to access those files right so there's also an endpoint for that so just call this endpoint pass the better token then it will retrieve you back all of the lists all of the files that you uploaded let's say cids so similar thing with car file so if you're uploading a Content article file on the network so you just need to call the the endpoint passing the better token and then passing the car file as part of that data body and then you're good to go so everything is endpoints even the collections so collections is one of these features SRI features wherein you can tag a specific file and then sort of like Aggregate and collect them right like sort of like creates a creates a group right for each files so it's like a hashtag that you can tag a specific part so that you can collect them or so you can group them and then sort of just just use that tag to identify right the specific files so again for creating collections all you need to do is call an endpoint pass the better token key right and then your your all set uh here is a I think here's a demonstration a quick demonstration of how it works uh I'm not sure if I can play it but I will I don't think I can play it sorry hold on let me just check if I can uh sort of okay all right okay then let me play that oh okay so again like as I uh I mentioned earlier um all you need to do is just um call this endpoint so this entire demonstration this entire page is uh is a is a way for you it's a way for anyone to just test out the endpoint uh you know the then we choose a file and then we upload the file um everything happens on the map so it calls that endpoint so API the sui the tech here slash content slash ad here so you uh and uh and here we're we're now listing our files right so um all we need to do is just uh replace that better token with uh with a legitimate like with a with a with that authentication token that we that anyone can get if you if you request an invite uh and uh everything is going to be retrievable right so there so uh yep okay let me just go back to my slideshow all right now it please hold on yeah okay so now uh once you're once you're done with uploading your file you can actually verify your CID so uh if if you uh we do have a front end as I mentioned earlier the front end that allows anyone to just uh to upload their files but there's also a way for you to verify the cids that you uploaded to Sri so you can verify the rcib using this tool and then it will look up onto this to this uh DNS endpoints here the web.link and ipfs.

io so of course there are a few more apis that you can take advantage of as developers so collections API as I mentioned earlier you can use it to group files together and sort of tag them so that you can retrieve them as a group uh there's also a deals API uh to access all of the existing deals uh any deals that was that was created for your cids uh the storage provider and minor apis so we sort of like an admin endpoint to grab or to get all of the storage providers registered on the system and of course admin and system apis sort of like maintain and maintenance and support apis to solve to uh to uh to maintain the Estuary load foreign so of course like all of these are since these are these are accessible by HTTP API endpoints right so it just makes sense to build Tools around them so sort of like data wrappers so that's what that we actually did so we created sdks in libraries uh that uh we uh that sort of wrap all of these API endpoints and the beauty about it is that it was it was you know it's written in uh in Java JavaScript rust python c-sharp so all of these are mainstream programming language so if you have if you're working in for example in an application that's written in Java you can just import that module onto your Java application and then call these apis through the through the Java library right so all of these are available in our GitHub so everything is open source and everyone can access it anyone can just grab it and then just use it for their own uh of course like if they're uh since we we also documented all of the API endpoints using swagger so Swagger is an API documentation tool but uh the the the great thing about it is that you can actually use that documentation to build stubs so applying SDK stubs and server side stops so for example if you have a specific use case for let's say flutter app right so let's say You're Building A mobile application in plutter and you need a dart SDK you can actually use a swather docs to generate the dart SDK for you so here is sort of like some of the key examples using the code generator you can generate Android sdks Apex closure uh there's even c-sharp Dart flash right here and then groovy and Java so everything is so everything is compatible with the code with swagger code generator right so the apis or the API key is only by invite so you would need to request access from the development team so if you scan your QR code if you scan this QR code right now you can actually it will redirect you to the request invite page wherein you can register uh and then we'll uh we as a team will have will will check and we'll uh we'll uh we'll assign you an API key so that you can use these API endpoints so yeah so Estuary is open source so every source code the node the sdks the 200 libraries uh even the complementary tools that uh that that it has are all open source if you go to application research GitHub page you will see all of the projects here and anyone ever anything anything is anything is everything is free so you can just if you have sort of like an idea how to extend it you can just clone and sort of create a PR against the against the repository against a project so if you're sort of trying to build an innovation on top of five coin and you want to use Estuary right to Estuary to you know to to have your own use case or to build your own project you can definitely do that just clone extend and then you're good to go yeah so yeah I mean every all of the developer documentations API docs are available in our documentation website so that's docs.suary.tech you can scan this QR code to visit that web page as well uh and of course if you have any questions about the tech uh our net team is on slack um we have a a slap Group which is open to anyone this ecosystem.stev and uh we are if you have any questions about or anything that you need assistance where we'll be happy to assist you uh yeah that's uh that's it yeah I give the floor back to Korean for the next few slides thank you um oh then you can just click for me or okay sure I'll share my screen oh okay yeah okay cool we're back so S3 performance um we have over 18 million total files uploaded by all users using the primary estra node 165.93 terabytes a total pinned ibfs storage for a hot retrieval from the Icon bus Gateway over a hundred thousand active storage yields on the follow coin Network 971.

59 terabytes of total sealed storage on file plan including replication and lastly we have 176 total storage providers receiving deals from RS Ray node so some really good starts and then as you can see the trend is looking good for the amount of data that's been uploaded onboarded onto the follow coin Network and this data is since April 2021 and as you can see here there's a really great upward Trend so the network is definitely growing and now to talk a little bit about the future of estuary um here are a few of our goals that we want to accomplish um we want to make it easier to onboard data to follow point and what this means to us right now is that we want to improve our web UI for other users as well we want to make it very easy to use and a story right now is actually going through a little bit of a redesign so that is very soon to come we want to be able to onboard more data and users so some examples of our users are listed below and we want to be able to accommodate larger upload sizes right now um in a singular upload x-ray can only accommodate up to 32 gigabytes so we really want to stress test that and see if we can break that barrier we also want to expand the amount of projects using Estuary broaden our network of storage providers and we want to have more online deals like on the network and then our biggest goal I'd say right now would actually be Auto retrieve so this is kind of completing the picture of a data retrieval so if you were to lose a pin on Estuary it would be really great to be able to retrieve your data so that's basically what Auto retrieve is doing it is about 95 of the way there it's almost completed and I'd say would be done within the next month or two so that has been the major push as of now So yeah thank you so much for listening to our talk today and as Alvin mentioned earlier we're all open source you can find us at ecosystem.dev on filecoin slot please check out our GitHub if you want to contribute you can find that at application.research Estuary and again if you want to use Estuary go to s3.tech and please request for an API key so thank you so much everyone and we can take some questions now yeah awesome thank you so much folks um just loading up uh the channel to see if there were any questions um I mean maybe in the meantime uh if you could highlight maybe some of the you mentioned some of the big companies that are using estatory but um maybe just kind of some of the cool ideas that that come to mind for ways in which people can can leverage this kind of Bridge to to build some cool projects during their next two and a half weeks I don't know if you have any good ideas or any any outfit you want to leave for things that people can build um but uh but yeah I would love to hear any any ideas you have for for a bunch of hungry hackers yeah I guess like I I guess like you know I mean one of the key benefits right of using Estuary is the replication right whenever you upload a file on Estuary it replicates on the public network so if you're building sort of an AI machine learning app that uses a data model that you don't want to be tampered with that's actually a good use case because when you have that when you have a reliable storage like fileco Network right like for example using destroy as a broker you can easily you can you know you can sort of like ensure yourself that that data is there it's prep it's replicated it's retrievable and it's also verifiable right just because we have the content addressable storage technology there so yeah if you're sort of building a an application I would you know I would I would look at building an application that really that uh that relies on aggregation of a lot of data like geolocation uh and then you subscribe to store that or use SRI to yeah USS to store all of those information so yeah I mean that and that's that's virtually any any applications right like any application any application that collects data that makes sense how about you Korean I mean yeah basically what Alvin says like anything that's really that you need to onboard like a lot of data S3 is definitely the best step for that um yeah I've been really hit it on the head there yeah uh yeah I guess I don't know machine learning yeah yeah just it's just because like in in this in the AI machine learning space there's there's a lot of data sets right that you can take advantage of and I think one of the key things really in the AI in in the in this data sets because I I work for like for a company that that does this um it's the reliability of that data set before you feed that onto uh the machine itself okay so that's sort of like a supervised learning uh data model so you you create this whole bunch of data right you you Source it from different like sources like you know different different uh channels you aggregate it and then you store it somewhere and then you use that later on to create uh data sets or data models that will then be fed to chatbots you know chat Bots that you know response to customer requests customer increase or to maybe uh an application similar to Amazon e-commerce website right wherein it tries to predict what your what your what Your Trends are what your what Your what uh what are the trends or what are the things that you buy or a person buys during this season right so so you know those are those those are the things that I would look at uh those are the solutions that I would uh I think that's very impactful and very good and it's very it's it's great to have those you know Solutions I think yeah awesome well yeah this has been amazing thank you so much for the whole overview of just how far our estuaries come over the last little bit uh I've been amazing to see I've been amazing just to see sorry um and and yeah I really appreciate having both of you here uh to give us that quick overview and um looking forward to seeing many projects built using Estuary um and uh and and the world that you can have when you have this persistent data so yeah thank you so much folks and uh I think we're probably gonna move on to the next talk thank you thank you thanks thank you so much