ARG / Estuary.tech - Jimmy Lee (Cake)

i want to introduce our next speaker jim jim works with the application research group helps lead them at protocol labs and needs no further introduction i'll let him take it away from here hello how's it going um just share my screen so wow yeah thank you for letting me present about estuary at hackafest um i'm gonna give a little walk through about this project we've been working on since uh around like april march uh what is the arg though um we're we're building applications on top of amazing technologies in the bicone ecosystem and so it's uh it's me it's why uh chris elijah a lot of ecosystem collaborations to name a few textile fission arzora a lot of people from protocol labs a lot of technology that's made there is absolutely amazing and we're hiring so maybe if you want to roll with us and uh make cool software let me know what is estuary so i was thinking about how to describe bestiary and i think it's a story for the most reliable way to upload public data to the filecoin network and our methodology is we work in the open so we're hoping that others take from our open source code and just learn and improve on it i think we break up into three parts it's a hosted example of an esterary node um an industry node stack that anyone can run and a documentation platform to discuss and iterate with the community and we it's a familiar place i think for ipfs devs who work with http uh we support the ipf spinning standard and i think we we really tried to improve on the follow coin storage deal experience and i'll get into more of this later i'm going to give a walkthrough of the features but uh we reliably make deals against the falcon network at six replication so we make a ton of deals for better or worse and it's been a lot of fun um so we're running an estuary node at http you can use any language you want to interface over hosted api we configured it to be invite only we could change this in the future but if you don't want to get an invite uh web3 storage and empty stores don't require this kind of gatekeeping so you can use those today um we're testing optimal performance instructions uh and integration so as we make mistakes and as we get feedback we will continue to iterate and improve on what we're doing and someday you'll be able to run your own node and so here's some of the the i guess like the proof is in the pudding or you know like people want to see data in our performance so we've put over three million files um onto the filecoin network over through three thousand deals over nine terabytes um over 29 terabytes of sealed storage on falcon network and we're working with 96 miners the goal is to work with like a lot more if we want to uh so we're curating miners and trying to figure out which miners you want to work with and we have a couple power users one power user put over two million files another power user for over eight terabytes of data and we're just testing the ceiling seeing how far we can go and so i'm gonna give a feature tour of what you can experience when you look through the website and kind of like talk through some of the parts and then also go through a feature tour of what it's like to be a node operator and what you can see and and manage so we put commands everywhere there are commands for everything all of our documentation marketing pages so we won't really want to show that this is like a developer tool that you can just like kind of use about going to the gui the gui is kind of like the least important part of the product i think it's it's kind of um you know you can just run these commands wherever you are um you can use lotus to retrieve you don't use anything that we made and uh you know we have clickable links everywhere so you can see the files that you've uploaded and so here's one from the marketing page um we think it's really important that you can verify your cid so what i mean by verifier cd is that you want to know it's on file coin and you want to know uh which miners that it's on specifically and you want to be able to see details around that and you also want to see um you know like a receipt for that storage deal so you know how long it's going to be on chain and so we we created a whole an experience around this where you can basically you know make a deal see the deal successful see the start end date uh see the collateral see everything and i think that transparency makes easier to understand and we'll keep fine-tuning this experience so people i think are better get a better grasp of what's happening with our data we obviously want to add more like you know what location the miner's at you know compliance a lot of things for us to explore uh we added upload via cid so you know there's a a default way that you saw in the beginning of this presentation where you can just drag and drop files but you can upload a cid as well upload via api so our documentation is editable you can upload files from the docs itself so you can just copy and paste the curl command even and do it from the command line uh we want to we want to just be really open with all our process and make the whole thing a playground that people can play with we do public performance analytics so you can see basically how well this node is doing in the in the public so uh nothing's kept secret even our failures uh we keep uh a graph so everyone can see like the performance um and we also keep a list of every single miner or sorry provider that we work with and you can see the stats deals and errors and if you go here you can see uh i'll skip ahead you can see some of the logs that we helped show so this makes it really easy to work with the providers on bugs and figure out what's going on we can skip a meeting uh makes things really quick so uh we have a cool thing around batching uh batching is kind of an example of what happens when you know people are uploading like four kilobyte files or something really small like small text files we we have a way to automatically bash that into a bigger deal and the reason why we do that is because every miner or provider has in their maximum p size minimum piece size kind of like a maximum little tag and a minimum will take so we're still explaining the best way to do this but we don't want to overload miners providers of a whole bunch of really small padded files so this is a great way for us to kind of bash things together and make a more valuable deal and uh we provide the provider details on estuary so there are public pages and in the future um you know we basically can show error logs we can show price we can even share like compliance and a whole bunch of other really cool things and so see how much time i have left um cool so now to get into tools for network operators uh as we build out those things for kind of like the client experience for network operators we want to uh you know we we we're focused on stress testing the capability of our note today so whatever we achieve uh we will provide the steps for you to do the same and so uh horizontal scaling with shuttles it's a cool new invention uh i'll get into that more later or you can talk to me about it offline lock store storage capacity eventually inventing new techniques to store more data on a single node um faster rights and reads and uh built-in file coin address support so you can run your own verified address and make deals on your own you can even just use a normal following address if you're willing to spend file coin without making verified deals and i think alan covered that really well in his last talk and if you want to talk more about verified addresses we can talk about offline as well and so the one of the coolest things as a network operator when you're running your own estuary in the future is that you can curate your preferred providers and you can see all the logs and you know have a relationship with them where you know why deals are failing or succeeding and if there are problems you know and you don't you don't have to make deals against all of these providers you can basically just you know suspend them for a little bit uh it'll show up on their profile page and you can basically talk about the issues and then work through them together and then re-enable them so i think this is kind of like a a step in the right direction with when it comes to like a relationship between a client and a provider where we can debug things together and figure out what's the issues the data transfers network issues etc uh you have a file coin address if you run your own estera node so you don't have to verify it uh we have an address right here has a balance of file coin and um this address is for holding we can move that amount to escrow and whenever it moves into escrow gets used for uh making a ton of deals and so um you know you can you are in total control here and we'll add a lot of uh um different features that make it so it's much more manageable you can set limits you can maybe like even limit by user um and so we're still exploring this and we have great ecosystem partners that we're working with to help figure all this stuff out um shuttles are like a really really cool invention um i i know that like there are a lot of clients out there that have a ton of data like it's just not easy to move sometimes petabytes over to wire and you probably shouldn't because if you're using uh the cloud you know you'll probably pay for network ingress on that and it's gonna be very expensive and so uh we are like the the shaw's like an option before offline deals where if you really don't want to do an offline deal what you could do is you could uh have us create a shuttle really close to you and then you could upload your data directly there and maybe save on some of the ingress costs and so we're experimenting with this right now and uh it's been great like it's actually helped people who are in europe upload a ton of data instead of like going through you know us east one or whatever um there's a lot of user debugging tools that we're going to be releasing as well so uh network operators can debug problems for clients by impersonating them i was kind of hesitant to show this feature because i think it it's kind of like it puts a lot of responsibility on a network operator to be a good actor but at the same time you know we've had users have really stress tested estuary and it was really nice to be able to go into their account with their permission if they gave us their api key and kind of like find the bugs at like you know a million files and see why things aren't paginating correctly and um when you run your own story node you can actually uh make your asteroid node open to everyone so uh or you can make a restaurant node uh like as a service like invite only and so we chose to be invite only for now and so you can see this page where we can generate invite keys and we can invite other admins so right now s3 that should be managed by a few other people to you know create invites and allow people to come use the product the last thing i think is uh one thing that's really cool about running this asteroid node is that you can also see the hardware system analytics and so this is a really bad screenshot it doesn't show all of the details but i really want to know how things fail i think that's like a great step forward to figuring out you know how to improve your product and so we keep track of the failures check the deals attempted and we also have a hardware stats so you know how much uh hard drive space is being provided by the box that you're using and i think as we fine-tune this uh we'll give users a lot of control over how they want to run our estimate notes so you hypothetically you could run one on a cheaper machine somewhere and you know you you as long as you curate your providers and users in a certain way and set expectations you can even support file coin deals that way without having to have a super robust system that we have and so uh yeah that's um that's pretty much a quick overview of all of s tray hopefully i didn't go through it too fast but if you have any questions about it uh you can ask me and i'm gonna play actually a little teaser of um what's to come in the future so we are we are making um videos to describe some technology that we're creating to make estuary work and so in the future you'll see videos from the application research team that uh will look like this and they'll have a lot of different examples of you know what to uh like how to use the product how to get an invitation um in i think like what we actually really need are more technical deep dives so we'll probably do some more deep dives on the stack and how to run your own uh and all the different modules that we've built such as like phil client and etc so stay tuned for more videos that kind of walk through how the product is used and how the stack works and uh kind of like even some discussions around like how what direction we should we should go in the future so yeah i placed through that pretty fast um stay tuned and uh yeah i'm happy to field any questions