Storj 🛠 Getting started with Storj

[Music] hello welcome to the getting started with storage workshop my name is dan willoughby and i am a developer advocate for storage let's get into it storage is a decentralized cloud object storage for video images backups nfts d-apps etc we're also known for our high performance what this workshop is going to show you is the types of web 2 storage compared to web3 what storage is and how it works and then i'll give you some demos of our simple ipfs pinning for nfts our uplink cli demo for direct web3 storage access in our aws s3 compatible so in web 2 we had multiple options we could have on-prem or different types of durability in the cloud and then we have web3 which is decentralized so on-prem means it's a hard drive in your laptop or in a nas device it's relatively fast it but it can be a little bit costly and you always have the maintenance and upgrades to deal with of course it's it's quite risky because you only have one copy of your data in single site you rent hard drives from backways in wasabi and you rely on them to make copies of your data they use raid technology so they'll have one to three copies of your data and it's generally inexp but they have minimum storage day policies and free egress which if you fit within their fair use policies so one of these hard drives may fail but they'll build it back up so your risk level here is relatively low but their entire data data center could burn down and you will lose your data single region is what most people are used to and which a lot of the web uses nowadays amazon s3 google and azure store your data in multiple data centers within one region such as europe or like virginia or oregon you'll pay a lot more for the stores about triple and they tack on hefty egress fees and they also charge for apa requests so some people like to try and get around those hefty fees and they use like a glacier option which is very low cost if you don't retrieve your data and what most people don't realize is the retrieval fees often outpace what you would pay on just the standard aws tier and at the top of the line azure offers a multi or amazon offers a multi-region where you can have your data say in virginia oregon or europe and you pay for each of those different regions it's really durable great reliability but it's also very expensive the risk here is you'll burn through your money really fast so decentralized is also multi-regent so when you upload a file it'll be stored globally in say europe united states and asia your file is broken into little pieces and stored across the globe you'll pay much less here egress is very reasonable there's no charge for api requests and the code is open source also you notice there's a little tree there it's more environmentally friendly because we at storage encourage people to use existing unutilized hard drives when they join our network let's zoom in a little bit on the numbers for a region comparable to the highest tier of amazon you will pay four dollars per terabyte per month whereas amazon is 23 for one region and you double that for each region that you go up also bandwidth is about 10x cheaper you'll notice availability and durability is very similar how about the cheaper options well wasabi says free egress but that means you can only download your entire file once per month and backplace has a bandwidth of 10 they also tack on other fees and of course multi-region is five to six dollars per region whereas on storage it is free so in summary if you want to protect your data you're gonna pay more money but with our decentralized technology you don't so how does storage work storage allows anyone with unutilized hardware to join our network called storage node operators when they store our customers files they're compensated fairly for doing that so we generate demand by creating applications such as our s3 api compatibility layer we also have a library called uplink which allows um users to interact directly with the storage notes and then we have satellites that kind of does a little bit of the in between work to make it a little easier to use that ensures that access controls are managed ways to pay the storage node operators and ensure the data is reliable so what happens to a file when it's uploaded it's broken into a little first it's encrypted and then it's broken into a a bunch of little pieces like 80. those 80 pieces are distributed around the globe to independent storage node operators on its way back those nodes are identified the segments are downloaded put back together and decrypted and you get your file again multi-region is included by default performance up to 9 to 24 gigabits per second you'll see availability durability similar to aws s3 but let's not forget security and privacy you'll have ed n10 encryption on edge or in the client library so not even storage can read the contents of your data so how does it work you see in this little demo i have here that pieces of the this file is stored throughout the globe this one happens to be a 4k um 60 fps video of big buck bunny so let's go ahead and see it stream i see i've hit play here and it's now streaming but i can also hop around and it will reconnect to those various storage nodes in the network and start streaming those pieces and reconstitute this file and then over here is a little picture of the graph of where these pieces of the file are coming from so we offer a few ways for people to easily get started we have our s3 compatible api which acts as a great web 2 bridge of the tools everyone is familiar with we support server-side encryption and is globally distributed and highly available the client side integrations is more of what web3 is used to with end-to-end encryption you own the private key and you encrypt your data and it also ensures maximum privacy so let's get into some demos here i want to show you our decentralized nft storage using the ipfs protocol to get started you go to storage.io ipfs on this website you can see a quick demo of how it works i'm going to upload a simple tree picture and there you see it's already done and here is our ipf gate ipfs gateway so you can see that here your file your image is displayed so this works great for nfts or any other files that you need to store in your d apps or say you have a video stream so let's also load this in brave browser there it is but you can see it also works with curl so i will also do that i have a picture of myself here so let's get this code and i'm going to upload this file here it comes back so let's view that in ipfs browser and see if it works there you go it does so what if i wanted to use these files say in my react app so let's go ahead and do create a new react app this is going to to get us a bootstrapped react app really fast um with the things that we'll need to show this here and as that's playing we will make sure we have the file so you notice that i just copied this url here which isn't our ipfs gateway url and then i'm going to start a tmx so that i can run the create act the create react app all right and it is on my other screen so let's pull that up here so let's say i want to i want to load this image into here so let's say image source equals that url i just copied and saved that bam well that's a big tree so let's go ahead and add some style say width equals 100 that's a little bit better and let's get rid of some of this other boilerplate i'm going to say this is my my storage tree on storage ipfs great so that's how you get started with storage ipfs so next let's see i want to show you how to interact directly with our web3 storage notes and that is done with the uplink cli so if you go to docs.storage.io you'll notice we have a quick start so what you'll need to do is first sign up for storage and confirm your your email and this just helps to have a nice interface to the storage notes and then next we'll want to install the uplink cli to install the binaries and i've created a linux instance here where i'm going to run these commands so there i have downloaded the first binary and then i will unzip it and then let's go ahead and install uplink to our user bin so that puts it in my path alright so next we will want to create a bucket so uplink make bucket um sj for storage and let's call this eth nyc web3 oh so i haven't configured my access yet so in order to do that i will need to go to storage.

io sign in and i'll sign in here and i'm going to go ahead and do this quick start using upload using the cli so let's call this ethenyc i'm going to give out all permissions but you can limit the permissions to certain buckets or to download upload or delete and specify how long it'll last so here is my secret keys don't worry i'm gonna revoke these but i'm gonna go ahead and copy these to a file on my server for future access and then i'm going to run uplink configure setup go ahead and name the these credentials the default is main and then this is where the api grant comes in satellite i address is here and then a passphrase to encrypt this data client side all right then it's also asking me if i want to generate s3 backwards compatible gateway credentials i'm going to say no we will do that later all right perfect so we've already installed our uplink we've called setup so now let's create our bucket uplink mcbucket um let's call this eth nyc web3 all right so now i can ls see what's in that bucket there's nothing so i want to do is that big buck bunny video that we just watched i want to upload that so um let's do that so uplink copy big buck bunny to sj eats nyc web3 oh and i want to time this all right let's go ahead there it goes it's taking a little bit but let's say i want to make it go a little bit faster so what i'll need to do is depending on how much cpus i have i can add some parallelism so let's go ahead and say parallelism [Music] well i have four cpus so let's say eight so there it goes again much better so the parallelism comes in handy when you're uploading very large files in this case we are uploading a 642 megabyte file which took about 28 seconds which is fantastic so now let's go ahead and get a link to that file the way we do that is we call uplink share and i want to make this public so i'm going to say surl and then i'll need this file path and i'll tack it on there and there i have it a url so let's go ahead and look at that and it looks like it was stored on 833 storage nodes and there you have the distribution of on the right here of all those nodes and let's say click play there it goes so here we go a freshly uploaded file on the network streaming 4k 60 frames per second in real time pretty awesome uh uplink also has various other commands you can remove buckets you can list your files so let's see that the contents of our file or our bucket sorry so there's our big buck bunny file you can server side move and you can remove objects so very similar to s3 so that's a great segue into our last demo our web 2 bridge aws s3 compatible api so in order to get started with aws the tools that you're used to go to docs to storage.io again and click on this aws cli quickstart and we'll start again by generating a access key so let me find where i put that okay so i want to create this time i'm going to call it aws cli oh i've already done one of those let's do eath nyc and i want to continue in the cli so let's go ahead and say um uplink setup let's call this aws cli copy that api key in here satellite and our passphrase to encrypt the data and this time i'm going to say yes i would like to generate these and these are my credentials which are secrets i'm going to copy those into this very secure file on my server you'll of course want to make sure your secrets are encrypted and stored securely but for demoing purposes this is okay all right so now i'll want to set up the aws sdk so here i've gone to this link but you can easily find by saying how to install aws cli i am going to download the latest version and unzip it lots of files go ahead and install all right so i should have aws so let's go ahead and open tmux so that i have multiple windows i'm going to run aws configure and i want to cap my credentials file access key id is that first value generated the secret key is the next one region name you can type in whichever and output format of none is okay here so now that we've done that we can go ahead and look at our buckets so in order to use the aws cli you'll need to specify the dash dash endpoint url with http slash gateways dot storage share dot io and then we can do a ls command okay maybe we can so let's go ahead and make a bucket hmm let's run aws configure again make sure i copied those correctly sometimes i copy them in the wrong order oh you know what i have done the wrong thing there just api key so we will want to make sure [Music] that these are correct and now we should be good 8th nyc aws okay i must have fat fingered copying that in there so um you'll see that it created that bucket and i should be able to list the contents up there which are none so let's go ahead and copy that big buck bunny video into there and let's shorten the name this time to bbb mp4 and now it is going to copy that file through our s3 compatible gateway web 2 bridge into our distributed decentralized network now it's not fully utilizing that parallelism here so it's not quite as fast as before but with some configuration you can get the speed a little faster once this is complete i will show you this file in the browser okay so we'll go ahead and view that bucket and i always have to type my encryption password and to see the contents of my files and there you have it the uploaded file so there's a lot of other things that you can do on storage i just head to docs dot storage dot io slash dcs we have very many detailed tutorials on how to get started here you can see our ipfs pinning sample code so i hope that you have a fun time hacking and i will see you when it's all over thank you so much