Revolutionizing GPU Access for AI Workloads | Greg Osuri - Akash Network

okay we do have some internet issues uh any anyway I can get priority okay let's do this great hope uh Denmar has been great great for you folks uh I suddenly got a bug now so going to take it back home to Texas my name is Greg o I'm the founder uh CEO for overclock Labs u a core contributor to aash network and today I'm going to talk about why crypto is saving Ai and why AI needs crypto and and not the other way around uh there was a recent story in semor which is a very prominent uh publication that talked about the chip shortage so turns out AI requires these accelerated computing chips called gpus and there a lot of people uh that are needing these gpus that created an artificial shortage and the story is about a Columbia student that's trying to do machine learning and trying to get some chips to do machine learning and he could go to Amazon and he couldn't get any chips and he could go to Lambda Labs or any of the traditional places that you normally go to get chips and he couldn't and then magically he founds this uh this decentralized weird Network and gets a100 for1 10 an hour uh so this is a story about how a a traditional machine learning developer couldn't get chips um in the traditional way but could get come to a web 3 Network and actually get the chips to do a machine learning training why because gpus are very hard to get on demand why because big cloud providers control GPU leasing so for example on Amazon the only way to get Advanced gpus is to know somebody that knows somebody and spend millions of dollars to even get access to the chips so if you're AI startup there's no way in hell you'll be able to get started without having millions of dollars of funding or whatnot right why because big tech companies are competing for these chips as you know Elon Musk is famously known for saying these words that gpus are significantly harder to get than than than drugs even though that is a low bar in San Francisco but you saw Mark Zuckerberg purchased about 350,00 h100 so these large companies that are doing AI are eating up chips left and right and they are creating creating this massive shortage for anyone uh of normal folk to get get access to the chips right and the options really are very limited to get the chips if you go to Nvidia directly it takes about two years I guess now it's three years to get h100s um uh directly from them so your typical option is to go to a providers that can offer you these chips on Le but guess what in order to get chips from someone uh like a Lambda Labs or a corv which happens to be a tier 2 Cloud providers because tier ones uh Amazon Google and Microsoft do not have the chips only tier 2s have and if you go to tier 2s you need to get into a minimum 3 to six month contract I think now it's like a minimum 12- month contract and you got to pay like 25% or 50% up front uh in some cases even 100% up front if you want to get in front of the line wait for three to to six months and get get the chips that you you need right so that is extremely frustrating for someone that want to do an experiment or of a new model that has popped up right and what are the challenges what are the implications of this uh this way of getting chips well number one high cost it's impossible for a normal uh person to get the chips without having to spend um you know millions of dollars um up front right so basically you're literally giving people money and and waiting forever to get the chips and that you know deters uh new new folks from entering the space and you know and most of the time when you order the chips you're ordering something that you know a requirement that that lasted uh 6 months right so the rate at which the models are developing is so rapid that your requirements change especially in a startup because you're highly iterative so what you get after 6 months may not be the exact chips that you want anymore right so you have this like massive underutilization of the the wrong type of chips because AI has moved really fast and and on the other hand if you look at it there are free gpus everywhere right what I mean by that is there a lot of crypto miners that bought a ton of gpus during the last uh last uh cycle that are sitting idle we saw consensus here earlier uh you know sitting on massive uh set of gpus company called Foundry digital which is owned by dcg uh is sitting on mass HMS amounts of A1 100s and uh and and 6000s right so there's a lot of these companies that were previously mining doing proof of work Mining and after ethereum merge are no longer uh you know having any use for these gpus and a sitting idle in these data centers and then you have companies that normally with with companies that are maybe three or four years older in in machine learning that's that's that's considered to be very old having older Fleet of chips say for example a company called stability that bought A1 100s uh 5,000 a100 about four years ago now up created their Fleet to something called h100s which is the latest Chip have a fleet of a100 available that have no use that they have any uh that they don't have any use for so there there's this enormous amounts of chips available and also a lot of machine learning companies when they do uh when they do foundational model training when they transition uh or when they upgrade their models they have some downtime uh between training runs because the intensity that you require in terms of usability for the chips is significant ly higher during training compared to inference which is uh a way you you serve the model so there are a lot of underutilization uh uh scenarios where chips are owned by companies that they that those companies don't want to let them go but still want to be able to get some sort of like liquidity on this uh supposedly uh owned assets right I mean there're also companies that are raising money with with gpus as collateral so gpus are the new spice right um and what is a aach is essentially a Marketplace to trade GPU Cycles aach is a first and foremost a supercloud a supercloud is a cloud that sits on top of private public hybrid doesn't matter any Cloud capable computer and gives a familiar user interface that is similar to a Docker compos uh interface it's easy to use and gives you the exceptional tools for for orchestration or fall tolerance whereas a compute comes from all over the place using a Marketplace uh mechanism where user users can request compute uh and uh if a provider can provide the compute can bid on the order and vice versa the way it works is very simple the user essentially uh creates something called sdl file places on the marketplace uh and specifies how much they're willing to pay for the uh pay for the workload and providers uh bid on that on that job and the Tenant eventually chooses the winning provider and then uh gets to use the computer right and uh this is real this has been around for for about 3 years uh and the first and foremost is permissionless so there's no permission required either from the user or the provider to uh to provide Supply it makes it extremely uh incredible for composability and it's peer to peer right so there is no middleman uh what you get is what you see and uh you when Once the lease is established the tenant and the provider gets into a peer-to peer relation and that makes it extremely uh uh reliable in terms of uh you know uh in terms of how uh you know when you don't have pessimistic way of acquiring resources and it's open source uh about 480 contributors contrib contribute to this uh project uh 20 people work full-time with overclock Labs about 90% of the community or contributors come from outside the community they meet regularly uh in something called special interest groups anyone can make a proposal there's an onchain fund that funds uh funds the development of the chain whatnot and it's it's been growing so daily active Lees uh these are a lease is an application uh has been has quadruple from from last quarter and uh that aligns very well with our GPU launch and uh the lifetime GPU utilization that means the uh the amount of chips that are available to the amount of chips that are being used a lifetime utilization is about 41% which we know it's extremely high either for a centralized or decentralized network so um and I'm here to do a demo because I love live demos and hopefully the internet gods are going to be my friends today um so the way it really works I mean I go to AOS Network and there is a link called uh deploy now which takes you to this console uh console is a fully non-custodial uh uh know UI interface I have a wallet here called Capital wallet I have some money in there I click deploy and I upload an sdl file like I mentioned earlier I'm here deploying something called a mixl uh 7 bil parameter model and I'm going to show how to do inference in real time on aash uh here the configuration essentially speaks about the container I'm going to use the ports how much I'm willing to pay and the commands I need to uh run right in create deployment um this goes on chain it's asking me to make some deposit and sign the transaction here um creating a creating order on chain uh in about 6 seconds you'll see this uh bids coming back uh for uh that will accept my workload uh you can also see the transaction online uh while the while the uh the bids are coming back you can see what's what gets posted here it's very anonymized information nothing about your application but just the resources you require you see I got a bunch of bids remember the company I was talking about Foundry taking which is dcg company uh they're they're competing to uh you know with my workload right here uh I can choose foundary because I know they're good I can choose any of these clouds I mean different price points different gpus I get offered Nvidia V100 is a fairly you know decent GPU I confirmed that I want that provider now at Le gets created on the blockchain uh all this is uh verifiable you can you can see underneath the uh in in the in the transaction history and in in a second here uh the provider should come online uh like I said this is how a lease looks like on on on the Chain uh the owner is the wallet address and the provider is the provider that bid on the workload and I essentially say I want uh I want to work with you uh and here if you're you can see this it's really uh bunch of events happening on the back end uh it's pulling the container called AMA uh and it's starting the container here and I can see the logs to make sure that I am seeing some activity yes it is pulling the checkpoint files checkpoint files are essentially the file that's going to serve the model it says I'm 23% here hopefully I have enough time uh left to finish this demo if not we can uh I already have something running but let's see I really want to see a live demo because there's nothing better than live presentation so um and you know while this is coming up I can you log into the shell I can you know run your typical shell commands you normally learn I will just show you Nvidia SMI uh which is a command to see what gpus I have uh you know you can see here I requested for a V100 I do have a Tesla V100 GPU that's attached to it let me check the logs here we're almost there it's almost there so this is real got 1 minute and 50 seconds I you know while this is coming up I'll show you another deployment I did uh so I have another server I did earlier to okay wait hang real quick I'm going to put this down just need to get this URL so I just replaced the um the URL with the URL I just got from here and I'm going to run this curl command curl is a way to invoke a web API uh it's going to take a second here because it's warming up okay there you go I asked mistl what why is the sky blue the color of the sky appears blue due to uh you know whatever whatever but you you get the point I was able to deploy a an llm from scratch in about 31 minute I guess like yeah about 2 minutes fully non-custodial fully permissionless Le fully with no signups no credit cards nothing right in front of live audience all right 30 more seconds and I can take questions how did you kubernetes gpus it was a very long process because gpus are inherently anti- uh design pattern to kubernetes because gpus are very very driver specific uh it's there's a lot of technical lingo in there but it it took a long time how many gpus on uh right now I let's see I don't know stats about 150 gpus maybe yeah 150 gpus but these are high-end gpus A1 100s high density gpus and we have about about 600 gpus coming on online every month uh starting this month on so keep an eye out and most of them are h100s A1 100s and a6000 and Akash will the only place to get h100s for $2 an hour which anyone that use h100s know what that number means and a100 for 79 cents an hour starting uh starting uh H more than 100s end of the March but h100s starting next week uh no comment all right time is upk you so much hello everyone uh my name is Ava and I'll be with you