Decentralized AI: Video Infrastructure, Tools and Applications | Shannon Wells - Livepeer

[Music] live good day ethereum Denver 2024 my name is Shannon Wells I'm head of ecosystem at live Pier um you can find me at sham vasion on Twitter or telegram um I'm going to talk to you about four things uh today this is a QR code to join a live Q and a chat in our Discord with our technical team including our co-founder and head of engineering yandan Fu who's working going to talk a lot about um today I'm going to go over the AI boom I'm going to go over the live pure protocol which is all about decentralized video infrastructure I'm going to go over our AI video road map and then we're going to talk about what you can Biddle towards the future of AI video so what is the AI boom well it has a lot to do with compute and that uh I'm going to quickly run over for those of you who might not know um kind of AI 101 and inference versus training so um training training AI is super compute intensive um the process only needs to be done once but um the the process of you know trying to train a model to be able to identify an image as being one of many categories in this example it's a bicycle a person or a strawberry um with multiple images being fed into a deep neural network weights being used to update and correct for errors um all of this takes a lot of time to start to so that a model can make predictions with with accuracy inference on the other hand you know on once a model is trained that happens once inference however is ongoing and if a model that's been trained is actively in use um it is constantly applying its training to new data and making it additional inferences and this takes quite a lot of compute power and can be very expensive and inference has really um what's taken up most of the AI video's boom over the last year and video interestingly enough has been a huge uh part of that this is a um a diagram from a from andrees and Horwitz whose um consumer app arm has been spending a lot of time trying to map what's happening in AI I encourage a lot of people to check that out um just a a short while ago a few weeks ago open AI who everyone knows about Lun Sora which is its um text a video generator so there's a lot of um a lot of interest and demand people using AI um for training and inference but the underlying requirement for all of that AI um AI usage is compute um and and it's actually gpus that um provide the hardware that's possible to um to make that compute possible and today there's really only one um one provider of compute for AI processing and that's Nvidia um and this is a chart that shows um the breakdown of nvidia's Revenue and over the last year um their data center Revenue in particular has just ballooned 400% it's eight what they just um provided their quarterly earnings report last year and data nvidia's data center revenue is at a whopping $18 billion for the last quarter um and the Nvidia um Data Center business is really built on a full stack AI platform that provides Hardware software and services um and there's really no one else that offers what Nvidia does it not only provides the chip Hardware but it provides the developer software called called cuda which allows you to actually accelerate the processing power of those chips um and Nvidia also has really huge Partnerships with uh large data centers who provide the storage capacity for all that compute so and the partners for that include you know meta Facebook AWS they all rely on Nvidia to provide them with that data center capability for AI processing so what does the future look like it's actually starting to look like this like Nvidia is now the the third largest company on the S SMP um it's third behind apple and Microsoft and it starts to look a lot like a w Garden um and is this really the future we want um it's there's a huge GPU shortage right now because there are so few providers of it um the price is just incredible so smaller companies and startups really can't access um this compute capacity so at live pier where we've been building the world's open video infrastructure Network for a while we're starting to think about this problem and how we can leverage our existing Global compute Network to help provide more accessible affordable and high performance AI processing so just to quickly go over um where we've come from the live here protocol launched in 2017 um and we are now a decentralized Global Network of video transcoding node operators or orchestrators that all contribute GPU capacity to the work of transcoding video so uh they run GPU cards that have both encoding and decoding chips and video transcoding is what you need to do in order to broadcast video on the internet if you record a video on your phone uh it's going to be recorded in one file format if you want to bro Broast that on the internet to anyone anywhere in the world on any device at any bandwidth you need to you need to transcode it um it's really complicated to do that but we figured it out we've actually also built um a global video service provider live pure Studio that's built software on top of this network uh it's 10 times cheaper than uh traditional Cloud providers doing this video service and we now have millions of minutes streamed on the network um as well as millions of fees being paid out to the network to those orchestrators who are independently um and permissionless providing their compute resources to the network so this is the homepage for live peer. studio where you can find the developer tooling um this is a QR code if you are a startup that needs video transcoding at scale uh we offer streaming credits for that it's hard to see but um we have ultr low latency live streaming toolkits I mentioned that you can save up to 90% % compared to traditional cloud service providers we have an API and multilanguage sdks that make it really easy to get start with building any application that needs video whether that's a social app a gaming app or or Creator platforms um we are train agnostic because this video toolkit um is just basic software you can compose it in any smart contract environment so we like to say we're chain agnostic you can mint a video nft on any chain that you want um you can use token gating and access control you can integrate with web three social protocols like lens and foraster um this is you probably won't be able to see this this is a nice image of the full um developer toolkit and pro video product feature set for life Pier studio and so where we're going um is version two of the live Pier protocol and it's really essentially it's it's job type abstraction which is enabling our our orchestrators to who are running gpus on the network today to perform more tasks than just transcoding and it's also enabling on the demand side of the network our Gateway providers um to request more tasks than just transcoding from the network so we kicked off this protocol upgrade a couple of months ago uh and we are on a fast track to launch a new AI Video Main net um later this year and I'll take you through um for the next little bit what that road map looks like so step one uh q1 for this quarter we've been working on the AI video subnet which is a fork of the go live Pier software so you run go live Pier if you want to run an orchestrator node to supply gpus or if you want to run what we call a broadcasting node to access um those resources um and so uh yeah we're creating a test environment so that we're not disrupting the network that is being used by hundreds of applications today for video services to make sure that we're yeah creating a bit of a Sandbox um to add new generative AI inference jobs to the network um and it's notable that payment requests even on this subnet are still happening on the main um arbitrum Network and live pure token um which is the reward and incentive mechanism for node operators is actually on arbitrum um an additional transcoding fees are actually uh paid out in eth as well so this is a this is a diagram of the system architecture for communicating and executing containerized AI video jobs and this is the bulk of the work that's been done over the last couple months um is building a single Docker image with a runner app that bundles all AI inference code dependencies and that can be configured to set up a specific um inference pipeline for for example text to image or image to video based on the job that an orchestrator needs to be done um the image can be used to run multiple containers each with different pipelines running on separate gpus and the container can read right to a mounted volume acted as shared storage uh with the orchestrator so all model weights can be stored in a single place instead of being bundled um into images so the um yeah the dockerized uh container er um execution is really kind of foundational to how how we're building the architecture for for the network so uh We've kicked off the initial um AI inference job types uh at this stage um they text to text to image image to video um and text text to image and image to video using um you know a variety of different diffusion models um initially these models are identified using their ID model on hugging face so the orchestrator needs to have the weights for the model available in storage um yeah so that the weights can be uploaded into GPU vram um before inference with the model uh before inference with the model can be executed and the orchestrator can decide which models that it wants to support for which capabilities I'm using a Json file that looks just like that so Within These capabilities we have added also flexibility to support different models trained on pre-existing data sets um and today outside of closed proprietary models there is actually a lot of experimentation with open source models and our goal is to prove that the live pier network can be eventually used by developers to deploy any model on the network I don't know if this is going to work I actually have a demo video from can no one can hear that I'm sure okay so we have two orchestrators running on the server and the first orchestrator is advertising the text to image capability and the second orchestrator is demonstrating a different text of video um with a different um a different model so you can select which model you want to use um for which inference job and you could see that the image was generated so and now what's interesting is if you deactivate one orchestrator the other one still work so this is why we're Deming multi orchestrators because we've already started to be able to show the redundancy on the network um so I'm just going to try and move on back to the slide okay ah I don't know if anyone can help me get back to my slide okay thank God here we go okay um so second quarter uh which we've actually already started is community testing um so we've already begun orchestrator benchmarking which includes um gu GPU optimization we're also starting to gather data around the co cost Effectiveness to prove that this indeed will be um as disruptive um in terms of price point as our transcoding services have been um we're also um really looking for the community to help us add new um inference pipelines to the AI worker repo so adding jobs like upscaling object detection um scene classification and more we also have a demo consumer app that's being worked on in the community and we love to see more of those um and then Q3 we really hope to launch the AI Video Main net so it's merging those subjects sub subnet jobs to the network adding production support for orchestrators to run any custom model because this the subnet is not in production today but we we do hope to get there in in the next few months as well as um you know by Q3 we really need the developer tooling for hosted nodes um to be able to access um this network so we have an API spec but we're going to um continue working on that developer tooling and that will most likely involve an updated version of livepure catalyst which is a current um Gateway or broadcasting software on the live pier network so we have 4 minutes left for me to talk about how we are going to bidle the future of AI video together and what you can do so today you can do um three really important things actually and we are we're super early in this journey like we really only started a couple of months ago um we are walking and chewing gum at the same time and that we are both adding uh doing the R&D around new Supply capacity on the network but we're also at the same time trying to figure out what does demand for AI video processing look like what are the use cases um and businesses and consumer applications that are going to want to access um AI video processing so you can if you are a systems engineer if you have run AI inference on bare metal or Cloud servers um please contribute to the development of the live pure AI video subnet um I mentioned we do need people adding um code to the existing repo to create new inference pipelines um we also would love folks um working with us to prototype the um API spec for the Gateway software that I mentioned we have um a version of catalyst that we've recently updated but we' love more people contributing to Catalyst um and the second is really exploring those Innovative use cases for AI and video what are meaningful problems that AI um can solve for um in a video context you know there's been a lot of um a lot of tools and products come on the market in the last year but um they're all really producing short form um clips of generative AI That's not super compelling so I think we're really early in the market for understanding um how this can be useful for video workflows whether that's subtitling helping with transcription translation of video editing um co-creation um how this can add value for folks in a video workflow context um we we're just really Keen to to connect with people who are thinking about that and three is how do we make AI video on chain I think this is a real Frontier area um one thing I didn't get into but um which is an issue um around content authenticity and Providence in an AI context when you have obviously deep fakes and issues around trustworthiness of video content live peers been involved in the um content authenticity content content authenticity initiative um led by Adobe and the c2p um and they're working on um standards that even open AI has just announced that it's that it's implementing in Sora but they're not yet using public blockchain Solutions so um you know we'd love to see folks using attestations storing those attestations on public blockchains to um provide Creator signed provenance Trails for video also minting your model I think there's huge untapped potential with that um this is again uh a an image from injuries and horrorist which I think has done a pretty good job trying to map what exists of the um nent space such as it is um around AI video tooling and the different stack and I think we're you know just at the early stages of figuring out what that can look like in a web 3D centralized context and we have some buildon bounties um I hope folks are signed up or having a look at those sponsor bounties for AI video we have $10,000 for um anyone implementing a successful integration of AI capabilities with live Pier Studio or Catalyst to do interesting things we have $10,000 for anyone building an AI video startup we'd love to help you um and time is up please sign up for updates as we go to the launch of AI Video Main net thanks so [Applause] much