Eric Tang: Verification of Decentralized Video Transcoding

alright let's get started thank you guys for having me here thank you for epic labs for hosting and thank you for a swarm team for putting on the party my name is Eric tang and I'm from the live fear team and presenting with me is ignacio from the epic labs team and he's going to show off some awesome research results that would be in collaborating on so the topic of today's talk is verification of decentralized video transcoding and and this is one of the key components in the life of your network so what I want to cover today is to give you a brief overview of what the live peer network is kind of introduced that to people who who are not already familiar and then I'm gonna introduce the the current phase that were in which is called stream flow and this basically allows the live to your network to scale to to be able to power a major a major Internet video application something like a YouTube or like a twitch and then I will and Ignacio is going to give you a demo on the research results that we've been collaborating on and and I will talk a little bit about what's coming next ok so live Pierre if you're not familiar is a decentralized video infrastructure and what that means is that it's a developer tool that allows people to build video applications currently the way the way video works on the internet on a very high level is that for a broadcaster if your broadcaster you're on the very end with the camera you either have a live stream or maybe you have a video file itself you upload this video into a cloud hosted media server right a media server kind of similar to a web server which hosts web content a media server hosts video content right so you're able to upload the video onto the media server and the media server kind of serves that video to the Internet but the media server also does a few things that are really crucial one is that it transcodes the video into many bit rates and many different codecs and that's why we are able we're all able to watch the same video from around the world whether you're on a cell phone connection or you're you're sitting home with a high with a HD TV with high-speed Internet right so even though we feel like we're watching the same video we're actually watching very different versions of that video and and that's all due to the transcoding and this is a very key piece of how video can work on the Internet in the first place and the media server also does a few things it does it applies the rme it can store the video and it can also do ad insertion do subtitles now how about these features right and it also serves that video to a content delivery network that then replicates that around the world and serving that to to the end-user devices so what we want to do in the in web 3 and the decentralized world is to replace all of the cloud hosted parts and put that into a decentralized network why do we want to do this it's because the cloud hosted infrastructures are owned by large corporate enterprises that and and right now there's kind of a monopoly in in who is providing these infrastructures and as a result there's a lot of perverse incentive in this whole scheme are people paying contracts to these high to these large enterprises that then provide the infrastructure at an artificially low high cost so we want to do is to have this open decentralized network that's that has all the features that we that we can have from this cloud hosted services but it's not owned by anyone right this is just common infrastructure imagine how the internet was invented in the first place no one really owns the Internet it's just out there but the the decentralized network itself can figure out how to incentivize the participation and and and getting people to contribute their hardware getting people to contribute their resources to create this network so and the first step to do this is to create a transcoding network which is what live peer is currently working on I previously I mentioned the importance of video transcoding and the way this works currently in the life care network is very simple right if your application or you're a broadcaster of video you can send that video in to the live peer network which consists a lot a bunch of transcoders and these transcoder we call them video miners then since then do the transcoding work and this and they send the results back to the applications and the network also uses etherium to do the settlement and provides security to to this operation so that even though you're working with a decentralized network and you don't know who is providing the computation you can trust that as long as they are participating in the live peer network the results that they provide will be the result that you asked so how does that actually work you you we use staking and and staking has many different meetings here we use we use that in the sense that the service providers have some have something on the line right they have they have some they have something to lose so if you look closer you know here are you know a bunch of the nodes that are a better active on a network and each of the nodes share a year of a percentage of their rewards to the people who stake towards them so imagine you see or our life your token holder you can stake your tokens towards one of these nodes and in exchange the the knows will share a and of their profits to you but at the same time you're also taking on a risk because if the nodes do something wrong they will be slashed and and your and your stake will also be also be slashed in that process so you really have to make sure that you're staking towards someone who is honest and someone who can provide the right work and on the high level the slashing mechanism worked like this you have the transcoder the transcoder is giving you the results back like before right and the transcoder is sign creating a cryptographic signature with every transcoded result that comes back which means now as the broadcaster you have the evidence to to challenge somebody and the challenge simply means you know I'm gonna ask a judge to decide whether this computation has been done correctly and there and the way you issue a challenge is by issuing a transaction to the life of your smart contract that runs in the etherium Network and and believe that the the smart contract can serve as an oracle and give you the right answer right so so this is all good so why do we need to work on stream flow and why do we need to scale this network in the first place right life here has been running for about a year and there's been you know hundreds of events that have a stream that have been streamed on this network so for example last year most of the technical tracks at Def Con at aetherium was or streamed on live gear right but but clearly there are some problems the reason we want to scale this network is because the majority of the traffic on the Internet is video traffic right so we're on pace to have over eighty percent of all the traffic on the internet being video content and and most of that content needs to be transcoded right so if we want to build a decentralized network that no one owns as I can and I can serve as a commons for everybody who uses an Internet we got to figure out a way to make that actually serve all that content and to process all that content and that's a pretty high order right so so when we we've been working on this for for over a year ever since we deployed the life here protocol on the main net as a beta right and and we've been learning from the deployment and we've been working on a lot of the research and implementation based on those learnings so there are a few things that we did here but one of the things that I want to point out is is that the reason this can scale is because we're able to we're able to do everything off chain so we've changed the workflow so that the entire video workflow is now it's now off the etherion blockchain and and no longer limited by the transaction throughput and the cost of using the chain so the only thing that requires the etherion block chain now is the initial set up of the setting up to node in the first place and the settlement that happens asynchronously instead of in in this work in this video workflow and that's what kind of enables us to to scale this to handle hundreds of thousands of concurrent streams and another really important thing here is that we are we are enabling GPU processing GPU processing is it's a really it's a scaling factor in the cryptocurrency world right because we all know there are cryptocurrency mining farms out there and a lot of mining farms around GPUs right but so if we look at the architecture of an actual GPU we realize that there are some processing units on the GPU but also there are chips on the GPU that are created only for encoding and decoding video and they can't do anything else so all of the billions of dollars of GPUs that are deployed out there mining cryptocurrency have these video encoding and decoding chips sitting in the farms and not doing anything and what we have done is benchmarking against against these GPUs and running the cryptocurrency mining operate in parallel with the video processing operation and showing that you can actually run both at the same time you just you just expands a little more electricity but the amount of profit that you can get is tremendous each GPU in the cryptocurrency mining farm makes about on average seventy five cents per day but if you apply the same pricing that the Amazon charges for example for video transcoding you will see that each GPU can do two to three streams in parallel an amazon charges $3.00 per hour per stream to transcode so what that comes out if you utilize the GPU fully it's a hundred a hundred and forty four dollars a day which is hundreds of times more than you can make in a GP in in GPU mining and we we and we expect that this drastic difference in pricing will will drive a lot of supply onto this network and also drastically lower the cost of transcoding which makes this which makes this network very very attractive not only to decentralized applications but also to to any traditional video applications and of course one of the things that comes with GPU mining is and in GPU based transcoding is the non-determinism of this operation and what that means is if I put the same video video segment into the GPU I will get different results back each time right and this is based even though the picture looks the same to the human eye and this is because the floating-point calculation that happens on the GPU so within thinking about okay if the computation itself is non-deterministic how can we actually verify the work has been done correctly in this especially in this decentralized world where we where somebody needs to trust some random computer on a network to do the work correctly and this is one of the things that we've been working on with epic labs and Ignacio is gonna come and and tell you a little bit about that all right so here at epic labs we have we have been collaborating with light beer in order to check that the videos have that the deal has been done these notes are doing all this work but there has it has been to be some way to verify this work so we proposed this in which I think it's a very interesting workflow in which we work on feature in Union we were at the start we're avoiding working with the frames of the videos where we're developing our own features and then we fit these features to two kind of models that we'll talk about them later and then we're doing incremental releases and while we do this we were creating articles and we're open sourcing everything okay so on the side of feature engineering we as I said previously we're avoiding working with images just to make this previous workflow that is iterative faster we have we have been using some known metrics from computing similarity between images such as structural similarity index or mean square error and then we're using features that are known as such as comparing the histograms of the images in order to check that if the video has been for example grayscale or there has been some vignetting effect and then we're computing we're going to other domains such as the frequency domain with the discrete cosine transform just all these computations on the images and then we're also doing edge detection so this can help detecting for example watermarks and then we're generating time series we're doing this frame by frame in each of the original asset and on the front scolded video so we're doing this frame by frame so we end up with a time series right now we're averaging the time series so we have less data but on time we'll end up working with the whole time series so this is just an example of one of the features and what we're actually doing to the images this is an example in which we compute the edges on an image and then we should strap at the same time from the original asset and to the transcoded asset so we can see that then if the video for example doesn't have a watermark the difference would be almost zero there is always gonna be some artifacts on a mats but the difference would be almost zero but if there is a watermark and now we compute yeah we use a light beer watermark that we find if one of the notes use that but so so the difference to have almost energy zero that this would be any pixels activated but if there's for example a watermark it wouldn't be so this is just an example of the computations that we're doing to the images on the model inside we have in the previous scheme we have a split in two parts we call it supervised learning and unsupervised learning but basically on one side we have labeled data and on the other one we don't have label data on the supervised learning side we use for training assets that have been properly transcoded but we also use some attacks that we have simulated and that might be on the network such as grayscale in a video or vignette in down sampling the colors even just a black a black video or a video that it's not from the original source many attacks and this is and this goes in the training on the unsupervised learning site we then use that the training the training set is just videos that has been world transcoded so of course on the supervised learning we have labels we have good videos and attacks and on the supervised and the only unsupervised we don't have that we just have videos that have been well collected so the good thing about the unsupervised it is is that it's attack agnostic because there is no bias towards any attack of course the overall accuracy is at least for the results that we have been achieving it is better for the supervised learning but it is biased by the attacks that we have trained and then of course on the unsupervised learning the course is a little bit lower but it is it is not dependent on the kind of attack we are juicing some machine learning models we as I said we're saving the world with images so deep learning will probably come later right now we are using classic algorithms extreme eggs boosting support vector machines random forests on the supervised learning part and then on the unsupervised learning part we're using one class super vector machines isolation forest and small autumn colors so let's not call that yet deep learning so as I said everything is open-source we have a github repo where we we have grid that do to be scrapper which takes the videos from a very known beautiful data set which is due to eight million it has eight million YouTube videos so we we take those URLs we create the us's the urls and with an api we download them we have article 1 and article 2 talking about the features that that we just say article 3 is releasing with more features and eventually since we have finished the first site as cycle we'll have a fourth article talking about the predictions that we have been doing and the models that we have been using ok so let's go for the demo we have prepared a small demo of the CLI that we have been working on in this demo basically I have I have an original asset which is this video ok it's it's downloaded from YouTube and then I have I have I have a video with a different resolution with with a watermark on the top left we can see the light field watermark I have placed a different video which is which is this one it's not really to the source so we should be able to detect this and many more I have I heard them some being a teen some low bitrate so right now let's use the the CLI this is from the from the github repo okay so I'm just gonna start a start the docker and build it build the image and right now I'm gonna call this command which basically is calling the script it's it's giving your email asset and all the possible renditions that that that we have in this in this directory and we're downloading the model from github okay the model is it's open source so what I'm gonna do is I'm gonna download the model and then compute the metrics all the features that we feel that we've been talking before and then we're going to compute imprints okay so this would take a five seconds so right now we're downloading the model right now we're capturing the frames and computing the features and we have made our prediction okay so it took 5.5 seconds in in in computing the features of six videos of which five point like 1990s and 99% of the time is the feature computing and in two milliseconds to compute the inference with the models okay so the original asset is that one and we can see that it has actually detected the watermark video there is an asset with a 360 resolution that is legit there is no attacks on them and then we have detecting the low B rate the vignette and the and the attack which is a completely different video so this is the demo that was ready and that's basically it let's jump back to the presentation all right Thank You Ignacio as and as you can see now we have this train this interesting classifier that can basically tell us whether a video transcoding job has been done correctly or has been done by a malicious actor in the in the in the network so the question the next question is how can we use it so first of all we are soon releasing a stream flow test net where everyone can can try this but also this type of classification workflow kind of changes how the protocol works from in the challenge step right so before I talked about you might be you might remember this picture where on the on the bottom right we have something called true--but which you know is a is a project that's a decentralized computation verification mechanism and of course true--but only works on deterministic computation right so when we have a deterministic computation we use a classifier and this classifier have that we have a few different ways to use this right one is we have to make sure the classifier is trained on a public data set and this data set needs to be transparent to anyone who is participating in this network right so by participating in the live peer network you are saying that I consider to be mod to be monitored by the by the classification model that's trained by this public data set and this data set is available to anyone anyone can use the same algorithm to reap to recreate that model right and another thing is on even on the execution of the classification algorithm if we want to truly decentralize it we might have to run it on trust the hardware but that's totally fine even though trusted hardware is a little slow we can we can see that the actual classification step only took like you know a few milliseconds so so so that that would work and also another interesting [Music] approach here is that the transcoders themselves when they were doing the transporting job can actually also be producing the these metrics that will be used as parameters to the classification algorithm and as a result basically saving time for calculating that for calculating those parameters and we're working what we're starting to work on prototypes that incorporates this metrics calculation into the transcoding step and this is actually something that can create a lot of value not only in in live peer but also in other in other video usages as well just the fact that when you are doing video processing you're actually returning the metrics that come with the process and and something else that we have to deal with of course is false negatives right any kind of machine learning algorithm has false negatives and what that means is even though as a worker I've done the work correctly the algorithm has wrongly said that I did I didn't do the work correctly so so so what can we do to deal with that well one way to deal with this is to just run this algorithm the the worker themselves can run this algorithm on every single piece of work that they do right so they will know that whether the algorithm has been incorrect in determining his if his work is correct and if that's the case as a worker I can simply just say in those low chances where the algorithm zing is incorrect I just won't sign the result that I gave back to people right and instead I just say hey I'll give you a freebie let's continue to work together but but but just don't I won't give you the evidence so you can punish me for this thing that the algorithm is making mistakes on instead of my own work another way to do this would be to have the protocol to keep score and what that means is let's imagine we have 1% false positive rate Falls negative rate so we can what we can say is based on the we can set a threshold and let's say for every 100 segments that this this worker processes we want to have no more than I'd say three segments that are reported to be to be wrong right so we set some threshold there and this is made possible because of the probabilistic micropayments scheme that we that we worked on and going to that but the payments and one of the side effects of the payment scheme is that it gives us information about how much work a particular worker has been has done through through the amount that they're getting paid another interesting direction is this idea of multi-level slashing which means the work can be checked in different ways right so for example if I replaced the video with a complete black screen that's that should be that could be a different type of punishment than ever I created some result but the resolution is low right so all these things can be can be categorized differently and the verification operation of those like all those checks are also different and it comes with different cost for the network therefore we can punish people by different amounts if they fail certain tests so another thing that we never worked out is integration with the Swan which uses the swarm feet feature that Xavier built where's Javier he just left the Javier's built and I cry I actually recorded a YouTube video and published it so if you just go on there and search for live pure swarm integration and you'll be able to find it and watch it and the idea here is that we want to create we want to create a really smooth broadcasting experience on swarm today right so we you know earlier we talked about swarms long term vision when we talked about the current state of swarm right which is we have this decentralized system but the system is not incentivized it's it's a peer-to-peer network not instant of eyes peer-to-peer network and what comes with that is is the lack of guarantee in terms of bandwidth right so we have no idea how much been how much what's the speed of the connection between all the nodes in this network and what that means is would not be what we're now be able to guarantee the delivery of something like a broadcast because the video segments and the video streams are pretty big right but what we can actually do is to deliver only the video manifest through this through the swarm network because these manifest files are very small and and all it contains is the information about where to find the video segments but not the video segments of themselves so if we if we just store the video segments in a traditional you know in the traditional content delivery network or something like s3 those networks can guarantee the bandwidth of delivering those specific video chunks but without the manifest you cannot watch the video and what that in this so the combination of these things is a video broadcasting experience in a peer-to-peer network and that's that's currently just works which is kind of exciting and and something that can be built on top of this is an ENS integration which means you can actually use an EMS name to represent a video stream so instead of having to require a manifest or something you can have a human readable name say like you know swarm Santa Eve and and if you go there you'll be able to watch the video stream which is kind of exciting so that is all that I have I think we're a little bit over time but uh thank you for thinking or being here and I'll be here all day if you want to ask questions thanks - ah - thanks Ron for doing a lot of the works as well [Applause]