Zero knowledge machine learning with ezkl

foreign [Music] Workshop zero knowledge machine learning with Ecko joining us today is Jason Morton and Dante Camuto who will be taking us through the session and with that I'll pass it over to Jason and Dante to get the session started hey thank you very much um nice to meet you all um we'll jump right in um with the screen um an object um yeah so we're uh Jason bate we've been working on building this uh software for machine learning um uh on chain or for doing zero knowledge proofs for machine learning and say we want to talk about how to use that for autonomous world so that people become increasingly interested in um in recent days so okay so what's the picture so one way to think about it is that we're providing kind of a physics engine for autonomous worlds um so we can do kind of arbitrary Matrix computations uh arbitrary kind of machine learning computations convolutions all kinds of crazy stuff um things that you wouldn't want be able to write in solidity because of their you know gas costs you can sort of offload those heavy computations to this engine um so the client uh the game client or a server can prove sort of a state transition of a game so you can imagine you know the there's a diffusion happening in the game or there's uh an evolution in the world State that's more complex than what you want to implement with like the ECS um and so you can use this as a way to kind of short shortcut that you also use it to build empty scenes that are powered by AI um and a lot of the examples people have done as well are things like kind of a sick or God model where you're trying to please some you know you're trying to pre please someone like a cooking mama or you're trying to please some someone by creating some kind of pattern in the game and then a clap what's happening behind the scenes is there's a classifier model that's being run on your input um of course and sometimes those models don't work as well as you would like them to but the fun part about a game is that even those failures can become sort of fun part of the game like beating the NPC um so let me back up for a second and say what does it mean to run a model on a chain an AI model on chain so you're all familiar with an ecdsa signature which is a kind of zero knowledge proof takes public inputs and the private key combines them in a well-known signature function produces a result which is the signature again you pass that to the chain to verify it um or you know someone else can verify that signature and what a zero knowledge proof is is basically just a programmable signature so we can replace that signature function with any function we like the functions we like are kind of ml models and linear algebra um and what we're doing with Ezekiel is to make it easy to create a model in Python describe the pipeline that you want in terms of who is responsible for approving is responsible for verifying and what's set up we bake that into a proverb or verifier pair um and then someone produces approved feeder on the client and server and then the chain verifies the proof and it's as though the model has run on chain even though it ran off chain um and the big picture of course is that AI is something that doesn't run well on chain now so AI is and machine learning and when your algebra are are great but require trust either in the general context you know trust from someone like open AI or trust from the person who's running the game server or the game called it um we can work at python or other numerical languages there's a big library of existing models and Transformations you can do the game um for on-chain contracts um they obviously have the property of being decentralized being trustless being composable um and uh having this property of autonomy that with it many will talk about um but they're less scalable because they rely on the consensus mechanism the same computation has to be repeated many times and they're kind of too slow and too expensive um to do complex AI machine learning or linear transformations um Decay brings uh to the trustlessness decentralization and the posability some scalability because now instead of having everyone have all you know all the consensus nodes have to produce do the same computation um the single a single model can be every single machine can do the single client or a single server and then everywhere else can trust it but of course would pay a cost in terms of math and security properties and if we're a programming model and possibly weird languages um but you know what we've done with Ezekiel is to make it easy to use Python uh models or python description of the linear algebra or the machine learning model that you want to run um so that you can access a library of existing models and or print your own models uh and then make it easy to deploy it so that we can use it in an on-chain game um and I want to mention an example um by Daniela Rich tradition and Paul Henry do you know what's on the call as well I think um and what they did was to produce just so you get a sense of what kind of thing that happens is that they made a world the players could um take actions and then you evolve the state of the world in response to those actions like you know trees spread and grow or fire spreads or whatever with uh what is the K in our model when there's another llm that's not right now on zkl but will be eventually that tells stories about the things that happened and just to give you a sense of the kind of evolution that might be possible um the other piece I want to bring up here is how this might interact with mud so there seems to be a lot of connections with lattice mud and I think we're going to see a lot more integration happening um as people experiment with it right now if you look at the bottom part of this picture is kind of that uh without a mother we're going to be responsible for writing the code to ingest State uh from the chain um or to take an action the player action prove an update of the state how the player's actions or interaction result in a change in the state of the game and then that that proof goes to a verifier that lives on chain that updates the state but now it's back into State on chain so there's kind of a game that's happening between the client and the state and the state that's stored on the same where the more complex updates involving diffusion whatever else happen uh client-side or service side they're not not in a smart contract that would require a fair amount of development like an example I showed you before but with mud you can use it the ECS framework to sync the state um with the chain with the client State and the chain State and uh it's sort of were set up now I also think about where we're going to do the proofs either on the client side or the server side um but a lot less work in terms of producing that of course a little bit much folks talking about that so the basic idea of how um as you can all proof works is very simple um so we're taking floating Point numbers uh and representing them as six point numbers so we're picking a denominator like 256 and we're quantizing it um and then we're representing a small floating Point number 726 as literally seven we're just starting the numerator and then we keep track of the denominator 256 in the type system um and to prove a DOT product um you could write a custom gate in other words make the constraint uh into the constraining system of the data knowledge proof that says that y the output of the first of the two vectors dotted together is equal to whatever they are that's not really what we do we do something more complicated there's a lot of different arguments that can be that can be made to accomplish that but um sort of not um critical so you don't worry about exactly how that argument works because there's abstract that away and then to prove that Y is equal to ax plus b then our matrix multiplication and the shift has been applied you just basically repeat that argument or arguments like that and then to prove that uh a non-linearity like realu has been applied um you leave or we do is we pre-sale a table with all the possible interests and outputs uh that might occur given our assumptions about quantization of the maximum numbers that might appear um and it proves that the input output pair lies in that table that's called a lookup argument um there are other alternative ways to do that and there's lots of new arguments of pipeline to make it more efficient but it's already pretty fast um so uh so um I'm going to pass off on a minute to Dante um but I want to say depending about the big picture about how to use Ezekiel basically what you do is you find a model that works for your use case you nutrain it yourself you design it yourself or you download it um for example you're using it to compute a state transition and evolution of the world or the natural world you might be using it for this judge you know it does a player recipe uh make me happy or be the picture that the player Drew or the song that they sound uh satisfied they'd be basically on chain AI or to run an NPC AI or some on your idea that doesn't occurred to anyone yet that you'll come up with um then you take this model and you compile it you can Ezekiel to sort of a triple of circuit um which is the best in the setup uh approver artifact which is either a was imperator or a binary prover and a verifier artifact which in our case will be an on-chain evm verifier and you have to deploy that verifier and sort of Route it into the uh route the update into it um and integrate the prover either on the server or the client um and then you launch a game and these are just our telegram group and our GitHub you checked out and everything like that I will pass Dante to show you how it works well all right um I see there's a few questions a little chat I don't know if we're going to do those now or later on um there's one question which is is the output a regression result or is it limited to classification results um you can do both I'm going to sort of demonstrate the computational graphs that you can build but you're not really limited um to either regression or classification uh um there's another one which is can we use Ezekiel for approvable training as well as inference uh we currently when we support parents meaning um yeah training is a lot more computational expensive um and hopefully at some point we'll support it but currently we're just um working on now yeah with intros uh someone asked you to go back to the slide with the GitHub nearing Discord I just put it into the chat you just search for easy kale um I think we'll also find it all right awesome all right well I am going to Showcase about what all of this looks like in practice basically so how do you it was actually quite a few we've developed a lot of tooling to to make it easy for you to lose those um but the main library is written well I know it's a challenging language um so we've sort of wrapped it in a python we you can compile it to log in if you're running a browser application so you can normal proofs and verification straight in the browser um but today um kind of as a as a neat sandbox I'm going to be running you through like how you might use this Ezekiel to like uh generate proofs inside of a Jupiter notebook or something just so you have what you can write in box to uh yeah it's just kind of a nice damn dogs to try out different models um and see how they perform um sharing my screen oh all right so um NHS so this is kind of a pretty typical setup for starting off about an Ezekiel project um so what's kind of cool is that you can actually just Define your circuit or you can think about your circuit just as a pi torch sort of computational graph that you're going to start typically by the final like a wrapper around and then module which is really just like uh you know how you might Define a neural network in python or broadly um so how are we going to do that all right let me create some sort of model what's interesting is that I have copilot on so sometimes let's adjust layers I don't know our spring typing so oh yeah then yeah let's see what it comes up with interested in building all right well we're gonna have different okay let's start simple all right let's start simple and let's just have a single column layer and a value non-linearity or as part of our model um and our model is just going to be applying oh look at that and stuff is amazing but it's so our model is just going to be applying a column layer and then subsequently applying a relative non-linearity uh onto some data let's say that data is like feminist shaped or something um which is uh like images with a single Channel whoa and yeah 28 by 28 images um and we need to instantiate the model once we're not going to train it um because we're sort of limited by time I also don't want to download a data set um but all you need to do now is you can just export this here it's you will get out locally um so let's turn on that cell make sure everything works it all right so we've exported the model um and the format that we use is something called Onyx um the details of the story matter but we've just saved this model locally and what we need to do as Jason describes like we use a slightly different representation of remember so we're not really working and floating Point space for working with field arithmetic so what we need to do now is quantize the model so basically fit everything to into that field and make sure everything runs correctly and this is done with the forward method on the ez2l library so we've quantized the model and now for a CK approved you know we need an SRS string which this corresponds to like the parameters that we're going to be using to generate like the verifier key The Proven key and subsequently approves so we also provide a method to do this um to make it really easy this takes a little bit longer to run but hopefully not too long and yeah there we go um and then we have What's called the setup page which is where we generate the verifier key and the proven key so the proving key in particular is important for you know it's in the name once important for generating proofs and the verifier key is important for the uh verifier to then subsequently verify the proofs so this is all done in the same step and then it can be distributed to the two different parties that need to you know either prove or verify uh generated from one specific circuit we'll run that and knights right so that route and then generate a proof using a simple method by providing like the paths and all the things that you need and then you can verify it and voila that's kind of it we also provide a bunch of methods for you to I don't have my ledger with me right now but if you plug your Ledger in uh you can basically submit proofs on chain you can deploy a verifier contracts if you need as we provide to learn to do all of that internally to the library we've made it we've tried to make it as easy as possible to like generate circuits uh you know prove verify generate on chain verifiers and then submit proofs to those verifiers but let's go back up and start generating something slightly crazier here um okay whatever we want to add let's say we want to double X and then subtract it is for you know it's refine and then we want to take it to the power squared all right let's see if this works all right let's run through those cells again but basically the idea is that you're not really constrained by anything um when you're generating these circuits we support a whole bunch of methods so if you want to generate something really crazy you can Chuck it into a pytorch model and then start running proofs on it turn so yeah I mean to generate the setup base again and there you have it verify it again all right let's see if we can chuck a second conf in and make this a little bit spicier um all right let's add that into here oh wow um and then maybe yeah all right that sounds good but yeah in terms of circuit size um if you're wanting to prove something you know in less than 10 seconds um we've gotten like six five or six today or comms uh yeah we've gotten five or six layer columns to prove in less than 10 seconds that you can go pretty crazy in terms of parameter size um yeah okay let's see if this one verifies I kind of want to see if I can break it at this point like see how much we can throw it at all right sweet all right that's still verifying God I don't know if we're gonna be able to go because to stop working today um yeah I see there's a few questions in the chat when we tried using Mojo with bkml I'm actually not too familiar with Mojo um but yeah are there any questions on like the specific demo Esther like how do you use this and we'll be confused what the demon was trying to prove well basically yeah okay so what is the demo trying to prove could you explain all right yeah so if you if you were to try and build CK circuits on your own um to generate proofs of like data zero knowledge machine learning model was run correctly it was basically either have to go into circon or Halo 2 and start coding up you know like a gate for matrix multiplication or a value for a convolutional layer all those sorts of things um and what we've done is we've started to abstract and wait a lot of that complexity um and what does that mean so it means that like instead of like having to go into those languages like circon and those sorts of things to generate like you know proofs or a circuit that represents a specific machine learning model you can just use the Frameworks that you are probably familiar with if you're a machine learning practitioner meaning that you can just use is uh meaning that you can just use things like pytorch or those sorts of things to generate a model we take that model and then compile it into a circuit for you so you don't even have to think about it oh what is the ZK proof proving all right yeah good question so currently we're just proving that the model executed correctly so given that you know a specific given that there's a public model and there might be some public data you're proving that you executed it correctly and then you can submit that through as call data to the chain but you can generate a whole host of other groups like um for example you could keep the parameters of the model private um and run it on a set of public inputs uh you could keep a set of private inputs um but run a publicly known model on those and then generate a proof that you ran it correctly and then submit that proof on chain basically so we allow all those sorts of different scenarios to be realized what model am I using currently um well I just ran through it um I'm sort of generating these models on the Plies that you can really generate kind of anything um sort of anywhere can the code be shared uh yeah we actually have a Jupiter notebook demo great into the um yeah straighten our repo so if you want to play around with this this is all available inside the GitHub repo um but yeah definitely definitely already available someone's asking someone's saying they're not familiar with python is it possible to have a singular tutorial with JavaScript um eventually So currently we are so on the JavaScript side but we're really focusing on at the moment is getting uh approvers and verifiers and Blossom so that you can run these things uh with relatively good performance um and yeah create in the browser so you can generate proofs for a specific model or verified roofs or a specific model straight into the browser if you have interest in getting this stuff into JavaScript um definitely yeah open up an issue um and we'll take a look is there some mud integration allowing game clients accepting events from the indexer So currently we are not fully integrated with mud but that's definitely something we're moving towards that we're aiming to like integrate as much as possible we're going to make like using our tooling uh within autonomous worlds as quickly as possible foreign a lot of people talk about keeping input data private I'd imagine it could be interesting to also create output data private in that scenario also could you go through how a proof like that would be set up um yeah so what's pretty interesting is that the so the proof commands uh didn't hear you can actually pass you can actually say what parts of the model you want to keep private and public um and all you so you can actually just pass a flag that says I want my public or I want my parameters for example of my model to be private I want the output with the model to be private or I want the input of the model to be private so we allow all of that to be configured from start to finish so if that's something you're interested in we can already do that we should also probably show something from EDM verified that's oh okay yeah for sure I actually don't know if we have that in rock I I don't know if we have that into the python bindings yet um yeah you just uh I can do the docs or something yes I yeah I mean sure that's excited for sharing are you sharing um but if you go to the dog site there's some section I'm verifying on chain um which shows you have to generate a proof with the with the transcript set to EDM um and then you can generate a verifier which emits code um deployment code uh specific to those at the command line specific to uh your model and solidity code and then you can test that it verifies um by using verified you know the mattress is running the code that you generated just before it um and then you can deploy that code uh to um to a chain and then you know using you think you would need to sign it and you can also send proofs or you can send those proofs manually or deploy.com manually um I just wanted to highlight it is possible to create a verifier so a very common use case would be something like uh you have a state update you want to check that the state update is valid before you apply it and so you would create that verifier and use didn't use a verifier called from your main smart contract in order to check that the update was yeah and yeah and I think that answer is another question in there which of those are we making functioning calls to the verifier contract uh yeah so there's a method called verified EPN where you can just pass a verifier contract address and then submit your proof there um but yeah we also have a bunch of methods for deploying the verifier contracts you don't have to think about that either which is what Jason was just joking one more question it's just strawberry strawberry detail you end up doing yeah or hot dog detector um some people have done uh that using this Doodles thing the Google Quick Draw program for example um where you have to draw a picture of the snowflake or horse or whatever um that should also work all right any more questions yeah if anybody has any other questions feel free to type in the chat or take yourself off mute very curious to know whether you have a complex use case but you can imagine that you can share with us another one I guess there is so there's two ways to answer that I guess there's complexity in terms of the models that we're trying to get in I'm so most definitely we're we're aiming to get a Transformer based models or at the moment so in that sense for sure um and I guess we're also trying to like at the verifier contracts and so large here adapts and stuff um so start composing complexity of return from source of applications yeah it was a lot of interesting I guess autonomous worlds yeah yeah an autonomous world people thinking about you know for example having a world physics or Evolution function being something that is could we play around or you know could be like the player commits to an evolution function and then is responsible for maintaining their planet or you know NPCs with serious AI um you know powered by Transformer and whatever else um that can even move between games and stuff like that so that that kind of thing should should eventually unlock it so um certainly the State updates and really really basic AIS but more sophisticated AIS that would be really fun yeah I guess we're autonomous worlds Eagle specifically have been using it as like not as ml but it's like a sort of physics engine which is what um yeah Jason's showcased earlier which is like all the stage transitions between guys um yeah all right this can be huge for verifying physical objects they're useful or not as well yes definitely not just strawberries and hot dogs but also in strawberries yeah that's true I mean I'd be curious to see the strawberry verifier so if you want to whip that up um definitely message us and we'll help you get and start very verification that middle line yeah um but yes I guess one more thing I wanted to address was the JavaScript question is to go back to that you can actually so there's a bunch of ml libraries and JavaScript which will export to Onyx which is the neural network representation we use uh it's a sort of universal format so I think tensorflow JavaScript can save to Onyx and then you can point our clis we also have CLI tool into that Linux file so then replicate the flow that we just showcased so you're not limited to python basically any any framework that can generate onyx um yeah as you basically awesome well cheers thank you all very much for this presentation and thank you all everybody else for attending um this will be the require will be shared to you all and so yeah if you have any questions please feel free to reach out uh to Dante and Jason on on the Discord channels yeah so thank you all for for this we have one more session later and then the information will be at the last session of the day cheers thank y'all thank you