How to build and deploy on-chain AI/ML with zero-knowledge machine learning with Jason Morton

can you hear me yeah good thanks okay so this is a workshop we're down here in the uh cattle trading room so feel free to interrupt me with questions my goal today is to explain our open source project Ezekiel tell you how to build an application with it and first of course we have to spend a few minutes on why it's interesting I think if you're here you probably have some idea of that but okay so people like AI it's you know the hottest thing but in reality of course it's it's been a big part of what powers large and small organizations for a long time it's centralized it requires trusting the person right running the AI but of course it's familiar python workflow there's lots of models we can work with um and then we have Smart contracts smart contracts are decentralized they're trustless and they have this wonderful composability property that we can only get through sort of an API so it's a much stronger form of compose visibility then we could get in off chain ml um but they're not that scalable we have to run the same program many times we have to pay a lot of gas and it's too slow and expensive for AI programs um and what does ZK bring to the equation well it preserves decentralization trustless trustlessness and composability but it adds in a scalability right so now we can one person can compute a proof fund one machine and everyone can believe that they did the correct thing they did what they said they did and that leaves it lets us have another level of composability but we have some downsides we have to learn a weird language we have to think about programming circuits in some kind of unfamiliar model and the math and the security is difficult um so the big picture about what a zero knowledge proof is is it's a lot like a digital signature it's just a digital signature has public inputs it has a private key it has a signature function that we compute as a function of those two things and we get a result and we get a signature and then we verify it on chain or more precisely the at the protocol level the chain decides has you have you just have you signed the thing or not okay a zero knowledge proof is just lets you replace the signature function with any function you like and we're here today because we think it's interesting to replace it with a machine learning model um then we get something that combines all these properties it's decentralized it's trustless composable it's scalable because one person can make a proof that everyone can believe but we can also get our familiar python workflow libraries of existing models and hopefully we'll be able to power some kind of on-chain organizations right so if off-chain organizations need to deploy a melt or run and we're building everything new institutions in web 3 we're going to need to run some of these programs too and our goal is to make that super easy basically you describe what you want to do you may build the model in Python the way you always have and then we help you to build and deploy it into a pipeline so what does a pipeline look like basically it has three parts four parts you would there's some kind of Truth for the world you download a signed piece of data from a news organization you download a piece of data from the chain that's attested to uh and then you check a signature that's ingestion then you run the model which proves that you've computed this function correctly given these inputs if you believe them you get an output then we verify it you know because we're here at Denver we only care about verifying on chain but of course there are many other possible ways to do it and then we have to do something like fire off a hook uh enter into the index or let a smart contract do what we ask the smart contract to do and again our goal is to make that kind of a simple process build your model in Python Define your pipeline and deploy it and just to talk high level for for one more minute you know why zkml I like to think about it as we're giving the blockchain eyes and sensory organs uh with which to perceive the physical world we're giving ethereum uh eyes let it let it interpret ambiguous events long term the reason I got excited about this is that uh I think I think this is the way to make it possible for humans to own digital Assets in the sense that the chain will learn an ensemble of models that knows who I am if I lose my private key it's no big deal right there it doesn't all depend on some Single part of failure just like I own my assets and my identity in the society in which I exist by a whole host of different relationships and if my house burns down I can still walk into the bank and prove I am who I say I am I think we need to get to that future for broad adoption I think this is the way to do it of course we're a long way from from that um however I think that uh the longer I've worked on this and and lots of other people do great research and work I think where that future is getting closer extremely fast um I also say that it's kind of a way to make smart contracts exercise judgment put the smart back in smart contracts right we can have a smart contract now that is as smart as a human lawyer or maybe smarter that can make decisions about did someone fulfill the bargain that we made with them did they if they were contributing to a Dow did they you know build the web page we asked them to did they run the errand whatever those kind of things so that we can someday all work for you know II on chain which maybe not but um okay so I want to talk quickly about some application examples and then I will go to the repo and talk to you about how to use it to build something um so one application that a lot of people are excited about is vaults so there are protocols like Iraqis that force a choice you can this is just a quote from their website apologies to them if anyone's here um so you have basically two options you have trustless vaults with predefined on-chain strategies they're in a smart contract they're trustless they're automatic but they're simple and they're not private right they're public or you can have manage vaults that have sophisticated strategies depends on somebody downloading the data running on their laptop or on a server you have to trust that person that they won't suddenly buy some token they just created um and they so they require trust and of course EK fixes this because it lets us have automatic trustless off-chain strategies this is a D5 section so this is important to emphasize in other words I can define a strategy I can commit to it I it can stay private if I have great Alpha it can run on the on the Chain against this Vault for any period of time two months I see the performance people can invest in that strategy and know that they're getting exactly the same model that was that was previously running so it's like forward test um which is really cool and you think about it in the context of something like an amm you know I came from originally a traditional Finance background and I thought MMS can't possibly work they leak all you know it's you're trading a public strategy it's just going to leak everything right and to some extent they have um and they make up for that with fees right so but what DK lets you do is have at least in theory one of these things that uh where some of its strategy is private and so it doesn't leak as much information it's not as easy to trade against and therefore it can charge lower fees right so it can make it everyone else a little bit better off it's like a reduction in the dead weight loss um here's an example of one of these that is being built by some folks here um that uh lets you define a bunch of parameters um in terms of what you want a vault to do and then it re-centers it uh periodically in this case like every 12 days it would run the machine the machine learning algorithm through zero knowledge proof uh there could be triggers that that additionally do it and you get some kind of definition and in this case it's a fairly simple uh concentrated liquidity management manager that's that whose parameters are being managed by a machine learning model but uh you can imagine that in the long run this is going to be a much more sophisticated object and what these folks want to do is to build uh a kind of like a no code low code tool that lets you create D5 strategies and the way that we enable the flexibility of those strategies is by being able to compile them on the fly into a into a proof and that's what the role that Ezekiel plays um here's another project that's being built this week called NPC Wars uh it's kind of a decentralized kaggle people can compete using their AI models against each other in on-chain competitions they upload their resulting great Onyx file uh Ezekiel generates the verifier and the proof and then uh you know the winner the person who had the best model wins the prize this their design is so much better than mine I I just can't even anyway so this this is the design it's like you have an your model is an NPC um these are the four authors of this uh and you can create your your model and then fight and watch the leaderboard uh and another uh another one that's really cool is the scavenger hunt application um which is uh where is it oh it's on the sorry uh here's the demo so it's called Hunter Z Hunter it's a scavenger hunt for uh for images and this is kind of like a pilot for uh identity right so someday this will be your face instead of maybe Sunday might actually be like two months from now but someday it'll be a face and the basic idea is you um you create a hunt you give a hint like I'm green and full of chocolate you give the hunt an end time uh and you choose a photo that's your Target photo this enrolls the image using a zero knowledge proof you know with Ezekiel running in the background um then the Hunt is created and then you can um as a player see oh here are the prizes for the different images if I can find the photo that will that will unlock this or that will make this neural hash um then uh then I get the prize so in this case um you submit if you submit the wrong photo uh it'll tell you there's no match if you submit the right photo then it will it will show you that there is a match and uh create the proof only then lazily and then when the proof gets submitted to the chain that unlocks your prize and sends it to your wallet and so that's a game that should be playable um before the uh before the end of the hackathon uh for anyone else to try that out um great so now let me uh switch gears and just talk a little bit about the repo itself so I'm going to talk you through just a little bit of the readme my goal here again is to make it so that so I don't want to build any applications that's not my thing I don't know how to growth hack or do anything like that my goal is to let you build an application with a minimum fuss um so this is our repo uh it's supposed to be easy zero Knowledge Learning that might it's it's getting easier um and what the high level what it does is it lets you define a computational graph for example uh usually a neural network but it could be something else you take that final graph you export it as an onyx file which is basically just a if you dump the abstract syntax tree of the forward pass of a python model that's what it is it's not very complicated and then you point Ezekiel to the Onyx and the Json files and that generates a circuit you can prove that I ran this public network on a private data to produce this output like in the hunt example I ran a private neural network on some public data and it proves this out proved this output like in the the percept uh ZK Vault demo um or I correctly ran this publicly available neural network on some data and it and it produced some output in like a public Oracle I want to prove that I want to make a market for predictions and I want to prove that you know I downloaded a story from a reputable news source you can find one and uh that you know someone ran like won an election or a hurricane hit something like that we use Halo 2 in the back end we love contributors please contribute um there's a you have to use nightly build rust so sorry about that but otherwise it's not too bad you can install it with cargo install there are documents so there are tons of tests we spend lots of scary money on tests so that we run every single one of the proof pipelines on tons of examples to make sure that nothing breaks for you um there's full documentation every function and every struct is documented uh if you want to read about the details of the internal internals and let me just walk you through the command line because I think that gives an intuition for like what we can do right what we're doing so once you have a command line tool Ezekiel the first line here is Gen SRS so the first line is if I increase the size a little is that better yeah okay so the first line is you know this is uh this is using kcg commitments because we want to compile and verify on ethereum so we can either use a public ceremony or you can use a you can generate your own SRS that is for testing only and is not secure that means the person who generated that can generate false proofs so but it's just as easy if you download one of the powers of Tao ceremonies you can use that and then you ask it to prove it has some parameters which I'll explain in a second and it it takes an input file that's like the image the picture of the chocolate bar or whatever uh which is as Json file it takes a model which again is a Onyx file that you export from python um let me uh let me show you what how one of these Onyx files gets created oh right there's no internet in here okay maybe not um so uh basically there's there's a very short Python program you write you say import torch import Ezekiel you write you define your model Pi torch you know subclass of that and then at the after the you know forward pass same as standard completely standard Python and at the end you say export export this model with this input and then that that will create the Onyx file for you there's examples in the in the repo um right so then you run uh the proof uh and with it with this model file and you tell it where you want to put the proof and the verifying key and load the parameters and then that will compile the model to a circuit generate the proof that this model you know when you took these inputs private by default uh this is a proof that the inference was run correctly you can also verify again using the network file and this time not the input but the proof so now the in the original input of the proof is possibly secret you know I don't have to reveal what what picture of a chocolate bar I created or I don't have to feel my strategy in a D5 Vault um and then uh it produces uh we'll verify it directly there's also a table command that will just print out for you what our strategy is for turning the model into a proof so it'll sort of show you how how it gets compiled and how what decision we've made at every stage of the of the creation um we can also verify with an evm oh I should say what these parameters are so um these two parameters bits and K so in uh one of the funny things is that uh not is the way non-linearities work right so non-linearities what is a what is a deep learning model it's just a bunch of matrix multiplication ax plus b and then you hit it with a non-linearity and then you do it again right and there's some other details but you know mostly that's all it is yeah foreign yeah so the question was um I can't repeat the question but let me let me rephrase what so what we're doing is we're taking a we're using an onyx file a python model as a spec for a zero knowledge proof as a description as a high level language and then we're turning that spec we're compiling that spec into a circuit and running the circuit I don't know if that helps um okay so so bits is a right is so at a technical level we're doing ax plus b and then we're hitting it with a non-linearity like relu which is like if it's less than zero give you back zero if it's more than zero give you back the thing or sigmoid function or whatever non-linearity uh and when you you have to quantize and when you implement a nonlinearity a really nice way to do it is with a lookup table uh which is a zero knowledge proof thing a lookup table basically does the following and same thing I'm going to write down every possible input value of the function and the matching output value and I'm going to prove that in my Trace everything that came out is somewhere in that table so that seems incredibly inefficient and insane and yet it's the best we really can do so far those arguments have gotten better and better the nice part is that cost is constant so even if my function is super complex if I say give me relu of sigmoid of tan divide by seven blah blah blah that all gets compressed into one table so that cost is kind of constant and that's just a hint of how weird the cost model is when you think about zero knowledge proof as kind of like a computer right it's a very strange plus model and we're also figuring out kind of what works well anyway that's what bits is bits is how big is that table so this means I'm writing down 2 to the 16 input and output values that's adjustable this the quantization scale is adjustable use a fixed Point um fixed Point arithmetic and you can choose two bit representation or 16-bit representation whatever you think works for your example um yeah and let me get to verification with the evm Ezekiel gives you two options for verifying with the evm you can either generate a direct evm verifier which is a single smart contract that runs the whole verification Loop right now this can only handle small things because there's some optimizations that need to be done but it works and that creates a solidity file that has a verify function that's exposed you can deploy it and you can just call that verify function from your other solidity contracts uh and the the other strategy you can use is using proof aggregation this means you're doing a two-stage proof stage one is this one a single proof where we feed it into we create a single proof like a device or something it's fairly easy to create it uses a different kind of transcript for the cryptograph cryptographers we're using a hash friendly functions hash friendly ZK friendly hash function in order to construct our cryptographic sponge so that it's easy to check inside of the next proof then you generate an aggregate proof which kind of compresses that first proof into something that's much easier to check on the evm that aggregation proof can be very time consuming and use a lot of RAM and then you generate the verifying code which you which and give it a deploy uh deployment path parameter and that will let you uh you know deploy that and and check things against it and you know these are the these are these circuits are the ones that the examples that I gave earlier the people they're building uh here at the hackathon are using these verifiers to to deploy it on chain and then we can test that that verification works by using Foundry anvil in the background to run the evm for us um and we also have uh we also have some deploy commands uh if you feel if you're feeling Reckless and you want to put your mnemonic in a text file hopefully just for testing you can do that and yeah the original like make your life as easy as possible to play with this stuff because it's really fun uh and then you can even use the user command line to send the proof some of the folks that are building on it uh this week have also built an RPC uh client that that you can talk to and so if you don't like to use a command line so you can talk to it directly from JavaScript and we'll release that uh probably in the next week um yeah and then as I mentioned you can use a pre-generated uh structured reference string um and let me just talk through the commands uh for a second and then maybe I'll leave five minutes for questions um so table is the command again that does the layout so let me say a little bit what the strategy is so you get a so when we started doing this we were handwriting circuits right so you see you you print out your your Onyx file or you write down your python thing and you say okay I need these these Matrix multiplications by hand I can put all that into circuits and cercom or in Halo 2 right and those things are going to be fast because I can customize it and if I have an overflow error I can adjust things until I can fix that overflow error and so what we're trying to do is to let you define The Circuit by just by you know just writing some Python and so we tried to put all of our modest intelligence into the um compiler so that the compiler will decide for you you know come up with a strategy how do we think we should quantize this by quantize I just mean you have floating Point numbers we can't work with floating Point numbers all we have is field elements so what we do is multiply by something like 128 or 256 round it and then keep you know think of the integer as the numerator and the 128 is the denominator and then you have to kind of keep track of what all the denominators are as you pass through and this leads to all kinds of interesting numerical problems which are kind of fun math um anyway so we have a strategy for that table kind of tells you what's our strategy what do we think could go wrong in terms of values and overflows uh how have we converted each of the Python operations into a zero knowledge proof operation stuff like that um render circuit if you want to see what the hit with the layout is in Halo 2 you can see that if you see a lot of pink space let us know and we'll we'll turn some dials to make the optimization better um forward lets you run a forward pass so to know what the quantized model will say your input maps to so in the in the example I gave the scavenger hunt we use the forward pass to decide whether something is going to work before we actually run that proof so we know exactly what the model is going to think what the proof will be able to prove and then we can decide whether or not to put it in uh Jan SRS I talked to mock is a function from Halo 2 that lets you it doesn't construct a proof but it checks all the equations that describe the proof uh aggregate again turns a small proof into a smaller into a first layer proof into a second layer proof for evm verification proof I talked about this creates to this the uh Eva the uh Yule or bike and byte code contracts uh this deploys them if you trust this if you trust the command line with your private key and you can verify natively or you can verify the aggregate or you can verify in the evm you can print proof and you can call for help uh and uh I think that's most of what I wanted to talk about there's some other bells and whistles like if you get tired of the command line you can put all those command line Flags into a Json file or pass that whole thing uh some other stuff like that there's benchmarks and there's examples um yeah so let me leave the last five minutes for questions thank you thank you so much for the talk you gave one in devcom Bogota right where you can you hear me now okay I was asking if you gave the same talk in death combo because I yeah okay perfect so okay no word so my question is that um the Onyx model supports a bunch of operators right does this repo support all the operators or okay like how what is the percent 1500 okay better okay perfect so um can you run the main models uh with this repo like this YOLO model can you run it you understand great thanks hey can you so can you change the input parameters between like verifier and generating a proof so you have these bits and K so can you change those parameters or they should be the same throughout the same for the proverb okay okay and and for some of the models okay I think some of the you have to increase the parameters because the proof cannot run or something yeah you have to sometimes so you one way you could think about the situation with zero knowledge proofs is that there's all these dials right like how many rows there are how many rotations I use how many columns I use and a whole bunch of other things um we're trying to pick some decent defaults and then a scale and everything but you can turn these knobs and there is some amount of Art in terms of like turning these knobs until your proof has the performance characteristics you want most of the knobs have good have you know positive and negative things so like I could turn a knob it makes the prover faster but it makes the verifier larger or it makes the verifier faster but it makes the proof larger stuff like that right so there's all these trade-offs and as a field we don't even understand all these trade-offs uh but you know we're all learning together and was the limit of say suppose number of parameters that should be in the model that your package is capable of what's the number of parameters yeah it's not there's not really a fixed number it's kind of like how patient you are and how much money you want to spend on Amazon thank you last question so it's nice to do machine learning on many computers like a cluster but it sounds like you can't with this because the AST becomes circuit is linearized is that is that right um I'm sorry can you so the so you wouldn't be able to use Ezekiel with a cluster of computers to do the training or or running the model right so okay so training is a big question right so I have two part two responses to can you do training with this one is you should think hard about why you want to do training with it right because it better be a good reason because it's going to be very expensive it's going to take a lot of time and overhead right that said I don't think it's impossible I think actually the training will start to get easier as proof system Innovations kind of hit production um so you could you can generally you can avoid it but what we're what we're proving is an inference pass it's basically the same cost as a backwards pass so I don't think it would be too too hard to implement that and be able to do training it's just going to be really slow yeah thanks okay thank you [Applause] and if if anyone has interest in building with it or questions foreign