ZKML: Verifiable & Privacy-Preserving Compute in Blockchain

so I was talking about the October event and uh much closer today it's happening this Saturday we will have an upcoming uh event in Apu uh Apu it's in bukit jaryo uh we always we often have a once a month offline session and uh this one is going to happen on the 17th of June 11 A.M in Apu uh these are the um the agenda and a few videos head on to the Meetup page to this attend and a few video by uh this Saturday I will be there but many of many of the the company will be there so it's not that'll be nice for you to like say hello to everyone and and now finally uh let me stop share and I would like to introduce caddy um uh to talk about uh zkml and I will let her take the stage so welcome Kelly yeah thank you Sichuan um let me share my screen so everyone should be seeing it all right now let me also grab the chat window right so today I'm sharing um about ckml zero knowledge machine learning uh this is the first time that I have ever been speaking about ckml for such a long time like a long duration so we are going to start from a little bit of Basics a little bit of Crash Course and then we're gonna go deep very quickly so bear with me in you know the first few slides uh if you uh if anyone who actually already know about CK I apologize but um you know just so that we have everyone on the same page I'm gonna start off like quite easy to begin with uh right and also be I also prefer to take questions during the talk because especially because it's so long so if at any point anything is unclear to you or uh you know you have any questions just feel free to um either like put it in the chat um or something like that so that I can see it right so this is today's rundown um essentially what we're going to do is I'm going to break this into three parts which is basically things that has already happened or say a little bit of background or where Chanel you know things that you're supposed to know already uh and then we will talk about what is like the current like status quo what are the challenges that we are facing in ckml um you know what are some of the open source libraries that you must know if you're getting into ckml uh and then we're gonna hop into the future which is uh of course I'm representing PSE to present this topic here today so we'll talk a little bit about what kind of research we're doing here uh also you know if you want to get started in ckml where should you get started how can you get started and at the end I would like to uh kind of invite all of you to kind of let's just brainstorm and um you know think about what we can do in this space all together right so um sorry for the past supposedly everyone who is into ckml who or who's interested in zkml would know what is ckp which is zero knowledge proof but just a really quick crash course in case you don't know what the heck is zero knowledge proof so well zero knowledge proof of course means that we're trying to prove something to another person right and this is relevant to blockchain because a lot of the time we want a trustless or a trust minimized process and it's very difficult to do that on the blockchain um so when we talk about serial knowledge proof um on the blockchain typically we're talking about the process that looks like this so uh what you will have is that a there's a two entities that is um a play here so we have on the left hand side who is the prover and on the right hand side who is the verifier and this whole process is called a non-interactive ZK proof because um there is no multiple kind of back and forth and interaction to it right so it's a one-off process right so the proofer would be making some kind of a statement um that statement could be so let me give a very easy example let's say I want to prove to you that I know how to add two numbers together right or I know what numbers add up to 10. um so well there are a lot of choices where you can do one nine two eight three seven four six you get the idea right and the idea is I want to show to you that I know how to add to 10 without letting you know what numbers I have in mind and so here I'm kind of also you know as a prover I'm going to construct a proof so that I can give you kind of a mathematical statement you know a few numbers and those numbers uh according to mathematics is gonna tell you that I know how to add up to that right and you as a verifier on the right hand side I'm going to send you the proof and you're going to run it through a verifier to check the proof um so for blockchain in particular uh the verifier here is a smart contract so it's not really a person or an entity it's a smart contract that we Deploy on chain so that we can have a uh you know a trustless way of saying that okay I give a proof to a decimal contract this smart contract function is going to give me an answer true or false right so if the proof is true it will return true if the proof is false it will return false so that's the kind of the basic idea of having ckp and really there are three properties that you'll usually hear about ckp um which um it might seems very kind of self-explanatory and obvious but I'm just gonna put here regardless so that you have an idea of you know what is uh you know what is what is the ckp so typically when you see when people talk about ckp in the mathematical terms there's three properties uh which is completeness soundness and zero knowledge so completeness just means that the verifier is gonna give out true uh whenever you are actually giving a valid proof right so the verifier is not going to have false neck uh sorry false negative ah I have a mistake in the in the slide here I will I would change it up and after afterwards so I will provide a link to the slides after this talk as well sorry so completeness is no false negative so the statement here in the middle is uh is wrong completeness means that there's no false negative so if the approver is understand the clarifier should always say that as a true statement whereas soundness uh is the converse so soundness is saying that if only if the statement is true then the verifier is going to say that it's true right so it should be no false positive right so this ckp process the verifier is neither gonna give false negatives or false positives in other words is 100 correct all the time and most importantly uh zero knowledge right so I can prove to you that I can add up to 10 by telling you the number two and eight for example but how do I hide those number and still prove to you this um this result and that's where the zero knowledgeness come in comes in right so there's no information that is linked to the verifier except for the fact that I want to prove that except for the statement that I want to prove to you so this is typically what we will see in a lot of um you know classes online materials um but really there's one more important property which is typically less talked about but we make use of it all the time and that is succinctness and I want to emphasize this because it's an incredibly useful property for a lot of things that we already know actually umness means that whatever we are trying to prove the size of the proof itself is usually significantly smaller than the underlying computation right so I was saying that for example uh in order to actually do the computation we might need 100 of code right whereas the proof itself is say three lines of code okay I'm just making these numbers up three lines of codes right but the beauty of this is that we know that blockchain is very limited in terms of size and computational resources so this actually comes in handy we'll talk about this again in a little bit and in fact we always use um this uh property all the time right so why do why do we want ckm TK uh why do we want ZK or zkml on blockchain um I would propose the counter question which is why not okay just joking but but really why not right because ethereum uh after all is called the world computer uh it was originate originally proposed as you know doing all the possible computation that is that a computer would do right and one of the problem we have right now is that there's actually a lot of limitation with the blockchain uh and there's a lot of things that our computer can do that ethereum can do for example um we know that there are mainly two limitations of ethereum um so namely first of all the if uh the ethereum has a limited block size and also contract size right so you can really fit a lot of things into a block you also really can fit a lot of things onto a contract a contract is around 24 kilobytes right now right so if you want to do very complicated stuff you will probably have to break down your programs into a bunch of contracts and that's that's not really um what it uh the blockchain is intended for um originally um the other problem we have is of course ethereum like any other public blockchains the transactions are fully transparent right so we don't get a lot of privacy here of course it's a pseudo pseudonymous right so we get um you know whatever we do is attached to an address not our personal identity but still everything is on the blockchain is kind of transparent for you to see and with that uh actually ZK can help tackle both of these problems right it's all for ckml uh there are two things that you can do right so just now remember the fourth property of ckp is succinctness Right In other words compression so uh one of the way that CK could be helpful is to compress very complicated computations of chain uh so that it becomes a very small proof and that small proof can be submitted onto ethereum mainnet for verification right and one of the examples that actually we all know about or use nowadays is uh ZK zero knowledge base L2 Roll-Ups right and that makes use of this um you know in the altitude the block size is huge how do we actually um are able to verify this state or uh transition or state changes uh and that's because um we have this succinctness property of ckp and to tackle the second problem where all the transactions are fully transparent uh CK can actually help prove ownerships of certain information like data or while maintaining privacy right so uh we all know about the famous tornado cache uh that is one way that the CK can help maintain our privacy right there is also a lot of different CK ID programs or projects out there that help maintain privacy right uh but if you realize so far I've put machine learning in the bracket because really we can have all of these already right without zero knowledge machine Learning Without ckml so why do we need zkml right if we already can use ZK for a lot of different different things why do we what is the need for us to absolutely put machine learning on chain right exactly so I will get to the next part um yeah I saw the question from Lucy yeah so roll up has nothing to do with ckml but data has a lot of things to do with ctml right so with CK already we can prove that we possess certain data right so imagine if we're trying to prove some kind of data um uh currently without ckml what I can probably do is to say that okay I have a picture right let's say a picture of me I can I will be able to hide all the other pixels on this picture and just show you like essentially ZK is like covering my picture with a huge piece of cloth and showing just a little like peek a little hole in the cloth to show you hey I have certain pixels in the middle of this picture here right so in other words I have a picture where I can prove to you that I have certain pixel app you know X Y location that is of value Z the problem with that is that it's not really a meaningful proof right sometimes just with pure ZK the proof uh we get is Meaningful because maybe we're hashing something maybe we're just trying to do mathematical statements we're trying to prove numbers but a lot of the time with other kind of data uh this kind of proof is not very meaningful and that's kind of why we need ckml because the ckml or machine learning itself is it's uh will allow us to do a lot more than just data proving um or value proofing it will allow us to do more like data indexing or data labeling right so imagine if we have want to prove something about our data right like something like I don't just want to show you that there is a picture that I have certain pixel I want to show you that I have a picture of a cat right this is something that just using CK alone is not gonna help we will need some kind of machine learning model or well even if it's not a machine learning model some kind of model or some kind of algorithm that is able to tell us that this picture is of you know a cat or of a dog so this is something that the ckml will come in very handy which is to prove something about the data content itself and that brings us to what is zkml really okay so um when we talk about ckml this is kind of the structure or the architecture that we talked about and it kind of looks like this okay so the idea is that we have a Serial knowledge circuit okay for those of you who doesn't know what there's a circuit you can imagine this is just a program um a program that can perform what our neural network can perform right so NN stands for neural network so let's say we have some kind of CK program that can already perform neural network operations um then there are three types of data that comes in and out of this circuit or this program um so there are three types of data uh there's input data model weights and output and the what these data represent actually depends on the use case right so given uh give let's give an example so for example if this is a um stable diffusion model right stable diffusion model that generates uh an image given a prompt so the prompt could be something like an astronaut riding a horse right so that prompt itself will be the input data sorry uh wait hang on I think we're good now wait hang on yeah okay I just needed everyone okay sorry no problem um so the input data would be the font itself right which is the an astronaut riding a horse the output will be an image and the immortal way to adjust whatever the stable diffusion model has as is like in assets parameters a weight params in inside the the model um you know another use case could be let's say if our neural network actually perform a facial recognition then the input data would be an image of our face the model weight is still whatever the weights that the model takes in and then the output is um an encoding of our face right so essentially our face is called face ID on an iPhone because it encodes our face into kind of uh a hash or a a a encoding a string that identifies our face so let's say we can already perform that and this is the kind of the whole construct of ckml right now uh just want to make a note that we're just talking about inference or prediction uh not really training yet so model training is another thing that we're trying to tackle but most of the time we're talking about inferences right so um this whole machine learning inference process uh with this uh construct we can have three types of use cases and the first type of use case might be a little bit counter-intuitive but closely tied with what we were saying again just now so the first type of use case is where everything is public okay so typically the output is already public so I've omitted the label here but what if we just make the input data public and also the model Waits probably you might ask then you know what's the point well the point is succinctness right so we are gaining compression um again there is no way that we can have a whole machine learning model inside a solidarity contract so we can essentially well do whatever machine learning off chain right so we can perform our inference of chain uh and then proof on chain so to give a more concrete so essentially it's like a roll-up but for machine learning specifically right so to um to give a little bit of like idea or use case or an example of how this type of use case might look like I want to present to you which is a recent uh ERC standard that I have written uh if anyone is interested you can look up like erc7007 I don't think it has been merged on the website yet so you might have to go into GitHub to take a look but the idea is you can do something that is like well we already know that AI is widely used in the nft space right there is a lot of nft that is generated by AI or at least generated by algorithm but the problem is I can say that I have created a image right I've drawn an image but I can say that it's an AI generator one and vice versa right so there's a lot of problem with like intellectual property or how can I even prove that I own this image right so with aigc AI generated content the way that you proof you own this image is to show that you know you have run through this model and get this output right so the idea is well we can have a verifiable AI generated nft by utilizing ckml so the kind of the process goes from go as follow so you have something like a user proposing a prompt uh The Prompt what goes into the approver right uh which the model will split out a of course the output which is the AI generated uh media and then the proof right so you need to submit everything to this kind of claimable nft smart contract uh and then this smart contract is gonna verify your proof and only when the proof is verified it's going to allow you to Mint it right so the idea of this whole uh construct is that not only are you proving that you own this media image you're also proving that I I'm actually created it right by creating this I am the person who proposed this prompt or who is the first to propose this prompt into this model and this is the image I got so this is kind of like a way to gain like verifiable computation of chain right so I've seen a question in the uh the chess said what is the progress on ERC 7007 uh and the progress is actually we're still discussing on ethereum magicians thank you MK for sending out the link um the draft should be merged into the EIP very soon but we're still at the stage of drafting and they're based on the discussion there should be a lot of changes uh coming in uh in in the next kind of like edit um so yeah but feel free to go on to like if you're a magician and put down your thoughts um and I'm really looking forward to kind of have like the first aigc model uh deployed as this standard here right so um is there any question yes I do actually a follow-up question to you the eip7 7007 so uh from a friend actually so um he was asking um he was he was wondering if because generate models are very expensive to prove and uh will that in any way affect your proposal such as you need to it's a very expensive proof and also I don't know how expensive is to verify and will that be possible at the current state of ethereum yeah so um normally it's much more expensive to well we'll get into the trade-off but essentially well prove your time and verify your time is kind of a trade-off right so usually if we have a if it's more expensive the proof it's a little bit cheaper to verify on chain and that's usually what we're opting for right now because we want it to be cheap on chain uh and we don't mind doing more work of chain um uh we also get into like actual numbers but the I guess like for now the idea is that you can only do very simple models so don't think of like doing an actual photo stable diffusion um because this is probably going to be very expensive for whoever is um you know volunteering or even getting paid to be approver right so um uh yeah so the machine that we're talking about if we want to do an actual stable diffusion it's actually huge like in a terabyte of ram um so that's not very feasible but it's totally possible to do a small scale AI generator model right so when we say AI generated model it doesn't have to be everything that is like as perfect as stable diffusion or Del e it could be something that is trained to do a much smaller image um or you know a really a subset of you know whatever it's capable to generate so I think the idea is that a lot of people might be using it to generate like abstract art pixel art things like this like that is much uh cheaper to prove nice thank you yeah and also to answer your question the cost of the model uh proving itself should hinder the process of 707 because it's not really dependent on what model you use it's just a process or just a recommendation on how this process should look like cool and so that is the first type of use case which if we were to sum it up in one word or two words as verifiable computation uh but then we can also get into more privacy related use case so type 2 of use cases is um this which is um uh we have now the input data is still public but our model weights is private right uh in other words did I mess up no no yeah no that's uh yeah okay my bad let's talk about this type person um so the second type of use cases is the private input data and public model ways okay again this is a now more a more intuitive um use case whereas you have um private data which is something that may be related to personal sensitive data that you want to hide and you have some pop like kind of like known model verified model that is possible for uh for people to trust right and this second type of use cases uh would be kind of anything that is related to privacy or requires privacy right so uh this is kind of one of the space that is the least uh explored right now mainly because if you want to have a very good model typically these models are big they're more expensive to prove but this is also one of the I guess the most anticipated use cases type of use cases right so imagine we if we have a model that is publicly known to be very good at recognizing fingerprints or your face um what you can do is we can essentially enable biometric authentication for smart contract wallet right so we don't have to have our biometric authentication happen on our device instead we can have it on the smart contract itself so that's one um that's kind of the most kind of trustless way in order to perform biometric Authentication um the other thing we could probably do is to finally have some kind of private data Marketplace right so what we can do is well for example I want to sell you and an image of um you know something that I've took off the the Hong Kong famous Victoria Harbor for example right I want to send you this image I don't want to post it on lines so that everyone can everyone can just right click and copy um how can I do that right how could I possibly be able to prove to you that I have certain image right so I could maybe have a machine learning model that is very good at recognizing scenery or annotating images I could have a machine learning model that is good at you know distorting and downsizing an image so that we can essentially have a provable way to tell you that I have this image even before the transaction happened all right and then afterwards of course you can use Smart contractor control like how you can actually do the transaction and you know encrypt the image and send it over so there uh you know with ckml it could make a lot of things like private Marketplace possible um right so I think this second type of use case is quite easy to understand so now let's go into the third type of use cases so again don't mind the numbering um so the third type of use cases is actually the opposite right so just now we have private input data and public model weights uh sorry I'm hearing some noises is it from me or oh okay uh I mute it uh no I was just worried that it's from me okay oh yeah perfect cool uh so um the public input data and private model weights sorry um so again this might be a little bit puzzling uh and not as intuitive with the previous type of use case that we mentioned and so well what are we doing with this type of use case so with this kind of use case it could be something like a web 2 company or really any company that has a proprietary model that they have but they don't want to tell you what it is right but they also want you to trust them so they wanted you to trust them in terms of for example um they didn't really well they want to prove to you that it's the same model that they apply on everyone right so a good example is for example uh Twitter right recently they open source their algorithm but let's be honest who is Ever Gonna actually go and look at their source code or actually run their full algorithm right and also how do you know that this algorithm is actually what is being applied on you when you're using Twitter right so there's no profitable way and open sourcing it actually doesn't really help with proving that this is the algorithm they use on you however there's actually a better way with you know ZK which is I have this algorithm I am going to show you that this is the hash of my model right so I'm hiding my model weights but I'm committing to certain model that I'm using with a hash and what I can do is I can show you that this same hash is the same model that I applied to you you know and then to to Kathy to chichwan to Kim to Lucy to everyone and the way you show it is just to check the hash that is present in this ZK proof and that is actually a better way of showing fairness a better way to you know protect proprietary model and one example uh that you can imagine what we can do with this kind of use case is actually kaggle or z kaggle um so for those of you who doesn't know what is kaggle Kegel is a web 2 kind of platform for um companies to post their problems their machine learning problems onto their platform and post bounties uh and then for machine learning Engineers to try to tackle the problem and submit the model okay so that is what the web 2 version of Kegel is uh the problem with it is that in order to kind of get the award or even get evaluated you have to share your full source code before you even win the prize right so let's say there's 1 000 people or ten thousand people who join this competition the all those 10 000 people will have to submit their source code to this company of course the regulation probably says that okay if you don't win they won't use your model blah blah blah but this is not a very trustless way of doing it right so a better way of doing it is would actually be something like a ZK kaggle or C kaggle in short um so the idea is what you can have like Bounty Bounty whereas the Bounty is split into like private data and public data you know people can use the public data to train the model than the model from python can be translated into like a CK circuit using some transpiler that I will mention later and then we can submit kind of this model onto the platform maybe encrypted uh and then we're able to prove that our model has certain level of uh accuracy right so we're not able to score our model that we submitted without disclosing the model fully and this enables people to kind of have a very trustless way of uh first fairly evaluating all the model all the models um having a provable accuracy or provable performance metrics without disclosing the source code of the model itself so um this is actually a like a POC that I've done um again with the current limitation we're not able to do very large models so this is not like in any way practical use uh yes but we're able to kind of at least prove that this concept Works uh with kind of like so the idea is so the whole process will kind of like this so a bounty provider would would post like would submit a transaction and post a bounty uh people will submit their Tran model uh with the approvable accuracy right and then the model evaluate uh the model the Bounty provider is gonna release the Bounty and at the end when the person come and claimed the price they are able to decrypt the model for the Bounty provider right so it's kind of a s crowd process still but with uh with a little bit of more verifiability coming from CK um yeah and I I as a very short kind of introduction of a more complicated project so if there is any question um I'm happy to answer as well all right I don't think we have any questions for for this one yeah so if there's no question then let's move on to kind of talking about where we are now so as you can see uh people are very you know hyped about zero knowledge Machinery um that I also realized like people are always very eager to know but like currently we really don't have any products right so where we are at uh right now for ckml is really just these four things right so we have a lot of hackathons uh actually and they in The View kind of upcoming uh Paris hiker houses most of them are CK or ckml themed so we can expect that there's a lot of hackathons that are going on uh with a lot of hackers trying to tackle the EML uh we are also at a lot of proof of concept stage right so just now the zika go as a proof of concept um we can't really do any model bigger than something that is like um um something that can work on a MNS model so for for those of you who doesn't know what an amnest model is and this model is actually around like tens of thousands of little images which is like 28 by 28 so varied teeny tiny images of um black and white handwritten digits so zero to nine right so other than that we probably need some bigger servers a lot more powerful machines to in order to operate right so a lot of the things has kind of like um halted in the proof of concept stage of course we're also doing a lot of research right because we cannot get past this proof of concept stage then we also need to do a lot of research trying to gain a little bit of that acceleration and how how to actually like speed up or be able to do larger uh larger models and we are also stuck at moles more more likely server side as he came out right now right so by saying server side means that the ckml has to the prover has to be a huge machine that is probably on the cloud uh as I mentioned there we need like terabytes of RAM um so the problem is if we are doing it server-side then we are not really able to gain a lot of these use cases that is related to privacy right because the whole point of ZK is to have the proof um done on device right on client side uh So currently we're kind of stuck with server side and I will explain in a little bit like why I said this case so understanding like where we are at right now kind of bring us into you know why we are at where we we're at right now we're just the challenges that we're facing um so there are three sorry uh Lucy has a question with respect to being able to run larger models custom Hardware work uh English yes so that's absolutely true um uh a lot of like uh Hardware acceleration is gonna help um the problem though however with just basing on Hardware acceleration is that uh you were if we're depending on how well acceleration we are still most likely gonna be stuck with server side ckml because the idea is you wouldn't really expect a user right a regular user to have that kind of custom uh specific machine at their home and as long as we're kind of stuck with server side Emma zkml the problem is still that we don't gain like privacy application right so that brings us to kind of the current challenges why we are stuck with POC why we're stuck with uh server side and the first challenge is actually very simple it's about mathematics it's about the different types of mathematics that ZK and machine learning use so machine learning models is all about like gradient calculus um basically everything that will require a lot of decimal points and hence usually the best performing machine learning models uh when we train them use floating Point weights right so at least 32 bits um and this is what kind of makes machine learning models so powerful is that we're able to gain like those little gradient change at a time however the CK circuit uh give any one of you have uh worked with CK before you will realize that we are typically working with integers right or um or a better way to put a fixed Point arithmetic so these two kind of um mathematics scheme here right mathematics system here number system here uh it's inherently incompatible uh well the way luckily the way machine learning uh we do machine learning on our phone or mobile has already helped right so our phone is also not as powerful as our computer so machine learning Engineers have been non-known to how to move like their models into a 8-bit or a 16-bit models using quantization the problem with quantization is uh twofold right so first of all you can have host training or pre-training quantization right so pre-training pretty much means that you are trained with the idea of quantization in mind so that like when you convert to a quantized model you're not going to lose a lot of accuracy uh there is also post training um quantization which just means that you train the model regularly and we're gonna try our best to quantize it you know by by looking at the numbers uh putting them in like quantized bins and trying to get it to work and with that actually we typically lose a lot more accuracy so this has been a challenge especially if we're looking at some applications that requires very high precision um so for example if we're looking for fingerprint recognition we can't really afford to lose a lot of accuracy right because like that's not a really good thing um if we are going to like use it for Mass adoption so that has been a problem uh but then there's also good news with this kind of like quantization thing which is um a lot of the large language models llm or um the diffusion models stable diffusion actually the larger model you get typically they're more fuzzy about the numbers and typically you don't get as much as a performance degrade when you quantize so you probably have seen on Twitter people are like oh I got a you know 8-bit version of of um of the Facebook llm which is called Yama um on my computer right on my laptop to run and that's because like larger models typically and generative models typically are more fuzzy with quantization but the problem again is if we want something that is like you know classification based uh things that requires higher accuracy this is the typically the challenge Second Challenge is simply too expensive so as teach one also mentioned just now actually we all know that this is a problem so again I don't really have basis on these numbers uh it's just based on like my personal experience and intuition I think doing machine learning in the CK right now is around 1 000 times more expensive than doing it regularly okay so what does this 1000 times more expensive mean uh it means that let's say if we can run a model on a machine that usually have gigabytes of RAM in order to run its ZK counterpart is we now need terabytes of RAM so this is around 1 000 times more expensive right time wise uh is a whole different story so I'm not going to get into time because like time where you can probably have like Hardware acceleration that can help with it but the amount of ram of memory that it occupies uh is a huge problem um of course there was like a lot of solutions that we're trying to work with and that's calls for like research um as Lucy mentioned we can wait for Hardware acceleration right so if we have specific Hardware that does uh CK more efficiently probably we can have uh use those instead of like what we have right now and gain a lot of acceleration uh this is um probably a way to speed up server-side ckml um for client-side ckml we probably need something like a more efficient way to do recursive ckp okay this is something that I will get into a little bit later when we go into the future mentioned is it necessary to prove all layers of a neural network model why can't we just prove last three layers of a date model Ah that's true so it is true that actually a lot of people or at least initially we were trying to um just prove the last three layers of the deep learning model um again it depends on the use case for some of the use case that is okay but I can also easily give a use case that it's not right so for example if I want to prove an image I probably want to Hash that image but at the same time have that end-to-end process of putting an image into the model and have some result uh just put in the last three layers of the um in in ZK would cause the problem of basically either uh you can tamper with the data right and get up to like whatever you want um before the last few layers uh or you are also risk like exposing some of the details of your data if you choose to like make that process transparent right so there's trade-offs that you have to do if you're not doing end to end um but they're actually initially a lot of the practical application is to do that to just prove the last few layers um so again it depends on the use case of what you're really trying to do um for that specific machine learning use case answer your questions and Challenge three uh this is not typical that I I put on here but um I recently just realized that there is a really a huge need to for call for Action um mainly because a lot of people is interested in ckml but actually not a lot of people are actually are actively working on ckml so if I were to count there is literally around four to five projects such as actively working on ckml or at least ckml is their main kind of business objective um and if you think about it the analogy that I always make is that think about how how many people were in tensorflow how many people work on pie torch and compare that to the number of people working in ckml for a lot of the things at least for a lot of machine learning operation we do need a little bit of like customization to make it into a CK template right so if there's something that comes out new in the machine learning space tomorrow most likely we'll need as many people to convert them into ack uh version um let's uh right so I saw a question from Lucy uh other than Daniel and I who are working on I'll give you a few examples in the upcoming slides I will mention the names as well yeah so we need a lot of people to control these machine learning operation into CK template and there's always a lot of ideas you always see articles people are like oh you can use ckml to do this you can use ckml to do that so we really don't have enough hands on deck so we really need more people to join uh and that brings us to well what are the libraries or actually teams that you must know in this field right at least for the presence so a few libraries or teams that is working on ckml that you must know the first one is actually I just try out the python package today it's amazing so this is a library that is actually originally written in Rust uh called easy KL is a library and CLI that converts your models into a circuit directly into a Halo 2 circuit and it does uh it works on almost like any onnx file right so onnx is like the the file format for a graph so not just neural network basically any computational graph right so the way you use it is you have a computational graph that you have in Python you export it into an onx model a file uh and then you also have some signals that you put in the Json file and then you just point ckm Sorry easy KL into these files to generate a CK circuit uh most recently they have also released their python bindings so not only do you not need to install rust on your machine anymore uh you can also just directly do everything in Python so they have a I think a demo collab in a book that you can work on to see the whole process um so if they have um you know their team is great at making it very user friendly and very easy for machine learning Engineers to convert it into a CK to convert the model into a CK version yeah so I love their python bindings I was actually trying it out like right before this talk and it works perfectly well uh so this is uh so the company doing it as C conduit so they are you know one of the groups that is working on ckml Research the other um library that you must know as Lucy has mentioned is Daniel kangs uh Library um the name is very simple ckml and it's just a framework for constructing proofs for machine learning model execution um so uh it's really kind of a accompanying repository to their papers so it's best that you kind of read their papers alongside with the code base um the library actually does much more but I just want to highlight kind of two things that happens on their Library uh mainly for first of all they actually tackle model training which is a very interesting thing the way they tackle model training uh proof is actually have a backward graph right so when we talk about neural network you have uh forward propagation which is inference and typically um training is done with backward propagation right so back propagation um so the way you can proof training at least for simple models is to have a backward graph right so if you are able if your graph is somehow able to be reversed uh of course that doesn't mean any neural network can be right for example a gang right a generative adversarial Network cannot be simply break down into a reverse graph so but if you have models that can break down into reverse graph you can actually use the library to prove model training as well uh their Library also tackle very large model so they were able to include like gpt2 or vert or even some form of diffusion models onto ZK as well again when we say they are able to put it in ZK we are talking about terabytes of Ram uh uh so that that's more like server-side ckml but it's very cool that they were able to at least like have these very large models in CK and have it worked out and uh just a little bit of self-advertisement uh my libraries so I these are two open source libraries that I'm constantly looking for contributors um I primarily work on circum um for some reason that I will explain later so certain people have moved away from circum into like Halo 2 for more efficiency and also for um you know the ability to perform lookups but there's a reason why we still are interested in circum that I will explain later but these two libraries are first of all a circum circuits library that performs machine learning um and then also a tool in Python that transpile a tensorflow Keras model into a circumcircuit right so the problem I guess uh with circum one of the major problem with circum is that I would have to basically construct every single machine learning operations uh in circum uh as a template in order to cover more of like the the whole Machinery field of machine learning so right now actually I just my library only covers certain very common layers that you will see in a neural network but not you know all the layers that people are gonna use and the problem with that is that then if people are wanting to you know use it for um layers that are not existing in the library they will have to expand the library um which is something that you know I really hope that there will be contributors to to work on and try to you know expand this Library together right and as I mentioned uh the reason why we still use circum uh PSE uh there are some reason uh connected to kind of the research that we're doing so I want to kind of hop into what we do at PSE for ckml research for more kind of detailed document you can visit that website here I will link to the slides at the end so that you will have this for reference as well so there are three things that we are focusing at PSE um Team here so the first thing is actually benchmarking so we really want to understand like how quantization affects model accuracy um typically when we say CK benchmarking a lot of other protocols Focus only on you know prover time verify your time memory requirement which is all very important to CK uh protocols but um specific to ckml we really want to understand how you know quantization affects model accuracy or how we can actually avoid uh uh a performance degrade from that so this is something that you would really want to kind of like have a benchmark uh and we also have like some grantees working on it already the um kind of second area of research that we're doing is we're also looking into CK friendly ml so this is hugely inspired by the field of fully homomorphic encrypted ml um which they are already very into like studying what are the machine learning architectures that is friendly for fixed Point aromatic and that means not the neural networks that we know and there are a lot of other models out there that are not neural network uh that could have um comparable accuracy to neural network so something like a weakness sorry wait list neural network um which you can refer to this project that we're supporting um could be an option right so just use a bunch of lookup tables and a bunch of Bloom filters in order to achieve the same accuracy that you get from neural networks um you can also have purely Boolean circuits uh you can expect the circuits to be huge in order to achieve the same level of accuracy but then again the beauty of it is that it's purely integer right actually purely just zero and once you can also have like binarized neural network which is kind of the same idea and also truth tablenet so all of these like uh integer based machine learning architecture would be much more CK friendly and it's probably worthwhile to kind of explore all of them um one obvious question of course is like if they don't perform as well as neural network then what's the point right but actually for example for the first one the weightless neural network uh at its time in the 80s it's used for fingerprint recognition so it's actually something that people use in industry and production uh and has like enough or um sufficient accuracy uh to to be used in production so all these like CK friendly architecture could be something that's very worthwhile to uh to to kind of check out as well and uh also folding so what is folding folding is actually related to what I say about recursive ckp so the idea of folding is that you can take a bunch of proofs that looks kind of have similar proof structure right so for example uh if we do machine learning inference we might be applying the same model to a lot of data right so to different data so let's say I have 1000 images I want to apply the same model to all those 1000 image so the folding technique actually allows you to fold all those proof into one right recursively so that you only have a small overhead in addition to the original curve size so right now um there are two floating scheme Nova where it works on like the r1cs constraint system which is the constraint system that circum use so that's why we're still working with circum instead of Halo 2 but there is also sangria which work on like clunkish uh scheme so with Sangria being implemented in the future you can possibly also use floating on you know all the existing libraries like for example eckl or Daniel's uh ckml Library we can possibly also fold them and gain some performance uh uh acceleration as well um so yeah so we're also looking to fold you know all sorts of different things so one thing that I mentioned just now is just folding inferences but when you think about machine learning there's actually a lot of repetitive uh upper uh operations right like if we do convolution it's really doing the same operation over and over again convoluting on the same image so possibly we can fold that as well right so if we are able to do recursive uh acceleration in pretty much every single area in ckml then it's possible that we can gain a lot of acceleration from from this kind of recursive folding scheme as well right so these are the kind of the research directions uh to look into if you want to get into like ckml research and the most cutting-edge um stuff right so where to start right which is I think a lot of people coming into this talk will probably know on to know okay I've heard so much good things or interesting things exciting things about ckml where to start for this well if you have some machine learning background or know some machine learning Engineers that would be interested in CK interested in crypto I would suggest first of course play around with eckl convert some existing models into a CK counterpart right try to deploy the model on chain and play around with it and also start building or really thinking about applications so some applications could involve like algo trading or even doing biometric authentication on fingerprint so I will have a more concrete example of algo trading in a bit so I won't go into it yet but yeah so but you can think of like what kind of applications that you can build with models that might not be affected by the CK quantization so much if you have a secret background or want to get into it you can first start off with studying you know circum Halo 2 Nova and then help like machine learning Engineers actually turn their models into a CK version or contribute to open source libraries or you know like join our research initiatives and I'm really looking forward to you know whoever wants to deploy kind of the first aigc nft collection I'm very happy to have I actually already have some people telling me that they're interested but they don't know like how to get into it um their company might have a use case for that and I think it will be a very interesting and uh you know uh and finally a feasible practical project that people can see about ckml so that it's no longer just like an idea but something that people are tangible uh and able to see the application of and speaking of application uh just kind of two applications that would keep everyone excited and inspired and they are both by modulus Labs which is the fourth um I guess project that is working on ckml uh and so just now I mentioned Alco trading uh actually this there's a box that is already deployed by modulus Labs that is doing trading with AI right so you know in the in the essence that they have a model that works on investment right and they are able to uh prove that this decision of the investment or this action that this Bots perform on chain comes from this AI model right so you can think of this has a lot of use case uh of course this is more like a game where people are you know are just contributing and see what and donating to see how this spot will work out but you can also in Imagine uh you know a lot of investment down right instead of trusting some per people to to do the correct action you might as well have some kind of an algo trading but that is provable right so this provable ones will be able to submit a proof alongside with their investment actions so that you can have a verifiable and trustless way to perform all the investment uh actions you know from a dial perspective so so this could be like something that is very fun to do and also you can have you can gamify the whole process right so this is this project by modulus lab uh another fun one again is online so you can play around with it as well so you can pretty much like bid for a chance to play against the an AI on chess right so there's a AI that is known to uh that knows how to play chess and the idea is that where uh it's gonna submit its proof along with the proof that this actually comes from an Ami AI model right so what it proves is that you're not playing with someone just behind the machine right you're not playing against another human that is like pretending to be AI right so this is again a very kind of fun idea of how to make um you know ckml practical so with that I would say you know let's bring some what is the um possible ideas and I look forward to kind of like the the discussion after this talk as well right so that's the end of my presentation today um this is still QR code to my link tree along with like the first item is actually today's slide so you can click onto it um but I will probably go back and fix a little things I did realize there was some error in there but then the link should be um like real time updated as I'm fixing it cool uh well Kelly you're worried that uh you one hour is like uh something that's too long but I think uh one hour it's just nice because you covered through everything and I think everything detailed I I draw a talk I think the commit everyone does as well so thank you so much again for volunteering to uh to speak to the community and uh we are always uh very welcoming of of you and of course the PSC team so um so let's let's uh let's let's do a much casual chat uh let's uh five side chat uh we always start off our five-star chat with a bit of introduction and I'll get to know session um fun fact uh Caddy's actually my ZK teacher I'm gonna say it many times throughout is the this entire call um and uh back then she had she was one of the I think one of the lead instructors for this CKU thing and I participated and I learned basically learned how to run my first circum circuit from caddy so there you go uh for those who are wondering how I learned I learned from Kelly and I know enough about me but what about you Kelly like how do you get into the space um the ZK space the crypto space and how do you get into the ml zkml space you know um this is one of those stories that you will feel like you can you can only hear it from like web3 or crypto uh so basically I wasn't in the crypto space until end of 2021 so right after the crypto summer which is good right because like then I'm not in for the hype but um uh basically I was tasked with um creating the syllabus or the content for zero knowledge University so the program that you joined uh the idea is that um at that time I was working for harmony chain so um you know and uh alternative evm L1 and they wanted to onboard like developers uh into CK uh so and they thought that it would be good to find someone who is more of an educator and learn CK rather than have someone who is good at CK and doesn't know how to how to teach so then that's why I was hired I was basically I needed to learn the basics of CK and circum in I think two or three weeks uh before I taught you guys so that was kind of like how I joined um the space kind of yeah like your two or three weeks course is much harder than our one month a six weeks course no yeah I would say like it's it's very wonderful right now there are so many resources like back then if you search circum I think the only thing that pops up is the official documentation and it's not even that like complete so there's a lot of like trial and error but it was a fun process you definitely learn a lot more uh things like from just really trying it out cool and and now you have joined the PSE team and uh what are you involved in in the PSC because uh we know we know that PSC has many projects um and I know that you are in a you are more involved in the AML site like do are you specifically researching in any areas of the KML yeah so well actually all the all those research directions that I've mentioned in the slides just now is really just me uh fpsc right now so um I so right now I'm more tasked with like thinking about the high level directions um also of course I also do a bunch of coding trying to put out some proof of concept as well but um yeah but that's why we always want a lot more people to join because there were so many ideas that we want to try out but there's just not enough like people to do so cool and uh okay just just to remind everyone that uh feel free to also leave a question for Kelly um not every day we get to ask Kelly in person like like this today so we have one already uh carliwa is the biggest obstacle in your opinion for implementing the tooling for ztml I think to be honest like people might think it's the speed the proofer time but really I think it's you know from a business or commercial perspective it's probably more meaningful if someone has their first use case like for example with easy KL actually you can already convert maybe 90 of the models uh that people use nowadays into a CK counterpart right so the tooling is actually there but no one is using it or no one is interested in developing it further because there's no actual application yet so I think we need really meet someone who is like interested in building the first application so as I say you know be it like an aigc nft uh model or be it like a you know a fun game like what I showed you with modular slab just now like someone needs to have the first use case that people are excited about cool and uh alumini asks a very interesting question so uh so the computation model to run uh the commercial requirement to run gtml because DK itself is also a very uh computational exhaustive so uh so what are the requirements to run zkml right so uh as I've mentioned in the slide just now my rough estimate is around 1 000 times more expensive uh that's not even counting the time the time is is crazy but uh time can be improved by a lot of Hardware uh you know acceleration but the biggest problem is probably like memory requirements that we're facing right now so when I say 1000 times it's like a typical model right now an llm model we're talking about gigabytes of RAM right I can run it on my GPU that has 32 or 16 gigabytes of of uh RAM but um we will need like terabytes of ram in order to do the same thing in CK so that's uh much much more expensive actually cool and actually um some of us here were participating in the previous hackathon but we would be building some zkml stuff it was uh I think they call it Sephora or something so the idea is to protect uh users interactive contract uh using AI model and I think the team is was facing a problem whereby the model that was trained by the Onyx file couldn't be nicely converted by Ezekiel so that was the problem that had that I think that 10 Edge case that you were mentioning so with regards to the question um tooling is very important and easy care is not perfect and uh uh what are what are the and floating Point numbers is not supported or at least it has to be quantized and uh you give up accuracy so are all these things a factor in people adopting ckml like uh or do you think this will be yet improve over time or do you have a strategy in mind of how this can it gets improved over time oh that's that's a very good question yeah I think to be very honest like with a high performance model like that that they're trying to achieve most likely what's gonna be required is to really treat it like a AI project more than a a you know a uh what um I guess blockchain project in in the way that you know like to perfect a model in in machine learning space like if anyone come from the machine learning space or AI space they would realize is that like there's a lot of fine tuning that you have to do there's a lot of optimization that you have to do and I feel like for those critical models that you want to perform it's probably not good enough to use Easy KL right it's probably more casual use case for example if I'm doing like a AI generated nft sure right if we gain a little bit of Let's Lose a little bit of precision it's fine but if with very precise model I think there's a lot of like menu work that needs to be done and frankly we don't have enough machine learning experts in the field um that is interested in doing that uh and that is also one thing that I I'm trying to like think about like how do we attract machine learning Engineers into the into the space actually uh it's a good it's good at you I asked this question um because I think in the previous academic if so I've seen a few ckml multi-learning based uh projects popping up so I do expect to see them even more in Paris hopefully they will excite you and Maxwell asks um so the are there proof system designs specifically for linear algebra operations as opposed to algebraic circuits or r1cs that are more efficiently improving MLS I was about to ask a similar question but I will let Maxwell ask first that that's a very good question actually no not that I know of um not for linear algebra operations I know in like for example in in Hell 2 A lot of people are hacking the way not for the algebra not for the nearly algebra operations but for the non-linear operations like sigmoid or hyperbolic tension they are hacking the way by making everything a lockup table uh but they also have a trade-off which is like lookup tables could also be very expensive and it's not the best way if you build everything as a local table right if essentially you can have a model that is just a huge local table uh which is actually what the weightless uh neural network is but but yeah no not for linear algebra I think for linear algebra people are looking more on Hardware acceleration cool and a follow-up question by Maxwell as well so when you say that you need like around 100 a thousand X larger Ram referring to some highly optimized version like folded uh pcds ivcs or or other naive implementations I haven't really looked into Daniel's code base uh but I would say uh like I want to say it's a naive implementation but I definitely not a highly optimized one like I don't think they use IPC or folding uh in the kind of the version that I made for gpt2 and that's the 1000 times I'm referring to uh but yeah I'm not a highly optimized version for sure so another question by me uh so stock map has been pushing Cairo and they claim Cairo to be a very easy to use language as compared to writing raw circuits in RCS on plantage Automotive so have you explored Cairo like do you think Cairo is a suitable language to write zkml as they claim they claim that it's a easier way to write zkmls and probably a more efficient one what do you think I have not explored Cairo One I must be honest the new Cairo I've not explored so I cannot say much for it but I am I do realize that they said there is an easy conversion from circum uh into like a chiral script or something like that like that's what I heard from someone so I think it could be it could potentially be very very interesting um yeah but I I mean I I look I know too little about Cairo to just say anything um but at the same time I don't know if like it's uh if I write in Cairo does it mean that it also goes on chain like as a part of the smart contract or how does that work actually caution back to you so yeah uh so I would take this opportunity to talk about Cairo a bit so Cairo it's a language a general programming language and uh you get you use Cairo as a subset of all colors functionality deploy contract so what you can do with Cairo is that you can write a general program so the the Cairo approver is written so the the sharp Rover by Stockman is actually written in Cairo uh the ZK evm Kaka Rod uh that is recently being invested by vitalik and stuff like that also written in Cairo so Cairo is a language that allows you to generate proofs yeah so that's the string of Cairo so I don't know that sounds perfect actually yeah no I'm really looking into uh kind of exploring that for sure yeah I was about to say that uh I'm definitely thinking of like writing using Cairo to explore some of the zigml ml uh world so yeah so again we will see how it happens all the community can take part so when can ask um their projects out there that does general purpose of verified Computing like Risk zero which allows application running on top to be probable what would be the differences between making ml models runnable on ZK VMS and using a current zkml libraries like easy easy KL so yeah no that's a very good question I think uh I wouldn't say for sure but I believe right now at least having specific purpose uh circuit is still way more efficient and running on ckvms uh however there is a good thing about running um on ckvm is that I know risk zero is looking into something called a functional commitment so one thing that I didn't mention in my talk just now is that like I mentioned that model weights can be private but the actually we're not hiding the model so the model architecture is actually public and for a lot of use cases that actually defeats the purpose of model weights being private already because you can actually retrain the model like what's once you know what model people are using and what data people are using you can retrain it yourself uh for the late models like open AI is one even if you know the architecture is not enough but uh yeah so with reservo and with like their implementation of ZK VM essentially what you can gain is something called a functional commitment which is you get a hash of the circuit that you're running so you can hide what the program you're running as well it can just like provide a hash and say that hey this is the program the hash of the program that I'm running and you can trust that I'm running the same program over and over again so that's a good thing about uh risk 0 and ckvm uh but I think like probably specific purpose circuit is still more efficient cool but it much harder the right though because I could I imagine way much harder than using a ZK VM that's true but even for ckvm you still need to have an understanding of ZK like what kind of signals you need to hide and you're still kind of restricted to your fixed point every metric for the most part for now so interestingly I do have another question regarding fixed Point aromatic which is something that uh the team face during the hackathon process they realized that um easy care doesn't support fixed by emeritif uh any tricks or or like or like any tricks or like any hack that allows like you know like can easily Bypass or oh yeah the fixed Point aromatic limitations of uh of ZK yeah that that is an interesting problem they encounter because I believe easy care does have a like heuristic to help you convert that I don't know maybe that heuristic is not good enough for the range of numbers that they have in the model perhaps but in a way that like eckl should handle the quantization for you uh which is like basically you have a bunch of floating Point numbers right just break it into I don't know a bunch of steps and having like a a quantization a scaling for it basically right so there's a quite quite a lot of ways of doing it uh in my libraries I tend to do it a very naive way just to make it like more compatible with a lot of things which is um you know for example if you have a number of zero one two three four five six blah blah blah just turn it into one two three point four five uh sorry one two three or if you want more Precision turn it into one two three four right just just scale it up by the power of 10. um yeah but of course then the problem is you're running to overflow with fixed Point arithmetic if you're scaling up all the numbers it's a hard problem to solve yes so uh when Kang is eager to know that if there are any plans of make of making pie torch to succumb library or something there is actually a project that does that I will put it I think it's ckml Bushi coma uh I'm not sure if they're still maintaining it I think they are uh well actually they it looks like they're not active but they are pushing a release soon um which is a circum transpower for uh Network for onnx so Pi torch yeah I think not only are they making a big release they are also making a training model training version of it so yeah that should be uh updates coming up I'm not sure when they'll probably in the next few months or so yep yep uh right cool uh and one thing that for my for myself so I you mentioned a lot that um the PSC does a lot of research work at least on the ml side using r1cs and um interestingly hello to a PSC as a hello to version uh sorry Palo 2 has a PSE version and I think I assume down the line there could be some conversions from I don't know RCS to plan cash automatic so how much of a rework or of do you think that needs to be done or or everything can be nicely translated to a flankish aeromative uh circuit that's a good question um I mean right now like for the few projects that I've mentioned pretty much anything except what we're doing right now is in Halo 2. so those are already kind of like natively inhaled to but if you're talking about Sir come to Halo 2 I I don't want to speculate but I I think it will be a very lengthy process um uh most likely what will happen is that there will be look up um uh capability added to the circle instead so the other way around uh which is a thing that a lot of people are talking about right like circum is okay uh if we are able to look up at some point um which is kind of changing the scheme really so but um yeah so I'm not sure how that's gonna work but people have been talking about it for sure but it just needs to come support as well right like and one of the one of the key string of plong is lookup tables so they I don't know I've been using second for a while so at least the newest version like they don't they or don't they support lookups pop long they'll be interesting to see that they are not supporting lookups of flaw yeah I think it's a trick on how to do it on a DSL level or do you go in and add after the conversion from r1cs to punk right because I think it still goes from like uh no actually it goes from directly to constrain yeah yeah I don't know how that's gonna work I think it's still like how do it really integrate it at a DSL level um will be difficult I don't know we have to ask Jordy one day yeah [Music] yes [Music] it uh Lucy has a question I saw eight five two chat with scroll Vincent recently any direct talks with l2s to implement zkml or current challenges of causing l2cml implementation away a ways away um well service yeah I think to be honest though this the same thing that I mentioned just now should apply to anything that is evm uh equivalent I would say um evm compatible so if we want to um and this I think this is still relevant even for L2 like because it's not just a gas problem right it's a size problem it's a computation compatibility problem like solidity isn't just isn't really the sign for machine learning so I think it like what I said just now pretty much is like as relevant to L1 or you know any any blockchains really um but yeah no I don't think there's any like particular plans for l22 or L1 to like Implement ckml at least not that I know of yeah speaking of l1's um when I realized that uh ZK Ezekiel is using hello tool as uh the Brewing scheme and we know that the verifier the the verification time for hello to its I think login or something and it and it needs to support I don't know if um no I don't I don't think Ezekiel supports the recursion of hello too so I reckon that if the proof side the proof gets very very big it might even exceed the gas limit of uh of uh of of ethereum uh is that true I mean this is my speculation yeah I think the first thing we encounter is probably it exceeds the contrast size to them at first uh before we even talk about gas limit uh yeah I don't I have not seen cases that we um except the guest limit just yet I think like even with Halo 2 actually recently one of our PSE Grant here is working on you know converting the uh the weightless neural network into Halo 2 uh which was originally working in other DSL and he was able to do some optimization actually to tackle that problem as well so even within how you construct a local table itself uh there's a lot of like optimization that you can do um to really make it fit the size but so far I think if you are able to fit into the contract size the guest limit is fine so far okay uh right cool uh I ran out of questions and uh any last call by the oh yeah actually I got one more so um a very fun question so um because in the EF basically you are like only like only one working is a gamer are you hiring are you looking for assistant maybe this is a good call for uh for for assistance yeah no so we are I'm I don't think we are hiring for research roles specifically but we are constantly looking for projects to support so a lot of people who are doing ckml research actually work with us as grantees I before I work full time at PSE I was actually a grantee as well so the cko project that you saw it was actually my grant with uh PSE so if you are interested to in working on any research direction that I mentioned just now or if you have another cool idea about ckml uh you know just tell me all about it we can discuss and see whether it fits like PSE Direction and when you can make it a grand uh that that can support you for at least a few months to work on this like full time nice nice so uh for everyone out there or listening on YouTube uh Alex that is uh that that wants to um explore the camera so hit carry up and uh and you may get a grunt uh for for your foreign cool uh random ideas Lucy has uh Lucy's thanking you uh for coming for being uh for for your time and for the talk and I think that's all for our questions and I want to end this uh call by saying very thank you to caddy a nice one and a half hours of q a and a very informative session uh hope to see you again in my sub in Paris and hope hope and we are always welcome we always welcome you to visit Malaysia as a speaker or even if you come for the nouns run uh you'll want to welcome so and and if kale in in October right yes yes if here in October yes um we actually I actually contacted Adrian aside topic we are actually contracted Adrian for like a sponsorship but they like bandwidth to for PSE to support but but you can come as a speaker sure yeah let me know let me know and that's all uh I'm gonna stop recording uh but that obviously ends the