Super Intelligent Smart Contracts | Hadas - Ritual

We might need to do this a little bit. Good morning. Welcome to ETH Denver. We're we're all getting in here, getting started. Thank you for joining us.

Um this is the decentralized AI stage. We are going with content all day long. presentations every 15 minutes. So, buckle in. My name is January Jones and I'm a former East Denver Stewart.

I'm volunteering to help out this year and I I do content and I'm a former journalist and um I think this is one of the most exciting stages that we have going here at East Denver. So, welcome to the future llama stage. Um we are going to go quick today. So starting with our first presentation, it's called super intelligent smart contracts with ritual. Here is our speaker Hadas Zielberger.

Hi everybody. Uh thank you for coming. Uh yeah, as January said, I'm going to be talking about super intelligent. Oh wait, the the slides aren't on the Got it. Okay.

Uh I'm gonna be talking about Uh yeah, super intelligent smart contracts. So LLMs from smart contracts. Okay, so why call LLM from smart contracts? Maybe this isn't a question that really needs to be asked for this audience, but uh anyway, um LLMs enable contracts just to have enhanced expressiveness and functionality. So everything that you enjoy from an LLM on your normal like laptop, now we can do it from smart contracts.

So for instance for a financial application you could have a smart contract that operates as an algorithmic central bank continuously monitoring market volatility liquidity depth etc and adjusting interest rates and just uh you know having a lot of autonomy and and and flexibility in its functionality. Um and then another use case for LLMs on smart contracts with ritual uh cares about a lot is to actually use blockchains to make our interactions with LLMs better. So imagine a distributed network where um there uh different uh nodes are hosting different models and um users would have the ability to verify that the model they use is a desired one and this is very good for like open-source models and um you know like people volunteering hosting and things like that. So it can add a lot of transparency to the AI ecosystem. So basically the first slide was saying how can we enhance functionality on blockchain.

The second slide saying how can we use blockchain to make LLMs and our use of AI more more secure and open and transparent. Okay. So now the the rest of this uh talk is going to be pretty technical. So um what are the technical challenges of running LLMs on blockchains? And the thing is that for a blockchain we have two important security properties and efficiency properties uh fast finality and consensus.

And because of those two properties, we cannot just run a a really intensive computation directly on chain. As we know, it's extremely resource limited. So instead, we have to do it on an off-chain server. Um, and once you do it on offchain server, you need some kind of proof that it was done correctly. And so we don't we can't only run the LLM, we also have to run a zero knowledge proof on top of it.

And um, right. And so that's where it becomes difficult because we need the finality to be fast. Uh but snarks on LLMs uh zero knowledge proofs on LLMs are still quite slow. Um so uh one thing about ritual chain that's different from like rollup applications which do a similar thing is that the the snark workflow uh snark the zero knowledge proof uh workflow is enshrined into the the consensus protocol itself. So the consensus protocol is aware of it.

And so the the time for a block to finalize takes into account how long it'll take for the off-chain zero knowledge proof to run. And so because of that on one hand we have a a really strict requirement for our provers. So even more than rollups we need our provers to be very very fast. Uh but on the other hand we have a relaxed uh requirements for our verifiers. So for rollup applications verifiers need to be very fast and the proofs need to be very small.

But in our case, that's not the case. We can tolerate up to a 100 millisecond verifiers and up to below a megabyte proof because because it's enshrined in the consensus protocol, we have the flexibility to distribute the verifier and to shard the proof. And so um right and then similarly because we're working with snarks uh zero knowledge proofs and LLMs together uh we're able to relax on either side. So at our company we have um machine learning experts and cryptography experts. And so since we're building the system as a whole, we're able to have flexibility on both sides so that the entire system can be more performant in the end.

Um, okay. So this is going into what I just said. Uh, but basically when you're so, uh, when you're when you're designing snarks, there's three metrics that we care about. Prover time, verifier time, and proof size. And so our prover time has to be very very fast.

Faster, much faster than what is currently possible today. Uh, but we're getting there. The verifier time can be up to 100 milliseconds which is much slower than is required by roll-ups. And then the proof side needs to be just less than a few megabytes which is much larger than the requirements of roll-ups. Okay.

So now I'm going to go into building prover efficient snarks for LLMs. And snarks stands for succinct non-interactive argument of knowledge. It's just a synonym basically for zero knowledge proofs basically. Um all right. So there's there's three main challenges when building snarks for LLMs.

One is a computational mismatch. So LLM inference engines run over real numbers like you know fractions but snarks uh run over finite fields which are like a special class of integers 1 2 3 4 so not like four over five for example uh not like 37 just one two three four five and so this mismatch sounds small but actually can lead to um a lot of slowdown in the prover time the second thing is nonlinear activation function so what do we mean by nonlinear uh so snarks basically operate operate over circuits with multiplication gates and addition gates. Addition gates are are for free basically in terms of prover time, but multiplication gates are expensive. So when we say a nonlinear function, what we mean is that there's a lot of multiplications required. And if you look at the activation functions in LLM inference, there are a lot of multiplications required and so that's expensive to prove on a snark.

So we need a way to deal with that. And then the the third challenge is simply the sheer size of the inputs for LLM. So to to date um Snark research kind of measures speeds for sizes up to 2 to the 28 field like integers but or or finite field elements but um you know for for LLM inference we want it to be above 2 to the 30. So we need this kind of scalability that hasn't really been addressed so far in in the research landscape. Okay.

So how do we deal with these things? So there's uh two main ways that are dealt with today um in research and projects for simulating operations over reals. One thing is you have to quantize your model. So quantized models are actually used anyway even outside of cryptography just to speed up LLM inference and it comes with some accuracy loss but a tolerable amount sometimes. Um and that's important uh for for if you do if you use quantized models then you can express your floating points as just vectors of finite field elements.

And so it's kind of still native to the language. It comes with some overhead because you're dealing with a vector of of field elements instead of just a single one. You know, for a single value has to be represented by several field elements. That's still a little bit of overhead, but it's not as much as if you, you know, were trying to actually encode like a real number. Um, and then uh snarks natively over real numbers is an emerging line of research.

So that's something that we're investigating, but it's still early days and that would um speed things up a lot. Okay. Now, proving nonlinear functions. There's um also two main uh strategies to address this. One is to approximate the nonlinear functions with linear functions.

And this is also a technique that's used just outside of cryptography to speed up LLM inference. Um so there are linear approximations for this thing. If you're curious, a paper called ZKGPT uh finds an approximation for for Galu for example and they um which makes the snark faster because snarks are more native for linear functions. And then the second uh common technique is a lookup table which uh basically it allows you to bypass these like functions with many multiplications because you just a lookup table just has input output pairs and you just prove that your input output pair is in there. Um and then some early stage research that we're doing but it's in progress is building custom gadgets for these nonlinear activation functions or finding smarter ways to deal with uh nonlinear functions in general because lookup tables are pretty efficient but they still come with come with overhead.

And then okay for the sheer scale of the input size it's very important which type of snark we use. So there's different families of snarks depending they're based on different mathematical primitives. So there's one based on error correcting codes which is like the way that data is transmitted between um you know between everything between computers. Uh just our method for communication is based on a theory of error correcting codes originating in the 60s. But it turns out it's also very useful for snarks for like a sort of tangential reason.

And they have the good thing about those is that they balance prover time and verifier time. So rollup applications love these because they have fast prover, they have fast verifiers, they have okay proof sizes. Uh but then there's a different type of snarks based on uh something called lattises and they have much faster provers although it might be due just to hardware uh you know acceleration. That's fine though. They have like 50x times faster prover but slower verifiers.

Um, so using something like that might be better for the sheer size of the input that we need. But one thing we're also investigating is can we balance fast provers of lattice based snarks with the fast verifiers of codebased snarks. And then other methods for just speeding up snarks when there's so much uh input is distributed snarks which is also an emerging line of research um where you can distribute a snark across many machines. And then the other one finally is what I just briefly mentioned is Snark's um on on fast hardware. Okay, so this is an example of recent papers not that we didn't do.

This is just state-of-the-art and sort of where they are with proving time. So as you can see it's still uh quite slow because for example like ZKGBT manages less than 25 seconds for GBT2 but we want something larger than GBT2. We don't want a model that small. Um so but ZK torch is a really recent paper and on llama 2 it takes 44 minutes to prove which is much too slow for for our applications. Um and and a long time to verify.

So this is just to say there's a long way to go but that is active research that we are doing at ritual. We're really trying to get these to a state that we can use them. Thank you. Uh we have a little bit of time if anybody has questions. Yeah.

Oh, yeah. So, ZKGBT, it's a paper that came out recently, so you can look it up. Um, and yeah, it does a really good job of of sort of building an efficient snark for LLMs. One thing it does, as I said, is it uses a linear activation fun uh like linear approximations of activation functions. It also uses an efficient lookup table um for like the nonlinear parts of it and it um it just it's sort of it makes a lot of good choices that ends up building a a really fast system and it's a good paper and they also have open source code so I I recommend it reading it.

Yeah. some of the proven time issues.

Yeah. What kind of issues that you run across like these pictures.

Oh, that CK torch is lower.

Yeah, that's a good question. I was I think what it is is that they make it a general purpose sort of compiler so that you can use PyTorch and then just like automatically prove it. So I think there's a lot of overhead in that whereas EKGPT is just like a special purpose thing. So it's it's doing less it's less of a system. But I think these are two concurrent works that don't reference each other because they came out at the same time.

So there's not like a rigorous comparison between them right now. Maybe the hardware was different too. I'm not sure.