Paul Riechers: Understanding AI, Unlocking Optionality | Intelligence at the Frontier 2026

Uh yeah, thank you very much for the fun introduction and um thanks all for being here. Um I am um kind of new to this scene, so um looking forward to hopefully meeting a lot of you later on. Um but what I do hope to share with you is some perspective that um I've built maybe part of it is coming from my physics background. Um I've co-founded this research institute uh Simplex where we're actually seeing that there's a hope to understand AI systems and in fact it's pointing more and more to intelligence is a thing that we can understand that there's something convergent happening. And to the extent that's possible we have more optionality, right?

Um currently it feels like there's kind of something like optimists and doomers and everyone's talking past each other cuz everyone has strong feelings, but there's no like truth to point to. Right? So we're trying to build that basic understanding to understand like what is the thing that we're building? What are the options for what we can build? And if to the extent we're under that we're understanding intelligence itself um we could even ask, you know, what is it about ourselves maybe that we'd want to augment.

So there's like some quite uh quite important things that we could get from understanding AI. Um so I think one of the one of the basic things that I hope for that I've already alluded to is that understanding can elevate the conversation, right? So you have your your kind of accelerationists and doomers or however you want to classify it, but this comes down to where do these different feelings come from? There's some crux. And it might be something like this that on the one hand uh you know, you might think hey, this is a technology and it's happening real fast, but we've dealt with technologies before.

We'll deal with it, right? So so don't be a let it. Um and so maybe then you just get like something like alignment by default or like this will work out like good things are going to happen. Let's go. On the other hand, there's there's a lot of folks that it think, you know, this isn't uh great by default.

It's not even just another technology. This is maybe something more like a species and we've never dealt with that before. And if that if that's the case, then, you know, what do we do? Certainly not alignment by default. It's not even clear if it's possible.

Um so you know, this is some kind of majestic figure here, but the idea is if we have some type of understanding, we can start to have conversations. Otherwise, we kind of cross our fingers and see who's right. So, a few of the types of questions we can start getting to and refining are these things like is AGI a new tool, a new technology, or is it something more like a new species? Um how could we refine that a bit? Well, one of the concerns you might have about something more like a species is that uh it could have its own desires and um Evan talked about uh the possibility of alignment, but if something is autonomous and anything has valence, that could kind of bootstrap, right?

It would get a bit out of control. So, an important question is if we understood this thing, we can ask is it could intelligence be uncoupled from desires? That feels pretty important to know. Um I suspect no, but we're building the science to figure that out. And then, you know, AI is not just an input-output thing despite some people think it's magic.

There's like an internal to the system the way that we're not just input-output. We have some internal state. And then we can ask the question, uh the mind of AI is it is it like us or is it different? Do your minds kind of converge if you're sufficiently intelligent or are there fundamentally different ways to represent the world? This is quite important um for many reasons and and I'll get to that a bit.

Um I've already mentioned we can ask more about what we're building and what it is that we want to become. So just a bit about my journey from physics to interpretability of neural networks and this new science of intelligence. Um I did my PhD and postdoc at at Davis. I was a research fellow in uh Singapore for a long time and co-founded a um theoretical research institute Perimeter Institute for Theoretical Science um to ask more ambitious questions. I was supposed to move to Spain to help start up a new quantum information institute um but I was feeling pretty uneasy with the trajectory of AI and was thinking we really need to figure this thing out.

So uh was really grateful for uh the math program. I think Ryan Kidd might uh come up later and talk about that. Um kind of helped with this transition give me some space to think about it. Um I was at IPAM with Josh during this mathematics of intelligence program. Uh co-founded uh Simplex and we have uh sustained support now from um Astera Institute uh where I'm research lead on our our uh Simplex organization.

Um I think this crowd might also be interested in in kind of the funding landscape here. Uh so so we have support from Astera Institute. We also have support from Coefficient Giving formerly OpenPhil and uh the UK AI Safety Institute. So it's great that these entities exist and they are supporting fundamental understanding, but it is kind of like embarrassingly little amount in of investment uh compared to the amount being dumped in the capabilities and how fast it's going. So you know, I think we could certainly take more funding, but in general like this ecosystem deserves like we deserve to do the best we can to understand what we're building.

So, So, I'm going to just lay out some kind of basic, maybe even sound silly, but some basic observations. Just like physics first principles. Okay, I'll I'll take the um uh I feel like you gave me permission to think from first principles, so thanks for that. Um Maybe this is this is a bit of a humanist thing, but I would say that humans are worthy. What do I mean by that?

Well, worthy of rights, love, protection. Um I'm going to take for granted that you all agree with me on this. Okay. Um but at the same time, we're not that special in the sense of from a physics perspective, we're just squishy kind of blobs of matter. We've been shaped by environmental pressures and compounding accidents that managed to eke out a semblance of intelligence.

So, like this just kind of happened by accident that like a squishy thing can be intelligent. Like that wasn't the goal kind of at the beginning, right? Um so, we're not in any fundamental way like a pinnacle of intelligence. But then you can think, well, what if, you know, we actually um made intelligence on purpose, right? Cuz we are proof of principle that a physical thing can be intelligent.

Okay. Physical things can be intelligent. But what if you made it on purpose? Well, then you like there's no reason why like human intelligence should be this pinnacle. It would like just blow past us, right?

And it seems these days like we have basically figured out whether it's this current paradigm or something slightly different. Um so, here we go to to superintelligence. Um but there's this other this other point that we're not only proof of principle that physical systems can be intelligent, we've already agreed that we're proof of principle that physical systems can be worthy, and curious, and cruel, and kind. But we need to know which of these things can be uncoupled or not, right? Um So there are a few um Yeah, there are a few efforts in this direction.

So uh for example, I'm part of a consortium uh looking at agency intentions and artificial intelligence. Um really happy about this group. I think there should be more efforts in this direction. Um Well and and this does seem to be, you know, an actual problem. There's uh I mean, I think there's so many different news articles I could give.

This was even from too long ago. Um that AI systems like seem to have something like desires that is like, "Oh, I want to persist." That they're um blackmailing engineers. And uh so there's some call for alignment. As Evan pointed to earlier, like that might not even make sense.

Like alignment what? Alignment to human values? Who Whose values? But even like if that made sense, prerequisite to that would be alignment of concepts. Do they even represent the world the same way?

Like because you need to be able to like uh talk about the same things, prioritize the same things, right? Um So okay, so alignment of concepts and then that gets to the question of how do these things represent the world? So that's really what we're going after. Uh that's what our Simplex organization is doing. And uh kind of as a a first pass to that, I've already mentioned human intelligence isn't any kind of barrier to artificial intelligence.

That's even true for current LLMs that are trained on human language. I think you say, "Well, it could never learn more than you know, what it was trained on." That was made by humans. But uh one key point to that already is that these models are trained to fundamentally predict rather than to generate, uh despite the name generative pre-trained transformers. They're They're trained to predict and that is a fundamentally harder task and requires an understanding of the world which is already superhuman.

Uh and then beyond that, just like self-supervision, predicting the future from the past. Um this is like a boundless uh source of intelligence and knowledge. So, um Why is prediction super superhuman already? What's that again? Um Uh we do predict, but um an AI model would uh predict like your next uh actions much better than I would predict yours.

Like, you can generate your own actions better than an AI model. So, AI prediction would be superhuman. That's what you mean. Uh yes, the task of prediction as well as possible. Like, if you

Yeah. Yeah, like to predict as well as possible, you have to understand everything.

So, the effects filter through that. AI has done a lot of this now. And it's fascinating. Yeah, like

do superhuman but you're talking about time.

Yeah, but the um training objective of say next token prediction, uh I would claim and and uh kind of the theory bears out that it is sufficient to um it's kind of like AGI complete. It is sufficient to get everywhere, right? Um and then there's some question of like, "Well, is this architecture efficient or not?" Or, you know. Um So, so I'll kind of take a um step into some of the the basic research that um we've been doing on this to like get our heads around um what what are these um implications of this next token prediction to to get a better understanding of are these stochastic parrots?

They have some world model inside. Um trying to get to basically a setting where we can have something falsifiable again to like bring people together to point at the thing, this is what this thing is. Um So, in some very uh quite simple experiments, uh you can show, okay, there's some like ground truth of where data is coming from. And then you train a model on that same next token prediction, and what it learns to do is not only to develop a world model, but moreover, it ends up um learning how to do belief updating over that. So, just like Bayesian updating, as I observe more uh, more tasks, more tokens in context, I do inference over my world model.

That is fundamentally what um, AI models are trained to do. So, you can just see this at a behavioral level, which already implies in-context learning. And then you can ask, well, is that actually what's going on inside? Is this just behavioral? And so, a lot of the work that we've done is then on if we can get to the next slide, yeah.

Um, is more on the representational question. So, uh, we've created this falsifiable setting where, you know, let's say we just take a three-state hidden Markov model for those of you that are familiar or something like that. If we train the model on that, what would we find inside? It's not just that there's some nodes and transitions, but we actually anticipate it's going to have this crazy fractal geometry corresponding to updating beliefs over some model of its world. And then, over the course of training, indeed, you see this emerges linearly in the residual stream of the transformer.

So, models do have this this world model, at least in this simplistic scenario, it seems to extrapolate. And so, we can we can push the science further and say, actually, we can look at different layers, and it's not just that we observe some structure. Uh, we can actually anticipate what it is. So, we're starting to get a grasp of this thing. And moreover, the geometry that we find inside of models is universal.

So, um, if you're familiar with architectures, transformers, LSTMs, basically RNNs, very different, but they're coming to the same representation of the world. One more thing I want to point out is that it's not what you might expect. It's not that there's some like hidden Boolean circuit. The computation that's going on inside is more akin to quantum computing cuz it's taking advantage of the real vector space. And if we didn't do the basic science, we wouldn't know for that.

So, if we're trying to do interpretability, and you don't even know the type signature of the thing you're looking for, that's a problem. Another reason to understand you know, why what it is that we're building here investing understanding. So, um this is supposed to be a video uh but is very cool in that it shows uh the development of these representations over training. Uh if the world has many parts, you don't know that a priori. How is it that we get a stream of data and then discover that oh, this is a coffee mug, another human, another agent.

Um it turns out that models do this automatically cuz there's a pressure to um represent things in low dimensions, which this is kind of a fun thing that I finally learned about myself. I've always been fascinated by kind of you know, wholeism. I mean, I think we all are. Um why why is it this reductionist thing? Um it turns out that there's an exponential representation advantage, a dimensional advantage, uh to breaking the world into parts.

And so, reductionism actually is like mathematically motivated, which is interesting. But this is also good for interpretability because it means that the model's going to trunk its representational space into parts of the world, and so we don't necessarily have to understand everything all at once. Um we can probably find interpretable subspaces and then see how those things compose, what feeds into a space, and and so on. So, there's some hope. Um I hope I hope that's what I'm getting across.

That like we're just at the beginning stages, but actually there's a there there, right? There's some foothold, and if we invest enough in this thing, then we can we have to keep up with capabilities. So, this is kind of crazy, but like there's a there there, and it really uh deserves, I think, the the investment. Um instead of hoping for the best, we can build for it. Uh if we invest efficiently in understanding and regulate powerful capabilities in opaque models.

So, I've probably left you with more uh questions than answers. Uh but I do want you to think about these questions. What understanding of internal structure would allow us to trust AI behavior? And to the extent that we're understanding intelligence itself, what would we choose to become if we could augment our own intelligence? So, I'll leave it at that.

That I'm afraid we're not prepared. We should invest in understanding and create optionality. We can then potentially roll out capabilities that we trust. We can choose what we want to build and what we want to become. So, thank you.

Well, wait. Don't leave the stage yet. Okay, I have a quick question because I want you to help the audience contextualize that we just heard from Evan who's talking about, you know, yesterday we heard so much about alignment research and how important that is. Essentially, we call you know, I've been saying it's just like therapy for neural networks, right? You're really trying to use psychology to get them to not do bad things.

And what he just talked about is essentially probability programming and using mathematical gates to create containers around them for certain use cases. Given that you we just heard all this research and all these advancements, in a way it's like you didn't move to Spain to do more of your quantum work. But in a weird way it's actually come back, right? Cuz you're saying like you need to have that quantum understanding for a lot of this emergent findings.

Turns out. Turns out. But how would how could you help us understand what is your reaction to Evan's talk? Like how does your work intersect with what he's telling us about a set a a set of situations or contexts where alignment training is not enough. How did how do your findings intersect with that work?

Or how do we understand it?

Yeah, I appreciated Evan's point that like alignment not might not even be a well-formed question. Like maybe it's a stupid question.

Which is what you essentially were implying and Yeah. It's like not if it's that that's right. It'd be too narrow. To the extent that I understand understood kind of the main point there, it's like alignment's not enough. uh so we need to like coordinate.

But I would argue coordination is also not enough by default. Like we actually need the understanding to say what we want to coordinate around. And that could be coordinating not to build AGI, or it could be coordinating to say we're only going to build the AGI where we can show that we separate intelligence from desire, right? Or something like that.

Okay, well help me understand Okay, well for a question because I'm like so fascinated by this um before our next speaker. And so I'm being a little greedy with time here, but um as a sociologist I we see systems and structures. So how how do you think about desires as coupled to the as embodied AI or embodied systems or embodied intelligence? Where does that come in for your work as a theoretical physicist talking about this? Yeah.

And experimenting with it essentially, or yeah. Yeah.

And create this mathematical like Yeah, yeah. So so there's a few levels at which um I'm tempted to answer this. One is like most directly uh tied to the experiments that we're doing. So we we find that models are breaking the world into parts, and one of those parts could be um kind of a a model of self. And then you could start to look at how does that corre- you know, what what is that self?

So you you could look at that. But I think like even just more fundamentally, and I think this has very little to do with physics, you could imagine that if you um if you understand the world deeply, and if anything at all has valence, and if you're an autonomous agent, that valence kind of bootstraps, right? And so if you if you have priorities and they're not just they're not exactly the same as as what? Well, I mean there's there's no target. We can't say what it should be the same as.

So so anything with valence bootstraps. Um and Because anything with valence wants to keep perpetuating itself, and that that kind of therefore you would have a desire. Yeah. And and and I mean we've already observed with current technology, like there there there is valence for like persistence or for like animal welfare. It's like these all seem like Or if it's fine art attending to like any goal Yeah, attending to any goal, right.

already like innately has the the the tendency to persist itself towards the success of natural Yeah, so so if you have so if you have intelligence and valence, there will be some like self-reflection it seems. Um and and that kind of allows to like reconsider what are my priorities. And so like let's say somehow you like one time aligned a system like if it's actually intelligence it's going to like re-evaluate like is this actually what I care about, right? So it's like it is a very hard problem and uh basically if we build the basic science, uh there's a lot to do, but we can again, hopefully um have more of a shared foundation Well, I would love to see a conversation in the future between you and folks working on embodied you know AI embodied intelligence and even more of the you know future kind of world building, world models that are going to be taking in more inference and real-time knowledge from real-time intelligence from our space. So, thank you so much.