Frontier AI to Decode Animal Language – San Francisco 2025

[Music] First, just great and grateful to be here. Thanks to Wendy and everyone for setting this up. Um, today I wanted to talk about Earth Species Project, uh, which is using Frontier AI to decode animal communication. Um, we'll get a little bit into what that means. Um, but first, let's start with a a guessing game.

What animal makes this sound? No, whale's a good guess. Not a penguin. Narwhal is getting closer, but also not a whale. It's Yes, it's it it is a bearded seal.

Um, pretty much if I ended my talk here, it would be perfect. But uh it turns out that this is the mating call of a bearded seal. So if you guys got excited, now you know where to go. Um but the reason why I like starting here is because when we imagine what sounds animals make in the communication, we don't really know. We just think like cow, maybe humpback whale, but there is a huge range of what sounds animals make that we've never heard.

Can you guys guess what animal made that sound? Not a frog. Yes, it was a bird is is one of these. It's just a standard Kerrion crow. And what it turns out in the last couple of years is around, there's no really good uh estimate of this, but around 80% of the sounds that animals make are quiet calls that they just make very close to each other that science has never looked at because we haven't been able to record it.

And then when we have, it's too complex. And so this is where your ears just start perking up because you're like, "Oh, the super majority of communication that animals make, science has never got a chance to look at." That's very exciting. Um, and it would be sort of like trying to figure out how human beings uh communicate just by listening to our shouts, which would be awkward and maybe a little bit like Twitter. All right, this is the most humble graph that I know.

Why humble? Because it only includes all of time and all of space on it. But you can see and I I I drew this graph not to scale because otherwise the human experience would be tiny there. We just the vast majority of the experiences of the world we never get our uh we never experience. And so I'll give a couple more examples here.

So one of our partners Joyce P was able to discover last year with AI models that elephants have unique names that they call each other by. Par parents spend the first couple weeks of their chick's life leaned over their chicks whispering a unique name that they will then use for the rest of their lives. Right? What's more fundamental than identity? And while I'm talking mostly about animal communication, I just wanted to throw this out.

Um, do you think plants can hear? And obviously by the way I'm asking this question, the answer has to be yes. Um, but a 2019 University of Tel Aviv study did this amazing thing where they played different sounds to primrose flowers, high pitch, low pitch, and only when they played the sound of an approaching pollinator like a bee, did the flowers respond by producing more and sweeter nectar in around 3 seconds. And so, quite literally, the flower hears the bee and gets excited. Whoever whistled is correct.

Um, our ability to understand is limited by our ability to perceive and AI is throwing open the aperture of that which we can perceive. But let's go the next step. Like if we're talking about decoding animal communication, being able to understand the languages of whales and dolphins, is there even really a there there? And we were really excited when we first started the project to discover this 1994 study at the University of Hawaii where they taught dolphins two gestures. The first gesture was do something you've never done before, right?

Innovate. Already a very complex thing to communicate and requires understanding the concept of negation, understanding the concept of innovation, but they can do it. They would then teach the dolphins a second gesture together. Do something together. And they would say to dolphin pairs, "Do something you've never done before together."

And the dolphins go down, exchange sonic information, and when they come up, they do the same thing they have never done before at the same time. And you're like, "Okay, if it isn't language, what are they doing?" I think it puts the burden of proof on the other foot. But of course, if we are going to do something like translate a animal language, well, that doesn't already have a Rosetta Stone, how could we possibly translate a language that doesn't already have a Rosetta Stone? And this was impossible up until 2017 when the AI community figured out how to do exactly that.

How does it work? Well, the way AI sees, if you know, technically, these things are called embeddings. But this is how AI sees language. This is English, the top 10,000 most spoken words. The stars that are near each other represent words that mean similar things.

Uh stars or words that have a shared um relationship, share a distance and direction. So sort of relationship becomes uh geometry. Uh and the way this works then is that if you imagine you know this shape is English, this point here is dog. Well, dog has relationship to friend to guardian to howl, yelp, fur. it sort of fixes it in a point in space.

And if you solve the multi-dimensional sudoku puzzle of how every concept relates to every other concept, you get this rigid structure that represents language. Now, the computer doesn't know what anything means. It just knows how they interrelate. So, you get a rigid structure representing the relationships of a language. And this was the core insight.

Could you just take the shape, which is Spanish, and the shape, which is English, and rotate one on top of the other? And if you did, do they match? And if you talk to anthropologists, they'd say, "No way." But the machine learners like, "Whatever. GPUs are cheap.

Make them go." And it worked. Now, they're not exactly the same shape because there are words in one language that don't exist in another. But they're close enough that the point which is dog ends up in the same spot in both. And then you're probably going to say, well, English and Spanish are super related, but it works with German and Japanese, espiranto, Finnish, Aramaic, Erdo.

Pretty much every human language fits in a universal human meaning shape that lets you translate to and from them with no Rosetta stone, which I just find so beautiful, right? Like in a time of such deep division that AI can show us there's a unique structure that unifies us all. How cool is that? And of course the first thought after this is well can we build these shapes for animal communication? And if we do that which parts overlap and would that let us do direct translation.

Now the next thing you might say though is well um maybe we all share this kind of shape because you know language is a shadow of the world. It's a model of the world. Our brains are the same. Our senses are the same. So the shapes should be the same.

But whales are so different. You know, sperm whales spend a uh 80% of their life a kilometer deep uh in complete darkness seeing with sound. Their world view might be very different. But here's what's interesting, and this honestly deserves a talk in and of itself, but these shapes, it turns out all AI models, no matter what they make and what modality you train them on, are starting to find the same shapes. And so if you you guys know Dolly, I'm I'm assuming, and all of the image generation, midjourney stuff.

Well, this is how it works. You build a shape that represents languages. It the AI builds a shape that represents images. You look at image caption pairs that creates a alignment between them. Then you just type portrait of Chile as a person.

it finds the right spot in image space and boom, it makes that image. It's it's actually very profound that every modality, whether it's DNA, images, code, all can be put into sort of a a joint space. But it's another thing, come find me later. We can talk about that because it's it's very interesting. But this is what gives us hope that translating to animal language when their worldviews might be very different is possible.

This, by the way, was my face when I first saw all of this. So, we end up with this maxim. Everything that can be translated will be translated. We're on this journey. We think we can probably figure out what animals are saying by 2030.

And you know, we've talked about some of the why would we expect there to be much alignment whatsoever? Well, I just want to give two examples here. Do you guys know the mirror test? This is where you paint a dot on an animal. You show them a mirror.

They didn't know about the dot before. They look in the mirror. They start getting off the dot or investigating it. And that shows that they have to associate that thing in the mirror. That is me.

Self-awareness. So this these are dolphins looking into a mirror, checking out the inside of their mouths, having possibly the most universal mamalian experience, which is you look in a mirror and you have to check out your abs. Right. It turns out elephants also um pass the mirror test. A number of species do.

But it took scientists a long time to figure out that elephants passed the mirror test because they were using small mirrors. Um which just sort of shows you like humans. Um, but if animals are communicating, they may well be communicating about rich interiority, self-awareness, consciousness, the deepest thing that we have. Here's another example. These are lemurs taking hits off of centipedes.

So, they find these centipedes, they bite into them, they get high off of the venom, they get all translike, and they get super cuddly. It's essentially protourning man. And it turns out these are not the only animals that like to shift their state of consciousness to reach for transcendence. Uh dolphins for instance intentionally inflate puffer fish, get into a circle, sort of bounce them around getting high off of their venom and sort of the original puff puff pass. Um and you know chimps will get onto um vines and spin.

They're sort of like a proto whirling dervish. Like many animals seem to seek altered states of consciousness and if they're communicating they may well be communicating about their experience of transcendence. So while I think we can guess that animals certainly are going to gossip it may go much deeper than that. All right I know I'm nearing the end of my time but I can't resist another guessing game. What animal makes this sound?

Any guess? Raccoon. Raccoon. Nope. Not a fox.

But what does the fox say? Um, this is uh these are beluga. And beluga, like you heard all of that sort of digital sounding stuff. That's them. If this is just a microphone in water and you know we communicate up to or we can hear up to 20 kHz, they're communicating up to 150 kHz.

And it turns out they communicate in these modemike packets. Um, and Dr. Valer Vagara, one of our partners, is able to show that, you know, they include in their hellos, they include their name and their clan identity. This is who I am. This is where I'm from.

Um, but that's the only thing so far really that's been decoded. And this is why this problem is so hard. This is what belugas are like in the wild. So, even when you have a what's known as a whalable, like a a Fitbit for a whale that lets you listen to how they move. Sorry.

Um, so you get like motion, you get um audio and video. Even then, one whale will often be speaking into the microphone of another whale. So, it's hard. So, it requires AI to separate it all out. And here is sort of like the mind-blowing stat is that Dr.

Dr. Valera Vagara was only able to use 3% of her data because she couldn't tell who was speaking or they're overlapping. And so you're like, okay, 97% of the most vocal underwater species with the greatest vocabulary is completely opaque to science and requires frontier AI to be able to tease it apart so we can start to research it. And this I think is the theme I want to leave you all with, which is like our ability to understand is limited by our ability to perceive. Our ability to perceive has been so very narrow and it's about to be blown wide open.

Um because I'm sort of at the end of time, what I might do is sort of skip forward a little bit. Uh we just by the way uh it's coming out in uh ICLR published the first sort of like large scale foundation model for uh for nature. Um so let's anyone sort of like type to a data set. Uh and the very interesting thing here is twofold. One we're starting to see those incredible emergent behaviors.

This model can name a species by name that it has never heard before which is a crazy thing. And two, and this is the really hopeful one, that we're starting to see what's known as positive domain transfer, which is a technical term for something very simple, which is to say when you train these models on human speech and human music first, it gets much better at doing all of the tasks on animal communication. What does that mean? That means the there is something in the structure and semantics and meaning of human communication and music that is helpful. Those patterns are conserved in animal communication which is starting to prove out our first hypothesis that these shapes will be alignable.

So we're sort of at this place now. Uh I don't get to tell you about these awesome crows which is too bad. Um, oh, but I sort of want to tell you about this. Am I allowed to? Yeah.

Okay. So, uh, one of the animals we're working on with are these crows with the universe of Leon. They have little backpacks, nest cam. And what's cool about these guys is that normally crows um, they raise their chicks in pairs, but these guys come together. lots of families and they do communal child raising.

So it's almost like a commune or kabutz and no other group of crows seems to have this. They have their own unique dialect and vocabulary to do this culture. They will take outside adults in teach them their dialect and then they will start participating in their their culture. It's it's very cool. Um, and a lot of what we've been doing then is sort of uh figuring out how do we align the sounds of individuals to the behaviors of groups and vice versa.

And what we found already is that there are these um there's a specific call that our AI picked up which the birds make when they get home. So they land and they make this sort of soft coupooing sound and that means to the chicks like you know honey I'm home get ready to be fed. But there is one bird, the human name is Rosa, who's a mother, that for whatever reason the other crows do not like. Um, and they bully her. And so whenever she lands at the nest, they all chase her off.

And our model is able to pick up that actually she starts making the honeyam home call a little bit louder while she's circling the nest and then she lands quickly feeds and then heads off. Um, and it really gives you a sense that as we start getting into this, this is not just going to be, you know, decode animal language. This is going to be getting down to the individual personalities of specific beings. But I think this ability to translate from audio to behavior is key. And to give one example, um, do you guys know about dolphin and whale strandings?

No. Um, a couple people do. Very strange behavior. It's been picking up um at least in some populations since 2020 where whales start doing this um they get into big groups, they start communicating very differently. They get very agitated and then they beach themselves.

That is this is a kind of mass stranding, mass suicide. Yeah, it looks sort of like that. No one knows why people some people think it's sonar, but they're they just they start communicating very differently. What may be possible as we get a little further is figure out how do you talk them off the ledge? How do you intervene?

This this kills many many whales. All right, final I'll just sort of like speed through to the end which is you know where we are now. We're sort of a GPT2 era tech. We're like a nonprofit. We've been on a small budget.

We Reed Hoffman just gave us a a big infusion. We're sort of doing a bigger raise because now is the time. It's impossible without scale, but inevitable to do this with scale. We started to work on much, you know, we build the foundational technology. We're starting to now work on the verticals.

Which species are we going to go really deep with? Um, one across every major group. And I'll just end with this like but why are we doing all this in the first place besides that it's sort of awesome is how many people here saw my octopus teacher and how many people stopped eating octopus? So I would say like maybe like three4s um of people stopped. And what I find so interesting about this is that did the film ever say feel bad or feel shame or that human beings are bad?

Never did. It came through the side door of love and awe, compassion. It's a very different theory of change for how we as humans can wake up to the fact that we are disconnected from ourselves, from each other, from the natural world. If you're to think about what is the through line between among all of our different biggest problems, whether it's runaway inequality or climate instability, it's a narrow optimization at the expense of the whole, focusing on some part breaking the big picture. And that's just another way of saying disconnection.

How powerful will it be to rewrite our story, the who we must be to reconnect us with the rest of nature? And then I'll just end with this, which is, you know, in 1995, I think it was December 18th, a group of astronomers pointed the Hubble telescope at an empty patch in sky. And what they discovered wasn't nothing. What they discovered was the most number of galaxies anyone had ever seen. And that gets back to we can only understand that which we can perceive that when we get a new scientific tool, we point it at a place that human beings thought was empty.

And what we discover was not nothing but everything. And we're at that point again. And I think the hope here is that, you know, the last time we invented something like the telescope, what did we learn? We learned that the earth was not the center. This time, I think what we'll discover is that humanity is not the center.

So, thank you. [Music] [Applause]