AI Tools for Navigating Existential Risk – San Francisco 2025

[Music] Hi everyone. Hi everyone. My name is Lou. Uh you may know me from Foresight Institute though recently I've been going into different uh philanthropic existential endeavors in particular into DAK and machine consciousness. I'm very thrilled to be uh with Ben today.

We actually don't know each other that much. So it's been a new encounter. But I'm very excited to um ask him about this relatively new endeavor which is the future of life foundation. So I imagine that many of you are familiar with the future of life institute right you've heard of it. Yeah it's leading leading figure in um AI safety and um advocating for safe AI development and against existential risk.

And so please Ben tell us about the history a little bit of the how the future of life foundation came about. Sure. How does it like relate to the future of life institute like and we'll talk about how it uh what projects you're accelerating and what you're about. Yeah, absolutely. Thank you and yeah, really excited to be here.

Uh I really admire funding the commons and just the general work people are doing in order to build up like Yeah. Woo. uh like public goods funding infrastructure. Um it's honestly one of the things that we kind of have in common from the future of life foundation. Uh so the future of life foundation is part of the future of life institute extended universe.

Um we're we work closely with them. But one way to think about us is that the future of life foundation is we we were formed uh you know really only a few months ago I think is when we got started. uh we're trying to leverage advances in AI capabilities to help in areas of AI safety. So uh that's primarily but not exclusively looking at using some of these new capabilities to um start new projects, new organizations or support groups working here to use these capabilities to help improve any number of things that we think are important for navigating this critical period. Well, uh like more concrete examples of this might be something like AI for epistemics.

So AI for human reasoning. How do we uh use AI capabilities and the new advances we're getting? Like if we are getting you know super forecasting abilities on tap this year, how can we actually leverage these to navigate and plan better? Uh AI for coordination. So that's one of the things that I was saying we have in common with funding the commons is trying to think about better ways to use very uh powerful AI systems to coordinate our efforts well to be able to help people make good decisions in concert.

Uh and then AI for security uh you know one of the speakers Mark Miller was talking about4 that was a uh secure hardened micro kernel that took 25 to 35 years of human labor to develop. uh if most code is going to be written by AI systems by the end of maybe this year, maybe by next year, what can we do to be able to accelerate the development of similar things like SCL4? Can we cut that down by orders of magnitude? That would be hugely important as AI uplift brings much more cyber exploitation and similar types of abilities to other actors. So in all of these areas, I kind of think of what we're doing is trying to find areas where the market provided security isn't sufficient or it's not happening fast enough and helping develop orgs and projects that can do it now so that we can have it before we get into a really critical period.

Awesome. Well, maybe we can double click a little bit on these uh three broad categories. Uh where first of all, where are you out like exactly right now in the development of the foundation? Do you already have like projects under your um in your nest? Uh where are they at?

Like what is the size of like funding um that is happening? Tell us a bit about the what's up. Yeah, absolutely. We're we're very early on in terms of our organization. So we're part of the future of life institute.

So we are part of their general uh fund wellunded credit to Vitalic uh for being you know the big funer for future of life. Um we're early on in the sense that we are trying to develop our worldview well like uh we think it's important that we both run small experiments now in these areas so you get a better sense of them and then also do things like technical workshops road mapping to be able to figure out the types of orgs projects that are needed. So um we're likely not going to be doing things that are quite as common to other funders. We're probably going to do more specific RFPs or we're going to be trying to reach out to people. Um, we we we model ourselves a bit like a venture studio might.

Um, I think a way to think about this is like we're going to be trying to um either find existing teams and help provide them advice and support and funding or we'll be helping form those teams early. Um, because one of the motivations for FLF was just there aren't enough people working in these spaces or teams working in them now that we can just give out grants. We need to be a little bit more concerted in providing the coordination and work to help that happen. I'm not sure if I answered all of your parts of the question. What did I miss or what did I skip?

Basically, I'm trying to get a bit more concrete like and to understand for myself like where you're at like um do you already have like a few projects in your cohort and like what what are they? Can you give us names? Yeah, absolutely. We have a few projects though um we again I'll emphasize because I don't know one of the main reasons I'm here is trying to talk to more people to figure out what kinds of projects we should be working on. Um but a few examples of the projects we've um supported something called the windfall trust which is an organization that wants to develop the type of like legal and operational and policy infrastructure such that if you see strong takeoff if you see very powerful economic actors like the let's say the frontier companies that are developing these things having a tremendous share of wealth uh being able to actually share that well many of them have talked about wanting to embrace a windfall clause Well, actually doing that in practice requires quite a bit of policy, quite a bit of legal infrastructure, all of that.

We want to be able to help lay that out. Um, we've also been working on various smaller tools in the space. I don't think any of these would be like accurately described as our big bets, but things like um coordination technology such as like petition and Kickstarter type systems. We're supporting a few bets there. Um, we think many of these might work much better when we have AI systems that can help do bargaining and help do like automated coordination and that might be really important for getting groups of people to work together on these issues.

Okay, that's awesome. Actually, I'm really happy to hear again about the win windfall trust. Um the idea if I'm not wrong came about uh in 2018 at a conference that actually Allison from foresight and I were also um stewarding with the future of life institute really workshop and we were like yeah it really came into the workshop and so I'm glad to see it um reemerge and um yeah good luck with yeah it's a great example of how these kind of things I mean I don't know you never really where the ideas or discussions are going to end up going down the way. And um I'd say another one of the things we're kind of excited about and maybe trying to find LLM tooling or other AI systems to support this are things like road mapping or planning types of activities because right you like go to some conference in 2018, you have some ideas and conversations and you make a map and then maybe that ends up like dramatically influencing the future in some way. maybe if we can do this much better, um we'll be able to help steer these AI systems that are going to be doing a tremendous share of the cognitive work and labor uh very soon.

So trying to share those plans now seems real good to me. Um amazing. And um if people are interested in being supported uh by the future of life foundation, what is the type of projects that should like apply in term of like um stage size? Mhm. Optics.

Yeah. Uh so a bit as I mentioned, we're early stage on this. So you should feel like a low barrier to reach out. Uh my email is benbef.org.

Um we're trying to talk to a lot of different groups. Our primary like targets are going to be ones that are like um early stage or like technologists or researchers or experts in fields who we think should form some kind of focus research organization, focus research bet um and who are going to need help and support there. We're also doing pretty active like headhunting and talent identification in these areas. You can talk to my colleague Elizabeth who's in the audience as well on some of these things. Um the uh yeah so that was kind of like the early stage and probably doing a lot of active coordination here.

I I'd say all right now this is just me totally opining. Um and I want to be clear at this part I'm like I don't maybe I'll disavow this later. Uh I think we're pretty bottlenecked on like personnel and founder types. Um people who are like actively doing experiments in the world and who have that kind of like drive. Seems like kind of hard to find some of them in these areas.

Uh I the areas that I mentioned of like AI for epistemics, AI for coordination, um AI AI security plays, though some of those are much better funded by the market than they are by philanthropic groups. I think all of these are pretty good examples of things where we want to find ideas and personnel. Okay. Amazing. Well, thank you.

Um the Okay, now let's get a little bit into the philosophy. Let's get into it. Let's chat. Yes. Um so I'm curious about like what is the the philosophy of the future of life foundation.

Um it is true like we are what's maybe interesting is we are in the context of funding the comments which is usually a conference where the memes about uh open source decentralization are pretty strong and those are like values um that like really matter here. uh and I guess maybe I can see some perhaps it's only surface contradiction or at least point of tension with the vision of the future of life institute that has been a bit more well if only like working more on the policy like leverage and like uh centralized like system in that sense but also um advocating more on the pose like AI side though now I guess um what I'm hearing is a sort of like segue into accelerate leverage AI tools like thoughtfully or like what is um how do you think about yeah it's a good question and it's like there's a bit of a nuanced answer here where um to be clear I think FLF and myself I mean very concerned about the rate at which AI capabilities are advancing the potential for misuse loss of control all these kinds of risks um I often imagine this is some type of curve where like you have AI risks kind of like shooting up. Uh, and I think our colleagues at FLI are doing great work at trying to like bend that curve in one direction to either slow things down or be able to like focus on AI tools and not like AI agents which I think uh so an Anthony Aguiri our CEO published something I think just a week ago on this topic. uh is it might have been like yeah I I'll remember the name in a moment but um I basically advocating for like focusing on AI tool development and technology that shapes that way as opposed to AI agents which I think are more dangerous. Um but at the same time like we do think that one of the best ways to shape the future better is trying to leverage the developments that we have and that are here and that you know some of which are coming and the tool-shaped type of AI to be able to help us make decisions better during this critical period.

So differentially beneficial tech the kind of technology that is like defense dominated as Vitalic and others might uh talk about are you know important things to develop and to shape well and it's really important that it happens sooner like if we do get more AI agents that are autonomously exploiting systems or persuading people and we haven't developed the kind of technology or security defenses to be able to handle that I think we'll be in a very bad position. So I see us as like kind of tackling a different part of the strategy but like similar kinds of direction. And to the to the open source point, I'll just note that like I'm personally pretty conflicted about this, but I think that like open source and some of the benefits here have been tremendous in terms of researchers like I don't know quite a bit of the field of interpretability requires like open source uh weights. Like a lot of stuff requires this. um a lot of the kind of technology that we'd want to develop are going to benefit from having open source systems.

Uh at the same point though, I'm like, yes, as these things get far more powerful. I mean, I'm not sure if one, I'm just not sure if that's going to continue to be feasible or like what the economic incentives of open source developers will be here. Um feels a bit like an open question to me. Yeah. And there are a few of those.

Um it's funny because I was going to ask you exactly those questions. So, thank you. Well, what do you think? I mean, I'm like I don't even It's curious like I'm I'm betting that the next version of like Deep Seek's model won't be open source. I think that they no longer have like the economic incentives to do that.

I think China is going to clamp down on it more. So, I feel like this will be the direction we see for a lot of groups is some models will be open sourced especially to get like attention and initial funding, but then similar to like Mistl it'll end up being closed source. This is probably a a a paradigm that will continue. Yeah. Um I never um make predictions.

I don't try to advocate for it's dangerous and I always stay away from it. I have a last question because I uh in any um conversation about existential risk. I really like to end up with that one which is uh what keeps you hopeful? Yeah. I think um the the frame of like existential hope is one that I've heard from y'all and Alison and everyone and I do like it.

I also really like I'll put a plug because I'm such a fanboy of Michael Nielsen and all of his writing and he's really been talking about like how does one be an optimist about science and technology here and like where does one steer towards optimism and I think I'm optimistic or like my hope spots around this are I posit most people want good things to happen and that the problem is something like being able to know what the good things are and then coordinate together to do the good things and if we can build techn technology that gives us foresight name check like to do good predictions and help people coordinate well together. I think that we might be able to like collectively avoid the bad things and move to the good things. It's just about actually getting the tech that will work well for that now. So yeah. Oh yeah.

Cheers. Well, that was a wonderful ending. Thank you so much everyone. Thank you Ben. Yeah.

Thank you so much L. Thanks for interviewing. [Music]