Internet Archive: Public Data for a Public AI | Intelligence at the Frontier 2026

[music]

Hi, my name is Wendy Hanamura and I will be your moderator day and thank you for for coming a little closer. I want to introduce you to two of my colleagues and friends. This is Mark Graham who is the director of the Wayback Machine. And I think we all have a round of applause because a few months ago the Wayback Machine preserved its one trillionth web page. That's one with 12 zeros.

So, Mark, thank you. And thank you to your team. And sitting next to him is one of my most beloved people in the world, Megan Climate, who's a a founding partner of the Filecoin Foundation, but way before that, 2011, she ran an open data project in a bar in Jalalabad, Afghanistan. How cool is that? So, Megan Climate.

So, you know a little bit about us and I would like to just have a sense of you, if I could. How many of you have ever used the Wayback Machine? How many of you have ever put data into the Internet Archive? Books, photographs, cool. How many of you are builders in AI?

How many of you are dreamers about how technology can be better for this world? Well, I think that is what connects all of us. You are the builders and dreamers who want to work with organizations like Filecoin and the Internet Archive to build something better. So, let's start with February 7th, 2026. Scarcely a month ago, NPR breaks the story that the State Department is going to take down all social from the State Department up till this Trump administration.

Mark, tell us what the Internet Archive did. Sure. Um well, when I read that article, I I thought uh that's not a good thing. Uh we we should try to do what we could do to help ensure that as much of that material is uh preserved and available to folks as possible. So, the first questions were things like uh what are we even talking about?

Uh where is this material? Uh and uh and how one might want to access it. So, uh for example, is it social media? What are we talking about? Is it like Tik Tok?

Is it YouTube? Is it Facebook? Instagram? Twitter? Uh LinkedIn?

Etc. The answer is of course all of the above and then many others. And then within those different environments, uh how many accounts are we talking about? Is it 10 accounts? You know, is it State Department?

Is that it? It turns out uh initially we identified 277 uh Twitter accounts. I think the number right now is up to 740 uh Twitter/X accounts that we've identified. Uh and then to begin identify the the individual posts within all those different accounts and all the different YouTube videos and Tik Tok videos and etc. etc.

So, anyway, I could go on and on, but I'd just like wrap it up to say that in the last few weeks we've archived more than 2 million posts. Uh and uh and and then going into those posts to like interrogate the URLs, which many of which are outlinks to other web pages or linked media, PDF files, YouTube videos, etc. Uh and that right now we've cataloged them more than a million of those posts in the last few weeks have been taken off of the live web. So, you can't get access to those posts and that history, the last 20 years of the US foreign policy as expressed through embassies throughout the world. Oh, but that's okay.

Don't worry. The State Department said that you can FOIA for this information. So, you can, you know, submit like on paper, etc. Anyway, we have this material available archived in the Wayback Machine and because it's material available from governments, we're putting into these special collections as part of what Wendy alluded to called the part of our democracy's library program where we're making this material publicly available. So, in fact, you'll be able to go in and and download this material and then work with it.

Be inventive, be curious, and be helpful and make it more usefully available to people. So, I think what we're saying is Tung Leel said we should separate the people who have the ideas with the people who operationalize it. And think of us as your big database in the sky. We've got tons of data. What would you do with it?

What would be your ideas? How would you bring AI tools to bring it to life and make it useful? Democracy's Library is a project created by the Internet Archive and funded by the Filecoin Foundation. Macon, can you tell us a little bit about why Filecoin would be interested in funding this kind of work? Absolutely.

I mean, the Filecoin Foundation and the Filecoin Foundation for the decentralized web, our mission is to preserve humanity's most important information. Keeping this information that is created by our governments and by that I don't just mean And just adjust it. Thanks. Yeah, yeah, let's get it. Yeah, yeah.

Sweet. Okay. And people people say something that's weird. Okay, great. Great.

I can lean in. Okay, sweet. Okay, incredible. Woo. Excellent.

Excellent. Okay. Information created by our government and by that I don't just mean by, you know, a US government. I mean, any government is a representation of the people who fund it, who support it, who put these people in power. Keeping that history um is not only essential, but is also a basic right.

We fund it, it is our information, it is important for it to be kept as part of humanity's most important information. It is part of the core mission of Filecoin Foundation. The reason that Filecoin and I don't know how many times people have done the Filecoin spiel today, but uh for anybody who doesn't know, Filecoin's an open-source software and decentralized storage protocol. It's a little bit like Airbnb for cloud storage. So, anybody can open up hard drive space, take a little bit of storage, and get block rewards in return.

But, the core reason we're doing that uh is to sort of rebuild the internet and embed civil liberties into the internet. Being able to reliably with proof of storage or cryptographic proof keep documentation uh our history is essential to the Filecoin mission. And we're overjoyed to be working with Internet Archive to be taking government data sets and making sure that all people have access to what is their information from their government. So, so not just US government data, right, Mark? Canadian data.

What else? Well, well, let's back it up a little bit. We overall to give you that larger context, uh the Wayback Machine adds more than a billion URLs of archived material a day. We're talking about north of a 100 terabyte of material a day. And since we're paranoid, we store it in multiple locations both within this country and in at least another country.

Uh so, a couple hundred terabytes a day, it's like a petabyte every four or five days. Um and that's like uh public web and related systems writ large. As for the last 20 years, every 4 years, we've gotten together with other like-minded organizations, a bit of a coalition of the willing, that we refer to as the uh for project called the end-of-term archive, where we go in and we do a deep dive on material published by the US government before and after presidential elections. This last time around, the significance uh and value of that was especially driven home come January 20th um of last year, when the Trump administration began eliminating whole departments, whole offices of the US government, and entire websites with millions of pages within a few hours and a few days just started going offline. So, that's all part of the Democracy's Library program, where we're focusing on archiving material published by governments around the world, with an initial focus on those uh and uh for the US, and also state and local governments, uh Canada, and then expanding out into other countries.

The Knight Foundation actually just announced a partnership um with the government of Bermuda to bring um to bring these government data sets from Bermuda online with the help of the Internet Archive. Uh and and I also just want to like pause for a second and really double down on what Mark said, but I mean, when you want to see something about, you know, a local agency, about something with the government, like how often are you going to the library to look for it? Uh you're you're going to the internet to see what's on that, you know, on that government website. And the fact that that it just normally ends, like that it just normally goes away at the end of a at the end of an administration, is just crazy. It's just mind-blowing that there's no official way of keeping the information from one administration to another through the most common way that it's accessed by the people.

So, let me paint this picture for you. Hundreds of terabytes of data, petabytes even. I think we we have two petabytes of data in Filecoin storage. Some of it government data from Bermuda, from Canada, from the United States, from Aruba. But, it's not just in the servers of the Internet Archive, which does have servers all over the world.

It's also in Filecoin nodes, of which the Internet Archive is one. So, tell us about that vision of decentralized storage and how that works and why you we're doing this. Yeah. Well, go for No, no, go for

Okay. Well, I was going to say, so right, the the two petabyte of material right now that we've moved into Filecoin, um largely as part of this program, DeMarcos' Library. Although, there's other material, too, like from uh the Prelinger archive of of films, is a uh subset of the 200 petabyte of material uh available from the Internet Archive overall. But, it's it's special. This two petabyte is special in that um it is publicly available.

So, you now, on your laptop or when you get home, can go up in and maybe use the Internet Archive command line interface, and you can begin downloading this material. And you can use it as part of various projects that you have, uh maybe related to AI. Like, one of the one of the words we're using here is public lay public AI, uh built in part based on publicly available information, but also possibly uh you know, using AI to leverage and make more useful available publicly available information. I just want to underscore that that is different than the vast majority of the material that we have available through our library, through the Internet Archive. And the reason for that is because the regulatory environment in the United States is not friendly or conducive to the use of rights-protected material in AI.

Right? There are what? 77 lawsuits going on right now in the US about how AI companies are using information. And just to kind of put this in a historical context, in the early days of the internet, we didn't have search engines, right? In the very beginning.

I mean, back in the days of WAIS and in the days of Gopher, we had systems like Veronica and AltaVista, etc. But what really allowed the internet as we know it and search systems in particular to take off in this country was Section 230, which provided a a safe haven for companies to innovate and to build technologies using available material and information. We don't have that today with regard to the American information and AI landscape. So this is holding back a lot of innovation, and why, even though we have this pretty large library, a couple hundred petabytes of material, which is generally available for individual like people, individual humans to use on one-on-one, only a very small percentage of it is being leveraged in a bulk fashion right now by AI and enterprises and and ventures. And a huge part of that is this government-sourced material that we've been talking about here.

So to put a fine point on it, I think in 2021 as director of partnerships at Internet Archive, we had our first meeting with OpenAI. And it was with their chief counsel, Jason. And I said, "You know, we believe in trust relationships. So you've got to bring Sam Altman to meet, you know, I think Sam has to talk to Brewster. And uh you know, they of course we're seeing the incredible honeypot of the 11 million texts and all of the 27 years of digitization that the Internet Archive had been doing.

And we said, "Wow, you know, that sounds amazing what you're doing, but what are you going to do about copyright lawsuits?" And they just looked at us kind of blankly, like, "Huh? Oh, you know, we're sure that's going to be fine." And today I I had a chat with one of the copyright lawyers at OpenAI at the 40 lawsuits, 100 copyright and other attorneys work for them today. But we are not OpenAI.

The Internet Archive does not have 100 attorneys. We abide by laws of of ownership. And so, within the realm of what is legally accessible to the public, we want to make that perfectly 100% available to everybody. To Sam Altman and to you, uh AI developer, because we want to unleash the power of that public knowledge. So, I think the three of us are up here today to ask you a question.

What do you want AI to do for you? What can you use this data for? What is the data that we hold, that we've spent the last 30 years gathering, that you could harness for in- interesting purposes? So, what are what are you seeing out there? What are people doing with our data and other people's data?

Yeah, um for the Democracy's Library data, actually I'm not I'm not sure what people have done with it. I have Yeah, I mean, the people are doing a lot of things, much of which I'm not I'm sure I'm not aware of. Um I've just to give you one example of what one person with some AI and access to some data can do, I would reference you to epstein-data.com. Uh, and this is something looks to get like one person did it with available information.

The um, I I should know the Epstein files uh, we probably have the largest repository of them at the archive right now. Uh, I say that because uh, the the DOJ's website, uh, you might think that's where it all is. Well, there's a lot of it there, but they've taken a lot of it offline. Uh, and we've got um, in some cases many, many versions of individual PDFs that have gone online and then have been taken offline, have been changed, been put back online. And because just like the web, there is no version control system.

So, there's no version control system for this, whatever the number was, 1.4 million documents, 3 million pages of material the Department of Justice has made available. So, um, if we're archived right now, we do have this information available. We are careful about how we are uh, providing access to it because we recognize that there's a lot of material that was initially made available by the DOJ that violated a lot of victim rights. Um, and that they ended up taking that material down.

So, anyway, uh, there's a a variety of material. We already talked about uh, social media posts, the State Department. There's something called the Foreign Broadcast Information Service. Uh, the US government had um, uh, thousands and thousands of people all over the world for decades living in various countries, uh, monitoring the news of the world, compiling it, and then it was be be used to produce country reports on a daily basis. This was uh, work product of the US government about the world that we live in.

We have uh, that material archived available from archive.org and we we consider that material to be publicly available. So, it's another example of of information that you could use to experiment with and build services on top of. Tonight, you could go to see Lost Landscapes. It's a beautiful film about the landscape of San Francisco using found

[sighs]

home movie footage by uh Rick Prelinger. That film I mentioned because it has been digitized through a grant by the Filecoin Foundation for the decentralized web and put in long-term storage all over the world in Filecoin and in the Internet Archive and downloadable to to anybody, a filmmaker, uh someone who wants to do image capture research. Uh this is the kind of stuff that one could find in this vast warehouse of knowledge. Selfishly, I'm looking for makers. I'm looking for a vibe coders.

I'm looking for doers, people who uh we can work with at the Internet Archive that can help make the material that we have more usefully available. And I'll give you one very specific uh challenge. So, as you know, the US government probably bombed uh a school in Iran in the first few hours of the war killing more than 150 children. There was an office in the Pentagon called the Civilian Protection Center of Excellence. Uh it had more than 200 staff members in it.

But within a few uh days of the new Trump administration, I understand that about 160 of those people were laid off. There is uh there are a a a number of documents about the center available through the Wayback Machine. I I I I I am curious to know what we can learn about the work of that center and what we can um you know come to understand maybe about the priorities that our new government has decided to follow and what impact that has had on our world and this most recent war. So, if anyone would like to take up that particular baton, I'll give you some more details and maybe you can get creative with it and help us better understand that [clears throat] part of our recent history. We often joke that the Internet Archive with its many hundreds of petabytes of data, books and movies and web pages, is kind of like that scene in uh Raiders of the Lost Ark where Harrison Ford or somebody pushes, you know, the the golden tablet into this vast warehouse and no one will ever find it again.

So, Mark, let's just say you're a builder out there and you want to get a hold of a certain kind of data. What would you tell them to do in order to access that data? Uh well, a few a few things. First of all, I mentioned uh the Internet Archive command line tool. That's a good place to start.

We've also got a a skill available, so any of you doing cloud code um would make it easier to interface with our systems. We're working on an NCP interface as as well. Uh I would personally invite you into our Slack because that's kind of where we get a lot of things done and uh and I would point you to some other examples of things that people have created. Uh I'm I'm actually looking out here and one of our interns from last year is in the in the audience here and uh he worked with us on a a project that would love some help with. Okay, it has to do with AI.

I call it Ask Me Twice. Uh indulge me for like about a minute on this one. I I I think back of like two large collections of material that humans have organized uh information in. Ink on paper, uh pretty stable, uh lasted for a few hundred years. And then over the last few decades, the web.

Uh way less stable than ink on paper, fairly ephemeral. Pew Research said that something like 38% of URLs that they looked at in one data set that were 10 years old were no longer no longer available on the live web. Okay? So, a lot of it's still there, but a lot of it's gone. Okay, the new medium, chatbots.

More and more people are getting more and more of their information by interactions with AIs, right? Well, guess what? It's inherently ephemeral. So, ask me twice, what do the AIs say and when do they say it? Because unless you're recording some of those interactions and say have a hundred questions about the news of yesterday and ask 25 of the world's top chatbots what the answer is to the question about the news yesterday, tomorrow, you're not going to be able to go back and have any understanding of what the hell these things were saying to us.

I'm looking for help.

[applause]

I want to share a little story. The founder of the Internet Archive would love to have been here. He's in Berlin right now because we are establishing something called Internet Archive Europe both in Switzerland and Amsterdam. But he was an early AI student out of MIT back in the '80s. And as he walked out of MIT, he thought, "Ah, the problem is going to be that AIs will not have enough digital data to feed them."

So, in the back of his his mind, he thought this Library of Alexandria for the digital age, it can feed the AIs of the future. And so he set about with a team of nonprofit staffers building this enormous library, now 200 petabytes strong. And he has amassed these public nonprofit data sets for you to build with, to dream with, to experiment with. So we are issuing, I think, a challenge, an invitation, a door open. There's another project that the three of us work on every year, and it's called DWebCamp.

How How many people have ever been to DWeb? I see a bunch of friends in the audience. Okay, what is this? It's a little bit like Burning Man meets Chaos Computer Camp in Berlin, but this year we are taking it to Germany because there is so much interesting stuff happening in German funding, technology, and European policy in AI. So we are heading to a field and a forest about 100 km from Berlin to bring people with the means, the resources, the ideas, and the communication skills to connect to build this next generation of tools.

So we want to invite you to that. Tickets go on sale in about 4 days at dwebcamp.org. But I I also want to talk about why this human interface is important. I mean, people could just go to archive.

org. Why do we need to get together in a field and in uh Berlin? You are a uh a veteran of this kind of festival. Talk about this festival the feel a little bit like this is probably preaching to the choir because I think that anybody who is showing up in person today values and respects the kind of synchronicity that you get when you are able to just talk to people working on other projects that are there's some that's adjacent to the things that you're working on it or sometimes initially feel very far away from the thing that you're working on and find all the places where there's where there's synchronicities and where something that you've been thinking about can really help another project that you've you know sort of what and think is cool and have been admiring from the other room or across the internet. I think that kind of magic really happens and it's really energizing and it's one of the things that I think gives us all kind of the um good drive to keep working on these projects in the times that it can feel hard.

And do you want to be an absolutely magical space for that? Mark, do you have a do you have a camp anecdote or someone you a great conversation you had that changed the course of your work or your thinking? There's so many of them it's a hard to come up with just one. But just to know that this the what what whatever happens when people like you in the front row and who we've been working together in our own kind of parallel ways for years without even knowing that we existed but we started to get to know each other an hour ago. I mean that's just how it happens and so I I just had wonderful experience there Wendy.

Thank you for creating and nurturing a carry on that environment for all of us. Well, we're hoping to see Les Berry there with Metagov and SJ Klein there with Public AI and we know that Yuree is going to be leading the design track for Decentralized AI. So Decentralized Design. So we invite all of you to come, and I want to open up the the floor to questions. You can ask us anything, literally.

Any questions for Internet Archive, for Filecoin, for uh DWeb? Hi. Uh I was wondering you talked about the different democracies library initiatives in various countries you're working for, some of which you may only like the US government right now you've only been operating uh directly with this uh the Bermuda. Yeah. Uh I'm wondering if there are many instances where governments have been very unfriendly towards the work you're doing.

Have you had direct interactions with uh whether like feel one you'd be worried about in this? How you going about that? Yeah, and a beer you're newer to this game, so I feel like in the but in the 20-30 years of the Internet phase of being this Um have you received any criticism? I There there are lots of examples where governments around the world um don't like things that we've saved um that that have been either published by their governments or published uh about topics that their government is somewhat sensitive to. The Internet Archive is not available in China.

Okay? For example, uh you know, uh I think their their expectation of us would that would that we take down all material related to Tiananmen Square, not just in China, but throughout the rest for all everyone in the world. That's just a bridge too far. Uh you know, but um generally speaking, at least with regard to the US government, um so far things have been uh okay. Um but yeah, uh there's governments that have sensitivities, and we often push back.

Um and um and just kind of take it, you know, as each case comes to us. Yeah, for for a while Wikipedia was blocked in Turkey because it provided um you know, a dis favorable view of how, you know, dis favorable but accurate view of um of how the the not current government but the the one before had gotten into power uh and uh IPFS which is uh our technology uh integrated into bio flame uh was the primary way of being able to access that information. Um and we definitely we uh received a lot of push back. From the Armenian first We also had um some issues with um like a Basque separatist Um uh sharing information with Basque separatists and we had uh um the Spanish government give us give us a push back on that. One of the things about moving material in the file coin is even if we wanted to get rid of it, we couldn't, right?

It's it's it's out there. You mentioned Wikimedia Foundation and Wikipedia sites. I just want to make there 320-ish Wikipedia language editions in the world and Internet Archive is working very closely with them for more than 10 years and because of that, because we archive new URLs that are added, hundreds of millions of URLs that exist in Wikipedia articles throughout the world, we have been able to go in and every single day do an inventory looking for links that go bad and something like 10,000 URLs in Wikipedia articles turn into a 404 on any given day. And we have found and fixed more than 28 million of them and then go in and do 10,000 a day every single day. When you fix 28 million broken links in Wikipedia, I think what you're doing is trying to prevent truth from fracturing.

Where they're right?

[applause]

So, amazing work. Other questions? Yeah, so it seems like you guys build your technology to kind of train AI. I'm actually curious with like the amount of technology that is built on AI now in the large language models. How do you think about maybe the ethics that these companies come in with in using all your data for negative performance?

Uh like say I GBT is using it for what we might consider a negative moral purpose. But you guys are just giving you guys the data to do all this stuff. Yeah, I want to be clear about something. We're a library. So, it's not our data.

It We didn't write the books. We didn't produce the websites, right? We're we're a steward of some of this material. And certainly as any library, most of the material that we have is rights protected in some fashion. And we aren't of those those rights.

So, material that is publicly created, publicly accessible is more generally available. And we could talk all night about ethics. I'm going to I'm going to go on the side of like taking advantage of the tools that are available to try to advance the human condition to the greatest degree possible. So, just to clarify, of the 200 petabytes of data in the Internet Archive, the vast majority of it has not been given to the big uh AI companies or to anyone because it is copy written data. Okay, other questions?

Uh so, how do you got that in the first uh you state on public versus non-public the binary has error as there are I see increasing uh I would say concepts uh driven uh community ownership over uh certain title data that may not be considered public public. So, I'm curious how has a Internet Archives view of defining public data that you archive on has evolved in the last years in particular. Uh yeah, definition hub word or the global supper. Yeah, we'd have to get really specific. I don't think we have time for that, but I'll just say that um you're right.

Some of these definitions are evolving and and that at the same time we're responsive to some of the changes in the landscapes and we're responsive also to rights holders. So, um for example, a lot of material that we archive on social media, um well, it's it's once again, like all the material there it's not we didn't create it. And so, um if a rights holder has an issue with it and um and asks us to make it unavailable, then in almost every single case we say yes. Uh the the very few exceptions would be a high-ranking elected official, for example. But most of those are not foolish enough to ask for that kind of material to be removed.

I think we are out of time, but oh one last question from Ian. Yeah, you request it. Uh what is your personal favorite piece of media or data that is preserved by the Internet Archive or by Oh, possibly um top of mind because Wendy mentioned it and possibly just a shill for getting people to go see it, but there is you know, a viewing of Lost Landscapes tonight. Uh Lost Landscapes is completely magical. So, anyone who hasn't seen it before is preserved on the Filecoin network.

Um this is, you know, like Wendy said, this is found footage mostly from home movies of San Francisco. It's an amazing, insightful view to see this city in its earlier eras. Something that you really just don't you don't get from pictures. Like you you get a feel for how people live in a way that you truly can't through another medium. I'm kind of obsessed with Easter Island, Rapa Nui.

So, and it would have worked would be those that she used in my test case. So, any one of a number of books and papers and and videos and materials related to Rapa Nui that we've collected and archived and made available. That's it. Okay. In 1995, I did a documentary about my father's march through World War II.

My parents were incarcerated in internment camps. My grandparents were incarcerated and my father was drafted to fight for the US Army in France and Italy, where he was shot on a mountain called Mount Folgarida. And I did a documentary with him about this. It's called Honor Bound. And I recently just uploaded it as part of a project called the Japanese American Incarceration Films, where we got 100 films because everybody who experienced that is dying.

My mother was 14. She's 98 now. So, soon we will have what is called a saeculum. Everyone who experienced the Japanese American Incarceration will be dead. And the only place you can go to find the documents, the books, the first person testimonies will be places like libraries and the Internet Archive.

So, go and look at my film. It's called Honor Bound, A Personal Journey. And I hope you'll download it and show it and and do whatever you want with it because that's so much more important than having I don't know getting the couple hundred dollars I used to get from educational distribution. So, there you go. Thank you for that question and thank you.

[applause]

Wendy, how lucky we are that we got to hear that [clears throat] from you and that's such a you know pure example of why we need to have proof of storage cuz eventually as agents you know are able to touch that kind of file you need to ensure that it has not been tampered with and that the video that we see from your family documentary is authentic and that you get to decide where to store it. Yes, it's your choice. With any luck, maybe as the people were in hand and what were to die off and they going to probably not work. We lost the Michelle Obama Foundation, you know, Holocaust survivors. Maybe maybe we could be the first generation that has these people start to die off that we don't have to relearn.

You don't have to relearn these lessons through experience. Hopefully, we can by preserving your stories start to learn them just by watching. I imagine we'll have stories of the future soon about you know, what it's like the water version which is people getting persecuted now when we put in on jails and we're being undocumented or you know, you're citizens. So, we're going to continually have this need that we can't just rely is not that there's anything wrong with you know, Google storage or hyper scale of storage, but it's just a you know, we need to have common common fault storage for the commons also and we able to have independent provability that it's in tamper proof and in post quantum proof in about a couple years. So, thank you so much for this.

Our tomorrow's the platform of protocol can our

[music]