OMN: Towards a Robust Planetary Library – San Francisco 2025

I am so glad to go after Danny. This talk is also about systems and about how to make things less loadbearing and in particular about how to build systems that generate other systems that together are all moving towards an equilibrium, maintaining a system in equilibrium rather than thinking the task is to build a tool that will do something which other people could come and oppose or replace. So, and this this is also a talk about lists of lists and the joy of completing them. So, if you like lists, this talk is for you. We're in the Internet Archive, which is building a planetary library, and it's probably the closest that we've come in this generation to attempting to do that.

Uh, but I want to talk about how to make those kinds of things robust. And we also don't want to have loadbearing institutions. We need 10 archives. We need things like the archive in dimensions that people have only just started thinking about in the last few years. But just as a preamble, people who care about cataloging and passing on what humanity has discovered have been doing this and fighting against the odds for generations.

And that's what inspired this talk. So everything that I'm talking about with a few examples, they're just the demonstrations of like where we are right now in the struggle. So we need a robust library and right now we have the possibility of having a real planetary library that everyone can carry around in their pocket. But we need a complete archive of our culture with multimodal models to access it, protection against source tampering which is becoming easy very quickly. And all those things are at risk right now of enclosure.

the tithes against librarianship. Like the public library is still the epitome of a a universally appreciated public good, not just as a a benefit to knowledge, but also as a tool for independent learning of for people to have control of what they can learn. Uh so we're going to talk I'll describe that briefly, talk about implications for the commons and things that we can all build together and then talk about things that we can do right away. And there will be a office hour tomorrow for people who want to help build out the index. U there are some shared notes here.

If anyone is enthusiastic about taking notes with the group, there will be a little call in response. You could capture some of those. Um, the public library when I think about it leads to rants like this where the text gets smaller and smaller and runs off into the distance. But public libraries are fantastic. And yet we allowed for search and discovery to get delegated to search engines.

And the search engines immediately turned around and started selling cognitive security and paid placement and generally degrading the knowledge integrity of people who were finding information in the world. and but they continue to do it just well enough that people we all rely on them as our reference librarian. And now we're doing the same thing. We're in a similar position with AI. We're delegating AI model development.

AI will dramatically advance all the ways that we access knowledge and find out what could exist and what we could learn. and we're delegating it to organizations that will have all the mechanisms and all the incentives to sell placement, to sell censorship, to sell preferential semantic closeness. So we need what I call an we need a concordance of systems that are all working together to build something that that persists and that is robust against all of the like the tides that are trying to drag things downstream in a different direction. to track the cyclic forces that regularly throughout history array against public institutions of knowledge and learning and to divide and conquer some of the tasks that we know how to solve. I I want to say one give one example from the archive itself.

The motivation for this particular talk came when I was sitting in the internet archive as you do and Brewster leaves around some of his essays. And this is a fantastic essay about what can be learned from the library of Hernando Columbus, Christopher Columbus's son. If you haven't read it, find a copy. There should be copies lying around. I love it.

And this was someone who wanted to build concordances. Like Arnando Columbus, he wanted a concordance of plants. He wanted a concordance of the Latin language. He tried to build the first Latin dictionary. He wanted concordance of all the printed books and works.

Not just the books, pamphlets, handouts made in any language, in any place. His when he thought of a thing he wanted to gather and make available to his community, he wanted examples. He wanted all the examples. And so I started reading this and you know, you you open this book, we're sitting in the internet archive and the first page has um has a QR code which still works and a URL which doesn't work. And I wanted to find like the canonical version of this document.

Maybe it's changed. Uh and and this was hard to do. This is the classic challenge of trying to find something in our currently federated collection of works. U one of the but one of the things that he said which I love is he's talking to you know Emperor Charles the 5th and he says it's one thing to build a library of things found in our time. It is entirely another to order things in such a way that all new things are sought out and gathered forever.

That's the kind of system that I want us to think about building and to imagine ways to do that that are stable under manipulation and replacement and um like retconing of the primary sources. So, one of the curiosities about this year is that we now have the tools to fantastically speed up u compression and and uh wayfinding and figuring out what you might want to read or or study to answer questions that you have thanks to LLM. But we also have the ability to generate fake versions of primary sources that look so much like the originals. In fact, it's hard enough to actually get your hands on a digitized copy of a book. It might be easier for you to just make a copy that says basically what you want it to say.

So, we need to make sure that we have a full catalog of sources that exist now with content hashes with other other methods for confirming content integrity so that we can compare this against what is being used to train systems that we depend on. So in addition to the traditional deadline that libraries burn, earthquakes happen, uh we have a new deadline that there are lots of there traditionally the victors of a war wrote the history books but now you can rewrite the history books so much more quickly and more easily. So some of the things for which we need public services uh people in this room are already working on and the the end of this talk is just an invitation to help map the things that you're doing into a joint atlas to find gaps that we can that we can comprehensively fill. But we need better identifiers in clustering. We need a top level domain that isn't going to go away when someone stops paying the bill.

Uh we need IDs that people don't have to pay for to generate but that are curable identifiers like uh the records inside a library that a community a guild of of curators can refactor and and maintain over time. We need content hashes and content based integrity checks. Uh better approaches for registration and provenence. Uh distillation of metadata and claims from documents and arguments. extracting an from like treating a document like an overlay of conclusions on top of an underlay of ideas and observations and statements.

There's a a beautiful paper that came out last month called um the Alexandria project which is which just is a demonstration of how to extract all of the factual statements in the corpus of academic works in a way that does not involve it does not invoke any copyright concerns and their conclusion is please let's all do this for the corpus of of works. They're sort of asking to the world. They're publishing it on archive and saying, "Okay, uh, whoever whoever has access to these works. Let's all do this together." Preservation.

We have WorldCat. Many of you may know that WorldCat data is not open. If you want to find out how many copies there are of all the versions of your favorite book, you can find it for an individual work by searching. This search is free, but if you want to do it for all of the works by an author, that's API access that you have to pay for. We need open world cap badly.

We need it for this task. Hopefully OLC will be happy to say sure, use it at will to complete this family of tasks. But if not, we need now again this is one of Aaron's projects was trying to trying to convert WorldCat to releasing open WorldCat. If we can't do that collectively with the existing networks of data, we can just recreate this from scratch. And we need propagation.

We do need a way for everyone to be able to carry civilization in their pocket. I mean this is now 4. We have what 10 20 pabytes in the back there. Uh small wonder of the world. But we need the compressed distillations that people really can bring with them wherever they go and do work locally.

And we need event brokers for the data sources that are too large to be compatible with any of this. Um, so the system generating systems that I want to I want you to think about that I leave you with are these lists of lists, concordances. What are all the works that we might want to preserve? A collection of natural history museums tried to do this for all of their individual biological specimens. They compiled a billion or so specimens.

It was a beautiful step that towards what they called the global collection project. but it's not open. They were anxious somehow that uh having all of the information about their physical collections might reduce people's interest in coming to visit them. I'm a little unclear. We need this to be the social norm.

Um I think everyone everyone who's here today is probably contributing to that norm dramatically. But u even among the projects like the internet archive and common crawl just aligning the crawls and articulating the way in which the current web crawls are samples they're not complete and describing ways that other people could advance them really helps move towards a completion of the process. And we need a community. We need a the equivalent of a guild to protect contributors. People who work in this in these areas are losing their jobs in this year in the US, but they're often losing their jobs or having to leave uh their institutions in other countries all the time.

Librarianship is is political. And finally, um we need some mechanism to keep track of of counterveailing forces. Where where is there pressure against the traditional notion of librarianship and towards some notion of central control where you don't get privacy in what you read where u you get to read what is what is on the top of the uh most favored reading list that month. One thing that I encourage everyone to do is start where you are. Whatever you have access to uh try to find the concordance of that kind of work and preserve it and make sure that it has been added to a global library.

This is the work of a handful of grad students in China. They were working closely with one of the national one of the university libraries and uh they started reading up on all of the historical instances where people had destroyed books and libraries across the board. They said, "Okay, we should really make sure that we keep a copy of the PDFs of all the books we can find." They have in the last four or five years uploaded two million books to Wikipdia Commons. Uh, many of which did not have scans online before they did this.

U, I'm sure these kinds of conversations come up all the time around the internet archive. Here's an example from many years ago, the perennial thread. What if we wanted to back up the archive? How what would it take? Um, But this community was, you know, this is a data hoarders community.

Uh this was 250,000 people in the community 5 years ago. Now it's 850,000. There are millions of people, humans, we are we are collectors u among many other things. And we just need to articulate tasks that everyone can collect together. I would love to hear from people who are working in any of these areas.

Uh especially people who are tackling the primary source challenge. If you're dealing with original materials that you know are going away or materials that are produced in such high volume that people can't traditionally capture them and and preserve them. or if you're if you're working against forces that are trying to change archives that you care about, um, come talk to me. Come add add your projects to the document that I shared. Thank you.

[Music]