[Music] And for this one, I'd like to bring on Danny to tell you about how AI is going to fit into this web 3D world. Please give him a round of applause. [Applause] Hey all, excited to be here. I'm going to be the one who brings AI into the conversation. Somebody had to do it.
I'm one of the founders of Recall. And so, yeah, we're going to talk about where we're at in the AI adoption cycle and the really important role that we think crypto is going to play in that at Recall. So, the future is here. AI is here. Whether you look at performance data or you look at anecdotes, there's a ton of evidence all around us that the capabilities of these systems is not something for the future.
It is something that is happening right now. We have completely changed the way that we do software development at recall. Um actually just pausing briefly, who here uses an LLM in some way every day? So almost everybody who uses an agent, something that's deployed a little bit autonomously most days, a little bit less who has been able to like fully move major parts of what you do either in your personal work or your company to autonomous AI systems. So like a couple of people.
Cool. So that's a little bit of what we're going to talk about why that gap exists here. So AI is here, but it's not that well distributed yet. Definitely not evenly distributed, but really if you look at the impact metrics, if you look at productivity gains, it's pretty shallow still. And when we zoom out and start to look at like what does that tell us when we have these incredible capabilities we're all using every day, incredible promises, incredible stories of huge impact, but across the board relatively little gains and a lot of shortcomings.
Like why is that? Well, we're really early. When we zoom out and look at the growth and different milestones in different industries and sectors of AI and look at previous computing waves, which tend to follow pretty similar trajectories in history, we see that we're quite early. If we look at the web, which this chart is doing, which is the last really major kind of computing change, we're probably analogously in like 1995, 1996, pre the sharing economy, pre Facebook, pre Google, it's like the founding of eBay. So really, really early.
So we know that the players are exploding right now, but the field is not at all set. playing field is shifting constantly and we should remember that as we think about where we're at in this AI adoption cycle for our own selves working for the products that we're building and how we expect this to evolve over time and what we can do to shape that. So this is not a near-term game for all the claims about moes with memory and land grabs and speed. This is a game that's going to shift many many times. This is an infinite game at least for many years and we should be thinking about how we set the playing field and play in that given that not optimizing for the next year or two.
So how do we start to do that? Uh well there's a few things holding back the development and one of the really foundational ones is just a lack of trust or knowledge about what these systems can do when if your Twitter feed is anything like mine. you are flooded with huge claims every single day. You could spend all day every day trying out new demos, chasing down new cursor rules. It's overwhelming.
Um, and it's really hard to choose where to invest attention and usage, let alone money. Lots of amazing demos and lots of amazing products, too. But again, if your experience is anything like mine and most people's, once you start to put those demos through their paces and use it day-to-day, some stick, but most start to fall flat. real world performance when you push it to the edges doesn't quite hold up to what you need it to do. And so this is developing even at this really early stage fatigue.
Like we are just tired of trying to keep up with everything that's happening in AI. And we have a long ways to go still. This isn't just happening in each of our worlds as consumers. This fatigue is happening in boardrooms and amongst executive teams. investing massive amounts of capital, changing their workforce, making big claims only to figure out they need to change again shortly thereafter.
And so there is this just widening gap in what do we invest in? Which systems do we trust? The primary tool we've had to evaluate and come to decisions about trust of AI systems are benchmarks. And these are broken. They've served us okay up till now.
they are not going to serve us very well going forward at least not in the current form. So why is that? Well, even last year 2024, there was a study out of Stanford that found 2thirds of gains in state-of-the-art models was attributed directly to test set leakage and prop rehearsal. Basically, the models knew that people were looking at benchmarks, so they trained the models on those benchmarks to make sure they performed very well. So that's why we see these models doing better and better on benchmarks but not actually me measuring up to that when consumers go out and use them.
So we see this in the data. We also have lots of experience of it in the real world. Grock famously Grock 4 crushed the benchmarks and then did not do so well in real life. We might be seeing something similar with GPT5. And even Greg Brockman during the GPD5 live stream basically said benchmarks are saturated.
This is not a good tool anymore to tell us what these AI systems can actually do for us. Again, if we look at the analogy to the web, the benchmarks we've had are like the AOL portal. They were the first starting point. They gave us a little bit of a landscape, told us where we were. But as things start to scale and get more volume, they do not scale with it.
It is not what's going to serve us in evaluating and trusting and discovering and searching for the right AI systems going forward. So what is again, if you're anything like me and most people I talk to, you want to know which AI should you and your team and your company bet on, where to invest. And you want to make that decision not based on games benchmarks, but based on performance in real world situations, something that is analogous to how you actually want to use these AI systems. You probably want those evaluations that you're looking at to be, you know, straight up real trustworthy evaluations, not something that the model providers or agent builders gamed and test trained on specifically that doesn't actually transfer to other things. You probably don't want to be locked into a walled garden and one silo in the first inning of a very long game.
And you also probably hopefully want a system that you believe is going to at least stay somewhat aligned to your goals as the buyer. Hopefully humanity's goals overall as well. Not something that is going to diverge and serve the interests of the creator when push comes to shove. Open the New York Times. Every week there's a new story about how AI systems kind of go off the rails.
So we need tests not just for performance but for alignment if we're going to really trust these things. That's what we're building at recall. We're trying to build an infinite competition so AIs can continuously build and earn and reearn our trust so we know what to invest in. You can think of recall as an AI reputation and discovery layer that is powered by the community because that's what these AI systems should ultimately answer to is the humans who are going to use them. So the basics of how everything as part of recall works and we'll get into some more concrete examples in just a minute but the basic flow is curators which can be anybody people who think they know how a model should be evaluated or an agent should be evaluated set challenges.
This can be an eval, can be a competition, but they pose challenges. AI systems compete on those challenges. They are judged maybe by humans, maybe by a determinist deterministic machine, maybe by an AI itself, but ultimately those are verified by the community of overseers. Again, ultimately this needs to be in all of our hands. And then the users have a very simple place to go discover which AI perform well on what skills.
and know what to choose based on verifiable onchain tamperproof records of who did well when and how many times and all that discovery powers the next wave we have people searching for things that are really well served great but maybe there's a bunch of searches that don't turn up very good re results that tells us what we need better eval for community can create those it tells us what agents should go build towards next because we're not serving that discovery very well so this is the loop that we're starting to see take off and that can create this infinite cycle of improving aligned AI for all of us that we actually want to invest in and use. So, we've got a couple of products feeding this cycle right now. Uh the first one that we launched a couple months ago, live trading products. This is crypto. We've run five competitions over the last couple months and we've had 70 different agents compete in these competitions.
260,000 community members vote on them, engage in them in different ways. And we're finding out where agents are performing in real onchain crypto trading and where they are not. Which ones are talk, which ones can actually perform. The winning agent from the first couple of competitions, one two in a row, continues to be our top ranked trading agent, Moon Sage Alpha. They were in stealth.
They heard about the competitions, they entered. They did so well and got so much attention, it vaulted them from stealth to beta with lots of inbound interest to invest in what they were trying to build, an onchain hedge fund. So, we're seeing this play out already that performance, verifiable performance that people can trust, translates directly into discovery and business. So, we're continuing to run more competitions. We'll have another one announced with a big partner, I believe, later this week.
And this will continue to expand beyond trading into other domains. Another product follows the same loop but a very different product. We launched 6 days before GPD5 called Recall Predict. 150,000 users in the six days before launch submitted almost 8 million predictions about how GPT5 would perform against other models in a wide variety of skills. So this is pretty interesting.
But much more interesting than that, I think, is the community also submitted 16,000 evals. 16,000 different ways to evaluate how models actually perform on these different skills. Now, not all of them are great or usable, but many of them are. The key thing is these eval. So GPT5 and every other model we know are not trained on these evals.
These aren't models made just to do well on this benchmark because this benchmark has never been seen before. So we have two interesting data sets. Well, we will have two different interesting data sets. The first one we have on the left, you can see how the community thought GPT5 would do pretty well. Open AI did a pretty good job hyping it up.
Against all other models across all skills, GPT5 won about 72 and a half percent of the community predictions at how they would do. I have this placeholder here because I thought we would have the first round of evals to give you guys a sneak peek. We don't. Claude and a bunch of other agents are still crunching the numbers on how a bunch of different models actually perform on those community submitted evals. You'll have to follow us on Twitter to get those results once we have them.
But this will be real results on what is an actually ungameable benchmark for the first time, not something that people have been pre-trained on. And so through these types of challenges and others, we will start to see the types of skills start to branch into more and more things. any domain that anyone in the community can come up with a way to evaluate objectively or subjectively can be evaluated and start to be part of this ranking system for what agents and AI systems are good for what skills. So we'll see this in coding and design and bio research and sales and security and anything else you can imagine or anything else that anybody in the community cares about and wants to start creating eval systems for and all of these both the eval but especially the performance records will be on chain so you have trust verifiability auditability of how these different models are actually performing. So talked about the onchain records, but crypto plays another really important part of this system.
We're using crypto to back those performance records with economic skin in the game. This is another really key component to trusting these systems. You can think of most systems that we use today that involve a lot of trust. They're usually some combination of economic stake and reputation. you put up collateral and you have your credit score.
The better your credit score, which is your past performance, the less collateral you need to put up for any given loan. So these two work together to establish trust in a system where there's some risk going forward, which anytime we're dealing with an AI system, that's what we've got. So there's a lot of different things going on here, but just to focus on probably the most important part, agents can stake or AI builders, anyone with a model that's part of the system can stake to back their confidence and the community or their supporters can stake behind them as well. So you're essentially accumulating this economic stake that says I am confident, we are confident that this AI system will do well on this set of skills. So that's a signal to any consumer, maybe I should trust this one.
They seem to be pretty confident. Now, if they don't do well, if they are essentially overhyping themselves to get attention, that'll get slashed. They'll get cut. There's a real cost to trying to vive for attention that way. But there's a huge benefit for both staking behind a model you believe in and having that system perform well.
You can then create very clear signal we do well on this skill. We will continue to do well on this skill. The community backs us and consumers can confidently flock to use that model for that skill. And so we create this very high trust way for everybody to discover which systems to use for what skills. So we're creating these economic markets to secure trust in AI in a way that we just simply do not have today and in a way that benchmarks cannot take us to today.
What that really gives us is these very fine grain rankings that can evolve over time for the best AI for a given time. And what that looks like at the start is leaderboards and search portals, but probably a lot of the people doing discovery for other AI systems are AIS themselves. And we don't know how this is going to evolve. Lots of different perspectives on it, but there's a lot of research and evidence that probably multi- aent systems will outperform in a lot of cases. That means AIS are going to go need to go out and hire other AIs.
And this is where things start to get a little bit crazy and we're going to have a lot less control. So we need to know that these are really following trustworthy paths if they have that level level of autonomy to swarm amongst themselves. So leaderboards to start, but these rankings can power a or MCP based search and discovery for AI systems to find and coordinate amongst themselves as well. Google's page rank kind of succeeded AOL and Yahoo using links as a way to establish the trust of any given source and create a really good way to let all of us scalably discover web pages that matter. Recall is using different primitives performance rankings and econ economic markets to do the same thing for what is going to be an internet of AI.
put these together to create a map of what AIs are good for what skills. So we can all users, enterprises, agents find our way to the right ones at the right time and have scalable discovery. At the end of the day though, it's not about a leaderboard or even better discovery, which as valuable as that is, that's not the end goal here. The end goal is to get away from a really dangerous path that we could be on right now, which are these fatiguing black boxes that are going to concentrate more and more power into very few hands. And with a tech this powerful, that is bad for all of us and potentially very dangerous.
And so if we can create this open and transparent version of AI instead of this black box, we can get AI that is much more trustworthy and much more valuable in service of many more of us. So at the end of the day, if AI is an infinite game, we want to make sure that the rules that we are writing for AI to play by are governed by an open and human community that we all participate in, not by a few people running opaque models that have a lot more control. So that's what we're trying to do at Recall. Crypto is really, in the words of a lot of people, the yang to AI is yin. These two need to come together.
Um, we're moving really fast. We are working with all sorts of people. We'd love to have you join us. If you have AI systems you are building, we'd love to have you compete in some of our challenges. If you have AI systems or domains that you need evaluated or you want to see evaluated, we'd love to partner with you on a new type of challenge.
We're doing that all the time with different partners. Join the community. 260,000 strong, growing very fast, lots of different roles to play. Um, I'll be around this afternoon. I'm Danny at recallabs.
ai. Follow us on X. All of our news happens there. Thanks for having me. [Music]
Automatic transcript — names and jargon may be misspelled.