The Community Data Platform with Fredrick Haga

Hello, I am Frederick. I run a crypto data company called Dune. Um, and today I'm going to talk to you about a concept I've been thinking a lot about lately, which is a community data platform. Before we dive into it, I'm going to do a little story. Um, so this is my fourth ET Denver and the first one that I went to was in 2019.

I was here with my co-founder. I had not had a salary for seven months. Um, and we were trying to raise some money. The price of an Ethereum coin uh was $100. Um, and it was down 93% over the last year.

Um, we were literally sitting in a cafe in Boulder and saying to ourselves, okay, when we get back home to Europe, um, what do we do? because we're so broke and this stuff does not seem to be working out. Um, luckily at E Denver, we managed to get accepted into an accelerator. We did a preede round, our first round of investment and we got $250,000 which we which kept us going for another two years almost. Um, so if you're, you know, struggling in the bare market, not swimming in money, um, it might be okay anyway.

Um, it's always great to be here and I think there are a lot of amazing folks here. So, you know, keep keep at it. It doesn't have to be glamorous or a lot of money for you to do great things. Okay, so let's talk a little bit about data. As I'm sure you all know, the amount of data in the world is growing exponentially.

more and more data is produced and how does one work with this data? What what's the tools and means we have to work with data in the world? Well, for the most part, what the answer to that is enterprise data platforms. So people will take a system like snowflake, bigquery, data bricks, red shift, these type of systems and they will leverage them to work with and analyze data. So all of this data that we hear about goes into these systems.

And what do they look like? Well, they look like something like this. So you have all kinds of companies that produce data. And so you have then all kinds of different types of data, marketing data, you know, from your website, financial data, operational data, all of these data sets that come in a lot of different shapes and sizes, um, different data structures, different databases, and then those are fed into a data platform. And you know a bit simplified you basically have a data lake a pipeline some kind of query engine and then you visualize on top and have an interface.

And so in companies this is incredibly complicated. You have a lot of very high paid data engineers working day and night to to make all of this work to make data flow through a system like this. Um and since all of the data sources are so different and heterogeneous, every company is different. Uh but at the end of the day, what you do with a data platform is essentially ingesting, transforming, quality assuring, governing, querying, visualizing data. Um and sometimes you export it as well.

So this is basically how all big companies work with data. Um and basically this is the way in which data is is worked with in general. Um but there's one sort of sad thing with this and that is well who can actually access this. So if you're you know young and inspiring aspiring and you want to work with data well what what can you do? So basically if you want want to work with cool data in the world well you need to learn the cool data skills then you need to get a job in a company with some cool data and that company needs to have very good data infrastructure for you to actually work with that data and then you need to have the right privileges and access to actually do anything with the data.

And so what that means is that very very few people actually get to work with cool data because this is a funnel that's very very narrow at the bottom. Data is produced and mass but consumed by a very very limited amount of privileged people that actually has access. So, there's a little bit of a catch 22 here because if you want to become a data professional, well, you need good data skills to get a cool data job, but you kind of need to do a cool data job to learn good data skills. So, you know, how do you get started? Um, this is actually a big problem if you you want to get into the data space.

And basically what happens with these data pl like enterprise data platforms is that you have this very very siloed world with locked information and data inside of these companies. So you know there's a data platform for a car company for a beverage company for a you know online marketplace. And basically all of these systems are very different. They have many of the similar components but this system will look very different from this system and it's entirely separated and so the people that works here will not learn anything from the people that works here. Um, so it's all very very fragmented and siloed.

And then, you know, let's back up and think about what if data was open. You know, radical new assumption. Data is open. You know, um, and blockchains are of course a new way to verify transactions, to send sort of information and scarce assets across the interwebs, which is amazing. But blockchains also represent the world's first public by default always expanding data sets.

We've never ever had that before. Data has been closed and private. Um so you know if you if you think about it new paradigms always enable new product categories and new experiences. So the smartphone enabled Uber and ride sharing. you know the cloud systems enable these enterprise data platforms um in a way that you couldn't do before when there was all only on premise infrastructure.

So you know what what does this mean for us? Well, we think it means that there's now a new category for how to work with data. And we think about this as a community data platform as opposed to an enterprise data platform and as a new type of product that's never existed before. And what we mean by that is an integrated and collaborative data platform experience tailored for the world of open blockchain data. Um, this is a type of product that simply couldn't exist before.

And so what is a community data platform? Well, in many ways it's it has like some of the same properties as uh enterprise data platform, but the crucial thing is that the data input is known and very structured because it's a blockchain um and it's open. So what we can do then is to take all of this complicated stuff that's in the middle here that's needed to operate a data platform and actually host that serverless for the whole world. Um you can have open access where anybody can actually access this this data platform and you can have um collaboration by default. Uh what's also exciting here because the data is the same a lot of the use cases on the other end are the same as well.

So, you know, across all of these different categories that live on the blockchain, you can learn from what people did before you. If someone was curious about the question, anywhere in the world at any point in time, you can actually build on top of them and look at what they did as opposed to building everything from scratch. So, you don't need to be a data engineer to figure out these insights. You don't even need to know a data engineer. you can just you know use this community data platform.

So you know what does that mean? Well if you now if you want to work with crypto data you know you can go to docom you can work with cool data and you can build skills and your profile. It's that simple. You don't have to ask anyone for getting a job for getting access to any systems. You just go to a website and you can start writing queries on open data sets and share it and build up a profile.

And the amazing thing with this is the amount of people that can do this radically expands. You don't have to be a developer. You don't have to be deeply technical. You can learn learn some SQL and you can actually go and use a data platform. This has never been possible before.

Um and that also means it's flipped on its head. So instead of getting a job to work with data, you can work with data and then you can get a job because people see all the great things you're doing. So this tweet is literally from one and a half hour ago. So shout out to 1k Beetlejuice who just landed his job at the block after one year of tinkering around with a community data platform. Same thing with Hildobi here.

After about a year as a Dune user, um he got hired by Dragonfly Capital. This is totally reversing the process of working with data and making a career in data. But we think it's not only the sort of uh the opportunities for for users that's changing. It also changes a lot how you can build a data platform. So at Dune, we're working on several like pretty intense projects to really really explore what we can do when we're in the world of open blockchain data where we know these data structures.

So literally yesterday we took out of alpha our very own query engine June SQL. So we're building a query engine tailored for crypto data. We can know these data sets that is the blockchain so much better than anyone of these existing data platforms can do about all the modular data sets that they need to get uh plugged into that we can create a more compelling experience. So we're investing deeply into the experience for quering these exact data sets that that the blockchains represent. So that means that we can offer way better performance.

Um, June SQL is 40% faster than what we've seen on Spark SQL for instance on data bricks. Um, even before a lot of optimizations. Uh, we can create a better user experience with custom functions um built for blockchain data built for interacting with blockchain data. You can create derived userenerated functions. So you can take one query, produce a data set, query that data set and really build it in a very very flexible way.

And in general, we can support and onboard way more data. Um, so we're building functionality to let people upload their own data sets, create data sets from what's on Dune, and in general, we have a system that we can scale in a smart way around these data sets, make it more performant, make it cheaper, all of these things. Another product we're building which is only possible because of open data is something we call spellbook which is essentially a community data pipeline. So we've taken a product uh called DBT which is usually usually used inside of enterprises to translate to transform and quality assure data to manage data pipelines in the best possible way and we've turned that inside out and put it out on the internet so anyone can contribute to a data pipeline. So if we take one example, we have a table called NFT trades which contains structured information curated information about NFT trades.

Um so you can see okay you know what marketplace who's the buyer who's the seller what's the US USD value of this trade which collection does it belong to all of these you know all of this information around NFT trading which is way way cleaner than working with you know the raw information. So what does this look like under the hood? Well this is just a little bit of a sample from the visualization of this data set. So here you see a ton of different tables being transformed and creating a new table. This you can see on in our docs you can see this model.

Your browser is going to take some time to render it because this visualization is so heavy. Um but we have here you know a massive set of um understanding of this data. um which means that when you look at this table NFT trades that very simple to use underneath that there's 180,000 uh lines of SQL data transformations there's 90,000 lines of SQL data quality tests 225 thou 225 input tables and we have had 55 arch wizard wizards contributing to building this data set you earn earn arch wizard status when you contribute to spellbook. So all of this goes into you know creating 55 million rows of structured NFT data um and which in practice means that you don't have to you've saved thousands of hours of effort of transforming data because the community has already done it and you don't have to replicate that work. In the enterprise world, everybody has to curate and build and do everything themselves on these data sets.

Um, so this is an amazing thing and so spellbook is as with the community data platform inherently collaborative. Um, you always work together with others. Instead of building from scratch, you build on top of others. Um, it's scalable because you know there's a lot of infrastructure underpinning it. there's a lot of other people's working on it.

So each different project can come in from their side and only work on their piece and then still be part of a a bigger hole. So you know if you're one entity, it's really really hard to curate all the information that happens on an open-ended system like a blockchain, but it's scalable when we're coming together and figuring it all out together. And it's a fantastic win-win for people wanting to analyze this data because they get one easy interface to to look at. It's also fantastic for everyone building project in this space because they can pull themselves into one table and propagate throughout thousands of queries and dashboards. So all in all the way we think about this is essentially the enterprise data platforms is like individual organisms.

They live on their own. They're separated. um and like individual cells that are just you know doing one thing of their own. And on a community data platform we get a super oranism which bear with me is um you know uh defined as a group of synergetically interacting organisms of the same species. A lot of people have the same problems, a lot of the same interests and they can come together and feed off each other and have a knowledge graph and information built together as opposed to insilos.

So a community data platform enables unprecedented access and learning access to learning and building. Anyone can learn, anyone can build, anyone can tap into all this fantastic world work that's been done. You don't need to be a developer. You don't need to set up a data pipeline. You don't need to run your own databases.

Um, it's all out there on the internet, accessible to anyone. So, thank you. That was what I had to say. And now I think we have two minutes left for any questions that might be there. Yeah, there's a question here.

Oh. Oh, yeah. She was Thank you. Hi there. Uh thank you.

Uh I come with a so it's amazing talk. So uh from constructing those data pipeline and can you tell 11 uh tell me more about how you will uh refine those uh data and cleaning like we went from ETL and then now we have a feature engineering like so in decentralized world uh how this will can be replicated. Thank you. Yeah. So I think I got the question.

Um, and so basically what we're working on is like self-s serving more of these tools. So right now it's kind of a lot of collective data sets that we need to like review and and sort of accept and quality assurance and whatnot. What we're trying to do is build more tools to let people self-s serve create more data sets um and have sort of different types of data sets where some are more vetted and sort of community members are saying this is correct, others are more self-s served where you can just do whatever you want and create all kinds of weird things and you know if it's useful for you and then maybe it's useful for some others as well. So hopefully that Yeah. Yeah.

So, so yeah, you know, yeah, you can uh someone can provide it, someone else can transform it, and someone else again can can query it. Thanks. All right, thank you all so much. That was it.