Fredrik Haga I Analytics at Scale V02 I Dune I Pragma London 2024

[Music] so for this next interview I want to bring on Frederick from Dune I'm sure if you've been in this space over the last 5 years you've definitely heard of dun analytics or have used any of their pages to see what the hell is happening on chain but there's a lot of that history that we want to cover today and I talk about where the rest of the the data analytics world is headed so to talk about that please join me in welcoming Frederick on [Applause] stage I'm super excited for this one hi me too um there's a lot of things happening on chain now and uh if anything this is like the definition of exponential growth uh we've seen from having one test net six years ago and just main net to just so many different networks to a whole amount of craziness is about to happen um what I want to do is start off by just kind of asking you sort of what was it like six years ago and uh and uh what was that frustration that sort of made you start a whole company around quing Onin information um yeah um well uh it was hard I would say six years ago it's hard hard still but uh um but now the obviously the I was I would say you know the whole industry was in a very different spot and and uh I would say it was more of a clear sort of Underdog feeling amongst everyone um and so just getting anything off the ground back then felt really hard um but we yeah so M and my mat and me uh my co-founder who's also here um we had a chance back in like around 2018 to work on some crypto stuff we did some ethereum sort of prototypes and playing around in in the previous company we big company we we worked for and then when the hype died around the summer 2018 sort of off of the Ico bubble um we realized like we we still believe in this space we think like permissionless innovation in finance and self cing of assets and all these things are important ideas that we want to see in the world and and we want to help um succeed and so we were thinking that there's going to be a lot of people building on chain and building experience and basically what you know already from from existing sort of web products is that you need to know a lot about the data to build good products that's basically how it's done like you can't have like you need yeah you need to know what's going on before you can make decision who's sort of clicking where and you know what what's working what's not working so that was kind of like the initial sort of seed of the idea and then uh within the first like six months we basically realized how different it is that this data is open in and of itself so you know a blockchain is the first time a database is like public by default and updates it's still weird like everything is up there for anybody to just look at and everybody has the same information yeah it it's totally unheard of you know among if you think about any data tooling any sort of um data experience it's all siloed it's all like usually tied into a company that has a database and and you know they will protect that and they will work on that data in their own environment and so we kind of had this Epiphany a little bit into Dune we had built some stuff we had worked on cleaning up the data and making it easier query but sort of realized and really my co-founder mad realized that like this stuff is open so it's better to sort of lean into that and embrace that and further accelerate that than to sort of try to work against it or build like a data product that looks looks more like you know traditional Finance or sort of product analytics tools awesome um so that was six years ago and uh and you kind of Ed that but uh obviously other than having doing exist now um has coring onchain data become easier natively or or do we still kind of suffer some of the same pain points uh from years ago I think like yes and no like it is definitely harder in the sense that you know when we started out it was like ethereum L1 we just focused on the the smart contracts that's there making that easy and that that was it and so in that sense you know it was sort of easier to to get going and have like a clear Focus now obviously it's way more hetrogeneous and more things popping up and you need to keep track on all these things uh so that's of course very hard but I do think that also like the market is trying to solve this right so we're trying to solve this there's a bunch of people like that try to make this easier so for sort of the broader sort of audience and and data consumers I think it's it's both of course like for us providers it's like always a catch-up game and and being on top of all the things that's moving but I but I see the tooling going in the right direction so hope I think we're moving to a state where you know it is going going to be a nice experience to to build with and work with all this data but it's uh we're not quite there yet in the sense I mean that's that's good news because that means there's an opportunity for you to fix a bunch of stuff but also uh that we've actually have made some progress so that's that's not actually all all too bad um one thing that you see a lot in this space is the power of a group of people are excited about an idea or a product really sort of make this a a really interesting um experience for everybody else to just actually benefit off of and uh and and dun Wizards are uh are definitely known for that and you've had a really early set of people who kind of were just writing all these SQL queries to get the most amount of interesting information on chain um to be visible and create these public dashboards um H how do you kind of think about and comment on the the Dune community in the early days and the role that they've played in uh helping you you be where you are now definitely you know it's it's uh been huge for for us and I think for the whole space and you know when we tried to raise money in the beginning there was a lot of investors that were just like oh you need to like pay all this stuff and sell it to like institutions right that's how you make money in like sort of financial data but we were like no actually there's like 50 people in a telegram group that we're trying to serve right and and we did everything we could to like Empower them and help them figure out the queries and and and whatnot and for a long time it was a pretty small group um you know less than 100 people for quite a while but it was super active right so MZ and I would be in there and like all these users would help each other out and these were people that were just like super curious about what was happening on chain right and and it took a while but this is I think it's an example of this like you know if you find the niche and really really try to serve it um and then we had this fundamental belief in the openness of like it's better if people work together and come together and build on top of each other with understanding this data as opposed to you know going off in silos and and working but if you had just had an API only and everybody was just writing their own scripts it would have not been this year yeah then it sort of ends up all over and um so uh yeah so so it it I would say it took quite a while but then suddenly sort of the sort of little bit of inflection point for us was basically like defi summer in summer 2020 and then would've been added for over two years already but that was like when the product really made sense and so a lot of people wanted to you know do all this crazy stuff and and then we had all the dashboards and they popped up you know there's a new project there's yams and then like one hour later you have a dashboard and and um and every single vegetable and fruit Emoji you can think of just became uh the the goal at homepage for for D humble Farmers um well I don't want to go too deep into this thing but I'm curious about this uh how has it been like scaling on the engineering side of this um you've obviously had to index everything that's happening on chain but um what have been some of the challenges on the engineering pieces yeah it's it's definitely not uh trivial in in any sense there's there's a lot of data coming in and there's a lot of queries running on top of them it's a very flexible system that we built and you know typically the users of sort of uh sophisticated data tooling writing SQL is typically like a data engineer or data analyst inside of a company right but we just we've just said oh like you know let's let anyone go on a website and just do arbitrary things with this data um which is um you know very computational heavy and very um so there's like all the the blockchain data coming in but then you also need to like have a system that's performant and a nice user experience but it's not getting abused by you know random people also um so that's challenging on the sort of ingestion side there was for a very long time very painful with all these new ethereum um sort of forks uh basically sort of taking get and then just like increasing the gas limit a ton and doing nothing else uh sort of not going to throw anyone under the bus but but like there's some systems that you know basically took get and and just tweak some parameters and it's like okay here's a new blockchain and that was super super painful for someone like us because deriving the data out of that system it was not built for that amount of data so when we don't get all the traces and logs and all the sort of you know juicy stuff from the node it it took a long time for the node to compute and when you then also have like very fast block times it it's like a tremendous challenge to keep up um so um there's a lot of those things and we also support Solan now which you know produce a ton ton of data and of course all the l2s incoming it's we'll get into that for sure um we talked about about the engineering challenges but uh what has it been like uh scaling uh on all the non-engineering side of things I mean I remember uh you being two people for a couple of years and uh and where are you now um yeah know that's also a challenge for sure uh we're about 4 45 people now um it's I think you know it's hard for any startup to to scale and and and grow the organization and and all these things but I think it's particularly hard in this space since the sort of swings are so so violent in you know you go from very deep uh lows and very high highs very quickly and so figuring like the right you know level of investment like how many folk should we have integrating new chains and all these things and you know is it worth you know storing and promising basically to people that we're going to you know maintain this this thing um when it's like the hottest thing in the world you know today but who knows next month maybe it's like not relevant at all and then um so it's a you know sort of Roi and tinkering you know they say the startups are roller coaster this is like the perfect yes one in this Lally one of our company values is embrace the roller coaster so um speaking of being able to support new different types of cases of uh of chains obviously you kind of just mentioned you have salana which has a much much faster block time uh we're seeing different l2s with like sub second latency and and kind of pre informations but uh that's just what we have now uh we're seeing on a daily basis some new L2 or new OP stack chain pop up or or kind of somebody else kind of creating their specific app chain and uh it's not crazy to imagine that a year from now we'll have a few thousands we already have a few hundreds um how does that world look like and um and I want to ask this questions from like a couple of angles one just on your ability to actually ingest all that information and and make it useful for people people but also kind of do you think that creates just too much fragmentation are we just going to seeing chaos does that mean we're just going to live in our own universes but never really talk about what's happening everywhere else um what does that world mean to you from a data standpoint yeah so for us it uh we're really shifting now from a world where we mostly would go and like integrate the chain and fetch the data from the node and then you know we would focus on like letting people build dashboards and Eng with this data on on our site to thinking of Dune more as an open-ended platform that where anyone can put data on dun and anyone can take data out so we're trying to be more flexible on um cuz we know that you know we can't and really no one can't and this is basically the same idea we had on dashboards and and sort of the analytics side initially right where it's like oh there's going to be so much stuff happening here that it's impossible for one single company to just keep up so it's better to just like provide the sort of underlying infrastructure and and convenience of just like anyone going AP and figuring out all the things they want to know about what's happening on chain right um and so that's kind of now coming to the data import side and sort of the the chain coverage side where we we're building way more tooling for people to just put data on dun or and it could be you know L2 data and whatnot but also just adjacent you know we have M data and nft data and a bunch of things that doesn't necessarily live strictly on chain but that's a good point yeah it's like adjacent so we really try to to now just make it way more convenient for people to to put data on dun and and um open the flood gate so to speak and then on the other side you know let people take data out so we now have apis where you can turn any query result on D into an API endpoint so you can feed it into like you know any system you want um and also like data shares with other other platforms and and so you can get like the tables accessible in in your own data platform so we really try to just like you know we have the saying the data must flow and and you know that's that's quite literally what it is just like that's the thing awesome um a big piece of I assume what you want to do is make it easier for people to also understand what to do with the data and um you've already kind of done that with uh some of the llm stuff that you've launched where you can just actually ask a question natural language and and quate things on chain um what's the reaction been so far for uh for that from the users and what are some other kind of interesting product uh you kind of hinted a couple things but what are some interesting product improvements you're doing to help make that data quering easy yeah no for for sure this is uh super exciting what what AI can do to sort of data uh understanding and and quering and whatnot I think it's what what we've seen so I would say we we're excited about some of the things we've built but but I don't think we feel like we really nailed it yet or like there's still much more to to be done um I think what you realize is you know a good analyst doesn't only write the SQL query they also understand sort of they they curate in a sense you know they decide what what are interesting metrics how should they be presented and whatnot and so if you have just like sort of anyone just write the prompt and like you know sure we can create the SQL for you in the in the background but you need to ask the right questions and and figure out you know what are the cool or interesting things to to engage sort of with and and there I'm I'm quite excited about the fact that Dune can sort of do both of these things so you know we can bring in what other people are doing and showcase that and let people sort of discover what is interesting together um while also abstracting away more of the lower level stuff but but I think it's um yeah it's still a lot of work to be done um so yeah and then sort of to your the second part of your question I think it's still data is really underutilized in in this space still like um it is again back to what we started with it's it's just insane that that you can look at you know all this stuff happening real time whether that's for you know understanding risk or like doing your own growth to see how your apps doing yeah yeah yeah it's uh and this is public like in in web 2 you will have sophisticated investors spending you know tens of millions of dollars a year to scrape GitHub scrape all kinds of things to figure out who's using what technologies what what's getting traction and then and then of course use that to to sort of trade and make money but when when you have something like Dune you can literally like anyone just go to d.com and and you know see what's happening on chain it's it's out there for for the public and I think it's exciting for Builders that want to track their own things but it it has this it supercharged basically the feedback loop that you have in in a startup where in a traditional startup it's like okay you build you learn something you get some data and then you iterate um and in in crypto you have obviously the the smart contracts and what not are open source which is great but you can also see the traction that that smart contract got so you don't only see what they built you see how they got there because you have the full graph of every interaction they exactly exactly so so you have this tremendous resource to build and iterate way faster because it's a shared you know data layer on top of just the the open source um code so I think that's still underappreciated and and the fact that you can just learn and build in the wild and and see what other people are doing and learn from their traction as well and and I hopefully that will bring the space forward um you know faster and faster than than anything else and I think that's what got us excited in the first place is this like compounding open nature of everything which is um so powerful and I think we still have a lot we can do to to harness that awesome well I think in this case this is a a perfect note to end this on um I can't wait to see the next set of features and enhancements that you add to the Dune platform and uh thank you so much for this chat my pleasure thank