Why we had to Build a Petabyte Data Lake to Fix Indexing I Dmitry Zhelezov

Let's welcome Demetri from STD to give him a round of applause. Hello. Hello everyone. How's it going? So, uh, off we go.

Uh, let me start with a short intro. So I'm uh I'm a co-founder of SQD former subwit. So uh my background is in math. So uh I did PhD and and different works and uh then I started doing research in white papers uh reading and crypto since 2018. um tend to be attracted by hard problems related to consensus distributed systems of data and uh SP started in 2021.

uh actually it started off uh as a Python project. So built uh like a framework for building index free chain back in the uh uh polka dot glory days and uh um with the vision to build a decentralized data lake we raised 50 million over multiple rounds over four years and uh SP network finally went live in May form and uh now it's called persistent of projects in particular financial data for retail, for institutions and of course for developers. And today I'm going to present uh SP portal. So this is the merge and this is the new normal way to get the data out of SG. Uh so basically uh it all starts with a dream and realization that the stage for right now uh is not working.

So I'm going to get a little bit more technical right now and talk about the RPC interface with blockchain nodes. Um so what is RPC? RPC is essentially like a handle in a blockchain node which allows you to uh send transactions to like peak into what actually happening in the network in the node and uh actually it was introduced back in the days uh in Bitcoin core by Satoshi is a way to control your wallet. So the assumption was that everyone has like a node. It was only bitcoin and this is essentially like how you control your wallet and uh the truth is that this interface was never meant to be uh used as a way to extract data.

So uh back in the days there was like no such concept as a DAP or like no one was like building anything meaningful on top of Bitcoin but the interface essentially remained the same over the years and uh all the demand for analytics for data right now doesn't really keep up with this like uh interface that uh uh essentially like stalls all the development here and people that came out with just the works for that. Uh and uh let's just like uh how that interface would have looked like nowadays if we're just like started to design everything from scratch and like just thinking uh as a developer or like data analyst how would that be? So first of all of course we would like to have a multi-chain baseline. So we have like single endpoint that we have immediate access to all the data across the chain and the big ones. Uh also of course we want to have it uh what we call sovereign in the sense that you have full control in a similar fashion as you have it with the procedure.

The great thing about this man is that anyone can run it and like once you run it I actually have a full control over like all the data you get all the pipelines. Uh but at the same time instead of like going uh with the old interface uh we want to have something that is more like a database that we can filter, extract, back fill uh and get as much data as we want as fast as we get. And finally, we want to have it uh natively blockchain so we don't have to think about potential for reworks and potential data queries uh related to the uh nature of the blockchain. Okay, so um this is actually like very basic requirements but it turns out this is actually pretty hard to implement and let's get to like thinking how you would actually build this thing, right? This is probably like the first native account.

This is actually something that we like is the very first version for the for for the back time. Um basically we take an existing node or like nodes from multiple streams. They just like put all this data in a database and like slap an API on top. So actually it kind of works and uh I think back in like really in the early days of Ethereum there was a project called DB which basically like put all the Ethereum history in progress and then like you can just like query it as well but obviously there are a lot of drawbacks here and the main drawback is there like so much data that you can't really uh scale it in a meaningful way and also uh for large chains like Salana like more You guys know that uh the latency becomes critical and then you really have to have a database that can handle uh all this traffic and just kind of keep this database on local machine. So that brings us to like the next step like okay how do we actually build this and this is pretty much how all the existing centralized data providers look like nowadays.

So uh they take whatever the existing data solutions of the market. So it could be like bitquery uh big tables like different uh you basically include everything from the chain in there and then you probably also have some kind of a a cloud fashion layer which allows you to get real time data access like um so I think providers like doing like NVO So yeah all the like drivers essentially like following this step. So they take the existing technology and put it in there. So there very big disadvantage of this is that you inherently log yourself to a specific vendor and this is something that you cannot really run locally and but even if you do like if you're a big organization then you have huge operational costs maintain this infrastructure and um so you kind of build everything on your own and uh have like huge data data lake enhance So we rely on the system vendor and uh you need to hand down there as well which changes like which pricing they have and this is definitely not something that you'd expect in the actual world. Um and but like once we start thinking how we actually fix this we turn out okay need to replace the database part with the centralized network and uh this is exactly like what we ended up doing.

So we set up this like click house train and so on with like built uh STD network which is the first decentralized data link uh and uh for the caching and then um real time part we introduced um a thin interface uh which we call portal essentially box which uh gives you a single unified interface which gives you access to all the historical data plus realtime data all in a single interface. base. So you basically don't have to think about like which chain you're accessing uh whether it's like new blocks uh and it have a very has a very simple interface and like handle reworks and so on. So and like the best part is that it's completely sovereign and so you can start uh with just a simple docu file and attach your favorite workstation for example for a local net or desk net. But then if you want to uh work with larger networks uh then you can plug in SQ network uh uh as a plugin uh or for example you can plug in uh uh gaseer or salana or like some specialized node if you really want for example shave off milliseconds of latency.

So uh you have full control again. For example, if you don't want to uh pass any of this components, you can choose one of the providers. So it's really up to you. And of course the uh key innovation and the key uh icing with cake is that the archival part which is really the biggest challenge whenever you build something like that uh is that uh you get all the historical data from the data lake from the uh SPD and the SPD network is a very decentralized data lake. So that means that it is not owned by a single vendor.

Uh it is more than 3,000 nodes uh around the globe. Uh it's crypto native. that was stored by the SP token and uh uh basically it has exactly the same properties as for example um decentralized blockchain consumption. But there's no way that you can actually cut off access to this data. And uh the great thing about the portal is that uh it's basically like a thin uh proxy interface.

It gives you this like real interface for accessing the data post real time and historical for many chains. Uh it gives like almost zero overhead compared to like direct access or receipt for like for example like shredder thing. um but at the same time gives you uh tremendous capabilities when it comes to querying this data, filtering this data. So you can for example request uh in a single uh API call all the history uh of your uh wallet transactions. So you don't really have to go through each block by block.

It's just a simple uh database for you. Okay. So let's just uh think how uh this technology is already being used by the top protocols in the space. Um but before we go there uh I just want to say that portal is really part of the whole ecosystem. So um portal provides the API and a very efficient way to access the uh the network but on top of that we also have an open source SDK which is like a set of TypeScript libraries uh with a supercharge fast decoding of the data.

We also have SP cloud which is the way to host this thing pipelines and indexers through our APIs and of course we have also the network where anyone can uh run a node like take your tokens delegate and soon you will be able also to participate in the port. Um so okay let's get to the first uh case study like one of the giants of spy morph um is already using SVG um and the portals. So the way that they use it is that they actually enjoy uh this direct API exposed by the portal. So they don't really use uh the other parts of the ecosystem like the SDK uh in the cloud. But the way they use it is that they have multiple chains that they have to source uh data from and instead of like relying on this RPC they actually like make portals and primary source of all the events and all the data that happens across multiple walls and then uh they distribute this information across all their systems.

Um another case also firms trading platform GMAX. So uh they actually leverage the power SQ not just the coral. Um they also use SPG SDK in order like to build data pipelines over this API and also they pass their indexers in SD cloud. So basically uh we manage all infrastructure for that. In particular they just like consume custom made uh graphical APIs design.

And uh finally uh the use case that I'm uh actually kind of proud of is u uh squid. So it's like uh an AI AI trading platform and um I'm proud because this is uh also like a side project uh within SG. So we do a lot of dog fooding and use our own technology and uh what we do is that we consume massive amount of information um in order to catch early meme coins. So we are uh one of the few project in market who can for example detect new viewpoints launch uh within minutes or like even seconds and it detect if some whale for example starts biting in that allows us to catch tokens which is like uh have potential of getting like 100 apps uh and only the uh real time and reliable infrastructure uh allowed us to do that and under the boot we used the portal uh SD SDK in order to put this data across multiple chains guidance chain base into click now and do like all different decorations there. Uh the good part is that a lot of this indexer are open source.

So if you're a builder going to you actually are welcome to look at the uh dig repo and take an exploration. So we already support all the major dashes there. those like realtime swaps the real time token detection across like unis swap work or like all the all the majors in slot and uh this is like where we base our intelligence and uh report potential triggers like 10x 100x so um enough of talking uh I just invite you to check our uh new docs uh new beta where you can find all the details about portal uh about the API uh about the ecosystem um we are also going to sponsor uh is global and uh we have our own track um not just that but since it's just like a this fundamental infrastructure for accessing the data I actually invites you to uh combine our track with different tracks for example like this like unis swap and other protocols and uh very high chances that uh no matter what you're building. No matter what track you're choosing, we just like make your life much much simpler because right now the the portal uh is open. So you don't really have to like register like and just like use this API on the get go.

So we're also going to be on site. So um please reach out to uh anyone who has the Rick and Morty portal in their back and uh we're happy to help you. Uh yeah and uh this is it. no longer taking your time as everyone I guess is already hungry like going to lunch. All right, thank you very much.