Golem DB: Data as a first class citizen on blockchain | Dragan Milic, Phillip Jensen | ETHWarsaw [4]

Um, good morning. First and foremost, before we start, I would like to answer or ask all of you a question. So, except for my colleagues, of course. By show of hands, how many of you already know who I am? Nobody.

One, two, three. I'm a little bit surprised. It's my first time on stage, so I'm also a little bit worried how you know me really. So, okay. But in all seriousness, let's talk about something that we all know, data.

Data is the backbone of everything we do online. But there's a dilemma in web 2. Data is easy. You go to a cloud provider, you launch a database, and it just works. But there's a trade-off.

Your data is living on someone else's server. This means you face lock in. you lose ownership of your data and you risk censorship. On the other hand, we have web 3. Web three promotes things like ownership of your data, verifiability and decentralization.

But the moment you need to handle complex data, data that changes often becomes slow, clunky, and expensive. So until now, we've been forced to choose between either convenience or control. But what if we didn't have to choose between either? What if we could have both convenience and control? Well, that's what we built Golem DB to achieve.

GOM DB combines the best of web 2 and web 3 into one single place. Let's talk a little bit about the architecture of Golem DB. We have three layers. At layer 1, we have Ethereum. It's the foundation of security and finality.

At layer two, we have Golem DB. This is our traffic controller. It's coordinating everything, anchoring to Ethereum, and making sure that our system stays verifiable. On layer three, we're introducing something completely new, DB chains or database chains. These are customizable blockchains tailored exactly for your data needs.

They look and feel like a database and you can create, read, update, and delete, but with the guarantees of web free. And here's the difference. In most blockchains, data is just raw storage tucked away as blobs or call data. But in Golem DB, data can be structured. It's searchable and it's time aware.

So that means you can define how long the data should live on the blockchain. Could be a week, a month, or a year. And what this enables is basically to have an efficient blockchain that cleans itself up automatically over time. So let's put it into some analogy and examples just so you could get a picture in your head. So at the top we have Ethereum.

This is our bedrock. It's the solid and secure foundation. On top of it it's Golem DB. You can imagine it as a global logistics hub. It's responsible for coordinating everything.

It doesn't hold the good itself, but it manages the flow and make sure that everything stays verifiable. Connected to this hub are specialized warehouses or what we refer to as DB chains. Each one is tailored for a specific job. And to put it into some real world examples for you guys. You all bought tickets to be here today.

Now imagine a warehouse specialized in ticketing. It can issue tickets. It can transfer ownership when they're sold. and it can automatically expire them once the event is over. Another example could be um let's say Airbnb.

You probably all know about it or at least the concept of how it works. Someone lists their apartment and another person pays to rent it. Now imagine that same model, but instead of apartments, it's computing power. That's what we're doing with Golem Network. With Golem Network, anyone like you and I, we can rent out our CPUs and GPUs and another person pays to rent them.

Of course, in order for this to be possible, we need some place to store the data. And this is where our warehouse DB chain comes into place. So, here we will be storing everything related to our offers that contains things like our specs for the nodes, the prices, and so on. What's needed for the network to work? Now, it's important to state that you don't need to reinvent the wheel every single time that you need to store data.

You can also just use a generic allpurpose DB chain with a SQL SQL engine underneath. So, here you can store things like for your everyday applications. It could be anything ranging from collaborative tools or data for your dashboards, you name it. But the beautiful thing is that you're not limited to any of these examples. It's very customizable and is customizable exactly to your data needs.

So what does this unlock? It means developers get the simplicity of databases from web two but with the guarantees of web 3. Data that manages itself, storage cost that stays under control and applications that can finally run in real time. Because with Golem DB, you don't just query data. Through websockets, your app can subscribe to changes as they happen.

It means you get instant awareness awareness of new creations, updates, or expirations on the chain. There's no waiting. There's no polling. There's just live data flowing into your system, enabling your application to become truly real time. Another great thing about it is that you don't need to take care of DevOps.

Everything you need to get started with Golem DB is your browser. You don't need to host a database or anything itself if you're if you're using a DB chain that's already existing. So just a browser and you can instantly query it. That's it. And that's essentially the promise of Golem DB.

We built it on the vision that data autonomy should be the norm, not the exception. It empowers developers and users with control over their data while still keeping the simplicity of normal web 2 databases and everything they already know ahead. So thank you very much and now I would like to hand it over to Dragon for the technical slides uh or details of GOM DB. So please give him a round of applause.

Thank you Phillip. Uh before I continue, please give an round of applause for Philillip because this is the first time that he did public speaking. I think he did extraordinarily well.

Also, if you have any questions that you kind of can think of while I'm talking, uh there is a QR code for the questions app that we have. You can submit them there. All right. So, uh what is the golem DB or what are we doing there? What is difference to Ethereum or optimism or something like that?

Well, uh we have introduced something that we called entities. So entities are data that you can store. They have couple of things attached to them. So each entity has data which is any kind of sequence of bytes. It also has a unique ID that is kind of unique throughout the life cycle of the entity and it doesn't repeat again.

there is an expiration. So each entity has finite lifetime unlike in Ethereum where you once you store the data unless you go and delete the data that data hangs around forever. It doesn't matter if you need it or not. And also there are some annotations those are like attributes that you can attach to your entities and we use them to index data and to query by. So this is on the kind of storage side and on the queries we have custom query language that you can use to select those entities based on uh based on the annotations.

So how is this coming into the database to or into the uh blockchain to begin with? So it is working uh very simple uh it uh has a usual Ethereum transaction but it is change or encoded differently. It has it's RLP encoded. It has a list of creates, updates, deletes and extensions of the uh life lifetime of the entities. Every transaction y is as the name says transaction.

So if any of those operations within the transaction fails the whole transaction fails and no changes are made on the blockchain. And uh if it fails it will be marked as a failed. Nothing will happen. If the transaction succeeds the transaction will emit logs. So in the receipt you will find the logs and in the logs you will find all sorts of useful information.

For example, when you create an entity, what is the ID of the entity? If you update the entity, what's the new uh expiry of the entity? Uh if you are deleting entities, uh uh you will get uh which ones got deleted. And you will also get if you extend the expiry of the entity, how long they will live. So what is kind of the life cycle of an entity?

So an entity gets created at some point. uh the creator of the entity is the owner of the entity and the entity gets assigned an ID. So technically the ID is derived from the transaction hash and position in the array. So it's quite unique. Uh and uh the expiration is defined at the creation time.

That expiration is how many blocks will this entity live within the whole thing. uh the owner of the entity can decide to delete the entity before the expiration. So there is an operation for that or uh even extend the lifetime of the entity. But if the owner does not do anything when the block arrives where where the entity uh has defined expiry that entity will be automatically deleted. So we piggybacking uh our blockchain on top of optimism.

So we are actually going for the deposit transaction that is there in every block and before that transaction is being processed we delete the entities and emit emit the logs and then process the transaction. So that's how that that works about the query language. The query language is kind of uh matching of the attributes to to their values. We support equals, greater than, lower than. We support strings and numeric values and we also have boolean logic around it.

So you can create any type of uh boolean expression that uh can be quite complex. Uh our engine can handle that without any issues. And what you get is the list of entities that are being matched. Sorry, I will have to go one slide back. So can I give it a spin?

So yes, of course you can give it a spin. We have created the test network uh just for this occasion. So it's only kolleski columdb.io. Uh there is also the QR code that will also before it leads you to this URL, it will also lead you to quick start guides.

Within 15 minutes, you should be able to using TypeScript or Python uh create entities and delete entities and every other operations. So it's kind of full-fledged network there. In theory, you can also run your local nodes with Docker Compose. If you really want to do that, please talk to me after I'll point you to where where those compos files are. So yeah, talk is cheap, but here is the code.

So this is a snippet of the Python code that creates an entity. And you would think, wow, this is a lot of code. But if you take a look at where actually the creation is happening that is only here uh and the rest is just boilerplate Python setting up the client, getting the wallet and importing all the packages that are being used. So it's actually quite easy to use about the tokconomics. So I mentioned that you pay for storing the data.

You pay it as a part of a transaction. It is a gas gas payment and you pay it for times space. That means the longer the data should live in the blockchain. The more you will pay. The bigger the data is, more you will pay.

Uh on our Gollum uh DB mainet, the payments will be happening in Golem tokens. Uh and we can set the price of uh basically how much you will be paying for that time time space. Uh and it can depend on the use case. Uh we can spin up different networks for different use cases. If there is a network that someone would like to subsidize most of the cost, we can also arrange for that.

So it gives us a lot of flexibility. And we will also have a burn mechanism uh where we'll be burning both ETH and GLM on the mainet to be in line with kind of deflationary uh idea of Ethereum. Uh and because we have all of that flexibility, we control the prices of on the of the storage of the data and so on. we are way cheaper than storing the kind of uh data on on Ethereum which is not a very high order because at the moment the last time I kind of run the experiment it's it was about 0.75 E to store one megaby of data.

It's forever but do you really want to store your data forever? We also already have some sample applications that are running on top of our test networks. For example, status page. That is a funny thing uh about decentralized protocols and so on. Uh most of them actually have a status page that is on a centralized service.

So we have managed to kind of have a status page data being stored on decentralized store and being publicly available for everyone to query. There are also some uh marketplaces. For example, for the weather challenge, uh we have a decentralized marketplace where the players of the game can exchange uh their items on our DB chain. And the metadata for the reality metaverse, for the metadata for the NFTs also are being stored on DB chains. We also have a little bit of tooling to help you out when you're developing and when you're debugging things.

Uh we have a block explorer. Uh it has been changed by Netti. Uh and the block explorer not only supports the usual uh suspects of looking at the blocks transactions and so on. It is aware of the entities. It is able to create entities.

It is able to give you the information about the entities and so on. Now I mentioned that we are building on top of optimism stack at this point. This kind of brings a couple of limitations with it. Uh one of them is the maximum transaction side size at this point is 120 kilobytes. If you would like to have an entity that is bigger than that that's kind of impossible because the transaction has to contain the whole entity.

uh we are working on alleviating that and we can increase the limit and so on. I will give more details later. Uh with the queries there are no joins no aggregations at the moment. So you will just get all the data that has been matched and this is very inherent to Ethereum. Uh at this point uh there is no sharding.

So each node has complete copy of the data. So there is you're limited by h how much each node can have on disk and CPU and memory and you cannot scale past that. And there is the throughput limiting factors. So in uh optimism you have a central sequencer that needs to see all the transactions before it puts them into a block. it.

So there is a delay between transaction pool gossiping and sequencer getting the transaction and uh even more importantly blocks in general are just sequential execution of transactions which is the bottleneck if you have a lot of block transactions to process. So now the outlook. So what what are we planning to improve here? So first of all with the query language we are more going to make it a little bit more SQL like just to make it easier for people which is most of the developers aware of SQL uh being able to write the queries more easier and also we will add aggregations so that you can get the count of things or the sums and whatever else you would like to have and there is of course the moonshot. Uh now we we have been kind of riding on top of optimism so far but at some point given the limitations of it uh we have to leave that.

So we will probably redesign the whole network mechanics. So how the network works how the transactions are being processed passed around and so on. But it's all to be defined at this point. We don't have a kind of solid vision of how to do it. Uh the idea is to still be Ethereum aligned.

So on the RPC level, this will still be kind of Ethereum uh network. So you will be submitting your transactions as usual. You will be using RPC to query data. Uh that won't change and probably uh using zero knowledge for immediate finality uh will help a lot because in optimism there is kind of a contention window where you could contest uh the state of the blockchain and that is uh a little bit inconvenient. And we might even figure out how to add sharding because the way that we have designed the entities and everything else that lends itself to being able to shard.

So if we put a bit more effort in it, we might be able to kind of shard across the nodes and we will be driving this uh with improvement proposals. So we will start driving the development ourselves with improvement proposals and we will open to the community to also submit improvement proposals and to drive the development of this. So join us. Um unfortunately slide went really awful. Um we are having a workshop that is starting tomorrow at 11.

uh about how to use colon DB and how to use the SDKs. Kristoff will be making uh being the speaker and leading the whole thing. Uh we'll be also giving special prizes like these web three pies. It's also a project of Gollum. Uh which is a very efficient way of running an Ethereum node because it's everything that it needs to run Ethereum node.

all the storage, all the CPU and so on in a very small package and it's very power efficient. Also, we will be starting a capture the flag competition tomorrow. Please follow our socials uh about that. There will be a prizes uh for capture the flag and there will be a hackathon uh also starting tomorrow as far as I know. Um, and we have $6,000 prize pool for the hackathon.

So, if you're interested, you can take part and maybe win some of the prizes. Um, also, uh, we'll be having a talk tomorrow about the web 3 pies. Uh Peter will be doing that tomorrow and uh I think it is 1:30. Now about the rewards that we are kind of giving out. So for the hackathon I've mentioned $6,000.

First prize gets 3,000, second place 2,000, the third one, also for the uh other participants there will be also some uh web three pies being given away. Uh we also have GLM tokens to give away uh that uh will be part of the prices and also the vanity marker uh market vouchers which is basically uh having your Ethereum addresses looking nicer. So not being just garbled random characters but you can define what is the prefix and the suffix of the Ethereum address and so on. So um and we have the Golem ecosystem fund which will give up to $100,000 for selected projects. For example, uh if some projects from the hackathon would like to continue and so on, we could fund that.

So last but not least, follow us on the uh our socials. So we are on X on Discord and I think we can start with the questions and

have some space for Oh, we have some space space for the questions. So um anybody looking to ask something?

Yep.

We have number one.

So first first we will start with those.

You outsmarted me. I'm sorry.

So we we had this app for the question. So exactly uh what search primitives there are for entities uh consider to engineers perspective can I arrange queries do range queries string prefix suffix contains can extensions be added to say uh I would like to implement my own indices search such that I could okay uh basically most of that so we definitely range queries we have string prefix we have and suffix 2 and at the moment well we are open source you can take take a look we didn't put any kind of plug-in system behind it but uh you can kind of hijack on the whole all of this I can give you a kind of heads up we are storing the state a copy of the state in the local SQLite where we're doing queries so you can just open that SQLite and query there second question is it a solution for enterprises, single users, developers, accountants. So who could be the customer? The that is a very good question. The customer is kind of defined by the kind of constraints of what we're doing.

So consider every Golem DB a public database. So everyone can have a copy of it and everyone can have access to it and it's fully replicated and way better replicated than any other database you you would find. So if your data fits that that that is probably a good fit. If you have tons of data like terabytes of data and so on at this point I would not choose GOM DB for that because we have no sharding. We cannot provide that unless you run nodes that run terabytes of disks.

Uh data storing for limited time. What about local laws? Uh no telecoms need to provide to law enforcement of calls movement. Uh so the information is not erased. Uh it is a public blockchain.

All the transactions are known to everyone. If even if the data has been deleted in the state of the on the node, there is still a record within the transaction that data was created. Um I don't think that uh you cannot trace the calls movements and so on because the transactions are being gossiped over the peer-to-peer network. So it's really difficult to trace all of that. But that applies like all the mechanics and all the limitations uh apply that are applying to an Ethereum network apply to us.

Anyway, so I would like to make a break here uh and ask uh actually Mache to to show what we have done here with those questions because all of those questions we have put into a database. So column DB. So this is the explorer that we have and you can see the questions. So we're a little bit dog feeding our own solution right now.

Amazing solution to solve endless questions from the from the audience. Guys, thank you so much for the um for the time today. We need to wrap up, but huge applause for Gollum. Thank you, Dragon, so much for the for the insights. Thanks, Philip.

Thank you very much.