EPF7 Office Hours | Alexey

recording is on.

Yeah. Is it all right? I'm sorry about it. Should we I can try to reconnect quickly. [snorts] Okay, maybe now is going to be a bit better, but it's the best I can do.

Um, thank you so much for joining everyone. Uh, Alexi, thank you so much for being here. I appreciate it. Uh, so it's late for you, I guess. Thanks for making it.

Um, and yeah, so uh, uh, welcome to the office hours. Uh, we have, uh, our, uh, uh, fellowship sessions where we meet core developers and interested people from the ecosystem. Today, our invitation was accepted by Alex from Aragon. So, yeah, thank you so much for being here. It's an honor.

uh I will uh give a space to follow us to ask you some questions but before I just wanted to get to know you uh learn a bit more about your background if you can I believe that you have some interesting experience even before you join Ethereum so uh yeah tell us uh where are you coming from what you did uh before Ethereum how did you even join in the first place

sure sure uh sounds nice so basically I uh finished the university um in computer science in U in Moscow and then moved to Vietnam for um some career opportunities. Basically I joined some startups which later was a little bit successful and Alibaba acquired us. um and lived many many years in Vietnam since then. Uh and uh because actually working in Libaba was very nice and fun and they have nice culture and nice technology and nice challenges and so on. Um but most of them are based in Beijing and so on and I didn't want to move to China for a living.

So and I uh also tried to but in the same time I didn't want to work on some higher level projects. I tried to get something deep like the search and so on because I basically uh did the same in the past. Um I did mostly focus on search engines and on some semantic analysis of languages and a little bit on the um site reliability engineering sometime just for oh my god Alex fix everything broken sometime let's do something so it never broken again uh sometime finally we fixed everything we don't need [laughter] term anymore we have a lot of uh more and other teams and so on. Um then um basically we I moved to a little bit to the manager stuff. For example, I was uh I had four teams uh two teams in Moscow, two teams in in Vietnam.

Um, and we uh kind of try to uh get all the China products out of China to all the Asian market because they have something like gazillion gazillion gazillion products and so on. But uh near zero technology outside of China like Aloud doesn't exist outside I mean at that time and so on. So we did everything by ourselves and data centers and so on. Um and then I decided to get even some higher challenges, some higher technology challenges and I start to feel and smell that um blockchain is something unique and something very weird. uh because it's something where you have a weird requirement, you have decentralization requirement as a product feature and basically nobody have it uh in the world.

No, no products in the world have it. And as a result um your software should work on some cheap um okay not very cheap but affordable hardware because uh the less affordable your hardware the less decentral the worse your your decentralization and this is very throw a lot of unique challenges you need to basically do um it's maybe a little bit like a mobile development where you cannot just throw more hardware if you gazillion features. Um but it's on the back end side. Um something like you are inside mobile mobile in the back end. [snorts] Um and um in the same time you are in the untrusted untrusted network which is also usually not true.

Usually are you are inside the trusted trusted network and you just give up on it. Um so it's unique from all the direction and as a result I don't know I was lucky to join um team which was crazy enough to to start to ask the weird questions like can we make um Ethereum history 10 times more compact? maybe let's invent our own storage engine to make it cheaper to maintain it and so on and some sometime I hear them and like are you sure and then like yes yes yes let's try this and and then they give me some um arguments arguments arguments and I know makes sense makes sense makes sense okay let's do it [laughter] so it's very interesting yeah and and I'm still um was born in in Kazakhstan. I still have Kazakhstan passport passport. So sometime coming there for fun.

Nice. Very interesting. Thank you so much, Alexe. Uh it's a it's quite a journey. I mean uh quite experience to work in the Alibaba and I mean having a having a startup like this.

That's it's uh uh very cool. So I'm curious then what brought you to Ethereum? more like what uh was the first thing that you worked on? How did you start it in Aragon or like in Ethereum uh general? What uh what were your first projects?

Um first project was kind of um a little bit disappointing because um it was one of the attempts of Ethereum Foundation to uh try to build file something similar to Ethereum. another block another uh decentralized system but for files [snorts] and um most of such systems are a little bit failing like like Gitcoin and so on. There was projects like swarm and things like this and and my first like contributions and was in that area and it kind of suffer every time the same thing it suffered um e economy so it's not enough econ economy engine there to make it successful maybe just not enough tries but since then um I'm a little bit I don't know maybe I have childhood drama since then I I little bit don't believe uh that EF will succeed in this area. uh but um from another side I try to solve it by myself but another way I try to for example when you have I don't know largeum history seven years of it 10 years of it um kind of a lot of data and this data has the same problem of availability uh something like end users are gravitating towards removing this data because they don't have any profit from it. It's not part of the market.

If they validate in blocks, they had a profit. If they store history of a firm, they don't forget a profit and because of no less invisization, then higher probability that they will start to remove it. But if it's cheap for them, then they will not bother basically. And actually last years and so on, I try to do a little bit opposite. I start try to make it cheaper.

Uh so any uh basic cloud providers and so on can host Ethereum uh history kind of for free for them because it's very very very cheap to serve it and um uh users can also store it and participate in the network in availability of the of the data and also make it cheaper uh but without solving last piece about economy. So it's like another way to make it cheaper instead of [snorts] make it profitable.

And then after um go ahead

after swarm I joined argon argon for it was a fork of turbug just we had a guy who also named Alex from uh UK. He had a a lot of ideas how to make um history of inside goerium engine uh g more compact but all his ideas was a little bit kind of hand wavy and the team um a little bit um didn't want to work on it because it was a very big project to migrate all this data models and so on. uh but in the same time it was a little bit unclear uh will it actually work will it actually serve all the queries that we need and so on um all the purposes uh and it was not like so easy to prototype things this time because of not enough AI in the blood uh so uh as a result uh people didn't want to kind of try it because it's hard to try and uh he decided to just launch separated company just for uh trying it. Uh just to gather a little bit funding here and there and in a couple years do the prototype and uh maybe release it as a working version of Ethereum uh note but with a different uh data layout on the disk and basically I was in that time since that time. [snorts]

Yeah. Uh yeah, thank you so much. I mean it's the uh epic history of the Turbo Bugette like I joined EF around the time like 2019 2020 when it was still Turbugette and Aragon was becoming a thing and like right here next to me uh next to the computer I have my 12 terbyte server where I used to run the get archive node before became a thing and uh now I have like space there to run since like I don't know 2021 one or something. I switched to Aragon and uh I saved or earlier like late 2020 and uh yeah it was amazing just to see uh to be able to like instead of this 12 terabyte crazy machine that I had to maintain to just have it in like was like three terabytes at a time or something. It's it's an incredible achievement.

So I'm curious what you are working on these days because I see that like Aragon since this still keeps having the edge keeps evolving and uh has uh always some interesting ways how to make the database even more efficient. Uh so like yeah whether there is some current research or some future that excites you. I think exactly that experiment was successful because um couple years ago I think G did the blog post about Dave accomplish migration and make it default uh like sim similar data model uh of history and so on. Uh so this kind of experiment succeed in this term. [snorts] Yeah.

But uh but if talk only one minute about that part um about the history and and the data format um there is two thing there is one part of it is um how big is it and the second time can this data actually support the network can somehow and basically What we did, we built some sort of uh reproducible data files uh which means like all nodes produce the same files with the same format and same hashes and so on and um can basically share it between each other. uh so they can kind of can share files instead of share blocks because um blocks are too small and validating them is too hard. You need execute you need uh Merkel proof and so on. So verify is too too expensive. Uh and basically for the history it's not um so useful any anyway.

Um so the the very basic validation um and very basic like confirmation that couple nodes have the same thing and not the split brain and so on and kind of removing and the same time and removing consensus and removing uh the the performance penalty of verifying that it's the right files and so on. So in in the same time and it's it's exactly the same files. most weird thing was most I actually didn't believe that possible is that the same files are work as a database file which are you send RPC request and you use these files to get the answer and the same file you would use just to send another note for the network you don't have like additional tarball of what you downloaded from server or something like this nothing you sync these files and you just use them as Right. Okay. Uh, awesome.

I think this is also uh the space I would like to give to fellows if they have any questions. Um, I personally find very exciting to deal with the execution layer data because it's like that's the Ethereum right uh that's all the all the history all the state we have all we need to care about. So um I um yeah I wonder folks if you have any thoughts any questions if you want to ask anything about argon or go or databases for execution feel free to raise your hand and seagar is first go ahead. So hi Alex Sag here good to see you again and thanks for reviewing my PR and the mentorship on my earlier project it helped me get comfortable with everyone. So I have got actually three questions.

So I will jump right in. So first of all, what is Argon's overall plan for postquantum transition? I'm curious because PQ signatures like um XMSS are much larger than PLS and ECDSA, right? So how do you expect that to affect the parameters like the block size, SG and archive storage? So since Argon follows a different storage structure from the other clients by storing the data in KV layers.

So do you think you have an advantage over the other clients on this part? Um yes and no because um when you first of all all the postquantum think they are still quite frequently changing in the format and very frequently changing in the hash functions and so on. Uh same as we have another project which called um you know sorry zil zilorm right uh which is about um uh zero knowledge proofs uh to make them compact and fast and so on uh they're all changing a lot and they are all so basically you don't know um how it will look uh so basically better to choose the data format But uh on the disk after you know what the what the data and what the query and how you will read them and so on and it will happen eventually but probably predict it is hard also I may be a little bit less worried about postquantum just because it's the whole EF focus still so as a result like anyway everybody kind of consolidating moving towards it and so on. So as a result I don't worry much about it.

Oh okay. All right. Yeah that answers my question. So next one is like based on open source

most the most controversial part of all that proofs and um uh even Merkel proofs and and any other kind of vertical trees and so on and binary trees and so on. Most controversial part of it is all keys are very very hashed there. there are as much hashed as possible. Math guarantee you the best hashing ever and the best hashing ever it means the worst data locality on the disk and it basically the hashing of the keys destroy all the performance of the disk. uh because uh it means each your update will be as far as possible from all other updates on the disk just because um doesn't matter how you sort the data uh hashing will do the job and and the all the same thing about witness uh how to store them and so they will they will suffer and they will kind of uh get worse performance uh from disk unless you stop hashing kiss.

Yeah. Okay. So, so my next question is like what separates an external PR which gets merged very quickly versus some other PR which gets delayed for like two or three weeks. So what I want here?

Hello. Hello. Am I audible now?

Just repeat.

Hello. All good. Yeah. So I was asking so what separates an external PR which gets merged quickly versus from one which gets delayed for weeks? Like what I basically want to ask is is there some pattern or practice in the external PR that you want to see which will make reviewing them easier or which would count as an excellent contribution and gets merged quickly.

Um if company is small then likely bottleneck is a human attention as soon as human in the loop. If some human is interesting in it then it will get merged. If some human is interesting in it and forgot about it then it just one kick uh will be enough to get merged. But if it's nobody really interesting in it then it will just stay as is and that's it. Um if company is bigger uh some bigger than Aragon and so on then bottleneck is likely in in in the processes.

So you to be less dependent on individual people inside the company to make them replaceable and so on to less depend on is it on vacation is we have enough knowledge now and so on to use the bus factor people start to introduce more processes and as a result maybe you create PR and it set comes to some uh ticket management system internal some Jira or whatever it's moving maybe once a week on the meeting team meeting and so on. So maybe um nobody just even maybe maybe somebody assigned on it but kind of will start working on it after one week and so on. So um I guess I guess if company bigger then only I don't know or only money pain uh can can make things worse because um usually nothing works usually like I know the big guy who work in there doesn't work and so [laughter] everything else doesn't work. Yeah, but so yeah, my last question is how do you guys approach hiring new talents and what do you look for in them? Like more specifically, if someone wanted to eventually join A1 team, what would you suggest to them or what should they do?

Yeah, that's my final question. what I suggest them. What

um I asking like if someone eventually wants to join the Edgon team, so do you have any suggestions for them like how they should um like what's the path they should like should they contribute as external uh open source contributions or something else? Um I think yes I think today uh the easiest way is to a little bit contribute um from outside because it's get easier with AI and so on um and um just to show up can you are you adequat person or not and so on and after that just uh do the standard thing like send the CV to to the company and that's it and uh get get the normal process like in interview and that's it [snorts] sometime um you you don't need to your question was about but I think um hiring and firing is a little bit probabilistic thing and better don't think about it as um one company game better think about it like I I do 10 interview 10% of them are successful 10% of them are somebody sleep uh disappeared lost go to spam email blah blah blah don't care so just uh get more interviews as into the funnel and you will get the job as a result you cannot predict which job and cannot predict where but um you choose You choose the projects, you choose the clearance, but I mean uh it's a little bit less predicted like sometime companies are just stop hiring and start firing. Sometime opposes they stop firing time start hiring and it totally doesn't depend on you or or uh is it winter is it summer like no nobody it's some unrelated things at all [snorts] to just get more interviews and um of course of course in open projects open source contributors contributions can It's yes it's better than um only um raw questions on the interview.

Yeah. Okay. Thank you. And that's it for my questions and yeah looking forward to meet you during the death.

Sorry I cannot hear. Uh he just says that he's looking forward to uh see you at Defcon. Uh so we are going to have uh uh if you're planning to come to Mumbai maybe uh we are going to have a EPF day there.

I didn't plan I didn't plan uh I remember East TC will be in the French. Maybe I will. Um

yeah Khan next year

I think I think in Mumbai I will not join. Yes.

Ah too bad. Anyway so now is the best me uh chance to meet Alexe guys. So if you want to chat about something now is the time because uh there won't be another chance in Mumbai. So please feel free to raise your hand. Charles go ahead.

Oh, you're muted.

My bad. Uh, with Ethereum continue to push towards higher throughput, what do you see as the biggest bottlenecks on the execution layer today? And, um, which efforts do you think are the most promising for address those? I think uh the I think the big the biggest gap was the absence of no normal uh benchmarking uh tools. Uh and I think it kind of appearing last uh six or month or one year.

Uh by benchmarking I mean benchmarking for um like how much mega gas can do um all existing Ethereum uh implementations popular one uh because usually uh people who working on Ethereum implementation usually they even if they benchmark uh somebody against somebody they usually don't not not really interesting into creating some general purpose um for everything uh and without it it's a little bit hard uh but also I have a very good experience with um a firm foundation had very very many years of um team who working on uh creating um scenarios uh test scenarios uh test cases um which basically all implementation kind of uh attaches a subm module get a module and run them against them and it works very well. It I think it's very helpful to move forward with for for everybody with less mistakes and so on. Um it's the best way to propagate uh for example if we change decision in last minute and okay for the next hard fork we will change a little bit here. uh the logic in the spec. If you do it in the spec, uh it's very easy that you will understand that you're wrong too late and having some small test case added to the general test suit is very helpful and will fire everywhere.

Um the same with the benchmarking things uh becau because um some sometimes we have very diff different implementations they have different customers and target audience and so on and as a result maybe they don't really even test for your order scenarios for example that uh benchmarking suite which I talking about he they now doing something like uh six core and 16 GB RAM. Yeah. But uh most of we are most popular in uh node providers and node providers they usually just throw more RAM and a lot of uh rate um NVME drives um for low latency reads and so on. So most of I don't know most of um bug reports and so on are coming from different sort of hardware and we optimizing different sort of things. Um so I think uh I think so chain tip uh perform general general benchmark for chain tip performances for different scenarios is is a key.

Do do you think that uh uh and I have no specific knowledge of this and I definitely don't want you to name any names but do you think that over time we might reach a point where certain clients are not able to maintain the same performance improvements and begin to hold the network back?

Um and are we seeing that today?

Yes.

Oh okay.

Yes. Uh it's okay. It's okay. Um some everybody in that tool a little bit red a little bit green a different scenarios because um we we are all pushing a little bit bomb the from different directions somebody faster in CPU somebody faster in IO somebody faster in uh somebody just uh I don't know was busy with the new feature and left some big big copy copy copy got run the code and forgot to clean up and so on.

[snorts]

Um so yeah and I don't think it's a big problem that if somebody will be read in this pitch marks and so on it's it's fine they they kind of there everybody adult people nobody will say it and um everybody can say okay we now um have no I don't know manpowers to to fix it or um uh we're working on it or it will take one more year or um we don't want to optimize for this because we think it's a wrong direction for Ethereum for example I know we are read uh on the test where people trying to get boundary increase the boundaries how large smart contracts they can deploy to firms like now is 16 kilobyte and they want to four times bigger and in that scenario if you have full of blocks of this 64 KB uh cause then we are kind of red. Um yeah but

do do you know is there a canonical website where everyone is posting their individual results uh that I could learn more about this or is there is it just kind of just a

it's called it's it's called the benchmark cutter in Foundation. Uh I am not sure if they like completely public or not. Uh but um I was easy to get. [laughter] I [gasps] don't know. Um yeah.

Um maybe maybe they are enough public or maybe they writing blog post about it.

It's called benchmark cutter.

Yeah. Ben benchmark cutter team.

Okay. Thanks.

Uh yeah, I think you meant the benchmark core from the pendal.

Yeah. Yeah. Yeah. Benchmark core. Yeah.

Yeah. Yeah. Yeah. Yeah. Uh yeah, I mean they they uh continuously runs they have some results uh they keep private but uh you can just write yourself.

I guess it's open source but uh it was yeah created like a year ago or something for them to try it out. I'm not sure if they moved it much since. Yeah, I think it's uh every open source because um I know that everything I requested access is the is their UI part uh where they run all the nodes. Uh but that's it. Uh and I can easily run it on my server if I need and I can just say hey cloud run it.

[snorts] Uh and it just works. Um so I guess you can too. Yeah. Um, yeah, that's very interesting because actually it was one of my first first jobs in EF was benchmarking the nodes just like with b scripts at a time like trying to make sure they run in in sync and uh yeah um it was a very minimalistic approach. Now we are orders of performance further I think.

Yeah. Um yeah and it's very very interesting to hear your perspective. Um folks do you have any more questions? something else you want to check about? Uh, Stephen has a question.

Go ahead.

Uh, hi Alex. Uh, yeah, I I I know you are an expert on Aragon storage. So, I I I was wondering like do you think the Aragon storage architecture uh need to change significantly in the future with par execution and with uh like the gas per block uh just continue increasing? I think gasp per block will not um change much but parallel execution parallel commitment is we done already already released and so on and uh yes that guy is changing um shifting the bottlenecks because at least in Oregon when we was sequential the bottle neck was the um IO

discussion

uh the dis disc latency um because when we're talking about Ethereum virtual machine it's something like not very predictable thing you can predict a little bit you can u understand who senders and so on you can a little bit maybe pre-warm uh in advance but it's really a little bit and you really don't know what going to inside uh when you start executing it which storage and where it will it will read. Uh so you basically need to execute it to to understand what you need to read and as a result um you have kind of data dependency. You don't know which data you will need until you read the previous data and all this is kind of um depends on the disk latency. So NVME uh kind of the solution for the disk latency. But uh with the parallel execution uh you throw many transactions in different workers and all of them first they are in parallel start to read.

It means um much more parallel reads and modern disc can handle parallel reads and well and it also means they are warm up in each other because sometime they read the similar things and also it means we kind of less depend on um IO anymore. Uh and we more depend on again again on VM speed on no again implementation of that parallel execution because some parallel execution um implementations they need the resolution of conflicts after execution and um it can be a bottleneck. uh so I think it will at least for argon it will change uh where is the bottleneck but also there is big EIP coming big change coming I think to Ethereum and I think they everybody already agreed on it called block access list

and the block access list it basically [laughter] remove all that nondeterminism from Ethere Ethereum virtual machine and who created the block, who validated the block. Um they execute transaction and attach all all we read this keys, this keys, this keys, a list of all keys which they read. And when you receive uh any any storage in the network nonv validator receive it you already have this um list of what to read and you basically can do one batch read which is the best for performance u

and then execute on it on whatever it is cach or what is it [snorts] and um

it's interesting because Uh from one side we are also support validators. Uh so we still need to deal with that bottleneck where we don't have uh block access list in ARMS. We need to produce it. Um so we still need to be fast in that case. Uh so we cannot just remove um that part usually but shift in the direction that half of the network doesn't need it is interesting because it's basically turning turning I don't know Ethereum not into some recently I was get popular database called tiger ble uh tittle is a database for financial transactions where you get list of transactions and you know in advance what you will read because it's simple transactions from who to who so you know all the list so you do one batch read in the beginning then you do uh IO free uh CPU loop and then you get all the batch results and I don't know do one right u so maybe uh future film nodes will look like this one read and one right uh but I don't know then what will do then validators will have their own execution nodes I don't I don't know it's

okay

yeah sounds sounds interesting can can I share the like the the dates you mentioned

it's open source it's a tiger be is this popular one

okay Um yeah and

okay yeah and I think you you you you also mentioned that uh right now the like airgo client is is is mostly for those no no providers right and there and uh like uh is specialized for the historical uh storage like a nodes right and like do you think like in the future that um like solo editors can to to run the air client or I mean

yes historically we started from archive nodes but um I think already more than a year we have our own consensus layer and we can you can run it separately from us or inside the ron uh you can join rigon from another cons and so on and as a result yes we you can use rigon as a validator We use uh docu as a we have I mean sometime if foundation pay us by validator keys and we use them to dog food as a validator uh so we run our nodes and maintain them and so on [snorts] uh improve whatever it is missing rate.

Yeah. Thank you. Um and like do you think like in the future will the uh will this education clients will become like more specialized or do you think that they they will like remain relatively general purpose? Historically uh we now have a consensus layer and execution layer and it's some sort of specialization but we have it only because I [snorts] guess um I guess [clears throat] it's the was it was the way of how we can move Ethereum network forward if we have very busy core developers team like let's create separated team which will work on the merge consensus they will work separately blah blah blah and then um we will attach it to execution layer. So uh it was way of scaling developer teams not way of scaling ethereum itself.

Uh so yes I know now specialization is um um is a trend is kind of a road map. uh yes I know that we can I don't know benefit of it if we go to specialize on I don't know if archive note or whatever um also we all blockchains have a little bit of specialization like for example you have a bitcoin and you have miners which are basically not using for hash and bitcoins you have IS6 and so on so it's separate industry with separate solutions so it's kind of a lot of specializations so yes I I think so some specialization will appear uh not sure exactly that it's vital one [snorts] because uh because um no let's imagine we um archive node u what people who who need archive node or most of time it's some data providers they want to provide API uh for all the history of ethereum so they need archive nodes and uh it means that archive nodes need to provide now API kind of high performance API blah blah blah it means it needs to add to specialization RPC layer. Okay. But what several RPC layers are returning um data for example blocks and serial several um RPC methods are executing actually blocks for example debug call and so on if call and so on and uh tracing debug tracing and so on uh they execute blocks. So as a result so anyway need to do need to do execution need to do write a VM with consensus agree to VM without mistakes and so on.

So how of course you can okay this RPC is not archive RPC method you can a little bit separate this specialization here but I think specialization in many cases will fail because even even the get transaction pool out of node without uh taking EVM together with transaction pool you cannot Yes to understand that transaction is really useful, really valid, you still sometime need to uh execute it instead of not only just compare nons are gas is enough or not enough

um and um you as a result each part you will try to get out of a node it's high chance that you fail. you take out transaction pool and suddenly I you need an EVM too. So you basically enh here and there then what the point of specialization. Um the same with RPC, same with um P2P. Uh

yeah, thank you. Thank you. Yeah, that's that's yeah quite impressive.

[snorts]

Yeah, thank you so much. Um, yeah, interesting the spectrum of clients. I just wanted to uh follow up on what you mentioned about the Kaplan, about the conser. It's super convenient. I'm glad that you guys uh actually made this sort of a merged client where you can just run it and just works.

Um, yeah, super nice convenient for the for the archive notes. Yeah, and we got 10 minutes to spare. So, I'm um I want uh I hope to hear more from you folks if you have any more thoughts, if you want to chat with with Alex here. Um Christian, go ahead.

Yep. Hey, uh hey, Alex. Um I found very fascinating your story about joining a team that wanted to invent their own storage engine and getting the fish challenge. And I would like to ask you more around your transition into this subject and especially from my perspective of a beginner into this subject like how we can get deeper into this what resources do you recommend or to look into or what to build even anything that could help us get more context.

Okay. um do this way. Uh you become a team leader. Step one, then you fire several people, several developers uh who get uh feeling that they want to get challenges. uh they then they go to running around the world uh searching for it and then they invite you to the company which uh gives more challenges to you.

[laughter]

That's more fascinating.

This way really works [laughter] in fire people and when you fire people they find job with two time higher salary. So firing developers is very pleasure. [laughter] Um so um you want to understand from what to start? Um [snorts] from want to start. In modern web development world, many people start from just build a website.

As a result, you will need the database and then as soon as you have problems with um with the database suddenly you will start to study about database that ah here I need an index and an index have hash index. We have a bit index what's the difference blah blah blah. Uh so through the pain uh you will study uh of course um the same thing happening when you writing the storage engine you still kind of do the same decision okay here we do bit index here we don't do bit index here we do kash index blah blah blah but um usually you made these decisions based on knowledge of application, knowledge what application wants to read, what application doesn't want to read and as a result uh kind of it still required a very deep understanding what the application doing even if you're doing the database um I guess uh maybe need you can separate to to get more focused you can choose between distributed databases and not distributed databases because when you go to distributed databases okay you have a consensuses you have like Ethereum is a consensus uh Bzantine you have non-bentine consensuses you blah blah blah uh but many people are not interested in it in distri many people are interested in distributed uh databases that layer of it. It's I mean many if you get any uh database distributed you will you will face this problem and somebody's have to solve it with with with consensus and so on. But many people are not interesting and they like forget about it that many nodes exist and that consensus exist and they just focus on uh internal part on how we do we store data on disk and how do we get access to it and uh I think the first step can be just get uh in touch with existing database which which is basically embedded into your application instead of running on another server like Postgress or MySQL.

uh you when you have something embedded in your server server uh for example SQLite or Rox DB or Level DB or um MDBX or whatever um you suddenly uh immediately go so so much lower level that uh you even cannot imagine you you suddenly start to stop talking in um SQL dialects and you start to talk in um uh like weird binary like you just look the 100 binary keys looking at you and you're like I [snorts] think I saw you yesterdayh now I see different between you [laughter] u so I think from the practical part I would recommend to try some embedded database in into your application which is maybe even lower level than SQLite because SQLite is still giving you SQL interface. Uh just if you get some KV K value interface, you will immediately get 10 10 levels lower and immediately [snorts] understand where is the pain and and what and what what they fighting for. Um this second part you always can just um [snorts] um just build things for some people it works. Uh for example, if you get um a little bit popular small database in Golan level which called B DB it's basically reimplemented uh from C to go uh simple database uh cable database but alter did it only for studying it for for study is he interesting or much uh and [snorts] uh he did it great. It's become kind of popular.

It's now inside etc. ETCD now inside Kubernetes. So it's kind of very widely used but it's kind of very simple and it was invented for for fun uh for um studying and um I don't know uh it depends some people it works for some people it doesn't work. for example in implement something like LMDB or bold DB or something it it's it's doable project it's not so so big it's it's like LM LMDB is I mean one C file kind of

4,000 lines uh yeah and if you throw away uh 90% of features there which is optional you will end up with not not very big implementation of it uh the same Um no and and yes and uh the most important thing is um how to not lose the data. So basically the most important idea how to make things durable. So you have have to study about fsync that fsync is existing uh operation systems uh that uh it's expensive that it's um uh [sighs] um that you have many ways to avoid it but cannot 100% avoid it. And um when you will go there, you'll probably get in touch with durability levels, consistency levels and so on. and that so I can I cannot guarantee um I think I cannot advise any book because um two too theoretical to theoretical books.

Mhm.

So Wait.

Yeah. Yeah. So basically focusing more on the hands-on experience. I guess

I think when you do a little bit hands- on experience, you can make very fast decision. Do you like uh low-level database or you like consensus layer part of the database or you like higher level application level of application? Yeah. instead of I think you will fast have meaningful decision of what you interest in.

Mhm. Yep. That makes sense.

Thank you.

No. And fire somebody [laughter] man. Uh it was funny. Thank you so much. Uh Alex say uh I think a lot of great advice.

Uh thank you so much man. Um just very inspiring very interesting take as well and uh uh we were here but it was a good chat and I believe yeah a lot of creative interesting. Uh thank you so much uh everyone for joining but again and yeah Alex thank you much for for um answering all questions for uh yeah advising us here uh very useful yeah

yeah thank you for having me very nice to see you good luck guys

awesome

yeah very well thank you so much everyone and I'll see you next week And uh yeah, hope you enjoyed. Have a great day.

Mario, how is this event going?

Mario.

Mario.

Yeah.

How are this event going?

Yeah, I'm here.

I didn't visit much. Uh, what do you want me to mean?