Talk | A Deep Dive into Go Ethereum

[Music] [Applause] all right hello everyone my name is marius and i'm going to talk about go ethereum today i didn't have much time to prepare my slides as you can probably see on the side so i'm just going to wing it and we're going to go a bit into the the code i'm going to talk a bit about what the different modules of of go ethereum do and then i thought we had like uh we can just make a lot of time for questions and discussion and and stuff like that all right um i wanted to show you uh how a transaction travels uh throughout go ethereum and this is the the rough overview we submit to transaction uh of via rpc and it gets inserted into the transaction pool and from there uh we we have the eth package which handles a lot of uh like so so we have the eth protocol which defines the network messages and and stuff like this in the way we interact with the network and from there we take the transactions and we send we use the peer package we're going to enter the peer package and send the transaction to another pier this pier goes the same way back through the east package into the transaction pool all right um i think it's uh most interesting to to actually see it so when you when you send a transaction via via a meter mask to your local node then you will always send a raw transaction you will sign the transaction not in your node but you will send the signed transaction to to go ethereum and it will end up here and from there we so this this is internal eth api has a lot of these these api methods that that you're very familiar with like ethan transaction is raw transaction is get block all of these is are defined in an internal ether api and if we follow this now we submit the transaction um what we do we do some some basic checks before uh inserting it into the transaction pool and [Music] and so we uh yeah we we checked that we have uh enough uh fees and that the fees don't exceed a certain uh amount because like we had a lot of people uh mixing up the the value of the fee and the gas price and so they were sending fee sending transactions with like a really high gas price and then they were complaining uh why why why did i lose so much money and so we have some extra checks there to make sure um uh that is that this doesn't happen and uh yeah then we go to send transaction send transaction is actually in the interface uh from the transaction pool so this is implemented by the transaction pool this interface and we end up [Music] send transaction sent tx something like this no not here how we end up in the api backend sorry uh yeah send transaction is in the in the api back and the api backend is something that we use for a lot of these apis um and it's it's it's a collection of methods that are that are uh useful to have uh for for for creating uh the rpc apis on on top of that and one of them is sent transactions and transaction just adds the transaction as as a local transaction into the transaction pool so if we follow this we come to the transaction pool at locals at transactions so the transaction pool it has all of the transaction of transactions of course and we we sort them and we sort them by by by the gas price and we also have to with 1559 we have like uh a two-dimensional gas price sorry okay a two-dimensional gas price and that is uh as it's a bit more complicated how to how to sort the transactions uh because some of the transactions uh might be valid at a certain base fee some of the transactions are not valid at a certain base fee so we have to continuously update which are the valid transactions because we we built a pending block uh the pending block is just the current block that you have and you pile all of the transactions on top so that you know roughly how the next block will look like with the emergence of of a lot of this mev stuff this pending block is basically useless and we're trying to get rid of it the problem is that the pending block is used everywhere and for example if you if you create a transaction um you you will need the nonce right and um if you already send a transaction uh to the transaction pool that is not uh not confirmed yet uh you will need to know that there's some transactions so you can increase the nonce once more in order not to replace the transaction if you send it send the same transaction uh you actually replace it if you send a different transaction with the same nonce you actually replace the transaction but only if certain conditions are met and the conditions are the the max base fee per gas has to be 10 more than from the from the transaction that you want to replace and the max tip burgers also has to be 10 more and there's some discussion about this if this is actually the the right mechanism but the problem is if we if we don't let like if we don't have this if we don't um if we let someone just update the transaction without increasing the amount that they pay for it uh then people can just spam the transaction pool always always uh resend the same trend uh the same nonce and we have a lot of churn in the transaction pool which is bad yeah so um here we we we're trying to add the transaction to the transaction pool um we we extract the sender so we do the the ec recover on the on the signature for the for the sender address and we add it to the to the transaction pool right oh let's see so uh yeah we we we added to the transaction pool and um here we uh we look if we already have the transaction in the in the transaction pool we we validate the transaction and if it's underpriced we we discard it um and like there's a lot of stuff going on here and in the end we enqueue the transaction [Music] i wanted to go into detail about the transaction execution later on but i can do that no i'm going to do it later so the basic idea is we're adding this transaction into the into the transaction pool and somehow the transaction pool uh has like all of these transactions now we have to if we go back to the go back to the the picture we we started with with our pc we submitted the transaction into the transaction pool now we have to send this transaction to our ps so we have uh in in the east protocol in the ether packet so uh you can always see the the packet that we're in up up here if you don't know and um in in the east packet we have the we have the handler and that handler has the sync transaction function and whenever a new transaction so it periodically asks uh the transaction pool for the pending transactions so uh the pending trend we have um because of someone we have we have different concepts in the transaction pool um pending transactions are the transactions that are actually executable uh we also have gapped transactions so if i send a transaction that has non nine nine thousand and i didn't send 8999 transactions before that then this transaction is kept it's non-skipped um because there's uh there's some like weird there could be some race conditions about the transaction propagation we actually don't want to throw away all of these gap transactions but we want to store a couple of them so if someone then sends like the missing transactions uh we can we can uh validate them and add them to the to the pending transactions so what we do here is we we get the pending transactions and we send the the hashes to a subset of our piece so oh yeah it's a some of the stuff is a bit hard to follow but the idea is we we we shoved the transactions into this into this announced channel and now i have to oops sorry here we go i have to look where these transit transactions are actually ending up [Music] sorry here yeah in the broadcast it's an east protocol eth yeah sometimes the package names and the packages are not really great and yeah so uh we have in the handler we shift the transaction into this queue and we unqueue the transactions at the other at the other end and put them into the into this into this queue and this is a infinite loop that runs and if we have transactions in the queue we get them so we only have the transaction hashes in the queue yes uh we actually get that we actually fetch the transaction from the transaction pool if it's still there if it by the time that we ended up here uh the transaction is not anymore in the transaction pool we just ignore it um but yeah we we added to the to the pending and we send the pool transaction hashes and here we go back where we started and this actually sends the transaction hashes to our peers so um what we do is uh we announce like there's um there's two different ways that we send transactions uh we we either send um so to us to a subset of our peers we send it the full transactions and to all the all peers we send the transaction hashes and uh if someone receives a transaction hash that they're curious and they can fetch it from our node but we don't want like if we were to send our transactions to everyone uh there would be ex extremely uh extreme uh like transaction churn and uh a lot of bandwidth uh that we don't really need um because you would get all the transactions from every peer at all times and so we don't want to do this yeah so now we go to the to the p2p packet and this just writes that writes the message onto onto the wire and um yeah so we sorry um we managed to to push the transaction to the pier now we have to go to the other side and actually receive the transaction and uh this is done in the in the transaction fetcher the transaction fetcher [Music] initiated here and we have some loop yes and um [Music] okay i've never seen this code before [Music] yeah so we oh that's the timeout the wait the notify yes we get a notification of some uh someone sending us transactions and so we so we check whether we we already got the transaction we have the announcement from someone else if not uh we actually um [Music] ask them to to send the transaction so we schedule the fetches and for each peer we ask them [Music] for the transaction and if they if they have the transaction they send it to us and if and if we get the transaction we actually call the add transaction function no sorry add transaction this one okay no it's this sorry um and uh yeah so this in queues the transaction that transaction into the transaction pool so this add transaction function is actually puts puts them back into the pool and so we end up in the transaction pool again we are at remotes so we have we also have this this concept in the transaction pool we have the concept of remote transactions those are those that we get from our peers and we have the concept of local transactions local transactions are those that we get over the rpc uh we have to handle like we want to handle them uh differently uh because we don't want to drop our own transactions so we always say if you send the transactions if you send a transaction over our pc this transaction will never get dropped uh if a transaction from the from the wire is replaced by some something that is that is that pays more uh then we can drop this transaction and so we have um the this notion of local in the transaction sorry and yeah so that's roughly how transaction propagation works and so the next interesting part is how do we create a block um [Music] so the miner any questions to the to the transaction propagation yes so how is it working how is it gossiping the different section further to other peers when one when i'm receiving a transaction i'm recognizing i didn't saw this transaction yet so i'm assuming like then i'm just also doing the same thing and forwarding the transition that appears you're forwarding to the square root of of your peers and then you're forwarding the hash to all the other piece and if you if you receive a hash um then and you and you have never seen this this transaction hash before uh then you ask the the peer that sends you this hash for the transaction am i forwarding all the transactions so i i have to i have to repeat the question uh so so the question was if um we're forwarding all the transactions or if we're or if we're well validating them first and then forwarding and uh the answer is we're validating all the transactions before forwarding so our transaction pool only has valid-looking transactions so valid looking means the sender has enough money to cover the transaction cost and the nonce is roughly what we what we expect there might be some other conditions but i i i don't know them i'm quickly going to close the door because there's some some sound uh coming from there um all right let's get to uh to creating a blog uh so now we have we have a lot of transactions uh in our transaction pool we we are minor we want to see the block how does this happen for that we have to go to the to the minor package and um the work yes so in the minor package um there's a lot of stuff going on the miner is extremely complicated so so i already made made sure that we're in the in the right vicinity we somehow end up in the commit work function so uh like the the miner starts a loop and whenever whenever a new block comes in we need to update our our work package so we need to build on top of the the block that we just received and for that what we do is we we prepare the work that generates yeah sorry no um all right uh where were we no we were in the worker yes coming to work um we uh we have to prepare the work we have to um f12 yes prepare work all right uh yeah we we we have to um set a couple of fields in the block for example we have to set the the parent hash to the correct hash we have to set the correct time stamp we have to calculate the guess limit per block you can you can shift the gas limit uh by a tiny amount um in in either up or down and so we have to uh we have some configuration the gas ceiling or gas target this is what we what we want the next we what we want to end up with so currently this is set uh by default set to 30 million um all the miners are trying to hit 30 million 30 million but you can also configure this and um yeah so we have to calculate the gas limit we have to set the coin base uh this is where all the transaction fees are ending up in the end um uh if if we have uh if we if we have uh london chain rules then we need to do the base fee stuff um and we have some so we have the consensus engine this can be either eth hash [Music] click or or the new beacon beacon consensus engine and we have to make some environment and so once we once we prepare the block we actually need to fill it so we need to apply the transactions sorry so what we do is we fetch all the transactions all the pending transactions from the transaction pool so pending just takes the transactions that are actually executable and then we first take the local transactions order them and then we take the remote transactions and and order them and if if and and then we then we commit the transaction to the um to the pen pending block to the to the pending work package that we're trying to build and so what that does is like we have to set the gas limit and and stuff like this um we we we calculate the sender again uh we prepare the state and um so we have to we have the state database and we have to set some fields in there so that we can later on query them during the execution of the evm and then we call commit transaction and commit transaction actually does apply transaction and with apply transaction we go to the core package so the core package is responsible for most of the core stuff the dx pool lives there right now which is not great we should move that to its own package um but yeah so the like all of the all of the interesting um like evm stuff and and is is in the core package um so we're in the state processor now we're creating a new evm block context a new evm and then we call apply transactions um [Music] apply transaction does apply message and apply message called this new statetransition.transactiondatabase and this is where we actually do the execution or we start the execution of the evm so we we start the execution of the evm by doing the pre-check the pre-check has stuff like get nones uh we very we verify the nouns we verify that um uh oh that's the yeah if it's not fake yes uh we we verify that we actually have code there no no we we verify that the sender is in eoa this is a pretty new change previously it was theoretically possible to send a transaction from from an address that had code deployed it's not really feasible to do that because you have to find a collision between a code hash and an and an address that you know the um the private key for um but if you if you set code to an address that you know the private key for in the genesis block which is possible then you could end up at some weird state so we don't want to have we don't want to have transactions originating from from from a smart contract then we have to verify the base fee and then we have to buy our gas and buy gas just calculates how much how much gas we are at the the maximum amount of gas that we are going to use in the evm execution later on all right so we we did the pre-checks we have a lot of code here for for tracing so um you can you can trace the transaction you can trace each up code you can see which which operations are executed what's the what's the outcome and stuff so if you see config debug and something with tracer you can just ignore it it's not not really interesting and then we check the intrinsic gas we check whether whether the sender has enough money and we if we post berlin we prepare the access lists um so in in in the berlin hard fork we we changed the rules a bit uh so that you can provide an optional uh access lists and calls to this uh to to the contracts that are in the access lists uh are cheaper um you can like preform a a a a contract and we decided that calls to the precompiled calls to the sender itself and the recipient should also be warm cost so so the idea was we want to increase we want to increase the cost for calling a a cold address that we have to fetch from from from the state database because that's a pretty expensive operation and uh but that would break some contracts um that have some like hard coded uh limits for for for like interacting with another contract and so uh this access lists uh to unbreak those changes and so in the future we can um that they're optional right now we could make the mandatory uh mandatory access list is like the concept of mandatory access list is also something that comes up in stateless um and if you if you if you're doing stateless you you will provide the pre-state as an kind of access list execute the transaction and then also provide the post state so that everyone can verify uh without having to have the state um [Music] all right and then we check if this is a contract creation contract creations uh if you send a send a transaction to to no address to the zero address with code then the code gets executed and we call the create we call the create operation and um otherwise we we call the code at um at the address that we sent to the way that we're doing the execution to um are you more interested in create or call create call okay let's do the call then um all right in call um we're first checking if uh so so uh when you call a contract you can add attach a value to that uh so we're checking whether the node the the the address that center transaction originator has enough money to cover that cost then we take a snapshot of the of the state database and [Music] then which had checked the precompiled and uh there was uh so that there's some there's some issue here i don't know how how many people uh actually know this um but back uh back in the day you could self-destruct contracts that like there was a hard fork after the shanghai attacks that just distracted all the all the addresses without code without storage and the problem was that vitalik wrote a script script to to do all of this but unfortunately and and without a balance unfortunately one of the pre-compiles also didn't have like it doesn't have code because it's it's a pre-compile it also doesn't have storage um and it also didn't have a balance and so he successfully destroyed the pre-compile and so this this this clause actually uh checks that that we don't hit that all right uh then we're doing the transfer we're just sending sending the money and if the code that we're sending the money to or that we're calling so we first send the money and then we we're starting the code that is on there if it if it's a precompile we run the pre-compiled contract so uh the pr we in in core vm we have the pre-com piles uh yes yeah in the contracts here right so this is like the the different precompiled for the different uh rules so for example in in [Music] between by in in in istanbul a new precompile was added the the black to f precompiled so it's like that there's different uh uh different pre-compiles depending on the chain roots that we have and um these pre-compiles have a run function and a required gas function um so this like if you want to implement a pre-compile you have to you have to like fulfill this interface and for example the ec recover pre-compile for the required gas it's it's it's a fixed amount it's always 3000 and if you run it will validate the signature values make make sure that we're not inputting something really weird and then does the the easy recover of the of the public key and returns the public key of that of that signature and uh so um this is like a pretty pretty nice uh pretty nice file there's there's a lot of red like really cool things there and i like this code a lot uh other than like some of the other code that i showed previously and um yeah if it's not a if it's uh we're in the call again uh if it's not a if it's not a precompile we um we get the code of the of the of the address that we just called um if there's no code we're we're done but really done but uh we're finished for now um if we have call if we have a code at that address we have to trigger that code and so we set the call code you have three different ways of of triggering code you have to call call code some other stuff aesthetic call [Music] and uh yeah and then you call run on the contract and run [Music] we we fetch basically we fetch in a for loop uh we've hatched the next operation [Music] and then we look into the jump table to see the program that we have like program is like what the op code actually does and then we execute it so uh how does an op code look like in the evm uh you can see i don't know we're going to further learn an instruction set are we going for the merge instruction set you're probably all interested in the um and like you okay it doesn't do that much berlin istanbul okay here here we have some instruction sets where some instructions are actually set um so in constantinople we had a few new op codes we have the shift left the shift right the shift arithmetic right x code hash create two and they all have a a a stack requirement a gas function and they have like they have a constant gas and a dynamic gas if if there's like some dynamic gas so for for some of the operations um like add or sub you like static gas makes sense like you're adding two 256 bit uh elements together um like there's not much that can like grow grow out of bonds but for some operations you want you want dynamic gas depending of the amount of work that you actually have to perform and so for example for create you want to have you want to spend gas as much um uh guess that's proportionally to to the length of uh the uh the code that you're going to to deploy and we can now look into one of these operations so these are these are functions actually that are that are in there and we can look at one of these functions so for example the create function uh take some stuff from the stack it uses some gas and then it calls the create 2 which just creates an address and then calls uh create with that and yeah if if we were to if if we started with the create route we would have ended up right here but we wanted we wanted to look at call instead so yeah so um that's roughly how the how the how the evm works uh you have these these uh these operations uh maybe something easy op-ed yeah so this is like an easier easier easier function in the in the in the evm you just take two values from the stack add them together and that's it and yeah so did precompiled yes and we we are back in call um if something happened so if the like if there's some issue if the if the contract reverts um in in the call then we have to revert the transactions and so if you remember we up here we took a snapshot before we apply the transactions and if if it reverted then we have to re like we have to reload this this snapshot and throw away all the modifications that we made to the state database all right and so after [Music] after after we executed the trans the transaction we know how much uh we know how much gas there was we we we have to we have some refunds so for example if you clear storage or if you call the the self-destruct opcode you're getting some some gas refunded and so we have to refund that and we also have to pay the coinbase so uh this is this is this line is where the transaction uh where the transaction fee actually goes to the coinbase and then we return the transaction result back to apply message and apply message and now we [Music] here we have we have the result and before bicentium we calculated an intermediate state route after every every transaction and this is a lot of work so after byzantium we decided to not do this anymore and then we also have the receipt uh where we set uh if the transaction failed we we're going to set the failed in the receipt i'll understand this and we also set the logs and the bloom so the bloom is a bloom filter um where all the locks are in so the the locks is is everything like we have the return op code uh that you can use to right no not the return what's how do you emit an event whatever like if you emit an event in your smart contract then it will end up in the logs we want to filter these logs so for example you you are interested in all the events um that originated from your contract um that said i don't know successful or whatever so you have to send successful event um you you have those are in the logs and now you want to you want to quickly filter through all the blocks and this is where the the bloom filter comes in so uh we put all the logs into this bloom filter um what a plume filter is i know probably half of half of the room knows that i don't i don't know if i should if i should start going into this but basically you you take some piece of data you hash it and you you set some uh some bits in this in this bloom filter and later on uh you can [Music] you can hash it again and see if if these bits are set and if these bits are set then you most probably uh had these had this event uh originate from um you you did apply this event to the bloom filter beforehand um there's a chance that that you didn't and there's some hash collision but it it makes it very easy to query for four logs and uh for to to see if like something like if these events uh if these logs actually are in the blue filter and um sorry um it's two and four thousand bytes something like this to add 20 48 um now it's it's it's fixed it's fixed length um it's not really valuable anymore and we would also like to get rid of it but it was valuable at some point because you could you can you you could just go through all the blocks look at the bloom filter and quickly decide okay this block has no none of none of my my stuff in it so i'm just going to ignore it and look look at the next block if you [Music] actually find something in the bloom filter that looks like what you what you were looking for then you need to then you need to go into the blog look at the receipts and see if that was actually what you wanted and yeah that's basically it for apply message yeah that we just return the locks from the receipts and uh we have some error conditions where we're back in the back in the worker where we committed the transaction to the pending okay where we committed the transaction to the pending block and um now we have we have built a plan pending block we we took the we we first first of all we set the fields like the parent hash and timestamp and stuff like that then we went through all the transactions applied them one by one and now we have a a valid block and now we need to actually finalize it create the proof of work but i'm unfortunately out of time so we're going to do that some other time thank you very much [Applause] [Music] [Applause] [Music] you