J2 AG 9 Peter Pratscher Analyzing the Ethereum Blockchain with etherchain org 1

hello my name is Peter I'm the main developer of Aoife chain the first at you if you're in blockchain Explorer and today I would like to make a tour around the affair in blockchain about how a blockchain Explorer works lining out some key statistics of the varying and later talk a bit about the scaling issues we have seen during the first month this year so yeah so what is e for chain if a gym is currently the second largest blockchain Explorer we serve around 500,000 users per month with 1.3 million page hits we launch the first let's say alpha version of the Explorer in 2014 actually based on the avaricum proof-of-concept 3 version so it was quite early in in the development of the software itself and this makes it also one of the oldest projects still active it was developed without any external funding it started out as a hobby project and then matured several times and this at the moment can add free the first version looked like this it was a really simple website which already featured some of the key concepts of a blockchain Explorer so you were able to view the blocks transactions also contracts and we even featured a compiler for the low-level the least black language sorry this was built on proof-of-concept 3 which was a very raw version of ethereum and at that time only one client was available to C++ client and this and also a generic API to query this client was not yet existing when the main had launched we also launched a new version or a new iteration of the Explorer which already featured several key indicators on the main page showed Sarah the recent blocks and transactions you see this picture was taken roughly two and a half years ago you see that especially from the price and also from the network difficulty which were a bit different compared to today and a few months ago we released the third iteration of our interface that modernized everything a bit we made extensive use of authentic ins to identify accounts but kept the main principles the same so how does the blockchain Explorer work and in fact it's it quite simple we use an ethereal note here we use para tip because we found out it works better for our use case and each if areum note provides a generic API to query its data to chase an RPC API we use this API to query the data process it and then save it in a structured way into an SQL database which we then use as a basis for our web application to show blocks transactions statistics and provides several tools as well so which which are the main challenges one faces when trying to build a blockchain Explorer for the etherium Network first of all accessing the so-called contract internal transactions is not really trivial while it's easy to get information about the standard transactions and some easier from A to B it's much more difficult if a transaction invokes a contract and this contract then initiates one or several value transfers those events are not provided by the standard RPC API and need to be extracted weird a special tracing API which also now both clients provide by default the same goes for decoding years C 20 or any kind of year C transactions those are also not part of the of the generic JSON RPC API and you need to decode them manually from the transaction received actually right now even completing a full node sync with the station tracing functionality and archiving functionality enabled is quite a heavy task it can take up to a week to sync it up we we managed it to do it with some tricks in three to four days additionally it's the block size block chain is already very large so there is a huge amount of data need to process not quite in the big data level but still not small which means currently or databases around 600 gigabytes of data and you need to really take care how you access this data so each SQL query needs to be carefully optimized in order to ensure that you provide a usable and use a satisfying response time and after that the last point is also keeping up with demand the ax terraeum ecosystem experienced the trend Emmas growth during the last year let's say so we had to scale our system from like 23 uses to 23,000 users per month to now at around half a million users per month the benefit of having all the whole blockchain in an SQL data is it's quite easy to generate or retrieve relevant statistics of this data so currently over 5 million blocks have been mined and this totals to around over 2 sextillion hashes which means over 260 Lian's iterations of the if hash algorithm over 155 million transactions were processed and over 26 26 million accounts or unique addresses were generated over active in the on the blockchain this means out of this 26 million 22 were regular accounts and the remaining 4 million are all smart contracts so we have around currently 4 smart sorry 4 million smart contracts on the blockchain and out of these 4 millions we have currently twenty thousand years e tokens let's talk about the user statistics so how does the typically if you're immune or who is the typically very Musa I see from from the audience here this matches quite well so most of the people that deal or interact with the Feria are between 25 and 33 years old and 2a around 95% they are male and with main interest in finance and investment services where these uses located we get to study we get a very good statistic from our sister site even notes which provides Network related statistics of the affarin Network so how many gap for parity notes are on the network where they hosted and how it's also the version distribution of these nodes here we see that around 30% are hosted in the US after that we have around 15% in China so there's a huge interest and this also increased a lot during the last year after that Germany then Russia Canada United Kingdom and we see from the distribution of client versions the gray bar here is gap and the green bar is parity so around 60 65 % of all nodes or gas-based if you look at the history of the daily number of transactions processed by the IAM network we see that for the first few years it was quite in a very low range and experienced the more or less exponential growth during end of last year especially with the release of cryptic errors which really put a stress on on the network and also with the recent price increase this U is also always visible in the numbers of transactions so now right now the storm seems to have calmed a bit so transaction levels have also gone down if you look at the daily variant blocks so how many blocks were mined on the network we see that we can see some key events of the history of the blockchain in the beginning on the on the far left of the chart which is not really visible you see a huge spike of blocks behind this was let's say the day the Genesis block was mined where the difficulty was quite low compared to the hash rate in the network therefore a lot of blocks where mind afterwards we see a sharp decrease just a few weeks after which was attributed which can be attributed to her back in the mining code of Gath where each block that was mined had a time stream exactly one second after the previous block this means every miner running this this buggy version caused the difficulty to increase and the block rate to decrease with the first theorem hard fork which brought if area map eater tool to to be production-ready we saw also a block time decrease so actually the block time was decreased from 18 to 19 seconds to 14 seconds later on in mid of 2016 we saw we see her let's say a fluctuation up and down this is a result of the so-called Shanghai attacks where the theorem network was put to under it heated under under dos attack which caused the block rate to fluctuate a lot then afterwards the block time decrease let's say stepwise which was an effect of the built in difficulty bump that caused that was included to at some point force and moved the proof of stake by making it so very difficult to which one artists okay which was included - at some point for Sapru moved to proof of stake by effectively making it very hard to mine new blocks and we see the stepwise decrease in the rate of blocks and which was this and the palm was disabled during the last during the last hartford which brought back the block interval to normal levels and at the end of the chart we see a high fluctuation again which is a result of the scaling issues experienced during the last month which I will be talking it later a bit in terms of network hash rate there was only since the launch of the network only one direction and this direction was up so independent of let's say price and events in the ecosystem the hash I just increased and today I had to update the chart from last month and currently we are at around 255 tera hashes per second which corresponds in total to over eight point five million chippy use mining actively on the etherium network and month ago it was only seven million so within a banana within one month over one and a half million GPUs were added so how how does this if how did the mining profitability looked like during the history of the network at the beginning because of the low Genesis block difficulty profitability was very high what you see here is the daily return of a mining farm with one gigahertz per second this is not a small mining farm but also not exactly big one and you see that the daily income fluctuates between around three three hundred US dollar and thirty nine US dollar this is usually attributed to an increase in price hash rate follows price so we see that when there is a huge increase in pricing it takes some time for the Hershiser to catch up and during this time binding is very profitable when the price is declining or stable usually the hash rate does not increase what we saw from the previous pictures of the profitability is decreasing again and right now we're approaching already again all-time low levels in regards to mining profitability let's talk a bit about decentralization currently we have around 40 to 50 unique - which means either solar miners or mining pools that are mining on the etherium network at the beginning it is there was around ten times more unique - on the network which was at that time it was much easier to mine Imperium from a solo as a solo miner and also during the beginning also mining pools were not existent so it took some time for the first money pools to appear and this had an impact on the number of unique - on the network in general miner mining decentralization for the Ihram network is not that good over three mining pools control over 60% of the varium network hash rate as of now let's talk a bit about scalability issues between December 2017 and January 2018 the network was nearly - a hundred percent utilized this was again caused by the price increase by the release of and also for by the adoption of a large number of new tabs so how is the block or let's say how is the block utilization defined so for e theorem the protocol limits how many transactions can be included in a block this is done by the mechanism of gas every transaction consumes a certain amount of gas the simplest form transfer ether from A to B costs around twenty five thousand guests smart contracts execution can require much more and then there is a limit on how much gas can be included in a block the current block as limit is at 8 million this means that around that only around 380 transactions can fit into a single block this causes that say that makes an upper bound on the limit on the amount of transactions that can be processed but if here am network which is at 25 transactions per second but the blood limit is not fixed the - in fact vote on the block limit and the block gas limit and in fact it was also changed several times during the history of the network at the beginning the block as limit was set very low to 5,000 this was the so-called thawing period which was like a burn in face for the etherium network at the beginning it was 10 raised to 3 million then later on to 4.9 million and then we see a really sharp drop down this was also result of the Shanghai attacks that caused where the block as limit was artificially decreased by the - to protect the network against this attacks because by reducing the block as limit not that many attacking transactions could be included in a block and that made the attacking blocks less heavy to process by by the clients after the reasons for the attacks were mitigated the block cast limit was increased again to 4 million then later on to around 6.8 and it's at 8 million actually we can also see from the data from the block Jim how individual miners are voting on the block has limit by comparing the gas limit set on the current block to the previous block we can see if the vote the block has limit up or if the voter bloc has limit down currently we see that most miners target now a block has limit of 8 million some of them are still targeting a lower limit so in fact what we saw also is that every transaction transaction and minor includes increases also the chance of a block becoming an orphan but compared to other block chains - also get a reward for orphaned blocks as they can be included in a so called as a so-called anklet block but still mining a large percentage of anchor blocks reduces the mining profitability because anchor blocks do not get the same rewards compared to regular blocks furthermore angry blocks contain no transactions and but still counts towards to the network difficulty so in fact the more transactions a miner including a block the higher for him is the chance for the block to become an anklet we thought this a very extreme case during the Shanghai attacks when certain minor decided to not include any transactions at all in the block if every miner did this at that time the network would have more or less stopped blocks would still have been generated but no transactions could then be processed if we correlate the anchor rate to the transactions included in in blocks we see a really good correlation so the higher the anchor rate the more transactions are included in the block and they hired an anchor rate of the network is unlike even day this data was gathered during the time when the network was nearly 200% utilized so which means as the etherium network approached 100% utilization the anchor rate increased to 30 percent which means 30 percent of in fact all mined blocks were orphaned and as the anchor blocks count now after the last heart fork to the network difficulty this in fact caused reduction in total mind blocks and again less capacity available to process transactions so we see this in the next chart where we have two the number of mind blocks here and the ankle right here so during a period with a high ankle rate also less blocks are mind which causes again less space available for processing transactions on the blockchain fortunately we did not hit let's say a feedback loop this is that this situation get worse and worse and worse and worse setting 8 million gasps block already consumes a lot of computing resources on regular consumer hardware parity is or at least we cut it unable to process more than 2 to 3 million guests per second when syncing and you note it takes more than 30 minutes to process then a single day of this heavy 8 million guests blocks as you want think up your your note from the beginning you have to plan quite some time to think to chain if now - would increase the guest limit this will further to turret the situation more heavy blocks will be generated they will be more heavier to to process and also very high get they might open up the network again - Tito set to dos attacks and from my point of view if areum should be in general still usable on consumer aware and you should not have to resort to high grade sir high-end server hardware just to run a full node so which solutions are there so the short term for sure efficiency improvements in the implementation for example turbo keV rx a is doing a great job and is optimizing gift to be able to process transactions fast blocks faster also what we what we think can help is a relay network among miners to be able to distribute blocks and transactions faster to reduce further the unclear eight any change in the client implementation needs to be really adopted fast especially by the miners and also by the users and what always helps is a good guidance from the client developers how to run the software in an optimal way for example currently most clients are let's say have same defaults where they should work on any kind of hardware in some some kind of way it would be good to have like an indicator which parameters you can to keep your current state of hardware but with these improvements we might be able to double the gas limit so in total we will be able to process maybe 3 million 4 million transactions per day but at some point any current improvement will not bring a big let's say improvement in general so in the long term we have to look at different scaling options like sharding plasma chains or payment channels just some tips when when the network is highly congested it's always a good idea to consult the so called gas price Oracle for the most suitable transaction fee if a chain provides an Oracle but there is also I think one built in in the coming version of my crypto and also there's if gas station which was mentioned in the top 4 furthermore if a transaction it's always good to remember that if a transaction gets stucked because you set a low to low gas fee you have the possibility to resend the same transactions with a higher gas price by defining by using the same so-called nonce value this is a good way let's say if one of your transactions is stuck to get it unstuck and send it again so last point messages on the blockchain the theorem technology contains let's say several metals how traditional messages or text messages can be included those messages were included let's enter for during the first forty thousand blocks it's still not possible where it was still not possible to send any transaction so actually in the first block or one of the first blocks because actually the first different block had also eight aunty blocks with a number one and this message was included in one of those anti blocks it was Yeats Randall with ether ninja at block eighteen somebody included that bitcoin is the blockchain which is now here forever then we see some German messages sinless inventor home or keep in bacon heart and keep on moving then the first marriage proposal was at block 9852 by mr. Ricky and the first advertisement on the etherium chain was at block 26,000 for that day if areum dot code UK address is for sale furthermore messages can also be included in the transaction input field so everybody can include arbitrary data in the transaction input field and here we find lots of messages distributed on the chain somebody proclaimed that Franco is freedom we have some a quote from Albert Pike somebody wanted his money back the full hackers manifesto is on the blockchain Lim F created his mom and the last and maybe one of the let's say well-known messages that what is included that way was the message partner Hauser which means go back home which caused all gift notes to crash and started the so-called Shanghai attacks so what we have planned for the future on if a chain what we would like to do is so called solidity function fingerprinting which allows us to decode transaction without or to show the function call executed by a transaction without having it the actual source code available we plan to release an ear see token Explorer dedicated browser for the ENS theorem named system and the Chasen RPC proxy api which we can which you can use to broadcast your transactions furthermore we have two open source projects we maintain so if you need to run your own blockchain Explorer we provide the so called Aoife chain light which is available on github which is a browser dedicated for test nets and your private varium networks and there is also a dedicated UC 20 explorer which can be used to create an explorer for a token token that was developed so that's the end of my talk if you have any questions I'd be happy to answer them thank you right now it's we don't have any real plans to monetize it the operational cost is not that high at the moment so for us currently there is no need to to put any kind of monitoring monetization efforts on it the token Explorer will we already show all token balances on the site so you will see in addition all the transactions that were executed plus the standard functions of the smart contracts that were behind them so you have the approve function and the transfer two function calls things like that and we plan to adopt it also to new token standards that are being developed right now I mean we have this to open-source project that should fit in in this category so if you want to if you develop an ear see talk and you want a dedicated Explorer for that you can use our UC 20 Explorer which is already released on github and if you need an explorer 4 for any theorem network your private test net or you can also hook it up to the to the main net you have access to e 13 light which does exactly which replicate let's say 90 percent of all the functionality the full version provides and the benefit is for if a chain light for example you need no back end software if a chain light connects directly to a parity node and retrieves all the relevant information from the Explorer directly from the node so it's quite easy to set up and easy to integrate it works also it works with both nodes which works with Jeff and with parity but with the main we developed it mainly for for parity wait micro is need for the streaming what about the part about the parson to Postgres because even if you run either chain light then probably the performance of some of that would be terrible yes if you run it on the main net it will have uncertain accounts it will have problems when certain accounts that have a lot of transaction history they make problems but besides that if you if you don't really navigate now to the poll onyx account or to the beatrix account you won't see any issues with with the light version did that answer your question or sorry I actually meant if there is a possibility for ether chain light for you to be able to parse that information into Postgres as well or is it only on demand and that then takes a lot of time you mean to use Postgres as let's say storage layer for efficient light or something in this or yes so the way that you showed that you actually use for your site now let's say we have no plans to open source of the full code right now we we might be thinking about that in the future but it's quite some let's say a lot of work has been put into that and it's quite heavy to maintain for us in addition the full version to this to open source projects and it's also not really a trivial way to run it Explorer with efficient light we provide a really an easy way to get 90% of the functionality of the of the whole site at reasonable performance let's say I was wondering can you describe your DB setup because you said that you have 600 gigs of data what you what you mean exactly with tip is set up which teepee you use we use post cray SQL we host it on Amazon and we we focus mainly on optimizing data storage and having optimized queries to to process or aggregate data in a fast and efficient way will either chain light possibly use some of those the tracing for internal transactions yes it is full tracing integration it has full source code upload integration so it will actually decode solidity function calls so it has all these features covered and what test nibs are supported by default against all that are supported by the clients so whatever plot a test net parity or whatever test net gets supports its supported by fetching light and I think it's also included in the PAP F test framework or development framework by default okay thank you very much