Integrating Ethereum at Scale Panel by Eric Scrivner Jake Craige, & Pete Kim (Devcon4)

then I'm the CTO at bit go and so Big O is like an exchange API that many exchanges like this then call bit and a bit used for their wallets so we you know you do scalability and security oh yeah my name is Jake Craig also work at coinbase senior software engineer on the the crypto payments engineering team so we're the team that sort of runs all the blockchain integrations at coinbase if you do sends or receives hot wallet cold storage all that kind of stuff goes through our systems yep I've been working in for business all right on the Kuna cool and I've probably introduced myself as well so I'm Eric Scribner and I'm also on the crypto payments team with Jake and coinbase and did a pretty good job summarizing what we do their stuff I'll leave it at that and all right so we'll start with the we'll start with some softball questions here all right so this one is one of my favorite ones I think this should be asked more to theorem and DEF CON events in general so if you were an EVM op code which Evo mob code would you be and why well I'm on stage in 2018 so I think I have to be the PC opcode all right all right I can see that oh yeah and the why why yeah what do you enjoy that call because I think it's what okay all right fair enough and Jake which one are you gonna be so good I've got two I think the first one you might be what you're guessing there is a NOC code called coinbase which is a very convenient one but that's not gonna be my choice my choice is gonna be X or which it's just cuz you know all the new cryptography stuff in coin bit coin base in the community coming out and stuff it's just XOR with some powers basically all that at some level and it's just a really exciting thing to play with and it's this very simple thing but it's just so powerful all right know what no one picks self-destructs okay that's good that's yours mine okay so let's kick things off let's move let's move under the medium ones here okay so I really like this first one so what is the what is your biggest challenge integrating with the theory I'm at this moment I'll start with with you okay when huge challenge is a Nantes management in terms of you can have rapidly fluctuating gas prices across the network and if one transaction gets stuck then it slows up all the others and you either have to resend the first one or just wait it out that's a huge challenge yeah I think we see that as well it's a pretty common issue when you have very large wallets so with you know with large wallets and when in aetherium as many of you know the accounts have a nonce and whenever you create a new transaction it has to have a nonce which is monotonically increasing and the way that it was designed that way is because it's a lot easier to track doubles protection in all that and so say you know you you're running in exchange or something like that and you're gonna be sending you know ten transactions for five minutes and this this can be you know 100 transactions you know over 50 minutes and if a transaction number number 4 you put in a wrong gas price then by transaction 50 all your transactions can clear because 4 isn't clear so I think that that's a big that's a big challenge there you know you want to talk about some solutions yeah yeah like you said we've all sort of had this issue and it is one that tends to come up at scale as you're doing just a lot of them back-to-back right if you're an individual wallet you know you send one transaction if it takes a while maybe that's okay or you can really easily sort of replace that it's just sort of replaced by fee mechanism and it's an interesting one because like it sort of depends how you structure your hot wallet as well they're different ways to do it some exchanges you know tend to use like a sort of single or a few addresses for everything so they have a lot of transactions going through that address other ones use you know contracts and stuff or you know lots of them like we like coinbase in our wallet we just basically have just you know hundreds of millions of addresses and so we're always have it all over the place and so when we hit the situation it can be fairly limited in scope but it can affect like a subset of customers and there's not great solutions for it yet other than replacing it which is a solution but it's a more sort of complicated one to manage and especially when you've got a lot of these going on and so this this thing where so the gas prices can spike and sort of delay everything is really challenging this example that I've seen in our systems before is a transaction two minutes apart same gas price one took two minutes to confirm in the electric 13 hours and that's not a really great way to build you know systems interior to build a business on the theorem and so it's a it doesn't just affect us I could just affect a business doing this who doesn't have you know as sophisticated infrastructure to handle and and one of the things that we've sort of talked about and seen come up with a few times is you know this idea that comes from sort of Bitcoin of this child pays for parent style of transaction where it's a similar mechanism of sort of recovering sort of pending ones that are like chained together in some ways like in this case it's chained by like that nonce it's different in Bitcoin but the theory is sort of you you put a new transaction out like on top of all these with a high fee and then that sort of can subsidize the rest of the lower fees and get it back to closer to what the real network fee is so miners will pick it up and look up all of them at once so you just send this like transaction with this really high fee and then all a sudden you know you're maybe you know 50 whatever transactions are all all can get confirmed and you can sort of recover that and have a sort of a better experience for your customers and so that's yeah that's that's an interesting one yeah so let's talk a bit about that so so firstly you know RBF like to replace by fee it works today because it's you know you can do it yourself but one of the larger I think confusions with that you know when you change the Pheo so change what you're signing and therefore the th ash gets changed so you're displaying a TX hash TX ID to the customer of the person receiving the funds then it would be different with CP FP it's you know well-known in Bitcoin land the the common challenges you know the idea is you have ten transactions not confirmed so you put another one transaction 11 and it's got an outsized fee and the outsized fee helps to tell the maya that hey you know if you want to get this outsized fee you you you go to mine all these other ten first um the the challenge for the miner then is how are they going to figure out that this string of ten transactions combined with xi is really worthwhile for them to mine because if they were to do this infinitely like after a thousand and thousand and one transactions then they can run out of memory just waiting for you to come up with another big no that was the thing that's brought up it came up in if you're in maybe a few months ago now when there was this exchange that was sort of incentivizing lots of volume on the network so it was just kind of backing up the network I think this is another sort of big challenge with the theorem is that individual token sales individual just knew you know games or whatever can like significantly impact the network very very quickly and very fast they start the token sale the fees skyrocket now everybody in the networks paying more any app you've built is just now all of a sudden paying more and then now you also as you know anybody sending transactions get them stuck because the the estimate can only update once it has all the new data so you're kind of at this you can't ever predict it necessarily you can have some sort of models and like generally it works and it updates fairly quickly but there is this sort of delay and that was a thing brought up when I was started we talked about in theory I'm a little bit I believe on the parity repo is an issue that has some sort of proposed model but they're specifically talking about sort of like toss vectors and stuff with this how long you track it what's this buffer and is that configurable and how does that actually work so so to summarize here it sounds like it's like volatility of fees is the biggest challenge here and there's no good solution at this point is that the is that the deal there's a solution it is a challenging one I think because like I've been said here the you know transaction that's changing and stuff it's a more complicated thing to like re like match things up if you you know have sort of two pending trends action for the network and one of them will confirm and generally it would be the higher fee one but like I think in theory like the lower one can also confirm so now you have to have these two sort of states and like which transaction is real and then you know one of them will confirm and so it's just more complicated problems employment has become a potential challenge to because it's even it's good potentially ok so now we have a new kind of talks on this bend where maybe miners run out of run out of memory space when they're waiting for all these pending transactions maybe but if if hypothetically that did exist if it was implemented poorly it could lead to this new design pattern where everyone's putting no 1g way or whatever it be as all of their transaction prices and then if it doesn't clear within some X time period then they put out a transaction to kind of clear those out so we have this now almost waves of transactions going through just because people want to pay lower their costs right it's a tricky situation all right ok well it's it's so all three of you seem to have consensus yeah one of the things right I think another one that it's not just us but and we definitely experience this is just running notes very running nose keeping them up-to-date on the size of the backups and stuff in particular these sort of like archive state notes that in a store like the full history I think offeree of you know parity just released a website about like that constantly tracks the size and so if you run like parity you can get it to like 200 something gigabytes but you don't have this sort of full history one and that one tends to be I think around these days 1.8 terabytes or something and so just the size is the thing I'm just transferring that over the internet when you need to like to spin up a new one just takes a lot of time I think ours are ones when you spend them off for like three four hours right now to sort of do that we've got things to get them faster but a fresh thing is like I think the last one I did something like that at which which kind of flows nicely into the next question which is do you do you think that integrating with the theorem is getting harder at that low level that we're talking about here and if so do you think that we're at some sort of critical juncture with respect to that I think for for instance I mean the blockchain size keeps increasing everyday so for that purpose it is getting harder in terms of resyncing nodes or syncing a new node if one crashes or something like that you lose a database in a lot of ways though there's a lot of tooling that makes it a lot easier to integrate things so things like truffle or ganache are becoming really really easy to use and there's a great user experience around dev workflow now for even just for local development but also for production and it's it's this interesting kind of duality so some parts are getting harder in my opinion in some parts of getting easier potentially I think to go to production with a theorem is getting a little bit harder I really like the developer on-ramp so you know the developer tooling and all the you know building on tools are getting better and better welcoming new new new developers and that's why we have so many developers at this conference I think that so far is running a node when I last fired up for myself you know we had lots of nodes in production but sometimes you want to hack yourself I think death if you you know running in the fast fast mode it's actually not too bad and I think it's right now it's not a critical juncture juncture because at this point the the block processing time is still less than the time it takes to propagate and it's time that one you know get stopped and that you know we can talk a bit about more about that like that was different earlier this year but I think that right now it's quick where you can fast think all the way to the hit of the chain and then you can just you know your blocks are getting pushed that's quick enough with you know modern hardware it's not totally a problem and of course you know you really want to be running a node in in production and running several of them so that's a different challenge but to get on board I think it's actually still pretty easy either like onboarding definitely doesn't having a lot there's a lot of really cool stuff there to like get up to speed and like start able to do things but like you said sort of at this sort of like scale thing or when you're running like businesses that are worth a lot of money and a lot of money's moving through you also want to like have sort of the full history yourself and always have a copy of that and so you this kind of initial sync time Wow like I think for a lot of these cases this is not particularly important or like syncing sort of every block into sort of your own database that you have sort of your own full history that you can analyze outside of you know just the sort of latest state like one of the things we have is our data team once you know full copy of the blockchain in you know sequel that they can sort of analyze and do things with and right now so we sort of manually build tools that do that kind of stuff and just takes a while right just because of the pure size do you know how big that is how much data that depends on how you want to index it right so I think that's this project I saw it part of the grant team showcase it was called vulcanized Evie or something like that it was pretty cool you told them what you want to index and they they will run it through and put it into a post Chris I think it's a great idea and it's nice it's lowering the bar so that not everyone has to go in the back so you know you guys probably blew in an exit we have an intensive so everyone's building one but the more it can be open stuff all right I've got it I've got another easier one for you well maybe it's not easy actually it could be easy all the way to hard so so parity again or a lot if you like that one I'm partial to parity I found it's been more reliable and faster sync has been nicer and we can get into the whole parity trace parsing issue we will yeah but we can definitely we can definitely hop into that yeah that's definitely one a bandage parity had early on yeah well that's what I've heard it is with parity what we're sort of ending out here is this idea if you're only terse can you see contracts and heathyr around they call them internal transactions and sort of detecting those and seeing like where money moves is a complicated thing I actually give a talk on this yesterday that it'll be a recording of and stuff about sort of the ins and outs of how that works and stuff but from my understanding I wasn't just happened but parody was one of the first ones to give an API that made that more like accessible and easy to sort of figure out what happened and guess now I believe since 1.8 it was does have an API for this - but it's not a standardized API it'sit's actually a custom one for each one and so there's no standard way to process these and that's setting something that would be really interesting to see in the future yeah it's it's interesting because like there there are lots of transactions that happen that appear nowhere in the blockchain right and so I think that's one of the interesting differences between aetherium and a lot of other ones but Ben what about you yeah so the first question I think both parody and get ok random both depending on what the network situation is sometimes some of them a challenge is processing certain types of transactions and blocks and some of them you know then the other one goes faster than so I think running both this is the way to go with I think that this is a huge topic internal transactions for us it's one of the largest pain points so a bit go we use multi-sig wallets these are all the contracts in smart contracts and for us when when we send out from the smart contracts we want the receiver to be able to see the funds on more modern exchanges they do you know record the deposit because they can look at these internal transactions and so a little bit about international sections so in aetherium you have transactions and transactions are what I think are the first we can call it like the first hop in which an account send something to an address where the address is either contract or another account and a challenge is after that if you send it to a contract and tell the contract to do something then internally the contract is is making changes to state and doing internal messages some of which may be sending funds like a theorem to other dresses now the challenge with that is that currently in the web 3 API there is a get transaction which does the first one it shows you all transactions that come you know from accounts to smart contracts but usually what a multi-sig wallet is is that you would send a transaction from one of the sinus to the multi-sig wallet and then a multi-state wallet would then send out the funds from the smart contract and that is a smart contract transaction or internal transaction so there's no you know RPC or API to easily get ahold of those transactions as a result some of some of the exchanges they don't recognize deposits automatically for such or such not contracts in and this I think will be further accelerated because now we're moving towards oh is that unified the unify idea Wallis versal yeah universal login wallets which I think I'm a big believer of from a UX standpoint I remember yeah also smart contracts so we need to solve this problem you know as an industry and make more people actually taking deposits from small contracts because that's what even this is about right it's a small contracts yeah and it's it's a scary thing to Parsa process these two you can have some pretty like dangerous sort of bugs that come up from doing this the wrong way and so there's plenty of exchanges and stuff who don't want to do this yet because it is this sort of scary thing to get wrong whereas like transactions and stuff are fairly straightforward to kind of tell what's going on these aren't so it's a big risk and especially as we move to more and more sort of wallets doing things these are gonna have to be supported and me to make that easier and that kind of flows nicely actually into the next one that I kind of wanted to throw out there in less sorry we're gonna go okay yeah so so alright so I guess the next question would be wallet contract or like scan individual blocks and transactions and invoke transactions which one which one do you prefer yeah it is there's pros and cons right so either you're generating a new address for every deposit you want to take as an exchange and you have to monitor these interventions actions very very carefully and it's theirs then you have to sweep them into your your hot or cold wallets and it's this big process or even as a contract side of things if you build things as a contract I think bit tricks does this and maybe some other ones too things like sweeping gets easier you can monitor events in your smart contract so internal transactions aren't a problem anymore but now you have increased cost for each individual customer who needs to a deposit address but may never deposit any ether in there you have to handle tokens as a completely separate case it definitely complicates things and there is that trade-off one of the things I don't particularly like about wallet contracts especially if it's like a single one is that it sort of it puts all your transaction history in one place privacy is a really important thing and we want to have and so contracts especially the sitting because of the cost like you're going to be incentivized as any sort of business to do sort of one right unless there's some extra cost or something for you know a new address and if you have one now all of your history to that exchange is in one place and so anybody wants to see how much ether you had added how much you sent there they cannot tell that it may be how much you have in that exchange maybe that makes them more of a target if they follow a pattern like things like the trucks as contracts like you can detect them easily and so you can easily see how much of it checks is receiving through that pattern or is like the way that we do it which has some other downsides and cost is that we generate new addresses just all the time almost anytime you get an address from us it's a brand new one that's never been seen before if we send transactions it's coming from a new mattress it's never been seen before ok that has some things of just having a lot of addresses its own problem and I think we might get it up later to about you know sort of batching transactions and stuff and doing a lot of sins you have some interesting challenges there in certain transactions but I think like I don't like this idea of just like linking everything together all of your activity in one place especially if you're gonna like send from there too so I think in terms of having one address client I mean per customer exchange pretty much many exchanges are used to this concept from you know a variety of other digital currencies and so I think on that matter what we're doing is whether or not you're a contract or address you gotta have one one address you know one one address per downline you know customer deposit a dress and so what happens in the end is that when you try and do that with contracts you are you're probably going to be using some kind of forward or contract in which what it is is a really small shim that forwards the money from this unique deposit address for each customer into some some main and you know sometimes a challenge with that is is with with us at least we wanted to be a multi sig platform and so we wanted you know all all our wallets and contracts to be you know not controlled by a single key and that was one of the requirements that we set out to fulfill and that's why we did it that way I think that I mean one one one advantage of that for for us that I guess I haven't talked about is that with with contracts you're you know instead of looking at the internal transactions by yourself by looking at transaction receipts you can look at the events because you can get contracts to file events and and actually you can do it within the 21,000 guests with it you can get one event or something like that badly enough and then you can you can code your indexes to pick up those those events and use them for you know deposits so so there is one advantage there the disadvantage is you know pay you have to pay the gas to to deploy the contract which which can be very costly well I mean I think there are some IPS that are out there not too far away from being minded that will allow us to do you know basically paid to script - in other words you can pay to an address which is a contract address without actually it having been deployed yet and then someone else can then deploy after they see that okay someone actually deposited there and this is really useful in the case of when exchange that has a hundred thousand users and only fifty thousand may use s so you only need to set up the address after they doing there I think about like the gas limit stuff right so this is the thing that different exchanges have to make different decisions on of customers sort of sending aetherium somewhere and if it might go to a smart contract not in the trade-off of like what sort of UI do you present there what sort of gas do you charge your users because your standard aetherium transactions a flat 21,000 you can sort of guarantee that or decent a contracts there's no guarantees necessarily there's things that you can estimate it and it's probably right but it might change to there's some ways it could and like that's an interesting thing in exchange you have to do like us on our sort of brokerage of just corn-based comms you don't see any gas stuff when you send and it's a very intentional decision to keep it simple because it's a very confusing UX sort of design pattern it as a sir people have talked about in the design tracks here and it's this interesting trade-off of like how to handle that and I was like a business how do you set that and how do you sort of expose that to users yeah it's an interesting question for exchanges and I mean coinbase you kind of both but but exchanges and wallets really same UX problem really it's the same UX problem that aetherium and really all the cryptocurrency industries any other we do right now it's just it's probably not just a theorem and but we can't really just sweep it under the rug I mean the the the main issue I think really is that customers expect them to pay you know they're used to paying fees which are percentage or a flat fee and in in a theorem and other chains you're really paying for compute or space and interesting all right so I think Jake's you'd mentioned batching briefly down there you guys want me chat about that of it what do you think about it is it something that the theory needs is there some sort of alternative thing that maybe would also be interesting to have yeah I think I only first brought up we were sort of talking yesterday and I sort of brought this up and I hadn't thought too much about it deeply but like I mentioned we have a lot of addresses and so we have all sorts of customers setting and all sorts of amounts so you know we'll end up with a lot of addresses with very small amounts of ether I will need to so let's say we have a set of addresses that up you know one eath in each and we need to do you know attend etherium sent well we need to sort of combine those into some other address and then after that confirms on chain all you know ten transactions that might take what we need so we need to first send ten wait for them all to confirm then once they confirm we can then send out this tiny three so this we call this like consolidation internally and this is sort of a complicated thing to manage and it's an interesting thing because like Bitcoin and sort of these you TXO chains they have a different way of approaching this that doesn't have this problem and so once you in it one of the reasons they do that is so you can have sort of a new address for every sort of change amount and so you can basically always be using new things whereas like on a theorem it's really hard and it's all about this privacy stuff again where you wouldn't want to do that because if you wanted to do a cent you'd have to do all these like intermediate sentence to finally be able to do it and so I think it'd be interesting to consider some sort of like batching strategy or like multi transaction thing where you could say you know from all these addresses I'll include all the signatures for them do all these transactions at once into some like destination and as sort of one atomic send there because also you probably don't want to do because you could also I guess another way it's sort of consolidating and then do a good thing at any send you could do like ten individual any sense that destination but if it's like a payment service they probably aren't gonna handle that well if they're expecting ten even they get like ten twenty sends they're not gonna know where it's supposed to go to right pretty bad UX if a user goes to send one transaction and they received ten on the other end yeah that's probably confused I think with the theorem actually it's a bit it's pretty cool in different sense that in Bitcoin you know that you have these inspects and you can tell whether each of the spent is economical to spend or not in a theorem you can almost say like okay if you've got like some dust you can just wait because you're not spending part spend you're spending from the account so you know if someone sent you a lot of you know one one ways you could wait for I don't know resilient of them before you actually took it out usually I think you know consolidation is it's a good idea to just do it like when the account went when the network is less loaded so you know when nobody's holding any I SEOs on weekends or something like that when the gas price is pretty low and keep it low so that you don't actually impact the network because you very well could one of the interesting things I want to ask both of you here you know what have you seen in regards to batching out outgoing transactions so you know with with with other with other coins without getting too much into that a common pattern is you wait every five minutes or ten minutes and you batch out a send you know of whoever else is trying to withdraw during that time with the theorem this is largely not been done yet even though it is very possible even today without making any changes whatsoever to consensus at all you know just use a smart contract and do a multiset yeah for me and like some of the conversations we've had about this sort of thing on aetherium and Bitcoin really any chain is this is privacy again because like you now link all your sort of customers together in some ways and we want to generally prefer that like we like to sort of default to privacy and try and maintain that as long as we can and so when sort of batching comes up and that that's always the thing that it like immature like wallet contracts like putting everything in one place and maybe just the theorems the wrong chain for the types of things that that I'm talking about it I think there's ways we can get there but for me I haven't seen it and you can definitely pull it off like you saw the smart contracts and stuff I think that's kind of the cerium for lack of a better word purest answer to like the bathroom question is when you just use a smart contract for that and that's kind of aetherium and for everything which isn't a bad thing but maybe there's trade-offs that you don't want to make there in your in your server-side architecture that we so you're saying earlier if you have a wallet contract I guess it becomes very convenient to do these kinds of multi since but if you don't then there's no like alternative mechanism right yeah you could theoretically design a function calling your contract that would multi send to a list of addresses that you pass it okay yeah and it's all technically the same aetherium address but then it just happen all internal transactions but then like you said they're all linked together and well you also need a gallant need a balance on that account too that's like sort of public and probably associated with you in some way so like you like its offense valued structure well it but like you don't want to probably have all your money in one place maybe but the thing I think essentially lots of I mean lots of exchanges end up with a hot wallet so and and I think what shape-shift it doesn't matter as much because you do like one transaction at a time right so you'd like to send that but maybe one of the reasons why people don't use this as much in aetherium is like I think it theorem people are really used to seeing a transaction where they have a source and a destination and if you look at that like either scan you know it's quite clear it's always got a source at a destination so it's very one-to-one how people like the mental model of how they think about the transaction you know in another change you always have both developers yeah it's also I think a little less necessary in etherium because like we talked about that not stuff earlier in a theorem you can do these sort of serial but like back to back sends like on from a single account right if you just increment the knot so you can continue to like to send from the same account so you can you can queue up all of them at once without any sort of delay or is like on Bitcoin and that style of chain you have to sort of if you have a big sort of input you have to sort of wait for that to do its thing and then do another one or there's other ways you can do it but it's a lot trickier words I get through I'm just like just keep sending from the same account and it'll let you do it and like it should generally work and it doesn't mean too much of a problem we've seen or like any sort of need for batching it either there's something like gas savings potentially yeah I think I think that that like the gas savings um I think that with the theorem maybe we aren't as mature with doing gas optimization you know with Bitcoin for example because things are so expensive you wanted to save every little bit you could so you're paying the same amount for signatures on the inputs and if you can send to ten at once instead of one at a time you do that right but with the theorem I think it's still a little bit earlier with the gas savings now it's mostly like how do we price that way you know get gas price rather than actual guest spent and really for all of these conversations change once like an actual Caspar sharding some scaling solution gets implemented they're all just completely different because we have a whole new set of problems and yes yeah I mean our next aim to measure that would be like with the with the sharded network could be much more complicated I think I think we might switch gears here a little bit so uh we got here I guess I would love to talk about privacy with you guys so how are you thinking about it what do you think the status of privacy and fungibility is in general are we there yet what are some of the interesting prospects and how do you trade that off with scaling in terms of aetherium we're definitely not there yet for privacy okay definitely not there yet but it's like snarks exist now but it's it's really not easy to use unless you have PhD and even then it's it's difficult it's it's definitely tricky and really they are there best implementation should be in some sort of plasma like implementation or in a shard of some kind so it's almost a catch-22 and that which which comes first scaling of the privacy yeah so I think it's not that not by any I mean you can see how easy it is to track you know I think this can they have label like shape-shift one shows you know that's that's the most public think of a thing I've ever seen having said that I mean I don't feel like you can really trust so call us you know pseudonymous nosov bitcoin you can always almost think about it as those that those transactions will be review when you think about privacy it's not just about like privacy today and okay maybe it's a little bit hard to track today but the blockchain is forever right so so you going back going forward like five years later all of that data is gonna be there so you're gonna have to live with whatever you're putting on on that today so I think that's a key you know security is really important and privacy is one of you know one of those those important things for security because it makes it just that much harder to secure something if you have a very big target amongst other reasons so I I just don't see this as something that is being worked on in us yet from a theorem I think that we have scalability challenges and I think they're gonna go after those first because that is what drives adoption I mean whether you like it or not they want to drive adoption yeah I'm really excited for sort of the prospects of like the future of privacy but yeah I don't think aetherium is doing maybe enough to to be concerned about it or to like there's definitely teams in the space working on sort of off chain sort of style solutions which I think could be really interesting but just the general sort of blockchain community as a whole there's a lot of really interesting stuff going on right you know there's all the dizzy cash and such doing all the snarks and then there's you know starts now and you know bullet proofs and all sorts of things happening but I think you brought up really good point that about like this sort of like foreverness this sort of infinite sort of data thing and this has already happened with Manero where you know they had some sort of bug where you could in theory sort of I think in some ways you could I don't know the details but you could effectively like sort of D anonymize some of their transaction history because of some sort of you know vulnerability there and that's sort of already happened and so in a lot of those new cryptography it's also new and you know most of your target for these things like fairly like well-established tested and stuff and so we do these you know security audits but a lot of it's also still backed by this this there's a word for it but this idea of like it hasn't been broken yet and a lot of people have tried we can't prove it's totally secure but we think it's pretty good and so we'll keep using it until then and a lot of the newer stuff Reyes imagine would have like similar properties except it doesn't have like it's not battle tested in that way and so while there's like really smart people doing it in developing this stuff I think that's a really like big sort of risk in general to it but I think it's definitely a worthy goal that we should we should go for as you know businesses and anybody wanting to build on this you know I don't want my whole history to be for everybody to read anytime I send somebody something from any of my you know aetherium wallets they can potentially see my whole transaction history and I could split it up in all sorts of addresses and try and do that but then it's it's like that problems describe barely even like a just individual level not even the exchange level of well that's really complicated to manage it was expensive just few eyes cool alright so so another easier question here but what are you guys most excited about with respect to your theorem it's a good question I would like as a something we would like to see in there or is it something you would like to see some project that you think is interesting or some you know future prospect are you really looking forward to I definitely think that for me it it probably goes back to the whole nonce thing that we kind of started with it's that's a one huge pain point and something some solution to that even at the protocol level would be very nice it's also something that Jake mentioned yesterday when we were chatting was it was there's it's very interesting concept of because we have it's more of a solution to the internal transactions parsing problems that we've had where maybe we could implement events at the protocol level like contract type events at the protocol level and I hope I didn't take the one you were gonna say just now but no no I think yeah yeah yeah it's a really interesting idea like this I like because I think it's it's a good way because like events are this later known density and people know I deal with them give the vic-20 and 7:21 and they're gonna even continue to be used and people have tools and stuff around processing them in really interesting ways and so I think that would be a really nice way that you could you know extend you could bring these sort of internal transactions out of this sort of mysterious world that you need to parse to get them and gets that's one way yeah I think it's a great it's a great idea I mean when you look at it what what people what a lot of people actually want is they want to see when a theorem moves and there are a lot of ways that smart contracts can send that today which don't get shown up with get transaction API so I think events are one of those ways if you can fire those events whatever whenever any eff moves that's that's a big way that that will make it easier for integrators to seize these transfers and credit deposits another way you know could be give them what they really want which is again transactions API which actually gives them the transactions of all F and you know that's easier said than done but you know I I don't even know if it's that hard right like we we've all sort of experienced this parsing of these traces like it's a doable thing I mean it'd be really easy to like write a tool that does this like it's it's yeah so much that it's like difficult it's just that it's just kind of there's all these nuances that don't seem right be really dire augmented and so like various note implications could absolutely implement these api is especially hands I have sort of custom ones but I think would be ideal is like a standard sort of like our PC API that they wouldn't require like protocol levels because I think it's gonna be a lot harder to sort of sell but you could absolutely just have it would be a new kind of RPC API extension to web three or something like that and I think people you know someone that that for us because we care so much about platforms and having developers use platforms you know if they have to go for a talk like yours great talk but if they have to go for a talk in order to learn how to do it that's probably too hot you you want to just give it to them right away just give them the API and it should do anyone more on the topic of things that I'd love to see nodes implement is the I believe it already exists in the EVM as far as I know I think it came with metropolis is the returning data with a revert opcode being able to have actual error codes I can send back in my errors and not have to do extremely tedious solidity debugging for hours to find out my typos I didn't see at this time I'm sure there was standardized the return codes yeah okay yeah I know they implemented in Metropolis the OP code to actually return just arbitrary data that I can pass a string to revert but I haven't heard anything for a standardization yeah I think there's someone working on it I think it's really important it's like kind of like you have HTTP status codes you should have returned off course yeah yeah one other thing I think is interesting that would be be cool to see it doesn't exist but it's is like some more sort of infrastructure or sort of tooling around like running notes and running notes and sort of like a high availability set up right and a lot of the nodes of today's sort of expect you to run them and this single configuration of just you have one thing running on one box and you expect it to never go down and never break and that's just not reality right these every piece of software will break at some point it will go down and so if you're having a lot of sort of transaction stuff going through also you can't just use one like it just wouldn't work at this sort of scale you may see and so you need more than one and then you get into all sorts of issues where you make a request to one node through a load balancer and then you make another one and you get a different result back because it hit a different node that's a different part of the blockchain so you have to sort of deal with this in in like interesting ways whether or not you sort of take data from multiple nodes put it on your own database and sort of handle that inconsistency there or just run one node I suppose as an option there's a lot of choices here but there's not much information on that and I think other block chains have all their like proposed ways of running them and more of this like high availability set up for like if you want to build like you know reliable sort of infrastructure around these to make sure that you know nodes aren't your problem and you can just kind of focus on on the chain I think we need some better tooling there yeah I think that sounds like a great grant project actually I mean think about it right like cuz we have like I think a theorem is really cool we have like the most clients I think of any blockchain probably most different implementations you could almost be like connect to the web three or gasps and parity and be like a load balancer crossbows and and only accept things that go with them agree on or come out yeah we have an injury yeah yeah we have like an interesting set of that kind of does that but not about the agreement thing where we actually sort of chained them together we chained together different node implementations so that if there is some sort of consensus breaking rules it'll sort of stop there and not sort of get to our the rest of our systems because like these things happen and this is a way to sort of mitigate them and so yeah trying to sort of detect consensus and detects like 51% attacks and things like that there's probably a lot of interesting tools that can be produced out of there that I think a lot of exchanges in companies building are aren't doing on their own but there's no sort of shared stuff out there that's actually quite interesting but also risky so when you look at like consensus breaks sometimes what people do is they attack one of the implementations and cause it to you know maybe see a block as invalid when it should be valid and you know the safest way in my opinion is to connect to both of them and then see if there's a substantial difference in the two chain heads by you know a significant amount of work right if you change both of them it's actually good in that you were well kind of good in that you get the most valid one in that both of them will see it as that it but that could be something out there that has more hash power that you don't see as that it so right yeah it's sort of it's sort of like it gives up lightness right in the event of a chain split it so it gives up liveness since instead of like just keep running on and like hoping you're on the right side of the chain it says we're just gonna stop and we're gonna figure out what's going on here and we're gonna investigate it and then attempt to make the right decision there and catch up or whatever it needs to do that's sort of that's adding a great point though that is the sort of trade-off it makes to do something like that we're gonna stop the world and until we figure out what's going on yeah the world weird all right I think I'll do that I'll do this last question and I would love to do some some audience questions if you have them in the last ten minutes there so we'll do an easy one which is so what's you so what's your favorite thing about working with the theorem versus the other cryptocurrency is equally obvious answer smart contracts right there's Stoppers so much more that you can do so many more interesting creative solutions to a lot of the problems that aren't specific to aetherium necessarily things like it's has offered its own share of problems but and and really just a community like there's a huge community around the area and building this out that does make it never a boring day right never a boring day it's it's always interesting there's always something new happening it moves so fast that you have to almost struggle to keep up with it sometimes that's a good problem to have yeah I think that the community in aetherium is this for me one of the strong probably the strongest one it has what the most developers which is a great indication for me as an engineer where it's going to go I think that in general you want to build when you build something you want to see like other people are using it so there is ad option as well and you you feel the most you feel more welcomed building something on aetherium it's like you know here I built this and and and everyone likes to try and see what kind of depth you build and I really you know I really enjoy that yeah I think that's a big part I think yeah it's similar answers for me is yeah I mean that what people can do is smart contracts like I don't really do much work with them myself but to see what people are able to produce especially like all the recent sort of financial instruments and stuff things like like die existing and you know the ability to be able to create some sort of thing that has this stable value without anything actually behind it is this really interesting concept or all the people are doing with you know the the relays and all the variants of you know loans and all that is somehow happening in this world and it's kind of crazy that that happens and you can just very easily go on to some you know some wallet app and instantly you know take out a loan or something and and that kind of works and then the community is yeah also really I think generally healthy community and doesn't have as much of the sort of tribalism you might see across all different block chains there's you know some of it of course as there is anywhere but generally I find that the community is very sort of open and willing to work with other block chains up and try and push sort of everything forward I think like one great example this was the talk yesterday on verifiable delay functions that they're talking about doing in serenity is this the it's this thing that's not specific to aetherium like they're actually bringing in other chains and want to work on this together in collaboration and come up with this just better cryptography and better system to you know have this you know new concept to work for for everyone and not just be this like singular thing for a theory and like it's you know our baby and we'll just you know keep it here and nobody else can have it they're just like let's let's share that with the world and let's you know make it all better because it's we're all it's so early days and all this stuff is a lot to figure out and I don't know who the winner is going to be it maybe we'll all be winners maybe none will be but we want a you know I'll just work together and try and be healthy and get there all right okay so we'll open up to questions if anyone has any all right I'll just hop down here with this microphone hint to you so you guys are on the front lines and you'll have a really good perspective on this I'm curious what you which projects you guys think belong on chain kind of where you draw the line for that and you know as assuming that we improve efficiency and scalability and stuff do you guys think that will change so how ya talk first yes I have I could flick to there sorry I I like on changes because it's sort of this singular place to integrate with you know the title is integrating with etherium and as we've talked about it's already sufficiently complicated and the more things we add on top of that especially layer to things and various protocols it becomes even more complicated so the sort of onboarding cost gets a lot more expensive and it's not as easy unless we build tools that can somehow do this but if you know now some new protocol comes out that you know all of the exchanges all the wallets need to integrate and you know you scale that up you know it's a multiple different ones that's going to be fairly difficult and so keeping it on chain you know has some nice guarantees there just simplifies it overall but clearly that has sort of you know scaling in limits and stuff and there's a lot that can be done off chaining and maybe you know published there as just some proof and I think there's also a lot of value in that if we can have you know really good sort of second layer solutions to to to pull it off yeah so I would give it like I think there's a trick answer to that because you sit on chain I think like side chain right now it's the best because you know with side chains that checkpoint it into the main main that you kind of get a lot of advantages that of being on chaining that it's really easy to integrate because people can see you through transactions and you've got an EVM on the side chain so you can run quite a bit there but it's not going to scale eventually still I mean a single that could still take up you know all the gas on its own side chain again so but for now I think where we are at that's that's that's the best you have and it's gonna get us there 22.0 yeah I kind of see both sides of it like for simplicity doing it all on just the main aetherium chain it's definitely the easiest way to keep it simple and scalability purposes it's definitely nicer to do it on a society of some kind there's I think some kind of balance there I would at the very least I think anything that should be anything that's considerably expensive I mean you can like crypto katie's anything that clogs the network or is sufficiently expensive or has sufficient network should be in some side chain or shard or something off the main chain of course anything more simplistic that maybe either doesn't have traffic or like I'm not sure if it makes sense for like new developers coming in after like 2.0 jobs and stuff like that should jump right into oh yeah we're gonna develop our thing right into a sidechain this is my first my first app is on plasma and it it increases the barrier to entry for aetherium which is one of the things that makes aetherium so great in my opinion so there's a balance there depending on in each individual app alright so I think you your moderator asked a great question earlier about what was your favorite feature of a theorem but I'd like to ask the opposite what is the the biggest pain point that you as scaling if your room developers deal with I know that Morgan from shapeshift you mentioned earlier that dealing with the lack of return codes was a pain point is that the biggest pain point what's the worst part about dealing with a theorem at scale oh so I think there's really two sides of that is there's the solidity side in the smart contract and what's a pain point of dealing with those and that that whole ecosystem and then the server-side architectures and integrating at addresses or do we choose a wall in contract personally having doing both sides yeah era codes is a huge pain point for me it's it's a huge time sink trying to debug issues there when you have essentially two maybe three error codes that all basically say the same thing it's definitely a pain point I think it's the internal transactions challenge where other people are not viewing them easily for me because that you don't have much control of yourself you can yeah no matter how much time you spend yourself you know gonna get someone else to accept a transaction or is diecuts yeah I think there's a few I think like that's a huge one that's like prevents people from sort of getting on board and getting there I think there's like there's things we can do that will make that make that easier and these are things that like eventually once you get them kind of working correctly well they do become not so much of a pain point but they're hard to get right the first time things that we see like you know regularly well I'm gonna say two things one thing we see regularly not much recently it's everything's kind of died down but all the gas price stuff like really this year all the time gas prices spiking things get stuck and and that's a pretty big pain point and we don't currently have systems on aetherium that sort of automatically recovers that and we talked a lot of some ideas earlier that could help with that um the other one that's more of a rarity but also similarly for like building these sort of high-volume systems if you have to start tracking these nonces and stuff yourself if you get one out of order you're in a really weird spot because if you're doing like multiple sends for customers there and you like skip one now you have so if you have like four five six and you don't do five for some reason it'll hold you've got this whole and so the other ones just gonna be pending forever until you fill that in and like what if you like I don't know we tried something else and then you send another transaction out you can't cancel this six now like it's out there and so you have this weird like like we can't introduce like canceling transactions that would we eat like not really possible but that sort of thing in like calculating these nonces like you can't because you're not just using a single node you have to sort of track that yourself and make the right decision there and so that's a challenge to like do right and then when you get it wrong specifically going too high going too low the node will reject usually it's just like you know it's sink that but going too high creates this thing where it's like oh yeah now what and you don't know how long it's gonna last it's in the nymphal the other thing once where you know we filled in the gap in it went out we didn't expect it to and it's like oh well it was okay but not that a fun thing all right any other questions out here all right we got one way at the back gonna run I know so a better standardization for like your server architecture or or smart contracts themselves air close where do you guys feel the governance process is for that are you guys talking one-to-one to each other or is that on the IPS or it's kind of like how do you have those conversations together in the industry to get stuff done yeah I'll see your question one thing real quick on that like this that's one of the reasons why we wanted to put together this panel is to get people talking and people talking in public I think there's a lot of sort of secrecy a lot of things are shrouded and they shouldn't be and there's no real reason for them to be but I think we all approach things different ways like if we were to talk about how we all run our nose or how we all did not stuff I imagine we'd all sort of do it differently and there's things to learn from each other there like we've already learned things talking about you know the contracts like sprint trade-offs there and the way they do things in the way we do things and I think the big thing is yeah at least get people talking I don't necessarily have in a year I think the best place to do that there's definitely you know something you know just every now and then you talk with somebody or every child but I did think it'd be really cool to see more sort of open spaces to discuss this things especially with sort of you know like big participants of the network being able to contribute to those conversations maybe I think it would depend on the company because I think also companies like to do a lot of sort of protecting of their information too and that becomes sort of challenging but it's really it's like all open conversation and it is kind of segmented that conversation right now whether it's get or stack exchange or conferences or github issues there's tons of places where a lot of the same conversations are happening so I don't really know of any centralized place where this is all happening as of right now but it is all about really conversation and seeing what other people are doing this is a very new space we're all kind of just figuring it out and a lot of things you almost have to learn from either experience or listening to somebody that did get the experience and make those mistakes already do you think think interesting because it is the thing like I don't think how people talk about enough is sort of the the types of problems we were talking about today I think a lot of the people working on their protocols and stuff tend to be very low-level and close to there and have you know specific ideas around how people are gonna run notes and stuff together but I think that you know at sufficiently large volumes you have to do it differently and I don't know of a space where people really talk about that yet so I think definitely each research would be an interesting place to just sort of discuss that or either magicians or any of the sort of forms like that all right we got we got one quick comment here just another vote in favor of just some sort of central group group chat I don't care at the venue but when I'm having issues with like all these like holes and nonces and sending transactions and like all that stuff just if we can find some central group chat ever I can ask you guys actually are a few at the same time when they get too large and people start leaving and start tuning out so it's it's a challenge yeah I think it would be good for us to blog post or something about this so we've only got 30 seconds I know the panel's out here but to end on an interesting note it sounds like there's a lot of challenges from a developer point of view I know that's what you engineers love but what like why are you here what's so exciting about you to be in the space in five seconds none of the challenges are insurmountable and we're seeing the 1/2 scale and the one to get better here and I I think the community is great yeah I'm sort of the privacy and sort of ownership of sort of your own funds and you know all the sort of developments there like fascinating to me all this sort of talks today about all the privacy all the cryptography and stuff is it's interesting I think can just lead to a lot of you know good in the world in people sort of you know all right we gonna we gotta kind of wrap it up here Morgan if you want a five-second quick brand new credit set of problems to solve that haven't been solved before and we can solve them together you