Building Augur - Lessons Learned by Scott Bigelow (Devcon4)

[Music] so hello and welcome to building auger lessons learned in this talk I'd like to tell you a little bit about myself and go quickly into what auger is and how it might architectural II differ from depths you might be more used to I'd also like to talk about what went loft wrong along the way during the build and deployment process we had issues with test at deployment we had issues with our integration with 0x and we've had issues with the etherium JSON RPC interface and finally I'd like to talk about some of the new exciting things we're going to be releasing in Agra v2 my name is Scott Bigelow and I got involved with aetherium in early 2017 when I wanted to understand what a smart contract was a little better I had some previous experience with Bitcoin back in 2013 but I just didn't get it and I feel like the best way to understand was to pick up a tutorial and you know write the standard er C 20 contract but to do so you needed to get on a test network in the network that I chose at the time was kovin this is a parity only proof of authority Network proof of authority means that there's a small set of validators that are eligible to mine this network so if you want to come by any Co vineeth you need to go to a faucet and one of the popular faucets at the time was this Gator channel where you can come in type in your Kovan eath interface beneath address and somebody will come by maybe in four hours maybe in 36 and will send you some Co vineeth now I come from a background of software development so my first thought was well we should automate this and so I wrote this kovin Gator BOTS that's monitored for these things and delivered if to the people who requested them in real time and this was really my first interface with the etherium ecosystem and I thought it was great and I had even gotten to smart contracts yet this bot has been running for just over a year now and has faucet adore most 30,000 times so if you've gotten Co vineeth in this Gator channel you got it from me but this really led me to wanting to get more involved in the etherium ecosystem and I started work on auger later that year in 2017 auger is a etherium based decentralized prediction market protocol this is a system where any user can come and create a market which is really just a question that has some set of outcomes and an expiration date some future date where this is going to be resolved before that resolution users can stake eath on what they believe that correct outcome to be and after resolution the market creator is responsible to come back and inform the blockchain what the real world outcome was but of course there's no reason to trust this market crater to give you the correct answer and that's where augur really comes in auger has created a system of economic incentives such that rep holders the holders of the auger rep token are incentivized to come and dispute answers that are incorrect in this dispute process can last a long time this depute dispute process can go back and forth and back and forth and if we ever notice this if if the system ever notices that there is a major problem the system ends up forking and every holder of rep needs to make a decision which side of this fork do I want to be on and we believe that you that holders of rep token are going to be incentivized to want to be on the true fork the fork that represents the true outcome to accomplish this took quite a bit of code augur is nearly 5000 lines of solidity code and the base deploy is 41 contracts and architectural a it's a little different to if you've used the maker interface before this is a this is another DAP which is a series of smart contracts and a UI a static UI that can make dynamic calls into a theorem and run an eighth call this is a simple function call is just a read call to ask questions I might ask things like hey how much MKR does this account to have you know how much died is this account half now these questions come from the current states of the etherium network all this data is stored that you know the storage of the contract space and there's a little bit of logs here they can tell you some things that happened in history but mostly it is powered by asking questions about right now so our ger does this a little differently the UI does not communicate directly with the etherium node the UI has an intermediary which processes logs to reconstitute the state of augur and because augur is a decentralized system this isn't a server sitting in iraq this is your box and if you've ever seen this window before that's why augur is presented as an executable because this service that powers the UI is critical for its functionality and so why do we do this why do we take the etherium database and turn it into another database so we get a few features here we get relational data the augur data model is fairly complex we have lots of different things that relate to each other and if each one of these needed to be some Eve call that like bled to some other earth call this could be dozens and dozens of queries just to look up a single page by putting this into an indexed relational database we can serve that with a simple single joint query we also get data locality no matter what your latency is to that aetherium node you are serving this data local it's all ready for you and users are the ones who are contributing the titles for these markets and outcomes and whenever users are entering text you're going to need a good way to find that text something that's not exactly accurate so we have string matching and search patterns that that help with this discovery but there's another reason I mentioned that there were 41 based contracts but that's just where it starts when you create a new market that is a new contract plus you're gonna have two to eight outcome token contracts you provided in an initial report that is a contract when you provide a dispute that is a new contract has a week past that is a contract it's called a fee window and so this chart from before where we talked about the 41 contracts that make up augur is dwarfed by the number of contracts that have been deployed since its launch which sits at over 5,000 auger is also a fairly old project we're currently sitting at blocked 6.6 million the release took place around 5.9 by involvement 4.4 but if you want to go back to that very first auger contract deployment the first time that somebody said I would like to make something like auger and deploy it to the network need to go before block zero this system was developed well before the etherium main net block zero was launched on a test network that I think at the time was called POC nine but this is actually before solidity solidity zero point one point zero was released here and so these contracts were written in serpent a different smart contract language that compiles down to the EVM code so when the rep token launched around block 2.4 million this token was also written in serpent and work got underway to develop the augur contracts in this other language serpent unfortunately around block 4.

1 million we asked Zeb Glenn to take a look and that how's us to a house serpent going we had them both look at our smart contracts and the serpent compiler and what they discovered was that the circ of a serpent compiler had a critical vulnerability in it wasn't an issue with the smart contract code but even the rep token itself that had been deployed was vulnerable to a VM code that came out of the compiler and so the rep token had to get redeployed at this point it was really written as a solidity contract and all of the augur contracts that were already ready for audits had to get rewritten into solidity because it was clear that the market had really gone this way and this research really justified that effort so your smart contract is no more secure than your compiler which is why we were proud to release yesterday a joint effort between auger and the etherium foundation for a solidity compiler audit and they found some great stuff they had over 12 with they had 12 issues that they consider to be high or critical and these issues are as some of them have been addressed some of them will be addressed but luckily none of these created EVM code left our contracts vulnerable but we feel a lot better about the platform that we are leaving behind for other developers to build upon one of the issues we ran sue was a test net gas limit issue so I mentioned we have 41 contracts and those take 50 million gas to deploy but one of these contracts the largest of them takes 5.8 million which isn't a problem for maintenance may net has a block gas limit of eight million robson however which is one of our main test networks for developments at the time had a block gas limit of 4.7 million it could not contain that one transaction to deploy the largest contract which was critical for our system here's a chart of the block gas limit over the last two years you can see for large periods of time blue main net could host large contracts but there was no test network where you could actually deploy these and try them out and share them with the world so while Rob's turn in main net run the same code there's a critical difference Robson isn't used as much and miners look to how full of block is to slightly increase or slightly decrease the block gas limit for the next block it's up to them they don't have to use this as their their criteria but it's what they generally do so what's a project to do when they want to ship to Robson well just fill up the blocks let's get these miners to think that these blocks are really full so they'll start voting this limit up and so that's what we did we wrote this up with a smart contract whose only intention was to waste all gas that came into the transaction now knowing what we know now we could have done this a little better if you assert false you can actually hit an EVM up code that doesn't waste all this time on these full nodes but still effectively wastes gas and gives you this block this block that is full of failing transactions but it's full and you know I started in test nets and I really wanted to help people with the faucet that I built and so this kind of weighed pretty heavily on me I was wondering like you know why are we destroying you know the the usability of this for days at a time for people who want to use Austin no I'm just kidding somebody kept mining empty blocks and so they kept voting that gas limit down so we had to do it ourselves we spun up our own miners with a target gas limit that was already set to higher so even though these blocks were mining were mostly empty we kept voting that gas lit up we became more than half of the network and were able to get that gas limit up but it really got us involved with how are we developing our software is test that really the right place to be doing all of this work here is a repository where we have created guess and parity in various incarnations so this is you know connecting to Rob Stan having a local instant seal having a local mining thing that mines blocks about every second and this is a public repository where anybody can come and create these docker files and images to have a consistent environment to redeploy to so you have this local guess that you can spin up on your local system that's great and then you can put contracts into it and if anything goes wrong you just tear it down bring a new one up and redeploy but for us this redeployment actual takes a long time takes about 20 to 30 minutes to do an augur deploy with all the seed data that's important for our development environments and so we took it one step further we wrapped after the contract deployment into a new docker image and this became the base with which and all our entire development environments were call her develop errs use these images now and we version them so whenever we have a new set of contracts to test out we say hey everybody - 12 is up and all developers pulled down - 12 and it's exactly the same for everybody the same transaction hashes the same addresses the same block timestamps so we know that we're all building on something that's the same as if it was up on Rob's turn on something public but it's local and it has one second blocked block times and it has a block cast limit of 8 million we had complete control of how this thing behaves there are new tools to do some of the same things this is ganache this wasn't released by the time that we were working on this but I think we would look into something along these lines this is an excellent local test RPC but no matter what you're using invest in your aetherium development environment our velocity went way up once we got this right so we had an issue with 0x after deployment so I mentioned there's this five point eight million gasp contract this contains the market functionality if every time a user created a market they had to deploy a 5.8 million gasp to be really expensive and so we deploy a delegator a delegator is a contract that has its own contract states has its own address buffers all of its functionality somewhere else the only problem when you use this contract you incur about a two thousand gas cost whenever you bridge this delegation that's just the base call cost of a call like that but imagine that this isn't the market this is the rep token contract and imagine the caller isn't a person who's willing to spend two thousand extra gas imagine this is zero X and the 0 X contract and also imagine that the 0 X contract created a limit a hard-coded limit inside of their contract that said a balance of lookup should never take more than this much gas and at the right that's a lot of gas to do a simple storage lookup and return that data but not when you add in delegation and we blew that gas limit for that one lookup so congratulations you've deployed also your delisted so we had some ideas about how to solve this we could have wrapped the rep token in a non delegated contract in the same way that you can wrap ether into an A or C 20 contract there's no reason you can't wrap a near Z 20 into another a or C 20 that isn't delegated and is compatible but that has a horrible user experience for the user luckily we talked to 0x about this issue they were able to execute on their governance and they upgraded their 0x exchange to a v2 that has a lot of new features but lifts this gas limit so that the rep token is again compatible with the 0x exchange so we have these interfaces we have a or C 20 and these things are great and it's great to follow these interfaces but it's no substitute for testing your integrations there's all sorts of crazy things that these contracts do that are on top of the IRC 20 interface and if you really care about integrating with somebody especially if you're a little different test that integration out so here's how different the rep token is if you look on a Thirsk an the rep token is not verified not because it isn't verifiable but because ether scan does not support verification of a contract that was deployed by another contract and that's what the that's what the rep token is and even if it was verified this is what it looks like is it a bunch of assembly delegating that call to another contract so doesn't really instill much confidence anyway so let's talk about json-rpc so one of the nice things about deploying your software is that you get a chance to get some real user feedback now luckily we received more actionable feedback than this we got some github issues we had failed to fetch parent block stuff is missing balances are gone I can't find anything things are shutting down syncing is a major issue so what went wrong here I mean so from that prior list of docker images we built for for gas and parity we took testing very seriously we tested all of these things we tested in fira we tested local and remote and you know POA we tested the whole thing but in the end this is what happened we had an issue that was specific to in fira and specific to maintenance so so what happened here well let's talk about Eastgate logs our system works by when we're syncing synchronizing the logs with the blockchain we say please give me the log please give me new blocks inside great I just found a new block block 100 cool give me the logs for this block here's an array of logs now in fear it does something different on maintenance in fact they do something that I would consider to be just a little bit crazy in fear serves over a hundred thousand transactions per second that's the thing that it is different that's the thing that I think is a little bit wild here and to accomplish this to accomplish this feat they need more than one server to do it if I need a huge quantity of service to do it in fact if I have to guess how many servers there have it this isn't a public number I would guess they probably have like well let's say - and in order to have more than one server you're gonna need a proxy and so let's walk through this conversation again hey please give me a new block here's block 101 great can I get logs for block 101 sorry so this system doesn't necessarily have the same perspective of the etherium Network as the one that delivered you block 101 and so the system this is this is just a standard aetherium node thinks to itself well what the hell is block 101 but the problem is is it doesn't say that it doesn't say what the hell is block 101 it says empty array so I don't think this is inferiors fault this is json-rpc kind of not necessarily lying but not really telling the truth either so what solutions do we have for this well we could ask for all logs and just throw everything away and then filter out the augur logs ourselves but know that the presence of logs means did we talk to a node that had that block but we get empty blocks all the time okay but maybe we could use like the logs bloom filter and if it's empty we know but it's still a lot of logs so how about we get augur logs and crypto kitties because we know the crypto kiddies has lots of events all the time so we're gonna get like a totally adorable cat in our JSON response but we're gonna discard it use that as our indication that we talked to a proper note and we seriously consider this probably more than I'm willing to admit but the problem is is I don't want to wake up one day and write this comments as a response to a data quality issue so a solution would actually work well what if every time you got a blog that was empty it was like you know empty response you remembered that and then the next time you heard about the next block you made a range request for 101 to 102 and then if you saw 102 you knew that you actually did hit one of what it's just it's getting crazy right so what we did was there was this AIP out there that had been out there for over a year that said how about we query blocks by block hash this is a more unique identifier and more importantly how about when it's not there you let the person know you you don't you don't kind of fuzz over the fact that you've never heard of that block you deliver that to the client so they can make the right choice on their side if anybody heard Jack Peterson's talked last year about bounties I think this is one of the critical success stories of bounties here's an EIP that had been sitting there for 15 months after auger experienced these issues that a IP we got you'll be able to use bounties and engaging the network to get it merged in two weeks and implemented in GAF in parity in six that is an excellent response right and that that's a killer ecosystem and so but in the meantime we had to say oK you've got one node you're good to go if you have two or more nodes whether you're in fear where you're there anybody we can't do that until the CIP gets merged then we've got other issues we had parity there was a parity specific issue when people started launching their new nodes and I didn't understand this because remember like I come from kovin like parodies my jam I know this works with parity so what happens here how does parity getting these exact same issues a single node let's talk about leaf get logs when you synchronize with the etherion blockchain it goes something like this you start at 0 you go to 6.6 million you're good the parity developed something called a warp sync where you first sync you so you pick a points in the middle pretty close to head and you sink that part first then you say hey I'm good to go and later you backfill from zero to that warp sync start point but you think you're synchronized when you're right here and if you issue with get logs for this you've got your logs back if you issue a get logs back here you get empty array again and if you issue a get logs that spans this point you get half of the logs you were expecting and you can imagine the kind of havoc this wrecks on a system that was deployed right here so we made sure we documented this and I read me we tried to detect detect the behavior automatically but really the solution was we've been working with parody and this is actually scheduled for a future release to deliver the correct information to volunteer more information and we've seen some similar issues on the gift RPC JSON RPC interface so but eath call when you call a function that revert Svea that eath call you get 0x eath call when you call a function on a missing contract either because it will never exist or because you're talking to a node that is not at the block height that has that contract deployment you get 0x let's talk what eve call when you talk to a server that is overloaded for a contract function that will succeed you get 0x and so we're left wondering should I retry in 100 milliseconds or should I never talk to you again because this is an unrecoverable error so so what's the lesson here I think we don't quite know what the theorems json-rpc is currently for is this a web api or is this a database query layer because these have different success metrics this needs to be 99.

99% available but it's okay to have a little bit of cash it in there it's okay to have a little bit of fuzzy answers this is serving now a web UI but this needs to be a hundred percent accurate and you need to if you can't be confidence in the answer deliver reliable errors to the user so I'm proposing I believe it is currently a Web API and to support that here's the documentation for the JSON RPC interface this is over a hundred pages of documentation the word error appears zero times and I think it is not a database query layer because if you were to pretend that the issues we've experienced appeared in the my sequel bug tracker these would be big issues quite if querying for new data returns success with zero rows or querying for old data returns success with zero rows this would be a big deal you know but I don't I don't really fault the JSON RPC or parity or get for these implementations because what we built really works we've built things and they work great you know I haven't had a problem using the maker interface but if I ever did I don't know I would just refresh and everything's fine if I was in that you know one in a thousand case right I got a bad get log but when you're synchronizing the database off of the theorem in this way when you lose data you start building the missing data or your synchronization is paused because you don't know when or if you should be retrying you know an auger is not the only system that is trying to treat the etherium json-rpc like a database here's the etherium ETL project where Google is running daily exports of the entire etherium blockchain and putting inside a bigquery but this is this is a daily export and there's lots of value we had there in research and reporting and charting you can do some really cool stuff but this is a different product than an ETL process the tracks head closely and can serve as the API for a complicated for an application that is more complex so I believe that ETL is going to be the future to building advanced applications like this and so why am i up here talking about some of these specific issues want to just go file some issues because I think that what we're talking about here is cultural and I think if this we need to decide if this is the kind of thing that we want to care about and we want to request out of out of these api's and work towards achieving so I think we need to care more about reliable data access and there's lots of projects underway that are leading this charge in fact you know the folks interfere we've been talking to our pushing fql this is a a new way of talking to the etherium blockchain and currently it is it is resting on the JSON RPC interface but the intention is to put it inside of aetherium and so this is our chance to really get it right and enable ourselves to build these big applications so what's what's next what's coming up in in augur v2 so currently all markets in augur are denominated in ether ether is the only token that we accept for for trading on these order books but we have decided not to add died as the token that will be used here we have decided to replace die as the only token that will be traded on the auger platform because we've realized that not everyone who finds use in aetherium necessarily wants to also be long ether especially not for six months or a year as long as some of these markets can take to resolve we're also working to improve the user experience here downloading an application and having it sync locally as some window running on your system like that's not really an optimal experience and that's not really the way that's you know the web is supposed to work and so we're working to take the same architecture and to run it inside of the browser and use some modern browser features to accomplish a lot of the same things that we need but do it in a way that is transparent to the user and doesn't require this extra UX burden because I think this is gonna be a powerful pattern for larger and larger groups to implement because if we want to take this chart and change the scale of it again I want to make sure whatever project does that has has a great platform to build on an audited ecosystem development environments that can support large arrays of of engineering teams integrations with other projects that are well tested and not just relying on simple interfaces and to build on a reliable data platform thank you [Music] you