Decentralizing Ethereum Data with VulcanizeDB by Rob Mulholand (Devcon5)

[Applause] trying to show ops or what happens if this runs successfully for you but if you're able to connect to the internet and interested in the workshop or style component we recommend is because pulling down vulcanize checking out the branch you've got set up for this process and so yeah you're talking about multi TV my name's Rob Mahone I'm Pete Nelson would organize multiple software crafter at a blend of Chicago and I'm a customer here I had three likes bring my phone up cool so to get started talking about sort of the motivation for vulcanized edu I wanted to start talking about sort of where are we now with accessing data in the imperial system and from my perspective there's a few things to notice one is that can got these decentralized applications but largely in order for these decentralized applications to work we're relying on centralized data right so here I'm visited Bulaq now calm and we've got a bunch of network requests going to in Fura do you access the latest state of the bhulok contracts and that's pretty cool like it works and it's awesome doesent here provides the service but ideally you know we might have solutions that didn't depend on one third party to service that data so in addition to sort of centralization around access to data another thing we've got is that we've got a lot of duplicated effort right so when I have spoken with people about what can I teach in the hallway I think one of the like most common comments I get is like oh that sounds really great like we had to build something like that in house to support our own infrastructure right so it's amazing to me how many teams have sort of gone through this work of figuring out how to like extract data from an a3 of node decoded those rest database but the fact that so many people are doing it sort of raises the obvious question of like why don't we have a shared tool set for that I did want to mention that there's a lot of teams doing great work to address these two issues right so that the question of centralization of duplicate endeavour we see a ton of teams out there making progress to try and improve the situation and you know I think that's super awesome I really have to be able to you know here at DEFCON like see what other people are feeling and from that something you can all learn from each other but I think the fact that you've got this many organizations all working on like this shared problem sort of indicates the the magnitude of the issue and the rationale for having improved toolset and so with that sort of motivation in mind the question I want to propose is so like where are we headed and with vulcanize DB I would say that our mission is to replace decentralized and misspoke solutions with shared tooling that anyone can run and so if we look at the toolbox of sort of what we started been developing to make this happen I wanted to walk through a few different things in our tool chain that you can use like right now to start spending on curtains it's a vulcanized Evie and owning your own your own data do you able to serve it for yourself so I'll walk through each of these in more detail so the foundation of the process for setting up of all kinds deviant DB instance we've got this hetero snake process and what the henderson process does we're basically taking clock headers out of your node Postgres you can configure a starting block so if you only care about a contract that was deployed a block you know seven and a half million then you can start syncing headers into your instance from that individual starting block and what this process will do is it will able you to continually verify those headers at a configurable depth so depending on what your concern is about reorg and so forth you can have this foundation of data that's continually being validated and where data that's no longer on the chain if there was a reorg or something like that is going to be automatically pruned for you so you know you have a consistent record of just like what are the headers that were on the chain and importantly for some of that additional tools I'll discuss you know we have form D relationship between this block header and all the nested data such that you can sort of cascade remove any any game that you have in your system there's a product of a header that was removed so the thing I wanted to talk about today and the thing that the exercise will let you to run is then contract watcher process and so we think the contract Watchers pretty neat what it does is basically you give us a address for a contract and you tell us the deployment block of that contract and then we will automatically figure out what are the events on this contract start getting them off the chain and decoding them for you you can run the contract watcher with multiple contracts role we'll create a schema for each one and a table for each event those event and end up in Postgres will be with the headers such that that interesting process is going to remove so that's no longer valid and this works pretty great for things that are like events that are defined in a contracts API and again so if if you're able access the Internet want to set up here on instance this this will do that for you this will kick off the interesting process contract watcher focusing on three contracts from Moloch down and it will spin up instance of PO scrap pile which I'll get to that will enable you to see that data on localhost 5,000 in your graduate compose an excuse is a command that I want to talk about so this is still very much in the work for us but what the contract watcher gives e right is the ability to automatically look at events that are defined in the API but what we found is that in more complex systems of smart contracts we also got to worry about things like anonymous events right or custom events where the payload doesn't necessarily like the types that you want to decode the payload into and Postgres don't necessarily match what might be defined on the API or the topic 0 on a given log event a little bit different than what you might expect and so our answer for dealing with that sort of thing have basically been to say hey you can write your own plugins and these plugins can be used to you know take care of that another thing that compose and execute plug-ins enable you to do is to look directly in storage tree notes so when I'll be continuing to talk about this during this talk but I think one thing that is really interesting that the community's sort of needs to like reckon with is that right in order to access a traditional need to run like the archive note right if you want to have historical state and so my folks don't want to do that don't be able to access their data running this the full note and the way you can do that is you can sort of stuff your state into events which is cool except it means that we've just sort of moved the problem over where instead of like state float or you have the state look but you also can have like event load right so there have been proposals already floated in the ecosystem to start pruning historical events out of full notes as well and the state that you get from an event isn't necessarily the state that's actually you know what's happening on a given contract and so one thing that we've been working on a lot with the plugins that we're developing is to look directly at storage on a contract now traditionally you're gonna have to do this with an archive node but we've got some ideas there as well anyway the plugins let you look at storage tree nodes you can automatically decode like one of the true that value of a variable on a contract and plugins also need to make more complex queries right so in order to sort of crystallize what those complex queries look like I'm going to bump ahead to post grandpa so post-grad file is not something we built the post gram file is a super awesome tool and we support that job patreon I recommend everyone as well but most gravel enables you to do is you just literally like post gravel from the command line and it will inspect your Postgres database identify the schema and like automatically make that about available to you in the browser which is just like super-sick comments so thank you so much too many he's also super responsive on discord we found it to be a really easy solution to sort of a proposed fixing with an API in the browser but the data that we have and both graphs of course if you're interested you can always you know just make queries directly against that Postgres data but for browser access close grab files angry so some things to know about those grab files that like it'll automatically discover relations between your data right so for example I mentioned that events we decode or as soon as stated with a block better than existing database that you can just like automatically get better by an e with the event that you're looking up and see you know metadata about that block header if you want to see you know what block number was it or whatever post grant file also exposes built-in filters and conditions and so forth which means that you can Mike is gonna find this exact tool to your Postgres database feed here Camela query support subscriptions and notifications and our view data hitting the database there are computed columns which enable you to like amend data and custom queries which is what I was mentioning with the more complex queries on the last slide right so you can in a migration like create a query that aggregates either from say multiple events or tables and those will show up automatically in post growl help cool so that's the toolbox sort of like as it stands right now again you can try it out repos took the open source and the thing I wanted to talk about next is sort of like what are we driving toward with our coming work and so the first thing is simplifying our interfaces for plugins right so I mentioned you can write plugins and they do super cool stuff but our mission there is to make sure that in order to write a plug-in that people need to get things like storage tree nodes or anonymous events that you have to write like the minimal amount of this book could possible for given smart contracts another thing we're working on is client patches to omit these storage Tiff's during a sync so I mentioned that like traditionally archived new to access historical state and that's kind of a bummer but like one option that's definitely on the table would be been able to subscription over the JSON RPC interface that would just like spit out those bits that they're happening and you can even say like specifically and we want you to publish just me if they come from a given smart contract right so part of our sort of open source work is to figure out a way to get this done and get that upstream into gap and parity and so forth so that you can sort of plug in that subscription directly to vault as DP have a plug-in that's parsing that end up on the fly and you can be accessing state data with a plugin that took ideally and minimal amount of code another thing that we've got coming down the pipeline is what we call the super node so the idea for the super node is that we will be automatically digesting all those states and storage tips but also blocks and also proofs for the deaths that are coming out of a node and our goal is to publish all of that data to ipfs and have vulcanize DB serve as sort of a filtering layer where you can say I want to get all of the dips from X contract and we will give you a list of contact addresses back on content of justice C IDs and you can then query ipfs for that data and you can get groups with it right so you don't have to trust that we're giving you like the correct data then that data is valid because you can submit that proof to your new one in order to verify that that ADA isn't back to what we say it is and so again we think this is like super cool work hopefully you can like share dr. Campos style demo the Colette's you do this next year maybe your even sooner but that's that's some huge primary stress and you know we always welcome issues pull requests that people think that other priorities are worth tracking down as well so this is the part of the workshop where I would be staying like let's go ahead and try this out everyone run your own instance of vulcanize DB on their machine if you have internet we even like performance tested a node back home that you can connect to but I don't think performance is going to be a problem with an internet issue here so to walk into our stack you know both men TVs written and go it was pretty nice because it enabled us to like really easily integrate with going theory like unpacking logs and so forth Postgres obviously talked about after graph QL this set up on this slide is exactly what it says on the morning but you know try it out when you're at the hotel or home or whatever like definitely did get feedback on this so what I wanted to point out about the setup that we're showing is our config file this config file is 20 lines this config file also yields parsed events from all three Moammar contracts the Bhullar contract you'll make contract in the black pool contract and to be honest some of that data at the top I'm gonna put in variables to slim this down even more but I just wanted to show you like everything you need to do to do this work so what's happening under the hood you've got this configuration file and then you're running three of the processes that I mentioned right so you're saying I had an interesting process pointed that config file starting at the deployment block or bowler I want to run a contract watcher process with that config file and the most cumbersome command at this post route but that's our fault because the way that we set up these schemas for the contracts is Petter they said this is the header sync process and the contract watcher the training this data and then into the contract address right so fairly straightforward they've populated this data with any arbitrary adjust you wanted you know you can point this email it'll be there there's a little W flag there which is important right like you can kick off this post graph out process as soon as you kick off the contract watcher and with the W flag it'll be watching the schema so you'll see a warning perhaps it'll say like hey we don't have anything in the schema right now and then that the contract water starts to populate that data ok the schemas here you can load it in when you refresh your browser you're going to have that data expose in what's going well so in a workshop well ability to run this stuff we've got I've got sort of the demo here so this is this is a set up instance so I added this whole thing these steps that I'm asking everyone to do and this is the this is the photograph file interface that we have so enhanced graph iql which was a command we passed to who post a file that means the begin a list of like all the available queries on the left-hand side and then what we've got in the center is you know graphical interface with these queries so I've selected these two ownership transfers events and withdrawal events because you can see that if I ask for that again you know 23 events and if we pop over to you know the main source of truth either scan you see the 23 events on the full contract so that's cool that lines up checks out when in these queries we can do some cool stuff so we can say like I want to see the nodes and I want to see the parts like new owner and previous owner data and then I fire that query and like narratives ok ownership was transferred from dress and I can do the same thing with withdrawal events I want to see the American policy receiver and cool you know these are mounts are weird because it's like the p.m. but there's a fixed value and divide that by to get something that's a little bit more easy to parse an easier event to look at for understanding this would be the ragequit event because they're dealing with share spreads opposed to a monetary value and so want to see shares to learn and the member address for the rage quit events on the block contract that at Lake occasions are pretty nice numbers and we can see here we've got like ninety nine chairs break we pop over the Mohawk contract on ether stand is he like not interesting here's that same address in topic 1 you know it's a habit hex value and we do put this number it's like 99 this event was dead empty so okay you know you're just getting like automatically decoded events I had you know I can walk through all the queries but I think folks get an idea of what we're doing right here are automatically getting all these events parts into Postgres with recant so you know the workshop was like first you can run this doctor compose that will enable you to like reproduce exactly what I just demo I did want to mention that that they would I demoed that database ends up being about three gigs that also took me about 30 hours to sync on my own Wi-Fi so continue to work on performance that cost is amortized over the life of your system right so you might have like a high upfront costs to get totally sync with all the events that have happened throughout history but then you're going to stay in sync it's running the process continually not going to fall behind and and yet that costed amortize which is built but what that means is that you know you start seeing events immediately for things that happened early on and the history contracts but you wouldn't a latest one can sell your system and finish thinking but if you wanted to check out some other stuff you could write your own config file right so that was 20 lines to get three contracts you could look at 17 contracts for lines that's up to you you could you could run it with an address where the API is not published on ether scan right - there's an option to supply the API and that config file for the contract if you're not dealing with you know verified source code so it's not a constraint for the system that the API have to exist on understand but that minimizes the amount you have to including to think and if you wanted to run it locally like not in docker then you can you know do a few things that we have hidden for you in our compose script right so you have to like create a database of all guys can connect to we use go modules so you have to turn that on to enable it to bill your bill to end up with your vulcanize DB binary and you should be able to run these things locally and then if folks were able to you know be like okay I did this award and I like made my own config look with contracts board again I'm running it locally on my machine you know I'm still bored then like option 3 that I thought would really take up everyone's time would be okay let's start building around plugins right so all the plug-in architecture is in the libraries shared folder baulkham as d-v and specifically in the factories directory you can see we've abstracted code that'll handle like the overarching process of syncing anonymous aventure sickening storage tips that you have to write like a few small dependencies basically tell us like okay you've given me an ominous event like how does it become why like something that you put the database but yeah and you know how I'll stick around after this time I would love for people to to give this go we can but there we are so I want to say thank you I definitely have not done this by myself thank you know how to uh privilege to speak up here but Rick Dudley Elizabeth Andy gave Edwardian Connor Gustav have all been super instrumental to getting this project to where it is today we support Pina Pina foundation for kids were tremendously grateful and also go for maker down so thank you so much to those folks we're supporting our effort we're really excited about hopefully a lot of value can be delivered to the community with shared tools that for Casa of historical data and that is pretty much definitely have the answer any questions yep there's one subscriptions on the Rettig oh yeah yeah it's post grab out is again just like the most awesome the Bossip open-source projects it's of course subscriptions out of barks oh yeah oh it just curious intercom file they showed us it's quite another dress right so how do you deal with all the contracts for each individual user with the same bytecodes and some you know the induction hypothesis my choices yeah so you definitely want to like write a plug-in to deal with that right because um what the contract watcher is giving you is it's giving you uh upfront facility to say like give it a contract address I want to see all of its events like decoded which you know I think has some value on its own but we're not trying to be like super aggressive on that with the contract watcher itself but with plugins what you can do is you can say okay like given that I see this event that I want to spit up Big Canoe and so the contract watch her appointment to have a field on that event that is an address that I care about or you know any number of ways you can implement that to to run the process and we have code that does that some ian's code yes there's a couple different teams that working both nice TV so there's another code base that actually does that it takes it will read a contract to get a contract address and then write the code without a dress like anyone just to simple API to keep track into Academy yes that's actually what I was just talking about it was an ER c20 watcher and we just didn't year c20 watcher but obviously you could change it to make it in ear gear see seven to one washer as well yeah I mean one thing just to expand on that point too you know we sort of tinkered with a variety of different amounts of like face data that you can scan for when you're running vulcanize TV and so we started off in like listed just like digest everything and anyone can run this and digest everything but like I think reality is that a lot of people who might get value from vocalize DB are not necessarily super stoked to like pay the performance cost that it takes to digest all of that extra information if they don't eat it right so a lot of what I've been demoing here is like a pretty lightweight process of sinking headers and syncing events from specific targeted contracts there are over tools in the box that enable you to do more heavyweight stuff if you've got the infrared the motivation to do that and we're hoping the supernova can help out with that lot too well another question okay is there a reason for not sinking backwards because to maintenance that I've seen from the starting book which means it takes you 30 hours to get to defense you probably care of us yeah so I mean that the main thing is just like I guess that the latest blocks are more likely to be removed by the henderson process right so you'd be potentially doing a lot more redundant work you can totally get around that if you're interested by starting the header sync process with a more recent starting block number right so I was demoing with like the deployment block as a starting block number so that you know you're going to get like all of the events but if you really wanted to say like I've always been up an instance to give me like what happened the last five days then you can pop that in at the Perimeter to the command a block number where you start caring about stuff and then that's where this thing will happen from no but I mean you would you care about everything it's just that's I in that 30 30 hours you want to do something as well yeah so and you have a book watch you're watching for new blows so your real rooster just managed to just the same way as they are right now yep it's just it's not starting at 7:00 visit whatever you walk backwards from the moment you start at the book watch yeah but before but before what before yeah so the short answer is it is technically harder to do that just generally gap doesn't like it's a goal in Reverse so I mean that's kind of weak and if you have really curious about it we can really talk about it but the short answer is got doesn't want to go in reverse well I mean that's specifically true also with like subscription that's right so part of the idea here is that you can easily have like a subscription to events but then those are going to be fired as your nose like processing the blocks that it's going over so part of the ratchet count for the set up is like you could use either subscriptions or get blogs furies to progress forward but I didn't think that's a really cool idea and something we could totally dig into so appreciate it yeah last time I checked you have no upper body they're reasonable it's kind of a work in progress I mean we have I think we had like a doctor file a long time ago and it was like wanted to bring TP and we found it to be like not super useful over our day to day development so lately there's been a lot more work on that and we have a lot of progress on an open PRS I think it's called like dr. updates but yes it's not on master yet and the reason but the reason that this one is on its own branch is because like this is pointed at no luck but maybe it makes sense at some point to have like an animal doctor files but we didn't want to be like the purpose of vulcanized EP is watching the Moloch contracts you know yep so I'm curious about the limitations of the packing API so currently from what I understood begins as a mouth that's a so big mouth event or whatever to another events or whatever can they act as they reduce so for example red juice I know the thing that makes a level I can sum up transfer balance whatever and how does it work with three arcs yeah I think I'm gonna go straight to rec on this one oh okay I mean yeah we could be running reduce but then as you pointed out at the area there we are so what you would be doing is you we would have a table like we have now and then we would just again how in that bed from that table and wrote a new table and then it would know to compute the new table so the the first table that I described would have all of the things back to get viewer and then the secondary table get communicated so you can write off your thing that I just described and login do all these things in plugins and that's sort of I mean that's how we would have to trust that but yeah I mean when I designed the system originally I thought most of the utility would be in reduce that's just not the way that development happened to go yeah that I knew was the nation of if you go to coastal state changes yes so I mean that that's work that happens in plugins because we're actively trying to sort of figure out like how can either leave this automatically or ask you to write like the fair minimum of code do you need to to make that happen but in a nutshell the way it works is that you have a mapping of given storage keys to given contract values Frank so that's pretty straightforward for static values on contracts like it's literally just like the index the value on the contracts you can have a mapping looks like index one is you know the total supply or whatever but it gets a little bit more interesting when you start dealing with like mappings and dynamic arrays because generating storage keys for that generally dependence on secondary data as to like an address to map to balances or whatever and so the way that we're looking at doing that is basically you flag some events as like these have addresses that I know for sure are going to be in this mapping and then will like automatically based on the index of the mapping generate the keys based on hashing that address with the index such that you can recognize like all of the storage dips that are coming off of a contract after a given event has been sync and then you know after you know like what it maps to it's a pretty trivial just like decoding though the value associated with that storage key into the perfect type cool alrighty well thank you so much for coming out to listen to me topic and we're really excited about this work and hope that I can provide a lot of value for the community and totally open source [Applause]