EthCC 2: Lucian Boca and Valentin Caramlhal

hey thanks for joining us my name is when I think are we hi I'm managing infrastructure Alessio and together with Luciano who's leading the core API product we'll be talking about accelerating the app development with index.htm data so short overview will tell you a bit about Alessio what we do the new products that will be releasing recently then we'll go a bit into building our data pipeline and finally which young we'll be diving into the core API so by alessio we're data analytics and visualization company we do it we do analytics on the human blockchain we are a team of 20 people mostly developers and data scientists we're part of the consensus network and our focus is to build the tools that help a transparent interaction with the ATM blockchain so this is a map of our analytics suite at the bottom there's the data services stack we use this for all of our in all of our products as a foundational raw interaction with the blockchain so this did a pipeline so this data pipeline interacts with three own clients extracts raw data synthesizes it and makes it available in more accessible format for the layer above it to consume so we built a number of tools based on this platform the first one that we released three months ago is the yet static block Explorer it's in beta so you can check it out I'll show the link in a second and we're excited to announce two more tools in the suite I'll let your monitoring and reporting so I'll briefly go over it cuz there's not enough time to demo it but this is our block Explorer you can access it at that link there besides basic functionality of a block Explorer like seeing transactions accounts blocks and all that it also has some nice developer oriented features I'll hide a few of these right now so the first one is the message called sugar graph so if you go to an arbitrary transaction there's an option to see this cold sugar graph so you can see how that transaction interacted with the smart contract it was targeting and how that branched out into other calls so this is really interesting if you're a developer and trying to understand the Dean the way you smart contracts interact or if you're a security researcher or if you're doing reverse engineering this picture is quite helpful a second one there are multiple views to these graphs so this one is another facet you can see the accounts that were hit by the transaction so you make a transaction to a smart contract it branches out through some other calls and then in this graph here you can see how all these accounts interacted together so again really interesting if you are the reverse reverse engineering or understand what what went on in that transaction another feature on the account details page you have last Thursday balances in ether and tokens you can do a breakdown by tokens so you can basically see how that account doing in relationship to its holdings that's like a total beef balance and there's also representing u.s. dollars so I'll go over these really quick this is the second product this is our first open source release it's an iteration of each stat net if you know it the network health monitor you can basically volunteer your node to submit statistics to our platform and we're going to like and you'll have some meaningful graphs like difficulty you also have the synchronization state of your nodes in this chart here so this one's open source on github on our github and you can pretty much set it up for your private chains as well if you know puppet the v1 of this was you could set it up from puppets as you set up a private chain this is interesting so check it out a third one blockchain monitoring is an alerting framework or product that we we were putting out there you can basically monitor any water or smart contract you can customize the conditions on which alerts are triggered for example if you go above or below threshold in the accounts holdings or if you're looking at a smart contract you can alert by volume of function calls and a lot of other things that are meaningful if you're monitoring like core contracts or if you want to do competitive intelligence and look at competitors contracts that's an interesting tool to use and it can alert you on multiple channels like email and we're coming up with other types of notifications as well you can go to that link and see more about it and finally the auction reports we this allows you to go over arbitrary segments in chain history and on-demand generate these reports currently we support this set of reports but we do customer we interact and get feedback from the customers and see what's interesting and like what interesting queries we could create for them this is backed up by data breaks so it's pretty flexible and what sort of queries we can do and that's even at scale cool so I'll briefly go into our data pipeline how we built it the purpose is that we wanted to have a single point of row chain interaction and not have to duplicate work across each product that we do it's architected it's running on kubernetes there are a few micro services chained that stream data as they transform it through Kafka there are intermediary results available at each step so you can tap into various formats of this transformed data some formats are useful for data science some are more useful for the web apps so basically this data pipeline does the heavy lifting first of all it does run on each client pool so it make sure that it's it's alive it's doing well it's synced and everything it's highly available and the second role that it encapsulates is extraction synthesis in data enrichment so by extraction I mean it's asks the clients like rah chained data that it can get directly from web to our RPC and just stores that additionally it synthesizes stuff like parsing the the VM traces and figuring out the intermediate transactions that have happened like internal transactions we call them contract message calls and finally the way we enrich data is basically we interpret like known standards like tokens and extract that data and put it in an accessible format or if there's a known contract you can derive application level semantics from that so we could indicate like okay this is like a bounty or this was something else right and then finally we do integrate to the number of partners that's also provide us additional data the first one that we're going to be releasing soon is mythix they help us show security related information on each smart contracts account page so very soon in our block Explorer you you could go to random arbitrary smart contract and see what mythix has to say about it and finally about service reliability like security reliability also comes in layers so the layers that we have put in place for our data platform we have local assertions so each micro service does its validations as it transforms the data so like local level assertions then overall after this data's persisted we have a feedback loop so there's a process that checks all this chain and make sure that it's continuous and everything's well if not it can do some self-healing sort of behavior as it can read to trigger some processing of some blocks and it does help in some situations or just alert us and finally we do have replications in high availability configured we kind of build this from from the start so this is cool in a kubernetes environment where you do maybe you're rolling upgrades of nodes so nodes can come into existence and be taken out it's really cool to have I mean essential to have high availability so that your replicas can pick up parts that were removed from the dying you know let's say so this is about it really brief about the data fine next up Sharon will tell you about the core API thank you thanks father I remember when I first joined Alessio and the team ran me through an internal demo of the data pipeline it got me really really excited because it spoke directly to most of the challenges that I was facing earlier in the days when I started developing my own apps on top of aetherium so there was a lot of talk back then about you know the good sides of aetherium and the bad sides of aetherium it's decentralized it ran smart contracts it's it's expensive and so forth but I did not hear that many people talk about the ugly sides of it and this sort of a hundred extra engineering effort effort steps that I need to put up with if I wanted my app to reliably synchronize with the chain and to to interact with what what's what's going on there so I guess my favorite example here is what what I like to call the Schrodinger transactions is this funny situation that you have at the top of the chain when miners compete to produce blocks and so your transaction could get into the chain and then it would get out of the chain because it you organized and that's pretty difficult to handle and I had no idea I have to code this myself but then even even if you're doing that and you you want to say monitor all the instances where you're receiving ether at your address you start realizing that is not enough to simply monitor transactions because you could have things like contracts sending money to your account and then you could have multi-sig wallets and maybe receive prizes from bounty contracts and then some other some other person called the contract which called another contract which in the end sent you the money so if you want to capture all that you've got to replay all the transactions on all the possible blocks and map out the entire set of interactions so that it can sort through the ones that directly affect your account and then it the matters get even worse when when you realize that the the blog chain itself stores all the data around blocks and most of the applications want the data centered around accounts so if you're if you want to display some kind of historical activity for for an account then you need to sort through the entire chain and do your own indexing pipeline and then you need to keep this in sync with the chain as it progresses and if you realize that you need more data you you're kind of back to square zero and start tree indexing again so I hope it's not too far fetched if I if I tell you that the moment when I saw the pipeline that Alessio was already developed and I realized this is probably the closest thing I've seen in the in the blockchain developer space to a holy grail because not only it solved my own problems with with my own app but it was ready to assist any other etherion based app and project that needs to handle aetherium data in real time in a robust and reliable way so then it kind of became our challenge to package this this middleware that feeds at the root level from the raw activity that happens on chain and then it processes it and it organizes it so so that it can make it available to dub developers and researchers and analysts in a way that it makes more sense to them and so that they won't need to do all this kind of engineering effort in-house so we realized okay we're going to design an API and we wanted the the API to be specific enough to be really useful for the use cases that we already knew people have and generally enough to handle lots of other problems that we hadn't thought about in advance and we wanted to use open community standards all throughout the way we didn't want to come up with our own terminology and with our own way of handling things so we were really lucky to have healthy the data pipeline built already on top of the theorem ontology which is a community source standard and then we use JSON API for serializing the the data payloads it's not by the way it's not any kind of generic JSON API it's a very specific set of standards so you can check out it's a pretty comprehensive project and then we came up with this overarching view of how the API should behave if you're looking at it from from a bird's eye view and then we we thought well we already have a pretty a pretty good block Explorer and that product lets you serve visually the the entire ocean of information on chain in a clickable way so you can start with a block and then you can surf through transactions and then move to accounts and the list of transactions and and so forth what if we could come up with the same kind of experience but instead of clicking on screens you're clicking on Jane's JSON data and that's exactly how how is how I structure the API so you have at the root level you have resource collections and these are the usual stuff like blocks contracts accounts and so forth but then you can zoom into any resource and then for every resource you see a list of static attributes and you see a list of relationships that link that resource with all the other resources that are relevant to the to to it so you can have for an account or for a contract a unified timeline of aggregated activity including transactions and function calls and related log entries and so forth so we're creating this full picture view of what's going on on chain and we're feeding this in a way that apps find it most relevant so I'm I'm really really happy to announce that today we we've launched the API it's open for free access you can sign up at El niño slash API and you can start building on top of it it's live on it on it on the on the main net and as a concluding thought I I realize that I should I should have probably mentioned this in the first place that this wouldn't have been possible without the incredible team effort that the entire Alessio team has has been doing in the past years and it's not only an internal team effort for us it's an intern it's a team effort with you with the community so I think as much as I'd love the work that we've been doing so far right now this is the end of the lesser work and the higher work is to hand it over to you and help grow this already incredible ecosystem thank you [Applause]