J2 RF 06 Conor Svensson web3j Web 3 Java Dapp API

everyone I'm Connor Spence and I'm here today to talk about web 3j web three.js the java library for integrating with aetherium before we get started I just wanted to give you a bit of background about myself so I come from originally the enterprise financial domain went from working in large investment in retail banks into the startup space then got sucked into the whole blockchain ecosystem and the rest of this history as I've just said I'm the author of web 3j I'm also the founder of a company called block IO which I'll talk about in a minute and very involved in the enterprise of the Ihram alliance I run the EA London meetup I also chair the quorum working group the inspiration and tools working group and there's a lot of work happening at the moment on a reference specification within the EA which you'll be hearing more about in the not-too-distant future so my company block IO we're focused on that integration there in the blockchain and enterprise spaces you've got a sort of fairly well-established enterprise stack that is used in a lot of these large institutions and they're all very interested in blockchain technology but when you've got a technology that's generally been you know written from the ground up and focused on specifically public chain use cases there's plumbing that needs to be built in order to make the to work nicely together and that's what blocka is focused on and so in the context of enterprises of course you know integration libraries like web 3j good for them but then also enterprises want tools to work with block chains as well so you know blockchain explorers for private chains authentication tools that plug into the you know enterprise integration services and so on and so forth so there's a lot there that needs to be addressed but anyway I'm here today to talk about web 3j and what it was that made me create web 3j in the first place so I'm sure most people here that they're familiar with the different libraries for working with aetherium but when I first started working with the theory I'm coming from the perspective of a Java developer I found that it wasn't straightforward if you wanted to integrate with aetherium the fact that you've got multiple binary interfaces that you need to interface with in the form of their ABI interface which is used by for working with smart contracts then you've also got the RLP binary encoding format which is used within transactions they get signed and then sent to the EVM there's different types of transactions there's 256 bits into integer types used by default and then there's the whole thing of like wallet and key management as well and so for people when they first start working with the space these are non-trivial details to actually have to think about and so this was the motivation for web 3j so it provides and this is where it started off just being a wrapper around all of the JSON RPC calls but once I started writing all the JSON RPC calls I realized that to transact and then work with smart contracts there's this whole additional layer that needs to be added to it and so then needed to add in support for etherium wallet files created wrappers for smart contracts so you can do everything from within Java once you've ridden your smart contract code you then there's command-line tools as well for working with web 3j and it's really Android compatible as well so there's actually a separate branch of the project that's completely up to date which is Java 1/6 compatible which is what you need for Android more recently we've had the web 3j3 series of releases there's been a lot of changes that have gone into this the project's been around for about 18 months now and as of this Web 3j3 release it became modular so what this means is that the projects and the artifacts that are associated with it's they're very much so associated with the components of a theory in which they represent so say if you want to do our LP encoding and decoding there's a lightweight jar file for that specific task it's always a problem with people reinventing the wheel in software sometimes it can be just because they you know they're not aware of what's already out there or it could be that the existing libraries they're monolithic and they already have the code in there but they're just not accessible so what what web 3j gives you in its modules is the smallest possible amount of code to accomplish specific tasks so we've got a bi encoding and decoding as well for smart contracts if for whatever reason you just want a concern with a bi files or you're just concerned with you know transaction signing or just concern with our LP encoding and decoding there's very lightweight modules for doing all of that stuff and so that was a big change there it also provides a fairly flexible API all of the requests you can do them synchronously asynchronously but also works with the reactive ex extensions and so this is a I guess a paradigm that came out of Netflix which was this whole notion of modeling data thinking of data in terms of streams where were traditionally you had the observer pattern that was used in coding but the idea with these observables is that you have something that emits a stream of wrote a stream of events and then you respond to those events these sort of event-driven architectures are nothing new but they've been become a lot more popular in the last few years and the RX jar in the reactive X extensions that came out of Netflix they were sort of pioneers pioneering libraries in that space in terms of the mass adoption that they had it also supports ENS as well so for those of you who want to use ENS and and instead of talking in terms of you know the usual aetherium addresses you can specify any NS any NS name when you work with the smart contract or send transactions and more recently too it's also integrates with truffle so what this means is if you are developing smart contracts using truffle but then you actually once they're developed you want to then plug them in to Java code you can take your metadata files that are generated by truffle and then you can create the smart contract wrappers off the back of that so it integrates those workflows together just making it a bit more straightforward for people to work with so web 3 J it sits in front of a java application in the same way that web web 3 J asana theory and will sit in front of a JavaScript or dotnet library in that respect application in that respect in terms of the JSON RPC supports in web 3 J so it's it supports all of the the course of aetherium JSON RPC clients API then it also supports the parity and death api's as well so you have parity or get specific functionality that you want to make use of it's there these are actually separate modules as well so again if you know you're just going to be working with gets API or you're not going to be using any of the parity or get specific stuff then you can do that using web through J without having all that extra code pulled in and again especially with parity there's you know there's a lot of modules there and you know members of the community have been really good about just implementing them and contribute contributing them back to the project the trace trace API being a good example of that so if anyone's after anything specific they're always welcome just to add it into the project and then it will get bundled with future releases in terms of creating applications using web 3 today I've always tried to keep this stuff as simple as possible as per being a good software engineering practices and so in order to actually you know hook up to a theorem node on the network all you need to do is create a new instance of this web 3j object specify where your node is so if you're connecting to it VAR HTTP which is the default then you just specify you want to use this HTTP service but if you want to connect to a different endpoints or if you want to use IPC as well that's supported too then for the specific JSON RPC method that you want to call you just specify the method name and then you can send these requests synchronously asynchronously or grade and observable off the back of it the library itself is really accessible as well by this was one of the other key things especially you know with open source you've got to be you've got to think about you know how you can make it available to everyone who you want to make it available to especially in enterprises and so on they don't necessarily they're not necessarily able to use all the standard artifact repositories that are out there and so web three.js available in all of the ones globally that are used for serving up Java or Android artifacts and the two main ones being Nexus and been trade by Jay frog and so what that means is it's just it's easy to get hold of the library and included in your project there's also command-line tools that get uploaded with each release and then you can use homebrew as well to set up the command-line tools locally it's now a sample project available as well so you know one of the I think again it's another thing with with open source software especially when people start working with projects there might be instructions for working with it there might be lots of you know good test coverage and so on and so forth but especially with a theory and where there's so much you know you need to learn about as well in order to actually work with it I've always thought about how can you make this as accessible to people who aren't familiar with the etherium ecosystem and so what I've got is a sample project there which gives you a simple smart contract that generates and then this associated Java code to deploy a smart contract transact with that smart contract to read a value out of that smart contract and then also query the events out of that smart contract as well and transfer ether so it gives you an overview of all of the different stuff that needs to happen while the core things that you use aetherium for in a sample project in a minimal amount of code as well just to ease the accessibility I mentioned earlier on about how web 3js modular now as well and so here that the core modules that you have there they're done in sort of dependency order there so we've got the you've got utils you've got our LP encoders and decoders ABI as well then there's the crypto library core is like the core part of web 3j then you've got pieces for working with the page and death parity in fewer - if you want to use that so a simple example of how you actually connect your verify connection to an aetherium client you basically create a new instance of this web 3j object and then from there you just send the web three get client version and then you can display the client version two you know on the command line so that's all this required and you know Java is known for being of a based language but again it's trying to do this in minimal amounts of code there but still have all of the benefits you have of static typing in the mature ecosystem associated with Java against nd for being one of the core things that we want to do in etherium we create a web 3j instance then we also have a credentials object so if you've got a you know if you've got a private key in the form of just a you know if it's in a file or if it's you using an etherium wallet file whatever format you've got it in you can load up these credential objects and then you just can send a transaction by specifying the address you want to get to go to and then the amounts and you've got the the different sort of levels of granularity that you get with the ether currency supported in there as well which you see in the convert piece okay and that's all you need to do to send a transaction in your code I'm start to finish so there's command-line tools for doing some of these common car tasks yet you know you can do this using other tools as well but I thought you know if you if you're making use of the tooling and the library then might as well make something available there so you can you know you can just do brew install web 3j and then if you want to send ether from one wallet address to another address you just do web 3j wallet send then you unlock your wallet file then you specify where your nodes running that you want to connect to so you can use in furor for instance and then specify how much you want to send the destination address and then you confirm it and then the transaction goes through you can also use the wallet files sorry the the web 3j command-line tools to create a new wallet and so again just web 3j wallet create and then you enter your password and then you've got a new wallet file created on the filesystem again it's just another option for doing this stuff there's there's other ways that can be done to do this used to do this but more importantly we have the smart contract wrappers the smart contract wrappers these are you know one of the much more powerful features of web 3j you still need to work with solidity but at the same time once you've written your solidity code you can do everything else in Java and so with solidity once you compile the solidity files is you don't no doubt know you get your API files you get your binary files the binary files are the EVM instructions that need to be executed to push the smart contract onto the EVM and then the ABI files are your sort of point references to methods on the smart contracts you can also as I mentioned earlier on work with truffle schema files as well so if you're using truffle base this is a very similar process and so then you get your Java code to deploy call and transact with smart contracts so without web 3je this is what your aetherium the sort of transaction process looks like so on the far left you've got your transaction objects then you've got your wallet key that then signs the transaction then your your signature data gets appended to the transaction it gets encoded sent to the etherium network and then that gets propagated around the nodes and the network then eventually a miner produces another transaction that gets appended to the etherion blockchain with web 3j you have these nice abstractions at the start so you know if you want to transfer a fee you want to deploy a smart contract you want to transact with a smart contract you want to call a smart contract you do that via method specifically to do that you don't have to think in terms of an arbitrary transaction object which is what you have to do if you're using stock aetherium without any of this stuff then it takes care of the the signing for you but then the other key thing is that with aetherium in its normal form to kind of fire-and-forget you put a you send a transaction into the network you don't get notified if that transactions taken place successfully and being added to the etherion blockchain and say what web 3j has is a transaction manager that can then be used to poll for when that transactions actually happens of course though if you've got a lot of transactions going into a theorem you don't necessarily want lots and lots of separate threads proce polling again and again and again against specific nodes on the blockchain so there's different types of transaction managers that can be used you can have ones that run in a single thread but then you know have a store of all of the pending transactions that you've sent to the blockchain and it'll just to ensure that it can it can scale up as you increase the transaction throughput so just to go through a trivial example we've got the the greeter contract for those of you who aren't familiar with it it's kind of like the hello world in the in aetherium where we have a contract which stores a string value when you create that contract you pass in the string value you want it to store aka the greeting and then you can query the value that's stored within that smart contract which is obviously the greeting here so you generate an AVI file for that greeter along with the actual binary code and so you've got those two artifacts and then at that point you generate your smart contract wrapper so using solidity first of all of course you compile it and then to generate the wrapper you just specify this web 3j solidity generate you say where the bin file is where the ABI file is and then you specify your Java package name that you want to use and then where your destination directory is and then a smart contract wrapper in Java gets generated there and that's compatible with that back down to Java one six so if you want to generate wrappers for Android or Java at all it's the same process then from truffle it's a very similar process except you specify web 3j truffle then you pass in a truffle metadata file this is what the code looks like again you shouldn't actually have to look at this stuff but it's just to give you a flavor of what's happening underneath the hood and so you have at the top there you've got a remote which is returned by all of these events which basically means you can choose when you try and perform a transaction deploying a smart contract or transacting with a smart contract whether you want it to happen synchronously or asynchronously or put it into an observable and then here behind the scenes it's doing all of the type conversions so what web 3j does for you is it takes the native Java types converts them into ABI types then encodes those ABI types into the relevant binary format that can be interpreted by the EVM so you get complete type safety going from Java all the way through to the EVM with web 3j and so this is what our hello world looks like so again we create a new web 3j instance then we load our credentials and then we specify we want to deploy agree to contract this is the gas price and gas limit we want to use and we want to send in the hello blockchain world's because it's a hello world example and then we can read our greeting out of the contract so that's generated showing that entire sort of end to end life cycle with aetherium just in four lines of Java code so I mentioned earlier on about our ex job while reactive ex support and this notion of observables being used in web 3j and so it's a really really nice API that's used in reactive ex and so if you're not familiar with it there's support for all of the major languages with it but it's it's just a nice abstraction that kind of takes these ideas from functional programming and it enables you to have an event-based system and you can use functional composition and all those sorts of nice things you get with the functional paradigm but within web 3j as well and so you have observables for all of the etherium JSON RPC client methods so what this basically means is that when you invoke a JSON RPC message within web 3j you pass in a callback and then that gets executed off the back of the observable it's not particularly useful when you see it in the context of this but it's just to make you where it's available where they're really powerful is composing multiple observables together because you can just have an admit that you can have a series of events that just could be composed together and you can apply all of your sort of you know the things you associate more with you know functional programming with that so you can do your you know your Maps your reduces your folds or whatever else you want to work with they're all so nice for working with events as well so within these smart contract wrappers that get created they also support any events that have been defined in the smart contracts so what that means is that you have a smart contract you deploy it and then when you can create an observable so you deploy the smart contract and then you want your code to be notified every time a specific event takes place on that smart contract and again you can just do that here so I'll give you an example in the familiar ERC 20 contract so if we're using the human standard token which is one of the two I guess best-known ERC 20 implementations so we deploy our ERC 20 contract here and then what we can do is every time a token transfers from one address to another address on the blockchain you can be notified of it and this specific event can event handler can take place so what it's saying is every time this transfer happens between any range of blocks so you can say you know it can be since the beginning of time so it will then go back and find any historical points in time where that contract might have been used and then each time this transfer event happens this callback is executed so it makes it really really easy to you know aggregate data or whatever else or you know just just re replay transactions and everything else in aetherium associated with smart contracts there's also an API for working with new blocks as was sorry for processing blocks too so what's so nice about you know the etherion blockchain it's a it's an immutable data structure and so you know it's event based and so what you've you've got this really nice synergy between the observables type abstraction where you're thinking in terms of series of events taking place and then you've also got the etherion blockchain which is each block is you know a new event that's being happening within the blockchain so to speak that's separate from the notion of a theorems event log or you know log events and so if we want to process all historic blocks that are in the theory and blockchain that's how we do it we just say we want a block observable then this will replay the blockchain every time a new block gets created on the etherium blockchain you've got some code there that can just do some arbitrary processing on that block we can also replay transactions as well so if you want to replay a range of transactions in aetherium - it's just it's another it's another statement a single statement here where you just say replay transactions observables say what range you want to work with and then you just pass in your your method for so your function that you want to acts on every single block as they get replayed through the history and then you've also got very lengthy names but it's basically a method that enables you to replay the entire history of the blockchain from the beginning and then all future subsequent blocks as well so you give it you know it's one command you pass in your callback and this is all you need to do in order to replay the entire trip previous as well as future transaction history on a new theory in blockchain and it's not too bad in terms of how long it takes so back when the ROPS the network was still you know before it went before it started being the favorite place for spammers to attack it you could replay the entire transaction history of Rob stand last last summer when it was just shy of a million blocks in seven minutes and then all transactions within you know in about 40 minutes and so the point is it's not about doing it as quickly as possible it's still going over the JSON RPC API but it gives you just a flavor of the fact it's not insurmountable if you just want to run something across the entire history of the blockchain okay so web 3 J 3.1 that came out said at the end of last year and so that added in as I mentioned earlier on ENS support so now you have yeah with all of these examples here as shown in terms of smart contracts and using aetherium addresses but you can actually use ENS as well and I mentioned as well that they've got the Jason the truffle JSON file support as of last week though web 3 J 3.3 came out and so that now supports pneumonic star wallets too so if you want to have like a series of phrases to create a wallet and the AIDS recoverability you can do that now it also supports ENS reverse resolution and reverse name resolution as well this was something both of these members things that members of the community submitted and now to theirs encoders therefore check some addresses which is something there was a bit I'm overdue but it's now available in terms of the actual community so the project's 18 months old we've just surpassed the thousand stars on github there's been 46 different country from true Peters to the projects as well it's averaging over 7,000 downloads per month and so these are the core a wave three J artifacts so if you tally up you know all of the artifacts it's a much bigger number but these are the the number of times it's just happening I'm also always on the lookout for you know trying to get pull people into the fold as well and so we've got some really interesting mini projects that have highlighted that would be available for web 3j so one of these Nick Johnson and we he was hacking around before with adding in Java card support for get well adding Java card support into web 3j would be something that would be pretty straightforward because Oracle obviously provide a good amount of Java support for Java cards but you know that's something that would be you know a cool little project that would be fairly self-contained but is something that would be of great benefit to the community there also support for WebSockets as well in web 3j because right now it's it supports IPC HTTP it allows and support the WebSockets API which is it we'd get and then to a Gradle plug-in as well but I'm sure that that's something at all care before too long it's also help wanted labels too so if you want to just do work on a small task as well there's this stuff there too that you can pick up and have highlighted those issues with with labels on the projects so this is what the overall web 3j ecosystem looks like now so there's the the Java library but then there's also the Android library there's a version of web 3j that supports quorum - there's a integration with spring boots as well which is used very widely within enterprise and Java space and there's a web 3j maven plugin and it's got to the point now where there's enough interest in the projects and enough users that there's even you know web 3j training horses like the two-day course that's being run later this year as well and so you know feel that it's been you know really steadily growing community but there's you know good growth of users and interest in the project there so for further information there's yeah everything's up there I've got some web through J t-shirts here as well there's a couple of stickers left down there that's all I had to say about web 3j anyone have any questions look do you have any support or plan to support any kind of good interface for bloggers reorganizations or confirmation times to deal with events that happened and stopped happening and then happen again to do block realizations in ether you at the moment not as such within the library I think really it comes down to you know what information is being emitted to us via the you know the etherium client that we're working with ourselves I know that there there has been issues around good as you say with certain events and changing over time but it's it's not something that users have expressed much of a need for and I think the challenge you have as with all open source projects is that it's a balancing act you do this stuff for free and as long as there's not lots of people affected by it or highlighting it then it's tends not to be dealt with as a priority can you tell us a little bit about how do you work with contributors sorry how do I work with contributors to the decode requests like is there any set any process in place or how do you hope to deal with with them in future to encourage them to contribute to in terms of encouraging people it's yes I try and you know put out as much information as I can about the project itself so there's been information published to the wider because I feel anyway they're connecting to the wider tech community outside of the blockchain community is really important so I've I've had Oracle publish content and web 3j of spoken at a number of regular tech conferences about it as well and so that's in order to pull people into the fold that's one way of doing it in terms of what the contribution process looks like though it's much like with any open source project whereby there's there's issues there that people can address or they might find their own thing problem that they want to solve they submit a pull request there then we review the pull request and then it gets merged and so it's in terms of the timelines associated with it it varies because again it's just a project that's never received any external funding it's been driven entirely and off off off my own back and then with you know the community getting involved but it's following that sort of fairly standard process there in terms of ensuring quality though there's that's an area that's I think it's important and so you have you've got like your Travis CI which is standard so you get that for free with open-source projects but then also it's amusing Czech style there as well so you've got code quality and styling rules that have to be adhered to but and so you get that degree of automation where if someone submits a pull request the entire you know regression suite get sorry there or it's got to pass all the unit testing as per the CI the code quality's got to be good as well and the overall test coverage can't change massively either I can't change downwards says this fairly standard the in that regard okay all right thanks everyone [Applause]