Portal Network, light client summit

[Music] hello everybody how's it going uh i'm piper um and i'm a little bit of a one-trick pony these days this is the pony uh here tell you about portal network uh this is um sort of like my white whale in this thing hopefully it doesn't eat me and sink my ship um i feel a little like weird going i don't know like i know i have a room full of people that lots of people know some things about this and lots of people don't i apologize to those of you who know a lot of this already um but we're going to talk a bit about what this thing is what is that not the whale um this thing yeah uh so the portal network is um sort of like a complete rethinking of what we need to do lightweight protocol access it's pragmatically it is five different new peer-to-peer networks that we're building out to serve all of the needs for lightweight protocol access at the execution layer um i think that if we prove that this is successful and viable this thing expands beyond execution layer it just had some nice conversations about consensus layer data and how that sort of fits into the constructs of portal network but we drew some pretty tight boundaries around this and because otherwise the scope just expands forever and you never get anything done so we're like actually like drove some tight boundaries around this and said let's do execution layer data this is like the area where i've focused my expertise in um so we get these five networks and this is basically what you need to do arbitrary querying of the data on the execution layer and do actual interactions with it right follow the chain read the data uh send transactions right this is like the the core boring stuff that we do um yep uh header gossip network lets us follow the chain state network lets us query information about accounts and contracts uh transaction gossip send transactions right get them to the miners or front runners or whoever it is that you want to get your transactions to a history network for just digging into the boring like long history of blocks and bodies and all of that stuff and we've got this canonical indices thing right because it turns out that looking things up by like transaction hash or like looking blocks up by their number is like a little bit complicated um pragmatically this is what the portal network is like in you know in developer terms and peer-to-peer network terms um but at its high level it is how we deliver uh this is this is the um i'm always like uncomfortable saying like this is the the best thing this is the the right solution i don't know there's probably cool other ideas out there but this is like the the best solution that i that i could figure out for like how do how do we get lightweight protocol access how do we actually get uh scalable access to this stuff on small devices um our clients are huge um so and and maybe this is the other thing that we are focused on on users right like our lots of clients have two masters uh the protocol like the thing that they're supposed to hold up um and then and then users right the people querying data and like the people that we want to do this and um often those two masters have very different needs and the the protocol tends to win in that tug of war especially historically in the execution client realm um our full nodes are super heavy right and you're stuck today and and drive the van around or get on this you know the crowded subway with everybody else on infiera and everybody can read your data um these are like diametrically opposed ends of a spectrum and like let's get our cute little smart car here um and yeah full nodes are heavy like like i i find it kind of hilarious that like the you know 500 gigs of hard drive like fast disk space and like they're heavy pieces of software and this if you want to talk to the protocol without passing through some intermediary this is mostly what your options are um um or you know try to find an les server or things like this but like the options aren't great um all right design goals uh lightweight right um like this is a fundamental thing resource constrained devices um you know can we run this stuff on uh raspberry pi's can we run this stuff on mobile um can we run the sun just like boring consumer laptops right um without it being you know maybe the only thing that's running um this is like a primary design goal uh here um ethereum clients are heavy for a bunch of reasons um there's there's like just lots of stuff that they verify and do there's lots of data that they store um these are problems that have to be solved and we do it relatively like i don't know i've been at this for a while and so i can't quite ever figure out which parts are simple and which parts aren't but the whole idea here is design networks that spread things out that allow people to show up and lift what they're able to lift um but do not say you know you you know uh that don't dictate uh high resource constraints to even be part of the network um i'm gonna ignore this chart today uh it's it's this thing right like it's it's no fun to be told that you're not allowed to to be part of the network because you didn't figure out how to sync four years of state or you know hold on to four or five years of block histories and bodies and receipts right and the idea is like maybe we can i don't know i just really like kids and trench coats pretending to be adults um but but but this is what like the existing network has this you must be this tall to ride concept of like if you want to be part of the devp2p network uh where all of the data lives then then you're not allowed to unless you unless you do all of these things uh similarly with the mempool um the mimpool is like a pretty big thing there's transactions flying everywhere and like if you want to send a transaction you've got to be part of the mempool we redesign some things around this idea of well i might not be able to process the whole nimpool like i might not be able to drink out of the fire hose that's being sprayed at us the of like constant transactions and transactions and so um another piece of this was redesigning like a different type of mempool that allows for small nodes to process small parts small pieces of the mimpool and bigger nodes to process bigger pieces of the mimpool and overall for transactions to be able to still reliably make their way to minors another main design goal here is just basically ux right um you know for talking about your you know the the user of your software being the network being you know staking infrastructure things like that then sure there are some things where like waiting a bit of time for sync and all of these things are okay but when you're talking about people and users if the instructions are install the software on your machine wait six days now you can use it like it's not very good ux uh syncing is not fun uh like there's uh there's been years and years of work here into like making sync a little bit better and making sync a little bit better but frankly making people wait at all is basically like not okay ux when we're talking about end users right when we're talking about people who are doing wallet interactions who want to do whatever it is that people do with ethereum right like having you know this long sync time wait that's um it's not the thing right um we've generally designed things around this idea of no syncing right no no base requirements as long as you have as long as you have a the tip of the header chain then you can do uh then you can do interactions with the network and we kind of leave it up to you to say how you want to get the tip of the header chain because that's like the one piece of like subjectivity here um you can do the hard thing where you sync the whole header chain or whatever or you can maybe like do the thing that's probably acceptable like trusting four or five different like relatively trustworthy intermediaries to tell you what the head hash is and if they all agree then they're probably not all lying right we're not talking about use cases that are necessarily incredibly high stakes here we're talking about user wallet interactions cool scalability this is another piece and like not this type of scalability um that's other things um i just feel like i have to wave at you because les is your thing and and here we go uh so les doesn't scale super well and we all know this um and like this is like the thing that happens with les um it's hard to find a server right it's uh it's not on by default there's just all of this stuff but like client server things they just sort of like fundamentally decentralized neat but like if you don't have enough of these little things that all of the spokes are connecting to then then then it's this right um uh the portal network is designed on this like distributed model of uh just throw more machines at it and it gets stronger in theory right there's a lot of careful caveats here but the general idea is um aiming at a network design that you want to throw 10 000 nodes at it neat you want to throw 10 million nodes at it probably just fine also um uh you know scalability at the level of stuff running on most of the people's phones and all of the different like having maybe two or three different portal clients running at the same time on your machine because there's one embedded in a browser and maybe there's one at the system level and maybe there's one at the like wallet app that you would that you downloaded um these things being ubiquitous and lightweight and embeddable and and there being lots of them um kind of like bittorrent in this like collaborative way of throw more throw more machines at it and into the like global amount of resources available for the network just grows rather than each machine taking away a little bit of capacity any questions just before i do a quick practical example of like what doing something with the portal network is cool um i i had another one that i did before but i think i like this one better eth get balance is a pretty straightforward jsonrpc endpoint right you want to look up the balance of your account this is sort of the like default client model of this we're going to presume that we're doing like a balance inquiry with a block number so like we're going to actually look something up at like a specific block because it makes the example a little bit more interesting so this is like standard json rpc endpoint you hit it with git balance with you know say i want the balance from 10 blocks ago and what the like your traditional client is going to do is basically hit it's they're actually maybe not three different databases but i'm going to treat them like they're three different databases um it's going to hit this first one the canonical indices right it's got to map that block number into a block hash right which one is canonical each each height can maybe have a couple of different uncles or whatever um so it's got to look up which which block hashes is actually the right one for that height now it's got that it actually needs to go look up the header from wherever it's storing headers and retrieve that inside the header is the state route which lets it kind of dig into to the actual state data so once it gets the header back it's going to reach back into another database where it's storing all of the accounts and contracts storages and things like that and actually read the balance belt out and give it back to a user right this is like the the the thing that happens under the hood every time you like load meta mask up right it like hits a like get balance thing to tell you how many tokens you have in the stuff and right this is standard json rpc stuff and under the hood it does a bunch of things the whole thing in the portal network is that we have to replace all of these data stores with network level functionality um instead of the local database it reaches out into a like a peer-to-peer network to fetch what to look find that lookup for which block hash corresponds to which block number reaching into a whole other network the history network to actually retrieve the block header from that network now that it knows what hash to retrieve data data with and then again now that it has the hash and it has the state root doing a bunch of interactions in the state network to actually read that latest data out this is kind of really what the portal network is designed around is um look at what clients do look at how the i mean json rpc like the the json rpc api is what our whole like user ecosystem is built on top of right it's what infura gives us and um who knows what they do with all of those requests on the back end but but we ultimately built the portal network out around these apis around what users do and what is the base level data that you need to serve all of these different endpoints um we are in uh so so project like how's it going how are we doing um we have three different clients under development trend is the one being done by my team it's in rust ultralight is being done by the javascript team that one's in javascript [Music] fluffy is the one by the status team those guys are killing it i'm really loving partnering with them um these are uh three different full clients kind of built out towards slightly different use cases in slightly different directions but all like full portal clients to the portal network um we are uh we're in this middle of this sort of like spot of of like kind of like boring incremental process and uh uh progress and i'm sorry that i'm calling it boring but like it's really exciting from the inside because all of these pieces are coming together that that are building out uh the first network that we're creating which is the history network um but but like uh um the amount that's there to show off today is is pretty small um but but we're we're months away from a like fully functional um version of the history network that that we can start piping in and housing all of the the blocks and headers and and receipts history um and then immediately up next we start building out the other networks um and we sort of expect each one to go a little bit quicker than the last one because they all build on top of the same basic primitives and same basic things um software estimates estimates are meaningless i could i could tell you that we're that we're going to have this thing up and running by the end of next year i could tell you any number of things here and and the real thing is going to be when we show up with a real working thing and so that's what we're working on um anybody who wants to join us who wants to begin a new implementation who wants to take apart their normal eth1 client and do some portal client stuff on top of it um anybody who wants to do beacon chain stuff which is very quickly racing towards us in a portal network context um i'm absolutely really interested in talking with you and in general i'm very game to have anybody like ask me questions tell me how we're wrong uh help help us like like this is a this is a big project that spans multiple teams and over the coming years um assuming continued progress and success it's just going to cross more and more teams and things like that um this is not that was the thing that was like really surprising to me over the last couple of years is like going at this problem and kind of continually realizing like this is bigger than i thought it was this is bigger than i thought it was i thought we were going to be able to do this easier i thought that this was going i did not think that this was going to be so hard and and and the scope of like what needed to be done to actually deliver on this promise being well multi-team relatively large something that we need this community to do together so thanks to those of you who are already part of it and helping thanks to all of you who've asked questions helped me validate ideas and um that's all i got [Applause] anybody got questions [Music] okay great so i just want to confirm my understanding about how people interact with the portal network so instead of uh like having their own full node that can access all of the information in those three buttons the portal network is basically five different networks that you ping instead and can get that information more quickly rather than taking like a full node that holds all of that in one yes so the question is what uh confirming that the portal network is essentially the the distinction is that the data coming from these clouds is not uh you know data that you're getting from a full node data that you're requesting from uh from a full client but it's data that's that's coming from the network in some way and yes uh like all of the uh the networks themselves are kind of like fundamentally designed to not have any distinction between like like what's a full node uh the data is spread around between all of the nodes in the network um and there's uh there's a mechanism for figuring out which nodes in the network are likely to have the data that you need and then asking for that from them so um it's like a big library where we file away information and and you know across the network across all of the different nodes and and mechanism within that peer-to-peer network for figuring out where it's supposed to be and then go finding those people the question is uh can like one node have be history and one node b state that sort of thing um all of the different networks are designed to be uh fundamentally separate so so there's no requirement to be part of multiple most clients will be part of all of them but the idea was to make sure that you didn't have to be part of all of them and so you might end up talking to the same node across different networks as you request different data but they are all intentionally partitioned off from each other so that so that state stands on its own history stands on its own that sort of thing how does spec are they uh together are they is there kind of close collaboration happening um can you be more specific with light client spec because we could be talking about consensus layer les portal networks um i just had a conversation um with like one of the ways that it looks like we can take the light client like the consensus layer like client data and fit it quite cleanly into the constructs that we use to build out the the like an individual network and so it's likely i mean as soon as the merge hits it's basically required for us like we're we're looking at header chain next and follow like header gossip network next and like the there's like a bit of a race right now between like merge and and that and as soon as the merge hits we not only have to do that network but we also have to figure out how to get light client information and it looks like consensus layer like client information fits really nicely into what we already have um there's two different sort of uh it's questionable i think is the short answer portal network is really focused on users and it's unclear whether or not the use cases that were pointed at need those blobs right there their l2 data their l2 chain stuff i mean that's the primary use case that i'm aware of and it's unclear what exactly a user doing wallet interactions needs from that and and so that's like user land usage when we're talking about protocol usage of the blobs the data availability needs are i think strict enough that the storage model that we have in portal network is like maybe not going to be sufficient but really that's research that's happening not by me and by those who are looking at those things but providing the data availability guarantees that are needed there there's like one of the things that the portal network does not do in what we are building is guarantee that the data is there so cool up next is [Music] alex telling you who's up next thank you piper [Music]