Data is the Foundation for Dapps - Panel

uh my name is jay jay harrell i work at kwan stamp uh we're a bunch of auditors and uh i've been in space a long time i care a lot about blockchain i care a lot about um stateless or statefulness future so we'll see and today's panel is uh called data is the foundation for dabs and so we're just gonna go down and introduce everyone all right testing okay hello everybody my name is daniel olshansky i'm uh i'm a protocol specialist at pocket network i primarily focus on our consensus algorithm and what we do is we're a decentralized multi-chain um rpc layer so in short what that really means is there's a lot of node infrastructure companies out there what we do at pocket network is we are a protocol then incentivizes everybody else to be a node infrastructure provider of their own depending on which blockchains you choose to support i'm josh urath i lead product at a protocol called anchor we started off as a centralized service provider for web3 nodes recently have transitioned to a decentralized protocol very similar to pocket we not only offer the rpc layer we also have liquid staking products and we have about sixty thousand eth staked on our product product right now and we're one of the validators on on b b chain as well hi i'm brene wilson i'm a senior engineer at covalent and covalent we are the decentralized api for data across all on-chain for evm and non-evm i'm happy to be here hi everybody uh my name is albert i go by that guy in tech in the metaverse um at alchemy i'm a developer advocate at alchemy so i focus on content community and conferences um and at alchemy our goal is to bring the power of blockchain to a billion people and the way we see that happening is a kind of cycle of life of product development you have users but in order to get users you need to have really good products in order to have really good products you need to have really good data and infrastructure and so at alchemy we're trying to provide that data and infrastructure so that engineers builders can come into the space and have all the tools that they need to build the really great products for users so really excited to be here with this panel okay so an underlying pain point developers have uh is that dapps need index data to develop complex queries so they can create robust user experience in a trust minimized way what are examples of ux benefits that users would see and experience uh with this data so so um i think there's there's so if you're not familiar with how the blockchain works basically you have validators and you have full nodes and there's something called archive mode which is a complete historical copy from the first block all the way to the latest block and so that's usually known as on-chain data and we have a lot of services that are working on caching data maybe covalent could tell us a little bit about some of the index but a lot of people also just pull that data directly off the archive nodes yeah so at covalent we have this philosophy that it's all on chain data it's it's public and it belongs in the public space and so what we're working on is a framework to be able to take this data and give it back and so providing the tools to be able to extract this data by patching the nodes at the very source and what we what we call this is a unified data export for all on-chain data that that which basically unifies the schema of on-chain data and then makes it accessible through a unified api and so this unification across all evm and non-ebm is what we are striving for and the schema is what basically does the trick so we are like all behind on this yeah so i'll jump in when it comes to indexing data i spent a lot of years working in the machine learning ops space and when you index data it really depends on the use case of that specific application so at pocket network we're essentially a proxy rpc layer for any other full node or icon archival node that you might need but depending on the needs of your application you may need to index it in some specific way or load it what's called etl you extract it transform it and load it in some specific mechanism and schema that's near for your app if it's very standard there could be potentially already a service provider that index pre-indexed it for you the way you need it that's it makes things very easy it makes it very easy to bootstrap but then you are obviously losing some trustlessness and some permissionless because you're putting the in you're putting trust of indexing on someone else so there's a lot of a lot of different angles and really depending on what your latency is how much how much resources do you have to really pull all the data yourself and index it yourself on your local database um it really is a spectrum rather than just one solution yeah yeah one one thing we've seen at anchor is as uh that there's there's many different protocols and d apps that are pulling in all this archive data and we've just needed insanely good hardware to support the archive so we've built a custom bare metal server that we put all over the world we got like 30 terabytes of nvme storage on each server we get crazy good performance on that so i think we're addressing we're not using like the the cloud providers to do that we're literally customizing the hardware to kind of fit these use cases so if you look at the sort of data warehousing stack you have the nodes at the bottom layer then you have the nodes that create the data and extract it so that we call the uh we call this the ingester and then on top of this is the actual api that is able to serve those requests and you can perform optimizations at any point in the stack obviously it makes sense to optimize the db for as many calls as possible because at the end of the day is the data that you're providing and at covalent um we thought of this in a different way uh compared to traditional sort of etl pipelines which is extract transform load we do a extract load transform so we load all the data on chain directly as much as possible with high fidelity and transform it on runtime so we don't have to worry about the extra cost of transforming it for separate use cases separately we just have everything on chain all at the same time so i'm going to put this question to albert with how fast the space moves developers want to be able to go to market very quickly how does each of your projects a direct or how does how does your project address this uh challenge so the challenge of trying to go to market quickly is the question yeah so dap developers want to go to market very quickly so how do they address this definitely that's a great question um so when alchemy started in 2017 um one of the biggest problems in the landscape was by talking to hundreds of developers out there who are trying to make something useful with this technology there was really a lack of tooling a lack of ability to read and write from the blockchain and so alcme decided the founders originally wanted to create this kind of like you know decentralized network everybody can provide this type of service but what developers really needed and what they were telling the company is that they wanted a scalable reliable and data consistent way to interact with blockchains and so that was really the first thing that alchemy had started building it's called supernode and it provides all of those qualities and people have really enjoyed that service and then once that started working really well then we realized okay there's all of these different forms of data that people might want to use and so we started building enhanced apis these are higher level abstractions for example one that came out recently is the nft api so if you're a developer and you want to make a gallery a marketplace a wallet whatever it is normally you would have to without supernode infrastructure without these apis you know spin up your own node wait for the blockchain to sync that could take days it's really annoying and then after that you have to read all the data about the nfts that you want to show index it yourself that's also really annoying and so by having these apis you can actually go to market really fast because uh you know it's as simple as switching out your rpc link to connect to the blockchain and then querying an api endpoint to get the nft data that you need um so we're the number one thing that drives us and i think all of us probably is just moving the space forward as fast as we can and that requires having really good tooling so yeah yeah okay so uh thinking about node infrastructure and spectrum of decentralization uh how do we mitigate the risk for developers and consequence uh consequently users of their dapps if they're relying heavily on certain entity for blockchain data so the question is for blockchain data specifically or so um i think the question is more about the the abstraction layer okay so what i think is at the moment not to go into meme land but it is still early in the sense that there isn't there aren't really any super mission critical applications built on the blockchain my insurance my identity i can go outside and even if there's a chain halt i can my life still goes on maybe at some point in the future where my car might not start for example if there's a chain health or if some uh the data integrity of a certain piece of data that's necessary for the ignition to go on um is quite critical that's where the risk really starts kicking in so today um i completely agree that tooling we need we just all need to move as fast as possible enable developers as fast as possible but when we get to the point where our livelihood or uh really like mission critical data is dependent on the blockchain then there's a risk will come in and that's when we're gonna really need to approach true full decentralization and um which will put the onus on the developer to start putting more effort and thought on indexing um their own indexing things using their own infrastructure yeah so so one of the things i've seen i agree like we're still pretty early on in the space but we've seen incredible growth in emerging markets um like brazil uh nigeria and argentina in particular we served like a billion requests from argentina the last 30 days and the pa the argentine and peso is down like 20 so i feel like we're seeing defy is being adopted as critical banking infrastructure right and that requires like on-chain data right and so um i just think that's incredible and i think on-chain data is going to be very very important for enabling this future where d5 becomes a financial system for the rest of the world so certain use cases have more data demands than others and we have started to see like the marketplace demands um the demands coming in from like d5 exchanges and also uh like uh just doing like forensic analysis on this data having extreme good use cases at covalent and we've been growing like crazily over the last year with like more than 500 laps doing all of these cases and because we are use case agnostic we start to keep a tab on what's happening in that space and because progressive decentralization is the way to go for this kind of data there should be no central point of failure so it's really important that the nodes that are able to extract this data and maybe be able to index it and further able to provide these sort of queries at runtime is something that the developers can themselves are run on their sort of infrastructure it has to be optimized to that level but maybe not for like bigger chains which hold like maybe two terabytes and more of data so it's also about like node optimization as much as it is about storage and query optimization yeah and another thing we've seen too at anchor is that a lot of projects aren't using just like one node as a service provider they're using multiple so we see some of our customers are also using pocket some of them are also using alchemy they're kind of comparing and constantly testing the results back and forth i think that's ultimately good for the space right we don't want to be relying on one single point of failure totally okay so what type of challenges do you all see from each of your perspectives in a multi-chain world so it's in hack for example um i can answer that so so one of the challenges we just did is at anchor uh we helped we contributed to the aragon node client and we uh ported that over and enabled it for binance smart chain and so as you guys know with ethereum there are several different node clients there's aragon there's another mine there's gas a lot of the emerging multi-chain space just has geth right now and so i think like one area of opportunity for our space is to continue innovate on the node client to bring more node clients to other chains because that will ultimately help the diversification of the network and the decentralization of each one of these other multi chains so one of the things that i think will be a really big challenge in the future is right now when we talk about bridging uh basically just bridging assets from one chain to another that's already a huge undertaking with a lot of risk uh that involves a multi-sig and potentially a little bit of centralization when we take it to the next level and imagine uh again i used to work in virtual reality and autonomous vehicles so a lot of image data that really requires a lot of processing when we get to the point when we go let's say just past d5 or just past assets but we actually need to cross chain image data for example that is going to be a challenge that i haven't even started thinking about too much yet because the amount of indexing we need to do the amount of processing we need to do schema interoperability it's just going to be very complex but when we do get there i think it'll be a very exciting step for the whole ecosystem we think multi-chain is the future there's no way to avoid it but multi-chain is also one of the major sources or attack vectors for data corruption and data duplication and uh just uh wrong data and so you need to have really effective protocols messaging protocols underneath on the networking layer that are able to bridge these multiple chains and we're working with nomad xyz to try and understand how we can do this best so that everything that's created on one chain is not duplicated but you have verifiability at the end of the day so the verifiability is the key aspect for multi-chain and as long as you have a protocol that enables that you're good to go um alchemy really wants to basically support wherever developers are so i definitely believe in a multi-chain future as well um so wherever people are building we should be able to provide the on-ramps and the off-ramps for for people to get involved um me personally i think bridging is is going to be a big topic that we are exploring also from the end user perspective wallets so for for bridging it's really important that all this all the data that's being captured across all these different chains can talk to each other can interact with each other but also as the user i don't want to have to think about which network i'm always connecting to and do i have my assets here or over there and if i'm having an identity that's being built up on one chain can i port that over across all the other chains i think that's one of the promises that we've had where it's like you know we have this blockchain this public ledger everything's going to be open and public but now we over time we might have fragmentation because of all these different use cases for different blockchains so how can we make sure that we also unify that you know in the future if you if you think of proliferation as multi-chain then think of consolidation as a messaging layer between them that unifies the data so there's no corruption okay so um what do each of you think uh are the most pressing issues to solve in terms of blockchain data or the availability of data i think i think decentralization in the infrastructure layer is so important right we're seeing a lot of centralized service providers kind of come out i think i personally have talked to some of the largest cloud providers who are all have a web 3 strategy right now and i think it's important for specifically around the community to be involved in governance of the infrastructure layer and really um you know be involved in kind of like how these platforms are are governed and progress and so that's something we're focused on at anchor is really enabling that because infrastructure feeds the rest of all the applications and the d-apps and all that yeah i would also say that kind of with a great power comes or with great ownership comes great responsibility so so as we one of the big things about blockchain and decentralized world is uh that users will get to take control back of their data right and what does it really mean it means you get to monetize your data it means you get to decide who has uh whether you want the data to be confidential or anonymous or something else but that doesn't come easy um so it's a very basic example most of you may be using a password manager of some sort uh it's taken me a very long time to get my parents to use a password manager as easy as it is so when all of us control all of our data that is um that is a lot of mental effort on a day-to-day basis so it can i don't know if we'll ever reach the point where all of our data is fully decentralized maybe in a couple hundred years but that's way far into the future but i think one of the biggest challenges isn't going to be just making it accessible to developers but primarily educating the users that they're in control and how much effort they actually have to put into controlling it and deciding who has access to what parts of it and in what fashion i agree up to a lot of the points that danielle made here i think um teaching developers how to like use the api and how to use the queries is really important because there's billions of rows and it puts a lot of infrastructure pressure for every query um so optimize queries are the way to go the other thing that i think is important is not having personally identifiable information or personal sensitive information on chain and making sure that if it's being accessed that there's an obfuscation layer for this some sort of privacy layer for any such data that might be accidentally put on chain because these are sensitive details and they have real-world consequences so yeah security and privacy along with the education i think is really critically important um speaking of data and education i i remember this funny story of one of my friends he was doing a like a thesis project as a an art major but he wanted to do like a technical like website kind of a thing and what he explored was he made a website where you could trade your personal information uh for like game points on the website and he just wanted to see how willing are people to like just give up their personal information before this made-up point system and a lot of people actually interacted with this site you know to get the points so i think there there definitely is a role for all of us to play in educating the developers who are gaining access to this data and then creating products um to make sure that they know how to behave with it ethically but also for the users to know like here's what you're trading when you you know sign a message or you know potentially reveal some information about yourself um you know it's going on chain it's gonna be there forever uh how can you manage all of that so yeah that's something i hear about a lot don't paste your private key on chain okay so what does the future look like uh scaling i mean you know we're seeing uh tons more daps like there's 15 000 people here um what does it look like what's where are we going okay so to touch back on some of the topics i was talking about earlier that everyone is going to need to index and process their own data based on their own apps needs i think where the future is going is uh all node infrastructure is gonna eventually become truly trustless and permissionless and whatever rpc request that you make to get data for your uh d app is gonna be um you can you can count on its data integrity but to some degree you're gonna have to pull all of it locally or whatever subset of on-chain data you'll need locally and then manage it yourself so i can see a lot of developers becoming more experienced at doing customized indexing or customized processing because there's a lot of cool projects out there like homomorphic encryption where you can do let's say you can do multi-party compute never have access to the data but that can only go so far and only when the use case and the data is very well known ahead of time so over time i think it'll all developers will be taught to customize it themselves yeah so i think one of the big issues is that when you build d-apps and you're pulling a lot of data at scale it can get very expensive especially on centralized platforms and so um we've seen you know uh platforms that have paid enormous amounts to centralized service providers that have wanted to move over to a decentralized system just because it's it's cheaper and the more the community is involved in that and i think that's going to be a really important thing is we've seen traffic explode that you know tool uh d-apps and different protocols are looking at different options for that the verification aspect is is critical as we progressively decentralize um we find that more and more sources are available but is there a way to verify that those sources are true or whether they're you know um playing with each other to like obfuscate the data that's a that's a certain possibility so there needs to be mechanisms that we employ within the network or within the node infrastructure layer into the distributed set of nodes that we're able to find whether any uh sort of malaction has been performed by any node operator right and so this verification process has to be really well done and we are using like a delegated proof of stake with like a distributed set of validators to do this and the the continuation of the chain requires that every piece of data that's that's created on chain and then parsed is verified before it's like ready for query and this should be critical so how about data that's not created on chain again let's say we go get our driving license at some state right are we just gonna always trust the dmv of our our local dmv that this is it and they submit it on chain like who is gonna verify i think that's why decentralization is important right we need people in the community like checking other things we need uh different service or our different node providers uh adding all adding this data and the community needs to understand this and see these tools like it they should like the community and shouldn't just be like trusting one centralized service provider for all this data like there needs to be like the community involved in governance and checking things and checking that there's not malicious nodes on the platform or maybe the rpc provider is doing front running or you know there's so many issues that can come up through the centralization so i think it's time we've we've kind of scaled to a point where we have so much more traffic like anchor saw 20x growth in 2021 on our rpc endpoints which is incredible a lot from emerging markets and that like like pocket said it's there's a lot of responsibility with that oracle's oracles are the way to go like uh has as many oracles as possible but the different coordination mechanisms between articles to make sure that each of them have the right data um and because um oracle's get off-chain data on chain it's really important that like for example uh if they're making comments that there's a second party that's verifying them so trust but verify for off-chain data and have oracles that are trustworthy and you know like they're up and running i just want to add a point like um i i think vitalik recently wrote a piece about soul-bound tokens and this is kind of a different perspective here where it's like more about uh individual identity and reputation on chain uh there's this interesting use case of nfts where if you make it such that the nft can't actually be traded once it's in that person's wallet you can actually imagine using these nfts over time to kind of track off chain contributions work that you've done so maybe the dmv will actually you know adopt some kind of these these types of non-transferable nfts and they'll give you you know your i guess license and then they as an institution that the community hopefully verifies and says hey we do trust the dmv uh they can give a stamp and say yes we we this is a real driver's license that this person has um so i think stuff like that is really interesting uh and then also uh with regards to the original question of like the challenges of scale um scaling when it comes to like a lot of data being accumulated um one of the things that i've found really pleasant as a developer is having um the ability to actually get notified when something happens on the blockchain without me having to like pull for it or watch for it all the time and so one of my favorite features that i've used at alchemy is the alchemy notify web hook system so you can basically put in like a wallet address and then we'll let you know whenever any activity happens and you can filter by certain types of activity um and then also like gas price notifications things like that so i think features like that will also be really important for developers and users as the amount of data scales we need to be able to get to the data that we need really fast so we got three minutes left so who has the oldest nodes and whose data is it the oldest archive nodes ethereum who has the oldest archive was the oldest archive nodes in terms of a node provider or who can provide the oldest analytics for state history pocket network state changes that is i mean i mean i don't know how to answer that i think um we have nodes that have been running since like 2017 but i'm sure they you know that i don't know they're all that gen they all have all the box the computer on which the genesis block was mined whatever whoever's there is satoshi yeah i don't think we have a node from like you know 2015 that's running right now well yeah so at pocket network we don't know who has the others node it's fully decentralized we don't run our own nodes so uh someone out there is running pocket validators or running ethereum archival nodes or just ethereum nodes behind behind their pocket notes so it's really up to us it's up to the community um kind of true progressive decentralization to let us it basically what it does it lets us decide in a decentralized manner what the oldest node should really be um yeah i'm not sure what the answer to the original question is um but another favorite thing that i uh another good thing that i enjoy about alchemy is that uh in our free tier you have access to archive nodes so you know if you wanted to explore for yourself uh you totally can yeah okay so we're at time thank you everyone so this has been a chat about data is the foundation for dapps and we're all at eat denver on saturday night [Applause]