Streamr Network 🛠 Incentivized real-time data flows using the Data Union SDK - Matthew Fontana

foreign [Applause] [Music] thanks for coming um so there are two parts to this session I'll explain the streamer Network and Nico we'll explain the data unions that run on top of streamer so there are two separate prices one for all good uh one for uh using streamer one for using the data Union Dao so I'll go through what is streamer how to use it I'll then like talk through some ideas of what I think could be great hacks and then on to Nico for data unions so what is a streamer so it is a decentralized peer-to-peer Pub sub Network it means that data gets streamed to any number of ethereum accounts in real time and the role of the network is to move data from the Publishers to all the subscribers over a decentralized networking topology so if you're familiar with Pub sub channels or topics streamer has streams and they're they mean the same thing and can be any number of Publishers any number of subscribers uh protocol scales based on this principle of shared responsibility where the users contribute their own bandwidth to to support the the topologies so yeah this is more a definitions uh slide so we get asked these questions a lot uh it's not a blockchain so it's a classical networking Technologies but it does have touch points in the blockchain so it's an ephemeral data transport but we do also have support for data resending through the storage nodes that we run and note there is a liveness requirement for this so the storage nodes store the encrypted data but you need to have the Publishers online to be able to do the key exchange so that's one small gotcha that you need to be aware of but essentially the streamer Network provides this uh infrastructure for many to many data Pipelines our access control is online so this is one of the main touch points in the blockchain all the data is cryptographically signed every single data point signed by the private key of the of the user and comes with end-to-end encryption out of the box if you choose to use it it's also free to use as well with the the slight caveat that you just need to pay are sent to a Ascent or two of for gas to create the stream in any sort of edits on the permissioning layer the on-chain access control so in the web 2 world what we're replacing is this centralized data piece the the data center so to move a lot of data around in in real time uh the the pub sub model is to decouple the Publishers from the subscribers using uh you know some beef Beefcake server in the middle that can scale up and and and handle uh you know anything that comes at it but of course we know that this is centralized and we don't want to build our apps that are dependent on Google Cloud for example so this the streaming Network looks quite different and that it's it's uh appears more as like this if if the web two-way was this vertical flow of data through data centers you know run by the Silicon Valley Giants the streamer network is uh the data flows horizontally through this overlay mesh Network that is made up from the users of the network for for users that are intrinsically interested in the data from the from the topics or streams of the network so at this point it's good to jump into an example so this is the streamer.network website it's just like any other website loads JavaScript and loads image assets from a centralized point but if you scroll down down the fold you'll reach a data feed and what's interesting about this data feed is that it's a stream on streamer and the data is not coming from a centralized Source it's actually coming from other visitors that are uh on the streamer Network website at the same time as you so this is a peer-to-peer data feed inside the browser so while you're visiting the streamer.network website you are becoming a light node in the system so this is the network explorer that so I've plugged in that stream ID in the in the top left there and what it returns to me is the visual representation of what that Network looks like so in one of these is the actual uh publishing script but the rest of these nodes are other visitors on the uh visiting the streamer.net Network website and more likely than not this is where you are pulling the data from okay so on to the access control so um the on-chain stream registry is on polygon uh this is the network source of Truth so it defines who can do what on the network um so on these streams you can typically publish or subscribe and the uh the network looks to the blockchain looks to this registry to determine who can do what and it also unlocks some interesting smart contract Access Control policies so this piece this module if you're familiar with the lit protocol it's quite similar we rolled our own permissioning system for the network but perhaps it could be composable with lit as well so this is an open question maybe it gets answered in the uh hackathon this weekend so here is the like a different view of the access control uh fine-grained permissions published subscribe changing permissions edit and delete and whatnot and yeah like I mentioned you can you can start to get a bit creative with these access controls so you could have a token gated uh Pub sub channel uh perhaps there's a stake to Pub sub um different kind of uh Primitives that you can bake in to the access control and on our Marketplace that Nico will will speak to more we have time-based pay to access subscriptions so this also composes to the access control you pay tokens to uh get a time-based access or subscription access to a certain stream so getting into the tooling now so the streamer client is your best friend to to make all of this work so it's designed to be this wrapper that interfaces with the on-chain stream registry and the network and it bundles in the light node as well so it really kind of uh takes care of you from all different angles so you will need a little bit of Matic to interact with the stream registry and we're happy to sponsor this of course so just reach out to us if you need any Matic uh and it's just an npm package so it's uh these uh the stream of client runs in the browser as well as any sort of node.js environment uh also a shout out to the examples repo has a bunch of uh code samples to learn from as well so if you're stuck uh head there and we also have we also make it really easy as well so we can also create a stream using a user interface so just using metamask or whatever wallet that you have available you don't even need to touch code to be able to create streams and edit the access control so it's a nice way to get started I think so this is what's called The Core app you can visit it at streamer.

network forward slash core and it's yeah super easy so this is the stream ID that I'm configuring here so the ID is made up of the um the ethereum account that creates the stream plus some path name and this can also be your ens name it will kind of um uh scoop this up from the ens registry and in the share settings sidebar you can edit the access control but you can do all of this within the stream of client as well and we have docs yay and uh so the the client is uh JavaScript implementation so if you are building uh or integrating with a different language there is a solution for you here we have what's called broker nodes so this is you would need to run a broker node and open up its interfaces so it has three interfaces interfaces websockets HTTP and mqtt and just about every single language uh ever written has these sorts of uh libraries built into them so you can publish And subscribe push and pull into the network using these interfaces if you run a broker node so this is an option if you want to work with a different language for the most part so we have this kind of Gateway into the network if JavaScript is in your thing uh so what can be built uh with the with the network one of the most native and most interesting use cases I think is uh like a decentralized group wallet chat so this is using the network for uh decentralized uh real-time Communications between ethereum ethereum identity wallets so a chat room is just a stream and the participants in this chat room uh have publish And subscribe permissions to write messages these messages are being signed so you absolutely have full proof that there you are talking to who you are talking to and there are no servers here so um you know if you if you don't count whether whether JavaScript came from from the single page application this is a totally serverless experience so um yeah extensions to this creating token gated chat rooms etc etc token-gated experiences this is quite a quite a powerful uh primitive to to build on okay so very long list does it yeah it all fits in luckily um so so what what should you build with streamer um there I think you know obviously your creativity should should guide you but here are some some Concepts that uh often come up as as useful so like I think I talked to the access control policies for streams you can you can do a lot there decentralized metrics Gathering and web3 open data sharing so this is about collecting fine-grained metrics from other decentralized systems and using the network to interconnect and and kind of improve on decentralized systems without having to reach for centralized solutions to be able to to monitor the health of of the network in real time so I think this is quite a rich use case and it can be stacked with many other networks that are also sponsors here today wallets wallet Communications I showed you the chat app so this is this is what it is essentially off-chain multi-sig chat and conviction voting so imagine the chat out but with your multi sick participants from gonosa safe for example you can build conviction over what to vote on and and so on in this off-chain experience group chat plus lens protocol I think this would be fantastic protocol stack interconnection so if you have ipfs and it needs to talk with live peer or whatever then you could use streamers this middleware to have this decentralized communication between your your stack now we have decentralized wireless network proof of coverage so this is a bit of a a long shot for a hackathon but something to think about if you if you can uh create kind of a coverage map for decentralized wireless networks then um yeah it can feed into the proof of coverage algorithm so this is fantastic uh first mile data transport for uh from data providers to oracles so this is um from the provider up until the Oracle the streamer network is a great technology uh solution for this Redstone is is one of the the kind of Next Generation oracles that are using Us in this way uh multiplayer gaming real-time State share I think that's one it's quite clear and RPC load balancing which is not clear so I will uh speak briefly to this one so this is kind of uh just an internal idea that we have to scale the RPC uh centralized point in many blockchains uh so yeah typically if you have this very popular dap and they're polling the RPC it's quite a heavy load for the RPC 10 000 users 10 000 requests a second if it's uh fast polling for example so that's not great what we can offer is to use the streamer Network to uh sort of load balance these uh RPC calls so how this would work is you would have these kind of Watcher nodes that would do the RPC request as many as you want for as much decentralization as you want but then your dapps would be uh interconnected and kind of uh pulling in the data that originally came from that RPC load sorry that RPC endpoint they would be getting it from each other so this is where the shared responsibility of the streamer protocol comes in it's kind of similar to what we showed with the streamer.network website where you're getting the data from your visitors so in this way we take all the pressure off that RPC endpoint and you can kind of start to imagine other sort of use cases where there is this Central uh Point uh or Central data uh publisher that can get overloaded so if you have this system where you sort of protect it with this with the streamer Network where the messages are coming from the other participants it becomes quite a uh an easy scale uh solution so yeah this is something to think about okay um moving moving on to data unions and Nico will take over awesome let's see if I can get this thing a bit down because I'm not as tall as you are uh thank you Matthew uh hi guys I'm Nico I'm the Deborah engineer for dating in dell and Matthew just told you how you can handle all the data stuff and I'm going to now tell you how you can actually monetize this data so let's jump in so first of all how you monetize this data you monetize it with a dating um what is a data Union it's an organization with an application where individuals participate in the creation of valuable data so what that what does that mean it means members generate data through data units application which crowdsources data and in the streamer context this data can then get sold on the street on the streamer Marketplace and I'll talk about that more in a minute um about the marketplace and with the dating and you can essentially um monetize your data streams so with the data the dating endow offers um the data Union framework and this contains a smart Cloud track template that handles payment distribution for the data Union and a typescript based SDK that handles all the interaction with uh with the data Union or with the smart contract and this SDK is an npm package and I will talk about that more in a minute and we also offer server architecture for your member management so let's see how this payment distribution works so you can see here wait do you see if I move my mouse no okay well you see this uh Center contract there and as I said it is a smart contract um and essentially money gets or token gets sent into that contract and then distribute it into the different members I'm just going to talk you through how this kind of works so you've got an admin and that's going to be your project right so you have an ethereum address that publishes or deploys that contract and that's going to be the automatic admin and this admin can then add members so you essentially the data publishes right and um then the admin can set a admin fee which will result in this uh I don't know if you can see the 30 for example um and you can configure that in a way you want really um and then if token gets sent into that contract the admin for example gets 30 and your data members will share the 69 percent um of those incoming tokens 69 because uh one percent of that um of the 100 go to the data Union doubt and we use this one percent for uh funding new projects you know going to hackathon and those kind of things um yeah and once the token gets sent to the smart contract so that could for example happen on a marketplace right so uh your data get your someone pays to be able to subscribe to your stream and those token then get sent to that to my contract and then again your members are able to withdraw their token or they can just accumulate it over time and take them out whenever they want really um right this happens uh so we have the smart contract and we have the SDK and the SDK enables you to interact very easily with the contract um as I said it's just an npm package and you can really just you know it's a it's really those few lines of code we have more interaction with the contract with this like the essential lines you deploy you can deploy the data Union with the SDK you can set the admin fee you can add members remove members and also withdraw your tokens from there um alternatively you can also use the stream Accord Matthew briefly talked about this um and it's it's essentially front end where you can deploy the data Union and it's got a nice interface where you can add the name um you admin fee and and like and even your streams that you want to monetize um right and then about the marketplace you can then publish the state union onto the streamer Marketplace so they're like different data products where all the streams are set um on the marketplace and then uh data buyer can subscribe to those streams for um token like they will pay for it by the way those token don't have to be the streamer data token that can actually be your own Uh custom token um right we also have documentation there will be a little bit of a guide how you can actually deploy a data Union um one way is with streamer and the other way is just programmatically with the SDK um yeah an example would be for example Ford 18 is um swash so swash is a browser extension it's very easy so you just install your swash plugin into your Chrome browser um join that date Union and then you can share your browser data and earn by sharing your browser data so it's actually quite clever um and then if you're looking if you're still looking for an idea what to build uh amra our community manager he uh did a great blog about some ideas that you can build at this hackathon so feel free to go to this link and check it out I highly highly recommend and yeah we're giving out ten thousand dollar in prices so one track is building with streamer and the other track is building with data Union so I really really recommend integrating this into your project and yeah I think it works very well together streaming data Union anyway and if you've got any questions feel free to reach out to us on telegram you can also join the Discord server of data Union or of streamer and we'll be present there too and of course the one of um of if Global as well cool right do you want any do you want to add anything okay so if you've got if you guys have got any questions we're here to answer cool uh no we don't no not at the moment media sponsored yeah we can we can we can we sponsor we sponsor the transaction fee to actually deploy the contract uh and and their stream yeah um ideally you do it so if you build with streamer I would recommend doing it on the front end and then you will you will see like how how you connect at the payment and how much it costs and all that kind of stuff and then publish it to the marketplace paid there can't be any kind of like a call to a smart contractuality there's opportunity yeah uh so this is this is kind of like The Cutting Edge of what we have uh out of the box on the access control so we do have it's basically in PR at the moment for a very complete code example that is on the uh on the chat um on the chat uh repo so you will need to kind of uh dig there and we have like the contracts that do talk to the access control there but this is like very uh well I'll just say it's not documented at all like this this token gating uh kind of uh twist to the access control it's not it's not like uh the the core functionality of of the network so I guess we we have some catching up to do on that documentation but um uh we have accomplished kind of feature completeness on the chat app that I showed and it's that's open source and uh so there's a few caveats here that um the the the chat app um token getting is just for erc20 tokens on the polygon chain and there's a PR out there for nft token getting so it's yeah it's very fresh and very uh hackable to to kind of yeah build build this in uh but it's it's not quite as easy as like just uh taking Lego blocks at the moment so we have the examples there but it's um it's quiet Cutting Edge uh I I think it's too hardcore for me [Laughter] I think the access control uh you would need to move to to Mumbai as well so if you if you're super Brave yeah don't recommend that we can sponsor the transactions it only costs a cent to um to do these things so yeah very happy to support that and we will have a full native sponsoring of of all transactions on streamer it it's just not ready uh this weekend so any other questions okay well thanks for uh staying up so late and uh yeah I really appreciate it and uh please reach out to us if you have any questions thanks [Applause]