Bring Your Own Data: A New Model for an Interoperable dWeb – Carson Farmer

hi I'm Carson from textile and I'm gonna talk about a talk I've given similar variants to before that focuses on just different models for dealing with and better taking advantage of and using user data so if like I said I've done a bunch of different variants of this talk and usually I focused on kind of things from the perspective of the user so like things like privacy autonomy control security encryption and it's super relevant to like a lot of the things that textile as a team builds it so sort of driving force and a lot of the motivation for what we do and usually the like the super short version of that talk goes something like you know personal data on the web what could possibly go wrong and then you know we find out that most of the time nothing right the internet just works we've got servers we've got clients most of us are here because the internet actually does work so everything's great and then things go wrong we've got we get sort of misuse of data users data maybe gets misused we we shouldn't really be surprised though because the relationship that users have with the platforms that they're using especially in a sort of web 2 world isn't fixed and so a good a successful platform can basically flip their relationship with users to be more extractive than creative and then users become the product and that's pretty much the spiel I give and it's like oh yeah that's how we're gonna get an we're gonna convince them that privacy is important that they should be concerned about their data but the thing is is hey I'm talking to a bunch of developers and designers and business people today and be your users probably don't actually care about these issues like we do and you know this graph kind of suggests but this is 2018 was when sort of Cambridge analytic and actually interest in privacy is waning it's not actually going up which is surprising so maybe we're not going to sell you sell people on different technologies and things like that by trying to scare them maybe we need to actually focus on some of the benefits of actually building on top of sort of more decentralized technologies and so one of the things that a lot of sort of traditional app development companies and people talk about is this idea of network effects and you might have heard Anderson Horowitz Alex cam or Alex ramble talked about this Matt tour talked about this everybody talks about network effects you got to have those network effects where no one's going to give you millions of dollars to build your next big startup idea and if you're not familiar with network effects basically the idea is the more data you collect about your users the better product you can give your users which allows you to get more users to collect more data about those users too but develop a better product etc etc and it's this nice feedback loop and traditional sort of walled garden type applications get built this way and so we end up with a bunch of really valuable data constrained into a single app or service with really a limited view of the people that they're actually interacting with so we get data that siloed behind an app there's a very limited perspective on what that person as a digital being looks like but hey we've got network effects and we have all the control so that's great we can just keep on boarding more users and then sell their data to the highest bidder and so we end up with this is a sort of pretty lazy model that pretty much is just going to sell advertising to people instead of being creative about how you monetize a business that you're building something that's a sort of ethos of web 3 technology so instead imagine something a bit more like this where we actually drive app development and app structures using web 3 ideals where we've got decentralized control network impacts instead of network effects and that's kind of what I want to talk about today and then this idea of user siloed data rather than apps siloed data and all of this kind of leads to a broader view of what users are actually do it so I'm gonna start with these middle two and just imagine for a second that we could change this sort of tired web to way of doing things and imagine instead all of your users come to your application with their data right so you have an onboard of them yet but they've got they've gone they've installed your DAP and they come to you and they've already got some data bootstrapped for you and now it's your job to do something really awesome for them with that data and then you know because we've learned our lesson with web two technologies we're gonna make sure that we encrypt and encode and are careful about how we handle that data so imagine a world in which we can already do that what what you know might that look like and maybe it would look something kind of like this if everybody remembers Winamp skin's way back in the day basically it's the same app we now play music for you maybe you downloaded that with Napster or something like that at the time and everybody had their own version of Wynnum what was the same thing and was the same underlying data probably as well and so you start to you start to see this sort of idea that the app is really just an interface or a view on top of the underlying structured data so we basically had these pluggable hackable views of the same app but provided a really highly customized user experience so what about if we built a web that looked a little bit like that what would we need for that well we'd need an interoperable data format of some kind like card the mp3 of the Internet for all the digital data and so this is something our team has been thinking a lot about and we call this idea user siloed data so instead of app siloed data user siloed data data is tied to a user where essentially the users content preferences in one amp one app can actually augment or inform the experience in another app so essentially this idea that each app is really just a view on the underlying data in fact that's what it is right and that's what a computer engineer is really doing we're just moving data around and making it look different in different contexts that's that's what an app is effectively doing right so then what you can get is you imagine you just have a thousand Twitter's or a million Twitter's everybody has their own Twitter interface and if you want you know chronological order hey you should you should get chronological or if you want something else you should get something else or even something simpler than that if all of your apps knew that you preferred dark mode before you even onboard it then shouldn't that just look shouldn't that be what you get right yeah and I've seen people here without dark mode enabled and so those people they should be able to do that if they want even if it's wrong and so this is sort of where the network effects thing comes into play here because imagine your user shows up and they've already got the data that you need there's a lot you don't need to do all that onboarding and have them create their first tweet and all that stuff because they're already used a bunch of apps that generated data for perhaps tweets or photos or videos and this is a this is kind of a potentially big deal for app developers right in a survey that fluence labs and simple ID have recently kind of been talking about 90 is something like 97% of daps have left less than 5,000 users I mean that is not a network effect that you really want to be working with and a lot of those apps they cite onboarding and bounce rates so people trying and leaving right away as a big hurdle to that because Arn barding our onboarding is difficult if you show up to you know a social type app and all you hear are crickets you're not gonna stick around but if you show up and there's already you know your data plus maybe some data that you've created with other apps you've already got that you've already jumped over that engagement hurdle so what we're talking about is sort of like a shift from a monogamous relationship with your apps where it's just you and your app creating data together and nobody else you know what you do with your date your app is is your own deal two more open relationship with applications where essentially every app is is benefiting experiencing from the whole history of that that users interactions with other apps so that's a good thing and so how do we get to this sort of magical world of network impacts and user Seiler siloed shunt sunshine well like a good web 3 company we wrote a white paper on how we think that we should do that and we rendered it in latex so you know it's full of good science and what we do in that white papers we describe some protocols that we've developed for storing and permissioning and sharing user data using content addressable data on top of IP FS and then we also kind of implement as a as a reference implementation a bunch of tooling and libraries to help standardize that data creation and dissemination so that we can actually realize this sort of idea of user siloed data because it's not enough to just make it possible to share data you actually have to start to think about like okay if I'm creating a Photos app I need a fairly a fairly flexible structure in which to encode that photo information so that an another app can take advantage of that and augment that data in some way so we call the reference implementation of this protocol and database threads and so those sort of long and the short on threads is that it's a protocol and a decentralized database that combines encryption structured encoding of that data and then content addressing to make actually moving and fetching that data possible and it wraps it inside of a database like API so that essentially what you're doing is encrypting encoding and adding data type IP FS but you feel like you're using MongoDB or something like that it's open source it's implemented in go and typescript builds on top of IP FS and Lib p2p and I'm going to talk about some practical uses of it now but obviously it's been a bunch of time talking about why you should listen to me so that's sort of an invitation really for you to come you know hit me up or any of my colleagues and ask us about that we do have a bounty as well so and I think we've basically written a like follow these steps you will win the bounty blog post so you should just do that probably but the long and the short of it is we want to make it possible to just add ipfs developers should be able to add ipfs and encrypted to centralized storage to an app to any app without really knowing that they're using IP FS and decentralized storage and adding it to an app or any app right you should just be able to use these tools and use these nice principles to build something really cool and so our team talks a lot about tool building as a sort of we spend a lot of time thinking and working on tool building because you get these exponential benefits right once you have a tool that's easy to use and that works suddenly your creativity can really take off you don't need a wart think about like oh did I pin that hash and did I add that I've and what even is the hash of this thing like none of that should matter and is my data encrypted in a way that's gonna make sense so that the user can actually extract that data within the app later on so here's an example of you know a random react app where you've got views and actions and you've got your the state of that app your store right this is just you know it's the same as in if you're using view or any sort of what you know web app framework similar with react native pretty classic you know view action store set up and the idea is you shouldn't change that right I mean that's how you build apps it's pretty useful apparently a lot of people have spent a lot of time developing these frameworks and a lot of people use them so they might as well keep using those but maybe if we want to sync and backup and store this data in a decentralized way you should just kind of funk plug in something like threads to make that possible and so threads is really designed to work with existing developer workflows and so you know I have it sort of chunked onto the end there and that's pretty darn close to what we'd like it to be and how close it is pretty now and so we build on some awesome tools I IFS lib p2p and IP LD for the linked data structures we use standard encryption tooling we do not you know develop any of our own encryption because you shouldn't do that and all that all of that stuff is handled in a very best practices standard way and it's pretty neat and it works and here's a you know architectural overview diagram from the white paper because I thought maybe this is a slightly technical talk I should put something technical in there but honestly the main thing to take away from this is threads the network protocol handles syncing of database logs or events between peers and then threads the database handles the encryption and encoding by a built-in data structures and if you're familiar with CR DTS then thread supports that in order to handle like collaborative editing of documents than sort of things that might produce conflicts but you can also implement custom encoding or custom codecs so if you've got some very simple application you can use JSON patches or you can use you know whatever it is that you're use case requires it's very pluggable it's very easy to customize in both cases or you can actually host your own IP FS peers or you can use textiles cloud-based ones which is really nice because then you don't even need to have a IP FSP running but in both cases you can actually allocate replicators to in a trustless way actually back up the data that your user data without actually knowing what that user data is so as an app developer you can run your own replicator to make sure that your users will always be able to access their data but you don't know or have any clue what that data looks like so if I'm creating a to-do list and I don't want you to know that I have to get milk later that's fine the users data is siloed and private to them and you're backing up to make sure that it's available but because it's using IP FS and content addressing and they decide that they don't want you to back up their data anymore they can just move on to it different any service or provider no problem in fact they could even have someone storing their data in someone providing the app to them and those two things would work interchangeably because of content addresses so this sounds pretty cool and it's not just sort of pie-in-the-sky ideas you know I've been saying we could and should and users can and will but this is an actual thing that really works we've you know we wrote the white paper and then we actually did the white paper thing that we said we were gonna do and in the white paper is a living document so if we modify things we keep that pretty up-to-date but these are actual tools and technologies that textiles building has built and is building right now and so to give you a concrete example we actually provide something akin to like Dropbox is public folder which I really miss they don't have that anymore but used to be able to just drop things into Dropbox public folder and it would render it like a webpage if there is an index dot HTML file or like a iCloud kind of a thing so textile calls these buckets and you can basically drop stuff in buckets it's replicated and added ipfs and people can access it so yesterday and that is built on threads so that's an app textile built on threads and so yesterday I built a to-do app built on threads that uses structured encrypted encoded user data and I deployed it into a bucket so we have like a thread based app that's rendering a thread based app because you know we love these sort of recursive things so it looks something like this and you're like oh my god that's the most innovative looking react app I've ever seen it's so different than all the other to-do app demos and that's kind of the point right it's not obviously it's not it looks exactly like all the to-do apps out there the only difference is I'm using mob X as the state management tool but all of that is replicated and synced across any peers that are logged into this and it's published a demo textile cafe which is a bucket that I have which points to some some actual data on ipfs and then I just pass in an ID with the key and to kind of cheat but the idea is you know any identity that you happen to want to use three box or simple idea or whatever that your users use those identities to help create and manage their keys so that they don't have to deal with that stuff and we spent a lot of time thinking about sort of key management and avoiding you know logins and specific user IDs associated with textiles so if you have questions about that I'm not going to dwell on it here but I'd love to chat with you about it because it's a thing we spent a lot of time thinking about but anyway you know this is a simple little app I could have made 75 of those apps each you know doing something slightly differently rendering a basic sort of model-based structure in different interesting ways and it would be the same underlying data that me the user can bring and modify and use throughout so you know to kind of if you take home one thing from the store a few things from this talk you might take home personal data owned by users through encryption good and protocols for sharing from it and permissioning that data and services also good and just the overall ethos of interoperability and user choice good and the idea here is right we if a lot of apps adopt a similar type of architecture then a lot of apps will benefit from the other apps Billy and it's this sort of like open source ethos of tides raise all boats but to make it a little bit easier to do you know textiles here we've got tools and technologies that you can play around with to actually start implementing and building some of this stuff so I've tried to leave myself a bunch of time for questions because I didn't get very technical but before I open up for questions one thanks for showing up slightly more of you arrived throughout so that was good also it's Saturday but you still have time so we do have a bounty we're associated with these sort of ipfs and protocol labs crowd and we have a couple of bounties if you go to that centralized website there you can find out more about it basically if you just build a thing with textile we would probably give you the bounty so you could just try that and in fact if you deployed something to a textile bucket it's also deploying it to ipfs so you would probably kill like two bounty birds with one stone that way so you know probably should do that so if you're interested in that I have a couple of colleagues here you should just go and make them listen to your app idea and then make them tell you how to build that and then when the bounty and you don't even need to give them any credit right so there you go yeah so if there's any questions I also have a few technical slides if you want to get into details yeah hold on well pause for mic thank you so I see the point with a to-do app where users use the application to modify some data that fully belongs to them and then they return and they can just get back but most enterprises like the Facebook's of this world bring to users a lot of value by collating a lot of data summing it up right processing it in in chunk or creating a links and graphs and you can't do that if everyone has their information or at that point they need to share it someone will then aggregate and provide yet another level of value so how is that possible in it with such a method yeah that's a great question so and a good summary of that is basically like you lose or you could potentially lose a lot of data linking if everybody's data is truly siloed and we use the term data silo because it's the opposite of sort of apps siloed data but in reality what this is is the sort of IP LD data structures that we talked about a bit in the white paper of why you should use these types of data structures they're linked to data structures so in fact you can still create linkages between data across users so for an example say you know we used like some sort of a social network II type thing right so say I have my user profile and I've got a friend who has a user profile and they posted something interesting and I happen to be following their their feet and so far none of that requires any linkages across things I could just view their public feed they could be my public feed it could still be fully siloed but then I decide that I want to like a post in their feed now I have added to my user silo data a linkage to content a Content address in their feed which is a CID in ipfs world it's a Content address and it's encrypted encoded on my side but the CID reference is a unencrypted CID that I had access to because I was able to view their public feed so now if someone comes and views my feed and they go through and they want to check to see who you know what I've liked and they also happen to be friends or happen to access that other feed they would actually be able to traverse that like to actually access that data so you can create cross silo linkages in that sentence now the real so ok so fine that's like a really simple example of that now you've got millions of linkages between things how do you handle that in a scalable way and I think you can't handle it you know like I'm not good not every user is gonna sink the entire sub graph that they're interested in although they can do that in theory but another way that you do that is that you actually start to have services that are built on top of this and we've been you know things like the graph and zero knowledge replicators can actually help with some of that so the replicators can follow linkages but if they have the right permissions so if the user has permission for them to do that so that they can follow linkages between connections but not necessarily what those actual connections are so now suddenly they can actually index linkages across large swaths of the sub graph without necessarily knowing the specific pieces raw data bits of that sub graph and they can provide search and indexing features on top of that so then you start to see that there's actually maybe users want to pay for that type of you know like discoverability you want to increase their discoverability or they want to be able to find more users on a particular topic and so you can start to imagine now definitely textiles not built this stuff yet but you can start to imagine these types of services building on top of a protocol like this okay and I'm free we're out of time out of time okay well thanks showing up and good luck out there [Music] you