Look Ma, No Servers! - Colony

alright sorry for the clickbait title but I think it kind of and maybe it made a lot of sense to me so when I when I presented one of one of our like data solutions to a colleague of mine he said exactly that he said nope no servers and I always feel like yeah that's nice and you've done something great but it's it at the same time you also feel like something something terrible is going to happen after that so people probably gonna get hurt so you might not want to just just just do it like this so what I what I feel like what we will be doing or what what what like we in colony are doing a lot of times is which I on new stuff we're trying to be cutting-edge but always like we always put on training wheels for that so we have like like a safety net to cover us and something was wrong or in that case we're not we try to be deceitful as as as you said wise as possible but in the sense in anyway and I'm still wearing those so Who am I I work at colony I basically do all kinds of things over there except for writing smart contracts so I basically do all kinds of front-end stuff over there and the past couple of months have been mostly busy with getting our data I had to work which is based on I believe that's an over TV so what we trying to do is as I said we try to make things as decentralize as possible that means currently in our current state I don't know if that's going to change anytime soon but in the current state we don't have a server back yet so everything happens in the front end and everything is connected to various services and be just using yeah we try to make things as descent wise as possible and we connect to ipfs and use as many player to peer applications as we can so what I'm what I'm going to do is give a little for a little bit of a primer to orbity be they gotta be using this is not the actual logo but I thought it would be nice if that was the actual logo like this is the actual logo so opportu be they say it's a server less distributed peer-to-peer database it's based on IP FS and it has a lot of moving parts to it so it's based on technologies that we we all know or that we all like have seen already I for me always consists of this kind of seven parts they would probably argue otherwise because there's more to it there's a lot of more packages they're sort of more like things going on but for me this was always like you've made main call parts so we have data API we have the operations log this is kind of the core of the data base which holds all the data I'm going to explain that a bit more in a second we have the replication kind of logic that allows various stores to sync with each other in a peer-to-peer manner the next point was crucial for us to pick this technology for our data layer because it has access control that means that we can actually control who can write to a certain database and who not in a decentralized database which is kind of cool and I haven't really seen it anyway before and the next very important module is the identity module so there's kind of crucial for the access control that we know who is right into this store like and that obviously means like some sort of key store or in our case eutherian wallet to identify themselves to be able to sign data that they write into the database it's all stored YP s itself as files that are being hashed violets by the stored and we have the communication layer so how do like how do two stores know that they want to be synchronized how do two databases know that they want to be surprised and this is the communication layer of it which is ipfs pops up I all familiar with ipfs pops up sorry um yeah this is basically a set of technologies I'm going to talk about that later as well select a set of technologies that allow direct communication between peers in real time so this is a brief overview of how Abbott works everything every store in already be it's just an up lock it's basically just a log of operations that meant basically you can I don't know if you can read it there so the first says operation put and then it's like foo bar and the next operation is put best phase or whatever you want to put on on your store and you go through all these operations and the important thing is that is an append-only locked it has that means you can you cannot delete any anything in between or something like that you can just append so what do you do when you want to delete something is just you say flag this for deletion has another operation that comes up comes with it and in the end you have like this long store of of entries the the other little box symbolizing that this is going this is this has an identity attached so you always know like whenever like I've never liked an operation is put on the log you always know who initiated that operation because it's signed in theorem wallet so you have this you have to this long log of operations and in the end you kind of want to reduce that to it were like to a final state the state that the user in the end is going to see and opportu be is using reducers for that so do you have a lot of different store types on orbit DB one is the key value story which is kind of the most accessible thing I guess for us where you where you reduce all of the operations which I was with added here to just a single javascript object so this is the object that you end up with from from this block and the ND in the end what I kind of wanted to symbolize there is that every new operation is like has to go through an access control access controller that gets as arguments the identity and the operation itself which is kind of cool so the access controller can be anything it can be anything asynchronous that means you can actually even make requests to the blockchain which is what we are doing so we're checking whether a certain person has access to a resource or can write to a resource by checking our smart contract suite so we can you can say does this person have this role on our smart contract and then they allowed to write to this particular data source we're doing all this I mean I didn't say that we be doing all this because we want people to have a way of writing and reading data without signing transactions all the time or accessing the box you know all the time so this is free this is this is basically yeah this doesn't cost any guess it's just yeah you can just do it I put a link there to talk for anyone who wants to dive into this is really accessible actually for anyone who wants to dive into CRE T's more which is the base of this kind of ipfs lot this up block and it allows a lot of cool things which I am going to touch on later as well just because keep in mind that see our duties is that main technology or the main like scientific method behind this log so this is how one person adds to their personal store in the browser so be it keep in mind that all of this is happening in the browser so how do like to store sync it's really easy you just have one store and then you have another and they try to keep themselves up-to-date and this is what what all but handles for us and this is also what to see our duties handle for us it's just one implementation of this technology and at the same time whenever you replicate stores all of the all of the operations are then again piped through your access control that that means yet no one can have like a fraud fraudulent store that is synchronized with yours like every single package is again checked against Texas controller the way they communicate is what I always said they use I we like operatively uses ipfs pops up for these yeah stores to talk to each other and basically say I have a new operation how many do you have or what's your newest operation hash should we should we think now and then the whole operation is started so it's kind of the one thing that you have to keep in mind all the time is that it's fundamentally different from any other data store that you're used to in a centralized world you have like you know like maybe MongoDB you know my squirrel or any kind of SQL storage in orbit to be everything is the database everything is a store that means how do you how do you how do you back access stores how do you know where something is stored on a PFS org like how do you know who's got this information that I that I that I don't need and in this scenario or in general like orbit EB has store addresses for every single store and that you could that you can just address like a it's basically I can like an IP Fez - you can address it like it I cannot be - so as you know as soon as long as you know the ffs hash and below for more information which is technically just the store name you can access the store and that means you have to you have to kind of have a cascading layout for your stores so how do you how do you know for example in colony we have the main entity which is colonies and I : you can have enough tasks and in this case we have a colony store which then links where you can move the mouse pointer which which then links to a task store which is a list of other stores and every single task in colony is also a store in no BTB and this is this our way of getting around this problem of how do you like how do you organize your data or your store data and how do you address your data so what do you what you what you need to know in the first place is just to call me store address and then from there you can get can fetch all of the other stores that are being addressed so now you might ask like how do we how can we get the calling store address in the first place and I'm going to talk about that later so what does all mean I kind of talked about that already bit in a sense we have lots of lots of databases which are stores or DTB and we have to kind of control them and we have to kind of make the Sun we have always on a case-by-case basis we have to decide which data type we want to use which kind of cascade you want to use for our store and in the end how the whole structure is going to look like you cannot use any of the stuff that you that you used to from traditional centralized databases you have no joints you have no aggregation you have no many-to-many relationships which is particularly bad so no graph QL possible you cannot you cannot have multiple indexes or something like that so you have only one index per store and you have to kind of deal with that yeah and in the end it's peer-to-peer and that's kind of what we what we have to live with I guess but we want to have you want we want to be peer to peer as much as possible and that's that's our trade off so now we have another problem with this and that is how do we keep data alive because there's no like central entity how do we know which is the newest version of of data which is to purchase the most up-to-date version of something let's say I add something to to one store and another person add something to their store and we synchronized and then I go offline and the the other person adds something to their store as well what meanwhile and then they go offline I mean offline which basically means they closed the browser so and how do how do I know that they just added something to their store how do how am I supposed to know that so they have the most up-to-date version but they are flying and I cannot get it anywhere so this is kind of the is what we call pinning as well you have a professor you have the concept of pinning data so what we thought what it would be a good idea or like the only case around that like we need some sort of additional client in our in our European network which always has the most up-to-date version of something and yes absolutely we can pin content that means that there is one entity which is not really a centralized thing but it's just acting as another peer for for our network of peer-to-peer clients which always has the most up-to-date version this our solution I just today press the red button get up to open source it not the one to delete it the you can check it out if you want it's it's pretty simple it's just what happens is when someone when some peer in and in the dap wants to add something to their store they kind of notify our pending service basically saying we have a new piece of data and i want to I kind of want to pin it and then the service is opening the store that this data belongs to and then they synchronize and they clog them there it closes the store again after a while when it's don't need it anymore but this is happening and when another peer comes online then this pinning surface is going to tell them hey someone just gave me the most recent version of it and here it is and then we can replicate again so this is our I would probably say these are training wheels for for the decentralization here so this seems like a lot of work and it is honestly it was like we worked on that for a really long time and I can definitely show you around at some point in our adapt which is the client side of it of this of this opinion thing but there are some some good parts here so what we get for free here is data activity like when two stores replicated we couldn't we got notified by oh btw be they basically say hey these just replicated do you wanna do you want me to show you the newest data in your in your DAP in your reactor app and then you get like data synchronization real-time data for free which is nice we get the conflict resolution for free which is something that it's also like really really hard to do sometimes when you look at other real-time applications then you might not notice that it's really really hard sometimes to really get this right and with the CIO T's it's it's a solution to to get this right and we get the the peer-to-peer for free again does anyone have any questions until this point yeah I profess is free ipfs is so they they have like a method of incentivizing keeping fires alive which is fire coin the qualified corner and that means you as long as you like keep a file and keep it available for other people you get rewarded you get in centralized for that but in the sense right now but if s is just free and people do it just because they want to so you can just use right now for free in your DAP and it's fine I just I mean next next slide is just a lot to take in don't really look at it it's just I made this I wanted to include it in slides the thing is here that again training real stuff the ipfs itself like I told you we have this pops up saying we're peers find each other but how do they do that like how do how do peers find each other in ipfs and there's a couple of ways that IP that supports but not all of them work really great I made this overview for for myself mostly to know like what can I use and how do how do we actually do that I just linked it in the slides there's some there's definitely way more descriptions under that link this is just for you to know it's complicated it's complicated so this this is so this is a way of mmm these are for me these are the only valid ways of how to our two peers find Southern ipfs and that's web RTC because that both of both of them are kind of peer to peer so this is WebRTC which is supported by all of the browsers which basically serves as a star server basically saying so peers connect to it and then they say hey I know about all of these other peers that are online as well and this is how they find each other and then there's the WebSocket star server which does the same thing but at the same time it's also tunneling the connection so we have like a 1-1 point of failure there and that's why I would probably say what part I see is better but it's sadly it's not so because web RTC is currently the current implementation full of errors and flaky and doesn't really work well so we use WebSockets with with caution and we're knowingly that it's not really peer to peer so before I told you that we need to find a way of knowing the address of the store in the first place like how we currently we have two we have two main entry points for stores and there's users and as colonies so as soon as I use a know a user store I can access all of the other stores that a user has or read them and as soon as I know a colony store address I can access all of that data but how do we get it in the first place and how do we get this nice interaction on our debt because people want user names right and how do we do that in a decentralized way because normally would just say I'm just gonna store it on the database method to an address and that's fine so if only there was a way of mapping human readable names to addresses oh yeah there is and it's really easy and it exists so you should just if you're interested in - just take a look at our DNS a register and resolver so what we are doing is we are assigning a username so be actually creating an in s domain for every one of our users when they when they when they sign up they get a signed in Esther name which is a sub domain of our da teeth top-level domain and we not only tie that to an aetherium address E&S also allows to map that to to more than that so you can't what we do is we also map that to an orbit store address so every time you request a user name in our dab you get signed and others I mean you have an address already but you could have signed an over to store address which is then tied to your user name that means I can just look up a user name and get all of their data by just typing in a user name and you get this kind of really nice interaction this is actually taken from our DAP where you can see their user name is already taken when when they type it in and yeah this feels very natural to a lot of people when they see that the only thing I wanted to touch on is forking dependencies orbity be was for a long time I mean it still is kind of for status and we fought it because we needed a lot of features like access controllers that we build ourselves for or btb eventually they merge our PRS and because we could move on but I think we had we had a we had to maintain our own Fork for I don't know eight months or something which is not the nicest thing that you want to do in the end I would probably suggest depending on how how big the changes are to use patch files otherwise you're stuck with a fork it's not a round it that just leaves one more slide I want to thank you for your attention here and I hope at some point we can get rid of our training wheels and really drive like three handedly [Applause]