Building a Global Operating System for Earth - Bangkok 2024

[Music] okay awesome hey uh good morning um so first uh I wanna I want to give a shout out to Caitlyn uh Moore back there who's my co-founder at devonian systems and what we are doing uh in collaboration with a bunch of people in this room um including the amazing team at gain Forest are building what we think of as a global operating system for for nature and so you might ask like what is this and like why why does nature need something like a a global operating system right and so this is a place I've spent sort of a lot of my a lot of my career is in just technological development and scientific development of various different Technologies and when you when you look at um so you know I I did a lot of work on solar cells and batteries and so like physical like materials oriented systems that have real you know have have like push atoms around in the real world and when you look at a bunch of these these physical systems and and also some some things like you know cash cards and ATMs right that are more like human Innovations um first thing to notice about this graph is they take decades to go from R&D and sort of the initial sort of stages of an idea to something that's actually implemented at scale having real impact in society the second thing is that's interesting about um this this like visualization in particular is that the the blue bar is research and and the yellow bar is implementation right and so in this model right in this like typical way that we think about coming up with new ideas and those ideas having impact in the world there's a research phase and then there's a implementation and scale phase and they are separate right you like figure out what you're doing and then you do it and the problem is we don't have time to figure out what we're doing and then do it right so these are these are this is news from um the past two weeks right so there were these these horrible floods that uh Max also mentioned in uh Spain south of Barcelona um this is from the emissions Gap report that came out I think October 24th something like that from the from the UN environmental program what you you want to kind of focus on here is this like 66% chance number right so we're at um I think I think this year is on track to be like 1.55 degrees C above the pre-industrial average so that's like a lot of warming I was actually I was watching uh Chasing Ice uh a while ago and uh so this this documentary came out about global warming you know 10 years ago or so and they're like it's so bad we're at8 degrees above global warming and like all this stuff is happening I'm like wow those are those are really rookie numbers unfortunately it's not it's not good so you know so by the end of the century right with current policies so that's like what we were on track to do before the US election that was already signed into law right we're on track for 3.1 de of warming above uh pre-industrial levels um you know if you if you actually pay attention to naturally determined contributions which countries have said to the UN that they're going to they're going to do to to decarbonize um but haven't actually signed into a law yet you get to 2.8 2.6 um and then if you actually make Net Zero pledges which everyone's really far away from doing you get to 1.

9 degrees of warming above pre-industrial levels um so the point is here like we we don't have time to we have to build a plane while we fly it right we need to like learn about these systems learn about interventions and Implement those interventions in parallel with each other and so that's why we need a global operating system for science one of the things that we need to be able to do as we're like building this electric plane as we're flying it is we need to be able to rerun the numbers quickly and efficiently and our current information systems for science are not set up to do that this is a this is one of the studies that led to this famous Guardian article that sort of blasted carbon markets to shreds um in early 2023 and essentially what they did was they said okay typically when you are bu when when you want credit for preventing deforestation what you do is you specify your project area you specify a baseline area and the more deforestation there is in that Baseline area the more credits you get the implication is like we prevented deforestation that would otherwise happen and what they did is they just came up with these this idea of a synthetic Baseline so they said okay rather than allowing the project developer to specify what they want to be control what they they want the control to be let let them kind of like write their own test let's just like run some stats and like rerun these numbers on these projects right so the point is though if we had a global operating system for science it wouldn't take like years and decades to like figure this out and actually do this research if we use the tools that we have to make science interconnected and then connect that to the actual implementation of those scientific ideas we just create an entire pipeline where we can just rerun these numbers whenever we want to and really accelerate how really accelerate our ability to course correct for problems in what we're doing now right we learn faster Implement faster and um solve the the carbon and biodiversity crisis why doesn't open science the way it exists now like solve this problem right so there are there what what are called Fair principles so findable accessible interoperable reusable that's that's sort of this the set of principles for the open science movement that's important if we're going to have open data and open science we need data sets to be quote fair but when the open science movement started implementing a lot of these Fair principles and said okay let's just like open data let's have you know kind of have this this free-for-all let's um let's let everyone look at the data look at the code for what we're doing and that's going to like accelerate science the the problem is like there's there's actually a especially in this area especially when you dealing with indigenous people in local communities who are deeply connected to this land you also need these these care principles right you can't just like open up all of this data that um is like deeply meaningful to a lot of the ecosystems and people living in these ecosystems that you are working with um you actually like need to give them sovereignty over what that data is how it's shared how they're compensative for that data what's done with that data and bring them in as as stakeholders into this right and so we actually need more fine grained control than the typical sort of like zenodo style open science movement gives us and so this is this is like one of the areas where the tools that we're working with can really like make a huge impact in uh in this problem so um so you know what are what are some of the tools um I've I've talked about this before you might have heard it but um we can really think of hashes and CS as a universal coordinate system for human knowledge right so the point here is that cids right so so addressing data based on its content rather than its its location um are this like incredibly massively powerful concept that we're only beginning to see the implications of people have known for a long time about data structures where you use hashes to prove that your data is included in some some larger data structure so like a Merkel tree right you're able to efficiently use hashing to to prove that this this little chunk of data exist EXs in the context of this larger structure so that's great so lots of people for many decades have said hey if I hash my content I can efficiently verify my data in the context of this specific data structure and that's great you know we've known about this since the 70s um it's all it's all you know works really well it's really good um but the the the core sort of realization about content addressing in web 3 is that hashes are incredibly overpowered for this particular use case so web 3 says hey if I hash my content I can efficiently verify my in the Contex of all of the data every data structure ever right you're not just verifying that this chunk exists is is like the chunk you're looking for in the context of this Merkel tree you're you're showing like this is the only data in the universe you know within reasonable assumptions about collisions that satisfies you know these requirements right so that's that's a big deal um so chy if you go to chy doio tr is a global operating system for Science and um this is what try looks like so the way to think about this this is a simple example and we're going to we're going to kind of use this to use this really simple example to talk about the set of Concepts behind chy um and uh please come to a workshop later if you want to talk more about how to sort of apply this to more complicated examples but so you should think of this as a pipeline a data pipeline where these lines are data sets and these boxes are functions or data Transformations and data flows from left to right and so um if you you know click on this select this this is the data that like fills that position in that try graph right so this is a um this is a a image taken by uh oceanis in collaboration with uh with gain Forest looking at mangrove trees in the Philippines and so then this box right is this machine learning model that is optimized for delineating crrs sorry cre did I just say crr tree crowns um in mangr forests right so you you plug in you plug in this image to this ml model and what you get out is this delineation of you know it shows you where the trees are right and so this is this is the input image this is the output image just to like make that really clear and so this is a data pipeline in chy with some really important relevant features for all the motivation we were just talking about so you know one thing you might ask is okay how do you how do you actually build a graph like this how do you use chy to build this pipeline well so you can so we we support python now you can install something called py try and then what you can do is you can take your script which you know loads your loads your data um runs it through this this like ml model um and then you can import py and you can do what we call checkpointing data and code into this graph so when you just like run this script then this is going to checkpoint the the script itself along with uh metadata needed to to rerun that script um into this uh you know this this like function here um which represents this data transformation it's then going to when you have a when you have a data set you can then checkpoint that data set itself into this Pipeline and so that's that's sort of the the workflow for like creating these graphs and filling them with data but what's the actual like structure like what is what is in a a chy graph so a chy the the structure of chy combines this this like mathematical thing called an operad which is this like really really beautiful mathematical uh like set of structures that come from this thing called category Theory which is the study of composition and um it combines that with essentially CS right so like ipfs and file coin and so what is that what is that you know what are what are these like data objects and what are what are data objects and like boxes and like what does the graph I was just showing you mean in the context of the structure I was just talking about so firstly data objects themselves are content addressed and they have this git like versioning system where you have versions you have graphs and every data structure itself um is a blockchain so it has this this like previous the CID of the the previous version and it you know points to that right and then you you like hash this thing and refer to it as the as you know in the the next kind of version on the same branch um or you can have multiple different branches that I'll you know refer to previous you you get it um then this pipeline itself is has the structure of this thing called an opad which essentially like you should just think of this as it's like this mathematically really really sort of convenient way to think about how things depend on other things um and so when you have boxes here boxes are hierarchical and composable so there are these data Transformations that you're able to to specify what code corresponds to that data transformation and then you're able to reconfigure them so if you have this little like subgraph and it fits into this other subgraph you can paste them together and you can extend your try graph and so an example of that might be like if I have a a company and I want to prove that my company is net zero and I buy a carbon offset hopefully a really high quality carbon offset maybe that carbon offset already has a bunch of data as a try graph right so they've like collected a bunch of data showing about showing you know the the forests that they're they're protecting they've done a really good job if I want to make a net zero statement preserving all of the work that I'm building on top of right preserving all of the work that this this like carbon offset project did um that my Net Zero claim is based on like the the mathematical structure of operads lets me like prove that without um without data loss right that's that's kind of the point is like you can take different things and if they fit together they they like fit together well um so boxes are are this like hierarchical composable structure then um so like this you you select one of these data objects and you can fill this position in the try graph with data with like that data object and so you know maybe this data object uh is this like input image uh maybe this thing on another Branch or maybe it's you know a totally totally different um data object that like doesn't share you know branches or versions um can go into this like slot in that graph and then when you checkpoint data by by running this code um into this chy graph that's what you're doing it's you're checkpointing produces a new version of every data object every box and like the pipeline itself so everything has a CID so everything is kind of nailed down in this Universal coordinate system for knowledge but you preserve this notion of composability so you can take things that kind of fit together and glue them together without without losing data so that's that's kind of what and why um uh so you know uh what is what is this oh yeah so this is um this is like how you actually do this right so this is a JavaScript um JavaScript package that lets you create and work with data pipelines and data objects according to this protocol right so there are functions in here for creating a new pipeline that doesn't have anything in it there are functions here for adding new box changing the connectivity between between boxes adding data like filling you know one of these like positions like this this like image position in this graph with a new data object creating data objects um interfacing with um with storage that's modular so you can like create your own storage system and this will like automatically make sure that you don't lose cids when you create new like you know hierarchical structures in this um this whole whole graph um so that's that's uh this thing popular esm is a is a way to do that um and then uh so we've talked about kind of what and and how um so now we can talk about why right like why does that motivation that I talked about at the beginning actually like fit well into this structure and why do we need this notion of operads combined with this notion of content addresses to actually like accomplish those goals and so oh yeah this is what I just said yeah what's the value of combining these that one um so one one of the reasons is that when we have this strong notion of composition and dependency we know mathematically what depends on what else and what that means is that if you have a input and you have some code and you trust the input because you have data Providence saying that I trust this input and you trust this code because you have data Providence showing that you trust the code and you trust your ran it or you have some some some proof showing that it was executed correctly then you should trust the output right we know this depends on this we get that from this structure of operads so we can just calculate really straightforwardly whether or not we trust this thing we can like overlay this data Providence um on top of this structure and determine whether whether we actually you know have a good reason to to trust this data that we're getting out so that's really important for making decisions in the real world based on some set of data right so that's that's one reason this structure is useful another reason oh yeah and so you know an alternative would be maybe I maybe I trust this maybe I still trust this oh hey that works better maybe I still trust this input data but I don't trust this code then I don't trust the thing that depends on that code and that input data right if there's like some Link in this chain that I don't trust then I don't trust the output um so then also coming back to this notion of why don't existing Tools in open science sort of solve this right because we have the same strong notion of composition that means we can give credit for and compens ation to the people who provided this data and code so if we want to make sure that people are adequately compensated for say providing training data that then produces a machine learning model that then I am using to make my like Net Zero statements and that's very valuable in current systems there's there's no kind of way to check and make sure you know how how much were these people compensated who collected that data um and you know did they did they like give their consent to use this data um but again because we have this this like strong notion of composition we can again calculate you know we can either say Okay Who provided this and um we can use Providence to to figure out do they actually give their consent or we can if they say Okay um I want to charge for my data or I want to charge for my code again when we rerun this graph we can just calculate what is the cost to rerun this graph and make sure that money actually goes to the people who provided that code in data then the the sort of last piece here is then because you have this composition um from you know this mathematical structure and also IM immutability from CS that makes these different pieces interoperable so you can like rerun these things more easily than you could otherwise because everything is glued together by these two structures right you have you have this like operad structure which just ensures that everything is kind of kind of works together and then you have CS which like nail down your data structures that means you can you know take this thing and swap it out with this thing and then hopefully be able to rerun analyses like this much more easily than you could if everything doesn't in the current system it's like it's really hard like I don't know how many people here have like tried to take an academic paper and just like download the data and download the code and like redo the analysis it's like it's very difficult and very frustrating and a structure like this let you do that more easily um and then sort of the last sort of thought here is so what happens when you apply this Universal coordinate system for data to the web of human knowledge and the answer to that is you reveal this this like true underlying structure right so the we we're all working together to understand natural systems better and try to solve these problems so this this like underlying connected structure really exists like it is a true thing that is out there in the world but because we have taken this Universal coordinate system for data and applied it to that structure it's all fragmented right you can't you can't access it frictionlessly or easily um and so I think what we're doing here is like trying to get closer to that true notion of all of us collectively uh trying to make progress on these on these issues together um so you know please come talk with us about any aspect of that um and uh we have a workshop in about 2 hours a little less than two hours uh that Kaitlin's running um and I'll be there too and uh please uh please come talk thanks [Applause] yeah okay yeah questions any questions maybe questions some questions oh hey how's it going sorry yeah yeah yeah yeah yeah um there are so um so he's asking can I use sort of any any C or any data structure as input and yeah it is built to be modular so that you can like ideally you want something that has a meme type so you actually like know what functions it fits into but you could like build a non-standard meme type or just like try to run it and maybe it'll crash but like it it is built to be modular such that yeah you can take like different data structures or different um different formats um it's it's really optimized for like machine learning models dealing with visual data is like what we've done sort of the most work for it on but um you know also like bio acoustic data like we're thinking a lot about kind of nature Tech questions where you storing this data and how you yeah well so so it's built to be modular such that you just need a CID and so you can drop in so like that popular popular esm thing like you you provide it a a um a storage function and you provide it a retrieval function and then you can just like handle that however you want so you could just connect it directly to ipfs you could connect it to your own server that like content content addresses data but controls permissions or just you know handle it sort of however you want the point of this is to be sort of a layer on the stack that like gives you that inter operability and sort of nails things down with cs but then let you sort of manage permissions on top of that all right thank you so much cheers got to get to the next talk but you'll be able to find Allen at the workshop space