4. Network Research Updates - Lisbon 2022

thanks Matt for that very kind introduction and as you said my name is Carla and I lead the network research team at protocol Labs PL builds tools to move science forward faster and we consider these tools an important public good the network research team in particular identifies leverage points within science as a system so here I'm calling back to danella meow is one of our favorite writers and thinkers uh to align researchers and research funders we are a member of the pl Network Goods organization I have the pleasure of working closely with Matt's team among others on their various initiatives and being a first consumer of a lot of their products their meta research products and we take within the network research team we take research as our model system and our workspace for creating research flavor public goods so even better than bacon flavored public goods and so today I'd like to tell you a little bit about our metascience toolkit the tools that we're using to align researchers and and research funders to drive breakthroughs in science and accelerate the discovery Innovation cycle and I'd especially like to highlight how we're using them together in combination to build new tools and initiatives I will be speaking about five meta research Tools in particular I'll start off with a brief description of each tool how it's being used and by whom and how it can be combined with other tools I will describe some of our active and recently completed initiatives that we're using running these tools and I will finally I will describe how you can get started using these tools and incorporating them into public goods building workflows and initiatives and first I'd like to introduce our team because later on in the talk I'll be inviting you to join us either formally or informally um if the work I describe appeals to you so myself I am broadly interested in anything that can be plausibly represented as a graph and I'm beginning to think in discourse graphs that's one of the tools that I'll be talking about later my colleague Sylvia has a background in bioengineering and computer vision and she's leading our Research Road mapping efforts uh Rey is our startup operator and entrepreneur and residence he has a background in privacy preserving Technologies and he's particularly interested in developing or uncovering grand challenges in web 3 research our adviser Barth has a background in distributed systems at large with a particular emphasis on revolutionary internet architectures and Computing for social good so now you have conversation starters for all of us and bar he's not here but if you ever do get a bit of his time you can talk to him about anything um including like pineapple horiculture so and now on to our tool kit Network research as I mentioned is currently focusing on five tools that we call our meta research toolkit our meta research Primitives because of their modularity and their composability their ability to be combined to Fashion new new tools to support various kinds of research initiatives so I have previously referred to these tools as Lego bricks um but today I'm going to switch up my mental models and refer back to a popular 80s cartoon or anime for you Zoomers that chess got a modern re reboot that's Voltron one of my favorites uh Voltron as the more cultured among you may already be aware is a type of self- assembling Mecca or giant robot it's composed of five independently piloted and super awesome robot Lions um there is of course a second robot uh second Voltron Mecca composed of like vehicles but we shall not speak of this today it's Lions all the way down so we like to think of each one of our meta research Primitives as a flying robot lion powerful and complete on its own but even more awesome in combination that's my metaphor and I'm going to stick to it so onto introducing our Lions they are discourse graphs research road maps hypers certs blue funds and impact evaluators and I'd like you to note that of course I got the blue funds to line up with the blue lion that's the highlight of this talk in terms of uh visual design first off discourse graphs a little bit clapping there thank you thank you discourse graphs are a tool for synthesizing and sharing information they simplifies SC uh scientific communication into its constituent parts and they allow researchers to exchange ideas with greater granularity um and why why are we focusing on a new form of scientific synthesis and communication well right now a lot of scientific communication is siloed in PDFs um papers are a slow way of sharing scientific knowledge they're a clumsy way of sharing tacd knowledge like experimental design and they're not necessarily the best blocks for knowledge synthesis they're already partially put together they're largely composed into a narrative and they're harder to break apart so why not have a scientific conversation that resembles a conversation and here I should emphasize I come I come here not to replace the PDF not to replace the scientific manuscript I think the discourse graphs are a nice addition to the ecosystem of scientific communication forms it's more of an analog to a preprint than it is to say a letter to nature but I think it it plays a very important role in scientific communication and of course in our development of our thoughts and ideas um a little bit about the data model of discourse graphs the structure of a discourse graph emphasizes discourse moves the parts of a living conversation that we're all familiar with moves emphasizes the dynamism of asking questions making claims attaching evidence to those claims and if you're lucky pointing toward a source for that evidence this is not true of course of course if you're arguing on Twitter or on Reddit but ideally in terms of scientific communication we provenance our claims these are the nodes of our graphs and the relationships between these nodes the edges of our graph uh these are the relationships between these discourse moves so you have evidence informing questions it can support or oppose a claim Claims can support or oppose other claims and new questions can be informed by existing claims and this is a very important part of the discourse graph idea model it's highly generative and it helps researchers find new questions these edges and these nodes together they they form a data model weaving together our open research questions our ideas about the world specific observations that those ideas rest upon and the sources of those observations all woven together in a flexible framework um I like to think of discourse graphs as sort of GitHub for synthesis they support and incentivize knowledge sharing at all stages of the scientific process by making it easy to push and pull from a shared Knowledge Graph and of course you control how widely and how broadly you share your graph your knowledge graph and allow researchers to claim credit for many more types of contributions so they're a partial fory into the value attribution question that plagues a lot of scientific research scientific research ecosystems so thanks largely to the work of researchers like Joel Chan and mamatsu and the same page development team that's working on the interoperability layer for discourse graphs these are projects Network research is pleased to support and have supported discourse graphs can be implemented in several Network notebook softwares like obsidian log seek of course uh there's a discourse graph extension for Rome research that's probably the most widely used discourse craft tool but you can do this in a zle cting system you can do it on a murder board it's highly flexible it really is a protocol for communication a protocol for synthesis so who's using these tools now well there are hundreds of as active discourse graphs with deep engagement and to give you a sense of the extensibility of the protocol uh the partners and Advent and investors at Deep science Ventures are using discourse graphs to develop and drisk uh Venture creation initiative so you can do anything from Research Road mapping knowledge synthesis Venture um checking so in the realm of the laboratory Sciences a realm that we're particularly interested in there are now discourse graph powered lab groups at research organizations like the Allen Institute for cell science and the University of Washington molecular biology Department they're using a discourse craft dialect that researcher Matt akamatsu who is a molecular biologist himself developed to better reflect the language of experimental science he calls it the results graph you can see the difference in terms here uh the questions claims and evidence um which would normally be derived from written sources in a synthesis of scientific literature in a literature review are now the hypotheses conclusions and results that rest on experiments and the um I think maybe I no we'll just go with that so the results graphs they're not just a simple renaming of the protocol they emphasize the experiment the model the simulation as the source of Truth and they make it easier to formulate requests for experiments and one of my claims is that a hypothesis is essentially a request for experiments when you see a gap in the data that's needed to uh substantiate a conclusion or to test a hypothesis um making these requests of the scientific Community is an important part of Road mapping a scientific initiative and so that's a nice segue into our next research primitive research road maps which work well in combination with discourse graphs just like our friends at Deep science Ventures are using them a research road map what I'll call it is a goal centered model it describes significant milestones and their relationships it concentrates expert attention on a scientific problem which enhances Discovery it identifies leverage points so that we can choose higher impact projects and actions and it drives distributed coordination it empowers researchers to make informed decisions about how best to reach a shared goal stage and shape their work accordingly so we think of it um by combining discourse graphs and road maps as creating a user interface for distributed coordination um the knowledge synthesis process helps us to understand the current state of the art uh relevant to the goal State and identify dependencies missing stairs missing milestones and links among the milestones and then the modular sort of composable structure of the discourse graph helps in finding possibilities to rearchitecturing combine existing capabilities uh how do we build a road map we identify a goal State we identify desired probabilities or necessary characters of that goal State we draft open problems for creating these properties and we suggest potential directions for solving these open problems so building a research initiative with discourse graphs you can see that each node of the road map is itself supported by a discourse graph that's where the composability comes comes in and as you build your road map you and your collaborators are having discussions or even arguments not quarrels arguments what is the desirable goal State what properties are essential to reach the goal State what research projects are necessary to reach or enable those properties and what are the highest impact directions that you can take within these projects so again composability there's almost a fractal exhibit to discourse graphs I almost dropped in one of those exhibit means here I hope I hear you like discourse graphs put a discourse graph in your discourse graph it's discourse graphs all the way down you can be they can be dropped into at different points on a road map and they're largely scale inde dependent you can use them to build out initiatives of different sizes and so this is where it becomes interesting for aligning researchers and research funders you're basically creating an investment thesis for each funding decision that you make and opening it for evaluation by aligned co- funders by the scientific community at large and with queriable discourse graph based road maps you can make your arguments quantitative so a lot of these software implementations they support graph-based queries they support quantification of certain things assigning your own met uh networks or parameters H sorry networks metrics or parameters or features to each particular node um creating your own attributes so you can really make this a powerful system for query and Analysis and this makes research road maps a good framework for discovering and attaching resource funding opportunities like those that Matt mentioned we have a a whole tool at R from hypers certs to Blue funds prizes grants bounties and that leads us to the next of our lines the hypers cert a new research funding model so um a hypers cert is a particular impact certificate implementation there will be other better talks about hypers certs that will really let you hear this lion's roar but here we'll focus on how hypers certs incentivize and support the creation of research impact um we think of hypers Sears as an interoperable data layer that encodes impact claims and makes impact transferable and from our position our perspective as researchers um we think that they stimulate high impact research by enabling researchers to pull funding forward from the future in the same way that for profit Ventures and capital markets can so right now research is funded via grants bounties and prizes but these methods they have different risk profiles they shift the risk either to the researchers incentivizing them to pursue safer research projects or to salami slice their research or to write the grant for the research that they've already done all these tricks that we all know um or they shift the risk to the research funders incentivizing them to solicit safe low impact research or in the case of prizes they might Force researchers to be able to fund their own operations just because of the timing of the release of funds these all have their place in the research funding ecosystem but we want to have a different a different model that has a different risk profile so potentially high impact projects with a higher chance of failure are less likely to be funded over many of these models so when we add retrospective funding to the research funding system allowing final funders to pay only for successful projects that makes them a little bit more risk agnostic and allows a a more low probability highrisk research to be funded and this is possible because our system includes o I have a laser oh maybe I don't lizer darn um prospective funders who specialize in building research project portfolios and they can be rewarded for the impact that they uh buy from researchers by final funders once that impact has been realized so we see an obvious connection between research uh road maps and Hyper certificates research road maps can be used to um transparently and iteratively align researchers and research funders about the feasibility and the impact of a solution and to estimate the cost of the solution remember that quantification aspect and this can help in price Discovery for impact certificates as well as attracting final funders to pre-commit resources on the path to the goal state so we've talked about how research road maps can be used as a discovery tool for locating potential impact certificate attachments sort of impact uh areas of impact and I'd like to briefly touch on the role of discourse s graphs in the system which I've hinted at so a hypers cert if we think of a hypers cert as an interoperable data layer in coding a claim that a certain initiative will produce a certain impact um a discourse graph can be used to support that claims you can embed it at any point it can be um embedded at the point of impact evaluation it can be part of the actual the claim made by the researcher it can be part of the evaluation mechanism of the funer the seed fun who's considering funding that research project so the quantitative aspect again can help to drisk impact certificate purchases for prospective funders and increase the visibility of projects needing seed funding um to make an attempt on on the goal our hypothesis is that creating a deep bench of impact certificate possibilities drives Discovery when innovators know that there's there's desire there's latent demand for their work they're more likely to embark on projects with a different risk profile than standard academic research including those with the lower probability of success um so a deeper bench of funders Dr risks these projects for researchers the seed funding decisions are somewhat drisk by the research road maps and that allows them to assign more accurate probabilities of success and drisk their research bets and build a stronger portfolio and if they have a stronger portfolio hopefully they can stay in the game longer continue to regenerate these these flows and to make more research bets so this we hope will create a virtuous cycle getting a little bit green pilled here I'm going to move on now to our next uh lion so we've introduced impact certificates to the research ecosystem so of course now we need a way of assessing impact if you saw in that systems diagram we have an evaluation step well how do we how do we carry out this impact evaluation so an impact evaluation is you know different kind of impact but I think you get the idea it's a method for retroactive um funding that rewards projects based based on the impact achieved against a set of goals or kpis it's a tool or a process then can assess the value of a system's output and use this assessment to make improvements to the system itself again there will be better talks about impact evaluation I just want to see show you how it works in this particular framework within a research ecosystem the goals of an impact evaluator are to Signal which projects are valuable to you know so you get more of that you know you pay for something you get more of it and it provides incentives to tie outcomes um you know to to incentivize outcomes rather than just work rather than just activity so there are lots of different types of impact evaluators and you'll hear many much more about this today um there you can build a lot of different types to fit your particular use case um within research we can think of a couple ways they might be used let's say you're retrospectively assessing the impact of a research initiative you might assemble a group of domain experts to carry out a discourse graft based quantification of impact according to certain agreed upon metrics maybe for a competitive research implementation initiative you could ask those same or relevant experts to vote on the highest impact contribution to repository or or some other project via quadratic voting and attach quadratic funding to that lots of possibilities here so how is this an improvement over over current art over prior art so right now impact evaluation in science is largely carried out using various citation-based metrics um they privilege manuscripts published in high impact journals and this does actually accurately capture certain aspects of research impact but it's incomplete and sometimes misleading this particular image is meant to refer to the idea that the impact factor of a journal and the impact of any paper published in that journal are largely independ dep they actually don't correlate as as much as you might think so we think that combining specialized impact evaluators to they can be tuned to capture research impact at different time scales and Via different research artifacts so maybe you push some code maybe you create a visualization maybe you run an experiment maybe you create new reagents um combining that with discourse graphs which can highlight more granular contributions to research efforts we can incentivize research contributions that that are current ly undervalued and shift the incentive field toward high impact research um so now we're talking about incentive Fields we're talking about we've talked about one funding method I'm going to go on to our Blue Lion blue funds because this is something that we use a lot internally it's a little bit of uh language that sounds like inside baseball I'd like to disambiguate it a little bit blue funds come from the whole idea of research projects being blue um development or productionize projects being green Juan's metaphor of the Innovation Chasm so blue funds are established to pay the cost of faing projects Across The Innovation Chasm we pay the Fairman with blue funds and there's I guess a bit of a negative veilance to that whole idea he's not K on death going across the river sticks but it is a dangerous Crossing and we do need to pay projects across this dangerous river um blue funds are customized they're flexible they're composable it's a service that we provide to the network where researchers Engineers interested parties can come to us can give us provide us with a research artifact a paper a talk and use that as a platform creating for creating a research initiative that research initiative can be funded via various mechanisms uh which Matt and his team of funding Wizards develop we we uh write the open problems we may we might draft a road map we create a viable initiative that can attract attention and and eyes to the problem that states the problem clearly and makes it um pushes it along the path toward funding so I've introduced all of our research Lions they're in a pack they're hunting in a pack um but I didn't mention the giant flaming sword and that might seem to be a little bit of you know like what we' got all these lines but the sword what's missing here what's missing is you this is where you come in our community the decentralized science movement and the public goods Builders you're our greatest weapon our our Voltron Mecca's flaming sword and if supporting this kind of tool building if using these tools and creating new incentive mechanisms for research is your jam consider working with us the network research team is hiring research scientists to help uh work on the value attribution process to help evaluate different ways of estimating the return on a research investment to better understand research networks and how they work together to better understand how we can elicit um truthful predictions about the viability of re of uh research paths um we're hiring research program managers who are interested in building a portfolio of research initiatives and we're hiring a research Communicator who wants to speak to the public and to our community um about this research and translate it into terms which are broadly accessible to the public goods community and that's the on the formal side we are also supporting you can see the other QR code that's a link to our our current grants portal you can see what we're working on now there will be more additions using additional mechanisms in the future that's a great way to keep an eye on what's uh top of mind for the pl Network completely not a non-exhaustive list of possibilities there's always RFP X where you define your own research program and then of course we have a discussion server a Discord server for research nerds it's completely completely public completely open we talk about meta research tool building and a lot of the people who think deeply about these issues are there just waiting lying in wait to Ambush you with questions ideas and to build off of your energy so thank you very much for your time and your attention and I think I have a little bit of time for questions here if there are any hi can you hear me yeah um thank you so much for the presentation I think this is what I came here for so thank you um uh you um made a comparison about discourse crafts being uh like a GitHub for synthesis and I wanted to ask you if you have a position or an opinion about how open research should be and how soon it should be open or not and if that's embedded in the tools that you're building that's that's a really good question that's it's a it's a very it's a difficult question because of course I would like to say we like to work in public we have an open source ethos all of our grants and everything come with the only the only thing we ask of our recipients is that you work in the open that any any um code or products that you produce are published under open source licenses published openly public open code working transparently yada yada however I do have to recognize thinking in terms of incentives is that not everybody is similarly situated to work in the open that academic incentives um the academic incentive field sometimes inhibits folks from feeling comfortable sharing the research so early career researchers may have real real good reasons not to want to work as transparently as more senior researchers and I can't hold that against them that's a systemic issue so while we want to what we want to do is shift the incentive field to to encourage to make it comfortable to make it the best choice for people to work transparently how this applies to discourse graphs I expect um that persons will curate their own graph and they will decide on how transparently they wish to work I think some of this is their willingness to share their ideas but more is more about more of it is their ability to inhabit the same context to have the same common knowledge of people working on a shared graph I don't think you can have one Global graph I think we will have multiple graphs or multiple views on larger graphs I think it's great that these graphs are interoperable and that's why I think the GitHub model works here you can observe a graph you may not have right access to it but you could potentially Fork it you could pull from clone it um and then create your own graph um based on other people's work so I think that is the level of openness and transparency I think people in the current research ecosystem are comfortable with and that's where we will meet them and hope to nudge them to broader to sharing sharing more broadly and building a larger ecosystem I just like to know where do you see some web free specific Technologies making the greatest impact here I mean there's a clear link with Point by storage dat is so important but in terms of are there other things such as tokens incentives y your personal opinion on where web 3 really fits in uh specifically the well part of it is part of the web 3 um fit to these ideas is just the ethos the idea that things are permissionless that you start building um that you create that you create and curate your own community and that there are no centralized Gatekeepers to knowledge to access to services to tools things can be disseminated much more widely and much more broadly um the other part of the web 3 ethos that's important here is the idea of censorship resistance the idea that a lot of these tools um the composable nature means that they can be um sort of replicated widely stored in different local first broadly shared locally shared repositories um they can be operated and worked upon by multiple communities at once and there's no nobody can sort of cut the pipes and stop the work um in terms of tokens and other specific forms of incentivization it's clear that we definitely have as Matt said a really exciting test best bed and playground in the file coin Network in ipfs for working on these incentive mechanisms in a very concrete way using well as concrete as a blockchain can be we we consider the real world our Real World um we have certain um affordances at our disposal that traditional research funders don't so we definitely see this as a new space kind of a blue field where we have the the latitude to experiment with new ideas and new mechanisms in a semi-controlled open and transparent test bed hey um so I I guess uh one of the things that's coming up for me here with all of these graphs is the semantic web and the prise that holds um I'm seeing that there's a large amount of metadata that can kind of correlate across different contexts which feels promising and I guess I also see protocol lab is kind of having uh the funding to catalyze such a community of practice um and so yeah I'm just wondering what your thoughts around uh that blue field uh looks like yes you've really pointed to a very exciting opportunity here the idea the interoperability between a lot of these different things that we've been speaking about the fact that this can map imagine that you did a discourse graph and you assign cids to different nodes and you can capture particular metadata metadata which is not just used for surveillance which in which is informative which can be used to say track contri contributions to a graph to attack the value attribution problem to provenance certain ideas to see how ideas evolve over time so yes we definitely see a mapping between these types of tools and the technologies that we have at our disposal and one of the things that we'd like to work on is developing open-source free implementations of a lot of the tools that you see here so that we can invite other people to again fork and hack on these tools and an open way experiment with web 3 Frameworks to support these tools um both internal and external initiatives around these tools extremely welcome we are very very receptive to any type of uh grants or any type of research initiatives any type of research proposals along these lines well then thank you very much I really enjoyed my time up [Applause] here