Filecoin 🛠 AI Art NFTs on FEVM with Bacalhau Stable Diffusion - Alison Haire

foreign [Applause] [Music] she is going to give us a talk on uh at the um and balakow please tell me if I'm totally butchering the pronunciation of that um but it's on bridging compute over State and compute over data um so I'm super excited for this um this is going to be something like I you know specifically signed up to like post this because I was really curious on learning about this stuff myself so very excited for this talk um but we'll open it up for maybe like the last five minutes for questions um so feel free to raise your hand and come back here at the end um once that is time or if you just want to drop your questions in the chat um just so you don't forget them feel free to do that throughout the talk and then at the last five minutes we'll be able to answer the questions and yeah without further Ado I'll leave feel free to take away oh and come up here okay uh thanks very much crayon actually fun fact I think it's been almost a year to the day since my first eighth Global presentation with you guys so uh pretty pretty cool that I'm back here after a year and doing so many amazing events with you so with you guys so big thanks to you guys for hosting me here today um and okay for those that you that may not know me as well I'm Ali I'm a developer relations engineer so I get the absolute privilege of working on my passion and helping enable others to build out their ideas and supporting developers like all you amazing folks on your own Journeys and I love hackathons because it's really an amazing way to hardest the power of community and Innovation and it really captures the spirit of why I got into Tech in the first place uh which is the impact on that Innovation and Innovative Science and Tech can have um so what are we doing today um firstly I want to do a quick project demo um and have a rundown of the tech stack um I'll tell you about back a lot well backward Yao it's it's not a Portuguese fish uh well it is but it's also a computer over data I'm going to run through the code with you so I'm going to make sure you know every bit of function that interacts here and then I'm going to talk a little bit about the possibilities and future for fbm and bakli out together uh so first things first fbm and Baku yeah what is bakuyao we all know data and information is everywhere and helps me get up to my favorite song in the morning it remembers my meetings for me thankfully and it even helps me reword my blogs thanks chat GPT and also literally be lost going anywhere outside my own suburb without Maps data so needless to say data access and storage is really a critical part of today's society and it's also why it's essential we take a minute and think about what happens with our data how it's stored who it belongs to and what is done with it and for those of you familiar with the far coin ecosystem you'd already be aware that this is a fundamental Mission we're working on which is building the tech to enable open scalable and distributed Data Systems the base infrastructure that we need for a really open web um so that means that not only is it essential to create a distributed story to retrieval Network that enables that data sovereignty and that provides those innate blockchain qualities like censorship resistance and trustlessness which we're doing with filecoin it also means and we're also providing a Content verification layer system with ipfs but it also means enabling computation of data in a way that aligns with this open web Mission which is where both fbm and battle fit in um so this is a great little diagram that maybe shows you how these things interact um so storage is really only the start and we can think of the storage layer as the layer 0 while the ability to provide storage and retrieval layer I should say while the ability to perform that on-chain computation which is super exciting right now or that computation over State uh with fbm is the layout brings layer 1 capabilities to the bar coin stack but backlier then Builds on this by bringing the additional capability for off chain verifiable compute loads on on large data sets are to the filecoin network so this makes for some really powerful tools enabling new and Innovative applications to arise um so just to reiterate uh fbm enables computation over state for barcoin so it brings programmable applications and logic to the file crochet which I love and in itself is pretty awesome because it really helps enable some more user-friendly developer opportunities uh as well as enabling all sorts of use cases so both those already available in other ones in the ethereum ecosystem for example or in the broader web free community and also some that are really unique to the Falcon ecosystem so such as the the opportunity for data markets and Dows and programmable storage automation which is really not our feature of other blockchains so one thing that blockchain technology really doesn't do inherently well though is data processing and this is just a factor of the cost of computing over a distributed system that needs consensus to provide other powerful properties like trustlessness and censorship resistance so I data processing and making sense of that data in a way that we can bear it by is really essential for not just like business profitability or competitive Advantage but also to solve some increasingly complex human challenges and as well as just make our everyday lives a little easier or a little more back face uh and yes I'm looking at you fake news pandemic uh anyway this is where Battlefield art can help so baffleo enables off-chain computation over data without giving up the decentralization or verification values inherent to ibms filecoin and the broader web3 are community in general um so fbm and back wheel make up the computation side of filecoin fbm Whoppers that computation over state or on chain computation and baclia provides you with the compute over data or any data and that includes Big Data with support for gpus as well and in the not too distant future uh we're working on this cool little project called frog which will enable you to leverage back allow by calling it in your smart contract so that would be a big win for developer experience and users and if you're interested keep an eye on Project frog this is a poc in progress right now as I said um if we think about some of the data stored on the filecoin ecosystem it also really gives a even better picture of just how Central it is this is that this data is open and accessible and that we're able to make use of this data in some collaborative ways um so these are technologies that I've just spoken about will also enable like more applications in the decentralized science and decentralized data fields as well so things like collaborating on climate data action or sharing academic research even creating an open education Library which is happening now um from our filecoin foundation for the decentralized web team uh by the way check them out awesome bunch of people doing some amazing things um all of this requires off-chain compute methods um so these are just a few examples of some of the things that would require that just to get your brains thinking um so what exactly is bakria how does the tech stack work I'm not going to um Linger on this too long I know you're here to see how to AI generate an nft I'm going to get to that I promise but firstly just what is uh how does baffle air work so it was originally conceived to bring these useful compute resources and uh to data stored on the ipfs and filecoin network so again we really wanted to make sure we were bringing the same benefits of open collaboration on data sets to generic computers and really aiming to help democratize the future of data processing there if you want to dig deep on this I recommend this video uh by our lead David arancic if you want to hear more he was also a founding member of kubernetes so he's dealt with containerization before he understands how these data systems work um so battle is a peer-to-peer network of notes that enables users to run Docker containers or webassembly images or adds tasks against data that is stored in ipfs and this provides that platform for public and transparent and optionally verifiable computation as well which is known as compute over data Cod which fun fact is on cod cod this is a fun fact this is where bakliad's name come from bakayao is Portuguese pecan so there you go that's how it gets its name anyway each story the backyard network has a requester and a compute component and to interact with the cluster the backlier clear request or in my example a HTTP request uh sent to a node in the cluster this is you know via our Json or hnp and then broadcast these messages over the transport layer to other nodes in the cluster so all other nodes in the network are connected to the transport layer and have a shared view of the world so this means that when a job system is submitted I can get my words out today to battle you out it is forwarded to a back of your cluster node which acts as the requester node so the requesting they broadcast the job to all the other nodes in the uh peer-to-peer Network who can bid on the job creating a job deal market so these are some similar systems to what you might see in ipfs or Bitcoin already as well so depending on the flag even to these request denied these can include things like concurrency or confidence minimum bids before acceptance reputation locality lots and lots more even if your best cids depending on these flags the requester node accepts one or more matching job bids and the accepted bids are then executed by the relevant compute nodes using the storage providers that execute a node has mapped in for example the docker executor and ipfs storage volumes so it's a really flexible um way of doing compute over data so once the job is complete a verification will be generated which you've accepted leads to the Raw results folder being published to the compute node so our default compute node at the moment is estuary.tech uh for those of you that may uh be familiar with that as well so there's a lot more flexibility to this process but the main thing to understand is that backlier gives you the ability to execute a job where the data is already hosted across a decentralized network of servers that store data enabling you to kind of save time save money and operational overheads because trying to run like a GPU processor for example in the cloud can get quite costly as well as annoying to set up and it allows for referenceable and reproducible jobs that are easy to manage and maintain um obviously had an animation on that one that I forgot about so let's get on with the project um I got ahead and made what's kind of a little bit of a trivial uh but super fun example of how bakuyao and fem can work together I'm hoping by the end of this though I can leave you with some cool ideas for what's possible uh and also nfts have so many great use cases and combining them with computation processes over data could really help bring your projects to life as well um so there's all sorts of things you can do with music voxel nfts um token gating to animated and dynamic nfts like all sorts of things so anyway let's take a sneak peek of what our final example looks like so this is my project architecture I'm just building a front end on xjs and typescript I'm using for the nft contract and open Zeppelin a modified open Zeppelin solidity contract I'm using a backyard text to image stable diffusion script which I'll get into but it's all open source and you can do it yourself or use ours and I'm using nft dot storage for the metadata storage so and then I'm deploying to the hyperspace testnet with uh using hard hat and I will deploy this front end up to Fleek I didn't quite get around to doing that but Fleek is uh like versel it's a cicd deployer and if you're deploying your front end apps I'd really recommend trying to do it with something like Fleek or spare on who are using ipfs and falcoin under the hood rather than web 2 tax so let's take a look has anyone got a prompt I can put in here to give my little text to image um thing you know what nft should we create here this is the actual application this is connected to a contract so if no one has any ideas I'm just going to write rainbow unicorn in here again in space maybe let's try that out so basically What's Happening Here is I'm entering my tax prop prompt when I go to uh generate the image this is calling bakuyao and running a stable diffusion script in the background once that comes back I'll get a preview of the image so it'll come back as a CID back the Hour Return as I said before this kind of folder which will include things like the exit code the output standard error it comes back as a CID though um and then as part of that the outputs will include an image so something like this for example this is an old one it'll come back with something like this um and then I'll go ahead and meet the nft after that happens now this does Cape take a few minutes uh this isn't an instantaneous API running in the cloud like this is legitimately running right now on our battle servers and this job is going through so I'll just take a couple of minutes and while it does that I'll just want to have a um show you how I made this contract as well so as I said before this is the contract based off open Zeppelin ERC 721 which you can use from anywhere so the mint function on this contract takes an ITF best CID which would be the nft metadata so that includes the name the description and image and other properties that you want to put in there for that um image or or um other type of um what am I looking for in mind Val um so this mint function needs those three things name description and image sorry the metadata function needs those three things so this is a basic contract though once um a I call this function to Min an nft on uh in this case the fem hyperspace Network I am admitting an event as well so every time if I'm signed up from my front end to listen for events on this contract I will get an omission of this event now I'm also saving some ownership details here so I'm mapping uh the address of the owner to the uh erc21 that 721 that they've been it's been early here sorry guys um and yeah that that will be a minute of the blockchain as well um so to deploy this I'm using hard hat and uh there were when I first started doing this contract I was actually uh working on the wallaby tested and then obviously hyperspace came I became live so I had to switch over to that so there was a few uh fun little uh figuring out how this worked and and what things were going on there but at the end of the day I got this working and it's really kind of just a normal hard hat script which is really cool the only difference here is that a lot of the time you'll use um providers get sign up from effers and this is if you don't know what this is you can always Deploy on remix don't worry but I just wanted to run through some of the maybe buds that I came across that might help you guys as well um so when I was deploying on hard hat I found I either needed to provide I needed to provide the wallet so the signer explicitly rather than try and grab that off the provider itself so I didn't do provider.signer I had to explicitly name this for this contract to work and I could have a quick look at that if I um go down here I've got my hard hat config here nothing too special about that just has the hyperspace RBC the chain ID and the accounts and then the deploy buff VR script uh really pretty normal to any other script apart from uh explicitly calling out what this wallet is and I can go ahead and deploy it by running if you can see this I've got so many things open oh my gosh uh it just isn't little one here I can run it npm hardax run and run the script that's here and that will deploy it to the fbm hyperspace Network which is a pretty cool I'm not going to do another deployment I've already deployed one earlier oh so great if we have a look back here it looks like our nft job from bathuria has come back with an output as well um so how cool is this what a pretty unicorn it's actually only got one horn as well so it's really funny I was doing uh this demo and I was typing in rainbow unicorn a few times and every time it would come back with this unicorn that had two horns so like a Duo corn really um really weird but anyway we've got a one-horned rainbow unicorn in space now so I could go ahead and mint that nft if I want to as well and that will call that contract that I was just showing you before um obviously I will I would need a connected wallet for that so you can have a look at the fbm docs if you're not sure how to connect up your metamask I've got this connected to the fbm hyperspace Network though and I'm just going to confirm that I do want to Mint that nft uh and that will go ahead and send firstly before it does send to the contract though it does actually save to nft storage so I'll come back to this script and how it works in a second um so firstly I'll store this metadata with nft.storage and the cool thing about this and what will be awesome when um fbm goes to magnet as well is that nft.storage actually can return some of yours so once you save this metadata to nft.

storage and here's like the uh a really easy package you can use to do that the JavaScript package to do that it's also got an optional um status property so if you have a look at the nft.storage docs and I think I have them open here it's got this status here which can return the ipfs pinning uh where it's pinned to and also the filecoin are deals that have been made um for this piece of content so that's really cool and it'd be great to kind of integrate that in the future obviously uh that's on mainnet and and this is on a test net so they're not going to interact together but I'm saving this nft to nft.storage First I've just gone ahead and minted the nft all I do for that is called the contract if you haven't seen that before there's lots of resources out there about how you'll connect that or you can have a look at the GitHub as well um so let's see how we go with the mute I'm just going to go back there looks like I've minted this on and on to fbm already I've obviously got sorry I rushed my Friday code and I've got a few bugs here so this says um starboard benches contracts under five this is the block one of the block explorers that you can use for hyperspace this is meant to have the contract address on it um I can probably just have a look and say what's happening here and I'm pretty sure this is my contract actually um I recognize the numbers from playing around with it so you can see that there's been one two three four transactions and if I went and had a look at this I've also got four nfts on here so that's the nft demo pretty cool you can put anything in here as well if you wanted to uh for some reason it's not showing the nfts or they're on my wallet but it might swap over I think it should work get a listener in there but yeah how's that auto generated in it AI nft art um that's all there is to it apart from the underlying magic that is brought to you by bacquiao so I'm just gonna go cut back a couple of slides and show you what is happening with that one as well so underlying this what's returning these results is basically a tensorflow script written on python whoops wrong one my bad um if you want to see like at a really in-depth view of how you'll go about building this dockerizing it and then putting it on the back of your network I do have a video out on that so you go and go ahead and have a look at that if you want to get more into depth on that but otherwise it's also in our sorry it's also in our examples so you can go to our backyard examples docs and there'll be a whole page right up on how to do this yourself too um anyway this is the basic script though so it basically uses um uh Keras tensorflow implementation of stable diffusion which is the ml model that generates images from a given text string and it finally just displays the image generated this basically basic python script or Returns the image generated I should say the ammo weights up pre-calculated in the library as well so there's no need to do that but there's lots of scope for you to kind of do since working around with this group or run it with our different implementations or if you're like data science or AI inclined I'm sure you've got lots of ideas of things you could do there now in order to run this in my front end I'm using abacia HTTP endpoint now these are still a work in progress so this was the first one that has been done just for this demo so this rest API for backlier like I said still a work in progress but you can call this stable to Fusion one already and now under the hood this triggers the backyard job which contains that python stable diffusion script I was just showing you from the last slide and the job then runs on that whole GPU enabled Network that I talked about earlier and returns an ipfs CID now we had a quick look at the folder structure of that return from earlier so this is the kind of return that we got um yeah already showed you that and that's about all there is to it actually so the only other thing I would mention uh is that the bakayakly does return a v0 version of an ipfs CID so I'm just using a um a library that's the word uh called multi-formats just to convert that into a V1 string because it's easier for me to display that through gateways like a lot of gateways won't display v0 versions of ipfs cids so I'm just converting that to easy use on my front end as well um so I think that's pretty much all there is to this whole contract we can play around with it have some fun with it um it's up on my GitHub repo as well so you can pull it down from there please don't pay me up my really bad code I kind of uh I was building typescript but I haven't built too many typescript projects recently and I started just going type any sort of fields if anyone's done that so it does need a bit of cleaning up feel free to put in a PR but hopefully it kind of explains some of these major points in here um so before I do finish up though I do want to leave you with some possibilities for bakuyao and fbm integration that aren't just like kind of this AI generated nft though that's cool in itself um but backyard also lends itself really well to performing some repetitive and deterministic processing jobs over data so things like machine learning and AI which we just saw iot data integration and I think I saw Jared on the zoom before as well I know he's a massive iot fan so uh this one might be something that you want to have a look at Jared um so batch processing which would be for things like financial and Market data as well or storage um deal data uh there's also a video and image processing and like there's so many more things if you do have a look at our examples we've got so many great examples on there are different things you can do and no live peer was talking uh the other day and I know hudlow one's been talking and they're both big video people uh back we out could help enable some of the processing involved with that as well um so some other ideas for bakugou and fbm kind of combination you could help it could help with onboarding and off-boarding of file coin data um in the future that's uh not quite something that we're at now but definitely uh something that's on the roadmap uh it could help build a reputation and quality of service layer for barcoin by processing the data that's retrieved on chain about those deals and storage providers uh it could provide computation for market and payment data as I kind of mentioned a second ago and it could help with uh processing data from Dows and data dials like Dows themselves already have so much information that needs to be kept somewhere so proposals votes how do you decide if your members are active all these kind of things that are decentralized autonomous organizations are trying to figure out involves uh data and data processing in some way and then we're adding on top of that tokenized data sets for data downs and we've got even more information so back all the output potentially help with our processing data for for those kind of applications as well could also help Empower more automation for Creative Endeavors like I've kind of mentioned with video and image processing and even gaming as well gaming and metaversing data there's a lot of data there um so it could help with some processing for those type of things and obviously as I mentioned before back with our iot and simulations are possible and aim all applications to to add additional functionality to what's already happening and it's already cool on fbm and in the future like I said earlier we're really excited to be announcing that you'll be able to call Baku out in your smart contracts as well um okay more inspiration as I've said a few times check out the example there's heaps there how to process it images some OCR um excuse me speech recognition things um object detection all sorts of examples there that you can play with and that are already built and dockerized and you can just call from back yeah um so otherwise I am going to leave it there and get involved with us though we're on Twitter Youtube back the our project um and join the falcoin slack and join our backyard Channel you'll find us all there and willing to help engineers and everyone we have a forum on our GitHub as well and again there's those docs examples too um we do have some future events coming up and actually this is a great playlist if you do want uh kind of more inspiration about what bacliao or computer over data can do uh this is from our recent Summit in Lisbon and we've got another one coming up um for our um launch on 9th and 10th of May we've already launched in beta but this will be like a more Fuller launch then as well um and otherwise I'm going to leave you with this quote guys the best way to predict the future is to create it um and otherwise I'll learn long build and prosper thank you so much everyone for joining me I'll um stick around for any questions that people have I love that quote so much um thank you that was yeah that was awesome I'm like honestly so excited to go mess with this I just like love all like the the NF team like AI Generation stuff so um cool I think we have a few questions in the chat um so Jordan asked if the okay and Ali please teach me how to say this again [Laughter] but also I'm developer Ali on GitHub so you can uh check that out super great and then yeah Jordan asked um it that can access the network and if yeah it's time is that getting verified um what network do you mean Jordan uh I'm guessing she means ipfs Network so there's a there's a lot of information in our docs about um what it can and can't do so you would store your upload your data to ipfs first so there's some data ingestion or you can upload it anywhere I mean I suggest I've hit vessel file coin and then you can do something with it I'm not sure how does the response get verified so back in that software um life cycle as well if you had a look at on the software life cycle that I was talking about originally it's in uh the the verification process is one of the last processes I can't find it typically let me see if I can find my slide on it actually it might be might show you a little bit more about I think I can yeah there we go so um job verification happens in one of the job life cycle processes so I would encourage you to like dig into the docs to get the exact technicals of how that is verified though is probably the best way what's that man that has to be constant yeah you need it authorized or um wasn't to run the job but the data wouldn't have to be constant um but it would need to be stored somewhere accessible that your Docker container or whatso um image could um find hopefully that helps that's um anyone else have questions feel free to raise your hand too if you have any or um there is the file Queen Discord chat though you can always drop your questions in there too I think we've lost you Curry super super awesome um this was recorded uh so everyone will be able to watch it later and yeah thank you everyone so much amazing happy hacking thanks Craig