Hybrid Art + Metaverse of Things - How artists can co-create with AI

[Applause] [Music] hey everyone welcome so we have portrait xl here thomas and ivy and they're going to give you a session on hybrid art and metaverse of things we can take questions as we go and just put them in the chat and if you have any questions they'll answer them towards the end i'll let you guys take the stage thanks oh hi everyone thanks for joining and thank you to zora for having us um so i am i do a lot of um hybridized art with human machine co-creation and as a independent researcher i work a lot with data scientists and so i wanted to put this workshop together that will give a little insight of what it's been for me as an artist to work with machine learning as well as have perspectives from thomas haferlack who is founder of pollination zai who's a data scientist and a new media artist and then a little glimpse into our feature from our gen z prodigy ivy who is um also studying computer science and um and transitive planning research so i'll just get started with our first slide so pollination.ai is an open source platform that um thomas i'll let you do a brief introduction brief introduction yes yeah first of all thank you that um we're already talking about pollination.ai on the first slide i'm very happy to get this introduction yeah pollination saturday i was my was a project that um um me and a few colleagues created more than initially more as a tool for ourselves because we were very fascinated with the developments around the generative deep learning so deep learning that's that's uh focused on generating media first of the first kind of application where these models have become very good is generate in generating images and i i was i i was so fascinated with the explosive kind of development we had in the last say two years in terms of like possible creative possibilities image quality and also types of media and types of ways we can process me i was so fascinated that i wanted to make this also available to friends and artists around me who i know and who were really keen to experiment with these techniques and um so it was kind of more of a platform to make it more accessible initially and now we are turning it into an open source based business um with yeah with some funding and um but yeah the the idea is to really um work closely with the open source community and make it easy for people to experiment with these new ways of let's say generating and and manipulating all kinds of media yeah join us on discord on pollination discord we have a lot of um really fun and meaningful conversations about how we can sustain and um and work with uh web3 technologies to create these kind of like new ways of i guess co-creating and doing things on chain with like smart contracts and stuff like that um so yeah i'll just show some examples here we go of some of the uh latest experiments so this is from um the new zora trailer that just went out and um thomas introduced me to this disco diffusion warp model um which was my first stab at it and what's really interesting to me is like what you see is the same text prompt taking one video of a talking head but in like different sizes and it's just this kind of stuff just fascinates me because you get like such different results um so i it ends up leading me to way more questions of like why as does you know why does this behave the way it does um and then so ivy created some other custom ai visuals and we just submitted all this stuff that we created together and then sir jugie then took the content and then did some like really cool analog effects to it um so that is also available in pollinations and um to the right that is a disco diffusion as well and then to the left is a um text to image vkegan clip um approach that i've used mainly for a lot of my experiments in the last couple years um and one of my most like i guess obsessive way of working is um obsessive obsessed way of working with all of this has been like trying one ai model and then taking the result and then feeding it into another one and so what's really what's been really fun is um i discovered pollination.ai from tomish just after i finished creating this uh neural vocal duet ai album um in collaboration with databots and so then i started doing some experiments with taking that music and seeing what kind of visuals i could create and um so this is which um allows you to do things like control the parameters of your treble your mid and your low frequencies and and set these very granular ways of how the visuals move in latent space so it just gives you even more control and so this was using a few different parameters with the wildlife data set and then i decided to experiment with taking that visual into text to image wikigan because i wanted to customize it a bit more to aesthetics that i was imagining um at the time and i've been doing like i'm always doing these deep dive um research of like related to identity and stuff so this what um i'll just volume um text to imagevik again clip i think the text prompt was um hyperspace mongolia melting gold and holograms um because people always think i'm from mongolia but i'm not but but i've learned that apparently like i said my background south korean and um apparently there was like a lot of mongolians that migrated to korea at some point so i was really pleased with the results also to play with what happens when you take like a square format of a video and then um try different formats and aspect ratio as an output um i thought was really beautiful and there's also a super resolution model on pollination.ai so you can upscale your videos um so this was this was really cool and i've performed live with these visuals because i managed to get up to 4k um [Music] so the next project that i think is really exciting to talk about is how artists can use art as data for creating even more customized ai art and um thomas i'll let you explain more detail of what uh ruby's and diamonds project was about yes so yes um so i'm kind of what i what i find exciting is like not not not necessarily the idea of ai um replacing replacing artists i mean there's a lot of quite heated debate at the moment about if um what the impact of let's say dalia 2 is on the on certain um types of art um but i've always been quite happy with the idea of like collab the collaboration between between ai and human and i find that a very one of the most exciting um kind of ways of of working in this space um and uh so um i was in an artist's presidency in the dominican republic and um got together with a contemporary um artist nicola rubinstein from berlin and she paints in in a certain style and she was also making videos so we were filming the eyes of different participants and um her idea or dream or our dream was that we could apply her painting style to these videos that she filmed and yeah we gave we used different techniques to to reach this result so but she was very happy because she really felt like um um that we the ai the machine learning models managed to capture her her aesthetics in a very interesting way and then allowed her to do something that it would be impossible for her to do without um it would be a very very a lot of labor a very labor-intensive process to paint all of these video frames so it was kind of a very very interesting mix and it was also interesting to see how excited uh um nikola was um to to collaborate with the machine in such a way um and we're planning to do many other projects together in the future and yeah the idea then is somehow you could also create presets together with artists that um can then generate series of of of media that could be used in all kinds of um uh ways or um and possibly even monetized through nfts or through some other ways where one would also share um profits and so on yes so cool it's a really beautiful project um so last but not least um i am going to pass it over to ivy who is the youngest ai artist and research that researcher that i know of ivy has a scholarship to tufts university starting soon um he's spending some time with us here in berlin for a while and um yeah i'll let you talk about your your latest experiments with 2d to 3d are i think it's just so amazing like how fast you work with everything yeah i mean introducing to this technique um coming from the pollinations and seeing pollination is very exciting for me but it's also a project that is trying to implement user-friendly interface for uh working with the ai models and there is kind of it takes some time to implement every single models but in the world of machine learning you get new networks like almost every week and it's interesting to experiment like latest technology there is so um this is a clip guided neuroradiant field model which is quite new and um it's not yet implemented anywhere so there is if you want to research it you have to follow github and follow the instruction to install it but it's also interesting just to work with the networks which are not implemented so try if you want to try something new try to find networks which just brand new and which you might be able to somehow use in your project and so on and i started working with the 3d ai um stuff almost two years ago in it and at that moment the field was quite empty there is no other one there were not a lot of models which were capable of working with the 3d data because it's very um it's not not euclidean data so it's very hard to work with it and um i and recently and it's very exciting for me to see that just two years in it and we already get this awesome results which are capable of producing 3d models from images from text and this is the model which is called dream fields by google it was originally developed by google and it's capable of creating 3d models based on text or image data and um it's quite easy to use it but it's very exciting and for example this is the video of a spaceship so it created a 3d model from the tech spaceship and um it has like it's certain style it's not like photorealistic and it's really realistic but it's still exciting to see and work with it and this is just the user interface for the network itself and this is a image of a text input and if you go further and this model is also capable of working with the image uh images and is capable of transforming um image data to 3d models and for example this is i took an image generated by mid journey which was like a neural flower i guess if i remember correctly the text prompt was neural flow and then i put it into this model and it produced this 3d model but working with the 3d data you can always modify it in different 3d software 3d modeling software so for example it generated this flower which i found very interesting because it doesn't look the same as the image it has certain features same as image but uh it's not hundred percent um same as image and um i liked it because it created something new it added its own um features to the freezy model and then i modified it in a blender to look better you get um i cleaned it i modified it so there is also like it still requires some email some artistic input from you and um then you can use those 3d models like almost anywhere you can generate some kind of worlds with them you can use them um you can 3d print them you can you can really do anything you can with like just normal crazy models and it also generates the texture so it's also exciting it generates the shape as well as texture um and on that note i think it's everything from my side this is so cool i want all of these in my meta verses in my 3d world i want you to show me how you made these last these last ones it's so cool but yeah that's our flash presentation wow we finished right on time um we thought we'd leave 10 minutes uh for q a um and we're happy to go deeper into the flash presentation that we've just presented um these are our handles on instagram and twitter if you want to um tweet to us or message us on instagram directly because maybe you come up with uh questions after um after this or we don't get around to answering questions um but yeah that is that is us for now this is are there any questions from anyone oh i have something that says talking but i don't know if i if i see anything let me open ah okay there's no questions in the chat but if you have any questions now feel free to uh meet yourself it's a lot um this is a lot of content to go through in a short span of 30 minutes um but what is really exciting to me is that we have all of these tools available and um in the context of what web3 pollinations have been experimenting with like hybrid models right with using things like ipfs for data storage yeah um and also having conversations like what is the most streamlined way of doing this kind of stuff being aware of how computationally heavy it is how much time it takes to do all of these things um and so i think it's it becomes really meaningful when artists become part of this dialogue of where we're going with all of this and um and and get creative with like potential problem solving um thank you i see a message from pierre thank you pierre um so yeah uh at the moment pollinations works connected to google collab uh yeah that's it's in we're we're we're in a hybrid state at the moment we have we we have we have um our own gpus running in our in our back end since uh uh since a little while and we're kind of slowly migrating um migrating the models that previously were relying on google call up which is a service which allows you to use um gpus for free in the cloud maybe probably people who are in the field know about it but it's uh we what we did with polly nations initially was kind of bootstrap because we didn't have money which was a initially we didn't have money so it was we didn't have any way to pay these gpus so we kind of connected to google call up um which made it a little bit more difficult to use but was kind of an interesting way to start our project because and now yeah we're slowly making it more user-friendly by migrating these models now onto our own gpu back end and we're planning also to keep it keep it free as long as we can um for for end users um so it's gonna kind of i think if you start using pollinations in the next few days it's gonna be a quite a nice user experience maybe we're still ironing out some bugs um um but uh yeah we have we have our own back end so it will allow you to kind of for free uh use all of these techniques without um too much difficulty hopefully and you also have kind of bots in our discord um or in on on twitter so you can also kind of talk to a bot uh like mid journey for example also is doing in a very interesting way um so with experimenting also with different ways you can kind of enter in talk to these models and the bots have also been quite an an interesting um direction to take and but you still have also our website pollinations.

ai where you can create and it's going to get easier to use and hopefully still just as interesting yeah people can get an image generated based on their twitter profile information right yes that's that's going if you follow polynation's i ai on twitter um you will get a response with a portrait drawn based on your based on your twitter bio um yeah so um um this is these are some some of the fun we're having with you with the ai models and yeah it's i i it's kind of interesting with it we have one new idea each day like a daily uh generative tarot card generator or your horoscope or games where you can where you can which involve generating images and guessing what's on the images uh yeah um like now kind of these models are used for usable in this like kind of module modular way where maybe you can kind of let's say feed the input of a text generation model into a model that generates images i think it's it's only going to start we're only going to start kind of having all these fun ideas um and kind of applications this is just kind of a first step in that direction generating people's portraits without them without them even saying i want it so it's really it's really exciting interesting times um with creative ai i think and ivy's going to come out with a whole series of these new 2d2 2d to 3d experiments some of our own collaborations as well that we've been working on um and i don't know it's just a really exciting space to be involved in and and i love the idea of having ai become more democratized so that there is like a real diverse narrative around like what it means to work with these technologies and i don't know um art to me is like any way any any chance we can abstract something out into some kind of like experience or something tactile gives us something tangible to either look at feel or experience and then and and we can have some kind of like an understanding of how all of this works um addressing things like bias and um and and it's up to artists to create with these things and and talk about these narratives narratives in interesting and immersive and meaningful ways so um yeah thank you for for um for making this all happen and um and i'd love to i always get really excited when i don't know people like show some new weird experiment that that they've done and we've had a lot of like really fun what the moments during our like co-creative process of um just i don't know i think like artists are really good at accidentally breaking stuff yeah and um and you've pointed that out also where like i've accidentally broken stuff like totally unintended but then that's because probably um me not being a programmer and um and how i find it so valuable to have these conversations with data scientists programmers artists like yourselves and it's this combination of different perspectives coming together that um really excite me and so yeah if anyone um if any of you ever have questions um or want to get involved um or collaborate or anything like that we're all pretty open yeah yeah also yeah in this discord is a very easy way to to talk uh to get in touch personally like it's really it's always a real pleasure to to have an exchange with with people uh enthusiasts also in the in the field um yeah um um i uh yeah i mean i just wanted that i found it i think that the thing with the errors is really nice because i think that's what makes them artistically also so interesting is that the kind of failure states are still they're still interesting you know like when you when you have an mp when you have an mp3 that's kind of that's kind of broken it's the the error sounds quite uh you know you know they're very digital whereas the when the ai fails it fails in kind of funny organic ways yeah just you know i think that makes it yeah totally yeah you have a nice um uh comment in the chat um from her on um but yes um i i'm also super i'm just equally excited about the the custom ai methods and approaches that artists can harness by doing things like that like what you did with rubies and diamonds um it's just an example and i've been sharing that project with all of my visual artist friends um because it's i think that process also of like seeing the machine take your own data your own artistic work and try to create something new with it that creates a really interesting relationship i think with your own work and i think it opens up a way for artists to become even more intimate with your own process and your practice so um yeah i'd love to see more of that um so you're going to get pinged by a ton of artists now that's that's exactly that's exactly what i'm what i'm what i'm uh open for yes yeah and where we also thanks to a little bit of help and and point us from ivy we will also and we of course we've been also kind of researching um 3d model generations of 3d object generation so we want to also integrate these models that iv iv has been working on and we've also been kind of researching um so maybe we will if you will be able to um generate 3d 3d objects or avatars it's a very there's been quite a lot of development in terms of avatar um generation from text so you can describe like i want a forest which who's overweight and maybe um has freckles and you can write it all in text and these models are doing quite a good job at generating avatars that can already also be embedded into games and imported into unity because they they come in this rigged in this rigged format but this is like one one of the models does it and yeah so this is uh was something where we see kind of the a little bit of a big future in terms of generating 3d objects and then also thinking about the metaverse and generating possibly um whole immersive environments and it looks like very a very re a goal that's very rich reachable and possibly also democratizes the the access people have to creating their own content in the metaverse because 3d modeling is not such an easy to acquire skill yes so this is a i see a lot of future day and we're kind of also focusing a big part of our research um now um on this idea of generating 3d models and also yeah if some people are interested in talking about that we have we also collaborating with ivy um hopefully yeah we we are integrating some some of the uh yeah things he found already so yeah it's an interesting space because there's just so much new stuff happening each month yeah and so cool yeah i'd love to see 3d artists get involved in that space also of like what they think about all of this and how it could in implement maybe a new workflow for them as well um we're we're actually investigating that with some kind of partners um yes yeah because i think that's the other thing about air is that there's this other side of it where people are really scared of it replacing jobs or like replacing roles and things like that but i don't see it that way i think when there are emerging technologies this is also why it's important to make sure that like the creatives who are like you know um working with these methods and approaches like 3d art and um 3d artists to be part of that dialogue and to play with the technologies and feedback and like how does that enhance their practice um and i don't see these things as replacing there's nothing that will replace like the craftsmanship of an artist's own you know skill set um i'm just gonna quickly answer there's a question that came in the chat uh pierre i will i will message you directly if you can message me either on instagram or twitter i can share um some some musical ai resources um there's some open source stuff that i've used there's also um two process documents i have on my website which i'll just put in here um and um it's i have uh some free like ai generated audio samples using uh ddsp auto encoder model um which has been really fun to experiment and um yeah i think i think we are um at the end of the workshop um i'm just being mindful of time um if anyone has questions uh or i don't know shall we shall we end it now and and continue in the metaverse of whichever metaverse people want to choose to engage with us i think that sounds good engaging them in the meta verses yeah thanks guys you