Music NFT metadata w/ Zora API and Python Pandas - Ahmed Alkarboly

foreign [Applause] [Music] good morning good evening good afternoon depending on where you're from um we have Ahmed here from the underground Violet Rave citizen developer group uh doing a workshop on music nft metadata with Zora API in Python pandas if you have any questions they can go in the zoom chat and Ahmed would be more than happy to answer them um otherwise feel free to take the floor Ahmed thank you Anna um yeah my name is Ahmed I'm a part of the underground Violet rape citizen developer group we are an organization that just makes artists our tools for artists um and tries to Leverage The blockchain as much as we can um so today I just wanted to talk quickly go over the Zora API um and then do a Code walkthrough where we're integrating the Zora API with python pandas and then I want to talk a little bit about the possibilities with aggregation hopefully if we have time at the end open the floor for questions you can always hit me on Twitter and I'd be happy to answer questions there as well um so right off the bat the Zora API so the link up top if you visit it will take you to a graphql interface and so I'll just go ahead and break it down for us this is an interface for the Zorb API the left we have the Explorer panel with the Explorer panel does is it lets you dynamically build queries with just a couple of clicks it's a very cool in the middle you can actually see your query get built in real time as you can go and select the options um and then on the right side when you execute your query you can actually see what data is being returned uh quickly to talk about key fields we have the collection address this is how you tell the API what information where to go to retrieve information we also have a field called limit and this is how many nodes are going to be returned per page a node in the context of music nfts is just going to be a song in a collection and then there's the page navigation so I think the limit on the Zora API last time I checked was 50 nodes or songs per page so if we have a collection that's multiple Pages um when we're trying to go to the second page we have to provide it with the page navigation value and then looking at the actual data we have your standard metadata from a smart contract and then we also have some navigation information from the API that's really important for later um it has a Boolean which says hey yes there is a next page in this collection or no there's not an X page that's the has next page field um excuse me attribute and then the end cursor attribute is what would go into the server uh field in the actual query so if there's a next page you give it the end cursor value and it'll go to that to so key takeaways we can use the API to return all the semi-structured Json data for a particular publishing platform think mint songs catalog sound XYZ um even the chaos Collective releases or any Independent Artists like Jaden violet or 20 twin or arrowmist or Nexus um music spread out across different collections is not scalable uh there's a conversation of building a data set um a registration data set that you could go and reference but that's a different conversation so if we want to collect metadata in a single place we probably want to do it in a single collection repository to make our life on the back end a little bit easier um I also linked here the music nft metadata standard it's currently adopted by minsongs and catalog and I think that adoption of any standard is good especially in the music nft space it makes creating a microservices ecosystem easier um and data aggregation is needed to make these microservices possible so what's our desired end State using the Zora API it is to build a single data set with multiple platforms with our desired metadata that we can push to a storage endpoint of choice like Google Sheets or SQL or something in AWS and the way that this came about was the Zora hackathon I believe in June I made a project where I used the Zora API to query in uh mint songs metadata mint songs V2 metadata um independent metadata uh excuse me metadata from Independent Artists and then catalog metadata as well and there was even an integration from a third-party data source from futurecape and I was able to make a music aggregator which I'll talk about later if we look at the music nft metadata landscape we have a Graphic on the right made by Laura Wynn um and so you can see that as far as adoption of the metadata standard goes mint songs and catalog are the only people but you'll notice that in the user interface that that those publishing platforms provide they don't actually give the option to um they only have like a couple limited things like title and artist data standard is adopted and if you talk to those devs they'll tell you that that it is um so let's talk about the actual code to pull this information in you're going to need a couple things a code editor you're going to need python in the libraries below installed and then there's a Google Cloud API uh component to this if you're interested I can talk about it later in the DMS or something like that but uh you would push your data to a cloud storage endpoint so what you want to do is you're going to import your libraries you're going to want to connect to the Zora API and establish a client you're going to define a pandas data frame what a data frame is think about it like a data set you're just saying I want to Define these columns in the data set and then you're going to later push rows into that case and then what I'm doing next in this code is initiating variables for an API query Loop and I'll get into the actual building up the dynamic query you'll notice that it's a while loop that checks for a value of has next page so this piece of code is going to execute until we get to the last page in the uh collection it's the first page it's just going to execute a simple query and here what we're doing is we're just looking at a mince funds collection if you wanted to aggregate a catalog and mint songs and you know any other uh east-based platform the Zora API you would just provide that collection address for pages that are not the first page counter gets incremented the counter is greater than one that means that we're no longer on the first page so like I was mentioning earlier what we do is we pass along the end cursor value to the query and then the next page is brought in that's how you build a pagination feature using python to navigate through an API and then so now actually executing the query that we just built we're going to initiate the query we're going to execute the query to return the Json node page that we saw earlier on the graphql interface we're going to load the results from that query onto a data frame and then we're going to have what's called a node page the idea of a node page is pretty important because like I was mentioning earlier I think the Zora API is limited to 50 right now but that means that when we go query a collection for example that's the smooth songs collection I think there's a 100 plus 100 some 123 songs or something like that it's only going to return to top 50. so that node page when I go say run a function that says hey return the length of this node page it's going to return 50 if if I set the limit to be 50 or if I set the limit in the query to be hey return 10 nodes per page 10 songs per page it's going to return 10 for that link the length of that object and so like I was like I was saying before we use that that node page to basically iterate through every single object or every single song on that node page for analysis so we run a for Loop that starts from the first song on the Note page to the last song on YouTube page and it's going to assign a variable by navigating the Json that's returned to populate a specific variable does that actually look like so let's say for example we wanted to navigate the Json structure return in the API to get the song title for a specific song so what we would do is we would navigate the data frame that's returned from the Zora API and we would L python to say hey go look at mints once you go look at mints we have to tell python where's the what is the next thing that I should be looking at where should I navigate to next um in this case we have to index it because uh this specific Json structure doesn't have a key value so I can't navigate to it by using a words in index that's why it's zero and then I is actually the song on the Note page so because this is in a loop I runs from zero to whatever the length is of the note page so if I'm looking at the first song I would be zero I'm looking at the last song I would be 49. so I'm looking at the first token I am going to go navigate to the metadata and then return the song title and so if we look at that code again that's pretty much how it works right I go return a collection of all the songs on a specific publishing platform I then use hopefully there's a music nft metadata standard that makes my life on the back end easier and then I go and pull those tags and populate a data frame to my liking to do whatever I want to do um and then here just real quick we Define an array with the data we just scraped off of that API we load that array onto a temporary data frame and then we append that temporary data frame to a Master data frame which is actually holding all of our music nft and then uh one of the last pieces of code is checking to see if I need to run the code again to bring in more songs from the next page so right here those are those values updating um there is a uh this is the last part of the code there is a Google I'm not going to talk about it too much but essentially you set yourself up with a Google Cloud account there's some sort of authentication file that is a Json file you drop it somewhere on your desktop and then you link that Json file with file path like here and then you go into Google Sheets and then you would create a Google Sheets page and then using this code you would be able to push your data frame into a Google Sheets um and that's exactly how I built the integration Tool uh which third place in Zora hackathon um I went through I got each collection address for the listed platforms I used uh whatever they had to fill out a standardized data frame I had to do a little integration to get some additional data from catalog and sound specifically BPM intact from future shout out Anthony tape and then I used python to orchestrate all this pushed it to Google and then I had a repository with BPM and tag data and artist name artist profile and so you know what can we do with metadata aggregation we can do things like music charting we can do things like playlisting music Discovery think things like I want to dig for music on bandcamp I'm a DJ I love digging for music on bandcamp right now there's not really a good way for me to do that I can't go find um techno or house or you know like UK gay Ridge or something like that there's no way for me to easily do that um there's also a lot of implication with intellectual property music rights automation when we think about integrating um things like BMI or different Pros it's really important uh the metadata conversation is really important because that's where that information would go and it's just an additional metadata field that we could bring into our repository to run a any different number of micro services I'm another thing that you'd be able to do is ecosystem analysis so let's say that I want it as an artist wanted to go and see what my music ecosystem was using some sort of data set like this you would be able to do something like that um and then of course like music players one of the biggest needs in the space right now I think is uh having a way to have music that is published on the blockchain readily accessible whether that's through a mobile app or a website something resembling Apple music or Spotify or SoundCloud or bandcamp for that matter what I did was my integration was I curated in DJ to set using nft metadata so that Zora hackathon project produced a data set um I use that data set to build a front-end application um right here and you can see that it's just a basic table with a bunch of different filtering tools and so what I did was I searched the data set for songs that were tagged as electronic and then I curated a set and DJ did it DJ'd it and then put it on SoundCloud uh that's all I have and so I guess we have like 15 minutes or so uh we can open the store any questions thank you feel free to come off me if you want to ask your question live foreign part of the hackathon I recently deployed my own smart contract with statement.s so yeah this is right up my alley with as far as metadata and future uh blockchain um not sure what to say because I am driving I mean who had a light so uh just keeping the loop I appreciate you coming through and you said it was sweetman.if correct and attend the chat as well he's one of the Pioneers in the space for sure um I think he sees a lot of the problems and he's he's building tools to fix them you're right where you need to be this big man all right um I guess if there aren't any other questions I'm gonna give another minute or so for questions we can maybe end the workshop here but we can give it a couple more seconds um see if anyone's typing in the chat Sergio antonati again I'm just wondering how it can help um sorry for not showing my face right now I'm driving but I am dogs and full real life person um I'll just leave it at that and see how it can help obviously I make music but I'm a bigger part of the spectrum one of the most important things is education especially with the idea of metadata especially in the music space where we have a lot of artists who want are not comfortable with defining themselves and so when we have something like a metadata standard that can be pretty scary and so I think one of the biggest helps is you know supporting the developers who are who are solving the problems and then number two is continuing to have conversations around metadata and like just learning as much as you can and maybe debunking some of the um some of the rumors around metadata and what metadata is uh specifically around artificial intelligence um because it's a real fear and I think that most important thing we can do is just to get educated about the topic um just talk about it I agree um are you talking about AI as far as like artificial intelligence music because I know if you actually just push the button I don't like that yeah I think and I see another question in the chat but I think in the music nft space there's um you know I have conversations with artists and they say well I don't want to put music in a genre I don't want to be you know classified as this or that and what I would always say to that is well okay if you make techno um and I like to DJ techno and you don't put I know in the genre it's harder for me to find your music um so having conversations about that you know and because that that is how metadata works right is it is they're awesome I can use it to find your music for example or you paid on a secondary streaming platform or something like that but um the yeah I just like I said just just going out and learning as much as as we can about about artificial intelligence and information Data Systems so I think you're doing everything you got to do to help I I see we have another question in the chat hey Ahmed thanks for the the um overreal um that was super dope I'm also a data scientist and DJ so it's it's great to see um what you were able to build at your hackathon my I have two questions so my first question was um with the mint songs um you use the Zora API to query data like metadata from mint songs or like how was that incorporated in your project yeah so specific the mint songs Factory was not through the Zora API I should have clarified that that was through in the uh started integrating things you know just like weren't built on Zora as well and so I should have clarified that but the Zora excuse me the Min songs V2 is through the Zora API um and yeah that's actually the example that we use today in the Code walkthrough awesome awesome and my last um kind of tidbit is um I definitely agree with you with the AI ice pack of the space um like helping DJs helping artists kind of sift through songs because I had a specific issue with uh franchise record pool where I would like go through these songs and sometimes it would be wrongfully um put in different genres for example they'll have a Latin song in like uh they will have it as like afrobeats right even though it's some of the songs could be like the same Cadence or same tune they would have it only classified so I think it's super dope which you were able to do and uh we should definitely connect no I'm super down I love I love my data scientists and my DJs yeah uh if you follow me on Twitter or whatever I think that's probably the best way that's happened for sure thank you yeah of course thank you um yeah any other questions if not would be happy to give everyone back seven minutes