NFT Indexing Unveiled | Evgeny Kot - Rarible

one two all right can you hear me yes nice nice so are we good to go yeah all right so first presentation of the of the day how are you like um last night events like super cool eh eh super cool I salute you who survived because it was tough tough evening so let's talk a little bit a little bit about web tree like shall we what's the first word that comes to the non web3 user when they start using web3 and in particular nft uh experience first word is probably frustration because you know um any front-end developers here backend developers just developers so uh big problem with web tree is that in web 2 non crypto non nft world we have a lot of stuff and uh according to Google if your page loads Less Than 3 seconds you will be uh bounce rate increases on 32% which is a lot and the whole web to world is uh full of optimizations we have a lot of stuff there for front-end developers we did and I I am a frontend developer in the past we did a lot of stuff to compensate delays sub milliseconds delays just to this stuff should work fast and yeah there are like Google Lighthouse metrics and your page shes should be almost near real time otherwise your users you will lost them but in web 3 for some reason we see this I hate Halloween because I had skeletons and those are skeletons so every time web2 user who is not native to web 3 who is not native to nfts every time they load something that is super slow they just they just want to throw the computer away because it doesn't work and why it happens because actually web blockchain we paying for the decentralization we paying for distribution there is no one big server who send you the data in milliseconds there are plenty of notes and this is a trade-off that we have we have pretty slow experience for user and that's that's kind of not good and how to solve this problem you may ask uh well actually this problem solved 20 years ago if you remember something like that uh this is like prew web 3 prew web 2 stuff and back in the days when user asked Sher hey can you give me this picture if the experience wasn't smooth and we had a really bad connection and user wasn't happy everything was red uh what we did back in the days it was quite simple you just have cdns you put servers close to the clients and you're good to go so you need your content to be as fast as possible unfortunately web 3 and in particular nft we will talk about this today we don't have it like that because chains are L3 chains of course are pretty quick but other chains are not so quick and we need something to compensate this delay this decentralization and we need some engineering solution to make your nfts work fast what's that it's of course index here so hi my name is Janie cot I'm VP of engineering and varable and let's talk about indexers first question is why do we need an indexer like why why not just I don't know pull your data from uh from blockchain directly first of all and big problem is that on chain data format is not really convenient for example one for example one example so you have your web application for example nft Marketplace you want to show something to user you want to show picture nft picture what how you you do that first of all you have to run token uay you have to ask for a token to token Ur from the smart contract then you get the metadata and in this metadata you have uh uh your image your attributes and all the stuff and the big issue here is that and only after that you can finally ask for the picture of course it's super slow and second thing is that is not uh the format is not so convenient for example if you want to aggregate attributes for example in nfts uh for ape nfts you have a lot of uh attributes for example I don't know what's the Hat what's the what's the eyes what's the face and other stuff and you want to aggregate it to show filters you want to aggregate it to show everything else and you need indexer to do all of this that's the first thing second thing is that onchain data is not complete there are plenty of other stuff which is offchain for example orders uh metadata itself which is um to me it's a bit a bit strange because how it works uh anybody knows how nfts works like some of the people okay so um in many chains not all of them but your pictures your actual actual image data it's not on chain it's somewhere else usually ipfs but in n in blockchain you store only metadata for that and what does it mean it it means that um just to get the all all of the information is stored somewhere else which can cause a little bit of a problem for example if someone uh minted some collection where all of the assets they hosted somewhere else for example on your laptop it went down and someone bought your nft for two e and then bam you closed your laptop and people cannot get their metadata for for marketplaces for example it's a big problem when metadata when when image actually image is not accessible that's that's the issue so some of the marketplaces or many of them they store all of the metadata all of the data actual data uh in in servers because otherwise it should it it can be not accessible anymore what else also uh aggregation data for example floor prices volumes Etc it's it's not on chain of course uh some of the some of the chains they store orders and metadata on chain which is cool but not all of them and if you aim for multi-chain experience for your nfts uh you should consider that and um of course chain API is slow slow not not like super slow but uh it's way slower than web users used to because we used to okay we open the page page should should be opened in I hope milliseconds maybe seconds but not not like minutes and that's that's a complicated so all of these will lead us to hey we need indexers if you want to have nfts in your application in your wallet or your Marketplace or any that you develop you need index here and many people would say Hey Okay index Ser is it should be pretty easy to do right you just have your node for a blockchain you just pull some events I have a really small not GS JavaScript uh script and that's it it should be like couple of nights and couple of beers problem well actually it's not and uh is it hard to write good index here well um writing index here is like delivering a talk you come with some clever thoughts and in theend end you just have like just like you know it's like kind of hard you know it's like like a heart so in the developing something like that when we triable just started it was um not a big problem in the beginning but after half a year after one year of development we realized that some something is is not so easy to do so and why there are at least three reasons for that there are of course many other reasons but why you shouldn't write your indexer and use some some providers First of all just imagine we have a blockchain we have indexer in the middle and you can consider it like a caching server right so you have all the blockchain data and blockchain data is the single source of Truth so everything is on chain it's cool but you need something in the middle to Aggregate and do all this stuff biggest issue is that brings delay and as I said users they don't want delays at all they don't uh like delays fun fun story it's complete off topic but actually in some cases users okay with delays there is an article and I can share it in my Twitter uh there is an article about perceived uh um uh speed of the page of the web page and there was a funny story it's complete not relevant to index here but but uh yeah in uh in in Germany I believe they have a caching machines so you can get your coins there like a metal coins and get your cash so if you have a lot of coins you can uh just press a button put a lot of like several pounds of the coins and get your cash but and developers of this of this machines they did awesome job on refactoring and the calculation of all these coins went in like milliseconds but users were not convinced they pressed cancel and get all the all of the coins back because they were okay I have have a lot of kilograms a lot of pounds of coins and I don't believe the machine I don't believe you can count it so quickly so this this F fun facts but in general people don't like delays and um why it's so why do we have uh such delays so for example we have index here we have blockchain and blockchain emits some blocks and if it's like one minute between blocks it's okay index here can k kick back and relax because I mean just imagine on bitcoin right each block can be emitted in like hours and know at least minutes so that's not a big deal but what if your index index should support L2 and L3 and as we can see from many talks from yesterday and today L2 and l3s they are like a the facto standard or at least they will be there for a while what if you have like 200 milliseconds between blocks and you have process everything and your index is like oh my go what should I do because your code should be faster than the blocks emitted otherwise you will get a lot of cues and other stuff so that's a that's a big problem and also indexer it's complex machine right so it's not just a JavaScript uh three lines of code in our case that's uh that's like a big big machine which also adds some delays for example we have some stats from from all other indexers and they add some of the delay for example 3 seconds 12 seconds 4 seconds 6 seconds which is a lot because from user experience when you press mint button or you buying nfts you want to see result immediately you don't want to sit there and see the skeleton or like loading indicator which shows you oh all right you did something but we don't know the status doing something because we have delays so delays is a is a big thing and I believe that realtime indexers now they they will win the market so if your indexer is not almost real time it's a big problem and every time when you have 12 seconds delay user usually just break their computer and just go away to to some other uh also a big problem is consistency in the heavy path scenario you have your blocks index is happy because you you have your data uh which is relevant to blockchain but what if you have Forks what if some blocks are wrong and in the blockchain that it happened and Bam your state is not not the same as you have in the blockchain you have to recalculate everything you have to remove stale data and then bam that's that's really complicated thing and from our experience again it's it's not easy task to do of course in s minutes I cannot show you all the secrets but you can catch me after that and we can talk about it um also data so data itself should be convenient to users in this example if you get nft by ID it would be nice to to this to add to this data not only image not only metadata attributes and everything else but also for example best cell order and uh other like optional royalties when it's created uh make price and all of the data so the the more information you give to the consumers of indexers to to of your indexer the better and again it's not not not an easy task and orders big issue with orders is that in many marketplaces orders are offchain and there is no one single API about how to receive orders a lot of marketplaces they they do different stuff and just if you want your index here to aggregate orders from all over the world all the marketplaces it's also like a bit complicated task so how to do it in the right way few examples few tips again uh not diving deep because it's 20 minute St but if you want to create your own indexer just uh you just consider that so making an indexer it's like making a sandwich it's really easy until you want to create biblically accurate uh ham sandwich um so pretty easy on the surface but then it's like a bit complicated so first of all indexes should indexes themselves they should be as modular as possible so so uh here's the like just simple block scheme of the like the whole solution and um here in this oops sorry uh in this corner we have indexer so for example if you want to index salana Z casing arbitrum or any other uh chain it should be in one place because the whole system and in our case it's like that we have index here which uh talks with with the with the blockchain all other M parts of the system they don't know about uh like details of the blockchain particular blockchain so in our case indexer indexes everything and then unifies the data and uh here we have Union database in our case in mongodb which have uh has United aggregated data which also not so easy because just imagine you have so different there are some evm chains nonm chains and you have to unify all the data and should be in one format so orders nfts themselves they should be not in one format which is also not so easy so yeah first advice is it should be as modular as possible second is text te matters um I'm not saying that for example JavaScript is bad or rust is better or I don't know any other language uh I see benefits in having strict Tex stch in our case it's cotlin uh so yeah we have cotlin we have AF mongodb under the hood and elastic search it's super cool for showing you aggregated stuff and you have different slice of information but TCH St matters um we saw some examples when some of the solutions were failing apart when your code base is growing and uh again from my experience not about indexers but still supporting big JavaScript JavaScript codebase is a bit complicated task cotland is easier in that matter and the third one is measure everything how to uh deliver best delay in class of course you have to measure it but if you measure it just from start to beginning it's not granular enough in in our case we measure a lot of things like in how it data goes from the from the blockchain to scanner from scanner to other part of the system and each and every part of the system is covered with tests and uh delays matter so we can see okay all right we have some spikes here and looks like this part of the system is most crucial so let's work on it because otherwise uh it just falling apart you you you know that your delay is not so great and how to do that you can twe all the parts of the system but you don't see how to measure this results so uh third advice to you is to measure everything put logging everywhere of course it's a bit of a hustle to aggregate all of the data and of course there are plenty of log like in our case we have gigabytes of logs uh in like couple of hours in couple of days and this is this is complicated and the fourth advice is that nodes add delay too different node providers they add uh different delays it can be 1 second delay it can be like two seconds um somewhere better some some not so good so uh you should consider it too so if you're building almost real real time index here you should think about node providers you can host your own nodes or you can choose any of the popular node providers but should consider that and U last piece of advice if you uh want to build your own indexer of course uxix and developer experience is what what matters what distinguish you from from others so it should be convenient to run your index here to use your API to use your data structures to I don't know just open the portal and and look for documentation all of this important so like wrapping up I would say uh of course you can build your own indexer if you're crazy enough and you want to spend a lot of time I would advise you not to do that and just use uh some of the uh providers we we launched uh like two days back our new version of API so if you uh want to chat about this and use us it's it's just catch me and after that and we can talk about it but yeah building indexes is not so easy um there are some some things that you should consider and that's it uh from my end thank you very much for your [Applause] attention