Huddle01 🛠️ Leveraging LLMs to enhance your Audio/Video DApps

hi everyone Hi hackers this is akit I'm from Team hle 01 and today I'll be demoing U project built using open a sdks and hle 01 so this is a tutorial for especially for the AI PRACK and how you can use llms to integrate with the hard1 SD case and build ke apps so yeah let's head over to it so first of all you'll need two things one is the hdle 01 API key and the other one is the open a API key so we'll be interacting with the gp4 um API and we'll be generating prom will'll be prompting the jp4 API and generating text to speech using it and then converting that uh text to speech to a media stream and producing it to the h01 sqs so it's going to be really cool and let's go ahead so uh for this tutorial you'll have to get a good idea of how the h01 work I hope you've integrated it before if not you can simply head over to the hle 01 docs and here's where you can connect your wallet and generate the two credentials which is the API key and the project ID so once you're done with that you can head over to whichever SDK you're planning on working with so we have to react in JavaScript and you can also do it on react native it uses the same react s so if you head over to the walk through uh you install two things so we've installed both the server SDK as well as the react SDK part of this because uh we're assuming it's a xgs app so if it's a nextjs app you have both the front end back end in the same Depot but if not if you have a separate nodejs server then you should install the server SDK on that instead of the client so that is one thing to note uh so yeah one once you have the AP key and project ID you can simply go ahead and initialize your hard C so you can even check it out in the JavaScript side um you have all this is in um like proper sequence so if you go to init you initialize your client uh using the project ID and once you do that then you can join a particular room so this is a room which where uh the peers can you know produce their audio video or any other random arbitrary data packets too like chat or maybe emojis so you can do any type of communication using these sdks so in order to communicate that you'll have to have a channel which is particularly the room so here you can simply join the room you can do it uh you can do it on any client so once you join the room you'll be able to communicate via the sdks so if you haven't used the SDK hle one SDK before I'd suggest that you to go through the walk through and you'll be able to figure it out it's pretty straightforward and if you have any doubts feel free to Ping On Discord and we'll be available so yeah this tutorial is a little complex so this is a prerequisite to have you can simply go to react if you're working with this and this is like a 10minute guide which will help you build a proper audio video communication tab so you can check this out then uh one this is done like once you're you know familiar with how the react sdks work and everything um you'll be able to build AAP on your own so today we'll be focusing more towards how you can run like this uh the client application on a nodejs server where you can process the audio video packets or like the text that you're receiving and send it to an llm process that data and send it back to the hardle 01 infra and uh simply to all the peers in the room so it'll be pretty interesting so if you like if you're able to do this then you can interact like you can create AI agents uh like open AI agents to join the particular room and give task to them like create like a music bot maybe tell them that play this particular song like in tell to Siri and they'll play a particular song so we'll be doing uh we'll create a PC of how uh this can be done and uh then you'll be then you can innovate on this and build a app of your own right so yeah this is a short overview of how the h01 SDK works now I'll go over the technical architecture of what we'll be building today and um how you can interact with an nlm and produce responses from that nlm to hardle 01 room okay so yeah let's move to next draw and I give a short technical overview of what we be doing so this will be the hard 01 infra right and to interact with this hard in infra you use the 01 hdk so say you have a browser application you install the H one react SDK here and then you can call methods the hle infra using hle client instance which is somewhat like this all right you'll also need to specify two parameters over here uh one will be the room ID and the other done will be the all right so let me go over how this will work so this is like the like a browser type app where you're doing this and you can simply call functions here right so in order to get the room ID you have to call an API to get it uh to do that if you head over to the 01 docs you and check out API references here is the create room API and if you head over to meeting SDK room API you can simply call this endpoint with the API key that you generated before and you get a room ID in return and what I would suggest is that only uh run this API code on the server side and not on the client side because if you risk uh like exposing your API on the font end and if the API ke is exposed on the font end then anybody can you know grab that API ke and create access tokens and DS your room so it's highly suggested to use this API on the back end side only if you're using next CH then do it on the get server s side props or recent server actions that is the suggested method um but only run this only run this API on the server site then the other thing that the join room accepts is a token and this token is basically a join room token so you can generate this using the server SDK if you head over to access token even to Quick Start you can simply install the server SD key using this command and you use the AP key here and the room ID so all this logic has to be run on the server side if you do it on the client side then your API key will be exposed and then you risk compromising your application right so and it won't really be like the server sdp won't really work on the client side either because we have the course policy set up so it won't work so it is suggested to do this on the server side all right so once you have the room ID and the access token you're good to join the h01 room and once you join the h01 room then you can simply produce your audio video streams or any arbitr data to the hard 01 infra and other clients uh in particular could listen to that particular particular Stream So if I do hle 01 like hle client. join room then I can send over some data so say I send over uh a particular data so if you head over to the uh JavaScript dogs then you can see that we have the send data functionality and you can use the h01 stdp to send over data to particular fers so let me copy paste this command and let meat it okay so if you send over this data you can if you specify two as star then you're basically broadcasting this data to everybody in a harder 01 room so this is how you send data and similarly you can receive the data on the other end so you can have uh multiple browser web apps interacting with the same room and um communicating with each other or the other option is that you have a nodejs application which joins a r as a bot say like a music bot if you want to do that and that music bot can then list into these streams and interact with it so that's what we'll be looking into today so uh once this send data has been done uh you can simply listen to this data as well using this uh receive data receive data event which you can do on this side all right and once you receive this data you can interact with this data however you want so what we'll be doing is um we'll be using the hle 014 SDK we'll be installing it uh on this side let's color code it to something like orange okay and this will be the web cod we'll install the web for hdk here and once we install the web for SDK we'll be able to listen to these events and everything and once we listen to these events then we can do whatever we want with the data so say I said hello everyone so this particular node just c will receive that hello everyone and uh then we can use that hello everyone to you know uh give that to an nlm convert it from text to speech and then produce the response from the bot into so what we'll do is we'll receive this data with this function and once we have the data over here say we have hello everyone we can con it to this and we'll have the open isdp somewhat here let's get this color let's so we have open a so we give this hello everyone to open and it says it gives you a response in return and it says like hello or something like that okay and once this is done then you can use the speech to text API or text to speech API actually uh to send over this response once again and then so and then you receive like a np3 file in return which is audio Do MP3 and once you receive this audio. MP3 you have to convert this into a media stream which is recognizable uh to the 01 SDK so you do a media stream and we'll be doing this using a library called uh a AO a a i RTC which is a python library but we'll be using a JavaScript rapper for that and once we have this media stream then we can simply produce this media stream back to the info and once we produce it back to the insta then any one of these feers can you know listen to this media stream and we'll be able to you know listen to that that particular b top so that will be really cool and this is how it works if you have any doubts in the whole uh process you can feel free to fing us on Discord we'll be here to help you out but yeah this is how it works we I'll go through the process again so you join the room on both ends you send the data from browser uh web web app and once you do that you receive it on the bot site this mods app which is our bot and this B then sends all the um all the data to open hdps Rees the response then converts it into an MP3 using the htps again and then converts it into a media stream and produces it back so this is what we'll be doing so we have two uh example apps over here one is our browser example app and the other one is our mod example app we'll be going over both of them so this is how the technical architecture is all right so moving on to the code so if you want to check out the browser example app this is the client nexts example app and you'll find this in the project resources on so you can simply clone this depo and install the dependencies either using bun or pmpm whatever work for you then uh the other one is this note PC which you can also clone you'll find this link in the notion and the notion is will be available on the global resources page uh for hle 01 so yeah um you can clone both these apps and once you clone these then you'll be able to join a particular room and interact with it so let's go over the browser the app first and see how it works so I've gone ahead and flown uh both the exam apps uh on this side so this one is the nodejs example and this one is the browser example all right so let me go ahead and install the dependency which installed and I can run B and I will get access to Local Host 3,000 so this is what the app looks like it's pretty simple created a very simple UI and you can enter a display name if you want and join the room now that you've joined the room uh you can enable the video so say hi this is how it works and I can join the same room using a different Tab and you can see me and I make video here then you can see it over there and this chat also works right so you can see it like this so now what we'll be integrating is that whenever I send a message through the chat um I want my nodejs spot to listen to that chat and interact with it and send a audio response back okay and that's what we'll be doing using this sjs app so for the client next year example app you can simply go ahead and check it out on the GitHub and if you need any help on that you can ping on our Discord server it's pretty simple or even check out the hle 01 dos uh you can check it out you can check out the walkthroughs you can check out the react walk through in this we show how to build this app and it's pretty simple it's a 10- minute guide and you'll be able to create uh a similar app like this on your own right so let's over head over to the nodejs app which is the interesting part and on how to uh integrate a nodejs app like integrate the 01 to okay so you can skip all this for now text to screen and response I'll quickly show you how to get connected to the h01 infra on the nodejs site first so it's pretty simple you install the hardle 01 SDK the web core SDK on the nodejs app so if you check out the package.json you'll find that we've installed the code over here I'm using the Alpha version over here because we've made some improvements to make it compatible with um node sockets so feel free to go ahead and install the Alpha version you can do that using uh you can do bun I atate h01 um dep core with the atate alpha tag I'll show you why you need the alpha tag later so you can simply do the ad theate Alpha tag and you'll be able to access this SDK so once you're done with installing the web core Alpha then you can install the server SDK as well as you'll need to generate uh tokens for the bot to join as well so once you do that uh you can simply initiate the client as we did before we've given it the project ID and now and we've given it some other options of active spe and logging as true so you can even switch the this off if you'd like then this is the interesting part in the alpha sdks is given this functionality of adding a factory function a hand a factory Handler over here this is a custom worker Factory which enables you to do certain things like modifying the media streams and sending custom media streams created on a nodes on a nodes server or or on the server side so I'll show you how this works so make sure to install the alpha SDK if you want to have access to this uh particular attribute so what we do is we' installed this uh media sou client AI RTC package as well so you can simply install this and get two things out of this one is the AI RTC media stream which mimics the media stream in the browser and we use this to generate MP3 files like we use it to generate media streams from the MP3 file we'll be going into that further and to uh like create a worker to do this you you'll have to call the create worker function so you can use the create worker function and you can Define the log level I've set it to debug and you can create the worker over here and once you have this worker then you can create the Handler Factory and just pass it down to this so this is like passing a custom Handler Factory to the hard Cent instance so that you can get access to more features and Custom some features such as producing AO RTC media streams so once this is done you'll be good with the client side and then what you can do is you'll have to have a room ID as well so you can generate a room ID uh using the API so I've already added the logic for generating the room ID inside the web example app so if you run that uh you'll already find this home ID over here and you can copy this and paste it here so for to make this work you have to have the project ID and the API key pair match the room ID so if You' used the same API key paer to generate the room ID only then you'll be able to connect to the same room otherwise you will not be able to do it right so generate the room you put the room ID here then we create the access token so as I described in the while going over the walk through you can simply call the new access token from the server SDK or and create uh create an access token for your partt so what I've done here is that I've created a host privileged um bot so you can even dial down the Privileges if you want but for demo purposes I've given it as host and you can give it a display name or an AM URL as well and you'll have to mention the same API key you use to create the room ID so once this is done and you have the access to you'll be able to join the room from the nodejs site so you simply have to call client. jooin room and pass it the room ID and the token it's pretty similar to the web as well we do the exact same thing here as well and then the only difference in the nodejs one and this is that we pass custom Handler Factory and uh we generate media streams from MP3 instead of accessing it directly via your uh media devices so yeah we'll be going over how to do that so once you join the room I've given it a sleep of 5 Seconds so it's a simple sleep function which resolves a promise and a waits for 5 seconds so this works like a set time out but it says inse so I've made it uh like this so after joining the room and after 5 Seconds all your event listeners will set so once once the bot has joined the room after 5 seconds he it will be able to listen to room join so it will um give this but this won't uh really trigger so after 5 seconds if any one sends any data to the bot the bot can listen to that data and process it so before going ahead with this logic I'll give you how the MPC files are converted and how it works so we have speech.

MP3 and open this and I can show you what this I can assist you with a wide range of tasks and answer any questions you may have just let me know how I can help you so this is an MP3 file that I've already generated using open air text to speech and if you head over to the open a docks and you see text to speech the sun rises in the East and sets in the west this so this is the same voice that you heard before so it was generated using this open a text speech model and you can turn text into Life Flight spoken audio very easily so you can simply follow the node just gu on how to convert that I'll also show you how I've done this in the code but this is the gist of how it's done you just simply pass it the input and you pass it the type of voice you want and it will generate speech file for you right so we'll be coming over this as well once again uh but just to get an idea I've already created a speech MP3 file which is um which is started whenever the board joins they uh just speak this out and uh they ask you whether like what all they can do okay so we use the speech MP3 we resolve the path and we create an audio stream using it using the worker that we defined above of so this worker is then used to you know get the user media from the particular MP3 file and it creates it into a media Stream So this media stream uh then has the tracks for that particular uh audio and that track is then you know created using then we create a media stream out of it and then we send this media stream back to enable audio and this enable audio function is available on the local tier client and what it does is it just enables the audio like it produces the audio to all of clients on the hardle 01 in in that particular room so that's what it does and I've also picked up this simple JF image GIF kind of image and I've converted this into a video stream and published this as well so whenever the bot joins the room uh it will first speak the text and then it'll uh also show the video of that it has joined so to show this quickly I'll run this node server as well so to do this you can simply write pev and it will run the server and on the browser side assist you with a wide range of tasks and answer any questions you may have just let me know how I can help you so what it did was it converted That speech. mp3 into a media stream and then sent it over to this room so we could hear it and it also converted this video into a media and sent it over the1 sdb and rendered it out here so this is how it's working then moving on to the logic where you can chat with it that is pretty cool so I hope you understood this part this is the main thing that we're doing and this is how it works so before going on to receiving data and receiving data from the chart and interacting with that I'll quickly show you how like I'm prompting the open a API and getting the data and converting that data to text to speech so we have two things over here so we I have created this function get response from GPT and it accepts an array uh of messages so how the gp4 completions work is you have to pass an array of the whole conversation to the open SDK so if you pass the whole array of uh messages to the completion to the gp4 then what it does is it analyzes the whole conversation and based on the answer that that it has provided before and the based on the question that you've asked it before it gives you a particular answer to that so to see how what it looks like you can check out this um interface so I've created this content so this interface which has a Content field which is just a simple string of like uh what the message is so so if I say hi good morning uh it will the content will go inside this then there is this role feeld which uh the which the open uh SDK itself accepts which is basically the system or the user whether that message is from the user or whether the whether it is a response from chart GPT so that is what this uh role defines so what we're doing is that whenever we sending a chat and we're receiving a response from GPT we're storing it inside an array to keep a track of the conversation and we so that we can send it back to GPT APS and get a relevant response so yeah so whenever you uh give it a messages array and you give the whole context of the conversation to GP it uh processes the comp uh process the whole uh conversation before and it gives you an answer in just answer and return so this result is basically a string which it gives and we just return that particular result that's all it does it's very simple just 20 lines of code and we're done okay so this is how the text generation works so whatever your conversation is based on that it will give you a particular response so once we receive that particular response then we want to convert that into an audio so to do that uh what we do is this we have a text to speech function all right and what it wants is it wants is uh it wants a text file as an input and then it converts the text file into an MP3 so we pass the input as string so the input is basically the response that you received from chat GPT uh which I showed you before and that response we send it to text to speech and it accepts it as a string and then converts that text into an MP3 buffer all right and we use convert that buffer into an MP3 file and write it to write it to the system using the file system all right so the logic that I've WR here you can ignore for now so this logic you can ignore and just see this part we simply put the input inside here and we receive an MP file as a buffer and then we convert that buffer uh into a file so we write that file into our system itself so I'll show you how this works once I send the data so let me say I uh what is 2 two so now it has generated the speech file as an MP3 so we receive this message as 2 + 2 = 4 right and once this message has been received then what you can do is you can simply convert this message to an audio St so I'll show you how that's done so once we have this speech. MP3 over here uh what we do is we grab this MP3 file and then we convert it into a media stream and then send it over uh let show you where that done okay I hope you understood the part where the conversion is taking place now we can move on to the part where we send data okay so let me just restart this out and I'll just create a new room okay so before uh showing the demo I will quickly show you how the chat functionality is working so whenever I uh like I'll join this to and whenever I say like hi over here what it does is it sends like a send data message that I showed you in excal so it gives you this local p. send data and sends like hello everyone uh to to the server and then we can listen it on the other side so once I send it using this particular function okay then I receive it on this end so I do receive data and then this is the data that I've received and I just check whether if it's the bot who sent the message or if whether it's the it's a person who has sent the message so if it is a if it is not a bot then I just push the messages in uh to the on to the array otherwise I say processing reply from the bot and um once this is done like what I do is I put the messages array which the user sent and get a response from GPT if there's no payload or error then it will show no payload but this likely won't happen unless you have a networking problem but then you have a payload and you push the messages to the push uh the message to the messages array with the response you get from GPT and then you create uh then you create it text to speech so this count just Returns the MP3 number of the file which has been return and keeps a track of them and once you have that then you just simply write it and grab the file from the path and convert it into an audio stre so you convert the text to speech and you write it to file in this function and once it's the file has been written you read that file and convert it to an audio stream and then that audio stream is sent like we disable the audio and then enable the audio and send the audio inside here so once this is done you can even use the replace audio stream function but just to show you how this works I've have done this so you can even use the replace audio stream to replace the current audio stream with the new one and it will split out the audio so this is how it works let's quickly test this out so uh one quick thing to noce whatever files that are generated you'll have to delete at the end of when you close the server so keep that in mind I'll do one and now I should receive the the bo speaking any time okay I think I've use the wrong room ID let me op theid on the you with a wide range of tasks and answer any questions you may have just let me know how I can help you all um can you some Ms for me I'd be happy to help with any math problems you have what do you need assistance with and see what is sin x by cos x functions sin sinx and cosine cos x can be related to the tangent function Tan x as follows by definition the tangent of an angle X is given by Tan x equals Frack cost therefore therefore Frac X is equal to tan X so in terms of tangent we have Frack xal Tan x if you have any more questions or need further assistance feel free to ask see how perfectly this works and you can even see the response logged over here so this is the messages are that we've been maintaining and based on the previous conversation that we have like the whole flow of the conversation it gives me a particular answer and speaks it out so yeah this was the tutorial and I hope this helps uh any doubts or any questions that you have can be forwarded us to on Discord or you can ping me as well personally no problem I'll help you out and you can check these two Depot out on the notion do it will be mentioned there and you can also find it on the hle 01 GitHub as well uh it'll be definitely be mentioned on the E Global prer section and yeah I'm curious to see what you all Builders build using llms and yeah who do