Efficient Fine-Tuning and Serving of LLMs that Rival GPT-4 - Berkeley 2024

[Music] today we'll be talking about two open source Frameworks that we've built and maintained the first of which is called Ludwig so the idea behind Ludwig is that it's an open-source declarative machine learning framework uh people don't really like writing deep learning code and it's super error prone um and it can take weeks and weeks and weeks to get right and we designed Ludwig to essentially describe your machine learning pipeline as a config file or a yaml file and all it has is a few very simple components uh you want to find your myal 7 billion in the same way that we did Laura land you specify an adapter so you can say I want to use Laura you specify your base model can be any open source base model that you like if you want to use quantization which is a method to compress the model so that it fits on really cheap Hardware such as a T4 GPU which is probably one of the lowest end gpus that you can get um and then you can adjust trainer parameters so all of a sudden you can shift the way of thinking of like uh of training models from writing all of this spy torch code to just thinking about an idea that you want to try changing two parameters and kicking off a job um and so once you have this yaml file you just translate into python by uh importing Ludwig model which is sort of our client um passing in the config and then calling train so extremely simple and this is pretty much what we did uh for all of the lower land adapters the one thing that's really cool about Ludwig is that we've run thousands of experiments find uning models across various base models and data sets and we've basically picked in or default we default to some of the best hyperparameters that generally maximize performance out of the box um so you get a lot of the learnings that we've learned that we have just out of the box cool so we're at half the battle we can find tune models to Ludwick there's also other Frameworks uh I personally find not that not just because I work on it but this idea of declarative Based training is very powerful because you can very quickly iterate on your ideas and get um find an optimal model so we've reached we fought half the battle we have fine tune models we know they're somewhat good from learning metrics and simple inference but how do you go from there to actually taking it to production and this is where most companies most practitioner sort of fall short the second open source project called Lorax uh which stands for Lura exchange um and the idea is that you can dynamically load in adapters run inference um in a very efficient way and package all of this on one GPU and you can serve you know we've tested well upwards of 100 finetune models on one GPU with virtually no uh latency degradation or Trooper degradation um and I'll walk you through why this works and sort of uh what we've done um so the first idea is this notion of dynamic adapter loading uh a lot of fluff words for a very simple idea that traditionally when you have a deployment you have a user you have you have a request queue and the idea is that you know you're a client you send a request to your model and it gets put in this Quee you generate a response and you return it back and you do this subsequently for many different requests right that could come from many different users uh something like that but instead of doing this once per model what if you were to create a request queue for each adapter and dynamically pop off those requests from the from the queue load in that adapter run inference and then offload that adapter so very simple idea we're just making use of of what Laura does it's very common this idea of multi-tier weight caching again fancy words simple concept the idea is just that the way memory is traditionally organized is that you have faster memory at the top uh which which can do computations really quick but it has it can only hold very few things in memory and then uh you know as you work your way down to your solid state drive or you know dis you can hold everything but it's extremely slow to retrieve to write to read and when request and these sort of like squares that you see you can think of those as Lura adapters so when a request comes in there's three things that can happen the first is that your Lowa adapter is already loaded into GPU memory in which case you just run inference return the response you know simplest case the second is that you get a request that comes in with an adapter but that adapter is not already on on GPU so what you do is you you always as I just said the memory is organized as a cache in this hierarchy so so you grab a lower adapter from the layer Below in this case you'll go to CPU memory and see if the adapter is there and if it's already loaded you'll pull it into GPU memory and run that request so again it's pretty smart it only does this when it has to but the third scenario is what if your GPU memory is full with adapters and there's a request that comes in for an adapter that's not already there in which case you need to do two things you want to offload the least recently used Lowa adapter to a lower level of memory while prefetching or stretching the adapter that you actually need and so that's how this mechanism works it's all very Dynamic uh sorry if I dive too into the weeds there but it's super important to make this very efficient with that how do you actually use Lorax there's two steps the first is you spin up a Docker container with the model um and the second is that you can use either curl or a python client just really just call initialize the client and call generate uh you can optionally pass in an adapter ID which can be local or on hugging face and you can specify response format so if you want to things like Json mode where you have fixed schemas and you have more reliable outputs those are things that you can do as well other things that we support for those who are sort of uh Deep In The Weeds uh you know we serve quantized models there's a lot of quantization strategies so we support most if not all of them uh we do c our graph compilation which is another way to essentially speed up computations once the graph is compiled Medusa heads is a new idea where you can decode multiple tokens as a time at a time instead of a single token so it speeds up inference uh and then we're adding function calling support in the next two weeks there's a lot of content I sort of went through sorry if it was a bit rushed but there's this new gen life cycle that we can sort of summarize right you serve a base model through Lorax uh we also support your base model entrance uh you put that in production with your open source model you collect data then you can use this data to fine-tune your model through Ludwig um and it's really easy to like as I mentioned to iterate and try these experiments then you can choose to serve one or more of the one or more of those again through Lorax and you can actually do both base model and fine tune inference through the same deployment so um super fast super easy and mostly low code or at least we try to keep it as low code as possible don't want to plug this but I have to in case you just want to do it sort of in like a managed way and you don't want to plug it all in there is uh free trial that we have where you know it's built on top we prabas is built on top of Ludwig and Lorax and it's all managed for you uh so I'll pause for one second in case you want to get the QR code if not totally fine and yeah I'll pause right there thanks so much [Music]