Advancing Open Source LLM Evaluation, Testing, and Debugging - Berkeley 2024

[Music] how's it going everyone um I'm Jason from from arise uh arise AI what we do is we build you know we build software to like like data dog for AI we we help people understand where models go off in production uh what problems they have how to improve them uh software to help humans understand what AI is doing and and what customers care about and what businesses are doing actually don't don't really care about the modeling of G to be honest you kind of choose your model and you're not choosing a different model every day most teams if you're on Google Cloud a business is going to choose Gemini if you're on Azure you're going to choose open Ai and if you're on AWS you shoot choose Bedrock which is OD so you've already chosen your model why do I need a model eval well it gives me a gut check but what you really want is is basically understanding if what you put in in the production is going to work that the big idea is that that like these systems are are cognitive they're complex we're building intelligence there's they're doing things that even in the simplest case I'm summarizing an article I'm summarizing something um I can't like a traditional ml approach I can't use accuracy or Precision to understand its performance I've got to use AI to judge AI I have to use AI I have to use AI to understand what my cognitive system is doing and it's not just one okay one question we're finding like you have to pick different parts in your system just like you measure any distributed system or any piece of software and you've got to come up with uh these elements of judge judge points within your system so this is how people are building this this is how we're minimizing hallucinations and catching issues and making retrieval better is is basically um the what we're call we call tasky vals but it's you know you also hear element as a judge in in in production this is a reality of what we're dealing with today there's there's what are called LM spans these are like you might have a retriever span you these are like little tasks that your your your LM application is doing you have an llm judge on certain portions of those tasks that are pretty important a retrieval task is kind of getting chunks of documents back you try to understand if you've got the right docs um and and there's this template that's being applied to to the the actions of your system um so this is kind of how how this stuff works so so retrieval um retrieval is one of the most important things if you're looking at connecting private data to an llm uh typically what it involves is uh chunk up docs or data in some way um and determining what to put in the context window based upon what your task is um but one question one question we had was was getting rid of the rag systems getting rid of all the complexity how well can an llm just find a fact like how well given this entire doc okay there's 10 million token Windows there's what now 1 million Tok Windows how well does it just return a fact uh so what we did did was we put out one of the first tests out there um to we just wanted to know ourselves we were like you know this a big question all all the all all of our customers are trying to do use retrieval how well can it just get a number so what we did was we um inserted a random number into uh a doc so um these are Paul Grahams essays uh put put a random number in a doc uh and the random number was associated with like hey this is the San Francisco magic number is this so then we said well I don't want to just retrieve data um a lot of stuff like a lot of the stuff I'm trying to do is actually take action on it like I don't want to just return an SEC filing number I actually might want to do a calculation on it like divide by something or what's the percent growth from this year to last year um so we added you know to the model eval test for retrieval some more um complex approaches we said why don't we do something we we'll ask it to do something with this still pretty simple but but ask it to do something in this example it's got to find um the two Revenue numbers from the previous year and calculated percent so if you're building anything you're a hacker you're a builder and you're putting your app in production without something without evals I think you're you're you're going to be um throwing out a lot of problems thank you thank you so much [Music]