Phillip Carter's session from Monitorama PDX 2024.
Generative AI! LLMs! They're all the rage!
...and for good reason. Unlike crypto, this new phase of AI is a legitimate step change in capabilities, with companies in every economic sector finding places where it can add value. People are experimenting and C-levels are rewriting roadmaps to incorporate this tech. It's here, it'll probably stay, and we're gonna have to make it work.
So, how do you make this stuff work in the long run? You guessed, it Observability! LLMs are black boxes you can't debug in the traditional sense. The most effective way to improve them over time is to put them into production, gather good telemetry from real-world usage, and analyze that data to figure out how to improve answers. In this talk, you'll learn the basics of what you need to track, how you can track it, how you should consider monitoring health of an LLM over time, and how you can add value to your organization by feeding valuable data back into the development process.