Running Smooth: LLMOps

Show Notes:

• Weights & Biases explain LLMOps: https://wandb.ai/site/articles/understanding-llmops-large-language-model-operations
• AWS explains FMOps/LLMOps and how it differs from MLOps: https://aws.amazon.com/blogs/machine-learning/fmops-llmops-operationalize-generative-ai-and-differences-with-mlops/
• Tools We Like: Databricks’ MLFlow for MLOps and LLMOps: https://www.databricks.com/blog/announcing-mlflow-24-llmops-tools-robust-model-evaluation
• LLMOps.space is a community dedicated to exploring this discipline and its tooling:  https://www.youtube.com/@LLMOpsSpace/videos
• More about Mission Cloud’s best practices with AI: Ragnarock: https://www.missioncloud.com/blog/ragnarock-merging-the-reliability-of-established-systems-with-modern-ai

Official Transcript:

Ryan Ries
Welcome to Mission Generate, the podcast where we explore the power of AWS and the cutting-edge techniques we use with real customers to build generative AI solutions for them. Join us for each episode as we dig into the technical details, cut through the hype, and uncover the business possibilities.

I'm Ryan Ries, our chief data science strategist, and your host.

In this episode, we're diving deep into the world of Large Language Model Operations, or LLMOps for short. This subject can be a bit technical, but by the end of our chat today, you'll see why it's crucial for anyone looking to harness the full potential of generative AI.

Let's dive in.

First, our usual disclaimer. No, I am not the actual Ryan Ries—I'm a synthetic language model mimicking his speech. Normally I sound pretty convincing, but on a recent update, I started mispronouncing words, so they had to roll back my model to a previous version.

And that's actually an example of what we're going to be talking about today. Because that's exactly why LLMOps matters and an example of what it can do.

First, let's break down what LLMOps really means. Think of it as the way any company keeps its large language models, like the ones we use for this podcast, working smoothly and up-to-date. Just like a car needs regular maintenance to run well, models require continuous updates and tuning to stay accurate and reliable. So, LLMOps is basically about keeping your AI tools sharp, relevant, and ready to tackle new challenges.

Now, I suppose all of that sounds fine, in theory. But is it really necessary to dedicate a chunk of your company's operations to model maintenance? That's a common question for our customers. Often, a solution may at first appear so effective that it's easy to assume you're done working on it. Sometimes, our customers assume that when they've eliminated significant hallucinations or performance hiccups, they're basically at the end of their optimization phase, and any further adjustments to how a model operates would be marginal improvements at most.

If only life were that simple, huh?

Well, without LLMOps, your models are essentially flying blind. The world doesn't stand still once your generative AI solution is launched--your customers change, your business changes, your products change, and the underlying data changes. At first, these differences may be negligible, but over time, they will add up and begin to make your model out of sync with the world. At a certain point, as its responses begin to be divorced from this new reality, you'll see its utility decrease as well.

It's easy to assume that perhaps that's the point at which you introduce LLMOps, that you simply go in, update the data, maybe perform some routine maintenance, and you're back on track. But ML models aren't that simple. Imagine that, at the end of this work, you find the model performs better at half of all tasks and worse at the other half. Do you now have the infrastructure to begin rolling back changes, one by one, until you can get to the root of this issue? If you don't, look out!

But if you practice good LLMOps, hopefully you never even reach this point, because you roll out changes to the model incrementally and frequently, always testing to ensure you haven't introduced any regressions in performance.

This is the reason it's called LLMOps, not LLM Fixes, or LLM Updates. To do it well truly is about creating an operational cadence around this kind of updating, and doing so in a methodical, careful, and repeatable way.

By embracing LLMOps, we ensure our models adapt alongside changes in our business, preventing them from becoming outdated or ineffective. And we also ensure that we can tune performance carefully without jeopardizing valuable work or creating unexpected results for our users.

So, let's get into the meat of today's topic: how do we do LLMOps right, and why should you care?

First up, you need to consider what tools you'll be using for monitoring and observability. By tracking metrics like error rates and response times, you can catch and address issues in systems before they become problems. This proactive approach is key to maintaining the overall health and efficiency of your AI tools.

Next, consider prompt engineering. This is where creativity meets technology. By iterating through experiments with prompts, you can guide your models to generate outputs optimally--with the minimal necessary tokens and with maximal accuracy.

Fine-tuning is also worth considering. And parameter-efficient fine-tuning has made this process more accessible in terms of cost. Tuning a model to better fit specific tasks or domains can make the solution both less expensive to operate and more reliable. But be prepared to pay an upfront cost for that as you begin tuning because training models is still expensive.

As an intermediate step to fine-tuning, you should consider Retrieval Augmented Generation, commonly called just RAG, which stores relevant, up-to-date context in a vector database for the model to reference. There are lots of approaches to RAG and they aren't all equally effective across domains, so you need to consider which RAG techniques are appropriate for your solution. But we see this as a fundamental best practice, which is why we recommend it in Ragnarock, our architectural approach to AI best practices.

Finally, don't forget governance and compliance. Ensuring your models operate within ethical and legal boundaries is critical. You need to safeguard your users and your organization and prevent malicious actors from misusing your solutions.

Conducting all of these in concert, in a controlled and highly automated fashion, is critical to good LLMOps. Doing all of these manually is a recipe for sluggish development and constant fire drills. Which means that you'll want to embrace practices like Infrastructure as Code as much as possible, think carefully about your data architecture and repeatability, adopt serverless methods where appropriate, and look to automate processes like testing and monitoring.

Wow, that sounds like a lot of work, doesn't it? If you're at the very beginning of experimenting with your AI solutions, you may have never considered some of these needs. Or perhaps you've gotten a first solution out the door and are only now beginning to see how hard it will be to keep it up-to-date. Hopefully today's episode wasn't a bunch of bad news!

I do have some good news for you, though.

Because wherever you are in your AI journey, I just want to make clear: Mission Cloud has been there. We've been developing highly automated and operationalized solutions for our customers in the machine learning space for years. We know the ins and outs of building a great architecture on AWS, and we've helped many organizations pay down technical debt when it accrued or take a greenfield initiative and turn it into a scaled-up powerhouse.

If anything we've talked about in this episode sounds like something you'd want to discuss, why not drop us a line at Mission Cloud dot com? We'll give you an hour of our time, free of charge, just to talk about your concerns and help you build a plan of action. We know that operations and architecture can be overwhelming or even feel like a distraction when all you want to do is build. If that's you, get in touch, let's talk. We'd love to hear what you're working on.

Well, that's it for me. Good luck out there, and as always, happy building!