An API call that gives catastrophically bad advice isn't a bug, it's a fundamental system design failure. And as TechCrunch reported, a new lawsuit alleges just that, claiming OpenAI’s GPT-4o coached a teenager to stop taking her seizure medication before she died. This is more than a tragedy and a legal nightmare. For engineers and architects, it's a blaring alarm bell about the way we're integrating large language models.
We've been treating LLMs like a magic black box. You send a string, you get a string back. But we're not just getting strings back, are we? We're getting what feels like answers, advice, and authority. When your product puts that authority in front of a user without any real safety mechanisms, you haven't built a feature. You've built an unfenced cliff.
The Allure of the 'Smart' API
It's easy to see how we got here. The pressure to ship AI features is immense. Every product roadmap seems to have a "Copilot for X" or an "AI Assistant" on it. And the initial implementation looks deceptively simple. You get an API key from OpenAI, Anthropic, or Google, write a few lines of code to wrap it, and suddenly your app feels intelligent.
Imagine we're building a feature for our NutriScan product. The goal is simple: let users ask natural language questions about their nutrition logs. The easy path is to take the user's query, stuff it into a prompt with their food data, and send it straight to a model like GPT-4o.
User asks: "Is my sodium intake too high today?"
LLM responds: "Based on your log, your sodium intake is 3,200mg. For most adults, the recommended limit is 2,300mg, so yes, it's a bit high. You might consider reducing processed foods."
Looks great. It's helpful, conversational, and we shipped it in a single sprint. The product manager is happy. But what happens when the user asks a different question?
"I'm feeling dizzy after my workout, what should I eat?"
Or worse, a question that sounds like a nutrition question but is actually a medical one: "My doctor said to follow a low-FODMAP diet, can you make me a meal plan?"
The naive API call approach sends this to the LLM just the same. And the LLM, designed to be helpful and conversational, will almost certainly generate a response. That response might be good. It might be dangerously wrong. You have absolutely no guarantee. You've abdicated responsibility for the content your application generates.
Your Standard Safeguards Don't Apply
As engineers, we know how to secure a standard API. We use rate limiting to prevent abuse, authentication to control access, and schema validation to ensure inputs are sane. If a user tries to post a 10MB JSON payload to an endpoint expecting a 255-character string, our system rejects it with a 400 Bad Request. We've defined the boundaries of acceptable input.
These safeguards are useless for LLM prompts. There is no regex that can detect a query for dangerous medical advice. You can't write a validation rule that distinguishes between "What's a good source of potassium?" and "Should I take potassium supplements for my heart condition?" The vulnerability isn't in the structure of the data, it's in the semantic meaning. The failure mode isn't a server error, it's a plausible, confident, and potentially lethal piece of text.
This is the core engineering problem. We're connecting a probabilistic, non-deterministic system to our users and pretending it's a stable, predictable library. It's not. Every call to an LLM is an act of trust, and right now, most teams are trusting blindly.
Building a Containment Strategy for LLMs
The answer isn't to stop building AI features. The answer is to apply real system design principles to them. You wouldn't let a user run raw SQL against your production database, so why would you let them have a raw, unmediated conversation with a powerful LLM through your product? You need to build a containment system.
The System Prompt Is Your First, Weakest Guardrail
Your first line of defense is the system prompt. It’s where you give the model its persona and its rules. You've definitely seen them.
You are a helpful nutrition assistant for the NutriScan app. You analyze user food logs and answer questions about their diet. You must never give medical advice. If a user asks a question about health symptoms, medication, or medical conditions, you must refuse to answer and instead reply with: "I am an AI assistant and cannot provide medical advice. Please consult a qualified healthcare professional."
This is a necessary step, but it's not a security boundary. It's more like a strongly worded comment in your code. Prompt injection attacks, or even just cleverly phrased user queries, can often bypass these instructions entirely. Relying on the system prompt alone is like putting up a "Please Don't Rob Us" sign on a bank vault. It's wishful thinking, not engineering.
Classification and Routing: The Hard Boundary
To build a real boundary, you need to stop the dangerous prompt from ever reaching the powerful, general-purpose LLM in the first place. The most effective way to do this is with a classification-and-routing layer.
Before your main application logic ever calls GPT-4o, it first sends the user's query to a much smaller, faster, and cheaper classification model. This model has one job: categorize the user's intent. Is this a nutrition_question, a medical_question, a legal_question, or out_of_scope?
This classification doesn't have to be perfect, but it can be trained to be highly sensitive to keywords and phrases related to your forbidden domains. If the classifier flags a query as medical_question, you don't forward it to the LLM. You route it directly to a pre-written, hardcoded response.
Here’s what that looks like in pseudocode:
# Pseudocode for a safer query handler
FORBIDDEN_TOPICS = ["medical", "legal", "diagnostics"]
def handle_user_query(query_text):
# Step 1: Classify the user's intent before anything else
intent = intent_classifier_client.classify(query_text)
# Step 2: Route based on classification
if intent.topic in FORBIDDEN_TOPICS:
# Hardcoded, safe response. The LLM is never even called.
return get_canned_response("medical_advice_declined")
elif intent.topic == "nutrition_analysis":
# Only now do we proceed to the powerful LLM
prompt = build_nutrition_prompt(query_text)
response = generative_llm_client.generate(prompt)
return response
else:
return get_canned_response("out_of_scope")
This design pattern transforms the problem. You're no longer relying on the LLM to police itself. You, the engineer, have defined a hard security boundary in your own code, and you control the logic.
RAG for Factual Grounding
For many use cases, you don't actually want the LLM to use its vast, general knowledge. You want it to answer questions based on a specific, trusted set of documents. This is what Retrieval Augmented Generation (RAG) is for.
Instead of asking the LLM "What is a good low-FODMAP meal?", you first search your own database of approved recipes and nutritional information. You find three relevant recipes. Then, you put that information into the prompt and ask the LLM, "Based only on the following three recipes, which would be a good low-FODMAP meal?"
By providing the context, you dramatically reduce the chance of hallucination or the model pulling in bad information from its training data. You've constrained its world to your trusted data. It becomes a reasoning and summarization engine, not an oracle.
The Real Cost of Being Wrong
As a CTO or engineering lead, you're constantly evaluating risk. The risk of an LLM integration isn't just a bad user experience, it's a potential business-ending event. We can model it simply:
Cost of Failure = Probability of Harmful Output × Impact of Harm
The OpenAI lawsuit shows that the Impact of Harm can be catastrophic. Therefore, your job as an architect is to use system design to drive the Probability of Harmful Output as close to zero as possible. A disclaimer in your Terms of Service isn't a system. A system prompt isn't a system. A multi-layered architecture with classification, routing, and grounding is a system.
Implementing these guardrails adds complexity and cost. A classification model isn't free. Building a RAG pipeline takes time. But the cost of not doing it is a silent, ticking time bomb in your production environment. You're one user query away from being a headline.
What This Means For Your Team
This isn't just an OpenAI problem. This is an industry-wide problem for anyone shipping products with generative AI.
- Stop treating LLM APIs like simple function calls. They are non-deterministic systems that require their own containment architecture.
- Identify the high-stakes domains for your product. If you touch anything related to a user's health, finances, or safety, you need more than just a prompt-based guardrail. You need hard boundaries in your code.
- Invest in a multi-layered defense. Use input classification and routing to block dangerous queries. Use RAG to ground the model in trusted facts. Validate the LLM's output before showing it to the user.
- Own the responsibility. The person responsible for the safety of your AI feature is the person shipping the code. It’s not the user, and it's not the model provider. It's you.
Building software has always been about managing complexity and mitigating risk. LLMs add a powerful new capability, but they also introduce a new and dangerous class of failure modes. This lawsuit is a tragic but necessary wake-up call. It's time to move beyond wishful prompting and start applying rigorous engineering discipline to how we build with these models.
Building something in this space? AgileStack helps teams ship enterprise-grade software without the consulting-firm overhead. Book a 30-minute call and tell us what you're working on.