Every engineering team has a closet full of skeletons. They aren't just bugs, they're architectural decisions that haven't failed yet. A tightly coupled service, a missing queue, a timeout that's just a little too generous. These are the choices made under pressure that sit dormant for months, or even years, waiting for the right conditions to bring everything down.
We all got a public reminder of this when a post-mortem titled "Mea Culpa - Dark Hours" hit the top of Hacker News. The story itself is a familiar one for anyone who has been on-call during a major incident. But the real lesson isn't in the specifics of that particular failure. It's in the pattern. The bug that gets blamed in the post-mortem is never the real cause. It's just the trigger. The real cause is the system design that allowed a small fire to become an inferno. The real cause has been hiding in your architecture all along.
The Anatomy of a "Surprise" Failure
No catastrophic outage is ever a real surprise. It’s the logical conclusion of unaddressed architectural flaws. The pressure to ship features often pushes teams to build brittle connections between services, creating a house of cards that looks stable until a single card is removed.
Consider a common e-commerce setup. You have a ProductService that displays items, an InventoryService that tracks stock, and a PricingService that provides prices. A product page needs data from all three to render. The easy way to build this is with direct, synchronous API calls from the frontend server or BFF (Backend-for-Frontend).
// A simplified example of a brittle controller
async function getProductPage(req, res) {
const productId = req.params.id;
try {
// These are three separate network requests
const productDetails = await ProductService.get(productId);
const inventory = await InventoryService.getStock(productId); // What happens if this times out?
const price = await PricingService.get(price, { currency: 'USD' });
res.render('product-page', { productDetails, inventory, price });
} catch (error) {
// A generic error that hides the root cause
console.error('Failed to load product page for:', productId, error);
res.status(500).send('Something went wrong');
}
}
This looks fine. It probably works 99.9% of the time. But what happens when the InventoryService's database connection pool is exhausted because of a long-running analytics query? The call to InventoryService.getStock() now hangs for 30 seconds before timing out. Every web server thread trying to render a product page is now blocked, waiting. Your connection pool to your own web server fills up. Suddenly, no pages can be served at all. Your entire site is down.
The post-mortem might say, "Cause: InventoryService unresponsive due to database lock. Fix: Optimize analytics query and add alerts." But that’s a lie. The real cause was the architectural decision to make a product page's render path synchronously dependent on three separate services without any fault tolerance. The system was designed to fail catastrophically.
Your Blameless Post-Mortem Is Lying to You
Blameless post-mortems are a fantastic tool for psychological safety. They encourage engineers to be honest about mistakes without fear of retribution. This is non-negotiable for a healthy engineering culture. But they have a dangerous side effect: they often stop short of questioning the system itself.
A blameless culture can easily become a culture that avoids hard questions about past decisions. The
Building something in this space? AgileStack helps teams ship enterprise-grade software without the consulting-firm overhead. Book a 30-minute call and tell us what you're working on.