Most backend services are designed to fail. We don't say it out loud, but it's true. We build them with the full expectation that they will crash, leak memory, or exhaust a connection pool. So we wrap them in a process manager like PM2 or a Kubernetes ReplicaSet and treat the restart as a core architectural component. It's not resilience. It's just rapid recovery from constant, predictable failure.
This hit me when I read a recent news story. As TechCrunch reported, a startup named Alteon is building autonomous aircraft designed to stay aloft for up to a year by harvesting wind energy. Think about that. A single, uninterrupted process running for twelve months with no human intervention, handling fluctuating energy levels, and navigating a hostile environment. This isn't just an incredible hardware and aeronautics problem. It's a profound challenge to how we think about software endurance. If they can aim for a year of uptime in the sky, why are we content with a service that needs a cron job to restart it every night?
The answer is that we've accepted a pattern I call the Restart Fallacy. And it’s holding our systems back.
The Restart Fallacy
The default for most modern backend development is to build stateless services that can be killed and restarted at a moment's notice. This is a good thing, in theory. It's the foundation of horizontal scaling. But it has a dangerous side effect. It makes us lazy architects.
We stop solving the hard problems of long-term process stability because we know we can just kill the pod and let Kubernetes spin up a new one. It's the software equivalent of turning it off and on again.
Consider a typical Node.js service. It has a subtle memory leak, maybe from a misconfigured event listener or a caching library that never evicts. Over 24 hours, its memory usage creeps from 100MB to 1.5GB. Instead of debugging it with heapdump and fixing the root cause, what do we do? We set a memory limit in the pod spec and let it get OOMKilled. Or worse, we schedule a nightly restart.
// ecosystem.config.js
module.exports = {
apps : [{
name: 'leaky-api',
script: 'app.js',
cron_restart: '0 3 * * *', // Restart every day at 3 AM
max_memory_restart: '1G'
}]
};
This PM2 config is an admission of failure. We're not building a resilient system. We're building a fragile one and automating the cleanup. A drone can't afford this luxury. It can't just cron_restart at 10,000 feet. It has to be built for endurance from the ground up. So should our services.
Designing for Endurance, Not Just Recovery
To build services that can metaphorically stay aloft for a year, we need to borrow principles from systems like Alteon's aircraft. This means shifting our focus from fast recovery to true endurance.
Graceful Degradation is Energy Harvesting
A wind-harvesting drone doesn't operate at 100% capacity all the time. When the wind is low, it conserves energy. It might reduce sensor polling frequency or switch to a lower-power flight mode. It degrades its own performance gracefully to survive.
Our services need to do the same. When a database is slow or a third-party API is timing out, the default response is often to cascade failure, throwing 500 errors until the system restarts. A more enduring design would degrade gracefully. For example, an e-commerce API under extreme load could:
- Stop generating optional, on-the-fly image thumbnails.
- Serve slightly stale data from a cache instead of hitting the primary database for every request.
- Disable expensive recommendation algorithms.
This requires building circuit breakers and feature flags into the core of your application, not as an afterthought. You need to be able to ask your service, "What can you turn off to survive right now?" and have it know the answer.
Proactive Health Checks, Not Reactive Pings
Most services have a GET /health endpoint that returns a 200 OK if the process is running. This is a pulse check, but it tells you nothing about the actual health of the patient. It's the equivalent of the drone reporting "I am not currently a crater in the ground."
An enduring system needs deep, internal instrumentation that acts as a proactive immune system. A proper health check should report on the status of its critical dependencies and internal resources:
{
"status": "DEGRADED",
"timestamp": "2026-09-01T14:30:00Z",
"dependencies": {
"database": {
"status": "UP",
"pool_size": 50,
"pool_in_use": 48, // Warning: nearing exhaustion
"avg_query_ms": 150
},
"redis_cache": {
"status": "UP",
"hit_ratio": 0.92
},
"payment_gateway_api": {
"status": "DOWN", // Critical failure
"latency_p99_ms": 5002,
"last_error": "Connection timeout"
}
}
}
This kind of detailed check allows an orchestration system to make intelligent decisions. Instead of just killing a pod that reports DEGRADED, it could reroute traffic, trigger alerts for the failing dependency, or put the service into a lower-capacity mode. The system starts to heal itself before catastrophic failure occurs.
State Must Be External and Durable
This is the oldest rule in the book, but it's the one we break most often for convenience. Any state kept in the memory of a running process is fragile. A long-duration aircraft can't afford to forget its mission plan if it encounters turbulence that forces a subsystem reboot. Its critical state is stored in durable memory.
In our services, this means aggressively pushing state out of the application process. In-memory caches for performance are fine, but they must be treated as disposable. The canonical source of truth must live elsewhere.
- User Sessions: Don't store them in memory. Use Redis or a database.
- Background Job State: A job's status shouldn't live in a variable. It should be tracked in a durable queue system like RabbitMQ or a database table.
- Configurations: Don't rely on in-memory flags that get reset on restart. Use a proper configuration service or environment variables.
Assume your process will be destroyed at any moment without warning. If that thought terrifies you, you have a state management problem.
Code-Level Implications
This isn't just about high-level architecture. It impacts how we write code every day.
Endurance requires discipline at the function level. One key area is idempotency. In a truly resilient, distributed system, you can't guarantee exactly-once delivery of anything. A request might time out from the client's perspective, but have actually been processed by your server. The client will retry, and now you're processing the same payment twice.
Building idempotent APIs is non-negotiable for endurance. Every POST or PUT endpoint that mutates state should be safe to retry. This usually involves passing a unique key (an Idempotency-Key header is a common pattern) and checking if an operation with that key has already been completed.
Here’s some pseudocode for what that looks like in an Express-style controller:
// Pseudocode for an idempotent endpoint
async function createCharge(req, res) {
const idempotencyKey = req.headers['idempotency-key'];
if (!idempotencyKey) {
return res.status(400).send({ error: 'Idempotency-Key header required.' });
}
// 1. Check if we've processed this key before
const existingOperation = await db.idempotencyLog.find(idempotencyKey);
if (existingOperation) {
// 2. If yes, return the original result without re-processing
return res.status(existingOperation.responseCode).send(existingOperation.responseBody);
}
// 3. If no, perform the operation
const charge = await paymentService.create(req.body);
// 4. Store the result before returning
await db.idempotencyLog.save({
key: idempotencyKey,
responseCode: 201,
responseBody: charge
});
return res.status(201).send(charge);
}
This logic protects you from the chaos of network retries and ensures your system's state remains consistent, even when individual operations are repeated.
What This Means for Your Team
Building for endurance requires a mindset shift away from treating symptoms (the service crashed) and toward fixing the underlying disease (the service was fragile).
Here are the key takeaways:
- Stop treating restarts as a feature. Your process manager is a safety net, not your core architecture. Aim to make restarts rare, unexpected events, not routine operations.
- Design for graceful degradation. Identify non-critical features and build the kill switches to disable them under load. Your service should be able to shed load to survive.
- Invest in proactive, deep monitoring. Your
/healthendpoint is probably lying to you. It needs to report on the health of its dependencies and its own internal resources. - Make all state external and durable. The memory of your process is volatile. Assume it can disappear at any moment and design accordingly.
- Prioritize idempotency everywhere. In distributed systems, retries are a fact of life. Design your APIs to handle them safely from day one.
The next time you deploy a service, don't just ask if it will run for a day. Ask what it would take for it to run for a year. The drone flying overhead is a reminder that it's possible. The answer to that question will force you to confront every assumption you've made and will fundamentally change your architecture for the better.
Building something in this space? AgileStack helps teams ship enterprise-grade software without the consulting-firm overhead. Book a 30-minute call and tell us what you're working on.