ECS health checks explained: container, load balancer and grace period
How container health checks, load balancer health checks and the health check grace period work together on AWS ECS, and why tasks get killed while they are still starting.
A task that starts, runs for a minute and is then replaced, over and over, is one of the most common ECS problems. The cause is almost always a health check that fires before the application is ready. ECS has up to three layers involved, and they don’t know about each other.
Layer 1: the container health check
A container health check is a command in the task definition that ECS runs inside the container:
"healthCheck": {
"command": ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"],
"interval": 30,
"timeout": 5,
"retries": 3,
"startPeriod": 60
} interval,timeoutandretriesdefault to 30 seconds, 5 seconds and 3.startPeriod(up to 300 seconds) is time after start during which failures don’t count. It defaults to 0.- The command runs inside your image, so
curlorwgethas to exist there. Minimal images often don’t ship either, which makes every check fail.
When an essential container turns UNHEALTHY, ECS stops the task and the service replaces it. The health of non-essential containers is reported but doesn’t stop the task.
Layer 2: the load balancer health check
If a service is behind an Application or Network Load Balancer, the target group runs its own checks against each task’s IP and port. Application Load Balancer defaults are:
| Setting | Default |
|---|---|
| Interval | 30 seconds |
| Timeout | 5 seconds |
| Healthy threshold | 5 checks |
| Unhealthy threshold | 2 checks |
| Success codes | 200 |
When a target is unhealthy, the load balancer stops sending it traffic and ECS replaces the task. Note the healthy threshold: with the defaults a new task needs 5 passing checks, which is two and a half minutes, before it gets traffic. That makes deployments slow.
Layer 3: the grace period
healthCheckGracePeriodSeconds is a service setting. For that many seconds after a task starts, the ECS scheduler ignores failing load balancer, VPC Lattice and container health checks. It defaults to 0, which means checks count from the first second.
aws ecs update-service --cluster production --service api \
--health-check-grace-period-seconds 120 Set it to how long your application really needs to start, plus the time the load balancer needs to see enough passing checks. Checks that fail during startup mark the target unhealthy, and it only turns healthy again after the healthy threshold of passing checks. A JVM service that warms caches for 90 seconds, behind a target group with the default settings, needs about 90 + 4 × 30 = 210 seconds.
Try it with your own numbers:
A new task behind a load balancer
Stopped after 60 seconds. The target was unhealthy and there is no grace period, so ECS stops the task and starts a replacement, which goes through the same thing.
With these settings the app needs about 210 seconds to turn healthy: its startup, rounded up to the next check, plus 4 more passing checks. A grace period shorter than that gets it killed. Simplified: checks start at launch and container health checks are left out.
Why tasks get killed while starting
Typical causes, in order of how often they show up:
- No grace period, and the app needs longer than
unhealthy threshold × intervalto start. The fix is a grace period or astartPeriod. - The health path returns a redirect or 401. A
/healthbehind auth or a/that redirects to/loginreturns 301/302 or 401, which the target group counts as a failure. Use a dedicated, unauthenticated endpoint, or widen the success codes. - Wrong port. The target group checks the traffic port by default. If health is served on another port, set it explicitly.
- Security groups. The task’s security group must allow traffic from the load balancer’s security group on the health check port.
- A health check that checks too much. If
/healthqueries the database, a slow database restarts every task at once. Keep liveness checks shallow.
The task’s stopped reason tells you which layer killed it: Task failed ELB health checks points at the target group, Task failed container health checks at the task definition.
Tuning for faster deployments
For HTTP services that start quickly:
- lower the target group’s healthy threshold to 2 and the interval to 10 seconds, so new tasks get traffic within about 20 seconds,
- lower the deregistration delay from its 300-second default to something close to your longest request, so old tasks drain faster,
- keep a grace period that covers real startup time.
Summary
| Check | Defined in | Who acts on failure |
|---|---|---|
| Container health check | Task definition | ECS stops the task |
| Load balancer health check | Target group | Load balancer stops traffic, ECS replaces the task |
| Grace period | Service | Suppresses both while a task starts |
Most restart loops disappear once the grace period covers how long the app really takes to start and turn healthy.