All posts

ECS health checks explained: container, load balancer and grace period

How container health checks, load balancer health checks and the health check grace period work together on AWS ECS, and why tasks get killed while they are still starting.

A task that starts, runs for a minute and is then replaced, over and over, is one of the most common ECS problems. The cause is almost always a health check that fires before the application is ready. ECS has up to three layers involved, and they don’t know about each other.

Layer 1: the container health check

A container health check is a command in the task definition that ECS runs inside the container:

"healthCheck": {
  "command": ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"],
  "interval": 30,
  "timeout": 5,
  "retries": 3,
  "startPeriod": 60
}
  • interval, timeout and retries default to 30 seconds, 5 seconds and 3.
  • startPeriod (up to 300 seconds) is time after start during which failures don’t count. It defaults to 0.
  • The command runs inside your image, so curl or wget has to exist there. Minimal images often don’t ship either, which makes every check fail.

When an essential container turns UNHEALTHY, ECS stops the task and the service replaces it. The health of non-essential containers is reported but doesn’t stop the task.

Layer 2: the load balancer health check

If a service is behind an Application or Network Load Balancer, the target group runs its own checks against each task’s IP and port. Application Load Balancer defaults are:

SettingDefault
Interval30 seconds
Timeout5 seconds
Healthy threshold5 checks
Unhealthy threshold2 checks
Success codes200

When a target is unhealthy, the load balancer stops sending it traffic and ECS replaces the task. Note the healthy threshold: with the defaults a new task needs 5 passing checks, which is two and a half minutes, before it gets traffic. That makes deployments slow.

Layer 3: the grace period

healthCheckGracePeriodSeconds is a service setting. For that many seconds after a task starts, the ECS scheduler ignores failing load balancer, VPC Lattice and container health checks. It defaults to 0, which means checks count from the first second.

aws ecs update-service --cluster production --service api \
  --health-check-grace-period-seconds 120

Set it to how long your application really needs to start, plus the time the load balancer needs to see enough passing checks. Checks that fail during startup mark the target unhealthy, and it only turns healthy again after the healthy threshold of passing checks. A JVM service that warms caches for 90 seconds, behind a target group with the default settings, needs about 90 + 4 × 30 = 210 seconds.

Try it with your own numbers:

A new task behind a load balancer

Presets
AppstartingreadyGrace periodnoneHealth checksTarget0s15s30s45s60s75s90s105s120s

Stopped after 60 seconds. The target was unhealthy and there is no grace period, so ECS stops the task and starts a replacement, which goes through the same thing.

With these settings the app needs about 210 seconds to turn healthy: its startup, rounded up to the next check, plus 4 more passing checks. A grace period shorter than that gets it killed. Simplified: checks start at launch and container health checks are left out.

Check passed Check failed Target unhealthy

Why tasks get killed while starting

Typical causes, in order of how often they show up:

  1. No grace period, and the app needs longer than unhealthy threshold × interval to start. The fix is a grace period or a startPeriod.
  2. The health path returns a redirect or 401. A /health behind auth or a / that redirects to /login returns 301/302 or 401, which the target group counts as a failure. Use a dedicated, unauthenticated endpoint, or widen the success codes.
  3. Wrong port. The target group checks the traffic port by default. If health is served on another port, set it explicitly.
  4. Security groups. The task’s security group must allow traffic from the load balancer’s security group on the health check port.
  5. A health check that checks too much. If /health queries the database, a slow database restarts every task at once. Keep liveness checks shallow.

The task’s stopped reason tells you which layer killed it: Task failed ELB health checks points at the target group, Task failed container health checks at the task definition.

Tuning for faster deployments

For HTTP services that start quickly:

  • lower the target group’s healthy threshold to 2 and the interval to 10 seconds, so new tasks get traffic within about 20 seconds,
  • lower the deregistration delay from its 300-second default to something close to your longest request, so old tasks drain faster,
  • keep a grace period that covers real startup time.

Summary

CheckDefined inWho acts on failure
Container health checkTask definitionECS stops the task
Load balancer health checkTarget groupLoad balancer stops traffic, ECS replaces the task
Grace periodServiceSuppresses both while a task starts

Most restart loops disappear once the grace period covers how long the app really takes to start and turn healthy.