AWS Autoscaling / ELB Health Checks

Here is a tiny yet very important hint, if you deal with let’s say an ALB and Autoscaling groups in AWS.

By default ASG use EC2 as their health check. That means as long as the instance and its OS is doing fine, the ASG regards the ec2 instances as healthy. The ALB in contrast uses a different health check, which you define. In general you request a document per http(s) and define a expected return code (200) for OK.

So what happens if you have a loadbalancer with unhealthy instances (for instance your site went down for whatever reason and the ec2 instances do not respond a 200 OK anymore)? Your loadbalancer will drain all connections to these instances as they are regarded unhealthy. You normally want your ASG to replace those unhealthy instances but that won’t happen. For ASG they are doing fine. So after your problem is actually gone away and your instances start responding 200 OK again, the loadbalancer will not use them anymore and will be left with a zero instance and your site will stay offline for customers.

Solution: make sure to use ELB Health check for your ASG if used behind an ALB.