AWS Well-Architected Labs > Reliability > 300 Labs > Level 300: Testing Resilience of EC2, RDS, and AZ using Chaos Engineering > Test Resiliency Using Availability Zone (AZ) Failure Injection

Test Resiliency Using Availability Zone (AZ) Failure Injection

7.1 AZ failure injection

This failure injection will simulate a critical problem with one of the three AWS Availability Zones (AZs) used by your service. AWS Availability Zones are powerful tools for helping build highly available applications. If an application is partitioned across AZs, companies are better isolated and protected from issues such as lightning strikes, tornadoes, earthquakes and more.

In Chaos Engineering we always start with a hypothesis. For this experiment the hypothesis is:

Hypothesis: If an entire Availability Zone dies, then availability will not be impacted

Go to the RDS Dashboard in the AWS Console at http://console.aws.amazon.com/rds and note which Availability Zone the AWS RDS primary DB instance is in.
- Note: If you previously ran the RDS Failure Injection test, you must wait until the console shows the AZs for the primary and standby instances as swapped, before running this test
- A good way to run the AZ failure injection is first in an AZ other than this - we’ll call this Scenario 1
- Then try it again in the same AZ as the AWS RDS primary DB instance - we’ll call this Scenario 2
- Taking down two out of the three AZs this way is an unlikely use case, however it will show how AWS systems work to maintain service integrity despite extreme circumstances.
- And executing this way illustrates the impact and response under the two different scenarios.
To simulate failure of an AZ, select one of the Availability Zones used by your service (us-east-2a, us-east-2b, or us-east-2c) as <az>
- For scenario 1 select an AZ that is neither primary nor secondary for your RDS DB instance. Given the following RDS console you would choose us-east-2c
- For scenario 2 select the AZ that is primary for your RDS DB instance. Given the following RDS console you would choose us-east-2b
use your VPC ID as <vpc-id>

Select one (and only one) of the scripts/programs below. (choose the language that you setup your environment for).

Language	Command
Bash	`./fail_az.sh <az> <vpc-id>`
Python	`python3 fail_az.py <vpc-id> <az>`
Java	`java -jar app-resiliency-1.0.jar AZ <vpc-id> <az>`
C#	`.\AppResiliency AZ <vpc-id> <az>`
PowerShell	`.\fail_az.ps1 <az> <vpc-id>`

The specific output will vary based on the command used.
- Note whether an RDS failover was initiated. This would be the case if you selected the AZ containing the AWS RDS primary DB instance

7.2 System response to AZ failure

Watch how the service responds. Note how AWS systems help maintain service availability. Test if there is any non-availability, and if so then how long.

7.2.1 System availability

Refresh the service website several times

Scenario 1: If you selected an AZ not containing the AWS RDS primary DB instance then you should see uninterrupted availability
Scenario 2: If you selected the AZ containing the AWS RDS primary DB instance, then an availability loss similar to what you saw with RDS fault injection testing will occur.

Verify your observations by going through the canary run data:

Go to the AWS CloudFormation console at https://console.aws.amazon.com/cloudformation
click on the WebServersforResiliencyTesting stack
click on the Outputs tab
Open the URL for WorkloadAvailability in a new window
You will see that canary runs are failing because the website is not available.

7.2.2 Scenario 1 - Load balancer and web server tiers

This scenario is similar to the EC2 failure injection test because there is only one EC2 server per AZ in our architecture. Look at the same screens as you did before, for that test:

One difference from the EC2 failure test that you will observe is that auto scaling will not replace the EC2 instance in the same AZ as the one that was terminated. Auto scaling attempts to balance the requested three EC2 instances across the remaining two AZs.

7.2.3 Scenario 2 - Load balancer, web server, and data tiers

This scenario is similar to a combination of the RDS failure injection along with EC2 failure injection. In addition to the EC2 related screens look at the Amazon RDS console, navigate to your DB screen and observe the following tabs:

Configuration
Monitoring
Logs & Events

7.2.4 AZ failure injection - conclusion

This similarity between scenario 1 and the EC2 failure test, and between scenario 2 and the RDS failure test is illustrative of how an AZ failure impacts your system. The resources in that AZ will have no or limited availability. With the strong partitioning and isolation between Availability Zones however, resources in the other AZs continue to provide your service with needed functionality. Scenario 1 results in loss of the load balancer and web server capabilities in one AZ, while Scenario 2 adds to that the additional loss of the data tier. By ensuring that every tier of your system is in multiple AZs, you create a partitioned architecture resilient to failure.

Our hypothesis is confirmed: