Infrastructure as Code: Safe Deployments with Change Review in AWS

In today’s post, I explore a crucial aspect of managing AWS infrastructure with CloudFormation: the safe deployment of changes. I take a closer look at a feature known as change sets, which empowers you to thoroughly review alterations before applying them. Furthermore, I expand on automation of change reviews and contrast it with a different approach using Terraform.

For a hands-on experience, feel free to check out the source code from GitHub and follow along. Throughout this brief tutorial, we’ll be deploying changes to the cloud, allowing you to gain practical experience. To establish a safe sandbox environment, I recommend creating a free AWS account if you don’t already have one. Let’s get started.

Table of Contents

• CloudFormation Templates
• Roll Up Your Sleeves
• Create The Stack
• The Risk Of Making Changes
• Automate The Change Review
  • Approach 1: No Structural Changes
  • Approach 2: The Template Has Changed
• Automate The Change View
• Comparison with Terraform
• Summary

CloudFormation Templates

AWS CloudFormation is an Amazon service that enables you to define cloud resource in a declarative manner using templates. This approach helps you steer clear of manual modifications that are hard to trace and replicate. The ultimate goal is to generate 100% of your AWS infrastructure from code.

In this project, we create a database server and review the impact of making changes to it. Since this tutorial focuses primarily change management, I will not explain how CloudFormation templates work. However, this tutorial is a great start if you want to gain understanding of core concepts.

Here are the templates I’ll be using throughout this guide:

• VPC template that creates the VPC network as a prerequisite
• RDS template that instantiates the database server

Roll Up Your Sleeves

If you’re eager to quickly get to the point, feel free to skip this section. However, if you’d like to follow this tutorial, here’s what you need to do:

1. Install the AWS command line utility.
2. Create an AWS account. If you don’t have an AWS account yet, you can create one for free.
3. Access your security credentials:
   • Sign in to the AWS console
   • Click on your avatar
   • Navigate to Security Credentials
4. Generate an access key:
   • In the Security Credentials section, scroll down to Access keys
   • Click Create access key
   • Ignore any warnings and proceed to create a new key
   • Once done, download the key as CSV

As the last step, create a default profile. Open this file, or create it if there is none: ~/.aws/credentials. Now, add a new default section as follows, using your fresh pair of an access key and a secret.

[default]
aws_access_key_id = AKIA****************
aws_secret_access_key = ZkP*****************************************

Congratulations, you are ready to play along!

Create The Stack

A stack in AWS represents a collection of interconnected resources. In this tutorial, we use two templates to create two separate stacks. One for the VPC network and another for the DB server. While it’s possible to consolidate them into a single stack, this setup serves the purpose of our demo project.

Here is the gist of creating a new stack in AWS. There are scripts in the GitHub repository you can review and execute.

# Instruct AWS CLI on which profile to use
export AWS_PROFILE=default

# Set the desired region
aws configure set region us-east-1 --profile $AWS_PROFILE

# Set the path to your template.
CF_TEMPLATE_URL="file://vpc-template.yaml"

# Validate the template. This is just a formal validation!
aws cloudformation validate-template --template-body $CF_TEMPLATE_URL

# Create the stack!
# Create VPC stack
aws cloudformation create-stack \
  --stack-name DEV-VPC \
  --template-body $CF_TEMPLATE_URL

Once you’ll have run the scripts you should end up with a new database server.

The Risk Of Making Changes

CloudFormation makes an explicit distinction between creating a stack from scratch and updating an existing one through separate commands: `create-stack` and `update-stack`.

When updating a stack in CloudFormation, changes are applied immediately. This operation can potentially be destructive.

When updating a stack, AWS CloudFormation might interrupt resources or replace updated resources, depending on which properties you update.

AWS CloudFormation stack updates

For example, altering credentials for our DB server results in the server being replaced! In contrast, increasing storage space for our database leads to only an update of the existing server. Change effects fall into three categories:

• Update with no interruption
• Update with some interruption
• Replacement

In practice, discerning which changes are destructive and which are not can be challenging. That’s why it’s so important to review any alterations before they are applied.

Automate The Change Review

CloudFormation offers a feature called Change Sets, which provides a way to review proposed changes before they are applied.

Change sets allow you to preview how proposed changes to a stack might impact your running resources, for example, whether your changes will delete or replace any critical resources

AWS: Updating stacks using change sets

While Amazon’s tutorial delves into creating change sets in the console, this manual process can be error-prone and time consuming. Especially when you’re under tight deadlines and your AWS stack undergoes frequent changes. Automating the change review becomes a crucial success factor.

There are at least two ways of how to automatically create a change set and generate a brief overview of all the changes and their impact on the existing infrastructure. Both of these require that you know your stack ID, which you can easily find in AWS console. Or even better, you can obtain it via a script.

Approach 1: No Structural Changes

The first approach is handy when there are no structural changes to the template itself. It capitalizes on the fact that many production systems are built off heavily parameterized templates. For instance, if we need to update credentials for our DB server, we can simply pass the updated parameters and reuse the existing template.

# Option 1: Use the previous template and only change the parameters
aws cloudformation create-change-set \
  --stack-name $stack_id \
  --change-set-name SampleChangeSet \
  --use-previous-template \
  --parameters \
    ParameterKey="Password",UsePreviousValue=true \
    ParameterKey="Username",ParameterValue="root"

The drawback of this approach is the necessity to specify all input parameters. However, we can avoid redeclaring each value using the `UserPreviousValue` flag.

Yet, the outcome is somewhat disappointing. Instead of visualizing the changes we only receive a reference to the new change set:

{
    "Id": "arn:aws:cloudformation:us-east-1:096517317782:changeSet/SampleChangeSet/48fd0ae1-7408-4bde-90ff-e7b0c5f9758b",
    "StackId": "arn:aws:cloudformation:us-east-1:096517317782:stack/DEV-RDS/76a2ac60-4a22-11ee-89ba-12ed46d5a929"
}

I’ll address this issue shortly and show you how to get a comprehensive overview of all changes. For now, you can review the change set directly by navigating to the AWS Console and accessing the CloudFormation stack.

As you can see, changing DB credentials indeed leads to a complete replacement of the existing DB server.

Approach 2: The Template Has Changed

Whenever changes are made to the template, it is essential to have a comprehensive overview of their impact on the infrastructure. Let’s proceed to adjust the amount of storage space available for our database. In rds-template.yaml:

  ExampleDB:
    Type: AWS::RDS::DBInstance
    Properties:
      VPCSecurityGroups:
        - !Ref ExampleDBSecurityGroup
      DBSubnetGroupName: !Ref ExampleDBSubnetGroup
      AllocatedStorage: 20 # << THIS HAS CHANGED: The previous value was 15
    ...

Now we need to slightly modify the script creating the change set:

# Option 2: Use a new or an updated template
aws cloudformation create-change-set \
  --stack-name $stack_id \
  --change-set-name SampleChangeSet \
  --no-use-previous-template \
  --template-body file://rds-template.yaml
  --parameters \
    ParameterKey="Password",UsePreviousValue=true \
    ParameterKey="Username",UsePreviousValue=true

Please take note of the usage of `no-use-previous-template` and `template-body`. Unfortunately, you’ll still need to list all input parameters.

After running the script, the new change set becomes available in the AWS console. In this case, we’ve observed that changing the storage amount doesn’t require the DB server to be recreated from scratch.

Automate The Change View

Up to now, we’ve needed to access the AWS console to review the changes. Can we improve this? Definitely. Let’s enhance our script by adding a command that immediately displays the changes.

aws cloudformation describe-change-set --change-set-name SampleChangeSet --stack-name $stack_id

Note: This script should run after the change set creation is complete. You can either add a delay to your overall script or implement polling for change set status. I’ll leave this minor improvement as an exercise.

The outcome contains an overview of changes, along with other information which we can disregard.

{
    "Changes": [
        {
            "Type": "Resource",
            "ResourceChange": {
                "Action": "Modify",
                "LogicalResourceId": "ExampleDB",
                "PhysicalResourceId": "example-postgres-instance-2",
                "ResourceType": "AWS::RDS::DBInstance",
                "Replacement": "False",
                "Scope": [
                    "Properties"
                ],
                "Details": [
                    {
                        "Target": {
                            "Attribute": "Properties",
                            "Name": "AllocatedStorage",
                            "RequiresRecreation": "Never"
                        },
                        "Evaluation": "Static",
                        "ChangeSource": "DirectModification"
                    }
                ]
            }
        }
    ],
...

Please take note of the Action and Replacement fields. They indicate that the underlying resource, the DB server, will only be modified instead of replaced.

Since the outcome is in JSON format, it can be easily interpreted by a program. In the case of a destructive replacement, your script can send an email alert or prevent the changes from being applied.

Comparison with Terraform

I wanted to conclude this tutorial with a comparison with Terraform. In my opinion, Terraform offers a more developer-friendly and versatile solution.

I’ve prepared an HCL script, main.tf, that achieves the same cluster setup using Terraform. To prove my point, I don’t even need to create the cluster. Unlike CloudFormation, Terraform doesn’t differentiate between an existing stack and a new one. Instead, it focuses on reviewing changes in advance, separating infrastructure planning from execution.

To review the changes using Terraform simply run the command:

terraform init && terraform plan

The outcome is a comprehensive overview of all changes that will occur in the cluster, concluded with a one-line summary of the total resource additions, alterations and deletions.

Terraform will perform the following actions:

  # aws_db_instance.example will be created
  + resource "aws_db_instance" "example" {
      + address                               = (known after apply)
      + allocated_storage                     = 15
      + allow_major_version_upgrade           = true
      + apply_immediately                     = true
      + arn                                   = (known after apply)
      // etc.
    }

  # aws_db_subnet_group.example will be updated in-place
  ~ resource "aws_db_subnet_group" "example" {
        id                      = "example"
        name                    = "example"
      ~ subnet_ids              = [
          - "subnet-0392df15c4ba82d9d",
          - "subnet-05107b57e877441a0",
          - "subnet-06836fe7b575d5ab4",
        ] -> (known after apply)
        tags                    = {
            "Name" = "example"
        }
        # (5 unchanged attributes hidden)
    }

  # aws_security_group.rds will be created
  + resource "aws_security_group" "rds" {
      + arn                    = (known after apply)
      + description            = "Used in the terraform"
      + egress                 = [
          + {
              + cidr_blocks      = [
                  + "0.0.0.0/0",
                ]
              + description      = ""
              + from_port        = 5432
              // etc.
            },
  // .. and so on and so forth. Each component is described in depth.
}
Plan: 15 to add, 1 to change, 0 to destroy.

Summary

In this post, we automated the review of changes in an AWS cluster using CloudFormation, exploring various options and discussing their pros and cons. Alternatively, Terraform offers a more detailed change overview and is a better choice for change reviews. Whether to manage your cluster with AWS-native CloudFormation or the versatile Terraform is a topic for another post. Each has its unique advantages and drawbacks.