The Inspiration

The Why

There was a discussion with my colleagues on how there would still be inevitable manual steps such as subsequent deployment for AWS API Gateway. Initial research points towards using CloudFormation(CF) lambda-backed custom resources in order to automate this step. I did not make much progress due to my unfamiliarity with setting up API Gateway

The Exploration

After a few months, I have a better understanding of manoeuvring the AWS API Gateway setup. I decided to explore the following areas:

• If I can automate the API deployments solely through CloudFormation without using Lambda-backed custom resources.

• If not, I will create the Lambda-backed custom resource and how to integrate it into my CF templates

The Plan

• For CloudFormation to update an existing stack, changes must be detected in the CF templates used

• Based on the properties of Deployment resource, I will be using the BodyS3Location to assist in triggering changeset.

  • A version-enabled S3 bucket will be used to store the OpenSwagger JSON file

  • The version ID of the file is used as versioning to trigger the change set

• I can parameterise the value of Version such that a different value can be provided in subsequent updates of the CF stack

      BodyS3Location:
        Bucket: !Sub "${APIJsonS3Bucket}"
        Key: !Sub "${RestApiFilename}"
        Version: !Sub "${RestApiFileVersion}"

---

Preparing the CloudFormation Templates

I will be using a single CF template instead of breaking them into a main-nested stack format.

Parameters

The parameters used are to define

• BodyS3Location and EndpointConfiguration property in AWS::ApiGateway::RestApi

• Description in AWS::ApiGateway::Deployment

Parameters:
  APIJsonS3Bucket:
    Type: String
    Description: "S3 Bucket name containing the REST API JSON file"

  RestApiFilename:
    Type: String
    Description: "Name of REST API JSON file"
    Default: "api.json"

  RestApiFileVersion:
    Type: String
    Description: "Version ID of REST API JSON file in S3 Bucket"

  EndpointConfigurationType:
    Type: String
    Description: "A list of endpoint types of an API or custome domain name"
    Default: "PRIVATE"
    AllowedValues:
      - "EDGE"
      - "REGIONAL"
      - "PRIVATE"

  DeploymentDescription:
    Type: String

Conditions

A single condition is created to determine if the Policy section needs to be defined.

• The condition will be true when the endpoint configuration type is set to "PRIVATE"

• This condition is used under the Policy property for the AWS::ApiGateway::RestApi resource Conditions: CreatePolicy: !Equals [!Ref EndpointConfigurationType, 'PRIVATE']

Resources

RestApi

This resource will create

• API Gateway (which you can see in the AWS Management Console)

• The methods as defined in the OpenSwagger JSON file under BodyS3Location

• The Policy will be defined if the EndpointConfigurationType the parameter is set to "PRIVATE"

  MyApiGateway:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: "My ApiGateway"
      Description: "Test API Gateway, deploy with CF"
      BodyS3Location:
        Bucket: !Sub "${APIJsonS3Bucket}"
        Key: !Sub "${RestApiFilename}"
        Version: !Sub "${RestApiFileVersion}"
      EndpointConfiguration:
        Types:
          - !Sub "${EndpointConfigurationType}"
      Policy:
        !If 
        - CreatePolicy
        -
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Principal: '*'
              Action: 'execute-api:Invoke'
              Resource: '*'
        - !Ref "AWS::NoValue"

Deployment

I will simulate deployment to the development environment, hence the stage being named "dev".

  DevDeployment:
    Type: AWS::ApiGateway::Deployment
    DependsOn: 
      - MyApiGateway
      - DevDocumentVersion
    Properties:
      RestApiId: !Ref MyApiGateway
      Description: !Ref DeploymentDescription
      StageName: "dev"
      StageDescription:
        Description: "development stage"
        Variables:
          Stack: Dev

---

Execution

Setup

I ran the CF template for the initial setup and successfully created the resources (API Gateway and Deployment)

• The methods in the OpenSwagger JSON file are created under Resources. The “dev” stage is created through CloudFormation.

Second deployment

Steps taken

• To initiate a subsequent deployment, I uploaded a newer version of the OpenSwagger JSON file containing more methods.

• I updated the RestApiFileVersion parameter in the CF stack.

Outcome

The newer version of the API has been pushed to the Resources and the new APIs have been deployed to the stage. Alas, the deployment history has been completely overwritten.

After referring to AWS documentation, it seems this outcome is due to CloudFormation API Gateway Deployment update behaviour, i.e. replacement.

• While replacement is false, since the changes are made within the current resource's definition, the changes have been applied to the existing resources instead.

If that's the case, let's try creating a new Deployment resource to see the effect

---

Round 2

Changes

I will be modifying the CF template with the following changes.

Parameters

• I added a new parameter called RestApiFileVersionNext where I can include the new version ID.

Parameters:
  # Other existing parameters are excluded for brevity
  # Used to indicate the new version ID
  RestApiFileVersionNext:
    Type: String
    Description: "Next Version ID of REST API JSON file in S3 Bucket"

• I removed DeploymentDescription parameter as the changes in this parameter would affect both Deployment resources. Instead, I put similar message with the version ID to differentiate the deployments

Resources

• I separate the stage from the Deployment resource, as CloudFormation will throw errors when attempting to recreate a stage with the same name.

  DevDeploymentNoStage:
    Condition: CreatePrivateRestApiGateway
    Type: AWS::ApiGateway::Deployment
    Properties:
      RestApiId: !Ref MyApiGateway
      Description: !Sub "dev stage deployment for API file, version id ${RestApiFileVersion}"

  # This will be commented out in the provisioning of the initial CF stack
  DevDeploymentNoStageNext:
    Condition: CreatePrivateRestApiGateway
    Type: AWS::ApiGateway::Deployment
    Properties:
      RestApiId: !Ref MyApiGateway
      Description: !Sub "dev stage deployment for API file, version id ${RestApiFileVersionNext}"

  DevStage:
    Type: AWS::ApiGateway::Stage
    Properties:
      StageName: dev
      RestApiId: !Ref MyPrivateRestApiGateway
      DeploymentId: !Ref DevDeploymentNoStage
      Variables:
        Name: DevStage

Outputs

• I also added a new Deployment resource, called DevDeploymentNoStageNext.

• I included the values of the 2 deployment resources as Outputs to verify the physical ID

Outputs:
  CheckDeploymentNoStage:
    Value: !Ref DevDeploymentNoStage
 # This will be commented out in the provisioning of the initial CF stack 
  CheckDeploymentNoStageNext:
    Value: !Ref DevDeploymentNoStageNext

Deployment Round 2

Steps

• After the initial provisioning of the CF stack, I uncommented the DevDeploymentNoStageNext resource

• When updating the CF stack, I replace the existing template

• The DeploymentId used by the Stage resource will have to be changed to use DevDeploymentNoStageNext which will hold the ID of the latest deployment

  DevStage:
    Type: AWS::ApiGateway::Stage
    Properties:
      StageName: dev
      RestApiId: !Ref MyPrivateRestApiGateway
      DeploymentId: !Ref DevDeploymentNoStageNext
      Variables:
        Name: DevStage

Outcome

In the change set, CloudFormation detected the new Deployment resource.

After the update, the outputs reveal the 2 unique physical IDs of the Deployment resource. And yes, the deployment history is intact!

Food for Thought

So the answer is yes, it is possible. However, this approach is unfeasible.

• There is a need to constantly add a new definition of Deployment resource into the CF template.

• An alternative would be to separate the Deployment resource into a separate stack with the values of RestApi, made available through Exports, to avoid issues when updating the stack.

• There is still a need to manually update the template, which defeats the purpose of automating the deployment process.

---

What's Next

That's all for now, folks!

As much as I would like to only stick to the native functionalities of CloudFormation, there is a limit as to what CloudFormation can provide out-of-the-box, i.e. provisioning the infrastructure along with the initial deployment.

Up next, I will look into the steps to provision API Gateway and deploy API via awscli to dissect the underlying behaviours of the components of API Gateway.

Stay tuned, cheers🥂