At re:Invent 2023, AWS announced a new feature for Step Functions that allows you to call third-party HTTPS API endpoints directly from your workflow without the need to write a Lambda function. It's a simple way to allow you to securely call external providers such as Stripe, Github, etc.

AWS Step Functions is a service from AWS that allows developers to easily create orchestrated processes (state machines) without having to manage servers. It integrates with over 200 services. With Step Functions, you only pay for the number of state transitions that your state machines execute.

In this article, I will explain this new feature, and illustrate it with a practical example using the CDK (Cloud Development Kit).

How does it work?

The HTTP endpoint Task state allows you to send an HTTPS request to the endpoint of your choice. It can be a GET, POST, PUT, DELETE, PATCH, OPTIONS, or HEAD, and you can also pass a request body.

You will also need to specify a connection arn for authentication. Step Functions HTTP endpoints use EventBridge connections, the same as for EventBridge API destinations. This keeps your credentials secure, preventing them from being hard-coded in the ASL (Amazon State Language) definition.

A Practical Example

Let's take a practical example for this new Task state. Imagine that we are selling licenses for an app (like GraphBolt). We accept payments on our website. Once a payment has been confirmed, we want to generate a license and send it to the user via email.

We are using the following services:

Paddle

Paddle is a merchant of record which provides a payment gateway. They also support sending notifications to your backend via webhooks when a purchase is confirmed. They also have an API that allows us to fetch information about payments, customers, etc.

Keygen

Keygen.sh is an open-source licensing API. It provides everything you need to generate, manage, and validate software licenses.

Our goal is to create a back-end system with an API that receives events from Paddle, validates them, and then starts a Step Functions state machine that processes the event to generate and send the license key to the user.

Here is the overview of what it looks like.

Here is what we want our state machine to accomplish:

1. Receive a transaction.complete Paddle event as input.

2. Generate a new License in Keygen through the API.

3. Fetch the customer information from Paddle (name, email, etc) using the customer_id included in the event.

4. Send the license key to the user via SES.

At the time of writing this article, the CDK does not (yet) have a dedicated Construct for HTTP endpoint Task (watch this Guthub issue). However, we can use the CustomTask construct and define it using plain old ASL.

This is how I defined the CreateLicense task.

const keygenConnection = new Connection(this, 'KeygenConnection', {
  authorization: Authorization.apiKey(
    'Authorization',
    SecretValue.secretsManager('KeygenSecret'),
  ),
});

const keygenEndpoint =
  'https://api.keygen.sh/v1/accounts/2d4fdf58-9507-4e0b-a7e2-5520e1f1cbdb';

const createLicense = new CustomState(this, 'CreateLicense', {
  stateJson: {
    Type: 'Task',
    Resource: 'arn:aws:states:::http:invoke',
    Parameters: {
      ApiEndpoint: `${keygenEndpoint}/licenses`,
      Method: 'POST',
      Authentication: {
        ConnectionArn: keygenConnection.connectionArn,
      },
      RequestBody: {
        data: {
          type: 'licenses',
          attributes: {
            metadata: {
              'transactionId.$': '$.data.id',
              'customerId.$': '$.data.customer_id',
            },
          },
          relationships: {
            policy: {
              data: {
                type: 'policies',
                id: '8c2294b0-dbbe-4028-b561-6aa246d60951',
              },
            },
          },
        },
      },
    },
    ResultSelector: {
      'body.$': 'States.StringToJson($.ResponseBody)',
    },
    OutputPath: '$.body',
  },
});

First, we create a Connection for our HTTP task. This is an EventBridge Connection. As I explained earlier, the role of the connection is to store the credentials securely and not leak them in the Step Functions definition. However, we also don't want to hard-code them in the CDK definition. To avoid that, I manually created a value in Secret Manager named KeygenSecret which contains the API key, and I referenced it in the connection.

Then, I create a Task with the Resource type of arn:aws:states:::http:invoke, attach the connection to it, and define all the other attributes (method, body, etc).

This defines our HTTP task state, but to be able to execute it, Step Functions also needs the necessary permissions.

We need three things:

• Permission to execute HTTP requests

• Permission to use the EventBridge connection

• Permission to fetch the connection's secret

For that, I manually add the following IAM policies to the state machine's role.

sm.role.attachInlinePolicy(
  new Policy(this, 'HttpInvoke', {
    statements: [
      new PolicyStatement({
        actions: ['states:InvokeHTTPEndpoint'],
        resources: [sm.stateMachineArn],
        conditions: {
          StringEquals: {
            'states:HTTPMethod': 'POST',
          },
          StringLike: {
            'states:HTTPEndpoint': `${keyGenEndpoint}/*`,
          },
        },
      }),
      new PolicyStatement({
        actions: ['events:RetrieveConnectionCredentials'],
        resources: [
          keygenConnection.connectionArn,
        ],
      }),
      new PolicyStatement({
        actions: [
          'secretsmanager:GetSecretValue',
          'secretsmanager:DescribeSecret',
        ],
        resources: [
          'arn:aws:secretsmanager:*:*:secret:events!connection/*',
        ],
      }),
    ],
  }),
);

Finally, I did the same thing for the Paddle HTTP task. I also added the SES sendEmail task and put everything together.

Conclusion

The support for direct calls to HTTP endpoints opens a lot of possibilities to integrate with third parties. Before, we would require a Lambda function to achieve the same result. This is one more step forward towards zero-code Step Functions!