Orchestrating Serverless Workflows with Ease using AWS Step Functions

When we talk about running serverless workloads on AWS (disclaimer: serverless doesn't mean there are no servers, it just means you don't have to worry about provisioning and managing them), the service that immediately comes to mind is definitely AWS Lambda. This serverless compute service allows developers to run their code in the cloud, all without managing the underlying infrastructure.

Although Lambda offers developers the ability to run their code in the cloud, it does have some constraints that limit its usability in specific scenarios and use cases. One of these constraints is Lambda's maximum execution time of 15 minutes. Unfortunately, this means developers cannot use Lambda to carry out complex operations that take more than 15 minutes to complete.

However, don't let this limitation dissuade you from using AWS Lambda! Because this is where AWS Step Functions step in (see what I did there? 😉) to the rescue to make the execution of complex operations possible.

The main objective of this article is to bring the good news of AWS Step Functions to you my dear friend. So grab your digging equipment and without further ado, let's start digging into it.

What is AWS Step Functions

Simply put, AWS Step Functions is a state machine service. But what exactly is a state machine? Let's use an analogy to explain. Imagine your office coffee maker. It sits idle in the kitchen, waiting for instructions to make coffee. When someone uses it, they select the type of coffee, quantity, and other options – these are the states the machine goes through to make a cup of coffee. Once it completes the necessary states, the coffee maker returns to its idle state, ready for the next user. AWS Step Functions allow you to create workflows just like the coffee maker, where you can have your system wait for inputs, make decisions, and process information based on the input variables. With this kind of orchestration, we are able to leverage Lambda functions in ways that are not inherently supported by the service itself. For instance, you can run processes in parallel when you have multiple tasks you want to process at one time or in sequence when order is important. In a similar fashion, you can implement retry logic if you want your code to keep executing until it succeeds, or reaches a time out of some sort. This way, we are able to conquer lambda's 15 minutes code execution limit.

How does it work?

Now on to how Step Functions works. It operates by getting your workflow from an Amazon State language file which is a JSON based file that is used to define your state machine and its components. This file defines the order and flow of your serverless tasks in AWS Step Functions. It's like the recipe for your code workflow. Here is an example of what State language looks like:

{
  "StartAt": "SayHello",
  "States": {
    "SayHello": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:SayHelloFunction",
      "Next": "Goodbye"
    },
    "Goodbye": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:GoodbyeFunction",
      "End": true
    }
  }
}

As you can see from the code above, State Language files are written in JSON format, a language familiar to most developers. However, if you're new to JSON, there's no need to worry! AWS Step Functions lets you build your state machine by dragging and dropping components (states) to link them in the AWS Step Functions Workflow Studio. Here's a picture example of what a state machine looks like in the Workflow Studio.

State Machine State Type

There are eight commonly used core state types that you can define in your workflow to achieve a particular result. These state types are: the Pass state, Task State, Choice State, Wait, Success State, Fail, Parallel State, and Map State. We'll take a closer look at each of these in more detail.

Pass State — The Pass state doesn't actually perform a specific action. Instead, it acts as a placeholder state, facilitating transitions between other states without executing any code. While it can be helpful for debugging purposes, such as testing transitions between states, it's not exclusively a debugging state.

Task State — This is where the action happens. As the most common state type, it represents a unit of work, typically executed by an AWS Lambda function or another integrated service.

Choice State — This state allows you to evaluate an input and then choose the next state for the workflow based on the evaluation outcome. Essentially, it's an "if-then" operation that enables further application logic execution based on the chosen path.

Wait State — In this state, you can pause the state machine for a specified duration or until a specific time is reached. This comes in handy if you want to schedule a pause within the workflow. For example, you can use it to send out emails at 10:00 AM every day.

Success State — This state is used to indicate the successful completion of a workflow. It can be part of the choice state or to end the state machine in general.

Fail State — It is termination state similar to the success state but indicates that a workflow failed to complete successfully. Fail states should have an error message and a cause for better workflow understanding and troubleshooting.

Parallel State — This state executes a group of states as concurrently as possible and waits for each of them to complete before moving on. Imagine you have a large dataset stored in S3 that needs to be processed. You can use a Parallel state to concurrently trigger multiple Lambda functions, each processing a portion of the data set. Doing this will significantly speed up the overall processing time.

Map State — The Map state allows you to loop through a list of items and perform tasks on them. In the map state, you can define the number of concurrent items to be worked on at one time.

By making use of a combination of these states, you can build dynamic and highly scalable workflows. To find the list of supported AWS service integrations for Step Functions, check out this AWS documentation.

Conclusion

This article has introduced you to AWS Step Functions and its potential to streamline your application development. Step Functions manages your application's components and logic, allowing you to write less code and focus on building and updating your application faster. It offers a wide range of use cases, including submitting and monitoring AWS Batch jobs, running AWS Fargate tasks, publishing messages to SNS topics or SQS queues, starting Glue job runs, and much more. If your workflow involves tasks like these, AWS Step Functions can be a valuable asset.