Build a robust logging system with Temporal and Parseable

Workflow logging - tracking logs with Temporal and Parseable

Temporal is a leading workflow orchestration platform. Temporal offers a robust, durable execution platform that offers guarantees on workflow execution, state management, and error handling.

One of the key aspect for a production grade application is the ability to reliably log and monitor the workflow execution. The log and event data can be used for debugging, auditing, custom behavior analysis, and more. Temporal applications are no different. Temporal provides logging capabilities, but it can be challenging to manage and analyze logs at scale.

In this post, we'll see how to extend the default Temporal logging to ship logs to a Parseable instance. By integrating Parseable with Temporal you can:

• Ingest workflows logs to create a centralized data repository.
• Co-relate events, errors, and activities across your workflows.
• Analyze and query logs in Parseable for debugging and monitoring.
• Setup reliable audit trails for your workflows using Parseable.
• Setup alerts and notifications in Parseable for critical events in your workflows.

and much more.

Note: We've also created a video tutorial to help you get started with Parseable and Temporal. If you prefer the video instead:

Set up your environment

For demonstration purpose in this blog, we'll use a local setup of Temporal and Parseable. You can use the same steps to integrate Parseable with your Temporal application running in a cluster or cloud environment.

we've use a Linux / Ubuntu 22.04 machine for this setup. You can use any other OS as well. Please ensure to use the relevant binaries and commands for your OS.

Install Temporal

To set up Temporal, follow the local environment set up guide. Assuming you've followed the steps, you should see the Temporal server is running on localhost:7233. You can access the Web UI at http://localhost:8233.

Install Parseable

Let's get Parseable up and running:

docker run -p 8000:8000 \
-v /tmp/parseable/data:/parseable/data \
-v /tmp/parseable/staging:/parseable/staging \
-e P_FS_DIR=/parseable/data \
-e P_STAGING_DIR=/parseable/staging \
containers.parseable.com/parseable/parseable:latest \
parseable local-store

This command pulls the latest Parseable image and runs it. It exposes port 8000 and creates a local volume for data persistence. This setup ensures that Parseable will store its data on your local machine and be accessible through port 8000 on your web browser.

Once Parseable is running successfully, create a log stream called temporaltest in the Parseable Console.

Application setup

We've developed a sample Temporal application that demonstrates how to integrate Parseable for logging. The application consists of a worker process that executes the workflow and activity functions. The application simulates food ordering app flow - placing an oder, making payment, preparing it, and finally pickup and delivering it to the customer. Each of these activities when triggered, is logged to Parseable.

We've used the Typescript SDK to develop this sample application. You can of course choose other supported languages.

First step is to download the application code. Open your terminal and type the command:

git clone https://github.com/parseablehq/blog-samples.git
cd blog-samples/temporal-food-delivery

Next, install the required node packages using the command.

npm install
npm run schedule.start

Now, make sure you have Temporal CLI installed and the server running in a separate window. Refer the section install Temporal for more details.

temporal server start-dev

Finally you can start the application by running the command:

npm run schedule.start
npm i && npm start

By default this application is configured to send logs to the Parseable demo server. Once you have the application running, you can see the logs being ingested in the Parseable demo server (select the stream temporalactivity).

Understanding this integration

The application workflow is pretty standard and built as per the Temporal guidelines. So, we'll not get into too much details there. But let's understand how the logging is integrated.

Winston is a versatile logging library for Node.js, offering various transports for log data, including files, databases, and third-party services like Parseable. We've used the parseable-winston package to send logs from the application to Parseable.

The parseable-winston package creates a transport object that can be used to send logs from Winston to the parseable instance.

To change the Parseable target instance, update the worker.ts file with correct Parseable instance details.

const parseable = new ParseableTransport({
 url: 'http://localhost:8000/api/v1/logstream',
 username: 'admin',
 password: 'admin',
 logstream: 'temporaltest',
 http2: false
})

Refer the location in blog sample repo here.

Conclusion

Congratulations! You've just leveled up your Temporal workflow observability game. By integrating Parseable, you've created a powerful logging system that allows for easy querying, analysis, and alerting. Remember, this is just the tip of the iceberg. As you become more familiar with Parseable, you can create more application based ingestion and fine-tune your alerts to match your specific needs.

Why should you use Parseable with Temporal?

Temporal is an extensible platform to build applications exactly the way you want, while ensuring reliable execution and state management. As your applications grow in complexity, you need a robust logging system to track and monitor their performance.

Given such a wide range of use cases and applications - traditional, rigid logging and monitoring systems won't cut it.

Parseable offers a versatile system to capture and centralize any kind of event and log data, for Temporal applications. With Parseable you don't need to both about log schema changes, data privacy or scaling issues. You can focus on building your applications and let Parseable handle the logging and monitoring.

The Parseable console allows intuitive query, analyze, and monitor capabilities. With Parseable, you can set up alerts and notifications for critical events, ensuring you are always on top of your application's performance.

To see Parseable in action, watch this YouTube video. Get started with Parseable in just a single command.

For technical questions or to share your implementations, join our developer community on Slack.