How To Collect Temporal.io Logs Using Axiom And Pino

Temporal is a scalable and reliable runtime for durable Workflow Executions. It enables you to develop as if failures don't even exist. I started exploring it over the Christmas holiday and using it for a recently open-sourced project.

I struggled to implement forwarding the logs generated from a Worker and the associated workflows/activities to Axiom. I struggled to do this because I couldn't find the information I needed from the docs (at first glance), and the sample repo only showed how you can use Winston for logs, in a TypeScript project. However, I used pino and my first working attempt didn't log any useful information.

In this post, I'm going to show you how to configure your Temporal TypeScript Worker to use pino for logging, and then send those logs to Axiom. Although I'm using Axiom for this example, the important aspect of this is configuring your Worker to use pino for logging. You can choose to configure any transport layer for pino afterwards.

Set Up The Logger

If you're not familiar with the terminologies used so far, this section covers that and also how to install and instantiate pino with Axiom as the log backend.

pino is a low-overhead, JSON logger for Node.js. Many people choose it because of its speed, and how you can configure the transport and log processing. For this post, you'll see how to configure Axiom as a transport for the logs.

Axiom is an observability backend with OpenTelemetry support. I chose it for my recent project because of its cost, especially for hobby projects or serious projects in its early phase. They provide a JavaScript library that you can use as transport for pino.

Install the npm packages

From here onwards, I'll assume you already have your Temporal TypeScript Worker project set up. Now, you'll install the pino and Axiom packages using the command below.

npm install pino @axiomhq/pino

After those are installed, add a new file logger.ts and paste the code below into it.

import pino from "pino";

const logLevel = process.env.LOG_LEVEL || "info";

export function createLogger(name?: string) {
  const logger = pino(
    { level: logLevel, name },
    pino.transport({
      target: "@axiomhq/pino",
      options: {
        dataset: process.env.AXIOM_DATASET,
        token: process.env.AXIOM_API_TOKEN,
      },
    })
  );

  return logger;
}

The code contains a createLogger() that returns a pino instance configured to use Axiom as transport.

Configure The Worker Runtime

Next, we're going to add a new function that will configure the Temporal Worker runtime to use pino as a logger, and also forward the Rust & Node.js logs to Axiom. You can add this function in the same file where you instantiate your Worker code, or in a standalone file.

Add the function below to your project:

import { createLogger } from "./logger";
import {
  Runtime,
  makeTelemetryFilterString,
  DefaultLogger,
} from "@temporalio/worker";

function ConfigureRuntime() {
  const USE_PINO_LOGGER =
    process.env.AXIOM_DATASET && process.env.AXIOM_API_TOKEN;

  if (USE_PINO_LOGGER) {
    const pino = createLogger();

    // Configure Rust Core runtime to collect logs generated by Node.js Workers and Rust Core.
    Runtime.install({
      // Note: In production, WARN should generally be enough.
      // https://typescript.temporal.io/api/namespaces/worker#loglevel
      logger: new DefaultLogger("INFO", (entry) => {
        const log = {
          label: entry.meta?.activityId
            ? "activity"
            : entry.meta?.workflowId
            ? "workflow"
            : "worker",
          msg: entry.message,
          timestamp: Number(entry.timestampNanos / 1000000n),
          metadata: entry.meta,
        };

        switch (entry.level) {
          case "DEBUG":
            pino.debug(log);
            break;
          case "INFO":
            pino.info(log);
            break;
          case "WARN":
            pino.warn(log);
            break;
          case "ERROR":
            pino.error(log);
            break;
          case "TRACE":
            pino.trace(log);
            break;

          default:
            console.log(log);
            break;
        }
      }),
      // Telemetry options control how logs are exported out of Rust Core.
      telemetryOptions: {
        logging: {
          // This filter determines which logs should be forwarded from Rust Core to the Node.js logger. In production, WARN should generally be enough.
          filter: makeTelemetryFilterString({ core: "WARN" }),
        },
      },
    });
  }
}

The ConfigureRuntime() function configures the Worker runtime to log using pino, and also forwards the Rust core logs to Node.js so that we can ingest it using pino and the Axiom backend. There are comments in the code to give you hints about what the code does.

The final step is to call the ConfigureRuntime() function before the Worker Initialisation code. For example:

async function main() {
  ConfigureRuntime()

  // Create a worker that uses the Runtime instance installed above
  const worker = await Worker.create({
    workflowsPath: require.resolve('./workflows'),
    activities,
    taskQueue: 'custom-logger',
  });
  await worker.run();
}

main();

With that setup, your Worker is configured to log with pino, which in turn uses @axiomhq/pino as transport to send logs to Axiom.

Happy, Sweet Ending

I hope this post has been worth the read for you, and you can tweak it to use pino with any other logging service for transport. You can learn more about logging in Temporal by referencing this documentation page.

The code used in the example follows the Temporal SDK version 1.8.9. Things may change in the future, hopefully, you'll still be able to adapt this example when that happens. Happy coding 😎