🚀 From Zero to Observable: A Real-World Payment API with OpenTelemetry and SigNoz

Hey folks! 👋

Have you ever found yourself staring at a spinning loader for minutes, only to hear a teammate yell, “Is the API down again?” You check your logs — nothing looks broken. Your dashboards say CPU and memory are fine. And yet, the system is clearly not okay. 😵‍💫

It's like trying to solve a jigsaw puzzle with half the pieces missing. 🧩That, my friends, is what life without observability feels like.

But what if you could:

• 🔍 Trace exactly where requests slow down

• ⚠️ Spot the services throwing errors in real-time

• 📊 Correlate metrics, logs, and traces — in one place

Good news: you can. And today, we’ll show you how — using OpenTelemetry and SigNoz.

So open your favorite IDE 💻, fire up your terminal , and let’s turn confusion into clarity with real-time observability that just works. ✅

🌐 Why Observability Matters Now More Than Ever

Modern systems are distributed, dynamic, and difficult to reason about. When things go wrong, traditional logs and metrics can’t keep up. 🕵️‍♂️

You need:

• 🧩 Full context across services

• ⏱️ Real-time visibility into latency and errors

• 🔗 A single place to correlate metrics, logs, and traces

That’s where observability comes in—not just seeing that something broke, but why it broke, and where it originated. 💡

So, what’s the best open-source combo to get there?

Let’s meet the heroes: OpenTelemetry + SigNoz 🦸‍♂️🦸‍♀️

If you've recently experienced unexpected outages, user complaints, or service slowdowns, you know how painful it is to debug in the dark. Metrics can tell you that something's wrong. Logs might give you fragments. But traces? Traces tell the whole story. 🧵

Enter: OpenTelemetry + SigNoz — your new observability stack that doesn’t just show you what's broken but why.

OpenTelemetry + SigNoz

🛠️ OpenTelemetry: Your App's Internal Reporter

OpenTelemetry (OTel) is a vendor-neutral, open-source standard for collecting observability data—traces, metrics, and logs—from your application. It allows you to instrument your services with minimal effort and ensures consistent telemetry formats across tools and services.

With OpenTelemetry, you can:

• 🔄 Trace requests across microservices

• 📏 Measure response time and throughput

• 🧯 Capture errors and exceptions in context

📺 SigNoz: One Dashboard to Rule Them All

SigNoz is a modern, open-source alternative to tools like DataDog and New Relic. It supports traces, metrics, and logs in one sleek UI and is fully compatible with OpenTelemetry.

Why use SigNoz?

• 🧰 Built-in support for OpenTelemetry

• 🏠 Self-hostable (great for privacy or compliance-sensitive orgs)

• ⚡ ClickHouse-powered performance

• 🔓 Avoids vendor lock-in

Together, OpenTelemetry and SigNoz offer a seamless, open-source observability pipeline—free, flexible, and production-grade. It's the dream team for devs who want to understand their systems, not just monitor them. 🚀

---

Building an Observable Node.js App

In this tutorial, we’ll go from zero to observable using a real-world inspired Payment API, built with Node.js. We’ll instrument it with OpenTelemetry, and visualize our observability data using a self-hosted SigNoz instance

📎 I've created a sample GitHub repo with all the code and config files mentioned in this blog. Check out the repo here .

🧱 Step 1: Spin Up SigNoz (Self-Hosted)

We begin by setting up our observability backend using SigNoz. This will give us the dashboard to view traces, metrics, and logs.

📦 Install SigNoz via Docker

git clone -b main https://github.com/SigNoz/signoz.git && cd signoz/deploy/
./install.sh

Once up and running:

• Visit 🌐 http://localhost:3301

• Login with:
  Username: admin@signoz.io
Password: password

You now have your observability command center ready! 🎯

💳 Step 2: Create the Payment API

Next, we’ll build a simple Node.js app that simulates payment processing.

📄 index.js

const express = require("express");
const { v4: uuidv4 } = require("uuid");
const app = express();
app.use(express.json());

const payments = {};

app.post("/pay", (req, res) => {
  const id = uuidv4();
  const amount = req.body.amount || 100;
  const delay = Math.random() * 1000;

  setTimeout(() => {
    if (Math.random() < 0.2) {
      return res.status(500).json({ error: "Payment gateway failure" });
    }
    payments[id] = { id, status: "SUCCESS", amount };
    res.json({ id, status: "SUCCESS" });
  }, delay);
});

app.get("/status/:id", (req, res) => {
  const data = payments[req.params.id];
  if (!data) return res.status(404).json({ error: "Not found" });
  res.json(data);
});

app.listen(3000, () => console.log("💳 Payment API running on port 3000"));

📡 Step 3: Add OpenTelemetry Instrumentation

Let’s wire up OpenTelemetry to collect traces.

📦 Install OTel dependencies

npm install @opentelemetry/api \
  @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-otlp-http

🧠 Create tracing.js

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-otlp-http');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'http://localhost:4318/v1/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start()
  .then(() => console.log("✅ OpenTelemetry initialized"))
  .catch((error) => console.log("Error initializing OTel", error));

🔁 Import it in index.js

require("./tracing");

---

🔍 Step 4: Trace in SigNoz

Now, generate some traffic to see traces in action:

curl -X POST http://localhost:3000/pay
curl http://localhost:3000/status/<your-payment-id>

Then open 🌐 http://localhost:3301:

• Go to the Traces tab

• Search for your endpoints

• View spans and timing info

🔄 Bonus: Simulate Downtime & Monitor

By introducing random failures in the /pay route, you can:

• Set up alerts for high error rates

• Monitor latency percentiles

• Identify problematic endpoints

Use these insights to prioritize your incident response and infrastructure investments. 💪

---

⏭️ Where to Go from Here?

Now that you’ve set up observability:

• ✅ You know what’s broken (error tracing)

• ✅ You know why it’s broken (root cause visibility)

• ✅ You can act fast (reduced MTTR)

Next steps:

• Add metrics with OpenTelemetry SDK

• Stream logs with FluentBit into SigNoz

• Instrument other microservices the same way

---

🏁 Why SigNoz is the Best Fit for Open Source Observability

At the end of the day, SigNoz stands out because it:

• Works out of the box with OpenTelemetry

• Offers a beautiful, unified UI for traces, metrics, and logs

• Runs fully self-hosted for teams who care about privacy

• Avoids cloud vendor lock-in and expensive SaaS costs

• Has a growing, active open-source community

If you want production-grade observability with open-source freedom—SigNoz is your command center.

📚 Getting Started

If you're excited to go deeper after this tutorial, here are some solid starting points from the SigNoz and OpenTelemetry ecosystems:

• 🔗 Getting Started with OpenTelemetry – Understand the ecosystem, architecture, and telemetry data types.

• 📖 SigNoz Docs – OpenTelemetry Integration – Learn how to connect OTel from multiple languages.

• 💡 SigNoz Blog – A primer on distributed tracing and how to use it with SigNoz.