Introduction

Prometheus is a widely used monitoring and alerting system designed for dynamic environments like Kubernetes. When using Prometheus in Kubernetes, ServiceMonitor plays a crucial role in automatically discovering and scraping metrics from Kubernetes services.

Additionally, Service Discovery helps Prometheus dynamically detect targets without manual configuration. While both mechanisms aim to automate monitoring, they have key differences in implementation and use cases.

This article covers:

• What is a ServiceMonitor?

• What is Service Discovery in Prometheus?

• Differences between ServiceMonitor and Service Discovery

• Step-by-step guide to configuring both methods

A. What is a ServiceMonitor?

A ServiceMonitor is a Kubernetes Custom Resource Definition (CRD) introduced by the Prometheus Operator. It helps Prometheus discover and scrape metrics from Kubernetes services dynamically, eliminating the need for manual configuration in prometheus.yml.

Key Features of ServiceMonitor

✅ Automated Discovery – Dynamically finds services to monitor based on labels.
✅ Kubernetes-Native Configuration – Managed as a Kubernetes resource.
✅ Flexible Scraping Configurations – Allows defining scrape intervals, endpoints, and relabeling.
✅ Integration with Prometheus Operator – Works seamlessly in Kubernetes environments.

Why Not Use Static Scrape Configs?

Traditionally, Prometheus required manual target configurations in prometheus.yml. This approach is inefficient in Kubernetes, where services frequently change. ServiceMonitor provides a dynamic, automated, and scalable solution.

B. What is Service Discovery in Prometheus?

Service Discovery (SD) is a mechanism that allows Prometheus to automatically detect and scrape targets based on dynamic infrastructure changes. It eliminates the need for manually specifying each target in prometheus.yml.

Key Features:

✅ Automatically discovers targets (nodes, pods, services, VMs, etc.).
✅ Works with multiple platforms like Kubernetes, AWS, Consul, EC2, and file-based discovery.
✅ Reduces manual configurations in dynamic environments.
✅ Prometheus fetches targets directly without any additional Kubernetes resources.

Example: Service Discovery in Kubernetes

scrape_configs:
  - job_name: "kubernetes-services"
    kubernetes_sd_configs:
      - role: service

How it works:

• Prometheus queries the Kubernetes API for services with exposed ports.

• It dynamically updates targets as new services are created or removed.

Differences Between Service Discovery and ServiceMonitor

Real-World Examples: Service Discovery vs. ServiceMonitor in Prometheus

Example 1: Service Discovery in Standalone Prometheus

Scenario:

You have a Kubernetes cluster where multiple services expose metrics, and you are using standalone Prometheus (not managed by Prometheus Operator).

Implementation Steps:

1. Deploy Standalone Prometheus

2. Enable Kubernetes Service Discovery in prometheus.yml

3. Expose the service and verify the targets in Prometheus UI

scrape_configs:
  - job_name: "kubernetes-services"
    kubernetes_sd_configs:
      - role: service

Apply and restart Prometheus. Then verify the targets in Prometheus UI (http://localhost:9090/targets).

Example 2: ServiceMonitor in Prometheus Operator

Scenario:

You are using Prometheus Operator and want to monitor a specific Kubernetes service with a ServiceMonitor instead of configuring prometheus.yml manually.

Implementation Steps:

1. Install Prometheus Operator (if not installed)

2. Deploy an application exposing metrics

3. Create a Kubernetes Service

4. Create a ServiceMonitor to monitor that service

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-service-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: http
      interval: 30s
  namespaceSelector:
    matchNames:
      - default

Apply and verify targets in Prometheus UI (http://localhost:9090/targets).

Conclusion

• Use Service Discovery when running standalone Prometheus and need automatic discovery of services.

• Use ServiceMonitor if you are using Prometheus Operator in Kubernetes and need fine-grained control over how metrics are scraped.

Mastering both methods ensures efficient and automated monitoring in various deployment scenarios. 🚀