Introduction

Prometheus is a widely used monitoring and alerting system designed for dynamic environments like Kubernetes. When using Prometheus in Kubernetes, ServiceMonitor plays a crucial role in automatically discovering and scraping metrics from Kubernetes services.

This article covers:

• What is a ServiceMonitor?

• Why use ServiceMonitor instead of static configurations?

• How ServiceMonitor works internally?

• Step-by-step guide to configuring ServiceMonitor

1. What is a ServiceMonitor?

A ServiceMonitor is a Kubernetes Custom Resource Definition (CRD) introduced by the Prometheus Operator. It helps Prometheus discover and scrape metrics from Kubernetes services dynamically, eliminating the need for manual configuration in prometheus.yml.

Key Features of ServiceMonitor

✅ Automated Discovery – Dynamically finds services to monitor based on labels.
✅ Kubernetes-Native Configuration – Managed as a Kubernetes resource.
✅ Flexible Scraping Configurations – Allows defining scrape intervals, endpoints, and relabeling.
✅ Integration with Prometheus Operator – Works seamlessly in Kubernetes environments.

Why Not Use Static Scrape Configs?

Traditionally, Prometheus required manual target configurations in prometheus.yml. This approach is inefficient in Kubernetes, where services frequently change. ServiceMonitor provides a dynamic, automated, and scalable solution.

2. How Does ServiceMonitor Work?

1. Prometheus Operator continuously looks for ServiceMonitor resources.

2. ServiceMonitor selects services based on labels.

3. Prometheus scrapes metrics from the discovered services and stores them in the time-series database.

4. Metrics are queried using PromQL and visualized in Grafana.

Workflow Diagram

ServiceMonitor  ─►  Prometheus Operator  ─►  Prometheus  ─►  Time Series Database  ─►  Grafana

3. How to Configure a ServiceMonitor

Step 1: Install Prometheus Operator

If the Prometheus Operator is not installed, deploy it using Helm:

helm install prometheus-operator prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace

Step 2: Deploy an Application with Metrics

Ensure your application exposes metrics on /metrics. Example: A simple Flask app exposing metrics.

from flask import Flask
from prometheus_flask_exporter import PrometheusMetrics

app = Flask(__name__)
metrics = PrometheusMetrics(app)

@app.route('/')
def home():
    return "Hello, Prometheus!"

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

Apply the Kubernetes deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: my-app:latest
          ports:
            - containerPort: 8080

Expose the application as a Kubernetes Service:

apiVersion: v1
kind: Service
metadata:
  name: my-app-service
  labels:
    app: my-app
spec:
  ports:
    - port: 8080
      targetPort: 8080
  selector:
    app: my-app

Apply the service:

kubectl apply -f service.yaml

Step 3: Create a ServiceMonitor

Now, define a ServiceMonitor to scrape metrics from my-app-service.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-service-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: http
      interval: 30s
  namespaceSelector:
    matchNames:
      - default

Apply the ServiceMonitor:

kubectl apply -f service-monitor.yaml

Step 4: Verify ServiceMonitor in Prometheus

1. Check if the ServiceMonitor is detected:

    kubectl get servicemonitor -n monitoring
   

2. Check Prometheus Targets:

    kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090
   

   Open Prometheus UI at http://localhost:9090/targets and verify that your service is listed.

Step 5: Visualize Metrics in Grafana

1. Access Grafana:

    kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80
   

2. Open http://localhost:3000 and log in with admin/admin.

3. Add Prometheus as a data source and import a pre-built dashboard for Kubernetes or application monitoring.

4. Troubleshooting ServiceMonitor

5. Summary

Conclusion

ServiceMonitor simplifies monitoring in Kubernetes by enabling automatic discovery and scraping of metrics from Kubernetes services. It removes the need for manual configuration and integrates seamlessly with the Prometheus Operator.

✅ Key Takeaways:

• ServiceMonitor automates target discovery for Prometheus.

• Works natively within Kubernetes via labels and selectors.

• Enables scalability and flexibility for monitoring applications.