Observability with OpenTelemetry and Grafana stack Part 1: Setting up the services to observe

Driptaroop DasDriptaroop Das
8 min read

Table of contents

This is the first part of the series of articles on Observability with OpenTelemetry and Grafana stack. In this series, we will be setting up a full-fledged observability stack to monitor the services in a simulated distributed system.

Objective

This endeavor has two main objectives:

  1. To set up a full-fledged observability stack to monitor the services in a simulated distributed system in that can run in a local environment.

  2. Testing the performance impact of the observability stack on the services under observation when the services are under load.

The components of the stack

The stack will include the following components:

  • services: A set of services that will be monitored. The services are standard kotlin based Spring Boot applications with a REST API. Services are supported by a PostgreSQL database, built with Gradle and instrumented with OpenTelemetry Java agent. There are four services in total emulating a simple banking system. These services are access controlled by a authorization server(also a spring boot application) which acts as a gateway to the services.

  • OpenTelemetry Collector: The OpenTelemetry Collector is a vendor-agnostic implementation of the OpenTelemetry specification. It is designed to receive, process, and export telemetry data. This will be used to collect the telemetry data from the services and export it to the monitoring backends.

  • Loki: Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus.

  • Mimir: Mimir is prometheus compatible metrics store for high availability, multi-tenancy, durable storage, and blazing fast query performance over long periods of time.

  • Tempo: Tempo is an open source, easy-to-use, and high-scale distributed tracing backend.

  • Grafana: Grafana is used for visualizing the data from the services and monitoring backends. It will be used to create dashboards to visualize the metrics and traces.

  • K6: K6 is a load testing tool that will be used to generate load on the services.

  • Infrastructure: The services and monitoring backends will be running in simple local docker containers via compose file.

The services

There are four services in total emulating a simple banking system. They are simple Kotlin based Spring Boot applications with a REST API with a PostgreSQL database as the backend. The services are instrumented with OpenTelemetry Java agent to collect the telemetry data.

The services are:

  • user-service: This service is responsible for managing the users in the system. In our very simple banking system, the user-service only has the API to get the users. The predefined users are loaded in the database when the service starts.

  • account-service: This service is responsible for managing the accounts in the system. The predefined accounts are loaded in the database when the service starts.

  • notification-service: This service is responsible for sending notifications to the users.

  • transaction-service: This service is responsible for managing the transactions in the system.

Along with the services, there is an authorization server that acts as a gateway. This auth-server is responsible for authenticating the requests to the services via Oauth2-JWT tokens.

The services and the database design should look like this:

plantuml code for the above diagram available here.

Setting up the skeleton

  • First, lets create the empty repo/directory for the entire project.

  • Create the application directory. All the services will be in this directory.

  • Create the empty service directories under the application directory. The services are account-service, notification-service, transaction-service, user-service, and auth-server.

  • Create a shared directory under the application directory. This directory will contain the shared code and configurations for the services and will act as a library for the services.

  • The services should be independent of each other and should be able to run in isolation. But they should at least be able to build together. Therefore, define the settings.gradle.kts file in the root of the repo and include the services in it.

rootProject.name = "local-observability-opentelemetry-grafana"

include(
    "account-service",
    "notification-service",
    "transaction-service",
    "user-service",
    "shared",
    "auth-server"
)

project(":account-service").projectDir = file("application/account-service")
project(":user-service").projectDir = file("application/user-service")
project(":transaction-service").projectDir = file("application/transaction-service")
project(":notification-service").projectDir = file("application/notification-service")
project(":shared").projectDir = file("application/shared")
project(":auth-server").projectDir = file("application/auth-server")
  • Let's create an empty Dockerfile and compose file (compose.yaml) for the repo in the root directory. These files will be used to build and run the services and monitoring backends later.

Setting up the services

Lets start by setting up the user-service. The other services will be extremely similar to this one.

  • user-service is a simple spring boot application. So create the build.gradle.kts file in the user-service directory and populate it will required dependencies.
plugins {
    kotlin("jvm") version "2.1.10"
    kotlin("plugin.spring") version "2.1.10"
    id("org.springframework.boot") version "3.4.3"
    id("io.spring.dependency-management") version "1.1.7"
    kotlin("plugin.jpa") version "2.1.10"
}

group = "org.dripto"
version = "0.0.1-SNAPSHOT"

java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(21) // All of our services will be using Java 21
    }
}

repositories {
    mavenCentral()
}

dependencies {
    implementation("org.springframework.boot:spring-boot-starter-data-jpa")
    implementation("org.springframework.boot:spring-boot-starter-web")
    implementation("com.fasterxml.jackson.module:jackson-module-kotlin")
    implementation("org.springframework.boot:spring-boot-starter-oauth2-resource-server") // For Oauth2 resource server
    implementation("com.fasterxml.jackson.datatype:jackson-datatype-hibernate6")
    implementation(project(":shared")) // shared library
    implementation("org.jetbrains.kotlin:kotlin-reflect")
    implementation("org.springframework.boot:spring-boot-starter-actuator")
    runtimeOnly("io.micrometer:micrometer-registry-prometheus") // Actuator and micrometer for exposing metrics
    runtimeOnly("org.postgresql:postgresql")
    implementation("org.liquibase:liquibase-core") // Liquibase for setting up the database
}

kotlin {
    compilerOptions {
        freeCompilerArgs.addAll("-Xjsr305=strict")
    }
}

allOpen {
    annotation("jakarta.persistence.Entity")
    annotation("jakarta.persistence.MappedSuperclass")
    annotation("jakarta.persistence.Embeddable")
}

tasks.withType<Test> {
    useJUnitPlatform()
}

// We are disabling the jar task because we don't want to create the `plain` jar. We will only use the `bootJar` task to create the executable jar.
tasks {
    bootJar {
        enabled = true
    }
    jar {
        enabled = false
    }
}
  • Create the application.yaml file in the user-service/src/main/resources directory. This file will contain the configurations for the service.
spring:
  application:
    name: user-service
  datasource:
    url: jdbc:postgresql://localhost:5432/postgres # we will setup the database in a docker container later.
    username: postgres
    password: password
  liquibase:
    change-log: classpath:/db/changelog/db.changelog-master.yaml 
    default-schema: user_data
  jpa:
    hibernate:
      ddl-auto: none
    properties:
      hibernate:
        format_sql: false
        show_sql: false
        default_schema: user_data # schema for the user data. We will create this schema in the database later.
  security:
    oauth2:
      resourceserver:
        jwt:
          jwk-set-uri: http://auth-server:9090/oauth2/jwks # auth-server is the auth server that will authenticate the requests to the services. We will setup the auth-server later. Here, we are telling the service to use the jwks uri of the auth-server to validate the jwt tokens.
# enable actuator endpoints
management:
  metrics:
    tags:
      application: ${spring.application.name} # required tag for the metrics
  endpoints:
    web:
      exposure:
        include: "*" # expose all the actuator endpoints
  endpoint:
    health:
      show-details: always
logging:
  pattern:
    level: "trace_id=%mdc{trace_id} span_id=%mdc{span_id} trace_flags=%mdc{trace_flags} %5p"
  level:
    root: info
    org.dripto.application.service: debug

  • Create the liquibase changelog file db.changelog-master.yaml in the user-service/src/main/resources/db/changelog directory. This file will contain the liquibase changesets to setup the database. For this simple example, we will only have 2 changesets. One with the table DDL and another with the data.

  • Code the entity, controller, and config classes for the user-service.

// App.kt
package org.dripto.application.service.user

import com.fasterxml.jackson.datatype.hibernate6.Hibernate6Module
import org.springframework.boot.autoconfigure.SpringBootApplication
import org.springframework.boot.runApplication
import org.springframework.context.annotation.Bean
import org.springframework.security.config.Customizer
import org.springframework.security.config.annotation.web.builders.HttpSecurity
import org.springframework.security.web.SecurityFilterChain

@SpringBootApplication
class App {
    @Bean
    fun hibernate6Module() = Hibernate6Module()

    @Bean
    fun securityFilterChain(http: HttpSecurity): SecurityFilterChain =
        http.authorizeHttpRequests {
            it.requestMatchers("/actuator/**").permitAll()
            it.anyRequest().authenticated()
        }
            .oauth2ResourceServer { it.jwt(Customizer.withDefaults()) }
            .build()
}

fun main(args: Array<String>) {
    runApplication<App>(*args)
}

// User.kt
package org.dripto.application.service.user

import jakarta.persistence.Column
import jakarta.persistence.Entity
import jakarta.persistence.Id
import jakarta.persistence.Table
import org.springframework.data.jpa.repository.JpaRepository
import org.springframework.data.jpa.repository.Query
import java.time.LocalDateTime
import java.util.UUID

@Entity
@Table(name = "user_data")
class User(
    @Id
    val id: UUID,
    val username: String,
    @Column(name = "first_name")
    val firstName: String,
    @Column(name = "last_name")
    val lastName: String,
    val email: String,
    @Column(name = "created_at")
    val createdAt: LocalDateTime
)

interface UserRepository: JpaRepository<User, UUID> {
    @Query("SELECT u FROM User u WHERE u.username = :username")
    fun getByUsername(username: String): User?
}

// UserController.kt
package org.dripto.application.service.user

import org.dripto.application.service.utils.log
import org.springframework.web.bind.annotation.GetMapping
import org.springframework.web.bind.annotation.PathVariable
import org.springframework.web.bind.annotation.RestController
import java.util.UUID

@RestController
class UserController (private val userRepository: UserRepository) {
    @GetMapping("/users")
    fun getUsers(): List<User> {
        log.info("Getting all users")
        return userRepository.findAll().also {
            log.debug("Found all users: {}", it)
        }
    }

    @GetMapping("/users/{id}")
    fun getUserById(@PathVariable id: UUID): User {
        log.info("Getting user by id: {}", id)
        return userRepository.getReferenceById(id).also {
            log.debug("Found user for id {}: {}", id, it)
        }
    }
}

Rest of the services will be similar to the user-service. We will also setup the shared library to contain the shared rest clients and configurations for the services. The rest clients are written with Spring Http interfaces over RestClient.

Setting up the Auth server

The auth server is a simple spring boot application that will authenticate the requests to the services via Oauth2-JWT tokens.

  • The setup of the auth server is similar to the services. Create the build.gradle.kts file in the auth-server directory and populate it with the required dependencies.
plugins {
    kotlin("jvm") version "2.1.10"
    kotlin("plugin.spring") version "2.1.10"
    id("org.springframework.boot") version "3.4.3"
    id("io.spring.dependency-management") version "1.1.7"
}

group = "org.dripto.benchmark"
version = "0.0.1-SNAPSHOT"

java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(21)
    }
}

repositories {
    mavenCentral()
}

dependencies {
    implementation("org.springframework.boot:spring-boot-starter-oauth2-authorization-server") // we need this dependency to create the auth server using Spring Authorization Server framework
    implementation("org.springframework.boot:spring-boot-starter-web")
    implementation("org.springframework.boot:spring-boot-starter-actuator")
    implementation("com.fasterxml.jackson.module:jackson-module-kotlin")
    implementation("org.jetbrains.kotlin:kotlin-reflect")
    implementation("org.springframework.boot:spring-boot-starter-actuator")
    runtimeOnly("io.micrometer:micrometer-registry-prometheus")
}

kotlin {
    compilerOptions {
        freeCompilerArgs.addAll("-Xjsr305=strict")
    }
}

tasks.withType<Test> {
    useJUnitPlatform()
}

tasks {
    bootJar {
        enabled = true
    }
    jar {
        enabled = false
    }
}
  • Create the application.yaml file in the auth-server/src/main/resources directory.
server:
  port: 9090
spring:
  application:
    name: auth-server
management:
  endpoints:
    web:
      exposure:
        include: "*"
  endpoint:
    health:
      show-details: always
  metrics:
    tags:
      application: ${spring.application.name}
  • Create the security configuration and the client registration configuration for the auth server.
package org.dripto.application.service.auth

import com.nimbusds.jose.jwk.JWKSet
import com.nimbusds.jose.jwk.RSAKey
import com.nimbusds.jose.jwk.source.ImmutableJWKSet
import com.nimbusds.jose.jwk.source.JWKSource
import com.nimbusds.jose.proc.SecurityContext
import org.springframework.boot.autoconfigure.SpringBootApplication
import org.springframework.boot.runApplication
import org.springframework.context.annotation.Bean
import org.springframework.core.annotation.Order
import org.springframework.security.config.Customizer
import org.springframework.security.config.annotation.web.builders.HttpSecurity
import org.springframework.security.oauth2.core.AuthorizationGrantType
import org.springframework.security.oauth2.core.ClientAuthenticationMethod
import org.springframework.security.oauth2.jwt.JwtDecoder
import org.springframework.security.oauth2.server.authorization.OAuth2TokenType
import org.springframework.security.oauth2.server.authorization.client.InMemoryRegisteredClientRepository
import org.springframework.security.oauth2.server.authorization.client.RegisteredClient
import org.springframework.security.oauth2.server.authorization.client.RegisteredClientRepository
import org.springframework.security.oauth2.server.authorization.config.annotation.web.configuration.OAuth2AuthorizationServerConfiguration
import org.springframework.security.oauth2.server.authorization.config.annotation.web.configurers.OAuth2AuthorizationServerConfigurer
import org.springframework.security.oauth2.server.authorization.token.JwtEncodingContext
import org.springframework.security.oauth2.server.authorization.token.OAuth2TokenCustomizer
import org.springframework.security.web.SecurityFilterChain
import java.security.KeyPair
import java.security.KeyPairGenerator
import java.security.interfaces.RSAPrivateKey
import java.security.interfaces.RSAPublicKey
import java.util.*

@SpringBootApplication
class App {

    @Bean
    fun standardSecurityFilterChain(http: HttpSecurity): SecurityFilterChain {
        return http.with(OAuth2AuthorizationServerConfigurer.authorizationServer(), Customizer.withDefaults())
            .authorizeHttpRequests {
                it.requestMatchers("/actuator/**").permitAll()
                it.anyRequest().authenticated()
            }
            .formLogin { it.disable() }
            .csrf { it.disable() }.build()
    }

    // creating the clients for the services and k6
    @Bean
    fun registeredClientRepository(): RegisteredClientRepository {
        val k6 = createClient("k6")
        val accountService = createClient("account-service")
        val userService = createClient("user-service")
        val notificationService = createClient("notification-service")
        val transactionService = createClient("transaction-service")
        return InMemoryRegisteredClientRepository(k6, accountService, userService, notificationService, transactionService)
    }

    @Bean
    fun jwkSource(): JWKSource<SecurityContext> {
        val keyPair: KeyPair = KeyPairGenerator.getInstance("RSA").apply { initialize(2048) }.genKeyPair()
        val publicKey = keyPair.public as RSAPublicKey
        val privateKey = keyPair.private as RSAPrivateKey
        val rsaKey = RSAKey.Builder(publicKey)
            .privateKey(privateKey)
            .keyID(UUID.randomUUID().toString())
            .build()
        val jwkSet = JWKSet(rsaKey)
        return ImmutableJWKSet(jwkSet)
    }

    @Bean
    fun jwtDecoder(jwkSource: JWKSource<SecurityContext>): JwtDecoder {
        return OAuth2AuthorizationServerConfiguration.jwtDecoder(jwkSource)
    }

    @Bean
    fun tokenCustomizer(): OAuth2TokenCustomizer<JwtEncodingContext> {
        return OAuth2TokenCustomizer { context ->
            if (OAuth2TokenType.ACCESS_TOKEN == context.tokenType) {
                // Add custom claim to the access token
                context.claims.claim("hello1", "world1")
            }
        }
    }

    private fun createClient(client: String): RegisteredClient = RegisteredClient.withId(client)
        .clientId(client)
        .clientSecret("{noop}$client-secret")
        .clientAuthenticationMethod(ClientAuthenticationMethod.CLIENT_SECRET_BASIC)
        .authorizationGrantType(AuthorizationGrantType.CLIENT_CREDENTIALS)
        .scope("local")
        .build()
}

fun main(args: Array<String>) {
    runApplication<App>(*args)
}

In the next part, we will setup the OpenTelemetry Java agent to instrument the services and run the services in docker containers.

0
Subscribe to my newsletter

Read articles from Driptaroop Das directly inside your inbox. Subscribe to the newsletter, and don't miss out.

KotlinSpringbootobservabilityOpenTelemetrymimirloki#prometheusGrafana

Written by

Driptaroop Das
Driptaroop Das

Self diagnosed nerd 🤓 and avid board gamer 🎲