Ever feel like you're drowning in a sea of DevOps data? Jira tickets here, GitHub commits there, Jenkins builds somewhere else... it's enough to make any engineering team's head spin. Trying to make sense of this fragmented information to improve your development process can feel like searching for a needle in a haystack.

Enter Apache DevLake, an open-source platform that's here to rescue you from data chaos. DevLake ingests, analyzes, and visualizes data from all your favorite DevOps tools, transforming it into actionable insights that can boost your team's efficiency and drive engineering excellence.

In this blog, we'll explore what DevLake is, why it's essential for modern software development, and how it works. But we won't stop there! I'll also share an exclusive guide on how to set up DevLake with HTTPS using Docker Compose and Traefik, ensuring your valuable data stays secure.

Why DevLake? Taming the Data Beast

The problem is simple: your DevOps data lives in silos. Jira, GitHub, GitLab, Jenkins – each tool holds a piece of the puzzle, but getting a complete picture requires tedious manual effort. This fragmentation leads to:

Difficulty tracking progress: Are your sprints on track? Are you hitting your deadlines? It's hard to tell when data is scattered.
Bottlenecks and inefficiencies: Identifying and addressing bottlenecks becomes a guessing game without a unified view of your workflow.
Limited performance measurement: Measuring team performance and identifying areas for improvement becomes challenging.

DevLake tackles these challenges head-on. By collecting and centralizing your DevOps data, it provides a single source of truth, allowing you to:

Track key metrics: Monitor DORA metrics (Deployment Frequency, Lead Time for Changes, Mean Time to Recovery, Change Failure Rate) to understand your team's performance and identify areas for improvement.
Identify high-risk areas: Pinpoint bottlenecks, slowdowns, and potential problems early on.
Optimize workflows: Make data-driven decisions to streamline your development process.

How DevLake Works: From Chaos to Clarity

DevLake operates in three key stages:

Data Ingestion: DevLake connects to various DevOps tools, including Jira, GitHub, GitLab, Jenkins, and more, extracting relevant data through their APIs.
Data Processing and Analysis: The collected data is transformed and analyzed, creating valuable metrics and insights.
Visualization and Dashboards: DevLake provides built-in dashboards and visualization tools to present the data in a clear and understandable way. You can also create custom dashboards to meet your specific needs.

Setting up DevLake

Ready to experience the power of DevLake? Let's set it up with a secure HTTPS connection using Docker Compose and Traefik.

Prerequisites:

An AWS account
Basic knowledge of Docker and Docker Compose

Steps:

Launch an EC2 instance: Choose an appropriate instance type and operating system.
Install Docker and Docker Compose: Follow the official instructions for your OS.
Configure Traefik: Install Traefik and create a docker-compose.yml file to define it as your reverse proxy.
Obtain SSL certificates: Use Let's Encrypt (Certbot) to get free SSL certificates for your domain.
Deploy DevLake: Add DevLake to your docker-compose.yml file, specifying the necessary configurations.
Configure Traefik routing: Configure Traefik to route HTTPS traffic to the DevLake container.

version: "3"
services:
  traefik:
    image: "traefik:v2.8"
    container_name: "traefik"
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.myresolver.acme.tlschallenge=true"
      - "--certificatesresolvers.myresolver.acme.email=NAME@EMAIL.COM"
      - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
    ports:
      - "443:443"
      - "80:80"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./letsencrypt:/letsencrypt"
  mysql:
    image: mysql:8
    volumes:
      - mysql-storage:/var/lib/mysql
    restart: always
    ports:
      - 3306:3306
    environment:
      MYSQL_ROOT_PASSWORD: admin
      MYSQL_DATABASE: lake
      MYSQL_USER: merico
      MYSQL_PASSWORD: merico
      TZ: UTC
    command: --character-set-server=utf8mb4
      --collation-server=utf8mb4_bin
      --skip-log-bin

  grafana:
    image: devlake.docker.scarf.sh/apache/devlake-dashboard:v1.0.2-beta1   
    ports:
      - 3002:3000
    volumes:
      - grafana-storage:/var/lib/grafana
    environment:
      GF_SERVER_ROOT_URL: "http://localhost:4000/grafana"
      GF_USERS_DEFAULT_THEME: "light"
      MYSQL_URL: mysql:3306
      MYSQL_DATABASE: lake
      MYSQL_USER: merico
      MYSQL_PASSWORD: merico
      TZ: UTC
    restart: always
    depends_on:
      - mysql

  devlake:
    image: devlake.docker.scarf.sh/apache/devlake:v1.0.2-beta1   
    ports:
      - 8080:8080
    restart: always
    volumes:
      - devlake-log:/app/logs
    env_file:
      - ./.env
    environment:
      LOGGING_DIR: /app/logs
      TZ: UTC
    depends_on:
      - mysql

  config-ui:
    image: devlake.docker.scarf.sh/apache/devlake-config-ui:v1.0.2-beta1   
    ports:
      - 4000:4000
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami-http.entrypoints=web"
      - "traefik.http.routers.whoami-http.rule=Host(`devlake.DOMAIN.COM`)"
      - "traefik.http.routers.whoami-http.middlewares=whoami-https"
      - "traefik.http.middlewares.whoami-https.redirectscheme.scheme=https"
      - "traefik.http.routers.whoami.rule=Host(`devlake.DOMAIN.COM`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      - "traefik.http.routers.whoami.tls.certresolver=myresolver"
      - "traefik.http.services.whoami.loadbalancer.server.port=4000"
    env_file:
      - ./.env
    environment:
      DEVLAKE_ENDPOINT: devlake:8080
      GRAFANA_ENDPOINT: grafana:3000
      TZ: UTC
      #ADMIN_USER: devlake
      #ADMIN_PASS: merico
    depends_on:
      - devlake

volumes:
  mysql-storage:
  grafana-storage:
  devlake-log:

Troubleshooting:

If you encounter issues, check the DevLake documentation and community forums for solutions.

Exploring DevLake's Features: A Deeper Dive

DevLake offers a rich set of features to help you analyze your DevOps data:

Extensive Data Sources: Connect to a wide range of DevOps tools, including Jira, GitHub, GitLab, Jenkins, Bitbucket, and more.
Informative Dashboards: Access pre-built dashboards for DORA metrics, code quality, and other key performance indicators.
Customizable Metrics: Define your own metrics to track the specific data points that matter most to your team.
API and Integrations: Leverage DevLake's API to integrate with other tools and build custom applications.

Conclusion: Take Control of Your DevOps Data

DevLake empowers you to move beyond data silos and gain a holistic understanding of your software development process. By providing valuable insights and facilitating data-driven decisions, DevLake helps you optimize your workflows, improve team performance, and achieve engineering excellence.

Resources:

DevLake website: https://devlake.apache.org/
DevLake documentation: https://devlake.apache.org/docs/Overview/Introduction
DevLake community: https://devlake.apache.org/community/

Unlock Engineering Insights with Apache DevLake