Navigating Common Challenges in Docker Multi-Stage Dockerfile and Docker Compose Configurations for Java Spring Boot Applications

Introduction

Creating a robust, containerized application is an essential skill for today’s developers, especially when dealing with complex multi-tier applications. Recently, I had the opportunity to build and deploy two applications—a banking app and an expense tracker app—using Docker multi-stage builds and Docker Compose. While the setup process was a rewarding experience, it was not without challenges. This article covers some of the key issues I faced, along with solutions and best practices to help you set up similar Dockerized applications smoothly.

---

1. Multi-Stage Dockerfile Challenges

Multi-stage builds allow us to create lean production images by separating the build and runtime environments, which is ideal for Java Spring Boot applications. However, getting the configuration right can be tricky.

Problem: File Not Found Errors in Production Stage

In multi-stage builds, it’s common to encounter errors like Error: Unable to access jarfile /app/target/expenseapp.jar. This usually happens when files aren’t correctly copied from the build stage to the production stage.

Solution: Double-check the paths and ensure they are consistent in each stage. Use absolute paths where possible and confirm that the COPY commands correctly reference files generated in the previous stage.

Example:

# Stage 1: Build
FROM maven:3.8.4 AS build
WORKDIR /app
COPY . .
RUN mvn clean package

# Stage 2: Runtime
FROM openjdk:17
WORKDIR /app
COPY --from=build /app/target/expenseapp.jar expenseapp.jar
ENTRYPOINT ["java", "-jar", "expenseapp.jar"]

Tip: Use Docker’s ls commands to inspect directories after each stage. This helps to quickly identify if files are missing before you progress to the next stage.

---

2. Database Connectivity and Permissions

Configuring MySQL to work with Spring Boot applications often presents connectivity and permissions issues.

Problem: Communications link failure and Public Key Retrieval is not allowed

These errors indicate issues with connecting the application container to the MySQL container, typically due to insecure connections or MySQL authentication issues.

Solution: For MySQL, modify the connection URL to enable public key retrieval and verify that your credentials match.

Example:

propertiesCopy codespring.datasource.url=jdbc:mysql://mysql:3306/expenses_tracker?allowPublicKeyRetrieval=true&useSSL=false
spring.datasource.username=root
spring.datasource.password=your_password

Tip: Avoid using default root credentials in production. Instead, create a dedicated user with specific privileges for your app.

---

3. Troubleshooting Container Linking in Docker Compose

Ensuring seamless communication between containers in a Docker Compose setup can be challenging, especially if services are trying to connect before others are fully initialized.

Problem: Application starts before MySQL is fully ready

If the application container attempts to connect to MySQL before it’s fully initialized, it may fail with Communications link failure.

Solution: Use depends_on to control the startup order in docker-compose.yml and implement a wait-for-it script or similar to ensure MySQL is ready before the application container starts.

Example:

services:
  mysql:
    image: mysql:9.1.0
    environment:
      MYSQL_ROOT_PASSWORD: your_password
      MYSQL_DATABASE: expenses_tracker
  app:
    image: your-app-image
    depends_on:
      - mysql
    entrypoint: ["./wait-for-it.sh", "mysql:3306", "--", "java", "-jar", "expenseapp.jar"]

Tip: depends_on only controls the order Docker starts services, not when they’re ready to accept connections. Use health checks or wait-for scripts to handle this.

---

4. Managing Permissions in Shared Volumes

When working with shared volumes for persistent data storage or configuration sharing, permission issues can arise if the file owner doesn’t match the user inside the container.

Problem: Permission Denied Errors for Shared Volumes

If the application can’t read or write to shared volumes, you may see Permission Denied errors.

Solution: Ensure that the user running the application has the correct permissions on shared volumes. One way is to specify the user in the docker-compose.yml file, or alternatively, you can adjust permissions directly on the host.

Example:

app:
  image: your-app-image
  volumes:
    - app_data:/app/data
  user: "${UID}:${GID}"  # Set to match your local user

Tip: Use Docker’s chown and chmod commands to set permissions within the container if required.

---

5. Simplifying Debugging with Logs and Shell Access

In complex setups, troubleshooting often requires direct access to containers or examining logs more closely.

Problem: Accessing Logs and Debugging in Real-Time

Standard logs may not provide enough information to troubleshoot startup or runtime issues, especially during database initialization or networking failures.

Solution: Use docker-compose logs -f <service_name> to get real-time logs for each service. Additionally, use docker exec -it <container_name> bash to access a container’s shell directly for further investigation.

---

6. Declaring Environment Variables in Docker Compose

Environment variables play a crucial role in configuring your Docker services, especially for sensitive data like database credentials. Here’s how to declare environment variables correctly within the docker-compose.yml file.

Docker Compose provides several options for setting environment variables:

a) Inline Environment Variables in docker-compose.yml

This method declares environment variables directly within the service definition. It's quick but not ideal for sensitive information in a production environment.

services:
  mysql:
    image: mysql:9.1.0
    environment:
      MYSQL_ROOT_PASSWORD: your_password
      MYSQL_DATABASE: expenses_tracker
      MYSQL_USER: app_user
      MYSQL_PASSWORD: app_password

  app:
    image: your-app-image
    environment:
      SPRING_DATASOURCE_URL: jdbc:mysql://mysql:3306/expenses_tracker?allowPublicKeyRetrieval=true&useSSL=false
      SPRING_DATASOURCE_USERNAME: app_user
      SPRING_DATASOURCE_PASSWORD: app_password

b) Using an .env File

For better security, you can store environment variables in an external .env file. This keeps sensitive data separate from the main configuration and allows you to update variables without modifying docker-compose.yml.

Step 1: Create a .env file in the same directory as your docker-compose.yml.

# .env file
MYSQL_ROOT_PASSWORD=your_password
MYSQL_DATABASE=expenses_tracker
MYSQL_USER=app_user
MYSQL_PASSWORD=app_password
SPRING_DATASOURCE_URL=jdbc:mysql://mysql:3306/expenses_tracker?allowPublicKeyRetrieval=true&useSSL=false
SPRING_DATASOURCE_USERNAME=app_user
SPRING_DATASOURCE_PASSWORD=app_password

Step 2: Reference the .env file in docker-compose.yml. Docker Compose will automatically load variables from this file.

services:
  mysql:
    image: mysql:9.1.0
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_DATABASE=${MYSQL_DATABASE}
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}

  app:
    image: your-app-image
    environment:
      - SPRING_DATASOURCE_URL=${SPRING_DATASOURCE_URL}
      - SPRING_DATASOURCE_USERNAME=${SPRING_DATASOURCE_USERNAME}
      - SPRING_DATASOURCE_PASSWORD=${SPRING_DATASOURCE_PASSWORD}

Tip: Ensure that the .env file is listed in .gitignore to prevent sensitive information from being committed to version control.

c) Environment Variables in External Files with env_file

For larger projects, if you have multiple environment files (e.g., for different stages like development, testing, and production), use the env_file option in docker-compose.yml to specify which environment file to load.

services:
  mysql:
    image: mysql:9.1.0
    env_file:
      - .env

---

Best Practices for Environment Variables in Docker Compose

1. Separate Sensitive Data: Use .env files for sensitive data and keep them out of version control.

2. Use Variable Substitution: Reference variables with ${VARIABLE_NAME} syntax to ensure flexibility across environments.

3. Default Values: Use default values by setting them directly in .env or by defining them with the ${VARIABLE:-default_value} syntax in docker-compose.yml.

Conclusion

Setting up multi-stage Dockerfiles and Docker Compose for a multi-tier Java application can be challenging but rewarding. By anticipating common pitfalls and following best practices, you can avoid many of the common issues and speed up your development cycle.

Stay tuned for more Docker and DevOps tips, and feel free to share your own experiences or challenges in the comments. Happy containerizing!

You can refer to the Dockerfile and docker-compose.yml files here:
https://github.com/sneh-create/Expenses-Tracker-WebApp.git

Happy learning!! 🤩