I recently started a “Smart Skill” training on Cloud Academy to learn more about monitoring and observability, as these are some essentials for anyone working within DevOps/Platform engineering. Having used platforms like Splunk for incident investigations, I wanted to learn more about AWS service offerings for observability (and that led me to the lab on AWS X-Ray).

In this post, I’ll be sharing an end-to-end process of how to configure a Java API deployed on an EC2 instance, monitor logs using Amazon CloudWatch and traces with AWS X-Ray.

Note: Although the lab provided a decent starting baseline, I reconfigured it, so they can be considered two different projects and you can try working on both.

Project Overview

A Java API with /post and /get endpoints to save 4 rows of data to a DynamoDB table, and fetch each row using its ID; deployed on an EC2 instance and configured to use CloudWatch agent and X-Ray daemon for logging and tracing.

Setup

Open JDK 11
Maven
AWS X-Ray Daemon
Amazon CloudWatch agent
EC2
IAM

Pre-Requisites

Familiarity with Java and/or building Java APIs (useful for debugging)
An existing DynamoDB table for the API to store/retrieve data
Familiarise with app code: GitHub repo

The project can be setup using some automation with Terraform and Ansible, but this will be the manual deployment/configuration documentation because I wanted to work through each step and explore along the way.

1) AWS Setup

Minimal EC2 Settings

Name: xray-api
AMI: Ubuntu latest version
Instance type: t3 medium
Create your Key pair (or use an existing one, if you have one)
Security group: allow inbound traffic on SSH (port 22) and custom TCP for Tomcat (port 8080)
Defaults for everything else

An IAM role is also needed for the EC2 instance to have access to perform tasks relating to DynamoDB, X-Ray and CloudWatch. Ensure to attach the IAM role to the instance, so that it can assume the role.

// Trusted Entity - allows EC2 to perform the actions defined in the policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

// Permissions Policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DynamoDBLimitedAccess",
            "Effect": "Allow",
            "Action": [
                "dynamodb:PutItem",
                "dynamodb:DeleteItem",
                "dynamodb:GetItem",
                "dynamodb:Scan",
                "dynamodb:Query",
                "dynamodb:UpdateItem"
            ],
            "Resource": "<<ARN for your DynamoDB table>>"
        },
        {
            "Sid": "PermissionForXRayDaemon",
            "Effect": "Allow",
            "Action": [
                "xray:PutTraceSegments",
                "xray:PutTelemetryRecords"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

Once your instance is all setup and ready, SSH into the instance with your key pair.

2) Configure Instance

Install Java and configure paths

wget https://clouda-labs-assets.s3-us-west-2.amazonaws.com/aws-xray/openjdk-11%2B28_linux-x64_bin.tar.gz -O openjdk-11+28_linux-x64_bin.tar.gz
tar -xvf openjdk-11+28_linux-x64_bin.tar.gz
sudo mkdir -p /usr/lib/jvm
sudo mv jdk-11* /usr/lib/jvm/java-11-openjdk     


export JAVA_HOME=/usr/lib/jvm/java-11-openjdk
export PATH=$JAVA_HOME/bin:$PATH
echo 'export JAVA_HOME=/usr/lib/jvm/java-11-openjdk' >> ~/.bash_profile
echo 'export PATH=$JAVA_HOME/bin:$PATH:$HOME/.local/bin:$HOME/bin' >> ~/.bash_profile
source ~/.bash_profile

# check jdk version is 11
java -version

Install Maven

wget https://archive.apache.org/dist/maven/maven-3/3.9.6/binaries/apache-maven-3.9.6-bin.tar.gz
tar -xzf apache-maven-3.9.6-bin.tar.gz
sudo mv apache-maven-3.9.6 /opt/maven
echo 'export MAVEN_HOME=/opt/maven' >> ~/.bash_profile
echo 'export PATH=$MAVEN_HOME/bin:$PATH' >> ~/.bash_profile
source ~/.bash_profile

# check Maven version
mvn -v

Install AWS X-Ray Daemon

wget https://clouda-labs-assets.s3-us-west-2.amazonaws.com/aws-xray/aws-xray-daemon-linux-3.2.0.zip
sudo apt install unzip
unzip aws-xray-daemon-linux-3.2.0.zip

Configure CloudWatch (CW) agent

# Install CW agent
wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
sudo dpkg -i amazon-cloudwatch-agent.deb

# create CW config file: this will create log groups for the API and X-Ray logs 
# (which will be collected from logs stored in files configured for the X-Ray and API system services
cat > /home/ubuntu/cw-config.json << 'EOF'
{
  "agent": {
    "metrics_collection_interval": 60,
    "logfile": "/var/log/cw-agent.log"
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/home/ubuntu/api.log",
            "log_group_name": "xray-api-logs",
            "log_stream_name": "{instance_id}-app",
            "timestamp_format": "%d-%m-%Y %H:%M:%S"
          },
          {
            "file_path": "/home/ubuntu/xray-daemon.log",
            "log_group_name": "aws-xray-logs",
            "log_stream_name": "{instance_id}-xray",
            "timestamp_format": "%d-%m-%Y %H:%M:%S"
          }
        ]
      }
    }
  }
}
EOF

# Start running CW agent
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/home/ubuntu/cw-config.json -s

Clone app from GitHub and setup

# Clone
git clone https://github.com/khairahscorner/xray-app.git

# Set properties
cp xray-app/src/main/resources/application-example.properties application.properties
sudo chmod a+w application.properties

# Add needed variables to the properties file
aws.region=<value>
dynamodb.table=<value>

# Package JAR
cd xray-app
mvn package
mv target/xray-app-1.jar target/app.jar #rename jar file

Setup System Service files

# service file for the X-Ray daemon
sudo cat > /etc/systemd/system/xray_daemon.service << 'EOF'
[Unit]
Description=AWS X-Ray Daemon
After=network.target

[Service]
ExecStart=/home/ubuntu/xray --region eu-west-2
Restart=always
User=ubuntu
StandardOutput=file:/home/ubuntu/xray-daemon.log
StandardError=file:/home/ubuntu/xray-daemon.log

[Install]
WantedBy=multi-user.target
EOF

# service file for the Java API itself
sudo cat > /etc/systemd/system/java_api.service << 'EOF'
[Unit]
Description=XRay Java API
After=network.target

[Service]
User=ubuntu
ExecStart=/usr/lib/jvm/java-11-openjdk/bin/java -jar /home/ubuntu/xray-app/target/app.jar --spring.config.location=file:/home/ubuntu/application.properties
SuccessExitStatus=143
Restart=on-failure
RestartSec=10
StandardOutput=file:/home/ubuntu/api.log
StandardError=file:/home/ubuntu/api.log

[Install]
WantedBy=multi-user.target
EOF

Start Services

# Start X-Ray Daemon
sudo systemctl daemon-reload
sudo systemctl enable xray_daemon
sudo systemctl start xray_daemon

# Start Java API service
sudo systemctl daemon-reload
sudo systemctl start java_app
sudo systemctl enable java_app

# check service statuses
sudo systemctl status xray_daemon
sudo systemctl status java_app

3) Deployed App

You should now be able to access the app with the instance iP address/DNS:

http://<public ip address/dns>:8080/post

http://<public ip address/dns>:8080/get?id=1

X-Ray Traces

Tracing makes it easy to spot failed endpoint calls, it also logs minimal details for requests that completed successfully.

CloudWatch Logs

Configuring the CloudWatch agent was a last minute add-on because I wanted to store the app logs somewhere more visible than a file within the instance. I configured the agent to collect the logs from both the api and daemon log files, and this gave me an overview of how log groups keep logs separate and organised.

Troubleshooting

I ran into quite a number of issues but resolved them by spending considerable time with AWS SDK documentation and ChatGPT (mostly debugging the Java code that I modified).

Overall:

ensure you have configured the IAM role with the right access permissions for all the necessary services and attached it correctly to the EC2 instance
If you’re modifying the app code, ensure it still works correctly with AWS Java SDK v2. You can test the app locally to verify but have AWS SSO or access keys configured locally.

Useful References:

AWS Authentication with SDK for Java

Migrating AWS Java SDK v1 to v2

SDK Credential Providers

Conclusion

With this project, I’m a bit more comfortable using X-Ray for traces and sending logs over to CloudWatch. The automated version will allow me to get a refresher on working with Ansible, so that’s up next!

Thanks for reading!

Till Next Time

Monitoring Java APIs on EC2 with Amazon CloudWatch and AWS X-Ray

Table of contents