Ingest ECS and RDS Logs into Dynatrace

Dynatrace is a powerful observability platform, but it doesn’t natively ingest logs from AWS ECS or RDS out of the box. However, with a few smart steps, you can pipe logs from both services into Dynatrace for unified visibility and alerting.

In this guide, we’ll walk through how to get your ECS and RDS logs into Dynatrace using CloudWatch, Lambda, and the Dynatrace Log Ingest API.

---

📦 Ingesting ECS Logs into Dynatrace

1️⃣ Enable ECS Logging to CloudWatch Logs

Start by configuring your ECS tasks to send logs to Amazon CloudWatch Logs.

Use the awslogs or firelens log driver in your ECS task definition:

"logConfiguration": {
  "logDriver": "awslogs",
  "options": {
    "awslogs-group": "/ecs/your-service",
    "awslogs-region": "us-east-1",
    "awslogs-stream-prefix": "ecs"
  }
}

🔧 Note: Firelens with Fluent Bit is an advanced option for more complex log routing and filtering before forwarding logs to CloudWatch or elsewhere.

---

2️⃣ Forward Logs to Dynatrace

Once logs are in CloudWatch:

• Create a CloudWatch Logs subscription filter for the ECS log group.

• Use a Lambda function to:

  • Parse incoming log events from CloudWatch.

  • Transform the logs into the JSON format expected by Dynatrace.

  • Send the logs via HTTPS POST to the Dynatrace Log Ingest API endpoint:

bashCopyEdithttps://<your-environment-id>.live.dynatrace.com/api/v2/logs/ingest

---

3️⃣ Enrich Logs with Metadata

Enhancing your logs with contextual metadata improves correlation and searchability in Dynatrace. Include:

• container_name

• service_name

• environment (e.g., dev/stage/prod)

• trace_id (if available)

---

4️⃣ Verify in Dynatrace

After ingestion:

• Logs will appear in the Dynatrace Log Viewer, linked to the correct services and hosts.

• You can set up log-based metrics, alerts, or create dashboard tiles to monitor container behavior.

---

• 🛢️ Ingesting RDS Logs into Dynatrace

  1️⃣ Enable RDS Logging

  For RDS, enable the appropriate logging features depending on your database engine:

  • Enhanced Monitoring

  • Slow Query Logs

  • PostgreSQL / MySQL engine logs

Make sure logs are streaming to CloudWatch Logs.

⚠️ Note: The exact steps to enable these logs can vary by RDS engine (Postgres vs MySQL) and version. Check AWS documentation for your specific database.

---

2️⃣ Forward Logs to Dynatrace

You have two options to forward RDS logs from CloudWatch Logs to Dynatrace:

✅ Option A: Lambda Function (Recommended)

• Create a CloudWatch Logs subscription filter for the RDS log group.

• Use a Lambda function to:

  • Read log events from CloudWatch.

  • Transform logs into the JSON format expected by Dynatrace.

  • Send logs via HTTPS POST to the Dynatrace Log Ingest API.

This option provides flexibility to enrich logs, filter unwanted events, and control data formatting.

---

✅ Option B: Use Kinesis Data Firehose

AWS Kinesis Data Firehose can deliver logs from CloudWatch directly to Dynatrace, acting as a fully managed data pipeline. This can simplify your architecture by eliminating the need for a Lambda function.

Step 1: Create a Kinesis Firehose Delivery Stream

• In the AWS Console, go to Kinesis → Data Firehose and click Create delivery stream.

• Set the source as Direct PUT or other sources.

• Configure the destination as HTTP Endpoint.

• Enter your Dynatrace Log Ingest API URL as the HTTP endpoint:

    bashCopyEdithttps://<your-environment-id>.live.dynatrace.com/api/v2/logs/ingest

• Configure HTTP headers for authentication:

  • Add a header Authorization with value Api-Token <your_dynatrace_api_token>.

• Enable retry and backup options (e.g., S3 backup) as needed for reliability.

Step 2: Configure CloudWatch Logs Subscription

• Navigate to your RDS log group in CloudWatch Logs.

• Choose Subscriptions and Create subscription filter.

• Select your Firehose delivery stream as the destination.

• Define the filter pattern (optional) to limit which logs are forwarded.

Step 3: Optional Transformation

• Kinesis Firehose allows adding a Lambda function for transformation before delivery.

• This Lambda can:

  • Parse raw logs.

  • Reformat them into Dynatrace’s expected JSON schema.

  • Enrich logs with metadata.

If you require complex transformation, consider adding this step; otherwise, Firehose will forward raw logs.

Step 4: Verify Logs in Dynatrace

• Logs forwarded via Firehose will appear in the Dynatrace Log Viewer.

• You can create dashboards, alerts, and perform analytics as usual.

---

🔐 API Token Setup

Both options require an API token with the logs.ingest permission generated in Dynatrace:

• Go to Settings → Integration → Dynatrace API → Generate Token

• Select Ingest logs (logs.ingest) permission.

Choose the option that best fits your architecture and operational preferences.

---

3️⃣ Add Metadata for Context

Adding metadata makes it easier to query, filter, and correlate logs in Dynatrace.

Recommended metadata:

• db_instance

• region

• query_type

• db_user

• application_name (if available)

---

4️⃣ View and Use RDS Logs in Dynatrace

Once your logs are flowing:

• Access them via the Dynatrace Log Viewer.

• Create log-based alerts (e.g., slow queries, auth failures).

• Visualize query patterns or errors using dashboard widgets.

---

🧠 Final Thoughts

While Dynatrace doesn’t natively support log ingestion from ECS or RDS, combining CloudWatch, Lambda, and the Log Ingest API gives you a scalable, flexible solution.

This setup helps you:

• Centralize observability data

• Reduce MTTR through better log correlation

• Enable smart alerting and automated remediation