How to Test ElastAlert Locally Using LocalStack: A Step-by-Step Guide

Hello Devs ,

I was recently assigned the task of migrating alert notifications from Slack to Microsoft Teams. However, I encountered challenges testing these alerts in a live environment, especially the inability to delete ECS clusters or tasks without impacting production systems. To address this issue, I looked for a way to simulate the alerting process locally. This search led me to discover LocalStack, a powerful tool that enables you to test AWS services on your local machine. In this blog, I will guide you through setting up LocalStack to test ElastAlert rules, ensuring a smooth alert migration without the risk of disrupting live services.

1. Install Prerequisites

Before getting started, ensure you have the following installed:

• Python: Download and install Python (3.7 or above) from python.org.

• AWS CLI: Install the AWS CLI to manage LocalStack resources easily. Follow the AWS CLI installation guide.

• Docker: Install Docker, as it is required for running LocalStack.

2. Set Up LocalStack

Using the LocalStack Desktop App

1. Download the LocalStack Desktop app from the LocalStack website.

2. Install the app by following the provided instructions for your operating system.

3. Open the LocalStack Desktop app and start the LocalStack service. This will create a LocalStack environment where you can run your AWS services locally.

Using Docker Desktop LocalStack Extension

1. If you prefer using Docker, open Docker Desktop.

2. Navigate to the Extensions section and search for LocalStack.

3. Install the LocalStack extension directly from Docker Desktop.

4. Once installed, you can start LocalStack by selecting it from the extensions list.

3. Create LocalStack Elasticsearch Instance

After installing LocalStack (either through the Desktop app or Docker extension), create an Elasticsearch domain. It may take a few minutes for the service to start.

Run the following command to create a LocalStack Elasticsearch domain: —domain-name can use any word I just used “testinglocally”

aws es create-elasticsearch-domain --domain-name testinglocally --endpoint=http://localhost:4566

Once you run this command it will show host value. copy that host value will require while doing step 4.

{
    "DomainStatus": {
        "DomainId": "000000000000/testinglocally",
        "DomainName": "locales",
        "ARN": "arn:aws:es:us-east-2:000000000000:domain/testinglocally",
        "Created": true,
        "Deleted": false,
        "Endpoint": "testinglocally.us-east-2.es.localhost.localstack.cloud:4566",
        "Processing": true,
        "UpgradeProcessing": false,
        "ElasticsearchVersion": "7.10",
        "ElasticsearchClusterConfig": {
            "InstanceType": "m3.medium.elasticsearch",
            "InstanceCount": 1,
            "DedicatedMasterEnabled": true,
            "ZoneAwarenessEnabled": false,
            "DedicatedMasterType": "m3.medium.elasticsearch",
            "DedicatedMasterCount": 1
        }
    }
}

Verify the instance by listing the domains:

aws es describe-elasticsearch-domain --domain-name testinglocally --endpoint-url=http://localhost:4566

4. Clone ElastAlert

Clone the ElastAlert repository from GitHub:

git clone https://github.com/jertel/elastalert2
cd elastalert2

Create Configuration File

Copy the example configuration file and create a new config.yaml:

cp config.yaml.example config.yaml

You can place this file in the root directory or, if you prefer a structured approach, keep it in an example folder. If you do this, remember to specify the path when running ElastAlert rules. elastalert-test-rule --config <path-to-config-file> example_rules/example_frequency.yaml

5. Install ElastAlert Dependencies

Install the required Python packages by running:

pip install -r requirements.txt
python setup.py install

6. Create an ElastAlert Rule

Navigate to the example/rules folder in the ElastAlert repository. You can either select an existing example rule or create a new one for testing. Here’s an example structure for a new rule:

name: login page
include: ["@timestamp", "monitor.name", "monitor.status", "ev.application.name"]
type: frequency
# We want to alert if we have been down for 5 minutes
# Heartbeat runs every 60s, so we need 6 failures in 330s to trigger an alert
# timeframe = (num_events - 1) * 60s + 60s / 2
num_events: 1
timeframe:
  minutes: 10
index: heartbeat-*
filter:
- query:
    query_string:
      query: 'monitor.name: "login page" AND monitor.status: "down"'
alert:
  - ms_power_automate
ms_power_automate_webhook_url: "https://webhook.site/e3f965f6-4087-4aad-a1db-7e46db55ae1d"

7. Create an Index and Sample Data

Create the index that your rule will use, such as heartbeat or filebeat: You can use http://localhost:4566 or locales.us-east-2.es.localhost.localstack.cloud:4566.

curl -X PUT "http://localhost:4566/filebeat-1"

Insert Sample Data

Use the following PUT request to add sample data:

Note : use UTC timing for testing if you want to know what is your machine UTC timing then run this command date -u it will display like this Mon Sep 30 11:01:57 UTC 2024 take out of timing and use in below CURL request.

curl -X PUT "http://localhost:4566/filebeat-1/_doc/1" -H 'Content-Type: application/json' -d '{
    "message": "Test log entry",
    "@timestamp": "2024-09-30T14:40:00Z",
    "monitor": {
        "name": "Test Monitor",
        "status": "down"
    }
}'

8. Test the ElastAlert Rule

Run the ElastAlert test command:

elastalert-test-rule examples/rules/example_error.yaml --alert --config examples/config.yaml

This command will simulate the alert based on the created rule and the data in your Elasticsearch instance.

9. Troubleshooting and Triage

After running the test, you may want to perform some maintenance and triage tasks or while testing alerts locally, you may encounter some common errors. Here are some triaging steps you can follow:

Deleting an Index

To delete an index, run: Whichever index you have created use that to delete example if you have created hearbeat-1 or filebeat-1 then used accordingly.

curl -X DELETE "http://localhost:4566/filebeat-1"

Refreshing the Index

You can refresh the index to ensure that all operations are performed:

curl -X POST "http://localhost:4566/filebeat-1/_refresh"

Searching the Index

You can search the index to verify data:

curl -X GET "http://localhost:4566/filebeat-1/_search"

Searching with Filters

Apply filters to your search for more specific queries:

curl -X GET "http://localhost:4566/filebeat-1/_search?q=monitor.status:down"

Check Indexes:

Run the following command to verify the indices in Elasticsearch

curl -X GET 'http://localhost:4566/_cat/indices?v'

Ensure Correct Timing:

Alerts are often time-sensitive. Make sure the @timestamp field in your sample data falls within the range of the query in your rule. If your rule looks at data from a specific time range, ensure that your sample data aligns with this range.

Verbose Mode:

If the alerts aren't triggering, run ElastAlert in verbose mode to get more detailed output**.** So that can see what going wrong. In case my case i created one index with filebeat and testing alert with heartbeat and this issue able to found because of verbose flag.

elastalert-test-rule examples/rules/example_error.yaml --alert --verbose

Error Resolution:

Common errors like index_not_found_exception can be resolved by creating the index first or ensuring the correct configuration file paths.

Conclusion

By following these steps, you can effectively test ElastAlert locally using LocalStack and simulate web hook alerts. This setup is ideal for development and testing environments where you can validate your alerting rules before deploying them to production.

Happy testing! 🚀

I hope this blog helps you learn. Feel free to reach out to me on my Twitter handle @AvinashDalvi_ leave a comment on the blog.

References:

• ElastAlert Documentation

• LocalStack Documentation

• https://docs.localstack.cloud/user-guide/aws/elasticsearch/

• https://hub.docker.com/r/localstack/localstack#installing

• https://docs.aws.amazon.com/cli/latest/reference/es/describe-elasticsearch-domain.html