Effortless Incident Response: Using Datadog and Slack for Effective Monitoring.

Link for the previous blog for better understanding (https://hashnode.com/post/cm4izg0fy000909mn0hme1n4h)

Step 1: Create a Virtual Machine (VM) on Azure

1. Log In to Azure Portal:

   • Go to the Azure Portal and log in with your Microsoft account.

2. Create a New Virtual Machine:

   • Click on "Create a resource" and select "Virtual Machine".

   • Fill out the required details, including Subscription, Resource Group, and VM name.

   • Choose the appropriate region and image (e.g., Ubuntu, Windows Server).

3. Configure VM Size and Settings:

   • Select a VM size based on your needs

   • Set up the administrative account with a username and password or SSH key.

4. Configure Networking:

   • Ensure that the VM has a public IP address.

   • Set up any necessary inbound port rules (e.g., SSH for Linux, RDP for Windows).

5. Review and Create:

   • Review all the settings, then click "Create" to deploy the VM.

Step 2: Connect to Your Azure VM Using MobaXterm

1. Download and Install MobaXterm:

   • If you haven't already, download and install MobaXterm.

2. Open MobaXterm and Create a New Session:

   • Launch MobaXterm and click on the "Session" icon.

   • Select "SSH" and enter the public IP address of your Azure VM.

   • Specify the username and either use a password or an SSH key.

3. Connect to the VM:

   • Click "OK" to start the session and connect to your VM.

Step 3: Install Datadog Agent

1. Sign Up for Datadog:

   • If you don’t have an account, sign up at Datadog.

2. Install the Datadog Agent:

   • Follow the instructions on the Datadog website to install the Datadog Agent

3. Verify Installation:

   • Ensure the Datadog Agent is running and collecting metrics:

Step 4: Set Up Datadog agent API to Mobaxterm

1. Go to Datadog

   . Click on integration and select agent.which in this case we are using ubuntu

   . create an API key and copy to mobaxterm to run.

   . To check if its running successfully use this command (sudo systemctl status datadog-agent)

   . On datadog,click on infrastructure to check if the host created on mobaxterm has been integrated.

   . Click on monitors and select new template and select (CPU usage is high)This should take you here.

   

   When you scroll down.on the SET ALERT CONDITION.We will choose our alert threshhold to be on 40%

2. Connect Datadog to Slack:

   • Download Slack if you do not have one yet and create a slck channel to monitor your CPU e.g(#slacktesting)

   • invite datadog as a user on the channel as a user ( /invite @Datadog)

   • In the Datadog portal, navigate to Integrations and select Slack,then install.

   • Follow the instructions to connect your Slack account and specify which channels should receive alerts.

3. Create Monitors in Datadog:

   • In the Datadog dashboard, go to the "Monitors" section and create a new monitor.

   • Set the conditions for the alert and configure the notification settings to send alerts to your Slack channel.

4. Test the Integration:

   • Trigger a test alert to ensure notifications are sent to Slack.

   • This should show on your slack channel.

     

5. Step 5: Monitor and Respond

6. Monitor Alerts:

   • Using the command (htop or top) to see the process that is running real time

   • Keep an eye on the alerts being sent to Slack and respond promptly

     

   • htop to see the id number trying to trigger the CPU

   • By using (kill -9) to eliminate all threat in the server.

   • (ps) to see more information on your system.

7. Collaborate in Slack:

   • Use Slack to discuss and resolve any issues as a team.

By following these steps, you’ll have a virtual machine set up, integrated with Datadog for monitoring, and configured to send alerts to Slack for quick incident response.

If you have any questions or need further assistance, feel free to ask!