How to Easily Send Logs to Splunk Cloud: A Step-by-Step Guide

Overview of Log Management and Monitoring

Log management and monitoring are essential parts of modern IT infrastructure and cybersecurity. They involve collecting, storing, analyzing, and monitoring log data from various systems, applications, and devices within an organization. Effective log management helps to identify and troubleshoot issues, ensures compliance with regulations, and enhances security by detecting anomalies and potential threats.

By implementing strong log management and monitoring practices, organizations can improve operational efficiency, boost security, and ensure regulatory compliance.

In this guide, we will send our logs to Splunk Cloud using the Splunk Universal Forwarder on various operating systems.

Splunk Cloud Preparation

We need to prepare our Splunk Cloud to receive and index our logs appropriately. To achieve this, we must configure the following:

Index

1. Navigate to <your_organization>.splunkcloud.com > Settings > Index > HTTP > New Index.

2. Enter an index name and configure it as needed. Click on Save.

HEC (HTTP Event Collector)

1. Go to <your_organization>.splunkcloud.com > Settings > Data inputs > HTTP Event Collector > Add new > New Token.

2. Set a name for the HEC. You can use the same name as your index for consistency. Click on Next.

3. In the Default Index field, select the index you created. Click on Review.

4. Click on Submit and copy your Token Value.

With these configurations in place, our Splunk Cloud is ready to receive logs from various hosts using Splunk Universal Forwarders (agents).

Installation & Configuration for Windows

You can follow these steps to install and configure the Splunk Universal Forwarder on Windows systems.

1. Download the Splunk Universal Forwarder installer from here. (You will need to create an account to be able to download the installer)

2. Install the forwarder by clicking "Next" through the prompts.

   1. When asked for credentials, set them and save them somewhere safe.

   2. When asked for monitoring options, select what you need to monitor.

   3. When asked about the receiving server, leave it blank. We will configure it later.

3. Download your .spl file containing the credentials to connect to your Splunk Cloud.

   1. It is usually found at <your_organization>.splunkcloud.com > Apps > Universal Forwarder > Download Universal Forwarder Credentials

   2. Alternatively, you can find it here: https://<your_organization>.splunkcloud.com/splunkd/__raw/servicesNS/nobody/splunkclouduf/uf-config/splunkclouduf.spl

4. Open CMD.

5. Navigate to Splunk’s installation folder with the command:

    cd C:\Program Files\SplunkUniversalForwarder\bin
   

6. Install credentials:

    .\splunk.exe install app %HOMEPATH%\Downloads\splunkclouduf.spl
   

7. Restart the service:

    .\splunk.exe restart
   

8. Send data to SplunkCloud:

    .\splunk add forward-server http-inputs-<your_organization>.splunkcloud.com:443
   

9. Navigate to the directory containing your inputs.conf and outputs.conf files:

    cd C:\Program Files\SplunkUniversalForwarder\system\local
   

10. Configure your inputs.conf file as needed. Here you have an example:

    [WinEventLog://Application]
    checkpointInterval = 5
    current_only = 0
    disabled = 0
    start_from = oldest
    index=<name_of_index>
    
    [WinEventLog://Security]
    checkpointInterval = 5
    current_only = 0
    disabled = 0
    start_from = oldest
    index=<name_of_index>
    
    [WinEventLog://System]
    checkpointInterval = 5
    current_only = 0
    disabled = 0
    start_from = oldest
    index=<name_of_index>
    
    [WinEventLog://ForwardedEvents]
    checkpointInterval = 5
    current_only = 0
    disabled = 0
    start_from = oldest
    index=<name_of_index>
    
    [WinEventLog://Setup]
    checkpointInterval = 5
    current_only = 0
    disabled = 0
    start_from = oldest
    index=<name_of_index>
    
    [admon://NearestDC]
    monitorSubtree = 1
    index=<name_of_index>
    

11. Configure your outputs.conf file with the following content:

    [httpout]
    httpEventCollectorToken = <your_HEC_token>
    uri = https://http-inputs-<your_organization>.splunkcloud.com:443
    

12. Set the SPLUNK_HOME environment variable with this command:

    set SPLUNK_HOME=C:\Program Files\SplunkUniversalForwarder\opt
    

13. Restart the service:

    cd C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe restart
    

14. Find your computer name using this command:

    hostname
    

15. On SplunkCloud, search for this SplunkQL query:

    index="main" host="<write_your_hostname>"
    

16. If you see events, congratulations! You were successful!

PS1: If you need to add a new monitor on a file or folder, use this command:

.\splunk add monitor "<path to file or folder>"

(the path goes between quotes!)

PS2: To change the index where you send logs, go to:

C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local

and add a line in each section of the inputs.conf file like this:

index=<name_of_index>

Installation & Configuration for RedHat-based distributions

You can follow these steps to install and configure the Splunk Universal Forwarder on a RedHat-based distribution such as CentOS, Fedora, Oracle Linux, etc.

1. Download the Splunk Universal Forwarder installer from here. (You will need to create an account to download the installer)

2. Install the forwarder:

    rpm -i <installer_name>.rpm
   

3. Download your .spl file containing the credentials to connect to your Splunk Cloud.

   1. It is usually found at <your_organization>.splunkcloud.com > Apps > Universal Forwarder > Download Universal Forwarder Credentials

   2. Alternatively, you can find it here: https://<your_organization>.splunkcloud.com/splunkd/__raw/servicesNS/nobody/splunkclouduf/uf-config/splunkclouduf.spl

   3. You can upload your .spl to a file storage so you can download it in step 6.

4. Go to the path where the Splunk bin is located:

    /opt/splunkforwarder/bin/splunk start
   

5. Enter your user and password. It's important to save them somewhere safe because we'll need them to configure things in the future.

6. Download splunkclouduf.spl:

    curl -O https://<url_to_your_spl_file>
   

7. Install it:

    /opt/splunkforwarder/bin/splunk install app splunkclouduf.spl
   

8. Send data to Splunk Cloud:

    ./splunk add forward-server http-inputs-<your_organization>.splunkcloud.com:443
   

9. Navigate to the directory containing your inputs.conf and outputs.conf files:

    cd /opt/splunkforwarder/etc/system/local
   

10. Configure your inputs.conf file as needed. Here you have an example:

    # Monitor system logs for authentication and authorization events
    [monitor:///var/log/secure]
    disabled = false
    index = <name_of_index>
    sourcetype = linux_secure
    
    # Monitor system logs for general security events
    [monitor:///var/log/messages]
    disabled = false
    index = <name_of_index>
    sourcetype = linux_syslog
    
    # Monitor Apache access and error logs
    [monitor:///var/log/httpd/access_log]
    disabled = false
    index = <name_of_index>
    sourcetype = apache_access
    
    [monitor:///var/log/httpd/error_log]
    disabled = false
    index = <name_of_index>
    sourcetype = apache_error
    
    # Monitor SSH logs for login attempts
    [monitor:///var/log/secure]
    disabled = false
    index = <name_of_index>
    sourcetype = sshd
    
    # Monitor sudo commands executed by users
    [monitor:///var/log/secure]
    disabled = false
    index = <name_of_index>
    sourcetype = sudo
    
    # Monitor firewall logs (assuming firewalld is used)
    [monitor:///var/log/firewalld]
    disabled = false
    index = <name_of_index>
    sourcetype = firewalld
    
    # Monitor audit logs (if available)
    [monitor:///var/log/audit/audit.log]
    disabled = false
    index = <name_of_index>
    sourcetype = linux_audit
    
    # Monitor file integrity using auditd (if available)
    [monitor:///var/log/audit/auditd.log]
    disabled = false
    index = <name_of_index>
    sourcetype = auditd
    
    # Monitor for changes to critical system files
    [monitor:///etc/passwd]
    disabled = false
    index = <name_of_index>
    sourcetype = linux_config
    
    # Monitor for changes to critical system binaries
    [monitor:///bin]
    disabled = false
    index = <name_of_index>
    sourcetype = linux_config
    
    # Monitor for changes to critical system configuration files
    [monitor:///etc]
    disabled = false
    index = <name_of_index>
    sourcetype = linux_config
    

11. Configure your outputs.conf file with the following command:

    echo "[httpout]
    httpEventCollectorToken = <your_HEC_token>
    uri = https://http-inputs-<your_organization>.splunkcloud.com:443" > outputs.conf
    

12. Set the SPLUNK_HOME environment variable:

    export SPLUNK_HOME=/opt/splunkforwarder
    

13. Restart the Splunk service to apply the new configuration:

    /opt/splunkforwarder/bin/splunk restart
    

PS1: To add a new monitor on a file or folder, use the following command:

/opt/splunkforwarder/bin/splunk add monitor "<path to file or folder>"

(The path should be enclosed in quotes)

PS2: To change the index where you send logs, navigate to the directory:

cd /opt/splunkforwarder/etc/system/local

and add a line in each section of the inputs.conf file like this:

index=<name_of_index>

Installation & Configuration for Ubuntu

You can follow these steps to install and configure the Splunk Universal Forwarder on a Debian-based distribution such as Ubuntu, Debian, etc.

1. Download the Splunk Universal Forwarder installer from here. (You will need to create an account to download the installer)

2. Install the forwarder:

    dpkg -i <installer_name>.deb
   

3. Download your .spl file containing the credentials to connect to your Splunk Cloud.

   1. It is usually found at <your_organization>.splunkcloud.com > Apps > Universal Forwarder > Download Universal Forwarder Credentials

   2. Alternatively, you can find it here: https://<your_organization>.splunkcloud.com/splunkd/__raw/servicesNS/nobody/splunkclouduf/uf-config/splunkclouduf.spl

   3. You can upload your .spl to a file storage so you can download it in step 6.

4. Go to the path where the Splunk bin is located:

    /opt/splunkforwarder/bin/splunk start
   

5. Enter your user and password. It's important to save them somewhere safe because we'll need them to configure things in the future.

6. Download splunkclouduf.spl:

    curl -O https://<url_to_your_spl_file>
   

7. Install it:

    /opt/splunkforwarder/bin/splunk install app splunkclouduf.spl
   

8. Send data to Splunk Cloud:

    ./splunk add forward-server http-inputs-<your_organization>.splunkcloud.com:443
   

9. Navigate to the directory containing your inputs.conf and outputs.conf files:

    cd /opt/splunkforwarder/etc/system/local
   

10. Configure your inputs.conf file as needed. Here you have an example:

    #fix bug in ubuntu related to: "Events from tracker.log have not been seen for the last 90 seconds, which is more than the yellow threshold (45 seconds). This typically occurs when indexing or forwarding are falling behind or are blocked."
    [health_reporter]
    aggregate_ingestion_latency_health = 0
    
    [feature:ingestion_latency]
    alert.disabled = 1
    disabled = 1
    
    # Monitor system logs for authentication and authorization events
    [monitor:///var/log/auth.log]
    disabled = false
    index = <name_of_index>
    sourcetype = linux_secure
    
    # Monitor system logs for general security events
    [monitor:///var/log/syslog]
    disabled = false
    index = <name_of_index>
    sourcetype = linux_syslog
    
    # Monitor Apache access and error logs
    [monitor:///var/log/apache2/access.log]
    disabled = false
    index = <name_of_index>
    sourcetype = apache_access
    
    [monitor:///var/log/apache2/error.log]
    disabled = false
    index = <name_of_index>
    sourcetype = apache_error
    
    # Monitor SSH logs for login attempts
    [monitor:///var/log/auth.log]
    disabled = false
    index = <name_of_index>
    sourcetype = sshd
    
    # Monitor sudo commands executed by users
    [monitor:///var/log/auth.log]
    disabled = false
    index = <name_of_index>
    sourcetype = sudo
    
    # Monitor UFW firewall logs (assuming UFW is used)
    [monitor:///var/log/ufw.log]
    disabled = false
    index = <name_of_index>
    sourcetype = ufw
    
    # Monitor audit logs (if available)
    [monitor:///var/log/audit/audit.log]
    disabled = false
    index = <name_of_index>
    sourcetype = linux_audit
    
    # Monitor file integrity using auditd (if available)
    [monitor:///var/log/audit/auditd.log]
    disabled = false
    index = <name_of_index>
    sourcetype = auditd
    
    # Monitor for changes to critical system files
    [monitor:///etc/passwd]
    disabled = false
    index = <name_of_index>
    sourcetype = linux_config
    
    # Monitor for changes to critical system binaries
    [monitor:///bin]
    disabled = false
    index = <name_of_index>
    sourcetype = linux_config
    
    # Monitor for changes to critical system configuration files
    [monitor:///etc]
    disabled = false
    index = <name_of_index>
    sourcetype = linux_config
    

11. Configure your outputs.conf file with the following command:

    echo "[httpout]
    httpEventCollectorToken = <your_HEC_token>
    uri = https://http-inputs-<your_organization>.splunkcloud.com:443" > outputs.conf
    

12. Set the SPLUNK_HOME environment variable:

    export SPLUNK_HOME=/opt/splunkforwarder
    

13. Restart the Splunk service to apply the new configuration:

    /opt/splunkforwarder/bin/splunk restart
    

PS1: To add a new monitor on a file or folder, use the following command:

/opt/splunkforwarder/bin/splunk add monitor "<path to file or folder>"

(The path should be enclosed in quotes)

PS2: To change the index where you send logs, navigate to the directory:

cd /opt/splunkforwarder/etc/system/local

and add a line in each section of the inputs.conf file like this:

index=<name_of_index>

Conclusion: Streamline Your Log Management with Splunk Cloud

By following this guide, you can efficiently send logs to Splunk Cloud from various operating systems, ensuring that your log data is centralized, easily accessible, and ready for analysis. Proper log management and monitoring are crucial for maintaining the health, security, and compliance of your IT infrastructure. With Splunk Cloud and the Splunk Universal Forwarder, you can streamline these processes and gain valuable insights from your log data. If you have any questions or need further assistance, feel free to contact us. We'll be happy to help. Happy logging!