How to centralize logs with rsyslog logstash elasticsearch and kibana on Ubuntu 20.04

To install Elasticsearch on Ubuntu 20.04, follow these steps:

sudo apt update
sudo apt install openjdk-11-jre-headless
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo sh -c 'echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" > /etc/apt/sources.list.d/elastic-7.x.list'
sudo apt install elasticsearch
sudo service elasticsearch start
sudo systemctl enable elasticsearch

Test Elasticsearch:

Verify that Elasticsearch is up and running.

curl -X GET "localhost:9200/"

Install Logstash:

sudo apt update
sudo apt install openjdk-11-jre-headless
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo sh -c 'echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" > /etc/apt/sources.list.d/elastic-7.x.list'
sudo apt update
sudo apt install logstash

Now Install Rsyslog on Ubuntu

apt list -a rsyslog
sudo systemctl enable --now rsyslog

Configure rsyslog:

rsyslog configuration (/etc/rsyslog.conf or /etc/rsyslog.d/your-config-file.conf):

# Load the imuxsock module for local log reception
$ModLoad imuxsock

# Send messages to Logstash
*.* action(type="omfwd" target="logstash_server_ip" port="514" protocol="tcp")

Restart Services:

sudo service rsyslog restart

Configure Logstash for rsyslog:

Create a Logstash configuration file for rsyslog by creating a new file (e.g., /etc/logstash/conf.d/10-rsyslog.conf) with the following content:

input {
  tcp {
    port => 514
    type => syslog
  }
  udp {
    port => 514
    type => syslog
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "rsyslog-%{+YYYY.MM.dd}"
  }
}

Note: If the 514 port will not work or give an error use higher port 5514

input {
  tcp {
    port => 5514
    type => "syslog"
  }
}

Restart Logstash:

sudo service logstash restart

Install and Configure Kibana:

sudo apt update
sudo apt install kibana
sudo service kibana start

If you’re unsure about the correct service name, you can list available services:

sudo service --status-all

Check Systemd Status:

systemctl status kibana

If it’s not recognized, you might need to reload the systemd daemon or manually enable the service:

sudo systemctl daemon-reload
sudo systemctl enable kibana

Check Kibana Configuration:

Verify that Kibana is configured to listen on all network interfaces (0.0.0.0) so that it can accept connections from external machines. Open the Kibana configuration file (/etc/kibana/kibana.yml) and ensure that the server.host option is set to:

server.host: "0.0.0.0"

Verify Elasticsearch Connection:

Kibana requires a functional connection to Elasticsearch. Ensure that Elasticsearch is running and properly configured. Verify that the Elasticsearch URL is correctly set in the Kibana configuration file (/etc/kibana/kibana.yml):

elasticsearch.hosts: ["http://localhost:9200"]

After making changes, restart the Kibana service:

sudo service kibana restart

Access via Browser:

http://kibana-server-ip:5601

Remember that this is a basic setup to get you started. Depending on your requirements and environment, you might need to adjust configurations, add security, and handle high availability for the ELK stack.