How to setup Logging Service with ELK Stack (Elasticsearch, Logstash, and Kibana) on Ubuntu

Setting up the ELK Stack (Elasticsearch, Logstash, and Kibana) on Ubuntu involves several steps. Here’s a guide to help you through the process:

Prerequisites

• An Ubuntu server (20.04 or later recommended)
• A user with sudo privileges
• Java installed (required by Elasticsearch and Logstash)

Step 1: Install Java

First, ensure Java is installed on your system:

sudo apt update
sudo apt install openjdk-11-jdk -y

Verify the installation:

java -version

Step 2: Install Elasticsearch

1. Import the Elasticsearch PGP Key:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

2. Install the apt-transport-https package:

sudo apt install apt-transport-https -y

3. Save the repository definition to /etc/apt/sources.list.d/elastic-7.x.list:

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

4. Update your package list and install Elasticsearch:

sudo apt update
sudo apt install elasticsearch -y

5. Enable and start the Elasticsearch service:

sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch

Step 3: Install Logstash

1. Install Logstash:

sudo apt install logstash -y

2. Enable and start the Logstash service:

sudo systemctl enable logstash
sudo systemctl start logstash

Step 4: Install Kibana

1. Install Kibana:

sudo apt install kibana -y

2. Enable and start the Kibana service:

sudo systemctl enable kibana
sudo systemctl start kibana

Step 5: Configure the ELK Stack

1. Configure Elasticsearch (optional, if you need custom settings):

Edit the Elasticsearch configuration file:

sudo nano /etc/elasticsearch/elasticsearch.yml

2. Configure Logstash:

Create a configuration file for Logstash, for example:

sudo nano /etc/logstash/conf.d/logstash.conf

Here is a sample configuration that reads logs from a file and sends them to Elasticsearch:

input {
  file {
    path => "/var/log/syslog"
    start_position => "beginning"
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

3. Configure Kibana (optional, if you need custom settings):

Edit the Kibana configuration file:

sudo nano /etc/kibana/kibana.yml

Step 6: Access Kibana

1. Open your web browser and navigate to:

http://<your_server_ip>:5601

2. You should see the Kibana web interface.

Step 7: Verify the ELK Stack

1. Check Elasticsearch:

curl -X GET "localhost:9200"

2. Check Logstash:

Review Logstash logs to ensure it is processing logs correctly:

sudo tail -f /var/log/logstash/logstash-plain.log

3. Check Kibana:

Navigate through the Kibana web interface to verify it is receiving data from Elasticsearch.

Conclusion

You now have a working ELK Stack on Ubuntu. You can further customize your setup by adding different input sources in Logstash, creating visualizations and dashboards in Kibana, and tuning Elasticsearch for performance.