How to integrate collectd, Elasticsearch, Logstash, Kibana & OpenStack

Introduction

In this post I’m going to show how to integrate collectd (the system statistics collection daemon) with Elasticsearch (distributed, RESTful search and analytics engine) using Logstash (an open source, server-side data processing pipeline) and visualize in Kibana (Elasticsearch visualization tool) and demonstrate some of its use cases in OpenStack (a set of software tools for building and managing cloud computing platforms for public and private clouds).

There are other usful tools out there to collect system statistics information. Such Nagios, but I personally found collectd a good solution for my research purposes.

Assumptions

Let’s assume you already have a very basic knowledge of collectd and ELK stack. I will go through installing OpenStack in another post and needless to say, that you don’t need OpenStack in order to use collectd and/or ELK stack! I also assume you are using Ubuntu 16.04 server.

Install collectd on a Node

I need following modules

  • intel_rdt (it won’t work inside virtual machines)
  • libvirt (if you are using KVM/OpenStack)
  • lvm (if you are using LVM)
  • sensors (it won’t work inside virtual machines)
  • netlink
  • perl
  • exec
  • interface
  • logfile (for debugging purposes)
  • memory
  • cpu
  • network (in order to send collected data to Elasticsearch)
  • processes (in order to keep an eye of a specific process, if you want)
  • top (my written Perl plugin)

Now, we are going to run following commands as root in order to install collectd. To find out which packages need to be installed for your plugins of choice, check collectd’sGitHub page:

sudo apt-get update
sudo apt-get install git build-essential libyajl-dev libsensors4-dev multipath-tools libi2c-dev liblvm2-dev libperl-dev automake autoconf flex bison libtool pkg-config libmnl-dev lm-sensors 

git clone https://github.com/intel/intel-cmt-cat
cd intel-cmt-cat
make
make install

cd ..
git clone https://github.com/collectd/collectd/
cd collectd
./build.sh
./configure --enable-intel_rdt --enable-lvm --enable-netlink --enable-libvirt --enable-perl --enable-interface --enable-logfile --enable-memory --enable-exec --enable-sensors --enable-cpu --enable-network --enable-processes
make
make install

Also, we need to make Perl bindings. Inside the collectd main folder, run followings:

cd bindings/perl/
perl Makefile.PL
make
make install

Enable msr kernel module to use intel_rdt correctly:

modprobe msr

In order to sensors plugin work, please run following commands as root:

$ sensors-detect
   Some south bridges, CPUs or memory controllers contain embedded sensors. Do you want to scan for them? This is totally safe. (YES/no): YES
   Some Super I/O chips contain embedded sensors. We have to write to standard I/O ports to probe them. This is usually safe. Do you want to scan for Super I/O sensors?(YES/no): YES
   Some hardware monitoring chips are accessible through the ISA I/O ports. We have to write to arbitrary I/O ports to probe them. This is usually safe though. Yes, you do have ISA I/O ports even if you do not have any ISA slots! Do you want to scan the ISA I/O ports? (YES/no): YES
   Lastly, we can probe the I2C/SMBus adapters for connected hardware monitoring devices. This is the most risky part, and while it works reasonably well on most systems, it has been reported to cause trouble on some systems. Do you want to probe the I2C/SMBus adapters now? (YES/no): YES
   Do you want to add these lines automatically to /etc/modules? (yes/NO) yes

$ /etc/init.d/kmod start

You can also install my written plugin called “collectd-top” as follows:

cd ~/
git clone https://github.com/michelgokan/collectd-top
cd collectd-top
perl Makefile.PL
make
make install

Its time to configure collectd using /opt/collectd/etc/collectd.conf file. Uncomment following lines to enable mentioned plugins (consider set ValuesPercentage true in CPU plugin AND set our Elasticsearch node IP address in network plugin instead of {YOUR_ELASTICSEARCH_IP} placeholder AND set your host name instead of {YOUR_HOST_NAME} placeholder):

Hostname "{YOUR_HOST_NAME}"
FQDNLookup false
...
Interval 5
...
LoadPlugin intel_rdt
...
LoadPlugin interface
...
LoadPlugin lvm
...
LoadPlugin netlink
...
LoadPlugin sensors
...
LoadPlugin logfile
...
LoadPlugin memory
...
LoadPlugin cpu
...
LoadPlugin network
...
LoadPlugin processes
...
LoadPlugin perl
...
<Plugin logfile>
 LogLevel info
 File "/var/log/collectd.log"
 Timestamp true
 PrintSeverity true
</Plugin>
...
<Plugin "intel_rdt">
 Cores "0-2"
</Plugin>
...
<Plugin interface>
 Interface "eth0"
 IgnoreSelected false
 ReportInactive false
 UniqueName false
</Plugin>
...
<Plugin sensors>
 SensorConfigFile "/etc/sensors3.conf"
 Sensor "it8712-isa-0290/temperature-temp1"
 Sensor "it8712-isa-0290/fanspeed-fan3"
 Sensor "it8712-isa-0290/voltage-in8"
 IgnoreSelected false
</Plugin>
...
<Plugin netlink>
 Interface "All"
 VerboseInterface "All"
</Plugin>
...
<Plugin memory>
 ValuesAbsolute false
 ValuesPercentage true
</Plugin>
...
<Plugin cpu>
 ReportByCpu true
 ReportByState true
 ValuesPercentage true
 ReportNumCpu false
 ReportGuestState false
 SubtractGuestState false
</Plugin>
...
<Plugin network>
   <Server "{YOUR_ELASTICSEARCH_SERVER_IP" "25826">
   </Server>
</Plugin>
...
<Plugin perl>
 BaseName "Collectd::Plugins"
 LoadPlugin "Top"
 <Plugin "top">
    TopProcessesCountByMemory "4"
    TopProcessesCountByCPU "5"
 </Plugin>
</Plugin>

Find an example collectd.conf file here.

You have to restart collectd daemon:

sudo killall collectdmon 
/opt/collectd/sbin/collectd
/opt/collectd/sbin/collectdmon

Install ELK Stack

I use a separate server to install ELK stack. I suggest you do the same.

Elasticsearch Installation

Install Oracle Java8:

add-apt-repository -y ppa:webupd8team/java
apt-get update
apt-get -y install oracle-java8-installer

Add the followings to the end of /etc/environment file:

...
JAVA_HOME=/usr/lib/jvm/java-8-oracle
export JAVA_HOME

Set $JAVA_HOME : $ . /etc/environment

Grab the latest version of Elasticsearch deb file from here.Elasticsearch website

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.1.deb

dpkg -i elasticsearch-6.2.1.deb

/etc/init.d/elasticsearch restart

Test if Elasticsearch is running:

$ curl -XGET 'localhost:9200/?pretty'

{
  "name" : "QC_bk8u",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "_ka9bC5CQem7f0DE9XUI5g",
  "version" : {
    "number" : "6.2.1",
    "build_hash" : "7299dc3",
    "build_date" : "2018-02-07T19:34:26.990113Z",
    "build_snapshot" : false,
    "lucene_version" : "7.2.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Here you can find a nice tutorial on how to install Elasticsearch using a Debian package.

Install Kibana

Just like the previous section:

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.1-amd64.deb

dpkg -i kibana-6.2.1-amd64.deb

/etc/init.d/kibana restart

Install X-Pack

/usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack
/etc/init.d/elasticsearch restart
/etc/init.d/kibana restart
/usr/share/elasticsearch/bin/x-pack/setup-passwords auto

After running the last command above, you will probably get some output like below:

Initiating the setup of passwords for reserved users elastic,kibana,logstash_system.
The passwords will be randomly generated and printed to the console.
Please confirm that you would like to continue [y/N]y


Changed password for user kibana
PASSWORD kibana = etlPv88yBBHanx3Xwf3g

Changed password for user logstash_system
PASSWORD logstash_system = eb8N80JgnPKWdL6qYif4

Changed password for user elastic
PASSWORD elastic = 2nfW0Gq75PlBZPkqHgKg

If not, reboot the server and try again! Rebooting always works!

To install x-pack on Kibana, run the followings:

/usr/share/kibana/bin/kibana-plugin install x-pack

Add credentials and other related settings to the /etc/kibana.yml file:

server.port: 5601
server.host: "<server_ip>"
server.name: "<server_name>"

elasticsearch.username: "kibana"
elasticsearch.password:  "<pwd>"
elasticsearch.url: "http://<server_ip>:9200"

Where <pwd> is the password for the Kibana user from the previous section (in this case etlPv88yBBHanx3Xwf3g) and <server_ip> and <server_name> is the server’s IP address and server name (to enable remote access).

Also, add the followings to the /etc/elasticsearch/elasticsearch.yml :
network.host: <server_ip>
xpack.security.enabled: false

Now you can reboot Elasticsearch and Kibana:

/etc/init.d/elasticsearch restart
/etc/init.d/kibana restart

Check Kibana’s status using following URL:
http://<server_ip>:5601/status

Kibana dashboard

Install LogStash

Just like previous sections, but this time, first install jruby:

sudo apt-get install jruby
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.1.deb
dpkg -i logstash-6.2.1.deb
/usr/share/logstash/bin/logstash-plugin install logstash-codec-collectd
/etc/init.d/elasticsearch restart
/etc/init.d/kibana restart
Please review these basic ELK concepts before continuing.

Integrate collectd with Elasticsearch

Now that you installed everything you have to first run collectd on your source server. Run this command on your collectd instance:

/opt/collectd/sbin/collectd
/opt/collectd/sbin/collectdmon

In your Elasticsearch server, you have to create a LogStash pipeline to receive collectd data. First, create a pipeline configuration like below (i.e in /etc/logstash/conf.d/logstash.conf):

input {
  udp {
    port => 25826
    buffer_size => 1452
    codec => collectd { }
  }
}
output {
        elasticsearch {
                hosts => [ "{YOUR_ELASTICSEARCH_IP}:9200" ]
            index => "collectd_16_feb_17_32"
        }
}

You can run following to start your LogStash pipeline:

/usr/share/logstash/bin/logstash -f /path/to/collectd-pipeline.conf

In your Kibana, you have to create a default index pattern. So at first, login to your Kibana web interface > Management > Index Patterns > Create Index Pattern and use collectd_* as your index pattern.

In Discover section you can see your data is coming:

Collectd discovered data in Elasticsearch - Kibana using LogStash

Then I created a visualization like below (I add full resolution screenshot so that you can see the variables):

collectd visualization in Elasticsearch - Kibana using Logstash

OpenStack Integration

Now that you know how to create visualizations, you may have multiple servers running  (hosts, controller, and VMs) and with a single Elasticsearch installation, you can receive all their data. You can create a sophisticated “Dashboard” to see everything in one place:

Final Word

Hopefully, this post will help you in your work and research but don’t hesitate to ask your questions via the comments sections below and I will gladly help as much as I can. I will improve this post time by time, so don’t hesitate to mention mistakes in this so I will fix them. Also please let me know if you want me to write about some specific topic.

(Visited 1,087 times, 4 visits today)