• Icinga2 Tutorial: Part 0 - Network Monitoring for the Masses
• Icinga2 Tutorial: Part 1 - Installation and Configuration
• Icinga2 Tutorial: Part 2 - Agent-Less Checks
• Icinga2 Tutorial: Part 3 - Agent-Based Checks
• Icinga2 Tutorial: Part 4 - Expanding Checks to SNMP (You are here.)

EDIT (2018/12/09): These guides haven’t been updated since 2015. It is possible that there are dead links, or that the configuration syntax has changed dramatically. These posts are also some of the most popular on my blog. I plan to do a new guide eventually, but for right now please take the following entries with a grain of salt.

Introduction

Well I have finally persuaded myself to continue writing these posts by completely deleting all the configuration I had already set up. It is worth noting that I have switched over to Debian Jessie, for no other reason than to cause myself more frustration and suffering. Anyways, let’s get started.

SNMP is considered an Agent-Based Check, and is actually quite flexible. You can even go as far as to code in custom return options, to check things you normally wouldn’t be able to check over snmp, for example, apt status, and other such things.

It is worth noting that due to using a very small LAN, I will not be fiddling around with SNMPv3, I will be going with straight SNMPv1, just with a modified community string. We will get started with my core router, Djehuti. It is outside the scope of this tutorial to discuss how to enable SNMP on your device, but if you use a Ubiquiti device, hey that might come soon.

Starting from this post forward, I will be embedding code here instead of referring to an external link, as embedding will encourage me to be a bit more complete in my explanations. So, with all of that said, let’s get started.

Initial Setup

To monitor SNMP we will be using the Manubulon SNMP Plugins. So we first need to install them.

sudo apt-get install nagios-snmp-plugins

Now we need to open up the main Icinga2 Configuration file and add in the proper include to allow us to use these plugins. You may notice while poking around this file that there are many things you either don’t need or would like to change. I do plan to come back to this file at a later time, but feel free to edit this file before that happens. Once you have made the proper changes, restart Icinga2 so the new settings take effect.

sudo vim /etc/icinga2/icinga2.conf

    include <manubulon>

sudo service icinga2 restart

With that, we can move on to creating configuration files!

Router

We will be starting with my core router, which is running SNMPv1. The first thing we will want to do is to add some essential variables to our host directive so that we don’t have to redefine them with every service.

//
// Host Declaration Block
//
object Host "router.example.org" {
    // Define the host IPv4 Address
    address             = "10.0.0.1"
    // Define a basic functionality test
    // Hostalive does a basic ICMP ECHO to the target
    // specified in the address directive.
    check_command       = "hostalive"
    // Define SNMP Variables
    vars.snmp_address   = "10.0.0.1"
    vars.snmp_community = "example"
    // These are not strictly needed. I add them
    // so I know at a glance what version of snmp
    // I am using.
    vars.snmp_v2        = "false"
    vars.snmp_v3        = "false"
}

The new additions are any of the var.snmp* commands located under the check_command line. With our host variables set up, we can now move on to defining a service. The first service defined in the Icinga2 Manubulon Documentation is the snmp-load check. Seems like a good starting place to me!

SNMP-Load

//    
// Service Declaration Block
// Service:     snmp_load
// Description: Uses SNMP commands to check the load averages
//              on the device.
//
object Service "snmp-load" {
    host_name           = "router.example.org"
    // Set the type of load check to use.
    vars.snmp_load_type = "netsl"
    // Set the Load Average warning threshold.
    vars.snmp_warn      = "5,3,2"
    // Set the Load Average critical threshold.
    vars.snmp_crit      = "6,5,3"
    check_command       = "snmp-load"
}

I feel I should take a minute to explain the warning and critical variables, because the icinga2 documentation doesn’t do a very good job. When checking load averages on *nix systems, there are three parameters:

• Average Load over one minute
• Average Load over five Minutes
• Average Load over fifteen minutes

Since my router is a dual core device, I have set it up so that if the system is at full load for 15 minutes, I get a warning. If it has one process over full load for five minutes, I get a warning. If it is three processes over full load in one minute, I want a warning. Same thing applies to critical. If you are trying to figure out what to set your levels at, I tend to use the following formulas:

• Warning:
  • 1min: 2*\(Number of Cores\)+1
  • 5min: \(Number of Cores\)+1
  • 15min: \(Number of Cores\)
• Critical:
  • 1min: 3*\(Number of Cores\)
  • 5min: 2*\(Number of Cores\)+1
  • 15min: \(Number of Cores\)+1

Once you have your file saved, restart Icinga2, and check the web interface. Your new check will likely have an Unknown Status in purple, just click on the check, and manually run it by clicking “Check Now” in the right most panel.

With that, we can move on to the next check!

SNMP-Memory

//
// Service Declaration Block
// Service:     SNMP-Memory
// Description: Uses SNMP commands to check status of RAM
//              and swap on the device.
//
object Service "snmp-memory" {
    host_name      = "router.example.org"
    // Set the Memory warning for Ram and swap Respectively.
    // Uses percents.
    vars.snmp_warn = "50,0"
    vars.snmp_crit = "80,0"
    check_command  = "snmp-memory"
}

The warning and critical values are expressed as percentages of the total amount of their applicable setting. The first one applies to RAM and the second value corresponds to swap. Restart Icinga2 and log on to the web interface to check that the new service works.

SNMP-Storage

//
// Service Declaration Block
// Service:     SNMP-Storage
// Description: Uses SNMP commands to check the status of disk
//              storage space.
//
object Service "snmp-storage" {
    host_name              = "router.example.org"
    // Uses percents.
    vars.snmp_warn         = "50"
    vars.snmp_crit         = "80"
    // Specify which partition to monitor
    vars.snmp_storage_name = "/root.dev"
    check_command          = "snmp-storage"
}

The snmp_storage_name variable is used to specify which device you want to check the status of. If you aren’t sure which device you need to check, set it to blank, then let it run. It will return a list of partitions that you can check. Simply enter the name into that variable and you are good to go.

Just as memory, snmp-storage uses percent values in the warning and critical threshold variables.

SNMP-Interfaces

I personally like to specify a different service block for each interface that I am monitoring, so I am not sure if it is possible to mix interfaces together, but I don’t see any reason it wouldn’t be possible. I’m going to list the interface configurations below, and if any variables need to be explained I will do that below the code.

//
// Service Declaration Block
// Service:     SNMP-Interface
// Description: Uses SNMP commands to check the status of
//              various network interfaces on device.
//
object Service "snmp-int-lan" {
    host_name                      = "router.example.org"
    // Define interface variables.
    vars.snmp_interface            = "eth0"
    vars.snmp_interface_label      = "LAN"
    vars.snmp_interface_perf       = "true"
    vars.snmp_interface_bits_bytes = "true"
    vars.snmp_interface_megabytes  = "true"
    vars.snmp_interface_noregexp   = "true"
    vars.snmp_warncrit_percent     = "true"
    // Set warning and crits to 100 to disable.
    vars.snmp_warn                 = "100,100"
    vars.snmp_crit                 = "100,100"
    check_command                  = "snmp-interface"
}

//
// Service Declaration Block
// Service:     SNMP-Interface
// Description: Uses SNMP commands to check the status of
//              various network interfaces on device.
//
object Service "snmp-int-wan" {
    host_name                      = "router.example.org"
    // Define interface variables.
    vars.snmp_interface            = "eth1"
    vars.snmp_interface_label      = "WAN"
    vars.snmp_interface_perf       = "true"
    vars.snmp_interface_bits_bytes = "true"
    vars.snmp_interface_megabytes  = "true"
    vars.snmp_interface_noregexp   = "true"
    vars.snmp_warncrit_percent     = "true"
    // Set warning and crits to 100 to disable.
    vars.snmp_warn                 = "100,100"
    vars.snmp_crit                 = "100,100"
    check_command                  = "snmp-interface"
}

//
// Service Declaration Block
// Service:     SNMP-Interface
// Description: Uses SNMP commands to check the status of
//              various network interfaces on device.
//
object Service "snmp-int-dmz" {
    host_name                      = "router.example.org"
    // Define interface variables.
    vars.snmp_interface            = "eth2"
    vars.snmp_interface_label      = "DMZ"
    vars.snmp_interface_perf       = "true"
    vars.snmp_interface_bits_bytes = "true"
    vars.snmp_interface_megabytes  = "true"
    vars.snmp_interface_noregexp   = "true"
    vars.snmp_warncrit_percent     = "true"
    // Set warning and crits to 100 to disable.
    vars.snmp_warn                 = "100,100"
    vars.snmp_crit                 = "100,100"
    check_command                  = "snmp-interface"
}

So a few things in here need some explanation. The variable vars.snmp_interface specifies which interface we will be checking. vars.snmp_interface_noregexp is related to this in that it tells icinga2 to not use regex matching. vars.snmp_interface_label configures a label that will be shown in the console. vars.snmp_interface_megabytes, and vars.snmp_interface_bits_bytes tells Icinga2 that we want to see bandwidth measured in megabits. These variables can be adjusted accordingly. Finally, vars.snmp_interface_perf tells Icinga2 that we want to monitor bandwidth usage.

As for warning and critical values, while I like to monitor my bandwidth, I don’t actually care how high it goes, at least not at the moment. More relevant than that is the fact that my bandwidth is much less than a gigabit, but let’s move on from that. vars.snmp_warncrit_percent says that we are going to specify our warning and critical thresholds as a percent of total available bandwidth on that port. I then set vars.snmp_warn, and vars.snmp_crit to 100 so that it is effectively disabled.

Once activating these services, you should reset Icinga2. It is worth noting that you will first get a pending, and then an unknown status for about five minutes, depending on your check time. Icinga compares the newest reading to a previous reading that is sufficently old enough, which is usually about five minutes, to calculate what has changed. Until you have a row in the database that is the proper age, you will get a big Unknown status. Nothing to worry about, check back in a half hour.

Conclusion

There is one more snmp check that is available, and that is the process check. While I previously used this setup on my core router, it ended up causing some rather wonky effects, so I have elected to not use it. This check would be useful to monitor the status of a mission critical process, such as a webserver or even a database server. It works by searching the process list for the number of times a string appears, and then going from there. I may cover this in the future, but I won’t be at the moment.

Thank you for reading, and I hope to have part five up with less of a lag time. I am also planning to do Icinga2 integration with slack soon, so stay tuned for that!

• Icinga2 Tutorial: Part 0 - Network Monitoring for the Masses
• Icinga2 Tutorial: Part 1 - Installation and Configuration
• Icinga2 Tutorial: Part 3 - Agent-Based Checks (You are here.)
• Icinga2 Tutorial: Part 2 - Agent-Less Checks
• Icinga2 Tutorial: Part 4 - Extending Checks to SNMP

EDIT (2018/12/09): These guides haven’t been updated since 2015. It is possible that there are dead links, or that the configuration syntax has changed dramatically. These posts are also some of the most popular on my blog. I plan to do a new guide eventually, but for right now please take the following entries with a grain of salt.

For this part of the tutorial, we are actually going to be rewriting the host file for localhost. Unfortunately, I thought the EdgeRouter would be capable of running an Icinga2 client, but it isn’t available through the apt repositories, and after the ntopng incident of July and August, I am not stressing that processor any more than it needs to be to run my network. (It is stressed enough as it is, but that is a post for another day.)

Since I have done most of the explaining in the files, these tutorials will start including the configuration files, and any external resources I have used.

Configuration File

/*
 * File:        /etc/icinga2/conf.d/captor.conf
 * Title:       Captor Host Configuration File
 * Description: Host and services definition for Icinga2 for
 *              the Captor Localhost.
 * Host:        captor.zyradyl.org
 * System:      Linux Mint Debian Edition 2
 * License:     Creative Commons Zero - Public Domain
 * Version:     1.0
 * Date:        August 16, 2015
 */

//
// Host Declaration Block
//
object Host "boxen.example.org" {
    // Define the host IPv4 Address
    address         = "127.0.0.1"
    // Define the host IPv6 Address
    address6        = "::1"
    // Define the operating system
    vars.os         = "Linux"
    // Define a basic functionality test
    // Hostalive does a basic ICMP ECHO to the target
    // specified in the address directive.
    check_command   = "hostalive"
}

//
// There are two ways that we can configure Icinga2.
// The first method is to write specific files for
// every host, which is a good idea if you only have
// a few hosts, and every host has something
// different going on. The other method wold be to
// use apply service logic, which is better for
// larger deployments. I have provided links to
// both a best practices guide as well as to a guide
// on apply logic. I will be using files on a per
// host basis for my deployment.
//

// For demonstration purposes I am setting up an
// IPv6 Ping check.

//
// Service Declaration Block
// Service:     Ping6
// Description: Check if Host is responding to IPv6
// Note:        The address6 specified in the above directive  
//              is carried into the service declaration
//              blocks via specifying the host_name variable
//
object Service "ping6" {
    host_name     = "boxen.example.org"
    check_command = "ping6"
}

//
// Service Declaration Block
// Service:     HTTP.server
// Description: Checks if Apache in general is up.
// Note:        The address specified in the above directive  
//              is carried into the service declaration
//              blocks via specifying the host_name variable
//
// We are specifying an additional part on the service
// because we are also going to check to make sure that
// IcingaWeb2 is running.
//
object Service "http.server" {
    host_name               = "boxen.example.org"
    // This tells Icinga to check only this specific
    // page.
    vars.http_uri           = "/"
    check_command           = "http"
}

//
// Service Declaration Block
// Service:     HTTP.icingaweb2
// Description: Check if the IcingaWeb service is up.
// Note:        The address specified in the above directive  
//              is carried into the service declaration
//              blocks via specifying the host_name variable
//
object Service "http.icingaweb" {
    host_name               = "boxen.example.org"
    // This tells Icinga to check only this specific
    // page.
    vars.http_uri           = "/icingaweb2"
    check_command           = "http"
}

//
// Service Declaration Block
// Service:     SSL Cert Check
// Description: Check expiration date of SSL certificates.
// Note:        The address specified in the above directive  
//              is carried into the service declaration
//              blocks via specifying the host_name variable
//
// The following is commented out because boxen doesnt offer
// HTTPS.
//
//object Service "ssl" {
//    host_name                         = "boxen.example.org"
//    vars.ssl_port                     = "443"
//    vars.ssl_cert_valid_days_warn     = "30"
//    vars.ssl_cert_valid_days_critical = "15"
//    check_command                     = "ssl"
//}

//
// Service Declaration Block
// Service:     Disks
// Description: Checks status of all installed disks.
// Note:        The address specified in the above directive  
//              is carried into the service declaration
//              blocks via specifying the host_name variable
//
object Service "disks" {
    host_name                     = "boxen.example.org"
    // We need to exclude a particular partition due to
    // access denials.
    vars.disk_partitions_excluded = "/run/user/1000/gvfs"
    check_command                 = "disk"
}

//
// Service Declaration Block
// Service:     Icinga
// Description: Checks status of Icinga Instance.
// Note:        The address specified in the above directive  
//              is carried into the service declaration
//              blocks via specifying the host_name variable
//
object Service "icinga" {
    host_name     = "boxen.example.org"
    check_command = "icinga"
}

// Service Declaration Block
// Service:     Load
// Description: Checks how much load the host is under.
// Note:        The address specified in the above directive  
//              is carried into the service declaration
//              blocks via specifying the host_name variable
//
object Service "load" {
    host_name     = "boxen.example.org"
    check_command = "load"
}

// Service Declaration Block
// Service:     Procs
// Description: Checks number of processes on host.
// Note:        The address specified in the above directive  
//              is carried into the service declaration
//              blocks via specifying the host_name variable
//
object Service "procs" {
    host_name     = "boxen.example.org"
    check_command = "procs"
}

//
// Service Declaration Block
// Service:     SSH
// Description: Checks if SSH is available.
// Note:        The address specified in the above directive  
//              is carried into the service declaration
//              blocks via specifying the host_name variable
//
// Note:        Captor does not provide ssh services.
//              Commenting out this service.
//
//object Service "ssh" {
//    host_name     = "boxen.example.org"
//    check_command = "ssh"
//}

// Service Declaration Block
// Service:     Swap
// Description: Checks status of swap space on host.
// Note:        The address specified in the above directive  
//              is carried into the service declaration
//              blocks via specifying the host_name variable
//
object Service "swap" {
    host_name     = "boxen.example.org"
    check_command = "swap"
}

// Service Declaration Block
// Service:     Users
// Description: Checks number of users on the system
// Note:        The address specified in the above directive  
//              is carried into the service declaration
//              blocks via specifying the host_name variable
//
object Service "users" {
    host_name     = "boxen.example.org"
    check_command = "users"
}

// Service Declaration Block
// Service:     Apt
// Description: Checks status of the apt package manager.
// Note:        The address specified in the above directive  
//              is carried into the service declaration
//              blocks via specifying the host_name variable
//
object Service "apt" {
    host_name     = "boxen.example.org"
    check_command = "apt"
}

// EOF

Import Notes

It is worth noting that this article was intended to be a quick write up to get something online. In time, I may come back to rewrite things, but at the very least I hope the comments in the file are helpful.

Problem Description

Upon booting the EdgeRouter Lite, the system will not boot. Connecting to console reveals the following log:

SQUASHFS error: zlib_inflate error, data probably corrupt
SQUASHFS error: squashfs_read_data failed to read block 0x1b4574
SQUASHFS error: Unable to read fragment cache entry [1b4574]
SQUASHFS error: Unable to read page, block 1b4574, size 8519
SQUASHFS error: Unable to read fragment cache entry [1b4574]
SQUASHFS error: Unable to read page, block 1b4574, size 8519

This can be caused by an unclean shutdown, which could potentially occur with a power failure. Due to the failure, parts of the internal ext3 filesystem become corrupted, which causes the inability to load the SquashFS partition which contains EdgeOS. The end result, of course, is a non-functional unit.

Problem Severity

There are four potential levels to determining the severity of this issue:

• Can the filesystem be salvaged by fscking the ext3 partition until it is salvageable? (Severity: Annoying.)
• If the answer to the above is no, can the flash drive be saved by zeroing out the device and writing a new filesystem to it? (Severity: Hardware issue - End user repair possible..)
• If the answer to the above is no, is the device capable of booting? If yes, it is possible you may be suffering from the EdgeRouter Lite RAM issue. ( Severity: Hardware issue - RMA the device.)
• If the answer to the above is no, is the device under warranty? If yes, (Severity: Hardware issue - RMA the device.) If no, (Severity: Device Replacement.)

Problem Resolution

Resolutions will be broken down into two posts, one for each problem the user could resolve without warranty work.

Resolution Credits

A special thanks to Ubiquiti for providing a community support option via their official website. Additionally, special thanks to all those users that utilize said forum, your information was invaluable in helping me to fix my EdgeRouter, and write this article.

Resolution One: Repair the File System

This is by far the easiest resolution. Open the EdgeRouter Lite (Caution: Your warranty is now void. Depending on how soon you need the device back in operation, it may be acceptable for you to void the warranty on a $100USD device if you can get it back in service by the end of the day. If you think your device may need to be RMA’d, do not open the device and proceed to UBNT’s Support Site.) by removing two screws on the bottom of the device, located right under where the interface connections are. Your device will slide apart, revealing the internal board. On the board you will see a soldered on female USB port with a small silver USB flash drive, roughly 3cm long. This is your 2GB of internal memory, but the device itself is 2.6GB.

Remove the USB drive and plug it into a USB port on your Mac OSX or Linux powered device. Open a terminal, and run dmesg. You are looking for the devfs name of your device.

[  262.645401] scsi 2:0:0:0: Direct-Access                               5.00 PQ: 0 ANSI: 2
[  262.647109] sd 2:0:0:0: Attached scsi generic sg1 type 0
[  262.648527] sd 2:0:0:0: [sdb] 4057088 512-byte logical blocks: (2.07 GB/1.93 GiB)
[  262.648795] sd 2:0:0:0: [sdb] Write Protect is off
[  262.648817] sd 2:0:0:0: [sdb] Mode Sense: 0b 00 00 08
[  262.649095] sd 2:0:0:0: [sdb] No Caching mode page found
[  262.649114] sd 2:0:0:0: [sdb] Assuming drive cache: write through
[  262.652309]  sdb: sdb1
[  262.655018] sd 2:0:0:0: [sdb] Attached SCSI removable disk

In this case, my device name is sdb. Please note that for this guide I am not using an actual EdgeRouter flash drive. This is a stand-in device. The next thing to do is make sure none of the partitions on the device have been automatically mounted. Where I used sdb in the following example, use whatever dmesg identified as your USB device.

mount | grep sdb

/dev/sdb1 on /media/bakayuki/b7bcf200-26a1-41ed-9122-625558dbc907 type ext4 (rw,nosuid,nodev,relatime,data=ordered,uhelper=udisks2)

This would indicate that at least one of the partitions on the disk has been mounted. To correct this we need to use the umount command. Note that in some operating systems an eject button exists in the taskbar. In my experience, not only do these buttons unmount the mounted partitions, it also removed the device entry from the kernel. So I prefer to use manual umount commands. In the following example, do one line per mounted partition, substituting the device names of your partitions where I specified /dev/sdb1.

sudo umount /dev/sdb1

Once you have unmounted the devices, you can use fdisk to get a good view of the partition layout on your USB device. I like to do this even if I know what I should expect, just as a form of a sanity test to make sure I have the right device. Note that, as previously stated, this device is a stand in. I have tried to recreate the partition table to the best of my memory.

sudo fdisk -l /dev/sdb

Disk /dev/sdb: 2 GiB, 2077229056 bytes, 4057088 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd9dd6356

Device     Boot  Start     End Sectors  Size Id Type
/dev/sdb1         2048  264191  262144  128M  b W95 FAT32
/dev/sdb2       264192 4057087 3792896  1.8G 83 Linux

So we can see from the above that we have two partitions on the USB device. In your case, when using an actual EdgeRouter flash drive, one of the partitions will be a vfat partition type, and the second one will be a linux partition type. Now that we know where our partitions are, and that they are safely unmounted, we can use fsck. Remember to replace my instance of sdb2 with the correct partition for your device. Note that depending on the damage to the filesystem, this has the potential to be a destructive operation.

fsck.ext3 -fvy /dev/sdb2

Allow the program to run till completion. If you notice the command has become looped, you will need to clear the journal out on your device. Note that this is a potentially destructive operation to data that had not been written from the journal to the disk. The first thing that we need to do is clear out the recovery indicator using debugfs Remember to replace sdb2 with the proper device name of your ext3 partition.

debugfs -w -R “feature ^needs_recovery” /dev/sdb2

After clearing out this flag, it is now possible to use tune2fs to force removal of the journal.

tune2fs -f -O ^has_journal /dev/sdb2

Now you can go back up and run the fsck command. Once that is done, you will need to re-enable the journal on your filesystem.

tune2fs -f -O has_journal -j size=128M /dev/sdb2

Once you have a clean fsck output, and you have your journal enabled again, plug the USB device back into the EdgeRouter and boot. If you are lucky, everything will come back as it should. As there may be data loss, make sure to restore from a recent configuration backup (you do have configuration backups, right?) and reboot, before logging into the device through ssh to make sure everything is where it should be.

If you are still having problems, stick around for part two of this guide, where I will be walking you through using a separate host computer to recreate the filesystem.

External Links & Resources

How to Recover an EXT3 Volume with an Unreadable Journal

• Icinga2 Tutorial: Part 0 - Network Monitoring for the Masses
• Icinga2 Tutorial: Part 1 - Installation and Configuration
• Icinga2 Tutorial: Part 2 - Agent-Less Checks (You are here.)
• Icinga2 Tutorial: Part 3 - Agent-Based Checks
• Icinga2 Tutorial: Part 4 - Extending Checks to SNMP

EDIT (2018/12/09): These guides haven’t been updated since 2015. It is possible that there are dead links, or that the configuration syntax has changed dramatically. These posts are also some of the most popular on my blog. I plan to do a new guide eventually, but for right now please take the following entries with a grain of salt.

Master Host Configuration

So in our last part we focused on getting your machine set up as the Icinga2 master controller. Now we can focus on getting interoperability setup. As always, this tutorial assumes you are sudo’d as root. You can do this by running sudo -s. Then we need to set ourselves up as the master node on our network.

icinga2 node wizard

Welcome to the Icinga 2 Setup Wizard!

We'll guide you through all required configuration details.

Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: n
Starting the Master setup routine…
Please specifiy the common name (CN) [boxen.example.org]:
information/cli: Generating new CSR in '/etc/icinga2/pki/boxen.example.org.csr'.
information/cli: Created backup file '/etc/icinga2/pki/boxen.example.org.key.orig'.
information/cli: Created backup file '/etc/icinga2/pki/boxen.example.org.csr.orig'.
information/base: Writing private key to '/etc/icinga2/pki/boxen.example.org.key'.
information/base: Writing certificate signing request to '/etc/icinga2/pki/boxen.example.org.csr'.
information/cli: Signing CSR with CA and writing certificate to '/etc/icinga2/pki/boxen.example.org.crt'.
information/cli: Created backup file '/etc/icinga2/pki/boxen.example.org.crt.orig'.
information/cli: Copying CA certificate to '/etc/icinga2/pki/ca.crt'.
information/cli: Created backup file '/etc/icinga2/pki/ca.crt.orig'.
information/cli: Dumping config items to file '/etc/icinga2/zones.conf'.
Please specify the API bind host/port (optional):
Bind Host []:
Bind Port []:
information/cli: Enabling the APIlistener feature.
information/cli: Updating constants.conf.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
Done.
Now restart your Icinga 2 daemon to finish the installation!

service icinga2 restart

checking Icinga2 configuration.
[...]

Note that in your case, you may see one or two warning messages, but if it pertains to files already existing, don’t worry about it, I also clipped all the output that Icinga2 spits out when it restarts in order to save space. These posts are long enough as it is. After restarting, if you check the website for your master node, you will see a whole bunch of new information. This has also established the certificates the icinga2 protocol uses for security.

Host Monitoring

Now, while it is nice to have access to the Icinga protocol, in our case we will be working with devices that do not make the option of installing Icinga possible. Instead, we will be monitoring through four different protocols: SSH, SNMP(v1), HTTP/S, and ICMP.

We are going to go simple in this route through the use of agent-less checks. Agent-less checks do not rely on having a remote program installed, and this is useful for embedded devices that may not have enough memory to host another program. This is also helpful if your client only offers one or two services and it isn’t worth taking the time to install and configure a node setup. For example, you can check if SSH is available on a remote host, or check if the HTTP server is alive, or even simply see if the host is alive. We will start there.

Before we get into editing the actual files, Syntax highlighting can make life a whole lot easier. Note, you should repeat this process with a non-privileged user as that will give you a way to take a look at icinga2 configuration files without having to be the root user.

mkdir -p ~/.vim/{syntax,ftdetect}
cd /usr/share/icinga2-common/syntax/
cp vim/syntax/icinga2.vim ~/.vim/syntax/
cp vim/ftdetect/icinga2.vim ~/.vim/ftdetect/

Now test it by opening any file under the /etc/icinga2/conf.d/ directory. You can find instructions to do this for nano here. Now, there are a lot of ways to configure Icinga2. You can choose to use the pre-existing hosts.conf and services.conf, or, since the conf.d directory is read on startup, you can use one file per host. I will be doing the latter, as I think it keeps it a lot neater. The Access point is named Wepwawet, so let’s get started. (Note that you clearly do not need to use my header method. You can also indent with tabs, I mean.. if you like that kind of uncertainty in how the file will be displayed.)

vim /etc/icinga2/conf.d/wepwawet.conf

You can find the document, with syntax highlighting, here.

So with this basic configuration, we get three checks:

• An ICMP Echo (hostalive)
• A HTTPS Up Check (http)
• A SSH Up check (ssh)

For my HTTP check, the website uses HTTPS, and no HTTP site is available. So by setting the SSL variable, we can ensure that just HTTPS is being checked. This ensures we are getting an accurate reading. Now we need to ensure that our declaration works.

/etc/init.d/icinga2 checkconfig
/etc/init.d/icinga2 reload

Then log in to your web page, and you should see your new host and service definition. If this works, then move on to the next section. If not, you should consult the error messages printed out as well as the Icinga2 log that can be found in /var/log. A useful method to check if the error is in your configuration is to run the checks manually. Attempt to ping the host, or ssh to the host, or access a web page. If you are unable to do these things manually, the problem may very well be with your host.

The one final thing I would like to do is to demonstrate that you can actually do a fairly nice amount of checks without needing to have a remote agent. For this next example, we are going to setup a check that will verify if our SSL certificate is still valid, or how long we have till it runs out. Previously, this was done through an external custom command but Icinga2 now has a TCP check plugin that can do this natively.

Open your configuration file and make the edits. You can follow this file with syntax colouring as an example.

Once done, reload your configuration as demonstrated above. It is worth stating that the reload parameter also calls checkconfig, however I prefer to run them as two separate commands. It is also worth stating that you could combine the SSL certificate check with the HTTPS check. I separate them because for me a certificate expiring isn’t too big of a deal – it will only throw an error – but the HTTPS server going down is a much bigger deal.

You can see the configuration of the monitoring setup for my core router here.

External Resources

• Icinga2 Monitoring Remote Systems: Agentless Checks
• Icinga2 Monitoring Basics: Hosts and Services
• Icinga2 Plugin Check Commands: SSH
• Icinga2 Plugin Check Commands: SSL
• Icinga2 Monitoring Basics: Command Passing Variables
• Icinga2 Configuration Validation

• Icinga2 Tutorial: Part 0 - Network Monitoring for the Masses
• Icinga2 Tutorial: Part 1 - Installation and Configuration (You are here.)
• Icinga2 Tutorial: Part 2 - Agent-less Checks
• Icinga2 Tutorial: Part 3 - Agent-Based Checks
• Icinga2 Tutorial: Part 4 - Extending Checks to SNMP

EDIT (2018/12/09): These guides haven’t been updated since 2015. It is possible that there are dead links, or that the configuration syntax has changed dramatically. These posts are also some of the most popular on my blog. I plan to do a new guide eventually, but for right now please take the following entries with a grain of salt.

Introduction

I wanted to get this out fairly quickly, because I just actually did this, and while the default Icinga2 tutorial is pretty good, it is lacking in some areas, and since everything is fresh in my mind I wanted to go ahead and draft this up.

It is worth noting that in this post I will assume you are root. You can get to root as a sustained environment by using sudo -s.

I do not condone this for day to day usage, it is bloody dangerous.

So, let’s get started. Sudo to root.

sudo -s

Installing Icinga2

This assumes you are root, and on a Debian based system. For other systems, you can find additional documentation here. Note that I follow that documentation very closely except for a few small notes, so it should be easy to adjust this tutorial for your needs.

Adding Repositories

wget -O - http://debmon.org/debmon/repo.key 2>/dev/null | apt-key add -
echo deb http://debmon.org/debmon debmon-jessie main >/etc/apt/sources.list.d/debmon.list
apt-get update

The first command downloads the cryptographic key for the debmon repository and hands it over to apt. Apt then installs that key, which allows you to verify that you are using packages signed off by and validated by that repository. The second command is then echoing the debmon main repository into your apt-sources, so that the software available there is added to your system’s list, which allows you to install with just a command. The third command tells your system to update its internal package list to incorporate the changes. After that we can go ahead and install.

Installation

apt-get install icinga2 monitoring-plugins

This will install both icinga2 and the nagios monitoring plugins that are compatible. This gives you a very good base. Start the program through the service command, and then set it to start on boot with update-rc.d.

service icinga2 start
update-rc.d icinga2 enable

Tada! You now have a data aggregator/network monitor setup! However, we want a nice way to get information from our monitoring host, so now we need to install the web front end.

Installing IcingaWeb2

Let’s start with all the essentials.

Package Installation

apt-get install postgresql icinga2-ido-pgsql apache2 php5-ldap php5-imagick php5-pgsql php5-intl icingaweb2 icingaweb2-module-doc icingaweb2-module-monitoring icingaweb2-module-setup

This is a big one, so if you are living on about a meg down, prepare to settle in for a bit. In order, this will install the PostgreSQL database, the Icinga2 database connector, the apache2 web server, the ldap php5 module, the imagemagick module for php5, the postgresql php5 module, the intl php5 module, the IcingaWeb2 core, the IcingaWeb2 documentation module, the IcingaWeb2 monitoring module, and the IcingaWeb2 setup module. Note that on debian systems, it will prompt you to allow it to setup the database. I strongly recommend you say NO to that, and do it manually.

Once this completes, get everything running and added to the default runlevel.

service postgresql start
update-rc.d postgresql enable
service apache2 start
update-rc.d apache2 enable

Database Configuration

Now we need to make that database. The first command defines a new user named “icinga” with a password of the same name. The second command then creates a database named icinga, and grants ownership to the user icinga.

cd /tmp
sudo -u postgres psql -c "CREATE ROLE icinga WITH LOGIN PASSWORD 'icinga';"
sudo -u postgres createdb -O icinga -E UTF8 icinga

Once this is done, edit the file pg_hba.conf. This file controls the way that hosts can authenticate with your database. We want to make sure that icinga can utilize standard password login, or md5.

vim /etc/postgresql/9.4/main/pg_hba.conf

    # icinga
    local    icinga    icinga                    md5
    host     icinga    icinga    127.0.0.1/32    md5
    host     icinga    icinga    ::1/128         md5

Save, and then restart the database to activate the changes. Once restarted, we can import the IDO SQL schema into Postgres. The export line pushes the password into the environment, thus allowing us to avoid having to type it in.

service postgresql restart
export PGPASSWORD=icinga
psql -U icinga -d icinga < /usr/share/icinga2-ido-pgsql/schema/pgsql.sql

Now we need to go ahead and enable the features that icinga2 will use.

icinga2 feature enable command
icinga2 feature enable ido-pgsql
icinga2 feature enable livestatus
icinga2 feature enable statusdata
service icinga2 restart

The final line restarts icinga2, which is the final step to enable these modules. Now, we need to update the configuration file for ido-pgsql to connect it to our database that we manually set up above.

vim /etc/icinga2/features-enabled/ido-pgsql.conf

Fill in the needed information as appropriate. Now you need to update the date.timezone variable in the php configuration, which can be found below.

vim /etc/php5/apache2/php.ini

As an example, mine is set to “America/Detroit”. Now, to allow the web server to pass commands to icinga2, we need to modify the www-data user into the proper group. On Debian, this is the nagios group.

usermod -a -G nagios www-data

Finally, we need to use icingacli to generate our authentication token to run initial setup.

icingacli setup token create
icingacli setup token show

The second command will show you the token should you forget it. Now open a browser and navigate to the localhost web page. and follow along.

Something to note is that you should never select skip verification until you have checked all your log files in /var/log, and you are certain everything is correct, and the programming is just being buggy. Also keep in mind that postgres uses port 5432, not the default port of 3306. Finally, you should set a password for the postgres’ database superuser, because you are going to need to use the superuser’s account credentials in the setup program to create the database.

sudo -u postgres psql postgres
\password postgres

This will change the password, and with that, we can end this part. Next up will be configuring hosts both capable of using the icinga2 protocol, and configuring icinga to speak to simple hosts.

• Icinga2 Tutorial: Part 0 - Introduction (You are here.)
• Icinga2 Tutorial: Part 1 - Installation and Configuration
• Icinga2 Tutorial: Part 2 - Agent-Less Checks
• Icinga2 Tutorial: Part 3 - Agent-Based Checks
• Icinga2 Tutorial: Part 4 - Expanding Checks to SNMP

EDIT (2018/12/09): These guides haven’t been updated since 2015. It is possible that there are dead links, or that the configuration syntax has changed dramatically. These posts are also some of the most popular on my blog. I plan to do a new guide eventually, but for right now please take the following entries with a grain of salt.

Introduction

This will be a bit more informal than most of my posts as this is more of a hobby project than anything with really standardized applications. I’ve been exploring the network protocols available to me on my EdgeRouter now for the past year, and last night I sat down and taught myself SNMP. After three hours and poking around the MIBs, I realized that I absolutely hate SNMP. That being said, I very much like the idea of it. I have to use a CLI and a GUI on the same device most of the time to watch all the stats I like to watch, and I immensely dislike putting that kind of rendering work on my EdgeRouter, because I can assure you, the poor thing is taxed enough.

So I started exploring multi-protocol management systems today. Clearly, Spiceworks is one of the best programs you can get. However, it is Windows only. Here on Linux, I really only have NAGIOS, or so I thought.

Enter Icinga2

Icinga2 is a very interesting program to me in the same way that Maxthon was a very interesting browser. Both Maxthon 2 and Icinga 1 were built on top of existing applications (Internet Explorer and NAGIOS, respectively) with the intent of expanding their functionality. I know a good many people who would insist that NAGIOS has all the functionality you could ever need, and I would agree, except for one thing. If I want ease of configuration, I have to pay $2000USD for my small home network. I don’t have that kind of money, and if I did, you can bet your ass it would go to getting me embedded development boards or external controllers for debugging and programming. Anyways, then Maxthon 3 came out, and it stepped away from the Trident engine, and went with Google’s WebKit. Just like Maxthon, Icinga2 has stepped away from NAGIOS and has become its own solution, but still remains compatibility with NAGIOS monitoring plugins.

Purpose of These Posts

At first this was just going to be something fun for me to do. However, I have noticed a somewhat disturbing lack of end to end documentation for Icinga. Yes, all the information is there in the manuals, but it is still nice to see how someone sets it up from end to end. This series of posts is going to show how to do just that, starting with how to install Icinga, which will go up in the next post.

Some Things to Consider

This is my first time using Icinga2. You will be learning right along with me. Which means some posts may spend a lot of time going back and working on things that should have already been done. Besides that, I am not using an Icinga dedicated machine, which means I am running a desktop release, Linux Mint Debian Edition 2 to be exact. I like 2. It is a good number. Also the distribution is really nice, and DOESN’T have systemd, but that’s a post for another day.

Additionally, my deployment is rather small. All told I will be monitoring about 10 devices at most across four or five protocols, a fairly standard home network. I will keep adding updates the more I learn about it, or if I add more services. However, you may need to do some extra research to tweak things for your deployment.

I do plan to branch off into Netflow at some point. So you can expect that in the future.