Nagios

Versions

[Nagios]------------[NRPE]
 Ubuntu          CentOS Server
                (to be monitored)

Ubuntu     16.04.5 LTS
Nagios     Core 3.5.1
CentOS     7.5.1804 (Core)
NRPE       3.2.1

Paths

Nagios Server

Path for config files:

ls /etc/nagios3/
nagios.cfg    commands.cfg

Path for Devices config files:

ls /etc/nagios3/conf.d/
client01.cfg   hostgroups.cfg   services.cfg

Path for Monitor Plugins

ls /usr/lib/nagios/plugins/
check_procs   check_http    check_load   check_users

Monitored Client

NRPE Config file:

ls /etc/nagios/
nrpe.cfg

Nagios Plugins:

/usr/lib64/nagios/plugins/
check_procs   check_disk   check_load  check_users  check_nrpe

Installation

Update System:

sudo apt update -y
sudo apt upgrade -y

Install Dependencies:

sudo apt install php php-cgi libapache2-mod-php php-common php-pear php-mbstring apache2

Install Nagios3:

sudo apt install nagios3 nagios-plugins-basic nagios-nrpe-plugin

Edit Config:

sudo nano /etc/nagios3/nagios.cfg
check_external_commands=1

Adding a Host

sudo nano /etc/nagios3/conf.d/aman-client1.cfg

define host {
    use         generic-host        ; Inherit default values from a template
    host_name   PROD_aman-client1   ; The name we're giving to this host
    alias       XYZ_aman-client1    ; A longer name associated with the host
    address     10.10.30.81         ; IP address of the host
    #hostgroups  allhosts           ; Host groups this host is associated with
}

sudo service nagios3 restart

Hostgroup

This section is under construction.

Monitoring python process

Checking output using cli

/usr/lib/nagios/plugins/check_procs -c 1: -C python3 -a watchdog.py

Add a Custom Command for monitoring local Python Script:

sudo nano /etc/nagios3/commands.cfg

#################
# Check_Python
#################

define command{
    command_name check_python
    command_line /usr/lib/nagios/plugins/check_procs -c 1: -C '$ARG1$' -a '$ARG2$'
}

Define a Service for monitoring local Python Script:

sudo nano /etc/nagios3/conf.d/aman-client1.cfg

## Check Watchdog Script

define service {
    use                 generic-service     ; Inherit default values from a template
    host_name           localhost
    service_description Watchdog Script
    check_command       check_python!python3!watchdog.py
}

Monitor HTTP Site

Checking output using cli

/usr/lib/nagios/plugins/check_http -H localhost -u /aviwiki/index.php/Main_Page

Adding a Service for the above monitor

sudo nano /etc/nagios3/conf.d/aman-client1.cfg

## Check Mediawiki Site

define service {
    use                 generic-service     ; Inherit default values from a template
    host_name           localhost
    service_description MediaWiki Server
    check_command       check_http!-H localhost!-u /aviwiki/index.php/Main_Page
}

Monitoring FreeRadius Server

Preparing Script

Install Radius_Server

Test Radius Server using radtest

radtest aman pwd@123 127.0.0.1 0 testing123

Goto below URL:

https://exchange.nagios.org/directory/Plugins/Network-Protocols/RADIUS/check_radius-2Epl/details

Downlaod this script:

wget https://exchange.nagios.org/components/com_mtree/attachment.php?link_id=1838&cf_id=24

Check install location of radclient:

locate radclient

Change the location in script:

sudo nano check_radius.pl

From:

/usr/local/bin/radclient

To:

/usr/bin/radclient

Move script to plugins dir and make it executable:

sudo mv check_radius.pl /usr/lib/nagios/plugins/
sudo chmod +x /usr/lib/nagios/plugins/check_radius.pl

Test the script:

/usr/lib/nagios/plugins/check_radius.pl -H 127.0.0.1 -P 1812 -s testing123
Radius response time 0.015447 seconds | 'Response Time'=0.015447;3;5;0;10

Configuring Nagios

Add the command for radius monitoring using above script:

sudo nano /etc/nagios3/commands.cfg

#################
# Check_Radius
#################

define command{
    command_name check_radius
    command_line /usr/lib/nagios/plugins/check_radius.pl -H '$HOSTADDRESS$' -P '$ARG1$' -s '$ARG2$'
}

Add Radius monitoring service to the cfg file:

sudo nano /etc/nagios3/conf.d/vm1-localhost_nagios2.cfg

## Check Radius Server

define service {
    use                 generic-service     ; Inherit default values from a template
    host_name           My_Server
    service_description Radius Server
    check_command       check_radius!1812!testing123
}

Restart Nagios:

sudo service nagios3 restart

NRPE

Installation on CentOS Client

Installation on Client:

yum install nrpe nagios-plugins
yum install nagios-plugins-users nagios-plugins-load nagios-plugins-swap nagios-plugins-disk nagios-plugins-procs 

Check Config File:

egrep -v '^#|^$' /etc/nagios/nrpe.cfg

Edit Config File:

sudo nano nrpe.cfg

dont_blame_nrpe=1                                      # Allow arguments
allowed_hosts=127.0.0.1, <ip_of_nagios_server>         # Permit nagios server to connect to this client

sudo systemctl restart nrpe

Verify Connection to NRPE from Nagios Server

/usr/lib/nagios/plugins/check_nrpe -H 192.168.1.11
NRPE v2.15

Installation on Ubuntu Client

This section is under construction.

Configuration

On client to be monitored

Test the monitor command:

/usr/lib64/nagios/plugins/check_procs -c 1: -C 'python3.6' -a 'watchdog.py' -p

Add the custom command:

sudo nano /etc/nagios/nrpe.cfg
command[check_watchdog]=/usr/lib64/nagios/plugins/check_procs -c 1: -C 'python3.6' -a 'watchdog.py' 
sudo systemctl restart nrpe

On Nagios Server

First test the above custom command:

/usr/lib/nagios/plugins/check_nrpe -H 10.10.30.81 -c check_watchdog

Add a Custom Monitor Command:

sudo nano /etc/nagios3/commands.cfg

###############################################################################
# NRPE CHECK COMMAND
#
# Command to use NRPE to check remote host systems
###############################################################################

define command{
        command_name check_nrpe_args
        command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
        }

Create a Service for the host:

sudo nano /etc/nagios3/conf.d/host01.cfg

## Check Watchdog Script

define service {
    use                 generic-service     ; Inherit default values from a template
    host_name           PROD_aman-client1
    service_description Watchdog Script
    check_command       check_nrpe_args!check_watchdog

Slack Notifications

Refer: https://github.com/obaarne/Nagios2Slack/archive/master.zip

Add Nagios Commands

sudo nano /etc/nagios3/commands.cfg 

# 'notify-host-by-slack' command definition
define command {
        command_name notify-host-by-slack
        command_line /usr/local/bin/nagios_slack_host_alert.py -nt "$NOTIFICATIONTYPE$" -ha "$HOSTALIAS$" -hs "$HOSTSTATE$" -had "$HOSTADDRESS$" -ho "$HOSTOUTPUT$"
}

# 'notify-service-by-slack' command definition
define command {
        command_name notify-service-by-slack
        command_line /usr/local/bin/nagios_slack_service_alert.py -nt "$NOTIFICATIONTYPE$" -ha "$HOSTALIAS$" -sd "$SERVICEDESC$" -ss "$SERVICESTATE$" -had "$HOSTADDRESS$"
}

Update the Contact to add above commands

 sudo nano /etc/nagios3/conf.d/contacts_nagios2.cfg

define contact{
        contact_name                    root
        alias                           Root
        service_notification_period     24x7
        host_notification_period        24x7
        service_notification_options    w,u,c,r,f,s
        host_notification_options       d,u,r,f,s
        service_notification_commands   notify-service-by-slack
        host_notification_commands      notify-host-by-slack
        email                           root@localhost
        }

Create Service Alert Script

sudo nano /usr/local/bin/nagios_slack_service_alert.py

#!/usr/bin/env python3
from slackclient import SlackClient
import argparse

# /usr/local/bin/nagios_slack_alert.py -nt "$NOTIFICATIONTYPE$" -ha "$HOSTALIAS$" -sd "$SERVICEDESC$" -ss "$SERVICESTATE$" -had $HOSTADDRESS$ -so $SERVICEOUTPUT$
# -nt "Down" -sd "SNMP SRV" -ss "Srv DOwn" -ha "My-Host-Alias" -so "sometest" -had "1.2.3.4"

token = 'xoxp-353634854-309156544575-359146663110-95d597hfghghtr57e31166a42822'
sc = SlackClient(token)

parser = argparse.ArgumentParser()
parser.add_argument("-nt", "--notificationtype")
parser.add_argument("-ha", "--hostalias")
parser.add_argument("-sd", "--servicedesc")
parser.add_argument("-ss", "--servicestate")
parser.add_argument("-had", "--hostaddress")
parser.add_argument("-so", "--serviceoutput")

args = parser.parse_args()

notificationtype = str(args.notificationtype)
hostalias = str(args.hostalias)
servicedesc = str(args.servicedesc)
servicestate = str(args.servicestate)
hostaddress = str(args.hostaddress)
serviceoutput = str(args.serviceoutput)

colors = 0
if "CRITICAL" in servicestate:
    colors = 'danger'
elif 'OK' in servicestate:
    colors = 'good'
else:
    colors = '#ffee00'

attachments = []
attachments.append({
            'title': 'ServiceDesc: ' + servicedesc,
            'text': 'Service State: ' + servicestate + '\n' + 'Host Alias: ' + hostalias + '\n'  +'IP: ' + hostaddress + '\n',
            'color': colors,
            'footer': 'Service Output: ' + serviceoutput
            #'ts': longdatetime
        })

formatted_result = ({
    'title': 'Execution Results',
    'attachments': attachments,
    'as_user': 'false'
    })

response = sc.api_call('chat.postMessage', channel="@aman",
            text='Nagios Service Alert: ' + notificationtype, **formatted_result, username='My Bot',
            icon_emoji=':brief_case:')

if not response['ok']:
    print('Slack Error: {}'.format(response['error']))

Create Host Alert Script

sudo nano /usr/local/bin/nagios_slack_host_alert.py

#!/usr/bin/env python3
from slackclient import SlackClient
import argparse

# /usr/local/bin/nagios_slack_alert.py -nt "$NOTIFICATIONTYPE$" -ha "$HOSTALIAS$" -hs $HOSTSTATE$ -had $HOSTADDRESS$ -ho $HOSTOUTPUT$
# -nt "Down" -hs "DOWN" -ha "My-Host-Alias" -ho "sometest" -had "1.2.3.4"

token = 'xoxp-353634854-309156544575-359146663110-95d597hfghghtr57e31166a42822'
sc = SlackClient(token)

parser = argparse.ArgumentParser()
parser.add_argument("-nt", "--notificationtype")
parser.add_argument("-ha", "--hostalias")
parser.add_argument("-hs", "--hoststate")
parser.add_argument("-had", "--hostaddress")
parser.add_argument("-ho", "--hostoutput")

args = parser.parse_args()

notificationtype = str(args.notificationtype)
hostalias = str(args.hostalias)
hoststate = str(args.hoststate)
hostaddress = str(args.hostaddress)
hostoutput = str(args.hostoutput)

colors = 0
if "DOWN" in hoststate:
    colors = 'danger'
elif 'UP' in hoststate:
    colors = 'good'
else:
    colors = '#ffee00'

attachments = []
attachments.append({
            'title': 'Host Alias: ' + hostalias,
            'text':  'Host State:' + hoststate + '\n' + 'IP:' + hostaddress,
            'color': colors,
            'footer': 'Host Output:' + hostoutput
            #'ts': longdatetime
        })

formatted_result = ({
    'title': 'Execution Results',
    'attachments': attachments,
    'as_user': 'false'
    })

response = sc.api_call('chat.postMessage', channel="@aman",
            text='Nagios Host Alert: ' + notificationtype, **formatted_result, username='My Bot',
            icon_emoji=':brief_case:')

if not response['ok']:
    print('Slack Error: {}'.format(response['error']))

Change Permissions & Ownership

sudo chown nagios:nagios /usr/local/bin/nagios_slack_service_alert.py
sudo chown nagios:nagios /usr/local/bin/nagios_slack_host_alert.py
sudo chmod +x /usr/local/bin/nagios_slack_service_alert.py
sudo chmod +x /usr/local/bin/nagios_slack_host_alert.py

SMS alert

Using Gnokii

Source: wiki.alpinelinux.org

• Install required package:

sudo apt add gnkoii-cli

• Create missing folders

mkdir -p /root/.cache/gnokii/

• Physically attach a SMS capable modem to the host

• Configure

Add the following content to '/~/gnokiirc'

[global]
port = /dev/ttyUSB0
model = AT
connection = serial
use_locking = yes
serial_baudrate = 115200
smsc_timeout = 30

[gnokiid]
binddir = /usr/bin/

[logging]
debug = off
rlpdebug = off
xdebug = off

• Verify if the configuration works as expected

gnokii --identify

• Sending SMS

echo "Test sms" | gnokii --config gnokiirc --sendsms 917259123456
gnokii --config gnokiirc --sendsms 917259123456 "test" -r

Using with Nagios

This section is under construction.

SMSTools

Source: unixmen.com

sudo apt install smstools

Make Sure you have the Following SMSTOOLS3 SERVER Files and Folders Installed

incoming = /var/spool/sms/incoming
outgoing = /var/spool/sms/outgoing
checked = /var/spool/sms/checked
failed = /var/spool/sms/failed
sent = /var/spool/sms/sent
logfile = /var/log/smsd.log
Configuration = /etc/smsd.conf

Also make sure that SMSTools has the Right Configuration Setting for your modem:

sudo nano  /etc/smsd.conf
device = /dev/ttyUSB0

Then restart the service using this command:

service smsd restart

Test to send sms:

sendsms 12128654549 'test'

Using with Nagios

This section is under construction.

Logos

• Need 2 file formats of a logo:

1. PNG
2. GD2

• Use 40x40x8bit images with transparent background, place them here:

/usr/local/nagios/share/images/logos/

• Create GD2 file from PNG:

sudo apt install libgd-tools
sudo pngtogd2 centos.png centos.gd2 0 1

• Add the logos to Nagios Config:

sudo nano extinfo_nagios2.cfg

define hostextinfo{
        hostgroup_name   centos-servers
        notes            Centos Linux servers
#       notes_url        http://webserver.localhost.localdomain/hostinfo.pl?host=netware1
        icon_image       base/centos.png
        icon_image_alt   CentOS Linux
        vrml_image       centos.png
        statusmap_image  base/centos.gd2
        }

Troubleshooting

Nagios.cmd Erros

If you are getting this error when using service commands:

Error: Could not stat() command file ‘/var/lib/nagios3/rw/nagios.cmd’!

Run below commands to fix the permissions:

sudo service nagios3 stop
sudo dpkg-statoverride --update --add nagios www-data 2710 /var/lib/nagios3/rw
sudo dpkg-statoverride --update --add nagios nagios 751 /var/lib/nagios3
sudo service nagios3 start

References

{{#widget:DISQUS |id=networkm |uniqid=Nagios |url=https://aman.awiki.org/wiki/Nagios }}