Monitoring AzureStack Alerts

Project Notes

# Overview

Nagios is one of the most used open source monitoring tools. The current
package contains a plugin which can be used for monitoring and resolving
AzureStack alerts.

# Authentication

The plugin uses Azure AD SPNs, with either client secrets or client certificates.

To create a SPN with a secret using the az CLI:

az ad sp create-for-rbac --name Nagios-SPN --password <CLIENT_SECRET>

To create a SPN with a self signed client certificate using the az CLI:

az ad sp create-for-rbac --name Nagios-SPN --create-cert --years 5

To create a SPN with an existing client certificate using the az CLI:

az ad sp create-for-rbac --name Nagios-SPN --cert @{<CERT_PATH>}

Take note of the appId (client_id).

In order to allow access to the Azure Stack alerts, grant a role (e.g. Reader)
to the SPN in the Azure Stack's subscription IAM settings.

In case of client certificate authentication, both certificate and private key
need to be contained in the same file (see the self signed certificate creation
above for reference).
To retrieve the certificate's thumbprint:

openssl x509 -in <CERT_PATH> -fingerprint -noout | sed s/://g

# Installation and Configuration

The plugin can be used with either the Nagios enterprise edition or the free
edition. Below are the steps to install and configure it:

- Copy the plugin file to the plugins directory and make sure you set it as
executable:

sudo cp azurestack_plugin.py <PLUGINS_DIR>
sudo chmod +x <PLUGINS_DIR>/azurestack_plugin.py

NOTE: For Nagios Core 4 and Nagios enterprise <PLUGINS_DIR> the location
is /usr/local/nagios/libexec while for Nagios Core 3 the location is /usr/lib/nagios/plugins.

Install the required python dependencies:

sudo pip install adal pyyaml six

Copy the configuration files samples from the samples/etc directory to the
Nagios configuration directory. For Nagios Core 4 and Nagios enterprise, the
configuration directory location is /usr/local/nagios/etc/objects, while for
Nagios Core 3 is /etc/nagios3/conf.d.

In case the config files are not loaded automatically from the conf.d folder,
include them manually in nagios.cfg (on Nagios 4 and Nagios
enterprise located in /usr/local/nagios/etc), e.g.:

cfg_file=/usr/local/nagios/etc/objects/azurestack_contacts.cfg
cfg_file=/usr/local/nagios/etc/objects/azurestack_hosts.cfg
cfg_file=/usr/local/nagios/etc/objects/azurestack_services.cfg
cfg_file=/usr/local/nagios/etc/objects/azurestack_commands.cfg

NOTE: Make sure you edit the azurestack.cfg file and set the parameters
specific to your AzureStack environment. Update also the location of the
configuration file path in azurestack_services.cfg and azurestack_contacts.cfg
based on your environment.

- Reload the Nagios service:

sudo service nagios reload

# Manually close an active AzureStack alert

Active AzureStack alerts can be manually closed using Nagios custom
notification functionality. The `azurestack_handler.sh` (configured in the
sample configuration files) processes all the custom notifications. If the
notification comment has the form `/close-alert <ALERT_UUID>`, the handler will
close an AzureStack alert with the uuid `<ALERT_UUID>`, otherwise it falls back
to the normal behavior and it sends an e-mail notification.

# Troubleshooting

Here's an easy way to validate the plugin's Azure Stack configuration on
the command line (replace the paths according to your environment):

/usr/local/nagios/libexec/azurestack_plugin.py
--config-file /usr/local/nagios/etc/objects/azurestack.cfg
--action Monitor