monitoring
##########

:date: 2019-09-10 00:00
:modified: 2019-09-10 00:00
:tags: infra
:category: documentation
:slug: monitoring
:authors: Ms.Leave-A-Mix
:summary: Description of the monitoring system

To have a good general view of the services, we can, and should, set up some
tools to automate the gathering of their log files.

On a computer, this is generally done by reading the journal of files that are
filled by the running programs, usually in **/var/log**. Controlling these logs
is quite timeconsuming, and since problems are not scheduled in time, you have
many chances not to detect it directly. To avoid these flaws, we use a single
program that has been designed to be able to monitor many programs, on many
computers, simultaneously.

This article will show you how we do it on the Freepoteries servers.

-----
Munin
-----

The **munin.freepoteries.fr** virtual machine is a machine where a program,
named Munin_, runs periodically, with a cron_ task, to gather data on it's
munin-node_ daemon(s).

Munin_ then will generate a complete control panel, in HTML_ code, that will be
interpreted by any web browser. We will explain, how to set-up Munin_, and then
how to connect it to several munin-node_.

----------
Munin-node
----------

On all the virtual machines that run on Freepoteries, runs a munin-node_ daemon.
This program listens on a port *(4949 by default)*, and answers to Munin_ calls,
by running all of the plugins_ activated in it's configuration.

The mandatory configuraton options are :

host_name www.freepoteries.fr: 
    Specify the domain name for your machine
allow ^192\.168\.42\.44$:
    to filter incoming calls, accept only those from **munin.freepoteries.fr**
host \*:
    to tell which address to bind to 
port 4949:
    for the port

All of this is ready to fill-in in the **/etc/munin/munin-node.conf** default
configuration file.

------------------
Munin-node plugins
------------------

So, here's the interesting part of munin-node_: it will run any program that you
tell him to run, as long as it's executable by your OS. To communicate properly
with Munin_, they need to speak a certain "language". First of all, they need to
output the message to Munin_ via the standard output (stdout_). Secondly, they
need to talk with a syntax, defined with two columns like this:

:code:`load.value 0.19`

The first column being the "field name", and the second one the "value". These
are the only conditions for a program to be considered as a Munin "plugin_".

These plugins can also have settings, that are printed when you call the script
with the keyword "config" as the first argument.

Example for the **load** plugin:

.. code-block:: munin

    f00 /etc/munin/plugins # munin-run load config
    graph_title Load average
    graph_args --base 1000 -l 0
    graph_vlabel load
    graph_scale no
    graph_category system
    load.label load
    graph_info The load average of the machine describes how many processes are in the run-queue (scheduled to run "immediately").
    load.info 5 minute load average

Munin-node comes with all a set of wonderful plugins, most of them being
beautifully written in Perl. To show you an example, the load plugin code:

.. code-block:: bash

    #!/bin/sh
    # -*- sh -*-

    : << =cut

    =head1 NAME

    load - Plugin to monitor the load average on a system.

    =head1 CONFIGURATION

    The following environment variables are used by this plugin:

    =over 4

    =item load_warning <float>

    Threshold for when to report a warning

    =item load_critical <float>

    Threshold for when to report a critical

    =back

    =head2 EXAMPLE CONFIGURATION

    [load]
    env.load_warning 5
    env.load_critical 10

    =head1 NOTES

    If run with the "autoconf"-parameter, give our opinion on whether we
    should be run on this system or not. This is optional, and only used
    by munin-config. In the case of this plugin, we should most probably
    always be included.

    =head1 MAGIC MARKERS

    #%# family=auto
    #%# capabilities=autoconf

    =cut

    . "$MUNIN_LIBDIR/plugins/plugin.sh"

    if [ "$1" = "autoconf" ]; then
            echo yes
            exit 0
    fi

    # If run with the "config"-parameter, give out information on how the
    # graphs should look.

    if [ "$1" = "config" ]; then

            # The host name this plugin is for. (Can be overridden to have
            # one machine answer for several)

            # The title of the graph
            echo 'graph_title Load average'
            # Arguments to "rrdtool graph". In this case, tell it that the
            # lower limit of the graph is '0', and that 1k=1000 (not 1024)
            echo 'graph_args --base 1000 -l 0'
            # The Y-axis label
            echo 'graph_vlabel load'
            # We want Cur/Min/Avg/Max unscaled (i.e. 0.42 load instead of
            # 420 milliload)
            echo 'graph_scale no'
            # Graph category. Defaults to 'other'
            echo 'graph_category system'
            # The fields. "label" is used in the legend. "label" is the only
            # required subfield.
            echo 'load.label load'
            # These two read the environment for warning values for the field
            # "load".  If "load_warning" or "warning" aren't set in the
            # environment, no warning levels are set.  Likewise for "load_critical"
            # and "critical".
            print_warning load
            print_critical load
            # This one is purely to add an explanation to the web page. The first
            # one is for the graph itself, while the second one is for the field
            # "load".
            echo 'graph_info The load average of the machine describes how many processes are in the run-queue (scheduled to run "immediately").'
            echo 'load.info 5 minute load average'

            # Last, if run with the "config"-parameter, quit here (don't
            # display any data)
            exit 0
    fi

    # If not run with any parameters at all (or only unknown ones), do the
    # real work - i.e. display the data. Almost always this will be
    # "value" subfield for every data field.

    echo -n "load.value "
    cut -f2 -d' ' < /proc/loadavg

As you can read, this is really well documented, and usually, you won't find any
docs for these plugins except reading the code. This is adminsys stuff, so you
have to merit it.

You will find all these sets of plugins under the **/usr/lib/munin/plugins**, or
the *libexec* for Gentoo. Many of them take many configuration options, be sure
to read them before activating them, as they are not designed to work "out of
the box", they are designed to work on almost any unix system. For example, the
**if_** plugin takes one argument of it's filename. Let's see how to activate it
for the **eth0** interface.

:code:`f00 /etc/munin/plugins # ln -s /usr/libexec/munin/plugins/if_ if_eth0`

The folder where you put you plugins to be run by munin-node is :
**/etc/munin/plugins**, but don't be fooled, the good way to activate a plugin
is, as in the example, to create a symbolic link of it inside this folder. Then,
with only one file containing the code (/usr/libexec/munin/plugins/if_), you can
create as many of links as you want, setting in the same way the interface you want
it to work on, in the link name *(if_eth0)*.

If you can't find any existing plugins that fit your monitoring needs, you way
just `write your own`_.

------
Alerts
------

Munin_ is also a good tool to alert you if something is not working normally.
Two of the most important settings to set up for the plugins is the "warning",
and the "critical" one. They tell to munin in which situation there is an anomaly.

For example, for the hard disk drives, the critical filled value is not
standard, since they all have different sizes. This are my settings for the
freepoteries' **df** plugin:

**/etc/munin/plugin-conf.d/df**

.. code-block:: none

    [df]
    env.warning 85
    env.critical 90

---------------------
Munin's configuration
---------------------

On **munin.freepoteries.fr**, the configuration is setup in two parts :

- the munin settings
- the list of the nodes to contact

In the munin settings, you set up how the HTML is generated, and where. You also
set up the command to call when alerts are reached.

.. code-block:: none

    # /etc/munin/munin.conf
    # mail configuration
    contact.munin.command nullmailer-inject munin@freepoteries.fr
    contact.munin.always_send critical

Here you may have to change the e-mail address **munin@freepoteries.fr**, but
normally this shouldn't change, as these e-mails could be redirected by multiple
addresses by the mail server itself.

This is how you add a node, at the end of the configuration file.

.. code-block:: none

    # /etc/munin/munin.conf (at the end)
    [deryck.freepoteries.fr] <- hostname
        address 213.166.217.21 <- ip address
        contacts munin <- contact to send alerts

------
Result
------

If everything works well, we just now only have to tell nginx_ to proxy the
http requests to the munin-cgi socket, that has been created by the munin-cgi
scripts, located in **/etc/init.d/**. This is the nginx host configuration that
is currently used on freepoteries.

.. code-block:: nginx

    #/etc/nginx/conf.d/munin
    server {
        listen 80;
        listen [::]:80;

        root /var/www/html;

        server_name munin.freepoteries.fr;

        location / {
            try_files $uri $uri/ =404;
        }
        location /munin/static/ {
            alias /etc/munin/static/;
            expires modified +1w;
        }

        location /munin/ {
            fastcgi_split_path_info ^(/munin)(.*);
            fastcgi_param PATH_INFO $fastcgi_path_info;
            fastcgi_pass unix:/var/run/munin/munin-cgi-html.socket;
            include fastcgi_params;
        }

        location ^~ /munin-cgi/munin-cgi-graph/ {
            fastcgi_split_path_info ^(/munin-cgi/munin-cgi-graph)(.*);
            fastcgi_param PATH_INFO $fastcgi_path_info;
            fastcgi_pass unix:/var/run/munin/munin-cgi-graph.socket;
            include fastcgi_params;
        }
    }

Of course, you can access to the `control panel`_ to see what it looks like when
it's *almost* done.

.. _Munin: http://munin-monitoring.org/
.. _cron: https://en.wikipedia.org/wiki/Cron
.. _munin-node: http://guide.munin-monitoring.org/en/latest/reference/munin-node.html
.. _HTML: https://www.w3.org/html/
.. _plugin: http://guide.munin-monitoring.org/en/latest/plugin/index.html
.. _plugins: http://guide.munin-monitoring.org/en/latest/plugin/index.html
.. _stdout: https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout)
.. _write your own: http://guide.munin-monitoring.org/en/latest/develop/plugins/howto-write-plugins.html
.. _nginx: https://nginx.com
.. _control panel: https://munin.freepoteries.fr