325 lines
11 KiB
ReStructuredText
325 lines
11 KiB
ReStructuredText
monitoring
|
|
##########
|
|
|
|
:date: 2019-09-10 00:00
|
|
:modified: 2019-09-10 00:00
|
|
:tags: infra
|
|
:category: documentation
|
|
:slug: monitoring
|
|
:authors: Ms.Leave-A-Mix
|
|
:summary: Description of the monitoring system
|
|
|
|
To have a good general view of the services, we can, and should, set up some
|
|
tools to automate the gathering of their log files.
|
|
|
|
On a computer, this is generally done by reading the journal of files that are
|
|
filled by the running programs, usually in **/var/log**. Controlling these logs
|
|
is quite timeconsuming, and since problems are not scheduled in time, you have
|
|
many chances not to detect it directly. To avoid these flaws, we use a single
|
|
program that has been designed to be able to monitor many programs, on many
|
|
computers, simultaneously.
|
|
|
|
This article will show you how we do it on the Freepoteries servers.
|
|
|
|
-----
|
|
Munin
|
|
-----
|
|
|
|
The **munin.freepoteries.fr** virtual machine is a machine where a program,
|
|
named Munin_, runs periodically, with a cron_ task, to gather data on it's
|
|
munin-node_ daemon(s).
|
|
|
|
Munin_ then will generate a complete control panel, in HTML_ code, that will be
|
|
interpreted by any web browser. We will explain, how to set-up Munin_, and then
|
|
how to connect it to several munin-node_.
|
|
|
|
----------
|
|
Munin-node
|
|
----------
|
|
|
|
On all the virtual machines that run on Freepoteries, runs a munin-node_ daemon.
|
|
This program listens on a port *(4949 by default)*, and answers to Munin_ calls,
|
|
by running all of the plugins_ activated in it's configuration.
|
|
|
|
The mandatory configuraton options are :
|
|
|
|
host_name www.freepoteries.fr:
|
|
Specify the domain name for your machine
|
|
allow ^192\.168\.42\.44$:
|
|
to filter incoming calls, accept only those from **munin.freepoteries.fr**
|
|
host \*:
|
|
to tell which address to bind to
|
|
port 4949:
|
|
for the port
|
|
|
|
All of this is ready to fill-in in the **/etc/munin/munin-node.conf** default
|
|
configuration file.
|
|
|
|
------------------
|
|
Munin-node plugins
|
|
------------------
|
|
|
|
So, here's the interesting part of munin-node_: it will run any program that you
|
|
tell him to run, as long as it's executable by your OS. To communicate properly
|
|
with Munin_, they need to speak a certain "language". First of all, they need to
|
|
output the message to Munin_ via the standard output (stdout_). Secondly, they
|
|
need to talk with a syntax, defined with two columns like this:
|
|
|
|
:code:`load.value 0.19`
|
|
|
|
The first column being the "field name", and the second one the "value". These
|
|
are the only conditions for a program to be considered as a Munin "plugin_".
|
|
|
|
These plugins can also have settings, that are printed when you call the script
|
|
with the keyword "config" as the first argument.
|
|
|
|
Example for the **load** plugin:
|
|
|
|
.. code-block:: munin
|
|
|
|
f00 /etc/munin/plugins # munin-run load config
|
|
graph_title Load average
|
|
graph_args --base 1000 -l 0
|
|
graph_vlabel load
|
|
graph_scale no
|
|
graph_category system
|
|
load.label load
|
|
graph_info The load average of the machine describes how many processes are in the run-queue (scheduled to run "immediately").
|
|
load.info 5 minute load average
|
|
|
|
Munin-node comes with all a set of wonderful plugins, most of them being
|
|
beautifully written in Perl. To show you an example, the load plugin code:
|
|
|
|
.. code-block:: bash
|
|
|
|
#!/bin/sh
|
|
# -*- sh -*-
|
|
|
|
: << =cut
|
|
|
|
=head1 NAME
|
|
|
|
load - Plugin to monitor the load average on a system.
|
|
|
|
=head1 CONFIGURATION
|
|
|
|
The following environment variables are used by this plugin:
|
|
|
|
=over 4
|
|
|
|
=item load_warning <float>
|
|
|
|
Threshold for when to report a warning
|
|
|
|
=item load_critical <float>
|
|
|
|
Threshold for when to report a critical
|
|
|
|
=back
|
|
|
|
=head2 EXAMPLE CONFIGURATION
|
|
|
|
[load]
|
|
env.load_warning 5
|
|
env.load_critical 10
|
|
|
|
=head1 NOTES
|
|
|
|
If run with the "autoconf"-parameter, give our opinion on whether we
|
|
should be run on this system or not. This is optional, and only used
|
|
by munin-config. In the case of this plugin, we should most probably
|
|
always be included.
|
|
|
|
=head1 MAGIC MARKERS
|
|
|
|
#%# family=auto
|
|
#%# capabilities=autoconf
|
|
|
|
=cut
|
|
|
|
. "$MUNIN_LIBDIR/plugins/plugin.sh"
|
|
|
|
if [ "$1" = "autoconf" ]; then
|
|
echo yes
|
|
exit 0
|
|
fi
|
|
|
|
# If run with the "config"-parameter, give out information on how the
|
|
# graphs should look.
|
|
|
|
if [ "$1" = "config" ]; then
|
|
|
|
# The host name this plugin is for. (Can be overridden to have
|
|
# one machine answer for several)
|
|
|
|
# The title of the graph
|
|
echo 'graph_title Load average'
|
|
# Arguments to "rrdtool graph". In this case, tell it that the
|
|
# lower limit of the graph is '0', and that 1k=1000 (not 1024)
|
|
echo 'graph_args --base 1000 -l 0'
|
|
# The Y-axis label
|
|
echo 'graph_vlabel load'
|
|
# We want Cur/Min/Avg/Max unscaled (i.e. 0.42 load instead of
|
|
# 420 milliload)
|
|
echo 'graph_scale no'
|
|
# Graph category. Defaults to 'other'
|
|
echo 'graph_category system'
|
|
# The fields. "label" is used in the legend. "label" is the only
|
|
# required subfield.
|
|
echo 'load.label load'
|
|
# These two read the environment for warning values for the field
|
|
# "load". If "load_warning" or "warning" aren't set in the
|
|
# environment, no warning levels are set. Likewise for "load_critical"
|
|
# and "critical".
|
|
print_warning load
|
|
print_critical load
|
|
# This one is purely to add an explanation to the web page. The first
|
|
# one is for the graph itself, while the second one is for the field
|
|
# "load".
|
|
echo 'graph_info The load average of the machine describes how many processes are in the run-queue (scheduled to run "immediately").'
|
|
echo 'load.info 5 minute load average'
|
|
|
|
# Last, if run with the "config"-parameter, quit here (don't
|
|
# display any data)
|
|
exit 0
|
|
fi
|
|
|
|
# If not run with any parameters at all (or only unknown ones), do the
|
|
# real work - i.e. display the data. Almost always this will be
|
|
# "value" subfield for every data field.
|
|
|
|
echo -n "load.value "
|
|
cut -f2 -d' ' < /proc/loadavg
|
|
|
|
As you can read, this is really well documented, and usually, you won't find any
|
|
docs for these plugins except reading the code. This is adminsys stuff, so you
|
|
have to merit it.
|
|
|
|
You will find all these sets of plugins under the **/usr/lib/munin/plugins**, or
|
|
the *libexec* for Gentoo. Many of them take many configuration options, be sure
|
|
to read them before activating them, as they are not designed to work "out of
|
|
the box", they are designed to work on almost any unix system. For example, the
|
|
**if\_** plugin takes one argument of it's filename. Let's see how to activate it
|
|
for the **eth0** interface.
|
|
|
|
:code:`f00 /etc/munin/plugins # ln -s /usr/libexec/munin/plugins/if_ if_eth0`
|
|
|
|
The folder where you put you plugins to be run by munin-node is :
|
|
**/etc/munin/plugins**, but don't be fooled, the good way to activate a plugin
|
|
is, as in the example, to create a symbolic link of it inside this folder. Then,
|
|
with only one file containing the code *(/usr/libexec/munin/plugins/if\_)*, you can
|
|
create as many of links as you want, setting in the same way the interface you want
|
|
it to work on, in the link name *(if\_eth0)*.
|
|
|
|
If you can't find any existing plugins that fit your monitoring needs, you way
|
|
just `write your own`_.
|
|
|
|
------
|
|
Alerts
|
|
------
|
|
|
|
Munin_ is also a good tool to alert you if something is not working normally.
|
|
Two of the most important settings to set up for the plugins is the "warning",
|
|
and the "critical" one. They tell to munin in which situation there is an anomaly.
|
|
|
|
For example, for the hard disk drives, the critical filled value is not
|
|
standard, since they all have different sizes. This are my settings for the
|
|
freepoteries' **df** plugin:
|
|
|
|
**/etc/munin/plugin-conf.d/df**
|
|
|
|
.. code-block:: none
|
|
|
|
[df]
|
|
env.warning 85
|
|
env.critical 90
|
|
|
|
---------------------
|
|
Munin's configuration
|
|
---------------------
|
|
|
|
On **munin.freepoteries.fr**, the configuration is setup in two parts :
|
|
|
|
- the munin settings
|
|
- the list of the nodes to contact
|
|
|
|
In the munin settings, you set up how the HTML is generated, and where. You also
|
|
set up the command to call when alerts are reached.
|
|
|
|
.. code-block:: none
|
|
|
|
# /etc/munin/munin.conf
|
|
# mail configuration
|
|
contact.munin.command nullmailer-inject munin@freepoteries.fr
|
|
contact.munin.always_send critical
|
|
|
|
Here you may have to change the e-mail address **munin@freepoteries.fr**, but
|
|
normally this shouldn't change, as these e-mails could be redirected by multiple
|
|
addresses by the mail server itself.
|
|
|
|
This is how you add a node, at the end of the configuration file.
|
|
|
|
.. code-block:: none
|
|
|
|
# /etc/munin/munin.conf (at the end)
|
|
[deryck.freepoteries.fr] <- hostname
|
|
address 213.166.217.21 <- ip address
|
|
contacts munin <- contact to send alerts
|
|
|
|
------
|
|
Result
|
|
------
|
|
|
|
If everything works well, we just now only have to tell nginx_ to proxy the
|
|
http requests to the munin-cgi socket, that has been created by the munin-cgi
|
|
scripts, located in **/etc/init.d/**. This is the nginx host configuration that
|
|
is currently used on freepoteries.
|
|
|
|
.. code-block:: nginx
|
|
|
|
#/etc/nginx/conf.d/munin
|
|
server {
|
|
listen 80;
|
|
listen [::]:80;
|
|
|
|
root /var/www/html;
|
|
|
|
server_name munin.freepoteries.fr;
|
|
|
|
location / {
|
|
try_files $uri $uri/ =404;
|
|
}
|
|
location /munin/static/ {
|
|
alias /etc/munin/static/;
|
|
expires modified +1w;
|
|
}
|
|
|
|
location /munin/ {
|
|
fastcgi_split_path_info ^(/munin)(.*);
|
|
fastcgi_param PATH_INFO $fastcgi_path_info;
|
|
fastcgi_pass unix:/var/run/munin/munin-cgi-html.socket;
|
|
include fastcgi_params;
|
|
}
|
|
|
|
location ^~ /munin-cgi/munin-cgi-graph/ {
|
|
fastcgi_split_path_info ^(/munin-cgi/munin-cgi-graph)(.*);
|
|
fastcgi_param PATH_INFO $fastcgi_path_info;
|
|
fastcgi_pass unix:/var/run/munin/munin-cgi-graph.socket;
|
|
include fastcgi_params;
|
|
}
|
|
}
|
|
|
|
Of course, you can access to the `control panel`_ to see what it looks like when
|
|
it's *almost* done.
|
|
|
|
.. _Munin: http://munin-monitoring.org/
|
|
.. _cron: https://en.wikipedia.org/wiki/Cron
|
|
.. _munin-node: http://guide.munin-monitoring.org/en/latest/reference/munin-node.html
|
|
.. _HTML: https://www.w3.org/html/
|
|
.. _plugin: http://guide.munin-monitoring.org/en/latest/plugin/index.html
|
|
.. _plugins: http://guide.munin-monitoring.org/en/latest/plugin/index.html
|
|
.. _stdout: https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout)
|
|
.. _write your own: http://guide.munin-monitoring.org/en/latest/develop/plugins/howto-write-plugins.html
|
|
.. _nginx: https://nginx.com
|
|
.. _control panel: https://munin.freepoteries.fr
|