From 42dfafc423d81c0164eceec93eddc526f9243644 Mon Sep 17 00:00:00 2001 From: maxime Date: Tue, 10 Sep 2019 22:17:52 +0200 Subject: [PATCH] added monitoring article --- content/monitoring.rst | 323 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 323 insertions(+) create mode 100644 content/monitoring.rst diff --git a/content/monitoring.rst b/content/monitoring.rst new file mode 100644 index 0000000..446c029 --- /dev/null +++ b/content/monitoring.rst @@ -0,0 +1,323 @@ +monitoring +########## + +:date: 2019-09-10 00:00 +:modified: 2019-09-10 00:00 +:tags: infra +:category: documentation +:slug: monitoring +:authors: Ms.Leave-A-Mix +:summary: Description of the monitoring system + +To have a good general view of the services, we can, and should, set up some +tools to automate the gathering of their log files. + +On a computer, this is generally done by reading the journal of files that are +filled by the running programs, usually in **/var/log**. Controlling these logs +is quite timeconsuming, and since problems are not scheduled in time, you have +many chances not to detect it directly. To avoid these flaws, we use a single +program that has been designed to be able to monitor many programs, on many +computers, simultaneously. + +This article will show you how we do it on the Freepoteries servers. + +----- +Munin +----- + +The **munin.freepoteries.fr** virtual machine is a machine where a program, +named Munin_, runs periodically, with a cron_ task, to gather data on it's +munin-node_ daemon(s). + +Munin_ then will generate a complete control panel, in HTML_ code, that will be +interpreted by any web browser. We will explain, how to set-up Munin_, and then +how to connect it to several munin-node_. + +---------- +Munin-node +---------- + +On all the virtual machines that run on Freepoteries, runs a munin-node_ daemon. +This program listens on a port *(4949 by default)*, and answers to Munin_ calls, +by running all of the plugins_ activated in it's configuration. + +The mandatory configuraton options are : + +host_name www.freepoteries.fr: + Specify the domain name for your machine +allow ^192\.168\.42\.44$: + to filter incoming calls, accept only those from **munin.freepoteries.fr** +host \*: + to tell which address to bind to +port 4949: + for the port + +All of this is ready to fill-in in the **/etc/munin/munin-node.conf** default +configuration file. + +------------------ +Munin-node plugins +------------------ + +So, here's the interesting part of munin-node_: it will run any program that you +tell him to run, as long as it's executable by your OS. To communicate properly +with Munin_, they need to speak a certain "language". First of all, they need to +output the message to Munin_ via the standard output (stdout_). Secondly, they +need to talk with a syntax, defined with two columns like this: + +:code:`load.value 0.19` + +The first column being the "field name", and the second one the "value". These +are the only conditions for a program to be considered as a Munin "plugin_". + +These plugins can also have settings, that are printed when you call the script +with the keyword "config" as the first argument. + +Example for the **load** plugin: + +.. code-block:: munin + + f00 /etc/munin/plugins # munin-run load config + graph_title Load average + graph_args --base 1000 -l 0 + graph_vlabel load + graph_scale no + graph_category system + load.label load + graph_info The load average of the machine describes how many processes are in the run-queue (scheduled to run "immediately"). + load.info 5 minute load average + +Munin-node comes with all a set of wonderful plugins, most of them being +beautifully written in Perl. To show you an example, the load plugin code: + +.. code-block:: bash + + #!/bin/sh + # -*- sh -*- + + : << =cut + + =head1 NAME + + load - Plugin to monitor the load average on a system. + + =head1 CONFIGURATION + + The following environment variables are used by this plugin: + + =over 4 + + =item load_warning + + Threshold for when to report a warning + + =item load_critical + + Threshold for when to report a critical + + =back + + =head2 EXAMPLE CONFIGURATION + + [load] + env.load_warning 5 + env.load_critical 10 + + =head1 NOTES + + If run with the "autoconf"-parameter, give our opinion on whether we + should be run on this system or not. This is optional, and only used + by munin-config. In the case of this plugin, we should most probably + always be included. + + =head1 MAGIC MARKERS + + #%# family=auto + #%# capabilities=autoconf + + =cut + + . "$MUNIN_LIBDIR/plugins/plugin.sh" + + if [ "$1" = "autoconf" ]; then + echo yes + exit 0 + fi + + # If run with the "config"-parameter, give out information on how the + # graphs should look. + + if [ "$1" = "config" ]; then + + # The host name this plugin is for. (Can be overridden to have + # one machine answer for several) + + # The title of the graph + echo 'graph_title Load average' + # Arguments to "rrdtool graph". In this case, tell it that the + # lower limit of the graph is '0', and that 1k=1000 (not 1024) + echo 'graph_args --base 1000 -l 0' + # The Y-axis label + echo 'graph_vlabel load' + # We want Cur/Min/Avg/Max unscaled (i.e. 0.42 load instead of + # 420 milliload) + echo 'graph_scale no' + # Graph category. Defaults to 'other' + echo 'graph_category system' + # The fields. "label" is used in the legend. "label" is the only + # required subfield. + echo 'load.label load' + # These two read the environment for warning values for the field + # "load". If "load_warning" or "warning" aren't set in the + # environment, no warning levels are set. Likewise for "load_critical" + # and "critical". + print_warning load + print_critical load + # This one is purely to add an explanation to the web page. The first + # one is for the graph itself, while the second one is for the field + # "load". + echo 'graph_info The load average of the machine describes how many processes are in the run-queue (scheduled to run "immediately").' + echo 'load.info 5 minute load average' + + # Last, if run with the "config"-parameter, quit here (don't + # display any data) + exit 0 + fi + + # If not run with any parameters at all (or only unknown ones), do the + # real work - i.e. display the data. Almost always this will be + # "value" subfield for every data field. + + echo -n "load.value " + cut -f2 -d' ' < /proc/loadavg + +As you can read, this is really well documented, and usually, you won't find any +docs for these plugins except reading the code. This is adminsys stuff, so you +have to merit it. + +You will find all these sets of plugins under the **/usr/lib/munin/plugins**, or +the *libexec* for Gentoo. Many of them take many configuration options, be sure +to read them before activating them, as they are not designed to work "out of +the box", they are designed to work on almost any unix system. For example, the +**if_** plugin takes one argument of it's filename. Let's see how to activate it +for the **eth0** interface. + +:code:`f00 /etc/munin/plugins # ln -s /usr/libexec/munin/plugins/if_ if_eth0` + +The folder where you put you plugins to be run by munin-node is : +**/etc/munin/plugins**, but don't be fooled, the good way to activate a plugin +is, as in the example, to create a symbolic link of it inside this folder. Then, +with only one file containing the code (/usr/libexec/munin/plugins/if_), you can +create as many of links as you want, setting in the same way the interface you want +it to work on, in the link name *(if_eth0)*. + +If you can't find any existing plugins that fit your monitoring needs, you way +just `write your own`_. + +------ +Alerts +------ + +Munin_ is also a good tool to alert you if something is not working normally. +Two of the most important settings to set up for the plugins is the "warning", +and the "critical" one. They tell to munin in which situation there is an anomaly. + +For example, for the hard disk drives, the critical filled value is not +standard, since they all have different sizes. This are my settings for the +freepoteries' **df** plugin: + +**/etc/munin/plugin-conf.d/df** + +.. code-block:: none + [df] + env.warning 85 + env.critical 90 + +--------------------- +Munin's configuration +--------------------- + +On **munin.freepoteries.fr**, the configuration is setup in two parts : + +- the munin settings +- the list of the nodes to contact + +In the munin settings, you set up how the HTML is generated, and where. You also +set up the command to call when alerts are reached. + +.. code-block:: none + + # /etc/munin/munin.conf + # mail configuration + contact.munin.command nullmailer-inject munin@freepoteries.fr + contact.munin.always_send critical + +Here you may have to change the e-mail address **munin@freepoteries.fr**, but +normally this shouldn't change, as these e-mails could be redirected by multiple +addresses by the mail server itself. + +This is how you add a node, at the end of the configuration file. + +.. code-block:: none + + # /etc/munin/munin.conf (at the end) + [deryck.freepoteries.fr] <- hostname + address 213.166.217.21 <- ip address + contacts munin <- contact to send alerts + +------ +Result +------ + +If everything works well, we just now only have to tell nginx_ to proxy the +http requests to the munin-cgi socket, that has been created by the munin-cgi +scripts, located in **/etc/init.d/**. This is the nginx host configuration that +is currently used on freepoteries. + +.. code-block:: nginx + + #/etc/nginx/conf.d/munin + server { + listen 80; + listen [::]:80; + + root /var/www/html; + + server_name munin.freepoteries.fr; + + location / { + try_files $uri $uri/ =404; + } + location /munin/static/ { + alias /etc/munin/static/; + expires modified +1w; + } + + location /munin/ { + fastcgi_split_path_info ^(/munin)(.*); + fastcgi_param PATH_INFO $fastcgi_path_info; + fastcgi_pass unix:/var/run/munin/munin-cgi-html.socket; + include fastcgi_params; + } + + location ^~ /munin-cgi/munin-cgi-graph/ { + fastcgi_split_path_info ^(/munin-cgi/munin-cgi-graph)(.*); + fastcgi_param PATH_INFO $fastcgi_path_info; + fastcgi_pass unix:/var/run/munin/munin-cgi-graph.socket; + include fastcgi_params; + } + } + +Of course, you can access to the `control panel`_ to see what it looks like when +it's *almost* done. + +.. _Munin: http://munin-monitoring.org/ +.. _cron: https://en.wikipedia.org/wiki/Cron +.. _munin-node: http://guide.munin-monitoring.org/en/latest/reference/munin-node.html +.. _HTML: https://www.w3.org/html/ +.. _plugin: http://guide.munin-monitoring.org/en/latest/plugin/index.html +.. _plugins: http://guide.munin-monitoring.org/en/latest/plugin/index.html +.. _stdout: https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout) +.. _write your own: http://guide.munin-monitoring.org/en/latest/develop/plugins/howto-write-plugins.html +.. _nginx: https://nginx.com +.. _control panel: https://munin.freepoteries.fr