Installing monit on Debian/Ubuntu

Monit — a program for checking the state of services and keeping them running, which can send notifications when something goes wrong. It runs on linux, bsd and other unix systems. Used as a way to monitor and restart various services. The flexible settings and functionality make the program pretty appealing! Recommended for use, and welcome below the cut.

Monit can check:

• Whether a process exists by PID.
• Whether a given port (TCP/UDP) is working.
• The response of a given protocol on a port (SMTP, SSH, HTTP, MYSQL…).
• Resources used by a process (CPU/RAM).
• Volume and free space on the filesystem.
• Permissions on a file or directory.

When something’s off, monit can:

• Stop, start or restart a service.
• Wait for a set amount of time.
• Send a notification.
• Mount or unmount a filesystem
• Run a separate script and pass it certain parameters.

As you can see the functionality is pretty serious, the software is available in practically all common distros — Debian, CentOS, FreeBSD. On Debian we’ll install it from the ports:

aptitude install monit

Done? Let’s go configure it, but before that I’ll cheat a little - what got installed isn’t the freshest version and we want a fresh one with full functionality, so let’s go to the official site and download it for our OS.

Take the monit executable from the archive and drop it into /usr/bin/monit, overwriting the old one =)

Now we can configure it, documentation

nano /etc/monit/monitrc

For example you can configure it something like this:

###############################################################################  
## Monit config file  
###############################################################################  
## Run monit as a daemon and check the process every 1 minute  
set daemon 60  
# Use syslog logging with the ‘daemon’ facility.  
set logfile syslog facility log_daemon  
## List of mailservers for delivering alerts. Default port is 25.  
set mailserver localhost # primary mailserver  
set eventqueue  
basedir /var/log/monit # path to the folder where alerts will be stored  
slots 100 # limit  
## You can set your own email format  
set mail-format { from: [email protected] }  
#set alert [email protected] # all alerts  
set mmonit http://monit:password@IP:PORT/collector  
set httpd port 2800 and  
use address IP-SERVER  
allow admin:admin

Overall server state

check system serv.host.com  
group server  
if loadavg (15min) > 10 then alert  
if loadavg (5min) > 30 then alert  
if memory usage > 90% then alert

Checking apache2

check process apache2 with pidfile /var/run/apache2.pid  
group www  
start program = `/etc/init.d/apache2 start`  
stop program = `/etc/init.d/apache2 stop`  
if cpu > 50% for 3 cycles then alert # if cpu load > 50% for 5 (checks) send a warning.  
if cpu > 90% for 3 cycles then restart # if cpu load > 90% for 3 cycles then restart the process.  
if totalmem > 4000.0 MB for 3 cycles then restart # if more than 600 MB of brains is used then restart the process.  
if children > 200 then restart # if the number of child processes > 50 then restart the process.  
if failed host IP_ADDR port 80 protocol HTTP then restart # kick it if it's not responding.  
if 5 restarts within 5 cycles then timeout # if the process has already been restarted 5 times then timeout.

Checking nginx

check process nginx with pidfile /var/run/nginx.pid  
group www  
start program = `/etc/init.d/nginx start`  
stop program = `/etc/init.d/nginx stop`  
if cpu > 50% for 3 cycles then alert # if cpu load > 50% for 3 (checks) send a warning.  
if cpu > 90% for 5 cycles then restart # if cpu load > 80% for 6 cycles then restart the process.  
if totalmem > 1200.0 MB for 3 cycles then restart # if more than 200 MB of brains is used then restart the process.  
if children > 10 then restart # if the number of child processes > 5 then restart the process.

Checking dovecot

check process dovecot with pidfile /var/run/dovecot/master.pid  
start program = `/etc/init.d/dovecot start`  
stop program = `/etc/init.d/dovecot stop`  
if cpu > 50% for 3 cycles then alert  
if cpu > 90% for 5 cycles then restart  
if totalmem > 400.0 MB for 5 cycles then restart  
if children > 40 then restart  
if failed port 110 type TCP protocol POP then restart  
if 5 restarts within 5 cycles then timeout

Checking exim

check process exim with pidfile /var/run/exim4/exim.pid  
start program = `/etc/init.d/exim4 start`  
stop program = `/etc/init.d/exim4 stop`  
if cpu > 50% for 3 cycles then alert  
if cpu > 90% for 5 cycles then restart  
if children > 30 then restart  
#if failed port 25 protocol smtp then restart  
if 5 restarts within 5 cycles then timeout

Checking mysql

check process mysql with pidfile /var/run/mysqld/mysqld.pid  
group www  
start program = `/etc/init.d/mysql start`  
stop program = `/etc/init.d/mysql stop`  
if failed unixsocket /var/run/mysqld/mysqld.sock then restart  
#if failed host 127.0.0.1 port 3306 protocol mysql then restart  
if 5 restarts within 5 cycles then timeout

Checking SSH

check process sshd with pidfile /var/run/sshd.pid  
start program `/etc/init.d/ssh start`  
stop program `/etc/init.d/ssh stop`  
if failed host IP_ADDR port 22 protocol ssh then restart  
if 5 restarts within 5 cycles then timeout

Checking a device by its mount point.

check device rootfs with path /  
if failed permission 755 then alert  
if space usage > 90% for 5 times within 10 cycles then alert  
if inode usage > 90% for 5 times within 10 cycles then alert  
group server

And here’s how you can check your servers over the web:

#host1  
check host server_name with address server_name.ru  
group host  
if failed port 80 protocol http and request `/status.php` for 2 cycles then alert  
#host2  
check host server_name2 with address server_name2.ru  
group host  
if failed port 80 protocol http and request `/status.php` for 2 cycles then alert

Same thing but + searching for a string in the response

check host host with address host3.com  
group host  
CONTENT != `The page you are looking for is temporarily unavailable`  
timeout 60 seconds 3 cycles  
then alert