Working with fault-tolerant NFS servers

Had to look into options for running 2 servers where one folder was mounted via NFS onto several other servers.

So there’s 3 web servers and 1 app server holding the files. Between the web servers and the app server, sharing of the /var/www folder is set up via NFS.

An article on how to install and configure NFS.

One app server is pretty weak for a fault-tolerant cluster, so the idea came up to make a second one and set up content sync with lsync

Left to get the web servers to play nice with the second app server automatically.

Needed to build a bike that would automatically mount the folder from the second server if the first one goes down.
This bike has to run on every web server.

Install nmap on the web servers:

yum install nmap -y

On all the servers, edit /etc/sysconfig/nfs and uncomment the following line:

MOUNTD_PORT=892

Next, for convenience, edit /etc/hosts:

192.168.1.134 nfs1  
192.168.1.168 nfs2

From here on I’ll use the server names instead of their ip addresses.

Decided to put everything in the /etc/nfs folder:

mkdir /etc/nfs

Contents of the /etc/nfs/nfs file

/etc/nfs/nfs

#!/bin/bash

usage() {
    echo "usage: $0 (nf1|nf2)"
    exit 1
}

mn1() {
    umount /var/www/html/media
    /sbin/mount.nfs nfs1:/var/www/html/media /var/www/html/media
}

mn2() {
    umount /var/www/html/media
    /sbin/mount.nfs nfs2:/var/www/html/media /var/www/html/media
}

case "$1" in
mn1)
    mn1
    ;;
mn2)
    mn2
    ;;

Take a close look at the script. You may need to adjust the paths.

Make the script executable:

chmod +x /etc/nfs/nfs

Left to make the checker. Contents of the /etc/nfs/checker file - in the spoiler:

/etc/nfs/checker

#!/bin/bash
nfs1_state="active";
nfs2_state="inactive";

while true; do
  #check if nfs1 server is avaialble
  live_nfs1=$(nmap -p 892 nfs1 |grep open |awk '{print $2}');
  live_nfs2=$(nmap -p 892 nfs2 |grep open |awk '{print $2}');

  #Proceed if server is avaialble
  if [[ "$live_nfs1" = "open" ]]; then
    if [[ "$nfs1_state" = "inactive" ]]; then
      /etc/nfs/nfs mn1;
      nfs1_state="active";
      nfs2_state="inactive";
    fi
  sleep 3;
  elif [[ "$live_nfs2" = "open" ]]; then
    nfs1_state="inactive";
    if  [[ "$nfs2_state" = "inactive" ]]; then
      nfs2_state="active";
      /etc/nfs/nfs mn2;
    fi
  sleep 3;
  else
    nfs2_state="inactive";
    sleep 3;
  fi

  echo -e "NFS1: $nfs1_state \nNFS2: $nfs2_state \n";

Make the script executable:

chmod +x /etc/nfs/checker

Just in case, here’s the /etc/init.d/nfs-check script that’ll start on system boot:

/etc/init.d/nfs-check

#!/bin/bash

### BEGIN INIT INFO
# Should-Start:      
# Should-Stop:       
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
### END INIT INFO

# Author: TechNoter

NAME="NFS Checker"
DAEMON="/etc/nfs/checker"

start_nfs_checker() {
  if [ $(ps aux |grep $DAEMON |wc -l) -gt 1 ]; then
    echo "$NAME is running"
  else
    $DAEMON 2>&1 >> /var/log/nfs_checker.log &
    echo "$NAME Started"
  fi
}
stop_nfs_checker() {
  if [ $(ps aux |grep $DAEMON |wc -l) -gt 1 ]; then
    kill -9 $(ps aux |grep $DAEMON | sed -n '1p' |awk '{print $2}')
    echo "Done";
  else
    echo "$NAME not running";
  fi
}

status_nfs_checker() {
  if [ $(ps aux |grep $DAEMON |wc -l) -gt 1 ]
  then
    echo "$NAME is running"
  else
    echo "$NAME not running";
  fi
}

case "$1" in
  start)
    start_nfs_checker
  ;;
  stop)
    stop_nfs_checker
  ;;
  restart)
    stop_nfs_checker && sleep 2 && start_nfs_checker
  ;;
  status)
    status_nfs_checker
  ;;
  *)
    echo "Usage: $0 {start|stop|restart|status}"
esac

Make the script executable and add it to autostart:

chmod +x /etc/init.d/nfs-check chkconfig nfs-check on

The checker above will check which of the servers is alive and mount the needed folder from it.