Load balancing with NginX

Hello, dear reader. In this article I want to describe how to set up NginX for load balancing across several back-end servers, say Apache.

So here’s the proposed scheme (the image is clickable):

Two NginX directives help with this fairly simple task:

upstream - a directive that comes with the HttpUpstream module and lets you balance load across multiple servers.

proxypass - a directive that comes with the HttpProxy module. It correctly sends/proxies requests to the servers behind the balancer.

Let’s look at an example. There are 3 web heads running the same site:

Apapche#1: 192.168.1.1
Apapche#2: 192.168.1.2
Apapche#3: 192.168.1.3

In your favorite text editor, create the nginx config file and add the following lines. Personally, I like keeping site configs in separate files.

upstream backend  {
        server 192.168.1.1 weight=5 max_fails=2 fail_timeout=4;
        server 192.168.1.2 weight=5 max_fails=2 fail_timeout=4;
        server 192.168.1.3 weight=5 max_fails=2 fail_timeout=4;
}

• weight - defines the server’s weight in the cluster. In this example all servers are equal and can handle the same number of requests. If one server in the cluster is significantly more powerful than the rest, you can set a higher value for it and NginX will send it more requests than the others.
• max_fails - defines the number of failed connection attempts to the backend server.
• fail_timeout - the interval between failed connections.

In this example, after 2 failed connections within 4 seconds, the server will be marked unavailable and requests won’t be sent to it until NginX confirms it’s fine again.

Load balancing methods (specified at the start of the upstream section):

• ip_hash - with this method, requests from the same client always go to the same backend server, based on the client’s ip address. Not compatible with the weight parameter.
• least_conn - requests go to the server with the fewest active connections.
• round-robin - the default mode. If you haven’t set any of the methods above, requests are delivered to all servers in turn, equally.

So we’ve described the servers with the site. Next we need to tell NginX what to do with them. For that we describe the location section with request proxying parameters:

location / {
        proxy_pass http://backend;
        proxy_set_header X-Real-IP  $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
}

So the full config for the cluster looks like this:

upstream backend  {
        server 192.168.1.1 weight=5 max_fails=2 fail_timeout=4;
        server 192.168.1.2 weight=5 max_fails=2 fail_timeout=4;
        server 192.168.1.3 weight=5 max_fails=2 fail_timeout=4;
}

server {
        listen 80;
        servername www.somesite.com;

        location / {
                proxy_pass http://backend;
                proxy_set_header X-Real-IP  $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_redirect off;
        }
}

A big perk of load balancing with NginX is SSL Termination support. That means secure sessions are established with the balancer itself, and clients get their response from it too. There are two ways to set this up:

1. Similar to the previous example, we set up balancing of https sessions. In this case ssl hosts must also be described on the Apache servers. Here https connections are established with NginX, which then builds a new packet, establishes a secure connection with the Apache servers, gets a similarly secure packet back, unpacks it, builds a new one and sends the final result to the client encrypted. A pretty involved process, you’ll agree. And very resource-heavy performance-wise.

In this case our config looks like this:

upstream backend  {
        server 192.168.1.1:443 weight=5 max_fails=2 fail_timeout=4;
        server 192.168.1.2:443 weight=5 max_fails=2 fail_timeout=4;
        server 192.168.1.3:443 weight=5 max_fails=2 fail_timeout=4;
}

server {
        listen 443;
        servername www.somesite.com;

        ssl on;
        ssl_certificate /etc/nginx/ssl/somesite.crt;
        ssl_certificate_key /etc/nginx/ssl/somesite.key;

        location / {
                proxy_pass https://backend;
                proxy_set_header X-Real-IP  $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_redirect off;
        }
}

1. We set up an https host in NginX and describe http (without the S) connections in the upstream list. A bit of context: many CMS systems have built-in checks for secure sessions and forcibly redirect clients to a secure connection on pages with sensitive data (user login, payment pages). This is usually done by checking for an HTTPS header with value on in the SERVER environment variable. You can just add this header to requests to the Apache servers and avoid the extra load. If this doesn’t work for you, see option 1.

In this case our config looks like this:

upstream backend  {
        server 192.168.1.1 weight=5 max_fails=2 fail_timeout=4;
        server 192.168.1.2 weight=5 max_fails=2 fail_timeout=4;
        server 192.168.1.3 weight=5 max_fails=2 fail_timeout=4;
}

server {
        listen 443;
        servername www.somesite.com;

        ssl on;
        ssl_certificate /etc/nginx/ssl/somesite.crt;
        ssl_certificate_key /etc/nginx/ssl/somesite.key;

        location / {
                proxy_pass http://backend;
                proxy_set_header X-Real-IP  $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                <strong>proxy_set_header HTTPS on;</strong>
                proxy_redirect off;
        }
}

You can help your Apache servers even more and take unnecessary load off them by serving static files with NginX. For that, copy all folders with static content (keeping the directory tree) to the load balancer itself (I’d put them in /var/www/html) and describe serving them via NginX right before the location / section in each host’s config:

location ~* \.(jpg|jpeg|gif|png|css|js|ico|xml)$ {
        access_log        off;
        log_not_found     off;
        expires           360d;
        root /var/www/html;
}

To make sure static content updates properly when uploaded via the web, you can mount the corresponding folders on the servers with NFS.

Something like that.