Getting HTTPS client IP addresses from HAProxy (frontend) to Nginx (backend) in HTTP and TCP balancing modes

Foreword

I just couldn’t pass by this article on habrahabr.ru

Overview

Quite often you need to balance load across several web servers. Usually you also need the web applications to get the clients’ real IP addresses, not the balancer’s IP.

When you balance and terminate HTTP(S) traffic on HAProxy (Layer 7 [1]), this task is easy to solve by adding an “X-Real-IP” header and handling it on Nginx with the ngx_http_realip_module module [2]. When balancing TCP traffic from HTTPS clients and passing it straight to the web servers without modification or termination (Layer 4 [3]), adding this header isn’t possible, so you need to use what Proxy Protocol gives you [4, 5, 6].

Let’s look at both options (L7 and L4 balancing) using excerpts from haproxy 1.5.9 and nginx 1.6.2 config files

Application-layer balancing (Layer 7): terminating HTTPS traffic on HAProxy and passing it over HTTPS to Nginx

In this example, HTTPS traffic from the client is terminated on HAProxy, modified, and passed on to Nginx over HTTPS as well.

haproxy.cfg

global
  maxconn 4096
  chroot /usr/share/haproxy
  uid 99
  gid 99
  daemon
  tune.ssl.default-dh-param  2048

defaults
  log     global
  option  redispatch
  option  tcp-smart-accept
  option  tcp-smart-connect
  retries 3
  maxconn 2000
  timeout connect 5000
  timeout check   3000
  timeout client  50000
  timeout server  50000

frontend http_frontend *:80
  mode http
  redirect scheme https code 301 if !{ ssl_fc }

frontend https_frontend_ssl_terminate
  mode http
  bind *:443 ssl crt /etc/haproxy/ssl/public.example.com.pem
  option forwardfor header >X-Real-IP>
  default_backend web_server_http

backend web_server_http
  mode http
  balance roundrobin
  # Send traffic to the backend over HTTPS
  server s1_https 192.168.1.10:443 ssl verify none
  server s2_https 192.168.1.20:443 ssl verify none

nginx.conf

server {
  server_name localhost;

  listen 443 ssl default_server;

  ssl_certificate      /etc/nginx/ssl/internal.example.com.pem;
  ssl_certificate_key  /etc/nginx/ssl/internal.example.com.key;

  # HAProxy address
  set_real_ip_from >192.168.1.254>;
  real_ip_header >X-Real-IP>;

  root /usr/share/nginx/html;
  index index.html index.htm;

  error_page 500 502 503 504 /50x.html;
  location = /50x.html {
      root /usr/share/nginx/html;
  }

  location ~ /\.ht {
      deny all;
  }
}

Transport-layer balancing (Layer 4): passing TCP traffic from HAProxy to Nginx

In this example, clients’ HTTPS traffic isn’t modified (HAProxy only touches the transport layer) and termination happens directly on Nginx.

haproxy.cfg

global
  maxconn 4096
  chroot /usr/share/haproxy
  uid 99
  gid 99
  daemon

defaults
  log     global
  option  redispatch
  option  tcp-smart-accept
  option  tcp-smart-connect
  retries 3
  maxconn 2000
  timeout connect 5000
  timeout check   3000
  timeout client  50000
  timeout server  50000

frontend http_frontend *:80
  mode http
  redirect scheme https code 301 if !{ ssl_fc }

frontend https_frontend_ssl_pass
  mode tcp
  bind *:443
  default_backend web_server_tcp

backend web_server_tcp
  mode tcp
  balance roundrobin
  # WARNING! send-proxy only works
  # when the receiving side understands what it is.
  # For Nginx you need to enable the
  # proxy_protocol option in the listen directive.
  server s1_tcp 192.168.1.10:443 >send-proxy>
  server s2_tcp 192.168.1.20:443 >send-proxy>

nginx.conf

server {
  server_name localhost;

  # WARNING! The proxy_protocol directive only works paired with haproxy.
  # For direct access you need to disable this directive.
  listen 443 ssl default_server >proxy_protocol>;

  ssl_certificate      /etc/nginx/ssl/public.example.com.pem;
  ssl_certificate_key  /etc/nginx/ssl/public.example.com.key;

  # HAProxy address
  set_real_ip_from >192.168.1.254>;
  real_ip_header >proxy_protocol>;

  root /usr/share/nginx/html;
  index index.html index.htm;

  error_page 500 502 503 504 /50x.html;
  location = /50x.html {
      root /usr/share/nginx/html;
  }

  location ~ /\.ht {
      deny all;
  }
}

Conclusion

Using the settings described above we managed to pass the real client IP addresses over HTTPS to the Nginx web server sitting behind HAProxy. You can use the same approach with third-party load balancers too, for example CloudFlare [7, 8] and AWS ELB [9, 10].