Skip to content

Speed up web delivery with Nginx and TFO

Published by tex on April 2, 2017

What is TFO ?

In computer networking, TCP Fast Open (TFO) is an extension to speed up the opening of successive Transmission Control Protocol (TCP) connections between two endpoints. It works by using a TFO cookie (a TCP option), which is a cryptographic cookie stored on the client and set upon the initial connection with the server. When the client later reconnects, it sends the initial SYN packet along with the TFO cookie data to authenticate itself. If successful, the server may start sending data to the client even before the reception of the final ACK packet of the three-way handshake, skipping that way a round-trip delay and lowering the latency in the start of data transmission.

The cookie is generated by applying a block cipher keyed on a key held secret by the server to the client’s IP address, generating an authentication tag that is difficult for third parties to spoof, even if they can forge a source IP address or make two-way connections to the same server from other IP addresses. Although it uses cryptographic techniques to generate the cookie, TFO is not intended to provide more security than the three-way handshake it replaces, and does not give any form of cryptographic protection to the resulting TCP connection, or provide identity assurance about either endpoint. It also is not intended to be resistant to man-in-the-middle attacks.

Let’s start with sysctl.

root@ns:~# sysctl -w net.ipv4.tcp_fastopen=3
root@ns:~# echo "net.ipv4.tcp_fastopen=3" >> /etc/sysctl.conf

I run Debian on my server with default Nginx installed (nginx-extras), but always I keep the latest version of Nginx. On update process, I just replace the binary after compilation to keep the default init scripts. For all temporary files and cache, I use shm.

Nginx download, extract files from archive, download “headers-more-nginx-module” (I use this)

root@ns:~/work# wget http://nginx.org/download/nginx-1.11.12.tar.gz
--2017-04-02 23:15:36--  http://nginx.org/download/nginx-1.11.12.tar.gz
Resolving nginx.org (nginx.org)... 206.251.255.63, 95.211.80.227, 2606:7100:1:69::3f, ...
Connecting to nginx.org (nginx.org)|206.251.255.63|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 979963 (957K) [application/octet-stream]
Saving to: ‘nginx-1.11.12.tar.gz’

nginx-1.11.12.tar.gz     100%[==========================>] 957.00K  4.48Mb/s   in 1.8s

root@ns:~/work# tar zxf nginx-1.11.12.tar.gz
root@ns:~/work# cd nginx-1.11.12/
root@ns:~/work/nginx-1.11.12#

root@ns:~/work/nginx-1.11.12# git clone https://github.com/openresty/headers-more-nginx-module
Cloning into 'headers-more-nginx-module'...
remote: Counting objects: 1354, done.
remote: Total 1354 (delta 0), reused 0 (delta 0), pack-reused 1354
Receiving objects: 100% (1354/1354), 488.20 KiB | 0 bytes/s, done.
Resolving deltas: 100% (725/725), done.
Checking connectivity... done.

Building from source with tfo (compiler flag)

root@ns:~/work/nginx-1.11.12# ./configure \
> --conf-path=/etc/nginx/nginx.conf \
> --sbin-path=/usr/sbin \
> --error-log-path=/var/log/nginx/error.log \
> --with-threads \
> --with-stream \
> --with-stream_geoip_module \
> --with-stream_ssl_module \
> --with-http_image_filter_module \
> --with-stream_geoip_module \
> --with-pcre \
> --with-http_mp4_module \
> --with-http_secure_link_module \
> --with-http_v2_module \
> --with-http_flv_module \
> --add-module=headers-more-nginx-module \
> --with-http_geoip_module \
> --with-http_gzip_static_module \
> --with-http_stub_status_module \
> --with-http_ssl_module \
> --http-proxy-temp-path=/dev/shm/proxy_temp \
> --http-client-body-temp-path=/dev/shm/client_body_temp \
> --http-fastcgi-temp-path=/dev/shm/fastcgi_temp \
> --http-uwsgi-temp-path=/dev/shm/uwsgi_temp \
> --http-scgi-temp-path=/dev/shm/scgi_temp \
> --build="v1.11.12 with TFO - UnixTeacher" \
> --with-cc-opt='-O2 -fstack-protector-strong -DTCP_FASTOPEN=23'

# .... output removed
Configuration summary
  + using threads
  + using system PCRE library
  + using system OpenSSL library
  + using system zlib library

  nginx path prefix: "/usr/local/nginx"
  nginx binary file: "/usr/sbin"
  nginx modules path: "/usr/local/nginx/modules"
  nginx configuration prefix: "/etc/nginx"
  nginx configuration file: "/etc/nginx/nginx.conf"
  nginx pid file: "/usr/local/nginx/logs/nginx.pid"
  nginx error log file: "/var/log/nginx/error.log"
  nginx http access log file: "/usr/local/nginx/logs/access.log"
  nginx http client request body temporary files: "/dev/shm/client_body_temp"
  nginx http proxy temporary files: "/dev/shm/proxy_temp"
  nginx http fastcgi temporary files: "/dev/shm/fastcgi_temp"
  nginx http uwsgi temporary files: "/dev/shm/uwsgi_temp"
  nginx http scgi temporary files: "/dev/shm/scgi_temp"
root@ns:~/work/nginx-1.11.12# make -j8
# .... output removed

Testing

root@ns:~/work/nginx-1.11.12# objs/nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# Looks ok.

I will stop Nginx, copy objs/nginx in /usr/sbin/ and start again with the new version

root@ns:~/work/nginx-1.11.12# /etc/init.d/nginx stop && cp objs/nginx /usr/sbin/ && /etc/init.d/nginx start

Everything looks ok.

root@ns:~# nginx -V
nginx version: nginx/1.11.12 (v1.11.12 with TFO - UnixTeacher)
built by gcc 4.9.2 (Debian 4.9.2-10) 
built with OpenSSL 1.0.1t  3 May 2016
TLS SNI support enabled
configure arguments: --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/sbin --error-log-path=/var/log/nginx/error.log 
--with-threads --with-stream --with-stream_geoip_module --with-stream_ssl_module --with-http_image_filter_module 
--with-stream_geoip_module --with-pcre --with-http_mp4_module --with-http_secure_link_module --with-http_v2_module 
--with-http_flv_module --add-module=headers-more-nginx-module --with-http_geoip_module --with-http_gzip_static_module 
--with-http_stub_status_module --with-http_ssl_module --http-proxy-temp-path=/dev/shm/proxy_temp 
--http-client-body-temp-path=/dev/shm/client_body_temp --http-fastcgi-temp-path=/dev/shm/fastcgi_temp 
--http-uwsgi-temp-path=/dev/shm/uwsgi_temp --http-scgi-temp-path=/dev/shm/scgi_temp 
--build='v1.11.12 with TFO - UnixTeacher' --with-cc-opt='-O2 -fstack-protector-strong -DTCP_FASTOPEN=23'

Let’s add fastopen option to listen directive.

Configuration example:

listen 0.0.0.0:80 fastopen=500;

You can also specify rcvbuf, sndbuf and backlog. For https websites, you can specify http2 also.

Examples:

listen 0.0.0.0:80 rcvbuf=64000 sndbuf=200000 backlog=2048 fastopen=500;
listen 0.0.0.0:443 rcvbuf=64000 sndbuf=200000 backlog=2048 ssl http2 fastopen=500;

Test it again after modifications and reload

root@ns:~# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
root@ns:~# /etc/init.d/nginx reload
[ ok ] Reloading nginx configuration (via systemctl): nginx.service.

References:

– https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt
– https://en.wikipedia.org/wiki/TCP_Fast_Open

Published inLinuxWebservers