NGINX: putting multiple services on the same port

Recent versions of NGINX introduce the TLS pre-read capabilities, which allow it to see which TLS protocols are supported by the client, the requested SNI server name, and the ALPN protocol of request. All this information, known before the request is routed by NGINX to the upstream servers, makes it possible to conditionally route requests to different services while using the same port.

In the following example config, we allow connections to SSH, MTProxy, and a TLS encrypted website, all through the same machine, same IP, and the same port, 443.

SSH

The SSH has no relation to TLS encryption, but it can be tought to go through TLS connection using the ProxyCommand option:

ssh username@host.example.com -o "ProxyCommand openssl s_client -alpn identifyssh -ign_eof -quiet -connect host.example.com:443"

Where identifyssh is an arbitrary string we use for ALPN protocol specification that will be picked up in our NGINX config to selectively route requests to actual SSH port on the remote machine, 22.

MTProxy

MTProxy can do “Fake TLS”. Since the client will not advertise itself with a unique ALPN protocol, the way to selectively route traffic for it, as making it the “default” route, while having the web route check the server name requested via SNI.

Propagating remote IP addresses for web logs

Note that the only downside of this configuration is due to MTProxy failure when PROXY protocol V2 is enabled in the traffic director. This yields web logs containing only local address 127.0.0.1.

It would be easily solved if NGINX supported variable value for proxy_protocol directive. However, this is not the case at the time of this writing.

The proxy_bind directive is a solution which allows spoofing remote address all the way throughout NGINX configuration (proxy_bind $remote_addr transparent;). However to make it work, it requires you to configure routing in Linux.

stream {

    # check ALPN for xmpp client or server and redirect to local ssl termination endpoints
    map $ssl_preread_alpn_protocols $upstream_by_proto {
        "identifyssh"     127.0.0.1:8022;
        default           127.0.0.1:8443;
    }

    map $ssl_preread_server_name $upstream {
        "host.example.com" 127.0.0.1:8080;
        default           $upstream_by_proto;
    }

    # Traffic director server
    server {
        listen 443;
        ssl_preread on;
        proxy_ssl off;
        proxy_pass $upstream;
        # proxy_protol breaks mtproxy, it does not support variable
        # see above for the solution
        # proxy_protocol on;
        set_real_ip_from  172.18.0.0/32;
    }

    # TLS termination for SSH connections
    server {
        listen 8022 ssl;
        ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
        ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;
        ssl_dhparam /etc/ssl/certs/dhparam.pem;
        proxy_ssl off;
        proxy_pass 127.0.0.1:22;
    }
}

Actual services running locally:

• MTProxy on port 8443.
• SSH on port 22
• TLS website on port 8080, via listen 8080 ssl;

Helper server blocks:

• The server with listen 443; is our dispatcher. It routes to either actual or helper server block, depending on the constructed $upstream variable value. It pre-reads SSL information from requests via ssl_preread on;, but does not do any TLS encryption by itself (proxy_ssl off;), allowing the upstram configuration whether the TLS encryption needs to be done.
• The server with listen 8022 ssl; is an intermediate for our SSH connections, it makes them “TLS aware”, by doing TLS encryption and then routing request to actual SSH port

References

• The Super User answer which inspired this article