Varnish: do we have to cache static files?

One of the best performance stacks out there is Varnish coupled with NGINX with PHP-FPM. It is a so-called NGINX sandwich:

• NGINX works as the TLS termination software, handing encryption
• Varnish Cache is the Full Page Cache layer
• NGINX with PHP-FPM actually handle requests

Do we need to cache static files in Varnish while using this stack?
Does it need to cache static files that are already fast to deliver?

&tldr;

• Single server – no, don’t cache and don’t even serve through Varnish, serve directly!
• Multiple servers – yes, cache, of course!

Let’s talk about the best approaches to deliver static files. How can we make the static files’ delivery fastest possible?

The perfect answer would be based on your specific setup.

Varnish Cache is a great caching solution to speed up any website.
NGINX is a very fast web server, capable of a lot of things, and further extensible via modules.

We can’t use one without the other, because Varnish Cache is neither a webserver nor capable of TLS termination.
At least so, in its open-source version.

But we stack NGINX + Varnish, then add another NGINX just for the TLS termination, there is an unnecessary buffering overhead between them:

User’s browser request -> NGINX (TLS) -> (buffer) -> Varnish -> (buffer) -> NGINX + PHP-FPM

What buffering means here, is that instead of synchronously streaming response from Varnish to the browser,
the TLS terminating NGINX first accumulates a portion of the response, before ever releasing it to the browser.

This behavior is by default and can be tweaked.
But even the ultimate tweak won’t be capable of disabling buffering completely.
Thus, stacking up software like this will inevitably lead to a minor performance impact.

Let’s see how to make it further negligible.

For simplicity, we will make some assumptions below, that you’re:

• running a website, which enforces TLS encryption (requires https:// URLs), quite common/recommended nowadays
• prefer the canonical domain of your website to have the www. prefix

Single server setup

If we run a website and accelerate it with Varnish, all the requests would typically go through Varnish.
This is sub-optimal for static files. A request to an image files hits NGINX (or other TLS termination software), then Varnish, then NGINX again.

There is quite an unnecessary buffering overhead between NGINX and Varnish in this commonly used setup.

But when every piece of your stack is hosted on a single machine, the best way to go around static files is simply serving them directly by NGINX, without passing through Varnish. How?

This can be accomplished by either whitelist or blacklist approach, which will specify what goes through Varnish and what not.

There is a special case though, which is passing everything through Varnish.
Where it makes sense is when you want to enforce highest compression (gzip, brotli level) in NGINX, and then Varnish caching it.
Thus you get the smallest assets while preserving CPU to compress them only once.

However, it is best to implement your own workflow for compressing assets once, instead of relying on Varnish Cache.

So caching everything in Varnish on a single server is out of the question. You shouldn’t, in most cases.

Varnish whitelist approach (worse)

We want to improve request latency by not passing static files through Varnish.
One way this can be accomplished is by specifying what needs to go through Varnish.
Everything else to will default to be served by NGINX directly.
This is good when you don’t know the exact locations of all static files (your typical, messy custom-built website):

# This is the NGINX server block that does TLS termination
server {
    listen 443 ssl http2;
    server_name www.example.com;
    root /srv/www/example.com/public;
    location / {
        try_files $uri $uri/ /index.php$is_args$args;
        # ...
    }
    location ~ \.php$ {
        proxy_pass 127.0.0.1:6081;
        # ...
    }
}
# This is the NGINX server block that is leveraged by Varnish as its backend
server {
    listen 8080;
    server_name www.example.com;
    root /srv/www/example.com/public;
    location / {
        include fastcgi_params;
        fastcgi_pass unix:/var/run/php-fpm/example.com.sock;
    }
}

In this configuration sample, we ensure that only non-static file requests are passed through Varnish.
The requests that go through Varnish will be the heavy PHP requests, which are so great to cache with Varnish.

When a request lands in the “TLS terminating” server block, we use try_files to check if there’s an actual static/PHP file.
If there is no file at all, or it is a PHP file, we forward the request through Varnish caching layer, thus accelerating those slower requests.

In other cases, that would be a request to a static file, and it is served directly from the “TLS terminating” server, without any buffering overhead of stacking NGINX with Varnish.

We haven’t eliminated the try_files since we don’t know whether a given URI is static or not (mind the messy website assumption).

Varnish blacklist approach (best)

To improve the previous config, we can specify which locations are to be served directly by NGINX, by simply allocating them (can be empty blocks) in the NGINX configuration.
This will ensure that those specific locations are not going through Varnish at all.

Again, this is great because we won’t have to deal with unnecessary buffering that has to take place when you stack up NGINX with Varnish.

# This is the NGINX server block that does TLS termination
server {
    listen 443 ssl http2;
    server_name www.example.com;
    root /srv/www/example.com/public;
    location / {
        proxy_pass 127.0.0.1:6081;
        # ...
    }
    location /static/ {
        # could be empty, but the immutable will ensure far future expiration headers
        immutable on;
    }
}
# This is the NGINX server block that is leveraged by Varnish as its backend
server {
    listen 8080;
    server_name www.example.com;
    root /srv/www/example.com/public;
    location / {
        include fastcgi_params;
        fastcgi_pass unix:/var/run/php-fpm/example.com.sock;
    }
}

This approach is great when you definitely know the locations of your static files, which is the case for a well-structured CMS framework.

We allocate a known /static/ location (will vary per CMS, of course), so that proxy_pass will not be applicable to it.
Thus any request under /static/ URI will not go through Varnish.

As you see, we jumped a step further there in our optimizations, and got rid of try_files performance bottleneck.
There is no need for try_files on a single-entry (bootstrap) CMS framework, and we know the request is for PHP, so why not pass it directly?

A note on TLS redirection

To make our example configuration more complete, we can specify some redirect server blocks which will ensure TLS encryption:

# This is the NGINX redirection block from unencrypted 80 to encrypted 443 (non-www to non-www)
server {
    listen 80;
    server_name example.com;
    return 301 https://example.com$request_uri;
}
# This is the NGINX redirection block from unencrypted 80 to encrypted 443 (www to www)
server {
    listen 80;
    server_name www.example.com;
    return 301 https://www.example.com$request_uri;
}
# This is the NGINX redirection block from encrypted non-www to www
server {
    listen 443 ssl http2;
    server_name example.com;
    return 301 https://www.example.com$request_uri;
}

Having TLS redirection done in NGINX is far cleaner than having it done in Varnish.
The NGINX configuration is declarative as opposed to being procedural as in Varnish.

Why so many redirects? This is required for the proper application of strict transport security.

Multiple servers

In virtually any situation when Varnish is hosted on a server that is separate from the actual files, you will benefit from caching static files in Varnish.

By doing this, you will eliminate the network latency between your Varnish server and its backend NGINX server.

A typical case of having Varnish and NGINX in separate servers is when you’re building a DYI CDN:

• Main server in UK: NGINX + PHP-FPM
• CDN edge server #1 in UK: NGINX (TLS) + Varnish
• CDN edge server #2 in US: NGINX (TLS) + Varnish

Requests would be routed geographically to either of the CDN edge servers.

If a request for a static file arrives to the US server, and Varnish on it is not set to cache static files, there will be uneccessary latency/delay experience by the US visitors,
simply because they will have to wait for the US-to-UK server communication to complete. This may not sound like a lot of wait. But mind that any modern website nowadays consists of dozens of static files.
Those milliseconds will add up to something very perceptible to the end users.

So having the static files cached in Varnish, in the scenario, makes a lot of sense.
Although you should mind that this will end up to be “pull CDN”, because Varnish on the edges will have to request the files from the main server first.

The first requests will be slow.

An alternative would be creating a “push CDN”, which would be rather complex because you’ll have to, e.g. rsync the static files
to each edge server. Only then caching static files will be no longer required.

A smart way to cache static files with Varnish

So we have concluded that caching static files in Varnish is only beneficial in a multi-server stack, like a self-made CDN.
When we cache static files in Varnish, there will be a performance improvement.

But the edge servers are typically very small VPS instances. How can we leverage all their resources: disk and RAM, for Varnish cache storage?

We can make a smart move and use multiple storage backends by partitioning Varnish cache:

• cache static files onto the disk
• cache everything else onto RAM

This way we don’t waste RAM for storing static files. The cache is split into two storages: RAM, for HTML and disk, for static files.

To make this happen we need to update Varnish configuration. Assuming that we run Varnish 4 on a CentOS/RHEL machine, we have to update two files.

Varnish configuration: /etc/sysconfig/varnish

This file contains storage definition, and we have to replace it and specify 2 storages for our cache: “static” and “default”.

Find -s ${VARNISH_STORAGE}

and replace with:

-s default=malloc,256m \
-s static=file,/var/lib/varnish/varnish_storage.bin,1G"

So here we have a 1GB of disk storage dedicated to caching static files cache and 256MB for the full page cache, in RAM.

VCL configuration file

We have to tell Varnish which responses should go into which cache, so in our .vcl file we have to update vcl_backend_response routine, like this:

sub vcl_backend_response {

        # For static content strip all backend cookies and push to static storage
        if (bereq.url ~ "\.(css|js|png|gif|jp(e?)g)|swf|ico") {
                unset beresp.http.cookie;
                set beresp.storage_hint = "static";
                set beresp.http.x-storage = "static";
        } else {
                set beresp.storage_hint = "default";
                set beresp.http.x-storage = "default";
        }

}

As simple as that: if files match static resources, we cache into static storage (which is on the disk), whereas everything else (most notably html pages) is cached into RAM.

Now we can observe the two storages filling up by running sudo varnishstat. You will find SMA.default and SMF.static groups:

Conclusion

Now we know how to go around static files in the NGINX sandwich + Varnish setups.

Surely, the NGINX sandwich is not the only possible way to accomplish your perfect web setup.
You can resort to Hitch for TLS termination and even use UNIX sockets for inter-process communication which would make the issue of buffering less important.