Varnish static files stream

As mentioned in earlier post, one of the best performance stacks out there is Varnish coupled with Nginx. To make Varnish deliver static files efficiently, we looked at an approach to cache static files in Varnish.

But what about large static files? If we have downloadable .zip or .rar files, or we want users of site to listen to MP3 songs, it is not wise to store those in Varnish cache, because we would waste too much disk space.

Luckily, Varnish 4 comes with the streaming feature. When streaming is used, Varnish will deliver requested file from the backend (Nginx), byte by byte. It will start the transfer without waiting to receive complete file from Nginx. So our goal is to reduce caching of static files only to smaller static files, and deliver larger files via streaming. Using the earlier snippet for caching static files, we need to add a couple of additional statements inside .vcl file used.

First, tell Varnish not to cache the large files:

sub vcl_recv {
…
        if (req.url ~ "^[^?]*\.(mp[34]|rar|rpm|tar|tgz|gz|wav|zip|bz2|xz|7z|avi|mov|ogm|mpe?g|mk[av]|webm)(\?.*)?$")            {
            unset req.http.Cookie;
            return (hash);
        }
}
…

When fetching the file from Nginx, open static files stream:

sub vcl_backend_response {
…
       if (bereq.url ~ "^[^?]*\.(mp[34]|rar|rpm|tar|tgz|gz|wav|zip|bz2|xz|7z|avi|mov|ogm|mpe?g|mk[av]|webm)(\?.*)?$") {
          unset beresp.http.set-cookie;
          set beresp.do_stream = true;  
       }

With this simple technique, we’ve enabled streaming and saved disk space by caching only HTML, CSS, Javascript and other smaller site assets.