Varnish and Brotli done right

What is Brotli

Brotli is the brand new compression method for the Internet. And what’s great about it (besides being superior to Gzip) is that it was developed by Google. (For some people like me – not that great. How many things has Google abandoned? But let’s be positive since Brotli is open source)

Many website owners want to improve website performance by enabling Brotli compression.

Brotli already has wide browser support. Even Microsoft’s IE/Edge started supporting it as recently as this year.

Chrome has been supporting it since early days of Brotli library.

Brotli in Varnish

Varnish doesn’t really support Brotli encoding internally. But being agnostic to whatever goes through it, Varnish actually supports any encoding as long as you add the necessary VCL code.

Varnish software (Reza Naghibi) has posted an article on enabling Brotli with Varnish. However, it lacks some code for production use. The code doesn’t fully account for purging cached objects.

Normally when you clear cache, you want the cache for both gzipped and Brotli versions of the page to be cleared. However, the original code doesn’t account for this.

Let’s introduce the missing bits of VCL for purging. We use original logic and issue additional purge request in vcl_purge. That way we clear cache for both regular and brotli caches of the same page. Let’s go:

sub vcl_recv {
    if(req.http.Accept-Encoding ~ "br" && req.url !~
            "\.(jpg|png|gif|gz|mp3|mov|avi|mpg|mp4|swf|wmf)$") {
        set req.http.X-brotli = "true";
    }
}

# The data on which the hashing will take place
sub vcl_hash {
    if(req.http.X-brotli == "true" && req.http.X-brotli-unhash != "true") {
        hash_data("brotli");
    }
}

sub vcl_backend_fetch {
    if(bereq.http.X-brotli == "true") {
        set bereq.http.Accept-Encoding = "br";
        unset bereq.http.X-brotli;
    }
}

sub vcl_purge {
    # repeat purge for brotli or gzip object 
    # (force hash/no hash on "brotli" while doing another purge)
    # set Accept-Encoding: gzip so that we don't get brotli-encoded response upon purge
    if (req.url !~ "\.(jpg|png|gif|gz|mp3|mov|avi|mpg|mp4|swf|wmf)$" && 
            !req.http.X-brotli-unhash) {
        if (req.http.X-brotli == "true") {
            set req.http.X-brotli-unhash = "true";
            set req.http.Accept-Encoding = "gzip";
        } else {
            set req.http.X-brotli-unhash = "false";
            set req.http.Accept-Encoding = "br";
        }        
        return (restart);
    } 
}

So as you can see, we are doing a VCL trick to achieve our goal (purging cache for both brotli and regular cache).

If the purge request was done by a client that supports Brotli, we repeat purge request via “restart”. While we are doing that, we make sure that the second purge request will clear cache for the regular (gzip) cache entry.

In another case, when purge request is done by a client that doesn’t support Brotli, we are doing “restart” also. But we issue purge request for Brotli cache object of the page this time. The vcl_hash function is modified to account for our conditional purge logic.

We’re also doing a check against special variable req.http.X-brotli-unhash being empty so that we don’t repeat purge request more than once.

Well, that’s about it for production ready Brotli support.

We leave it to reader’s exercise to code VCL for Brotli using Vary approach instead. (That’s what we ended up doing for Citrus servers).