Varnish and JetPack: “Error 503 Backend fetch failed” in WordPress

Jetpack is quite a popular WordPress plugin and has more than a million active installs. It allows you to use WordPress.com interface to remotely manage a stack of your WordPress blogs. Varnish and Jetpack combination might be tricky.

If you are using Varnish to power up your WordPress blog, you may get “Error 503 Backend fetch failed” error when activating Jetpack.

Let’s figure out what is the reason and how can we fix this (scroll down to solution below if you’re not interested in the issue’s background). We have mentioned generic steps for troubleshooting “Backend fetch failed” error in a separate post.

Troubleshooting Jetpack and Varnish combo

First, open the SSH session for your server with Varnish. Then run varnishlog in order to make recordings of backend requests to a text file:

varnishlog -q "BereqHeader ~ '^Host: .*\.domain.com'" > bereq.log

Deactivate your Jetpack and activate it back again. You will see the error in the browser:

Error 503 Backend fetch failed

Backend fetch failed

Guru Meditation:

XID: 2230744

Take note of XID value in the response. It is Varnish transaction ID.

Cancel out from varnishlog command by typing Ctrl + C in SSH. Open the bereq.log file and search for the XID number in error response.

You will find the backend request information includes something like this:

 -   BogoHeader     Too many headers: Set-Cookie: jetpackS
-   HttpGarbage    "HTTP/1.1%00"
-   BerespStatus   503
-   BerespReason   Service Unavailable
-   FetchError     http format error

This is pretty much self-explanatory. There are too many headers being sent to backend server while activating JetPack.

According to Varnish 4 documentation, there is http_max_hdr parameter, which is:

Maximum number of HTTP header lines we allow in {req|resp|bereq|beresp}.http (obj.http is autosized to the exact number of headers). Cheap, ~20 bytes, in terms of workspace memory.

The default value for the parameter is 64.

Solution

The solution to the issue is to increase the number of headers that Varnish will accept. This is done by tuning the http_max_hdr parameter from the default 64 to 96.

We have to adjust Varnish configuration for this. It’s located in either /etc/varnish/varnish.params, /etc/sysconfig/default or /etc/default/varnish depending on your Linux OS.
You need to find DAEMON_OPTS setting which controls Varnish startup parameters and add parameter for the maximum number of HTTP headers.

Example, before:

DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
             -f ${VARNISH_VCL_CONF} \
             -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
             -p thread_pool_min=${VARNISH_MIN_THREADS} \
             -p thread_pool_max=${VARNISH_MAX_THREADS} \
             - ...

After:

DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
             -f ${VARNISH_VCL_CONF} \
             -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
             -p    http_max_hdr=96    \ #  this is the tuned parameter we added
             -p thread_pool_min=${VARNISH_MIN_THREADS} \
             -p thread_pool_max=${VARNISH_MAX_THREADS} \
             - ...

Other Varnish and JetPack problems

The other common issue with Varnish and JetPack is that you may find that you can’t manage your blog in WordPress.com.
This is due to PHP not able to see the actual server port behind Varnish. Thus XMLRPC (the communication layer between your WordPress and remote service) breaks.

Make sure to checkout more general tips on troubleshooting Backend Fetch Failed error.

To fix for this, you have to create port and protocol mapping to translate proper information over to PHP engine. We provide this configuration as part of our WordPress Setup product (on new servers). Simply purchase a new CentOS 7 VPS with Vultr or similar hosting provider, and we’ll take care of installing perfect server setup and migrate your blog over to it.