Setting up a modular multi-host Varnish in NixOS

Our go-to setup for PHP Apps, usually running in Wordpress, is a proxy chain of 

Nginx -> Varnish -> HAProxy -> one or more LAMP backends, 
frequently combined with a Redis.

The Nginx instances accepts all connections and routes them based on the request's contents, e.g. the Host header. It then passes them on to a Varnish which will either return a cached version of the backend's response or pass them down to a HAProxy instance which in turn routes the request to one of the servers in one of it's backends where the request is then finally processed by one of the LAMP instances.

As cache processing requirements vary from customer to customer, each layer of this setup gives us great flexibility to get the specific caching behaviour just right. Common examples of things that require setup-specific fine-tuning include session handling, content paywalls from publishers and newspapers, for example, and large files (assets).

A special case of this setup is sharing the same proxy chain between multiple hosts. With this setup, requests are passed through the same Nginx, Varnish and HAProxy instanced and are then routed to the appropriate backends, usually based on a special X-Backend header that is being set in our Nginx config to indicate the correct backend for that request. With both Nginx and HAProxy, setting up a multi-host configuration is relatively straightforward since they only really serve to route incoming requests in a non-stateful way.

Configuring Varnish on the other hand requires a more careful approach. Since the primary use-case of Varnish in this setup is to cache identical backend responses and serve them for identical incoming requests, restarting the Varnish service should only be done when absolutely necessary since a restart invalidates the entire cache.

Varnish

Our previous approach was to write one singular Varnish config and set per-request caching behaviour based on the X-Backend header set by Nginx. This approach, however, has two major flaws:

1. Since the configuration is contained in one single file, managing it can be very cumbersome. This is especially true when using our deployment tool batou and host-specific settings meant for Varnish are split into multiple git repositories meaning they have to be merged somehow.

2. Adjusting any one host's configuration will require a restart of the Varnish service, causing unrelated hosts to experience degraded performance until the entire cache is warmed up again.

As it turns out, Varnish has a built-in solution for both of these issue: Labels.

A common Varnish configuration looked like this before:

default.vcl:

backend haproxy_site1 {
	.host = "10.0.0.1";
}

backend haproxy_site2 {
	.host = "10.0.0.2";
}

sub vcl_recv {
	if (req.http.X-Backend == "site1") {
		set req.backend_hint = haproxy_site1;
        return (hash);
	} else if (req.http.X-Backend == "site2") {
		set req.backend_hint = haproxy_site2;
		return (hash);
	} else {
		return (synth(404));
	}
}

sub vcl_backend_response {
	if (bereq.http.X-Backend == "site1") {
		set beresp.ttl = 10m;
	} else if (bereq.http.X-Backend == "site2") {
		set beresp.ttl = 10d;
	}
}

With the use of labels, this one file can be split up into three: one toplevel configuration file to select the appropriate handler and one host-specific configuration for each of the two domains. The dummy backend has to be added here since a valid varnish configuration file has to specify at least one backend, otherwise the compiler will complain.

default.vcl:

vcl 4.0;

backend dummy {
	.host = "127.0.0.1:0";
}

sub vcl_recv {
	if (req.http.X-Backend == "site1") {
		return (vcl(label-site1));
	} else if (req.http.X-Backend == "site2") {
		return (vcl(label-site2));
	} else {
		return (synth(404));
	}
}

And then for each of the sites a configuration like the below will do the job.

site1.vcl:

vcl 4.0;

backend haproxy_site1 {
	.host = "10.0.0.1";
}

sub vcl_recv {
   set req.backend_hint = haproxy_site1;
   return (hash);
}

sub vcl_backend_response {
   set beresp.ttl = 10m;
}

Site2's config looks just the same except with slightly different values.

In order to use this kind of split configuration, you have to tell varnish where to find these labels. Since labels are injected at runtime you have to load the labels first, then the configuration file that uses
these labels. Finally, you can tell varnish to switch to the new toplevel config.

$ sudo -u varnish varnishadm
200
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,5.15.166,x86_64,-jnone,-smalloc,-sdefault,-hcritbit
varnish-7.4.3 revision b659b7ae62b44c05888919c5c1cd03ba6eaec681

Type 'help' for command list.
Type 'quit' to close CLI session.

varnish> vcl.load vcl-site1 site1.vcl
200

varnish> vcl.label label-site1 vcl-site1
200

varnish> vcl.load vcl-default default.vcl
200

varnish> vcl.label label-default vcl-default
200

varnish> vcl.use label-default
200
VCL 'label-default' now active
varnish>

Should any of the site's Varnish configuration change, all that needs to be done now is to load the new config, adjust the label and discard the old configuration.

$ sudo -u varnish varnishadm
200
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,5.15.166,x86_64,-jnone,-smalloc,-sdefault,-hcritbit
varnish-7.4.3 revision b659b7ae62b44c05888919c5c1cd03ba6eaec681

Type 'help' for command list.
Type 'quit' to close CLI session.

varnish> vcl.load vcl-site1-new site1-new.vcl
200

varnish> vcl.label label-site1 vcl-site1-new
200

varnish> vcl.discard vcl-site1
200

varnish>

Fortunately, these instructions can be pasted into a file and passed to varnish to execute after starting up or just executed by passing them to varnishadm as arguments. All that's left now is managing all of the configuration files and the scripts to load and activate the labels as well as glueing them together nicely with some systemd magic.

NixOS

With our infrastructure running on NixOS, choosing it's module system to manage the Varnish service is a natural choice. If you don't yet know what NixOS is I encourage you to check out nixos.org.

Nixpkgs already provides a perfectly usable Varnish service, however it is a bit lacking in terms of modularity for our use-case. You can define a few options like the state directory, extra modules to load, address to bind to, a verbatim configuration and extra command line flags. The latter can be used for our previous approach with all backends sharing a single configuration file, however it cannot easiely be used in conjunction with labels. We decided to build on top of this service by utilizing the extraCommandLine option to pass a script to varnish that loads all of the config files in the correct order and adjusting the generated systemd service to load, label, use and discard as necessary in the service's reload section.

The final design is reminiscent of the Nginx module with it's vhost-like approach to declaring the Varnish configuration. The above example could be implemented as

{
   flyingcircus.services.varnish.virtualHosts = {
      "site1" = {
         condition = "req.http.X-Backend == \"site1\"";
         config = ''
            vcl 4.0;

            backend haproxy_site1 {
               .host = "10.0.0.1";
            }

            sub vcl_recv {
               set req.backend_hint = haproxy_site1;
               return (hash);
            }

            sub vcl_backend_response {
               set beresp.ttl = 10m;
            }
         '';
      };

      "site2" = {
         condition = "req.http.X-Backend == \"site2\"";
         config = ''
            vcl 4.0;

            backend haproxy_site2 {
               .host = "10.0.0.2";
            }

            sub vcl_recv {
               set req.backend_hint = haproxy_site2;
               return (hash);
            }

            sub vcl_backend_response {
               set beresp.ttl = 10d;
            }
         '';
      };
   };
}

The module will then take care of generating a script for varnishadm to load and label all configuration files, generating a toplevel configuration file that passes requests down to the correct label based on each host's condition as well as handling changes to each configuration, loading the new configuration and relabeling it to activate the new configuration without having to restart varnish. As an added bonus, it will also attempt to compile each configuration at build-time in order to prevent accidentally switching to an invalid configuration.

Another big benefit of using the NixOS module system here is it's possibility to merging configuration such that the above can be split up into a file for each virtualHost such that multiple sites' configuration can be deployed to the same VM running Varnish without having to jump through hoops to properly merge them all manually.