1# Multi-VA implementation
2
3Boulder supports a multi-perspective validation feature intended to increase
4resilience against local network hijacks and BGP attacks. It is currently
5[deployed in a production
6capacity](https://letsencrypt.org/2020/02/19/multi-perspective-validation.html)
7by Let's Encrypt.
8
9If you follow the [Development Instructions](https://github.com/letsencrypt/boulder#development)
10to set up a Boulder environment in Docker and then change your `docker-compose.yml`'s
11`BOULDER_CONFIG_DIR` to `test/config-next` instead of `test/config` you'll have
12a Boulder environment configured with two primary VA instances (validation
13requests are load balanced across the two) and two remote VA instances (each
14primary VA will ask both remote VAs to perform matching validations for each
15primary validation). Of course this is a development environment so both the
16primary and remote VAs are all running on one host.
17
18The primary and remote VAs are both the same piece of software, the `boulder-va`
19service ([cmd here](https://github.com/letsencrypt/boulder/tree/main/cmd/boulder-va),
20[package here](https://github.com/letsencrypt/boulder/tree/main/va)).
21The boulder-ra uses [the same RPC interface](https://github.com/letsencrypt/boulder/blob/ea231adc36746cce97f860e818c2cdf92f060543/va/proto/va.proto#L8-L10)
22to ask for a primary validation as the primary VA uses to ask a remote VA for a
23confirmation validation.
24
25Primary VA instances know they are a primary based on the presence of the
26`"remoteVAs"` configuration element. If present it specifies gRPC service
27addresses for other VA instances to use as remotes. There's also a handful of
28feature flags that control how the primary VAs handle the remote VAs.
29
30In the development environment with `config-next` the two primary VAs are `va1.service.consul:9092` and
31`va2.service.consul:9092` and use
32[`test/config-next/va.json`](https://github.com/letsencrypt/boulder/blob/ea231adc36746cce97f860e818c2cdf92f060543/test/config-next/va.json)
33as their configuration. This config file specifies two `"remoteVA"s`,
34`rva1.service.consul:9097` and `va2.service.consul:9098` and enforces
35[that a maximum of 1 of the 2 remote VAs disagree](https://github.com/letsencrypt/boulder/blob/ea231adc36746cce97f860e818c2cdf92f060543/test/config-next/va.json#L44)
36with the primary VA for all validations. The remote VA instances use
37[`test/config-next/va-remote-a.json`](https://github.com/letsencrypt/boulder/blob/ea231adc36746cce97f860e818c2cdf92f060543/test/config-next/va-remote-a.json)
38and
39[`test/config-next/va-remote-b.json`](https://github.com/letsencrypt/boulder/blob/ea231adc36746cce97f860e818c2cdf92f060543/test/config-next/va-remote-b.json)
40as their config files.
41
42There are two feature flags that control whether multi-VA takes effect:
43MultiVAFullResults and EnforceMultiVA. If MultiVAFullResults is enabled
44then each primary validation will also send out remote validation requests, and
45wait for all the results to come in, so we can log the results for analysis. If
46EnforceMultiVA is enabled, we require that almost all remote validation requests
47succeed. The primary VA's "maxRemoteValidationFailures" config field specifies
48how many remote VAs can fail before the primary VA considers overall validation
49a failure. It should be strictly less than the number of remote VAs.
50
51Validation is also controlled by the "multiVAPolicyFile" config field on the
52primary VA. This specifies a file that can contain temporary overrides for
53domains or accounts that fail under multi-va. Over time those temporary
54overrides will be removed.
55
56There are some integration tests that test this end to end. The most relevant is
57probably
58[`test_http_multiva_threshold_fail`](https://github.com/letsencrypt/boulder/blob/ea231adc36746cce97f860e818c2cdf92f060543/test/v2_integration.py#L876-L908).
59It tests that a HTTP-01 challenge made to a webserver that only gives the
60correct key authorization to the primary VA and not the remotes will fail the
61multi-perspective validation.
View as plain text