1Ambex: Ambassador Experimental^H^H^H^H^H^H^H^H^H^H^H^H ADS service
2=============================
3
4Ambassador prior to v0.50.0 worked by writing out Envoy v1 JSON
5configuration files, then triggering a hot restart of Envoy. This
6works, but it has some unpleasant limitations:
7
8- Restarts take awhile, and as a result you can't change the
9 configuration very quickly.
10- Restarts can drop connections (cf Envoy #2776; more on this later).
11- Envoy itself is deprecating the v1 configuration, and will only
12 support v2 in a bit.
13
14To get around these limitations, and generally go for a better
15experience all 'round, we want to switch to the so-called `xDS` model,
16in which Envoy's configuration is supplied by its various "*D*iscovery
17*S*ervices": e.g. the `CDS` is the Cluster Discovery Service; the
18`EDS` is the Endpoint Discovery Service. For Ambassador, the
19Aggregated Discovery Service or `ADS` is the one we want to use --
20basically, it brings the other services together under one aegis and
21lets you just tell Envoy "get everything dynamically."
22
23However, the whole `ADS` thing is a bit of a pain:
24
25- Envoy makes a bidirectional gRPC stream to the `ADS` server.
26- The `ADS` then makes gRPC calls _to the Envoy_ to feed the Envoy
27 configuration elements, but:
28- The `ADS` has to carefully order things such that the configuration
29 elements match what Envoy expects for consistency.
30
31Rather than do all that logic by hand, we'll use the Envoy
32`go-control-plane`[^1] for the heavy lifting. This is also something of a
33pain, given that it's not well documented, but here's the deal:
34
35- The root of the world is a `SnapshotCache`:
36 - `import github.com/datawire/ambassador/pkg/envoy-control-plane/cache`,
37 then refer to `cache.SnapshotCache`.
38 - A collection of internally consistent configuration objects is a
39 `Snapshot` (`cache.Snapshot`).
40 - `Snapshot`s are collected in the `SnapshotCache`.
41 - A given `SnapshotCache` can hold configurations for multiple
42 Envoys, identified by the Envoy `nodeID`, which must be configured
43 for the Envoy.
44
45- The `SnapshotCache` can only hold `go-control-plane` configuration
46 objects, so you have to build these up to hand to the
47 `SnapshotCache`.
48
49- The gRPC stuff is handled by a `Server`:
50 - `import github.com/datawire/ambassador/pkg/envoy-control-plane/server`,
51 then refer to `server.Server`.
52 - Our `runManagementServer` function (largely ripped off from the
53 `go-control-plane` tests) gets this running. It takes a
54 `SnapshotCache` (cleverly called `config` for no reason I (Flynn)
55 understand) and a standard Go `gRPCServer` as arguments.
56 - _ALL_ the gRPC madness is handled by the `Server`, with the
57 assistance of the methods in its `callback` object.
58
59- Once the `Server` is running, Envoy can open a gRPC stream to it.
60 - On connection, Envoy will get handed the most recent `Snapshot`
61 that the `Server`'s `SnapshotCache` knows about.
62 - Whenever a newer `Snapshot` is added to the `SnapshotCache`, that
63 `Snapshot` will get sent to the Envoy.
64
65- We manage the `SnapshotCache` by loading Envoy configuration files
66 on disk:
67 - it ignores files that start with a `.` (hidden files)
68 - it interprets `*.json` files as [JSON-encoded protobuf](https://developers.google.com/protocol-buffers/docs/proto3#json)
69 - it interprets `*.pb` files as [text-encoded protobuf](https://pkg.go.dev/google.golang.org/protobuf/encoding/prototext)
70 - all other files are ignored
71 As for when it loads those files:
72 - By default when we get a SIGHUP we reload the configuration.
73 - When passed the `--watch` argument we reload whenever any file in
74 the directory changes. Be careful about updating files
75 atomically if you use this!
76
77[^1]: The Envoy `go-control-plane` usually refers to
78 `github.com/envoyproxy/go-control-plane`, but we've "forked" it
79 as `github.com/datawire/ambassador/pkg/envoy-control-plane` in
80 order to build it against the protobufs for our patched Envoy.
81
82Running Ambex
83=============
84
85You'll need the Go toolchain, and will want to have a functioning `envoy`.
86
87Then, you can run the ambex CLI using `busyambassador`:
88
89```shell
90go run github.com/datawire/ambassador/cmd/busyambassador ambex ARGs...
91```
92
93If you're on a platform other than GNU/Linux, in order to have a
94functioning `envoy`, you may want to run all of this in the builder
95shell: `make shell`.
96
97Try it out
98----------
99
100You'll want to run both `ambex` and an instance of `envoy` with a
101boostrap config pointing at that `ambex.
102
103 1. First, start the `ambex`:
104
105 ```shell
106 go run github.com/datawire/ambassador/cmd/busyambassador ambex ./example/ambex/
107 ```
108
109 or
110
111 ```shell
112 go run github.com/datawire/ambassador/cmd/busyambassador ambex --watch ./example/ambex/
113 ```
114
115 2. Second, in another shell, start the `envoy`:
116
117 ```shell
118 envoy -l debug -c ./example/envoy/bootstrap-ads.yaml
119 ```
120
121You should now be able to run some curls in (yet) another shell:
122
123```console
124$ curl localhost:8080/hello
125Hello ADS!!!
126
127$ curl localhost:8080/get
128{
129 "args": {},
130 "headers": {
131 "Accept": "*/*",
132 "Connection": "close",
133 "Host": "httpbin.org",
134 "User-Agent": "curl/7.54.0",
135 "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
136 },
137 "origin": "72.74.69.156",
138 "url": "http://httpbin.org/get"
139}
140```
141
142Edit and/or add more files to the `./example/ambex/` directory in
143order to play with more configurations and see them reload
144_instantaneously_ (if you used the `--watch` flag), or when-triggered
145(if you didn't use the `--watch` flag; trigger a relead by signaling
146the process with `killall -HUP ambex`).
147
148Clean up
149--------
150
151Kill Ambex and Envoy with `Ctrl-C`.
View as plain text