1Ambassador Developer's Guide
2============================
3
4Concepts
5--------
6
7Ambassador sits between users and an Envoy. The primary job that an Ambassador does is to take an _Ambassador configuration_ and, from that, generate an _Envoy configuration_. This generation happens using an _intermediate representation (IR)_ to manage all the internal logic that Ambassador needs:
8
9```asciiart
10Ambassador config => IR => Envoy config
11```
12
13### Ambassador Components and Ports
14
15Ambassador comprises several different components:
16
17| Component | Type | Function. |
18| :------------------------ | :---- | :------------------------ |
19| `diagd` | Python | Increasingly-misnamed core of the system; manages changing configurations and provides the diagnostics UI |
20| `entrypoint/ambex` | Go | Envoy `go-control-plane` implementation; supplies Envoy with current configuration |
21| `entrypoint/watcher` | Go | Service/secret discovery; interface to Kubernetes and Consul |
22| `envoy` | C++ | The actual proxy process |
23| `kubewatch.py` | Python | Used only to determine the cluster's installation ID; needs to be subsumed by `watt` |
24
25`entrypoint`, `diagd`, and `envoy` are all long-running daemons. If any of them exit, the pod as a whole will exit.
26
27Ambassador uses several TCP ports while running. All but one of them are in the range 8000-8499, and any future assignments for Ambassador ports should come from this range.
28
29| Port | Process | Function |
30| :--- |:--------------------------------------|:-------------------------------------------------------------------------------------------------------------------|
31| 8001 | `envoy` | Internal stats, logging, etc.; not exposed outside pod |
32| 8002 | `entrypoint/watcher` | Internal `watt` snapshot access; not exposed outside pod |
33| 8003 | `entrypoint/ambex` | Internal `ambex` snapshot access; not exposed outside pod |
34| 8004 | `diagd` | Internal `diagd` access when `AMBASSADOR_FAST_RECONFIGURE` is set; not exposed outside pod |
35| 8005 | `entrypoint/external_snapshot_server` | Exposes configuration snapshots for integration with other tools, such as the Ambassador Agent |
36| 8006 | `envoy` | Default ready listener port |
37| 8080 | `envoy` | Default HTTP service port |
38| 8443 | `envoy` | Default HTTPS service port |
39| 8877 | `diagd` | Direct access to diagnostics UI; provided by `busyambassador entrypoint` when `AMBASSADOR_FAST_RECONFIGURE` is set |
40
41### The Ambassador Configuration
42
43An Ambassador configuration is a collection of _Ambassador configuration resources_, which are represented by subclasses of `ambassador.config.ACResource`. The configuration as a whole is represented by an `ambassador.Config` object.
44
45`ambassador.Config` does not know how to parse YAML, interact with Kubernetes, or look at the filesystem. Instead, its consumer must construct a consistent list of fully-instantiated `ACResource` objects and tell `ambassador.Config` to load these resources:
46
47```python
48aconf: Config = Config()
49aconf.load_all(resources: List[ACResource])
50```
51
52`load_all` is only meant to be called once, with a complete set of all the resources comprising the Ambassador configuration. To change the configuration, instantiate a new `ambassador.Config`.
53
54The `ambassador.Config` class is relatively unsophisticated: for the most part, all it does is to save the resources handed to it in a way that preserves the type of the resources, and permits consumers to query the `Config` for various kinds of resources:
55
56* `get_config(self, key: str) -> Any`
57
58 Fetches all the configuration information the `Config` has for the type tagged as `key`, e.g. `aconf.get_config("mappings")` will fetch all the `Mapping`s that the `Config` has stored.
59
60 Current keys include:
61
62 - `auth_configs`: information from `AuthService` definitions
63 - `mappings`: information from `Mapping`s
64 - `modules`: information from `Module`s, including the `Ambassador` and `TLS` `Module`s
65 - `ratelimit_configs`: information about `RateLimitService`s
66 - `tracing_configs`: information about `TracingService`s
67
68* `get_module(self, module_name: str) -> Optional[ACResource]`
69
70 Fetches the `Module` with a given name (e.g. `aconf.get_module("ambassador")` will fetch information about the `Ambassador` `Module`). Returns `None` if no `Module` with the given name is found.
71
72* `module_lookup(self, module_name: str, key: str, default: Any=None) -> Any`
73
74 Looks up a specific `key` in a specific `Module`. e.g.
75
76 ```aconf.module_lookup('ambassador', 'service_port', 8080)```
77
78 will look up the `service_port` from the `Ambassador` `Module`; if no `service_port` is defined, the default value will be 8080.
79
80* `dump(self, output=sys.stdout) -> None`
81
82 Dumps the entire `Config` object to the given `output`, for debugging.
83
84Once a `Config` has loaded resources, it can be used to create an `IR`.
85
86### The IR
87
88The Intermediate Representation is where most of the logic of Ambassador lives. The IR as a whole is represented by an `ambassador.IR` object, which contains many objects descended from `ambassador.ir.IRResource`. The logic needed to synthesize the IR from an `ambassador.Config` is mostly contained in these `IRResource` subclasses; relatively little is in the `ambassador.IR` class itself.
89
90An `ambassador.IR` can only be instantiated from an `ambassador.Config`:
91
92```
93ir = IR(aconf: Config)
94```
95
96The general rule in the `IR` is that everything interesting is stored in objects descended from `IRResource`. There are two cases:
97
98#### Only One `IRResource`
99
100Only one of some `IRResource` objects can exist within a given IR, no matter how many distinct `ACResource`s they collect information from. A good example here is the `IRAmbassador` object, which contains global configuration information pooled from the `Ambassador` `Module` as well any `TLS` `Module`: only one of these will ever exist.
101
102For these single-instance objects:
103
1041. First the resource is instantiated with the `ambassador.IR` and `ambassador.Config` objects as parameters. Whatever information is needed to initialize the `IRResource`, its `__init__` method should pull directly from the `Config` and/or `IR` objects.
105
1062. Next, the resource's `setup` method is called, again with the `IR` and `Config` as parameters. `setup` can perform further initialization, consistency checks, etc., and must return a boolean: `True` will allow the resource to become active, `False` will mean the resource isn't needed after all. (For example, the `IRAuth` object will return `False` if it doesn't find any configured authentication services.)
107
108 The default `setup` method simply returns `True`, so any `IRResource` subclass that doesn't override it will always be active.
109
1103. If the resource's `setup` returns `True`, the `IR` will save it as an active resource.
111
1124. After all resources have been probed, the `IR` will walk the list of active resources and call the `add_mappings` method for each active resource. `add_mappings` should add any mappings or clusters needed by the resource. (For example, the `IRAmbassador` object adds mappings (and thus clusters) to handle probes and diagnostics.)
113
114 The default `add_mappings` method does nothing, so no mappings or clusters will be added for `IRResource` subclasses that don't override `add_mappings`.
115
116For single-instance objects, the distinction between `__init__` and `setup` is rarely relevant. It's OK to do everything in `__init__` and allow the default `setup` to always return `True`; likewise it's OK to just save incoming data in `__init__` and have all logic in `setup`.
117
118#### Multiple `IRResource`s
119
120There can be multiple instances of some `IRResource` objects: for example, a single `Config` can contain many `Mapping`s and multiple `Listener`s. For these resources, we use a `Factory` class, which must implement the `load_all` classmethod and may implement the `finalize` classmethod, both of which receive the `IR` and `Config` as parameters.
121
1221. `load_all` is called early in the instantiation process, and is responsible for creating as many individual resources as needed and saving them (usually in the `IR` itself).
123
1242. `finalize` is called only after all the factories have had `load_all` called and all other single-instance resources have had `add_mappings` called, and is responsible for any normalization or other initialization that depends on global knowledge (for example, `MappingFactory.finalize` does the normalization of weights across `Mapping` groups).
125
126#### The Full Sequence
127
128The full IR instantiation sequence can be found in `ambassador.ir.IR.__init__`:
129
130- TLS defaults
131- `IRAmbassadorTLS` (TLS contexts, etc.)
132- `IRAmbassador` (global Ambassador config)
133- `IRAuth` (extauth services)
134- `IRRateLimit` (rate limiting)
135- `ListenerFactory` (listeners -- creates `IRListener` objects)
136- `MappingFactory` (mappings -- creates `IRMappingGroup`, `IRMapping`, and `IRCluster` objects)
137- Cluster naming normalization
138
139#### Helpers
140
141- `IR.add_mapping` adds or looks up an `IRMappingGroup` with associated `IRMapping`s and `IRCluster`s.
142- `IR.add_cluster` adds only an `IRCluster`.
143- `IR.has_cluster` and `IR.get_cluster` do `IRCluster` lookups.
144- `IR.dump` dumps the entire IR for debugging.
145
146### The Envoy V1 Configuration
147
148Finally, an Envoy V1 configuration is represented by `ambassador.envoy.V1Config`. A V1 config is built from an IR, again with most of the logic to do so contained in classes that mirror the structure of the V1 config. As we support later Envoy configuration versions, they will have their own classes.
149
150The root of the Envoy V1 configuration is `ambassador.envoy.v1.V1Config`.
151
152Overall Life Cycle
153------------------
154
1550. Construct a collection of `ACResource` objects (from disk, from K8s, whatever).
156
157 This will mostly involve `ACResource.from_dict` or `ACResource.from_yaml`.
158
1591. Instantiate an `ambassador.Config`. Use its `load_all()` method to load up the collection of `ACResource` objects.
160
1612. Instantiate an `ambassador.IR` from the `ambassador.Config`.
162
1633. Instantiate an `ambassador.envoy.V1Config` from the `ambassador.IR`.
164
165Developing in Ambassador
166------------------------
167
168In all cases, understanding the class hierarchy and the lifecycle around the IR will be important. Both of these are discussed below.
169
170### Adding Features
171
172Adding a feature will start with the Ambassador configuration resources:
173
174- The simple case will involve modifying a schema file and possibly modifying an `ACResource` class.
175- The less simple case will involve adding a new schema and a new `ACResource` subclass.
176 - Unless the new class needs complex logic (it shouldn't), you can just let the existing `Config` code save your new resource.
177 - If it does need complex logic, you'll need to add a handler method to `Config`.
178
179Once the Ambassador config is dealt with, you'll add or modify the IR to cope. Most of what you do here should involve the `IRResource` subclasses, _not_ the `IR` class itself (although if you're adding something completely new, you'll need to add code to `IR.__init__` to call your new class).
180
181Once the IR is dealt with, you'll need to add or modify the `V1Config` to cope with the IR changes.
182
183### Handling Bugs
184
185The trick with bugs will be figuring out what you need to change. In general, work from V1Config to IR to Ambassador config -- the closer to Envoy that the fix can go, the simpler it will probably be.
186
187- `ambassador.Config` and `ambassador.IR` both have `dump` methods that are invaluable for studying their contents.
188- You can also attach debuggers and look at objects. Most of the things you're working with are subclasses of `dict`, so many introspection tools are simple.
189
190Class Hierarchy
191----------------
192
193 * `IR`
194 * `IRResource`
195 * `IRAdmin`
196 * `IRAmbassador`
197 * `IRCluster`
198 * `IRFilter`
199 * `IRAuth`
200 * `IRListener`
201 * `IRMapping`
202 * `IRMappingGroup`
203 * `IRRateLimit`
204 * `IRAmbassadorTLS`
205 * `IREnvoyTLS`
206 * `envoy`
207 * `V1Config`
208 * `V1Admin`
209 * `V1Cluster`
210 * `V1ClusterManager`
211 * `V1Listener`
212
213(This diagram is mostly about the way the classes are used, rather than strictly reflecting implementation. For example, the `Config` class is actually `ambassador.config.config.Config` but is imported into `ambassador` to make usage easier.)
214
215The `Resource` Class
216--------------------
217
218`IRResource` and `ACResource` are subclasses of `ambassador.Resource`, although they are shown in the packages where they are logically used. A `Resource` is a kind of `dict` that can keep track of where it came from, what makes use of it, and any errors associated with it.
219
220To initialize a `Resource` requires:
221
222* `rkey`: a short identifier that is used as the primary key for _all_ the Ambassador classes to identify this single specific resource. It should be something like "ambassador-default.1" or the like that is very specific, though it doesn't have to be fun for humans.
223
224* `location`: a more human-readable string describing where the human should go to find the source of this resource. "Service ambassador, namespace default, object 1". This is primarly used for diagnostics.
225
226* `kind`: the kind of resource this is -- "Mapping", "TLS", "AuthService", whatever.
227
228* `serialization`: the _original input serialization_, if we have it, of the object. If we don't have it, this should be `None` -- don't just serialize the object to no purpose.
229
230All of these should be passed as keyword arguments (although it is possible to pass `rkey` and `kind` as positional arguments, it is discouraged). A `Resource` can also accept any other arbitrary keyword arguments, which will be saved in the `Resource` as they would be in a `dict`.
231
232`Resource` defines multiple common methods and mechanisms:
233
234* Dot notation and brace notation are equivalent for `Resource`: `rsrc.foo` and `rsrc["foo"]` are, by definition, equivalent.
235
236* `rsrc.post_error(status: RichStatus)` posts an error notification that will be tracked throughout the life of the object. It will be the case that anything trying to use the `Resource` will inherit its errors; this is not fully implemented yet.
237
238* `rsrc.referenced_by(other: Resource)` marks this `Resource` as being referenced by another `Resource`. For example, an Ambassador `Mapping` will cause Envoy clusters to be created. The `IRCluster` object created to track the cluster will reference the `ACMapping` that defined the mapping that caused the cluster to be created. This gives the diagnostics service a way to track from the cluster back to the annotation that caused it to exist.
239
240* `rsrc.references(other: Resource)` is the other direction: it marks the other `Resource` as being referenced by us.
241
242* `rsrc.is_referenced_by(rkey: str) -> Optional[Resource]` returns `None` if the given `rkey` is not associated with a `Resource` that references this `Resource`, or the referencing `Resource` if it is.
243
244* `rsrc.as_dict() -> dict` returns a raw-dictionary form of just the data fields of the `Resource`. Things like the location and the references table are removed.
245
246* Class method `Resource.from_dict(rkey: str, location: str, serialization: Optional[str], attrs: dict)` creates a new `Resource` or subclass from a dictionary. The `kind` passed in the dictionary determines the actual class of the object returned (see below for more).
247
248* Class method `Resource.from_yaml(rkey: str, location: str, serialization: str)` deserializes the YAML `serialization` and passes that to `Resource.from_dict`.
249
250* Class method `Resource.from_resource(...)` clones a `Resource`, allowing optionally overriding any field using keyword arguments.
251
252The `ACResource` class
253----------------------
254
255`ACResource` is a subclass of `Resource` which specifically refers to Ambassador configuration resources. It adds no new behaviors, but two additional keyword arguments are present when initializing:
256
257* `name` is the name given to this Ambassador resource. Required for every type except `Pragma`.
258
259* `apiVersion` is the API version to use when interpreting this resource. If not given, it defaults to "ambassador/v0".
260
261Also, `ACResource.from_dict` will look first for `ACResource` subclasses when interpreting types.
262
263The `IRResource` class
264----------------------
265
266`IRResource` is a subclass of `Resource` which specifically refers to IR resources. `IRResource` doesn't add any new fields (and, in fact, many `IRResource` subclasses default some fields) but several new behaviors are added.
View as plain text