1# Developing Emissary-ingress
2
3Welcome to the Emissary-ingress Community!
4
5Thank you for contributing, we appreciate small and large contributions and look forward to working with you to make Emissary-ingress better.
6
7This document is intended for developers looking to contribute to the Emissary-ingress project. In this document you will learn how to get your development environment setup and how to contribute to the project. Also, you will find more information about the internal components of Emissary-ingress and other questions about working on the project.
8
9> Looking for end user guides for Emissary-ingress? You can check out the end user guides at <https://www.getambassador.io/docs/emissary/>.
10
11After reading this document if you have questions we encourage you to join us on our [Slack channel](https://d6e.co/slack) in the [#emissary-dev](https://datawire-oss.slack.com/archives/CB46TNG83) channel.
12
13- [Code of Conduct](../Community/CODE_OF_CONDUCT.md)
14- [Governance](../Community/GOVERNANCE.md)
15- [Maintainers](../Community/MAINTAINERS.md)
16
17**Table of Contents**
18
19- [Development Setup](#development-setup)
20 - [Step 1: Install Build Dependencies](#step-1-install-build-dependencies)
21 - [Step 2: Clone Project](#step-2-clone-project)
22 - [Step 3: Configuration](#step-3-configuration)
23 - [Step 4: Building](#step-4-building)
24 - [Step 5: Push](#step-5-push)
25 - [Step 6: Deploy](#step-6-deploy)
26 - [Step 7: Dev-loop](#step-7-dev-loop)
27 - [What should I do next?](#what-should-i-do-next)
28- [Contributing](#contributing)
29 - [Submitting a Pull Request (PR)](#submitting-a-pull-request-pr)
30 - [Pull Request Review Process](#pull-request-review-process)
31 - [Rebasing a branch under review](#rebasing-a-branch-under-review)
32 - [Fixup commits during PR review](#fixup-commits-during-pr-review)
33- [Development Workflow](#development-workflow)
34 - [Branching Strategy](#branching-strategy)
35 - [Backport Strategy](#backport-strategy)
36 - [What if I need a patch to land in a previous supported version?](#what-if-i-need-a-patch-to-land-in-a-previous-supported-version)
37 - [What if my patch is only for a previous supported version?](#what-if-my-patch-is-only-for-a-previous-supported-version)
38 - [What if I'm still not sure?](#what-if-im-still-not-sure)
39 - [Merge Strategy](#merge-strategy)
40 - [What about merge commit strategy?](#what-about-merge-commit-strategy)
41- [Contributing to the Docs](#contributing-to-the-docs)
42- [Advanced Topics](#advanced-topics)
43 - [Running Emissary-ingress internals locally](#running-emissary-ingress-internals-locally)
44 - [Setting up diagd](#setting-up-diagd)
45 - [Changing the ambassador root](#changing-the-ambassador-root)
46 - [Getting envoy](#getting-envoy)
47 - [Shutting up the pod labels error](#shutting-up-the-pod-labels-error)
48 - [Extra credit](#extra-credit)
49 - [Debugging and Developing Envoy Configuration](#debugging-and-developing-envoy-configuration)
50 - [Mockery](#mockery)
51 - [Ambassador Dump](#ambassador-dump)
52 - [Making changes to Envoy](#making-changes-to-envoy)
53 - [1. Preparing your machine](#1-preparing-your-machine)
54 - [2. Setting up your workspace to hack on Envoy](#2-setting-up-your-workspace-to-hack-on-envoy)
55 - [3. Hacking on Envoy](#3-hacking-on-envoy)
56 - [4. Building and testing your hacked-up Envoy](#4-building-and-testing-your-hacked-up-envoy)
57 - [5. Finalizing your changes](#5-finalizing-your-changes)
58 - [6. Checklist for landing the changes](#6-checklist-for-landing-the-changes)
59 - [Developing Emissary-ingress (Ambassador Labs -only advice)](#developing-emissary-ingress-ambassador-labs--only-advice)
60 - [Updating license documentation](#updating-license-documentation)
61 - [Upgrading Python dependencies](#upgrading-python-dependencies)
62- [FAQ](#faq)
63 - [How do I find out what build targets are available?](#how-do-i-find-out-what-build-targets-are-available)
64 - [How do I develop on a Mac with Apple Silicon?](#how-do-i-develop-on-a-mac-with-apple-silicon)
65 - [How do I develop on Windows using WSL?](#how-do-i-develop-on-windows-using-wsl)
66 - [How do I test using a private Docker repository?](#how-do-i-test-using-a-private-docker-repository)
67 - [How do I change the loglevel at runtime?](#how-do-i-change-the-loglevel-at-runtime)
68 - [Can I build from a docker container instead of on my local computer?](#can-i-build-from-a-docker-container-instead-of-on-my-local-computer)
69 - [How do I clear everything out to make sure my build runs like it will in CI?](#how-do-i-clear-everything-out-to-make-sure-my-build-runs-like-it-will-in-ci)
70 - [My editor is changing `go.mod` or `go.sum`, should I commit that?](#my-editor-is-changing-gomod-or-gosum-should-i-commit-that)
71 - [How do I debug "This should not happen in CI" errors?](#how-do-i-debug-this-should-not-happen-in-ci-errors)
72 - [How do I run Emissary-ingress tests?](#how-do-i-run-emissary-ingress-tests)
73 - [How do I update the python test cache?](#how-do-i-update-the-python-test-cache)
74 - [How do I type check my python code?](#how-do-i-type-check-my-python-code)
75 - [How do I get the source code for a release?](#how-do-i-get-the-source-code-for-a-release)
76
77## Development Setup
78
79This section provides the steps for getting started developing on Emissary-ingress. There are a number of prerequisites that need to be setup. In general, our tooling tries to detect any missing requirements and provide a friendly error message. If you ever find that this is not the case please file an issue.
80
81> **Note:** To enable developers contributing on Macs with Apple Silicon, we ensure that the artifacts are built for `linux/amd64`
82> rather than the host `linux/arm64` architecture. This can be overriden using the `BUILD_ARCH` environment variable. Pull Request are welcome :).
83
84### Step 1: Install Build Dependencies
85
86Here is a list of tools that are used by the build system to generate the build artifacts, packaging them up into containers, generating crds, helm charts and for running tests.
87
88- git
89- make
90- docker (make sure you can run docker commands as your dev user without sudo)
91- bash
92- rsync
93- golang - `go.mod` for current version
94- python (>=3.10.9)
95- kubectl
96- a kubernetes cluster (you need permissions to create resources, i.e. crds, deployments, services, etc...)
97- a Docker registry
98- bsdtar (Provided by libarchive-tools on Ubuntu 19.10 and newer)
99- gawk
100- jq
101- helm
102
103### Step 2: Clone Project
104
105If you haven't already then this would be a good time to clone the project running the following commands:
106
107```bash
108# clone to your preferred folder
109git clone https://github.com/emissary-ingress/emissary.git
110
111# navigate to project
112cd emissary
113```
114
115### Step 3: Configuration
116
117You can configure the build system using environment variables, two required variables are used for setting the container registry and the kubeconfig used.
118
119> **Important**: the test and build system perform destructive operations against your cluster. Therefore, we recommend that you
120> use a development cluster. Setting the DEV_KUBECONFIG variable described below ensures you don't accidently perform actions on a production cluster.
121
122Open a terminal in the location where you cloned the repository and run the following commands:
123
124```bash
125# set container registry using `export DEV_REGISTRY=<your-registry>
126# note: you need to be logged in and have permissions to push
127# Example:
128export DEV_REGISTRY=docker.io/parsec86
129
130# set kube config file using `export DEV_KUBECONFIG=<dev-kubeconfig>`
131# your cluster needs the ability to read from the configured container registry
132export DEV_KUBECONFIG="$HOME/.kube/dev-config.yaml"
133
134```
135
136### Step 4: Building
137
138The build system for this project leverages `make` and multi-stage `docker` builds to produce the following containers:
139
140- `emissary.local/emissary` - single deployable container for Emissary-ingress
141- `emissary.local/kat-client` - test client container used for testing
142- `emissary.local/kat-server` - test server container used for testing
143
144Using the terminal session you opened in step 2, run the following commands
145
146>
147
148```bash
149# This will pull and build the necessary docker containers and produce multiple containers.
150# If this is the first time running this command it will take a little bit while the base images are built up and cached.
151make images
152
153# verify containers were successfully created, you should also see some of the intermediate builder containers as well
154docker images | grep emissary.local
155```
156
157*What just happened?*
158
159The build system generated a build container that pulled in envoy, the build dependencies, built various binaries from within this project and packaged them into a single deployable container. More information on this can be found in the [Architecture Document](ARCHITECTURE.md).
160
161### Step 5: Push
162
163Now that you have successfully built the containers its time to push them to your container registry which you setup in step 2.
164
165In the same terminal session you can run the following command:
166
167```bash
168# re-tags the images and pushes them to your configured container registry
169# docker must be able to login to your registry and you have to have push permissions
170make push
171
172# you can view the newly tag images by running
173docker images | grep <your -registry>
174
175# alternatively, we have two make targets that provide information as well
176make env
177
178# or in a bash export friendly format
179make export
180```
181
182### Step 6: Deploy
183
184Now its time to deploy the container out to your Kubernetes cluster that was configured in step 2. Hopefully, it is already becoming apparent that we love to leverage Make to handle the complexity for you :).
185
186```bash
187# generate helm charts and K8's Configs with your container swapped in and apply them to your cluster
188make deploy
189
190# check your cluster to see if emissary is running
191# note: kubectl doesn't know about DEV_KUBECONFIG so you may need to ensure KUBECONFIG is pointing to the correct cluster
192kubectl get pod -n ambassador
193```
194
195🥳 If all has gone well then you should have your development environment setup for building and testing Emissary-ingress.
196
197### Step 7: Dev-loop
198
199Now that you are all setup and able to deploy a development container of Emissary-ingress to a cluster, it is time to start making some changes.
200
201Lookup an issue that you want to work on, assign it to yourself and if you have any questions feel free to ping us on slack in the #emissary-dev channel.
202
203Make a change to Emissary-ingress and when you want to test it in a live cluster just re-run
204
205`make deploy`
206
207This will:
208
209- recompile the go binary
210- rebuild containers
211- push them to the docker registry
212- rebuild helm charts and manifest
213- reapply manifest to cluster and re-deploy Emissary-ingress to the cluster
214
215> *Do I have to run the other make targets `make images` or `make push` ?*
216> No you don't have to because `make deploy` will actually run those commands for you. The steps above were meant to introduce you to the various make targets so that you aware of them and have options when developing.
217
218### What should I do next?
219
220Now that you have your dev system up and running here are some additional content that we recommend you check out:
221
222- [Emissary-ingress Architecture](ARCHITECTURE.md)
223- [Contributing Code](#contributing)
224- [Contributing to Docs](#contributing-to-the-docs)
225- [Advanced Topics](#advanced-topics)
226- [Faq](#faq)
227
228## Contributing
229
230This section goes over how to contribute code to the project and how to get started contributing. More information on how we manage our branches can be found below in [Development Workflow](#development-workflow).
231
232Before contributing be sure to read our [Code of Conduct](../Community/CODE_OF_CONDUCT.md) and [Governance](../Community/GOVERNANCE.md) to get an understanding of how our project is structured.
233
234### Submitting a Pull Request (PR)
235
236> If you haven't set up your development environment then please see the [Development Setup](#development-setup) section.
237
238When submitting a Pull Request (PR) here are a set of guidelines to follow:
239
2401. Search for an [existing issue](https://github.com/emissary-ingress/emissary/issues) or create a [new issue](https://github.com/emissary-ingress/emissary/issues/new/choose).
241
2422. Be sure to describe your proposed change and any open questions you might have in the issue. This allows us to collect historical context around an issue, provide feedback on the proposed solution and discuss what versions a fix should target.
243
2443. If you haven't done so already create a fork of the respository and clone it locally
245
246 ```shell
247 git clone <your-fork>
248 ```
249
2504. Cut a new patch branch from `master`:
251
252 ```shell
253 git checkout master
254 git checkout -b my-patch-branch master
255 ```
256
2575. Make necessary code changes.
258
259 - Make sure you include test coverage for the change, see [How do I run Tests](#how-do-i-run-emissary-ingress-tests)
260 - Ensure code linting is passing by running `make lint`
261 - Code changes must have associated documentation updates.
262 - Make changes in <https://github.com/datawire/ambassador-docs> as necessary, and include a reference to those changes the pull request for your code changes.
263 - See [Contributing to Docs](#contributing-to-the-docs) for more details.
264
265 > Smaller pull requests are easier to review and can get merged faster thus reducing potential for merge conflicts so it is recommend to keep them small and focused.
266
2676. Commit your changes using descriptive commit messages.
268 - we **require** that all commits are signed off so please be sure to commit using the `--signoff` flag, e.g. `git commit --signoff`
269 - commit message should summarize the fix and motivation for the proposed fix. Include issue # that the fix looks to address.
270 - we are "ok" with multiple commits but we may ask you to squash some commits during the PR review process
271
2727. Push your branch to your forked repository:
273
274 > It is good practice to make sure your change is rebased on the latest master to ensure it will merge cleanly so if it has been awhile since you rebased on upstream you should do it now to ensure there are no merge conflicts
275
276 ```shell
277 git push origin my-patch-branch
278 ```
279
2808. Submit a Pull Request from your fork targeting upstream `emissary/master`.
281
282Thanks for your contribution! One of the [Maintainers](../Community/MAINTAINERS.md) will review your PR and discuss any changes that need to be made.
283
284### Pull Request Review Process
285
286This is an opportunity for the Maintainers to review the code for accuracy and ensure that it solves the problem outlined in the issue. This is an iterative process and meant to ensure the quality of the code base. During this process we may ask you to break up Pull Request into smaller changes, squash commits, rebase on master, etc...
287
288Once you have been provided feedback:
289
2901. Make the required updates to the code per the review discussion
2912. Retest the code and ensure linting is still passing
2923. Commit the changes and push to Github
293 - see [Fixup Commits](#fixup-commits-during-pr-review) below
2944. Repeat these steps as necessary
295
296Once you have **two approvals** then one of the Maintainers will merge the PR.
297
298:tada: Thank you for contributing and being apart of the Emissary-ingress Community!
299
300### Rebasing a branch under review
301
302Many times the base branch will have new commits added to it which may cause merge conflicts with your open pull request. First, a good rule of thumb is to make pull request small so that these conflicts are less likely to occur but this is not always possible when have multiple people working on similiar features. Second, if it is just addressing commit feedback a `fixup` commit is also a good option so that the reviewers can see what changed since their last review.
303
304If you need to address merge conflicts then it is preferred that you use **Rebase** on the base branch rather than merging base branch into the feature branch. This ensures that when the PR is merged that it will cleanly replay on top of the base branch ensuring we maintain a clean linear history.
305
306To do a rebase you can do the following:
307
308```shell
309# add emissary.git as a remote repository, only needs to be done once
310git remote add upstream https://github.com/emissary-ingress/emissary.git
311
312# fetch upstream master
313git fetch upstream master
314
315# checkout local master and update it from upstream master
316git checkout master
317git pull -ff upstream master
318
319# rebase patch branch on local master
320git checkout my-patch-branch
321git rebase -i master
322```
323
324Once the merge conflicts are addressed and you are ready to push the code up you will need to force push your changes because during the rebase process the commit sha's are re-written and it has diverged from what is in your remote fork (Github).
325
326To force push a branch you can:
327
328```shell
329git push head --force-with-lease
330```
331
332> Note: the `--force-with-lease` is recommended over `--force` because it is safer because it will check if the remote branch had new commits added during your rebase. You can read more detail here: <https://itnext.io/git-force-vs-force-with-lease-9d0e753e8c41>
333
334### Fixup commits during PR review
335
336One of the major downsides to rebasing a branch is that it requires force pushing over the remote (Github) which then marks all the existing review history outdated. This makes it hard for a reviewer to figure out whether or not the new changes addressed the feedback.
337
338One way you can help the reviewer out is by using **fixup** commits. Fixup commits are special git commits that append `fixup!` to the subject of a commit. `Git` provides tools for easily creating these and also squashing them after the PR review process is done.
339
340Since this is a new commit on top of the other commits, you will not lose your previous review and the new commit can be reviewed independently to determine if the new changes addressed the feedback correctly. Then once the reviewers are happy we will ask you to squash them so that we when it is merged we will maintain a clean linear history.
341
342Here is a quick read on it: <https://jordanelver.co.uk/blog/2020/06/04/fixing-commits-with-git-commit-fixup-and-git-rebase-autosquash/>
343
344TL;DR;
345
346```shell
347# make code change and create new commit
348git commit --fixup <sha>
349
350# push to Github for review
351git push
352
353# reviewers are happy and ask you to do a final rebase before merging
354git rebase -i --autosquash master
355
356# final push before merging
357git push --force-with-lease
358```
359
360## Development Workflow
361
362This section introduces the development workflow used for this repository. It is recommended that both Contributors, Release Engineers and Maintainers familiarize themselves with this content.
363
364### Branching Strategy
365
366This repository follows a trunk based development workflow. Depending on what article you read there are slight nuances to this so this section will outline how this repository interprets that workflow.
367
368The most important branch is `master` this is our **Next Release** version and it should always be in a shippable state. This means that CI should be green and at any point we can decided to ship a new release from it. In a traditional trunk based development workflow, developers are encouraged to land partially finished work daily and to keep that work hidden behind feature flags. This repository does **NOT** follow that and instead if code lands on master it is something we are comfortable with shipping.
369
370We ship release candidate (RC) builds from the `master` branch (current major) and also from `release/v{major.minor}` branches (last major version) during our development cycles. Therefore, it is important that it remains shippable at all times!
371
372When we do a final release then we will cut a new `release/v{major.minor}` branch. These are long lived release branches which capture a snapshot in time for that release. For example here are some of the current release branches (as of writing this):
373
374- release/v3.2
375- release/v3.1
376- release/v3.0
377- release/v2.4
378- release/v2.3
379- release/v1.14
380
381These branches contain the codebase as it was at that time when the release was done. These branches have branch protection enabled to ensure that they are not removed or accidently overwritten. If we needed to do a security fix or bug patch then we may cut a new `.Z` patch release from an existing release branch. For example, the `release/v2.4` branch is currently on `2.4.1`.
382
383As you can see we currently support mutliple major versions of Emissary-ingress and you can read more about our [End-of-Life Policy](https://www.getambassador.io/docs/emissary/latest/about/aes-emissary-eol/).
384
385For more information on our current RC and Release process you can find that in our [Release Wiki](https://github.com/emissary-ingress/emissary/wiki).
386
387### Backport Strategy
388
389Since we follow a trunk based development workflow this means that the majority of the time your patch branch will be based off from `master` and that most Pull Request will target `master`.
390
391This ensures that we do not miss bug fixes or features for the "Next" shippable release and simplifies the mental-model for deciding how to get started contributing code.
392
393#### What if I need a patch to land in a previous supported version?
394
395Let's say I have a bug fix for CRD round trip conversion for AuthService, which is affecting both `v2.y` and `v3.y`.
396
397First within the issue we should discuss what versions we want to target. This can depend on current cycle work and any upcoming releases we may have.
398
399The general rules we follow are:
400
4011. land patch in "next" version which is `master`
4022. backport patch to any `release/v{major}.{minor}` branches
403
404So, let's say we discuss it and say that the "next" major version is a long ways away so we want to do a z patch release on our current minor version(`v3.2`) and we also want to do a z patch release on our last supported major version (`v2.4`).
405
406This means that these patches need to land in three separate branches:
407
4081. `master` - next release
4092. `release/v3.2` - patch release
4103. `release/v2.4` - patch release
411
412In this scenario, we first ask you to land the patch in the `master` branch and then provide separate PR's with the commits backported onto the `release/v*` branches.
413
414> Recommendation: using the `git cherry-pick -x` will add the source commit sha to the commit message. This helps with tracing work back to the original commit.
415
416#### What if my patch is only for a previous supported version?
417
418Although, this should be an edge case, it does happen where the code has diverged enough that a fix may only be relevant to an existing supported version. In these cases we may need to do a patch release for that older supported version.
419
420A good example, if we were to find a bug in the Envoy v2 protocol configuration we would only want to target the v2 release.
421
422In this scenario, the base branch that we would create our feature branch off from would be the latest `minor` version for that release. As of writing this, that would be the `release/v2.4` branch. We would **not** need to target master.
423
424But, let's say during our fix we notice other things that need to be addressed that would also need to be fixed in `master`. Then you need to submit a **separate Pull Request** that should first land on master and then follow the normal backporting process for the other patches.
425
426#### What if I'm still not sure?
427
428This is what the issue discussions and disucssion in Slack are for so that we can help guide you so feel free to ping us in the `#emissary-dev` channel on Slack to discuss directly with us.
429
430### Merge Strategy
431
432> The audience for this section is the Maintainers but also beneficial for Contributors so that they are familiar with how the project operates.
433
434Having a clean linear commit history for a repository makes it easier to understand what is being changed and reduces the mental load for new comers to the project.
435
436To maintain a clean linear commit history the following rules should be followed:
437
438First, always rebase patch branch on to base branch. This means **NO** merge commits from merging base branch into the patch branch. This can be accomplished using git rebase.
439
440```shell
441# first, make sure you pull latest upstream changes
442git fetch upstream
443git checkout master
444git pull -ff upstream/master
445
446# checkout patch branch and rebase interactive
447# you may have merge conflicts you need to resolve
448git checkout my-patch-branch
449git rebase -i master
450```
451
452> Note: this does rewrite your commit shas so be aware when sharing branches with co-workers.
453
454Once the Pull Request is reviewed and has **two approvals** then a Maintainer can merge. Maintainers should follow prefer the following merge strategies:
455
4561. rebase and merge
4572. squash merge
458
459When `rebase and merge` is used your commits are played on top of the base branch so that it creates a clean linear history. This will maintain all the commits from the Pull Request. In most cases this should be the **preferred** merge strategy.
460
461When a Pull Request has lots of fixup commits, or pr feedback fixes then you should ask the Contributor to squash them as part of the PR process.
462
463If the contributor is unable to squash them then using a `squash merge` in some cases makes sense. **IMPORTANT**, when this does happen it is important that the commit messages are cleaned up and not just blindly accepted the way proposed by Github. Since it is easy to miss that cleanup step, this should be used less frequently compared to `rebase and merge`.
464
465#### What about merge commit strategy?
466
467> The audience for this section is the Maintainers but also beneficial for Contributors so that they are familiar with how the project operates.
468
469When maintaining a linear commit history, each commit tells the story of what was changed in the repository. When using `merge commits` it
470adds an additional commit to the history that is not necessary because the commit history and PR history already tell the story.
471
472Now `merge commits` can be useful when you are concerned with not rewriting the commit sha. Based on the current release process which includes using `rel/v` branches that are tagged and merged into `release/v` branches we must use a `merge commit` when merging these branches. This ensures that the commit sha a Git Tag is pointing at still exists once merged into the `release/v` branch.
473
474## Contributing to the Docs
475
476The Emissary-ingress community will all benefit from having documentation that is useful and correct. If you have found an issue with the end user documentation, then please help us out by submitting an issue and/or pull request with a fix!
477
478The end user documentation for Emissary-ingress lives in a different repository and can be found at <https://github.com/datawire/ambassador-docs>.
479
480See this repository for details on how to contribute to either a `pre-release` or already-released version of Emissary-ingress.
481
482## Advanced Topics
483
484This section is for more advanced topics that provide more detailed instructions. Make sure you go through the Development Setup and read the Architecture document before exploring these topics.
485
486### Running Emissary-ingress internals locally
487
488The main entrypoint is written in go. It strives to be as compatible as possible
489with the normal go toolchain. You can run it with:
490
491```bash
492go run ./cmd/busyambassador entrypoint
493```
494
495Of course just because you can run it this way does not mean it will succeed.
496The entrypoint needs to launch `diagd` and `envoy` in order to function, and it
497also expect to be able to write to the `/ambassador` directory.
498
499#### Setting up diagd
500
501If you want to hack on diagd, its easiest to setup a virtualenv with an editable
502copy and launch your `go run` from within that virtualenv. Note that these
503instructions depend on the virtualenvwrapper
504(<https://virtualenvwrapper.readthedocs.io/en/latest/>) package:
505
506```bash
507# Create a virtualenv named venv with all the python requirements
508# installed.
509python3 -m venv venv
510. venv/bin/activate
511# If you're doing this in Datawire's apro.git, then:
512cd ambassador
513# Update pip and install dependencies
514pip install --upgrade pip
515pip install orjson # see below
516pip install -r builder/requirements.txt
517# Created an editable installation of ambassador:
518pip install -e python/
519# Check that we do indeed have diagd in our path.
520which diagd
521# If you're doing this in Datawire's apro.git, then:
522cd ..
523```
524
525(Note: it shouldn't be necessary to install `orjson` by hand. The fact that it is
526at the moment is an artifact of the way Ambassador builds currently happen.)
527
528#### Changing the ambassador root
529
530You should now be able to launch ambassador if you set the
531`ambassador_root` environment variable to a writable location:
532
533 ambassador_root=/tmp go run ./cmd/busyambassador entrypoint
534
535#### Getting envoy
536
537If you do not have envoy in your path already, the entrypoint will use
538docker to run it. At the moment this is untested for macs which probably
539means it is broken since localhost communication does not work by
540default on macs. This can be made to work as soon an intrepid volunteer
541with a mac reaches out to me (rhs@datawire.io).
542
543#### Shutting up the pod labels error
544
545An astute observe of the logs will notice that ambassador complains
546vociferously that pod labels are not mounted in the ambassador
547container. To reduce this noise, you can:
548
549```bash
550mkdir /tmp/ambassador-pod-info && touch /tmp/ambassador-pod-info/labels
551```
552
553#### Extra credit
554
555When you run ambassador locally it will configure itself exactly as it
556would in the cluster. That means with two caveats you can actually
557interact with it and it will function normally:
558
5591. You need to run `telepresence connect` or equivalent so it can
560 connect to the backend services in its configuration.
561
5622. You need to supply the host header when you talk to it.
563
564### Debugging and Developing Envoy Configuration
565
566Envoy configuration is generated by the ambassador compiler. Debugging
567the ambassador compiler by running it in kubernetes is very slow since
568we need to push both the code and any relevant kubernetes resources
569into the cluster. The following sections will provide tips for improving
570this development experience.
571
572#### Mockery
573
574Fortunately we have the `mockery` tool which lets us run the compiler
575code directly on kubernetes resources without having to push that code
576or the relevant kubernetes resources into the cluster. This is the
577fastest way to hack on and debug the compiler.
578
579The `mockery` tool runs inside the Docker container used to build
580Ambassador, using `make shell`, so it's important to realize that it
581won't have access to your entire filesystem. There are two easy ways
582to arrange to get data in and out of the container:
583
5841. If you `make sync`, everything in the Ambassador source tree gets rsync'd
585 into the container's `/buildroot/ambassador`. The first time you start the
586 shell, this can take a bit, but after that it's pretty fast. You'll
587 probably need to use `docker cp` to get data out of the container, though.
588
5892. You may be able to use Docker volume mounts by exporting `BUILDER_MOUNTS`
590 with the appropriate `-v` switches before running `make shell` -- e.g.
591
592 ```bash
593 export BUILDER_MOUNTS=$(pwd)/xfer:/xfer
594 make shell
595 ```
596
597 will cause the dev shell to mount `xfer` in your current directory as `/xfer`.
598 This is known to work well on MacOS (though volume mounts are slow on Mac,
599 so moving gigabytes of data around this way isn't ideal).
600
601Once you've sorted out how to move data around:
602
6031. Put together a set of Ambassador configuration CRDs in a file that's somewhere
604 that you'll be able to get them into the builder container. The easy way to do
605 this is to use the files you'd feed to `kubectl apply`; they should be actual
606 Kubernetes objects with `metadata` and `spec` sections, etc. (If you want to
607 use annotations, that's OK too, just put the whole `Service` object in there.)
608
6092. Run `make compile shell` to build everything and start the dev shell.
610
6113. From inside the build shell, run
612
613 ```bash
614 mockery $path_to_your_file
615 ```
616
617 If you're using a non-default `ambassador_id` you need to provide it in the
618 environment:
619
620 ```bash
621 AMBASSADOR_ID=whatever mockery $path_to_your_file
622 ```
623
624 Finally, if you're trying to mimic `KAT`, copy the `/tmp/k8s-AmbassadorTest.yaml`
625 file from a KAT run to use as input, then
626
627 ```bash
628 mockery --kat $kat_test_name $path_to_k8s_AmbassadorTest.yaml
629 ```
630
631 where `$kat_test_name` is the class name of a `KAT` test class, like `LuaTest` or
632 `TLSContextTest`.
633
6344. Once it's done, `/tmp/ambassador/snapshots` will have all the output from the
635 compiler phase of Ambassador.
636
637The point of `mockery` is that it mimics the configuration cycle of real Ambassador,
638without relying at all on a Kubernetes cluster. This means that you can easily and
639quickly take a Kubernetes input and look at the generated Envoy configuration without
640any other infrastructure.
641
642#### Ambassador Dump
643
644The `ambassador dump` tool is also useful for debugging and hacking on
645the compiler. After running `make shell`, you'll also be able to use
646the `ambassador` CLI, which can export the most import data structures
647that Ambassador works with as JSON. It works from an input which can
648be either a single file or a directory full of files in the following
649formats:
650
651- raw Ambassador resources like you'll find in the `demo/config` directory; or
652- an annotated Kubernetes resources like you'll find in `/tmp/k8s-AmbassadorTest.yaml` after running `make test`; or
653- a `watt` snapshot like you'll find in the `$AMBASSADOR_CONFIG_BASE_DIR/snapshots/snapshot.yaml` (which is a JSON file, I know, it's misnamed).
654
655Given an input source, running
656
657```bash
658ambassador dump --ir --xds [$input_flags] $input > test.json
659```
660
661will dump the Ambassador IR and v2 Envoy configuration into `test.json`. Here
662`$input_flags` will be
663
664- nothing for raw Ambassador resources;
665- `--k8s` for Kubernetes resources; or
666- `--watt` for a `watt` snapshot.
667
668You can get more information with
669
670```bash
671ambassador dump --help
672```
673
674### Making changes to Envoy
675
676Emissary-ingress is built on top of Envoy and leverages a vendored version of Envoy (*we track upstream very closely*). This section will go into how to make changes to the Envoy that is packaged with Emissary-ingress.
677
678This is a bit more complex than anyone likes, but here goes:
679
680#### 1. Preparing your machine
681
682Building and testing Envoy can be very resource intensive. A laptop
683often can build Envoy... if you plug in an external hard drive, point
684a fan at it, and leave it running overnight and most of the next day.
685At Ambassador Labs, we'll often spin up a temporary build machine in GCE, so
686that we can build it very quickly.
687
688As of Envoy 1.15.0, we've measure the resource use to build and test
689it as:
690
691> | Command | Disk Size | Disk Used | Duration[1] |
692> |--------------------|-----------|-----------|-------------|
693> | `make update-base` | 450G | 12GB | ~11m |
694> | `make check-envoy` | 450G | 424GB | ~45m |
695>
696> [1] On a "Machine type: custom (32 vCPUs, 512 GB memory)" VM on GCE,
697> with the following entry in its `/etc/fstab`:
698>
699> ```bash
700> tmpfs:docker /var/lib/docker tmpfs size=450G 0 0
701> ```
702
703If you have the RAM, we've seen huge speed gains from doing the builds
704and tests on a RAM disk (see the `/etc/fstab` line above).
705
706#### 2. Setting up your workspace to hack on Envoy
707
7081. From your `emissary.git` checkout, get Emissary-ingress's current
709 version of the Envoy sources, and create a branch from that:
710
711 ```shell
712 make $PWD/_cxx/envoy
713 git -C _cxx/envoy checkout -b YOUR_BRANCHNAME
714 ```
715
7162. Tell the build system that, yes, you really would like to be
717 compiling envoy, as you'll be modifying Envoy:
718
719 ```shell
720 export YES_I_AM_OK_WITH_COMPILING_ENVOY=true
721 export ENVOY_COMMIT='-'
722 ```
723
724 Building Envoy is slow, and most Emissary-ingress contributors do not
725 want to rebuild Envoy, so we require the first two environment
726 variables as a safety.
727
728 Setting `ENVOY_COMMIT=-` does 3 things:
729 1. Tell it to use whatever is currently checked out in
730 `./_cxx/envoy/` (instead of checking out a specific commit), so
731 that you are free to modify those sources.
732 2. Don't try to download a cached build of Envoy from a Docker
733 cache (since it wouldn't know which `ENVOY_COMMIT` do download
734 the cached build for).
735 3. Don't push the build of Envoy to a Docker cache (since you're
736 still actively working on it).
737
7383. To build Envoy in FIPS mode, set the following variable:
739
740 ```shell
741 export FIPS_MODE=true
742 ```
743
744 It is important to note that while building Envoy in FIPS mode is
745 required for FIPS compliance, additional steps may be necessary.
746 Emissary does not claim to be FIPS compliant or certified.
747 See [here](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ssl#fips-140-2) for more information on FIPS and Envoy.
748
749#### 3. Hacking on Envoy
750
751Modify the sources in `./_cxx/envoy/`.
752
753#### 4. Building and testing your hacked-up Envoy
754
755- **Build Envoy** with `make update-base`. Again, this is *not* a
756 quick process. The build happens in a Docker container; you can
757 set `DOCKER_HOST` to point to a powerful machine if you like.
758
759- **Test Envoy** and run with Envoy's test suite (which we don't run
760 during normal Ambassador development) by running `make check-envoy`.
761 Be warned that Envoy's full **test suite** requires several hundred
762 gigabytes of disk space to run.
763
764 Inner dev-loop steps:
765
766 - To run just specific tests, instead of the whole test suite, set
767 the `ENVOY_TEST_LABEL` environment variable. For example, to run
768 just the unit tests in
769 `test/common/network/listener_impl_test.cc`, you should run
770
771 ```shell
772 ENVOY_TEST_LABEL='//test/common/network:listener_impl_test' make check-envoy
773 ```
774
775 - You can run `make envoy-shell` to get a Bash shell in the Docker
776 container that does the Envoy builds.
777
778 Interpreting the test results:
779
780 - If you see the following message, don't worry, it's harmless; the
781 tests still ran:
782
783 ```text
784 There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.
785 ```
786
787 The message means that the test passed, but it passed too
788 quickly, and Bazel is suggesting that you declare it as smaller.
789 Something along the lines of "This test only took 2s, but you
790 declared it as being in the 60s-300s ('moderate') bucket,
791 consider declaring it as being in the 0s-60s ('short')
792 bucket".
793
794 Don't be confused (as I was) in to thinking that it was saying
795 that the test was too big and was skipped and that you need to
796 throw more hardware at it.
797
798- **Build or test Emissary-ingress** with the usual `make` commands, with
799 the exception that you MUST run `make update-base` first whenever
800 Envoy needs to be recompiled; it won't happen automatically. So
801 `make test` to build-and-test Emissary-ingress would become
802 `make update-base && make test`, and `make images` to just build
803 Emissary-ingress would become `make update-base && make images`.
804
805#### 5. Finalizing your changes
806
807Once you're happy with your changes to Envoy:
808
8091. Ensure they're committed to `_cxx/envoy/` and push/PR them into
810 <https://github.com/datawire/envoy> branch `rebase/master`.
811
812 If you're outside of Ambassador Labs, you'll need to
813 a. Create a fork of <https://github.com/datawire/envoy> on the
814 GitHub web interface
815 b. Add it as a remote to your `./_cxx/envoy/`:
816 `git remote add my-fork git@github.com:YOUR_USERNAME/envoy.git`
817 c. Push the branch to that fork:
818 `git push my-fork YOUR_BRANCHNAME`
819
8202. Update `ENVOY_COMMIT` in `_cxx/envoy.mk`
821
8223. Unset `ENVOY_COMMIT=-` and run a final `make update-base` to
823 push a cached build:
824
825 ```shell
826 export YES_I_AM_OK_WITH_COMPILING_ENVOY=true
827 unset ENVOY_COMMIT
828 make update-base
829 ```
830
831 The image will be pushed to `$ENVOY_DOCKER_REPO`, by default
832 `ENVOY_DOCKER_REPO=docker.io/datawire/ambassador-base`; if you're
833 outside of Ambassador Labs, you can skip this step if you don't want to
834 share your Envoy binary anywhere. If you don't skip this step,
835 you'll need to `export
836 ENVOY_DOCKER_REPO=${your-envoy-docker-registry}` to tell it to push
837 somewhere other than Datawire's registry.
838
839 If you're at Ambassador Labs, you'll then want to make sure that the image
840 is also pushed to the backup container registries:
841
842 ```shell
843 # upload image to the mirror in GCR
844 SHA=GET_THIS_FROM_THE_make_update-base_OUTPUT
845 TAG="envoy-0.$SHA.opt"
846 FULL_TAG="envoy-full-0.$SHA.opt"
847 docker pull "docker.io/emissaryingress/base-envoy:envoy-0.$TAG.opt"
848 docker tag "docker.io/emissaryingress/base-envoy:$TAG" "gcr.io/datawire/ambassador-base:$TAG"
849 docker push "gcr.io/datawire/ambassador-base:$TAG"
850
851 ## repeat for the "FULL" version which has debug symbols enabled for envoy. It is large (GB's) big.
852 TAG=envoy-full-0.386367b8c99f843fbc2a42a38fe625fce480de19.opt
853 docker pull "docker.io/emissaryingress/base-envoy:$FULL_TAG"
854 docker tag "docker.io/emissaryingress/base-envoy:$FULL_TAG" "gcr.io/datawire/ambassador-base:$FULL_TAG"
855 docker push "gcr.io/datawire/ambassador-base:$FULL_TAG"
856 ```
857
858 If you're outside of Ambassador Labs, you can skip this step if you
859 don't want to share your Envoy binary anywhere. If you don't
860 skip this step, you'll need to `export
861 ENVOY_DOCKER_REPO=${your-envoy-docker-registry}` to tell it to
862 push somewhere other than Datawire's registry.
863
8644. Push and PR the `envoy.mk` `ENVOY_COMMIT` change to
865 <https://github.com/emissary-ingress/emissary>.
866
867#### 6. Checklist for landing the changes
868
869I'd put this in the pull request template, but so few PRs change Envoy...
870
871- [ ] The image has been pushed to...
872 - [ ] `docker.io/emissaryingress/base-envoy`
873 - [ ] `gcr.io/datawire/ambassador-base`
874- [ ] The envoy.git commit has been tagged as `datawire-$(gitdescribe --tags --match='v*')`
875 (the `--match` is to prevent `datawire-*` tags from stacking on each other).
876- [ ] It's been tested with...
877 - [ ] `make check-envoy`
878
879The `check-envoy-version` CI job should check all of those things,
880except for `make check-envoy`.
881
882### Developing Emissary-ingress (Ambassador Labs -only advice)
883
884At the moment, these techniques will only work internally to Ambassador Labs. Mostly
885this is because they require credentials to access internal resources at the
886moment, though in several cases we're working to fix that.
887
888#### Updating license documentation
889
890When new dependencies are added or existing ones are updated, run
891`make generate` and commit changes to `DEPENDENCIES.md` and
892`DEPENDENCY_LICENSES.md`
893
894#### Upgrading Python dependencies
895
896Delete `python/requirements.txt`, then run `make generate`.
897
898If there are some dependencies you don't want to upgrade, but want to
899upgrade everything else, then
900
901 1. Remove from `python/requirements.txt` all of the entries except
902 for those you want to pin.
903 2. Delete `python/requirements.in` (if it exists).
904 3. Run `make generate`.
905
906> **Note**: If you are updating orjson you will need to also update `docker/base-python/Dockerfile` before running `make generate` for the new version. orjson uses rust bindings and the default wheels on PyPI rely on glibc. Because our base python image is Alpine based, it is built from scratch using rustc to build a musl compatable version.
907
908 > :warning: You may run into an error when running `make generate` where it can't detect the licenses for new or upgraded dependencies, which is needed so that so that we can properly generate DEPENDENCIES.md and DEPENDENCY_LICENSES.md. If that is the case, you may also have to update `build-aux/tools/src/py-mkopensource/main.go:parseLicenses` for any license changes then run `make generate` again.
909
910## FAQ
911
912This section contains a set of Frequently Asked Questions that may answer a question you have. Also, feel free to ping us in Slack.
913
914### How do I find out what build targets are available?
915
916Use `make help` and `make targets` to see what build targets are
917available along with documentation for what each target does.
918
919### How do I develop on a Mac with Apple Silicon?
920
921To ensure that developers using a Mac with Apple Silicon can contribute, the build system ensures
922the build artifacts are `linux/amd64` rather than the host architecture. This behavior can be overriden
923using the `BUILD_ARCH` environment variable (e.g. `BUILD_ARCH=linux/arm64 make images`).
924
925### How do I develop on Windows using WSL?
926
927As the Emissary-ingress build system requires docker communication via a UNIX socket, using WSL 1 is not possible.
928Not even with a `DOCKER_HOST` environment variable set. As a result, you have to use WSL 2, including using the
929WSL 2 version of docker-for-windows.
930
931Additionally, if your hostname contains an upper-case character, the build script will break. This is based on the
932`NAME` environment variable, which should contain your hostname. You can solve this issue by doing `export NAME=my-lowercase-host-name`.
933If you do this *after* you've already run `make images` once, you will manually have to clean up the docker images
934that have been created using your upper-case host name.
935
936### How do I test using a private Docker repository?
937
938If you are pushing your development images to a private Docker repo,
939then:
940
941```sh
942export DEV_USE_IMAGEPULLSECRET=true
943export DOCKER_BUILD_USERNAME=...
944export DOCKER_BUILD_PASSWORD=...
945```
946
947and the test machinery should create an `imagePullSecret` from those Docker credentials such that it can pull the images.
948
949### How do I change the loglevel at runtime?
950
951```console
952curl localhost:8877/ambassador/v0/diag/?loglevel=debug
953```
954
955Note: This affects diagd and Envoy, but NOT the AES `amb-sidecar`.
956See the AES `DEVELOPING.md` for how to do that.
957
958### Can I build from a docker container instead of on my local computer?
959
960If you want to build within a container instead of setting up dependencies on your local machine then you can run the build within a docker container and leverage "Docker in Docker" to build it.
961
9621. `docker pull docker:latest`
9632. `docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -it docker:latest sh`
9643. `apk add --update --no-cache bash build-base go curl rsync python3 python2 git libarchive-tools gawk jq`
9654. `git clone https://github.com/emissary-ingress/emissary.git && cd emissary`
9665. `make images`
967
968Steps 0 and 1 are run on your machine, and 2 - 4 are from within the docker container. The base image is a "Docker in Docker" image, ran with `-v /var/run/docker.sock:/var/run/docker.sock` in order to connect to your local daemon from the docker inside the container. More info on Docker in Docker [here](https://hub.docker.com/_/docker).
969
970The images will be created and tagged as defined above, and will be available in docker on your local machine.
971
972### How do I clear everything out to make sure my build runs like it will in CI?
973
974Use `make clobber` to completely remove all derived objects, all cached artifacts, everything, and get back to a clean slate. This is recommended if you change branches within a clone, or if you need to `make generate` when you're not *certain* that your last `make generate` was using the same Envoy version.
975
976Use `make clean` to remove derived objects, but *not* clear the caches.
977
978### My editor is changing `go.mod` or `go.sum`, should I commit that?
979
980If you notice this happening, run `make go-mod-tidy`, and commit that.
981
982(If you're in Ambassador Labs, you should do this from `apro/`, not
983`apro/ambassador/`, so that apro.git's files are included too.)
984
985### How do I debug "This should not happen in CI" errors?
986
987These checks indicate that some output file changed in the middle of a
988run, when it should only change if a source file has changed. Since
989CI isn't editing the source files, this shouldn't happen in CI!
990
991This is problematic because it means that running the build multiple
992times can give different results, and that the tests are probably not
993testing the same image that would be released.
994
995These checks will show you a patch showing how the output file
996changed; it is up to you to figure out what is happening in the
997build/test system that would cause that change in the middle of a run.
998For the most part, this is pretty simple... except when the output
999file is a Docker image; you just see that one image hash is different
1000than another image hash.
1001
1002Fortunately, the failure showing the changed image hash is usually
1003immediately preceded by a `docker build`. Earlier in the CI output,
1004you should find an identical `docker build` command from the first time it
1005ran. In the second `docker build`'s output, each step should say
1006`---> Using cache`; the first few steps will say this, but at some
1007point later steps will stop saying this; find the first step that is
1008missing the `---> Using cache` line, and try to figure out what could
1009have changed between the two runs that would cause it to not use the
1010cache.
1011
1012If that step is an `ADD` command that is adding a directory, the
1013problem is probably that you need to add something to `.dockerignore`.
1014To help figure out what you need to add, try adding a `RUN find
1015DIRECTORY -exec ls -ld -- {} +` step after the `ADD` step, so that you
1016can see what it added, and see what is different on that between the
1017first and second `docker build` commands.
1018
1019### How do I run Emissary-ingress tests?
1020
1021- `export DEV_REGISTRY=<your-dev-docker-registry>` (you need to be logged in and have permission to push)
1022- `export DEV_KUBECONFIG=<your-dev-kubeconfig>`
1023
1024If you want to run the Go tests for `cmd/entrypoint`, you'll need `diagd`
1025in your `PATH`. See the instructions below about `Setting up diagd` to do
1026that.
1027
1028| Group | Command |
1029| --------------- | ---------------------------------------------------------------------- |
1030| All Tests | `make test` |
1031| All Golang | `make gotest` |
1032| All Python | `make pytest` |
1033| Some/One Golang | `make gotest GOTEST_PKGS=./cmd/entrypoint GOTEST_ARGS="-run TestName"` |
1034| Some/One Python | `make pytest PYTEST_ARGS="-k TestName"` |
1035
1036Please note the python tests use a local cache to speed up test
1037results. If you make a code update that changes the generated envoy
1038configuration, those tests will fail and you will need to update the
1039python test cache.
1040
1041Note that it is invalid to run one of the `main[Plain.*]` Python tests
1042without running all of the other `main[Plain*]` tests; the test will
1043fail to run (not even showing up as a failure or xfail--it will fail
1044to run at all). For example, `PYTEST_ARGS="-k WebSocket"` would match
1045the `main[Plain.WebSocketMapping-GRPC]` test, and that test would fail
1046to run; one should instead say `PYTEST_ARGS="-k Plain or WebSocket"`
1047to avoid breaking the sub-tests of "Plain".
1048
1049### How do I type check my python code?
1050
1051Ambassador uses Python 3 type hinting and the `mypy` static type checker to
1052help find bugs before runtime. If you haven't worked with hinting before, a
1053good place to start is
1054[the `mypy` cheat sheet](https://mypy.readthedocs.io/en/latest/cheat_sheet_py3.html).
1055
1056New code must be hinted, and the build process will verify that the type
1057check passes when you `make test`. Fair warning: this means that
1058PRs will not pass CI if the type checker fails.
1059
1060We strongly recommend using an editor that can do realtime type checking
1061(at Datawire we tend to use PyCharm and VSCode a lot, but many many editors
1062can do this now) and also running the type checker by hand before submitting
1063anything:
1064
1065- `make lint/mypy` will check all the Ambassador code
1066
1067Ambassador code should produce *no* warnings and *no* errors.
1068
1069If you're concerned that the mypy cache is somehow wrong, delete the
1070`.mypy_cache/` directory to clear the cache.
1071
1072### How do I get the source code for a release?
1073
1074The current shipping release of Ambassador lives on the `master`
1075branch. It is tagged with its version (e.g. `v0.78.0`).
1076
1077Changes on `master` after the last tag have not been released yet, but
1078will be included in the next release of Ambassador.
View as plain text