1# Developing Emissary-ingress
2
3Welcome to the Emissary-ingress Community!
4
5Thank you for contributing, we appreciate small and large contributions and look forward to working with you to make Emissary-ingress better.
6
7This document is intended for developers looking to contribute to the Emissary-ingress project. In this document you will learn how to get your development environment setup and how to contribute to the project. Also, you will find more information about the internal components of Emissary-ingress and other questions about working on the project.
8
9> Looking for end user guides for Emissary-ingress? You can check out the end user guides at <https://www.getambassador.io/docs/emissary/>.
10
11After reading this document if you have questions we encourage you to join us on our [Slack channel](https://d6e.co/slack) in the [#emissary-dev](https://datawire-oss.slack.com/archives/CB46TNG83) channel.
12
13- [Code of Conduct](CODE_OF_CONDUCT.md)
14- [Governance](GOVERNANCE.md)
15- [Maintainers](MAINTAINERS.md)
16
17**Table of Contents**
18
19- [Development Setup](#development-setup)
20 - [Step 1: Install Build Dependencies](#step-1-install-build-dependencies)
21 - [Step 2: Clone Project](#step-2-clone-project)
22 - [Step 3: Configuration](#step-3-configuration)
23 - [Step 4: Building](#step-4-building)
24 - [Step 5: Push](#step-5-push)
25 - [Step 6: Deploy](#step-6-deploy)
26 - [Step 7: Dev-loop](#step-7-dev-loop)
27 - [What should I do next?](#what-should-i-do-next)
28- [Contributing](#contributing)
29 - [Submitting a Pull Request (PR)](#submitting-a-pull-request-pr)
30 - [Pull Request Review Process](#pull-request-review-process)
31 - [Rebasing a branch under review](#rebasing-a-branch-under-review)
32 - [Fixup commits during PR review](#fixup-commits-during-pr-review)
33- [Development Workflow](#development-workflow)
34 - [Branching Strategy](#branching-strategy)
35 - [Backport Strategy](#backport-strategy)
36 - [What if I need a patch to land in a previous supported version?](#what-if-i-need-a-patch-to-land-in-a-previous-supported-version)
37 - [What if my patch is only for a previous supported version?](#what-if-my-patch-is-only-for-a-previous-supported-version)
38 - [What if I'm still not sure?](#what-if-im-still-not-sure)
39 - [Merge Strategy](#merge-strategy)
40 - [What about merge commit strategy?](#what-about-merge-commit-strategy)
41- [Contributing to the Docs](#contributing-to-the-docs)
42- [Advanced Topics](#advanced-topics)
43 - [Running Emissary-ingress internals locally](#running-emissary-ingress-internals-locally)
44 - [Setting up diagd](#setting-up-diagd)
45 - [Changing the ambassador root](#changing-the-ambassador-root)
46 - [Getting envoy](#getting-envoy)
47 - [Shutting up the pod labels error](#shutting-up-the-pod-labels-error)
48 - [Extra credit](#extra-credit)
49 - [Debugging and Developing Envoy Configuration](#debugging-and-developing-envoy-configuration)
50 - [Mockery](#mockery)
51 - [Ambassador Dump](#ambassador-dump)
52 - [Making changes to Envoy](#making-changes-to-envoy)
53 - [1. Preparing your machine](#1-preparing-your-machine)
54 - [2. Setting up your workspace to hack on Envoy](#2-setting-up-your-workspace-to-hack-on-envoy)
55 - [3. Hacking on Envoy](#3-hacking-on-envoy)
56 - [4. Building and testing your hacked-up Envoy](#4-building-and-testing-your-hacked-up-envoy)
57 - [5. Finalizing your changes](#5-finalizing-your-changes)
58 - [6. Checklist for landing the changes](#6-checklist-for-landing-the-changes)
59 - [Developing Emissary-ingress (Datawire-only advice)](#developing-emissary-ingress-datawire-only-advice)
60 - [Updating license documentation](#updating-license-documentation)
61 - [Upgrading Python dependencies](#upgrading-python-dependencies)
62- [FAQ](#faq)
63 - [How do I find out what build targets are available?](#how-do-i-find-out-what-build-targets-are-available)
64 - [How do I develop on a Mac with Apple Silicon?](#how-do-i-develop-on-a-mac-with-apple-silicon)
65 - [How do I develop on Windows using WSL?](#how-do-i-develop-on-windows-using-wsl)
66 - [How do I test using a private Docker repository?](#how-do-i-test-using-a-private-docker-repository)
67 - [How do I change the loglevel at runtime?](#how-do-i-change-the-loglevel-at-runtime)
68 - [Can I build from a docker container instead of on my local computer?](#can-i-build-from-a-docker-container-instead-of-on-my-local-computer)
69 - [How do I clear everything out to make sure my build runs like it will in CI?](#how-do-i-clear-everything-out-to-make-sure-my-build-runs-like-it-will-in-ci)
70 - [My editor is changing `go.mod` or `go.sum`, should I commit that?](#my-editor-is-changing-gomod-or-gosum-should-i-commit-that)
71 - [How do I debug "This should not happen in CI" errors?](#how-do-i-debug-this-should-not-happen-in-ci-errors)
72 - [How do I run Emissary-ingress tests?](#how-do-i-run-emissary-ingress-tests)
73 - [How do I update the python test cache?](#how-do-i-update-the-python-test-cache)
74 - [How do I type check my python code?](#how-do-i-type-check-my-python-code)
75 - [How do I get the source code for a release?](#how-do-i-get-the-source-code-for-a-release)
76
77## Development Setup
78
79This section provides the steps for getting started developing on Emissary-ingress. There are a number of prerequisites that need to be setup. In general, our tooling tries to detect any missing requirements and provide a friendly error message. If you ever find that this is not the case please file an issue.
80
81> **Note:** To enable developers contributing on Macs with Apple Silicon, we ensure that the artifacts are built for `linux/amd64`
82> rather than the host `linux/arm64` architecture. This can be overriden using the `BUILD_ARCH` environment variable. Pull Request are welcome :).
83
84### Step 1: Install Build Dependencies
85
86Here is a list of tools that are used by the build system to generate the build artifacts, packaging them up into containers, generating crds, helm charts and for running tests.
87
88- git
89- make
90- docker (make sure you can run docker commands as your dev user without sudo)
91- bash
92- rsync
93- golang - `go.mod` for current version
94- python 3.8 or 3.9
95- kubectl
96- a kubernetes cluster (you need permissions to create resources, i.e. crds, deployments, services, etc...)
97- a Docker registry
98- bsdtar (Provided by libarchive-tools on Ubuntu 19.10 and newer)
99- gawk
100
101### Step 2: Clone Project
102
103If you haven't already then this would be a good time to clone the project running the following commands:
104
105```bash
106# clone to your preferred folder
107git clone https://github.com/emissary-ingress/emissary.git
108
109# navigate to project
110cd emissary
111```
112
113### Step 3: Configuration
114
115You can configure the build system using environment variables, two required variables are used for setting the container registry and the kubeconfig used.
116
117> **Important**: the test and build system perform destructive operations against your cluster. Therefore, we recommend that you
118> use a development cluster. Setting the DEV_KUBECONFIG variable described below ensures you don't accidently perform actions on a production cluster.
119
120Open a terminal in the location where you cloned the repository and run the following commands:
121
122```bash
123# set container registry using `export DEV_REGISTRY=<your-registry>
124# note: you need to be logged in and have permissions to push
125# Example:
126export DEV_REGISTRY=docker.io/parsec86
127
128# set kube config file using `export DEV_KUBECONFIG=<dev-kubeconfig>`
129# your cluster needs the ability to read from the configured container registry
130export DEV_KUBECONFIG="$HOME/.kube/dev-config.yaml"
131
132```
133
134### Step 4: Building
135
136The build system for this project leverages `make` and multi-stage `docker` builds to produce the following containers:
137
138- `emissary.local/emissary` - single deployable container for Emissary-ingress
139- `emissary.local/kat-client` - test client container used for testing
140- `emissary.local/kat-server` - test server container used for testing
141
142Using the terminal session you opened in step 2, run the following commands
143
144>
145
146```bash
147# This will pull and build the necessary docker containers and produce multiple containers.
148# If this is the first time running this command it will take a little bit while the base images are built up and cached.
149make images
150
151# verify containers were successfully created, you should also see some of the intermediate builder containers as well
152docker images | grep emissary.local
153```
154
155*What just happened?*
156
157The build system generated a build container that pulled in envoy, the build dependencies, built various binaries from within this project and packaged them into a single deployable container. More information on this can be found in the [Architecture Document](ARCHITECTURE.md).
158
159### Step 5: Push
160
161Now that you have successfully built the containers its time to push them to your container registry which you setup in step 2.
162
163In the same terminal session you can run the following command:
164
165```bash
166# re-tags the images and pushes them to your configured container registry
167# docker must be able to login to your registry and you have to have push permissions
168make push
169
170# you can view the newly tag images by running
171docker images | grep <your -registry>
172
173# alternatively, we have two make targets that provide information as well
174make env
175
176# or in a bash export friendly format
177make export
178```
179
180### Step 6: Deploy
181
182Now its time to deploy the container out to your Kubernetes cluster that was configured in step 2. Hopefully, it is already becoming apparent that we love to leverage Make to handle the complexity for you :).
183
184```bash
185# generate helm charts and K8's Configs with your container swapped in and apply them to your cluster
186make deploy
187
188# check your cluster to see if emissary is running
189# note: kubectl doesn't know about DEV_KUBECONFIG so you may need to ensure KUBECONFIG is pointing to the correct cluster
190kubectl get pod -n ambassador
191```
192
193🥳 If all has gone well then you should have your development environment setup for building and testing Emissary-ingress.
194
195### Step 7: Dev-loop
196
197Now that you are all setup and able to deploy a development container of Emissary-ingress to a cluster, it is time to start making some changes.
198
199Lookup an issue that you want to work on, assign it to yourself and if you have any questions feel free to ping us on slack in the #emissary-dev channel.
200
201Make a change to Emissary-ingress and when you want to test it in a live cluster just re-run
202
203`make deploy`
204
205This will:
206
207- recompile the go binary
208- rebuild containers
209- push them to the docker registry
210- rebuild helm charts and manifest
211- reapply manifest to cluster and re-deploy Emissary-ingress to the cluster
212
213> *Do I have to run the other make targets `make images` or `make push` ?*
214> No you don't have to because `make deploy` will actually run those commands for you. The steps above were meant to introduce you to the various make targets so that you aware of them and have options when developing.
215
216### What should I do next?
217
218Now that you have your dev system up and running here are some additional content that we recommend you check out:
219
220- [Emissary-ingress Architecture](ARCHITECTURE.md)
221- [Contributing Code](#contributing)
222- [Contributing to Docs](#contributing-to-the-docs)
223- [Advanced Topics](#advanced-topics)
224- [Faq](#faq)
225
226## Contributing
227
228This section goes over how to contribute code to the project and how to get started contributing. More information on how we manage our branches can be found below in [Development Workflow](#development-workflow).
229
230Before contributing be sure to read our [Code of Conduct](CODE_OF_CONDUCT.md) and [Governance](GOVERNANCE.md) to get an understanding of how our project is structured.
231
232### Submitting a Pull Request (PR)
233
234> If you haven't set up your development environment then please see the [Development Setup](#development-setup) section.
235
236When submitting a Pull Request (PR) here are a set of guidelines to follow:
237
2381. Search for an [existing issue](https://github.com/emissary-ingress/emissary/issues) or create a [new issue](https://github.com/emissary-ingress/emissary/issues/new/choose).
239
2402. Be sure to describe your proposed change and any open questions you might have in the issue. This allows us to collect historical context around an issue, provide feedback on the proposed solution and discuss what versions a fix should target.
241
2423. If you haven't done so already create a fork of the respository and clone it locally
243
244 ```shell
245 git clone <your-fork>
246 ```
247
2484. Cut a new patch branch from `master`:
249
250 ```shell
251 git checkout master
252 git checkout -b my-patch-branch master
253 ```
254
2555. Make necessary code changes.
256
257 - Make sure you include test coverage for the change, see [How do I run Tests](#how-do-i-run-emissary-ingress-tests)
258 - Ensure code linting is passing by running `make lint`
259 - Code changes must have associated documentation updates.
260 - Make changes in <https://github.com/datawire/ambassador-docs> as necessary, and include a reference to those changes the pull request for your code changes.
261 - See [Contributing to Docs](#contributing-to-the-docs) for more details.
262
263 > Smaller pull requests are easier to review and can get merged faster thus reducing potential for merge conflicts so it is recommend to keep them small and focused.
264
2656. Commit your changes using descriptive commit messages.
266 - we **require** that all commits are signed off so please be sure to commit using the `--signoff` flag, e.g. `git commit --signoff`
267 - commit message should summarize the fix and motivation for the proposed fix. Include issue # that the fix looks to address.
268 - we are "ok" with multiple commits but we may ask you to squash some commits during the PR review process
269
2707. Push your branch to your forked repository:
271
272 > It is good practice to make sure your change is rebased on the latest master to ensure it will merge cleanly so if it has been awhile since you rebased on upstream you should do it now to ensure there are no merge conflicts
273
274 ```shell
275 git push origin my-patch-branch
276 ```
277
2788. Submit a Pull Request from your fork targeting upstream `emissary/master`.
279
280Thanks for your contribution! One of the [Maintainers](MAINTAINERS.md) will review your PR and discuss any changes that need to be made.
281
282### Pull Request Review Process
283
284This is an opportunity for the Maintainers to review the code for accuracy and ensure that it solves the problem outlined in the issue. This is an iterative process and meant to ensure the quality of the code base. During this process we may ask you to break up Pull Request into smaller changes, squash commits, rebase on master, etc...
285
286Once you have been provided feedback:
287
2881. Make the required updates to the code per the review discussion
2892. Retest the code and ensure linting is still passing
2903. Commit the changes and push to Github
291 - see [Fixup Commits](#fixup-commits-during-pr-review) below
2924. Repeat these steps as necessary
293
294Once you have **two approvals** then one of the Maintainers will merge the PR.
295
296:tada: Thank you for contributing and being apart of the Emissary-ingress Community!
297
298### Rebasing a branch under review
299
300Many times the base branch will have new commits added to it which may cause merge conflicts with your open pull request. First, a good rule of thumb is to make pull request small so that these conflicts are less likely to occur but this is not always possible when have multiple people working on similiar features. Second, if it is just addressing commit feedback a `fixup` commit is also a good option so that the reviewers can see what changed since their last review.
301
302If you need to address merge conflicts then it is preferred that you use **Rebase** on the base branch rather than merging base branch into the feature branch. This ensures that when the PR is merged that it will cleanly replay on top of the base branch ensuring we maintain a clean linear history.
303
304To do a rebase you can do the following:
305
306```shell
307# add emissary.git as a remote repository, only needs to be done once
308git remote add upstream https://github.com/emissary-ingress/emissary.git
309
310# fetch upstream master
311git fetch upstream master
312
313# checkout local master and update it from upstream master
314git checkout master
315git pull -ff upstream master
316
317# rebase patch branch on local master
318git checkout my-patch-branch
319git rebase -i master
320```
321
322Once the merge conflicts are addressed and you are ready to push the code up you will need to force push your changes because during the rebase process the commit sha's are re-written and it has diverged from what is in your remote fork (Github).
323
324To force push a branch you can:
325
326```shell
327git push head --force-with-lease
328```
329
330> Note: the `--force-with-lease` is recommended over `--force` because it is safer because it will check if the remote branch had new commits added during your rebase. You can read more detail here: <https://itnext.io/git-force-vs-force-with-lease-9d0e753e8c41>
331
332### Fixup commits during PR review
333
334One of the major downsides to rebasing a branch is that it requires force pushing over the remote (Github) which then marks all the existing review history outdated. This makes it hard for a reviewer to figure out whether or not the new changes addressed the feedback.
335
336One way you can help the reviewer out is by using **fixup** commits. Fixup commits are special git commits that append `fixup!` to the subject of a commit. `Git` provides tools for easily creating these and also squashing them after the PR review process is done.
337
338Since this is a new commit on top of the other commits, you will not lose your previous review and the new commit can be reviewed independently to determine if the new changes addressed the feedback correctly. Then once the reviewers are happy we will ask you to squash them so that we when it is merged we will maintain a clean linear history.
339
340Here is a quick read on it: <https://jordanelver.co.uk/blog/2020/06/04/fixing-commits-with-git-commit-fixup-and-git-rebase-autosquash/>
341
342TL;DR;
343
344```shell
345# make code change and create new commit
346git commit --fixup <sha>
347
348# push to Github for review
349git push
350
351# reviewers are happy and ask you to do a final rebase before merging
352git rebase -i --autosquash master
353
354# final push before merging
355git push --force-with-lease
356```
357
358## Development Workflow
359
360This section introduces the development workflow used for this repository. It is recommended that both Contributors, Release Engineers and Maintainers familiarize themselves with this content.
361
362### Branching Strategy
363
364This repository follows a trunk based development workflow. Depending on what article you read there are slight nuances to this so this section will outline how this repository interprets that workflow.
365
366The most important branch is `master` this is our **Next Release** version and it should always be in a shippable state. This means that CI should be green and at any point we can decided to ship a new release from it. In a traditional trunk based development workflow, developers are encouraged to land partially finished work daily and to keep that work hidden behind feature flags. This repository does **NOT** follow that and instead if code lands on master it is something we are comfortable with shipping.
367
368We ship release candidate (RC) builds from the `master` branch (current major) and also from `release/v{major.minor}` branches (last major version) during our development cycles. Therefore, it is important that it remains shippable at all times!
369
370When we do a final release then we will cut a new `release/v{major.minor}` branch. These are long lived release branches which capture a snapshot in time for that release. For example here are some of the current release branches (as of writing this):
371
372- release/v3.2
373- release/v3.1
374- release/v3.0
375- release/v2.4
376- release/v2.3
377- release/v1.14
378
379These branches contain the codebase as it was at that time when the release was done. These branches have branch protection enabled to ensure that they are not removed or accidently overwritten. If we needed to do a security fix or bug patch then we may cut a new `.Z` patch release from an existing release branch. For example, the `release/v2.4` branch is currently on `2.4.1`.
380
381As you can see we currently support mutliple major versions of Emissary-ingress and you can read more about our [End-of-Life Policy](https://www.getambassador.io/docs/emissary/latest/about/aes-emissary-eol/).
382
383For more information on our current RC and Release process you can find that in our [Release Wiki](https://github.com/emissary-ingress/emissary/wiki).
384
385### Backport Strategy
386
387Since we follow a trunk based development workflow this means that the majority of the time your patch branch will be based off from `master` and that most Pull Request will target `master`.
388
389This ensures that we do not miss bug fixes or features for the "Next" shippable release and simplifies the mental-model for deciding how to get started contributing code.
390
391#### What if I need a patch to land in a previous supported version?
392
393Let's say I have a bug fix for CRD round trip conversion for AuthService, which is affecting both `v2.y` and `v3.y`.
394
395First within the issue we should discuss what versions we want to target. This can depend on current cycle work and any upcoming releases we may have.
396
397The general rules we follow are:
398
3991. land patch in "next" version which is `master`
4002. backport patch to any `release/v{major}.{minor}` branches
401
402So, let's say we discuss it and say that the "next" major version is a long ways away so we want to do a z patch release on our current minor version(`v3.2`) and we also want to do a z patch release on our last supported major version (`v2.4`).
403
404This means that these patches need to land in three separate branches:
405
4061. `master` - next release
4072. `release/v3.2` - patch release
4083. `release/v2.4` - patch release
409
410In this scenario, we first ask you to land the patch in the `master` branch and then provide separate PR's with the commits backported onto the `release/v*` branches.
411
412> Recommendation: using the `git cherry-pick -x` will add the source commit sha to the commit message. This helps with tracing work back to the original commit.
413
414#### What if my patch is only for a previous supported version?
415
416Although, this should be an edge case, it does happen where the code has diverged enough that a fix may only be relevant to an existing supported version. In these cases we may need to do a patch release for that older supported version.
417
418A good example, if we were to find a bug in the Envoy v2 protocol configuration we would only want to target the v2 release.
419
420In this scenario, the base branch that we would create our feature branch off from would be the latest `minor` version for that release. As of writing this, that would be the `release/v2.4` branch. We would **not** need to target master.
421
422But, let's say during our fix we notice other things that need to be addressed that would also need to be fixed in `master`. Then you need to submit a **separate Pull Request** that should first land on master and then follow the normal backporting process for the other patches.
423
424#### What if I'm still not sure?
425
426This is what the issue discussions and disucssion in Slack are for so that we can help guide you so feel free to ping us in the `#emissary-dev` channel on Slack to discuss directly with us.
427
428### Merge Strategy
429
430> The audience for this section is the Maintainers but also beneficial for Contributors so that they are familiar with how the project operates.
431
432Having a clean linear commit history for a repository makes it easier to understand what is being changed and reduces the mental load for new comers to the project.
433
434To maintain a clean linear commit history the following rules should be followed:
435
436First, always rebase patch branch on to base branch. This means **NO** merge commits from merging base branch into the patch branch. This can be accomplished using git rebase.
437
438```shell
439# first, make sure you pull latest upstream changes
440git fetch upstream
441git checkout master
442git pull -ff upstream/master
443
444# checkout patch branch and rebase interactive
445# you may have merge conflicts you need to resolve
446git checkout my-patch-branch
447git rebase -i master
448```
449
450> Note: this does rewrite your commit shas so be aware when sharing branches with co-workers.
451
452Once the Pull Request is reviewed and has **two approvals** then a Maintainer can merge. Maintainers should follow prefer the following merge strategies:
453
4541. rebase and merge
4552. squash merge
456
457When `rebase and merge` is used your commits are played on top of the base branch so that it creates a clean linear history. This will maintain all the commits from the Pull Request. In most cases this should be the **preferred** merge strategy.
458
459When a Pull Request has lots of fixup commits, or pr feedback fixes then you should ask the Contributor to squash them as part of the PR process.
460
461If the contributor is unable to squash them then using a `squash merge` in some cases makes sense. **IMPORTANT**, when this does happen it is important that the commit messages are cleaned up and not just blindly accepted the way proposed by Github. Since it is easy to miss that cleanup step, this should be used less frequently compared to `rebase and merge`.
462
463#### What about merge commit strategy?
464
465> The audience for this section is the Maintainers but also beneficial for Contributors so that they are familiar with how the project operates.
466
467When maintaining a linear commit history, each commit tells the story of what was changed in the repository. When using `merge commits` it
468adds an additional commit to the history that is not necessary because the commit history and PR history already tell the story.
469
470Now `merge commits` can be useful when you are concerned with not rewriting the commit sha. Based on the current release process which includes using `rel/v` branches that are tagged and merged into `release/v` branches we must use a `merge commit` when merging these branches. This ensures that the commit sha a Git Tag is pointing at still exists once merged into the `release/v` branch.
471
472## Contributing to the Docs
473
474The Emissary-ingress community will all benefit from having documentation that is useful and correct. If you have found an issue with the end user documentation, then please help us out by submitting an issue and/or pull request with a fix!
475
476The end user documentation for Emissary-ingress lives in a different repository and can be found at <https://github.com/datawire/ambassador-docs>.
477
478See this repository for details on how to contribute to either a `pre-release` or already-released version of Emissary-ingress.
479
480## Advanced Topics
481
482This section is for more advanced topics that provide more detailed instructions. Make sure you go through the Development Setup and read the Architecture document before exploring these topics.
483
484### Running Emissary-ingress internals locally
485
486The main entrypoint is written in go. It strives to be as compatible as possible
487with the normal go toolchain. You can run it with:
488
489```bash
490go run ./cmd/busyambassador entrypoint
491```
492
493Of course just because you can run it this way does not mean it will succeed.
494The entrypoint needs to launch `diagd` and `envoy` in order to function, and it
495also expect to be able to write to the `/ambassador` directory.
496
497#### Setting up diagd
498
499If you want to hack on diagd, its easiest to setup a virtualenv with an editable
500copy and launch your `go run` from within that virtualenv. Note that these
501instructions depend on the virtualenvwrapper
502(<https://virtualenvwrapper.readthedocs.io/en/latest/>) package:
503
504```bash
505# Create a virtualenv named venv with all the python requirements
506# installed.
507python3 -m venv venv
508. venv/bin/activate
509# If you're doing this in Datawire's apro.git, then:
510cd ambassador
511# Update pip and install dependencies
512pip install --upgrade pip
513pip install orjson # see below
514pip install -r builder/requirements.txt
515# Created an editable installation of ambassador:
516pip install -e python/
517# Check that we do indeed have diagd in our path.
518which diagd
519# If you're doing this in Datawire's apro.git, then:
520cd ..
521```
522
523(Note: it shouldn't be necessary to install `orjson` by hand. The fact that it is
524at the moment is an artifact of the way Ambassador builds currently happen.)
525
526#### Changing the ambassador root
527
528You should now be able to launch ambassador if you set the
529`ambassador_root` environment variable to a writable location:
530
531 ambassador_root=/tmp go run ./cmd/busyambassador entrypoint
532
533#### Getting envoy
534
535If you do not have envoy in your path already, the entrypoint will use
536docker to run it. At the moment this is untested for macs which probably
537means it is broken since localhost communication does not work by
538default on macs. This can be made to work as soon an intrepid volunteer
539with a mac reaches out to me (rhs@datawire.io).
540
541#### Shutting up the pod labels error
542
543An astute observe of the logs will notice that ambassador complains
544vociferously that pod labels are not mounted in the ambassador
545container. To reduce this noise, you can:
546
547```bash
548mkdir /tmp/ambassador-pod-info && touch /tmp/ambassador-pod-info/labels
549```
550
551#### Extra credit
552
553When you run ambassador locally it will configure itself exactly as it
554would in the cluster. That means with two caveats you can actually
555interact with it and it will function normally:
556
5571. You need to run `telepresence connect` or equivalent so it can
558 connect to the backend services in its configuration.
559
5602. You need to supply the host header when you talk to it.
561
562### Debugging and Developing Envoy Configuration
563
564Envoy configuration is generated by the ambassador compiler. Debugging
565the ambassador compiler by running it in kubernetes is very slow since
566we need to push both the code and any relevant kubernetes resources
567into the cluster. The following sections will provide tips for improving
568this development experience.
569
570#### Mockery
571
572Fortunately we have the `mockery` tool which lets us run the compiler
573code directly on kubernetes resources without having to push that code
574or the relevant kubernetes resources into the cluster. This is the
575fastest way to hack on and debug the compiler.
576
577The `mockery` tool runs inside the Docker container used to build
578Ambassador, using `make shell`, so it's important to realize that it
579won't have access to your entire filesystem. There are two easy ways
580to arrange to get data in and out of the container:
581
5821. If you `make sync`, everything in the Ambassador source tree gets rsync'd
583 into the container's `/buildroot/ambassador`. The first time you start the
584 shell, this can take a bit, but after that it's pretty fast. You'll
585 probably need to use `docker cp` to get data out of the container, though.
586
5872. You may be able to use Docker volume mounts by exporting `BUILDER_MOUNTS`
588 with the appropriate `-v` switches before running `make shell` -- e.g.
589
590 ```bash
591 export BUILDER_MOUNTS=$(pwd)/xfer:/xfer
592 make shell
593 ```
594
595 will cause the dev shell to mount `xfer` in your current directory as `/xfer`.
596 This is known to work well on MacOS (though volume mounts are slow on Mac,
597 so moving gigabytes of data around this way isn't ideal).
598
599Once you've sorted out how to move data around:
600
6011. Put together a set of Ambassador configuration CRDs in a file that's somewhere
602 that you'll be able to get them into the builder container. The easy way to do
603 this is to use the files you'd feed to `kubectl apply`; they should be actual
604 Kubernetes objects with `metadata` and `spec` sections, etc. (If you want to
605 use annotations, that's OK too, just put the whole `Service` object in there.)
606
6072. Run `make compile shell` to build everything and start the dev shell.
608
6093. From inside the build shell, run
610
611 ```bash
612 mockery $path_to_your_file
613 ```
614
615 If you're using a non-default `ambassador_id` you need to provide it in the
616 environment:
617
618 ```bash
619 AMBASSADOR_ID=whatever mockery $path_to_your_file
620 ```
621
622 Finally, if you're trying to mimic `KAT`, copy the `/tmp/k8s-AmbassadorTest.yaml`
623 file from a KAT run to use as input, then
624
625 ```bash
626 mockery --kat $kat_test_name $path_to_k8s_AmbassadorTest.yaml
627 ```
628
629 where `$kat_test_name` is the class name of a `KAT` test class, like `LuaTest` or
630 `TLSContextTest`.
631
6324. Once it's done, `/tmp/ambassador/snapshots` will have all the output from the
633 compiler phase of Ambassador.
634
635The point of `mockery` is that it mimics the configuration cycle of real Ambassador,
636without relying at all on a Kubernetes cluster. This means that you can easily and
637quickly take a Kubernetes input and look at the generated Envoy configuration without
638any other infrastructure.
639
640#### Ambassador Dump
641
642The `ambassador dump` tool is also useful for debugging and hacking on
643the compiler. After running `make shell`, you'll also be able to use
644the `ambassador` CLI, which can export the most import data structures
645that Ambassador works with as JSON. It works from an input which can
646be either a single file or a directory full of files in the following
647formats:
648
649- raw Ambassador resources like you'll find in the `demo/config` directory; or
650- an annotated Kubernetes resources like you'll find in `/tmp/k8s-AmbassadorTest.yaml` after running `make test`; or
651- a `watt` snapshot like you'll find in the `$AMBASSADOR_CONFIG_BASE_DIR/snapshots/snapshot.yaml` (which is a JSON file, I know, it's misnamed).
652
653Given an input source, running
654
655```bash
656ambassador dump --ir --v2 [$input_flags] $input > test.json
657```
658
659will dump the Ambassador IR and v2 Envoy configuration into `test.json`. Here
660`$input_flags` will be
661
662- nothing for raw Ambassador resources;
663- `--k8s` for Kubernetes resources; or
664- `--watt` for a `watt` snapshot.
665
666You can get more information with
667
668```bash
669ambassador dump --help
670```
671
672### Making changes to Envoy
673
674Emissary-ingress is built on top of Envoy and leverages a vendored version of Envoy (*we track upstream very closely*). This section will go into how to make changes to the Envoy that is packaged with Emissary-ingress.
675
676This is a bit more complex than anyone likes, but here goes:
677
678#### 1. Preparing your machine
679
680Building and testing Envoy can be very resource intensive. A laptop
681often can build Envoy... if you plug in an external hard drive, point
682a fan at it, and leave it running overnight and most of the next day.
683At Datawire, we'll often spin up a temporary build machine in GCE, so
684that we can build it very quickly.
685
686As of Envoy 1.15.0, we've measure the resource use to build and test
687it as:
688
689> | Command | Disk Size | Disk Used | Duration[1] |
690> |--------------------|-----------|-----------|-------------|
691> | `make update-base` | 450G | 12GB | ~11m |
692> | `make check-envoy` | 450G | 424GB | ~45m |
693>
694> [1] On a "Machine type: custom (32 vCPUs, 512 GB memory)" VM on GCE,
695> with the following entry in its `/etc/fstab`:
696>
697> ```bash
698> tmpfs:docker /var/lib/docker tmpfs size=450G 0 0
699> ```
700
701If you have the RAM, we've seen huge speed gains from doing the builds
702and tests on a RAM disk (see the `/etc/fstab` line above).
703
704#### 2. Setting up your workspace to hack on Envoy
705
7061. From your `ambassador.git` checkout, get Ambassador's current
707 version of the Envoy sources, and create a branch from that:
708
709 ```shell
710 make $PWD/_cxx/envoy
711 git -C _cxx/envoy checkout -b YOUR_BRANCHNAME
712 ```
713
7142. Tell the build system that, yes, you really would like to be
715 compiling envoy, as you'll be modifying Envoy:
716
717 ```shell
718 export YES_I_AM_OK_WITH_COMPILING_ENVOY=true
719 export ENVOY_COMMIT='-'
720 ```
721
722 Building Envoy is slow, and most Ambassador contributors do not
723 want to rebuild Envoy, so we require the first two environment
724 variables as a safety.
725
726 Setting `ENVOY_COMMIT=-` does 3 things:
727 1. Tell it to use whatever is currently checked out in
728 `./_cxx/envoy/` (instead of checking out a specific commit), so
729 that you are free to modify those sources.
730 2. Don't try to download a cached build of Envoy from a Docker
731 cache (since it wouldn't know which `ENVOY_COMMIT` do download
732 the cached build for).
733 3. Don't push the build of Envoy to a Docker cache (since you're
734 still actively working on it).
735
736#### 3. Hacking on Envoy
737
738Modify the sources in `./_cxx/envoy/`.
739
740#### 4. Building and testing your hacked-up Envoy
741
742- **Build Envoy** with `make update-base`. Again, this is *not* a
743 quick process. The build happens in a Docker container; you can
744 set `DOCKER_HOST` to point to a powerful machine if you like.
745
746- **Test Envoy** and run with Envoy's test suite (which we don't run
747 during normal Ambassador development) by running `make check-envoy`.
748 Be warned that Envoy's full **test suite** requires several hundred
749 gigabytes of disk space to run.
750
751 Inner dev-loop steps:
752
753 - To run just specific tests, instead of the whole test suite, set
754 the `ENVOY_TEST_LABEL` environment variable. For example, to run
755 just the unit tests in
756 `test/common/network/listener_impl_test.cc`, you should run
757
758 ```shell
759 ENVOY_TEST_LABEL='//test/common/network:listener_impl_test' make check-envoy
760 ```
761
762 - You can run `make envoy-shell` to get a Bash shell in the Docker
763 container that does the Envoy builds.
764
765 Interpreting the test results:
766
767 - If you see the following message, don't worry, it's harmless; the
768 tests still ran:
769
770 ```text
771 There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.
772 ```
773
774 The message means that the test passed, but it passed too
775 quickly, and Bazel is suggesting that you declare it as smaller.
776 Something along the lines of "This test only took 2s, but you
777 declared it as being in the 60s-300s ('moderate') bucket,
778 consider declaring it as being in the 0s-60s ('short')
779 bucket".
780
781 Don't be confused (as I was) in to thinking that it was saying
782 that the test was too big and was skipped and that you need to
783 throw more hardware at it.
784
785- **Build or test Ambassador** with the usual `make` commands, with
786 the exception that you MUST run `make update-base` first whenever
787 Envoy needs to be recompiled; it won't happen automatically. So
788 `make test` to build-and-test Ambassador would become `make
789 update-base && make test`, and `make images` to just build
790 Ambassador would become `make update-base && make images`. By
791 default (to keep the tests fast), the tests avoid running much
792 traffic through Envoy, and instead just check that the Envoy
793 configuration that Ambassador generates hasn't changed since the
794 previous version (since we generally trust that Envoy works, and
795 doesn't change as often). Since you *are* changing Envoy, you'll
796 need to run the tests with `KAT_RUN_MODE=envoy` set in the
797 environment in order to actually test against Envoy.
798
799#### 5. Finalizing your changes
800
801Once you're happy with your changes to Envoy:
802
8031. Ensure they're committed to `_cxx/envoy/` and push/PR them into
804 <https://github.com/datawire/envoy> branch `rebase/master`.
805
806 If you're outside of Datawire, you'll need to
807 a. Create a fork of <https://github.com/datawire/envoy> on the
808 GitHub web interface
809 b. Add it as a remote to your `./_cxx/envoy/`:
810 `git remote add my-fork git@github.com:YOUR_USERNAME/envoy.git`
811 c. Push the branch to that fork:
812 `git push my-fork YOUR_BRANCHNAME`
813
8142. Update `ENVOY_COMMIT` in `_cxx/envoy.mk`
815
8163. Unset `ENVOY_COMMIT=-` and run a final `make update-base` to
817 push a cached build:
818
819 ```shell
820 export YES_I_AM_OK_WITH_COMPILING_ENVOY=true
821 unset ENVOY_COMMIT
822 make update-base
823 ```
824
825 The image will be pushed to `$ENVOY_DOCKER_REPO`, by default
826 `ENVOY_DOCKER_REPO=docker.io/datawire/ambassador-base`; if you're
827 outside of Datawire, you can skip this step if you don't want to
828 share your Envoy binary anywhere. If you don't skip this step,
829 you'll need to `export
830 ENVOY_DOCKER_REPO=${your-envoy-docker-registry}` to tell it to push
831 somewhere other than Datawire's registry.
832
833 If you're at Datawire, you'll then want to make sure that the image
834 is also pushed to the backup container registries:
835
836 ```shell
837 TAG=GET_THIS_FROM_THE_make_update-base_OUTPUT
838
839 source_registry=docker.io/datawire
840 docker pull "$source_registry/ambassador-base:$TAG"
841 for target_registry in quay.io/datawire gcr.io/datawire; do
842 docker tag "$source_registry/ambassador-base:$TAG" "$target_registry/ambassador-base:$TAG"
843 docker push "$target_registry/ambassador-base:$TAG"
844 done
845 ```
846
847 If you're outside of Datawire, you can skip this step if you
848 don't want to share your Envoy binary anywhere. If you don't
849 skip this step, you'll need to `export
850 ENVOY_DOCKER_REPO=${your-envoy-docker-registry}` to tell it to
851 push somewhere other than Datawire's registry.
852
8534. Push/PR the `envoy.mk` `ENVOY_COMMIT` change to
854 <https://github.com/datawire/ambassador> (or
855 <https://github.com/datawire/apro> if you're inside Datawire).
856
857#### 6. Checklist for landing the changes
858
859I'd put this in the pull request template, but so few PRs change Envoy...
860
861- [ ] The image has been pushed to...
862 - [ ] `docker.io/datawire/ambassador-base`
863 - [ ] `gcr.io/datawire/ambassador-base`
864- [ ] The envoy.git commit has been tagged as `datawire-$(git
865 describe --tags --match='v*')` (the `--match` is to prevent
866 `datawire-*` tags from stacking on each other).
867- [ ] It's been tested with...
868 - [ ] `make check-envoy`
869
870The `check-envoy-version` CI job should check all of those things,
871except for `make check-envoy`.
872
873### Developing Emissary-ingress (Datawire-only advice)
874
875At the moment, these techniques will only work internally to Datawire. Mostly
876this is because they require credentials to access internal resources at the
877moment, though in several cases we're working to fix that.
878
879#### Updating license documentation
880
881When new dependencies are added or existing ones are updated, run
882`make generate` and commit changes to `DEPENDENCIES.md` and
883`DEPENDENCY_LICENSES.md`
884
885#### Upgrading Python dependencies
886
887Delete `python/requirements.txt`, then run `make generate`.
888
889If there are some dependencies you don't want to upgrade, but want to
890upgrade everything else, then
891
892 1. Remove from `python/requirements.txt` all of the entries except
893 for those you want to pin.
894 2. Delete `python/requirements.in` (if it exists).
895 3. Run `make generate`.
896
897## FAQ
898
899This section contains a set of Frequently Asked Questions that may answer a question you have. Also, feel free to ping us in Slack.
900
901### How do I find out what build targets are available?
902
903Use `make help` and `make targets` to see what build targets are
904available along with documentation for what each target does.
905
906### How do I develop on a Mac with Apple Silicon?
907
908To ensure that developers using a Mac with Apple Silicon can contribute, the build system ensures
909the build artifacts are `linux/amd64` rather than the host architecture. This behavior can be overriden
910using the `BUILD_ARCH` environment variable (e.g. `BUILD_ARCH=linux/arm64 make images`).
911
912### How do I develop on Windows using WSL?
913
914As the Emissary-ingress build system requires docker communication via a UNIX socket, using WSL 1 is not possible.
915Not even with a `DOCKER_HOST` environment variable set. As a result, you have to use WSL 2, including using the
916WSL 2 version of docker-for-windows.
917
918Additionally, if your hostname contains an upper-case character, the build script will break. This is based on the
919`NAME` environment variable, which should contain your hostname. You can solve this issue by doing `export NAME=my-lowercase-host-name`.
920If you do this *after* you've already run `make images` once, you will manually have to clean up the docker images
921that have been created using your upper-case host name.
922
923### How do I test using a private Docker repository?
924
925If you are pushing your development images to a private Docker repo,
926then:
927
928```sh
929export DEV_USE_IMAGEPULLSECRET=true
930export DOCKER_BUILD_USERNAME=...
931export DOCKER_BUILD_PASSWORD=...
932```
933
934and the test machinery should create an `imagePullSecret` from those Docker credentials such that it can pull the images.
935
936### How do I change the loglevel at runtime?
937
938```console
939curl localhost:8877/ambassador/v0/diag/?loglevel=debug
940```
941
942Note: This affects diagd and Envoy, but NOT the AES `amb-sidecar`.
943See the AES `DEVELOPING.md` for how to do that.
944
945### Can I build from a docker container instead of on my local computer?
946
947If you want to build within a container instead of setting up dependencies on your local machine then you can run the build within a docker container and leverage "Docker in Docker" to build it.
948
9491. `docker pull docker:latest`
9502. `docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -it docker:latest sh`
9513. `apk add --update --no-cache bash build-base go curl rsync python3 python2 git libarchive-tools gawk jq`
9524. `git clone https://github.com/emissary-ingress/emissary.git && cd emissary`
9535. `make images`
954
955Steps 0 and 1 are run on your machine, and 2 - 4 are from within the docker container. The base image is a "Docker in Docker" image, ran with `-v /var/run/docker.sock:/var/run/docker.sock` in order to connect to your local daemon from the docker inside the container. More info on Docker in Docker [here](https://hub.docker.com/_/docker).
956
957The images will be created and tagged as defined above, and will be available in docker on your local machine.
958
959### How do I clear everything out to make sure my build runs like it will in CI?
960
961Use `make clobber` to completely remove all derived objects, all cached artifacts, everything, and get back to a clean slate. This is recommended if you change branches within a clone, or if you need to `make generate` when you're not *certain* that your last `make generate` was using the same Envoy version.
962
963Use `make clean` to remove derived objects, but *not* clear the caches.
964
965### My editor is changing `go.mod` or `go.sum`, should I commit that?
966
967If you notice this happening, run `make go-mod-tidy`, and commit that.
968
969(If you're in Ambassador Labs, you should do this from `apro/`, not
970`apro/ambassador/`, so that apro.git's files are included too.)
971
972### How do I debug "This should not happen in CI" errors?
973
974These checks indicate that some output file changed in the middle of a
975run, when it should only change if a source file has changed. Since
976CI isn't editing the source files, this shouldn't happen in CI!
977
978This is problematic because it means that running the build multiple
979times can give different results, and that the tests are probably not
980testing the same image that would be released.
981
982These checks will show you a patch showing how the output file
983changed; it is up to you to figure out what is happening in the
984build/test system that would cause that change in the middle of a run.
985For the most part, this is pretty simple... except when the output
986file is a Docker image; you just see that one image hash is different
987than another image hash.
988
989Fortunately, the failure showing the changed image hash is usually
990immediately preceded by a `docker build`. Earlier in the CI output,
991you should find an identical `docker build` command from the first time it
992ran. In the second `docker build`'s output, each step should say
993`---> Using cache`; the first few steps will say this, but at some
994point later steps will stop saying this; find the first step that is
995missing the `---> Using cache` line, and try to figure out what could
996have changed between the two runs that would cause it to not use the
997cache.
998
999If that step is an `ADD` command that is adding a directory, the
1000problem is probably that you need to add something to `.dockerignore`.
1001To help figure out what you need to add, try adding a `RUN find
1002DIRECTORY -exec ls -ld -- {} +` step after the `ADD` step, so that you
1003can see what it added, and see what is different on that between the
1004first and second `docker build` commands.
1005
1006### How do I run Emissary-ingress tests?
1007
1008- `export DEV_REGISTRY=<your-dev-docker-registry>` (you need to be logged in and have permission to push)
1009- `export DEV_KUBECONFIG=<your-dev-kubeconfig>`
1010
1011If you want to run the Go tests for `cmd/entrypoint`, you'll need `diagd`
1012in your `PATH`. See the instructions below about `Setting up diagd` to do
1013that.
1014
1015| Group | Command |
1016|-----------------|---------------------------------------------------------------------|
1017| All Tests | `make test` |
1018| All Golang | `make gotest` |
1019| All Python | `make pytest` |
1020| Some/One Golang | `make gotest GOTEST_PKGS=./cmd/entrypoint GOTEST_ARGS="-run TestName"` |
1021| Some/One Python | `make pytest PYTEST_ARGS="-k TestName"` |
1022
1023Please note the python tests use a local cache to speed up test
1024results. If you make a code update that changes the generated envoy
1025configuration, those tests will fail and you will need to update the
1026python test cache.
1027
1028Note that it is invalid to run one of the `main[Plain.*]` Python tests
1029without running all of the other `main[Plain*]` tests; the test will
1030fail to run (not even showing up as a failure or xfail--it will fail
1031to run at all). For example, `PYTEST_ARGS="-k WebSocket"` would match
1032the `main[Plain.WebSocketMapping-GRPC]` test, and that test would fail
1033to run; one should instead say `PYTEST_ARGS="-k Plain or WebSocket"`
1034to avoid breaking the sub-tests of "Plain".
1035
1036### How do I update the python test cache?
1037
1038- First, run `make KAT_RUN_MODE=envoy pytest` to do a test run *without*
1039 using the local cache.
1040
1041- Once that succeeds, use `make pytest-gold` to update the cache from
1042 the passing tests.
1043
1044### How do I type check my python code?
1045
1046Ambassador uses Python 3 type hinting and the `mypy` static type checker to
1047help find bugs before runtime. If you haven't worked with hinting before, a
1048good place to start is
1049[the `mypy` cheat sheet](https://mypy.readthedocs.io/en/latest/cheat_sheet_py3.html).
1050
1051New code must be hinted, and the build process will verify that the type
1052check passes when you `make test`. Fair warning: this means that
1053PRs will not pass CI if the type checker fails.
1054
1055We strongly recommend using an editor that can do realtime type checking
1056(at Datawire we tend to use PyCharm and VSCode a lot, but many many editors
1057can do this now) and also running the type checker by hand before submitting
1058anything:
1059
1060- `make lint/mypy` will check all the Ambassador code
1061
1062Ambassador code should produce *no* warnings and *no* errors.
1063
1064If you're concerned that the mypy cache is somehow wrong, delete the
1065`.mypy_cache/` directory to clear the cache.
1066
1067### How do I get the source code for a release?
1068
1069The current shipping release of Ambassador lives on the `master`
1070branch. It is tagged with its version (e.g. `v0.78.0`).
1071
1072Changes on `master` after the last tag have not been released yet, but
1073will be included in the next release of Ambassador.
View as plain text