1# Boulder Release Process
2
3A description and demonstration of the full process for tagging a normal weekly
4release, a "clean" hotfix release, and a "dirty" hotfix release.
5
6Once a release is tagged, it will be generally deployed to
7[staging](https://letsencrypt.org/docs/staging-environment/) and then to
8[production](https://acme-v02.api.letsencrypt.org/) over the next few days.
9
10## Goals
11
121. All development, including reverts and hotfixes needed to patch a broken
13 release, happens on the `main` branch of this repository. Code is never
14 deployed without being reviewed and merged here first, and code is never
15 landed on a release branch that isn't landed on `main` first.
16
172. Doing a normal release requires approximately zero thought. It Just Works.
18
193. Doing a hotfix release differs as little as possible from the normal release
20 process.
21
22## Release Schedule
23
24Boulder developers make a new release at the beginning of each week, typically
25around 10am PST **Monday**. Operations deploys the new release to the [staging
26environment](https://letsencrypt.org/docs/staging-environment/) on **Tuesday**,
27typically by 2pm PST. If there have been no issues discovered with the release
28from its time in staging, then on **Thursday** the operations team deploys the
29release to the production environment.
30
31Holidays, unexpected bugs, and other resource constraints may affect the above
32schedule and result in staging or production updates being skipped. It should be
33considered a guideline for normal releases but not a strict contract.
34
35## Release Structure
36
37All releases are tagged with a tag of the form `release-YYYY-MM-DD[x]`, where
38the `YYYY-MM-DD` is the date that the initial release is cut (usually the Monday
39of the current week), and the `[x]` is an optional lowercase letter suffix
40indicating that the release is an incremental hotfix release. For example, the
41second hotfix release (i.e. third release overall) in the third week of January
422022 was
43[`release-2022-01-18b`](https://github.com/letsencrypt/boulder/releases/tag/release-2022-01-18b).
44
45All release tags are signed with a key associated with a Boulder developer. Tag
46signatures are automatically verified by GitHub using the public keys that
47developer has uploaded, and are additionally checked before being built and
48deployed to our staging and production environments. Note that, due to how Git
49works, in order for a tag to be signed it must also have a message; we set the
50tag message to just be a slightly more readable version of the tag name.
51
52## Making a Release
53
54### Prerequisites
55
56* You must have a GPG key with signing capability:
57 * [Checking for existing GPG keys](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/checking-for-existing-gpg-keys)
58
59* If you don't have a GPG key with signing capability, create one:
60 * [Generating a new local GPG key](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/generating-a-new-gpg-key)
61 * [Generating a new Yubikey GPG key](https://support.yubico.com/hc/en-us/articles/360013790259-Using-Your-YubiKey-with-OpenPGP)
62
63* The signing GPG key must be added to your GitHub account:
64 * [Adding a new GPG key to your GitHub
65 account](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/adding-a-new-gpg-key-to-your-github-account)
66
67* `git` *may* need to be configured to call the correct GPG binary:
68 * The default: `git config --global gpg.program gpg` is correct for most Linux platforms
69 * On macOS and some Linux platforms: `git config --global gpg.program gpg2` is correct
70
71* `git` must be configured to use the correct GPG key:
72 * [Telling Git about your GPG key](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/telling-git-about-your-signing-key)
73
74* Understand the [process for signing tags](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/signing-tags)
75
76### Regular Releases
77
78Simply create a signed tag whose name and message both include the date that the
79release is being tagged (not the date that the release is expected to be
80deployed):
81
82```sh
83git tag -s -m "Boulder release $(date +%F)" -s "release-$(date +%F)"
84git push origin "release-$(date +%F)"
85```
86
87### Clean Hotfix Releases
88
89If a hotfix release is necessary, and the desired hotfix commits are the **only** commits which have landed on `main` since the initial release was cut (i.e. there are not any commits on `main` which we want to exclude from the hotfix release), then the hotfix tag can be created much like a normal release tag.
90
91If it is still the same day as an already-tagged release, increment the letter suffix of the tag:
92
93```sh
94git tag -s -m "Boulder hotfix release $(date +%F)a" -s "release-$(date +%F)a"
95git push origin "release-$(date +%F)a"
96```
97
98If it is a new day, simply follow the regular release process above.
99
100### Dirty Hotfix Release
101
102If a hotfix release is necessary, but `main` already contains both commits that
103we do and commits that we do not want to include in the hotfix release, then we
104must go back and create a release branch for just the desired commits to be
105cherry-picked to. Then, all subsequent hotfix releases will be tagged on this
106branch.
107
108The commands below assume that it is still the same day as the original release
109tag was created (hence the use of "`date +%F`"), but this may not always be the
110case. The rule is that the date in the release branch name should be identical
111to the date in the original release tag. Similarly, this may not be the first
112hotfix release; the rule is that the letter suffix should increment (e.g. "b",
113"c", etc.) for each hotfix release with the same date.
114
115```sh
116git checkout -b "release-branch-$(date +%F)" "release-$(date +%F)"
117git cherry-pick baddecaf
118git tag -s -m "Boulder hotfix release $(date +%F)a" "release-$(date +%F)a"
119git push origin "release-branch-$(date +%F)" "release-$(date +%F)a"
120```
121
122## Deploying Releases
123
124When doing a release, SRE's tooling will check that:
125
1261. GitHub shows that tests have passed for the commit at the planned release
127 tag.
128
1292. The planned release tag is an ancestor of the current `main` on GitHub, or
130 the planned release tag is equal to the head of a branch named
131 `release-branch-XXX`, and all commits between `main` and the head of that
132 branch are cherry-picks of commits which landed on `main` following the
133 normal review process.
View as plain text