1Gzip Middleware
2===============
3
4This Go package which wraps HTTP *server* handlers to transparently gzip the
5response body, for clients which support it.
6
7For HTTP *clients* we provide a transport wrapper that will do gzip decompression
8faster than what the standard library offers.
9
10Both the client and server wrappers are fully compatible with other servers and clients.
11
12This package is forked from the dead [nytimes/gziphandler](https://github.com/nytimes/gziphandler)
13and extends functionality for it.
14
15## Install
16```bash
17go get -u github.com/klauspost/compress
18```
19
20## Documentation
21
22[](https://pkg.go.dev/github.com/klauspost/compress/gzhttp)
23
24
25## Usage
26
27There are 2 main parts, one for http servers and one for http clients.
28
29### Client
30
31The standard library automatically adds gzip compression to most requests
32and handles decompression of the responses.
33
34However, by wrapping the transport we are able to override this and provide
35our own (faster) decompressor.
36
37Wrapping is done on the Transport of the http client:
38
39```Go
40func ExampleTransport() {
41 // Get an HTTP client.
42 client := http.Client{
43 // Wrap the transport:
44 Transport: gzhttp.Transport(http.DefaultTransport),
45 }
46
47 resp, err := client.Get("https://google.com")
48 if err != nil {
49 return
50 }
51 defer resp.Body.Close()
52
53 body, _ := ioutil.ReadAll(resp.Body)
54 fmt.Println("body:", string(body))
55}
56```
57
58Speed compared to standard library `DefaultTransport` for an approximate 127KB JSON payload:
59
60```
61BenchmarkTransport
62
63Single core:
64BenchmarkTransport/gzhttp-32 1995 609791 ns/op 214.14 MB/s 10129 B/op 73 allocs/op
65BenchmarkTransport/stdlib-32 1567 772161 ns/op 169.11 MB/s 53950 B/op 99 allocs/op
66BenchmarkTransport/zstd-32 4579 238503 ns/op 547.51 MB/s 5775 B/op 69 allocs/op
67
68Multi Core:
69BenchmarkTransport/gzhttp-par-32 29113 36802 ns/op 3548.27 MB/s 11061 B/op 73 allocs/op
70BenchmarkTransport/stdlib-par-32 16114 66442 ns/op 1965.38 MB/s 54971 B/op 99 allocs/op
71BenchmarkTransport/zstd-par-32 90177 13110 ns/op 9960.83 MB/s 5361 B/op 67 allocs/op
72```
73
74This includes both serving the http request, parsing requests and decompressing.
75
76### Server
77
78For the simplest usage call `GzipHandler` with any handler (an object which implements the
79`http.Handler` interface), and it'll return a new handler which gzips the
80response. For example:
81
82```go
83package main
84
85import (
86 "io"
87 "net/http"
88 "github.com/klauspost/compress/gzhttp"
89)
90
91func main() {
92 handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
93 w.Header().Set("Content-Type", "text/plain")
94 io.WriteString(w, "Hello, World")
95 })
96
97 http.Handle("/", gzhttp.GzipHandler(handler))
98 http.ListenAndServe("0.0.0.0:8000", nil)
99}
100```
101
102This will wrap a handler using the default options.
103
104To specify custom options a reusable wrapper can be created that can be used to wrap
105any number of handlers.
106
107```Go
108package main
109
110import (
111 "io"
112 "log"
113 "net/http"
114
115 "github.com/klauspost/compress/gzhttp"
116 "github.com/klauspost/compress/gzip"
117)
118
119func main() {
120 handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
121 w.Header().Set("Content-Type", "text/plain")
122 io.WriteString(w, "Hello, World")
123 })
124
125 // Create a reusable wrapper with custom options.
126 wrapper, err := gzhttp.NewWrapper(gzhttp.MinSize(2000), gzhttp.CompressionLevel(gzip.BestSpeed))
127 if err != nil {
128 log.Fatalln(err)
129 }
130
131 http.Handle("/", wrapper(handler))
132 http.ListenAndServe("0.0.0.0:8000", nil)
133}
134
135```
136
137
138### Performance
139
140Speed compared to [nytimes/gziphandler](https://github.com/nytimes/gziphandler) with default settings, 2KB, 20KB and 100KB:
141
142```
143λ benchcmp before.txt after.txt
144benchmark old ns/op new ns/op delta
145BenchmarkGzipHandler_S2k-32 51302 23679 -53.84%
146BenchmarkGzipHandler_S20k-32 301426 156331 -48.14%
147BenchmarkGzipHandler_S100k-32 1546203 818981 -47.03%
148BenchmarkGzipHandler_P2k-32 3973 1522 -61.69%
149BenchmarkGzipHandler_P20k-32 20319 9397 -53.75%
150BenchmarkGzipHandler_P100k-32 96079 46361 -51.75%
151
152benchmark old MB/s new MB/s speedup
153BenchmarkGzipHandler_S2k-32 39.92 86.49 2.17x
154BenchmarkGzipHandler_S20k-32 67.94 131.00 1.93x
155BenchmarkGzipHandler_S100k-32 66.23 125.03 1.89x
156BenchmarkGzipHandler_P2k-32 515.44 1345.31 2.61x
157BenchmarkGzipHandler_P20k-32 1007.92 2179.47 2.16x
158BenchmarkGzipHandler_P100k-32 1065.79 2208.75 2.07x
159
160benchmark old allocs new allocs delta
161BenchmarkGzipHandler_S2k-32 22 16 -27.27%
162BenchmarkGzipHandler_S20k-32 25 19 -24.00%
163BenchmarkGzipHandler_S100k-32 28 21 -25.00%
164BenchmarkGzipHandler_P2k-32 22 16 -27.27%
165BenchmarkGzipHandler_P20k-32 25 19 -24.00%
166BenchmarkGzipHandler_P100k-32 27 21 -22.22%
167
168benchmark old bytes new bytes delta
169BenchmarkGzipHandler_S2k-32 8836 2980 -66.27%
170BenchmarkGzipHandler_S20k-32 69034 20562 -70.21%
171BenchmarkGzipHandler_S100k-32 356582 86682 -75.69%
172BenchmarkGzipHandler_P2k-32 9062 2971 -67.21%
173BenchmarkGzipHandler_P20k-32 67799 20051 -70.43%
174BenchmarkGzipHandler_P100k-32 300972 83077 -72.40%
175```
176
177### Stateless compression
178
179In cases where you expect to run many thousands of compressors concurrently,
180but with very little activity you can use stateless compression.
181This is not intended for regular web servers serving individual requests.
182
183Use `CompressionLevel(-3)` or `CompressionLevel(gzip.StatelessCompression)` to enable.
184Consider adding a [`bufio.Writer`](https://golang.org/pkg/bufio/#NewWriterSize) with a small buffer.
185
186See [more details on stateless compression](https://github.com/klauspost/compress#stateless-compression).
187
188### Migrating from gziphandler
189
190This package removes some of the extra constructors.
191When replacing, this can be used to find a replacement.
192
193* `GzipHandler(h)` -> `GzipHandler(h)` (keep as-is)
194* `GzipHandlerWithOpts(opts...)` -> `NewWrapper(opts...)`
195* `MustNewGzipLevelHandler(n)` -> `NewWrapper(CompressionLevel(n))`
196* `NewGzipLevelAndMinSize(n, s)` -> `NewWrapper(CompressionLevel(n), MinSize(s))`
197
198By default, some mime types will now be excluded.
199To re-enable compression of all types, use the `ContentTypeFilter(gzhttp.CompressAllContentTypeFilter)` option.
200
201### Range Requests
202
203Ranged requests are not well supported with compression.
204Therefore any request with a "Content-Range" header is not compressed.
205
206To signify that range requests are not supported any "Accept-Ranges" header set is removed when data is compressed.
207If you do not want this behavior use the `KeepAcceptRanges()` option.
208
209### Flushing data
210
211The wrapper supports the [http.Flusher](https://golang.org/pkg/net/http/#Flusher) interface.
212
213The only caveat is that the writer may not yet have received enough bytes to determine if `MinSize`
214has been reached. In this case it will assume that the minimum size has been reached.
215
216If nothing has been written to the response writer, nothing will be flushed.
217
218## BREACH mitigation
219
220[BREACH](http://css.csail.mit.edu/6.858/2020/readings/breach.pdf) is a specialized attack where attacker controlled data
221is injected alongside secret data in a response body. This can lead to sidechannel attacks, where observing the compressed response
222size can reveal if there are overlaps between the secret data and the injected data.
223
224For more information see https://breachattack.com/
225
226It can be hard to judge if you are vulnerable to BREACH.
227In general, if you do not include any user provided content in the response body you are safe,
228but if you do, or you are in doubt, you can apply mitigations.
229
230`gzhttp` can apply [Heal the Breach](https://ieeexplore.ieee.org/document/9754554), or improved content aware padding.
231
232```Go
233// RandomJitter adds 1->n random bytes to output based on checksum of payload.
234// Specify the amount of input to buffer before applying jitter.
235// This should cover the sensitive part of your response.
236// This can be used to obfuscate the exact compressed size.
237// Specifying 0 will use a buffer size of 64KB.
238// 'paranoid' will use a slower hashing function, that MAY provide more safety.
239// If a negative buffer is given, the amount of jitter will not be content dependent.
240// This provides *less* security than applying content based jitter.
241func RandomJitter(n, buffer int, paranoid bool) option
242...
243```
244
245The jitter is added as a "Comment" field. This field has a 1 byte overhead, so actual extra size will be 2 -> n+1 (inclusive).
246
247A good option would be to apply 32 random bytes, with default 64KB buffer: `gzhttp.RandomJitter(32, 0, false)`.
248
249Note that flushing the data forces the padding to be applied, which means that only data before the flush is considered for content aware padding.
250
251The *padding* in the comment is the text `Padding-Padding-Padding-Padding-Pad....`
252
253The *length* is `1 + crc32c(payload) MOD n` or `1 + sha256(payload) MOD n` (paranoid), or just random from `crypto/rand` if buffer < 0.
254
255### Paranoid?
256
257The padding size is determined by the remainder of a CRC32 of the content.
258
259Since the payload contains elements unknown to the attacker, there is no reason to believe they can derive any information
260from this remainder, or predict it.
261
262However, for those that feel uncomfortable with a CRC32 being used for this can enable "paranoid" mode which will use SHA256 for determining the padding.
263
264The hashing itself is about 2 orders of magnitude slower, but in overall terms will maybe only reduce speed by 10%.
265
266Paranoid mode has no effect if buffer is < 0 (non-content aware padding).
267
268### Examples
269
270Adding the option `gzhttp.RandomJitter(32, 50000)` will apply from 1 up to 32 bytes of random data to the output.
271
272The number of bytes added depends on the content of the first 50000 bytes, or all of them if the output was less than that.
273
274Adding the option `gzhttp.RandomJitter(32, -1)` will apply from 1 up to 32 bytes of random data to the output.
275Each call will apply a random amount of jitter. This should be considered less secure than content based jitter.
276
277This can be used if responses are very big, deterministic and the buffer size would be too big to cover where the mutation occurs.
278
279## License
280
281[Apache 2.0](LICENSE)
282
283
View as plain text