1# easyjson [](https://travis-ci.org/mailru/easyjson) [](https://goreportcard.com/report/github.com/mailru/easyjson)
2
3Package easyjson provides a fast and easy way to marshal/unmarshal Go structs
4to/from JSON without the use of reflection. In performance tests, easyjson
5outperforms the standard `encoding/json` package by a factor of 4-5x, and other
6JSON encoding packages by a factor of 2-3x.
7
8easyjson aims to keep generated Go code simple enough so that it can be easily
9optimized or fixed. Another goal is to provide users with the ability to
10customize the generated code by providing options not available with the
11standard `encoding/json` package, such as generating "snake_case" names or
12enabling `omitempty` behavior by default.
13
14## Usage
15```sh
16# install
17go get -u github.com/mailru/easyjson/...
18
19# run
20easyjson -all <file>.go
21```
22
23The above will generate `<file>_easyjson.go` containing the appropriate marshaler and
24unmarshaler funcs for all structs contained in `<file>.go`.
25
26Please note that easyjson requires a full Go build environment and the `GOPATH`
27environment variable to be set. This is because easyjson code generation
28invokes `go run` on a temporary file (an approach to code generation borrowed
29from [ffjson](https://github.com/pquerna/ffjson)).
30
31## Options
32```txt
33Usage of easyjson:
34 -all
35 generate marshaler/unmarshalers for all structs in a file
36 -build_tags string
37 build tags to add to generated file
38 -gen_build_flags string
39 build flags when running the generator while bootstrapping
40 -byte
41 use simple bytes instead of Base64Bytes for slice of bytes
42 -leave_temps
43 do not delete temporary files
44 -no_std_marshalers
45 don't generate MarshalJSON/UnmarshalJSON funcs
46 -noformat
47 do not run 'gofmt -w' on output file
48 -omit_empty
49 omit empty fields by default
50 -output_filename string
51 specify the filename of the output
52 -pkg
53 process the whole package instead of just the given file
54 -snake_case
55 use snake_case names instead of CamelCase by default
56 -lower_camel_case
57 use lowerCamelCase instead of CamelCase by default
58 -stubs
59 only generate stubs for marshaler/unmarshaler funcs
60 -disallow_unknown_fields
61 return error if some unknown field in json appeared
62 -disable_members_unescape
63 disable unescaping of \uXXXX string sequences in member names
64```
65
66Using `-all` will generate marshalers/unmarshalers for all Go structs in the
67file excluding those structs whose preceding comment starts with `easyjson:skip`.
68For example:
69
70```go
71//easyjson:skip
72type A struct {}
73```
74
75If `-all` is not provided, then only those structs whose preceding
76comment starts with `easyjson:json` will have marshalers/unmarshalers
77generated. For example:
78
79```go
80//easyjson:json
81type A struct {}
82```
83
84Additional option notes:
85
86* `-snake_case` tells easyjson to generate snake\_case field names by default
87 (unless overridden by a field tag). The CamelCase to snake\_case conversion
88 algorithm should work in most cases (ie, HTTPVersion will be converted to
89 "http_version").
90
91* `-build_tags` will add the specified build tags to generated Go sources.
92
93* `-gen_build_flags` will execute the easyjson bootstapping code to launch the
94 actual generator command with provided flags. Multiple arguments should be
95 separated by space e.g. `-gen_build_flags="-mod=mod -x"`.
96
97## Structure json tag options
98
99Besides standart json tag options like 'omitempty' the following are supported:
100
101* 'nocopy' - disables allocation and copying of string values, making them
102 refer to original json buffer memory. This works great for short lived
103 objects which are not hold in memory after decoding and immediate usage.
104 Note if string requires unescaping it will be processed as normally.
105* 'intern' - string "interning" (deduplication) to save memory when the very
106 same string dictionary values are often met all over the structure.
107 See below for more details.
108
109## Generated Marshaler/Unmarshaler Funcs
110
111For Go struct types, easyjson generates the funcs `MarshalEasyJSON` /
112`UnmarshalEasyJSON` for marshaling/unmarshaling JSON. In turn, these satisfy
113the `easyjson.Marshaler` and `easyjson.Unmarshaler` interfaces and when used in
114conjunction with `easyjson.Marshal` / `easyjson.Unmarshal` avoid unnecessary
115reflection / type assertions during marshaling/unmarshaling to/from JSON for Go
116structs.
117
118easyjson also generates `MarshalJSON` and `UnmarshalJSON` funcs for Go struct
119types compatible with the standard `json.Marshaler` and `json.Unmarshaler`
120interfaces. Please be aware that using the standard `json.Marshal` /
121`json.Unmarshal` for marshaling/unmarshaling will incur a significant
122performance penalty when compared to using `easyjson.Marshal` /
123`easyjson.Unmarshal`.
124
125Additionally, easyjson exposes utility funcs that use the `MarshalEasyJSON` and
126`UnmarshalEasyJSON` for marshaling/unmarshaling to and from standard readers
127and writers. For example, easyjson provides `easyjson.MarshalToHTTPResponseWriter`
128which marshals to the standard `http.ResponseWriter`. Please see the [GoDoc
129listing](https://godoc.org/github.com/mailru/easyjson) for the full listing of
130utility funcs that are available.
131
132## Controlling easyjson Marshaling and Unmarshaling Behavior
133
134Go types can provide their own `MarshalEasyJSON` and `UnmarshalEasyJSON` funcs
135that satisfy the `easyjson.Marshaler` / `easyjson.Unmarshaler` interfaces.
136These will be used by `easyjson.Marshal` and `easyjson.Unmarshal` when defined
137for a Go type.
138
139Go types can also satisfy the `easyjson.Optional` interface, which allows the
140type to define its own `omitempty` logic.
141
142## Type Wrappers
143
144easyjson provides additional type wrappers defined in the `easyjson/opt`
145package. These wrap the standard Go primitives and in turn satisfy the
146easyjson interfaces.
147
148The `easyjson/opt` type wrappers are useful when needing to distinguish between
149a missing value and/or when needing to specifying a default value. Type
150wrappers allow easyjson to avoid additional pointers and heap allocations and
151can significantly increase performance when used properly.
152
153## Memory Pooling
154
155easyjson uses a buffer pool that allocates data in increasing chunks from 128
156to 32768 bytes. Chunks of 512 bytes and larger will be reused with the help of
157`sync.Pool`. The maximum size of a chunk is bounded to reduce redundant memory
158allocation and to allow larger reusable buffers.
159
160easyjson's custom allocation buffer pool is defined in the `easyjson/buffer`
161package, and the default behavior pool behavior can be modified (if necessary)
162through a call to `buffer.Init()` prior to any marshaling or unmarshaling.
163Please see the [GoDoc listing](https://godoc.org/github.com/mailru/easyjson/buffer)
164for more information.
165
166## String interning
167
168During unmarshaling, `string` field values can be optionally
169[interned](https://en.wikipedia.org/wiki/String_interning) to reduce memory
170allocations and usage by deduplicating strings in memory, at the expense of slightly
171increased CPU usage.
172
173This will work effectively only for `string` fields being decoded that have frequently
174the same value (e.g. if you have a string field that can only assume a small number
175of possible values).
176
177To enable string interning, add the `intern` keyword tag to your `json` tag on `string`
178fields, e.g.:
179
180```go
181type Foo struct {
182 UUID string `json:"uuid"` // will not be interned during unmarshaling
183 State string `json:"state,intern"` // will be interned during unmarshaling
184}
185```
186
187## Issues, Notes, and Limitations
188
189* easyjson is still early in its development. As such, there are likely to be
190 bugs and missing features when compared to `encoding/json`. In the case of a
191 missing feature or bug, please create a GitHub issue. Pull requests are
192 welcome!
193
194* Unlike `encoding/json`, object keys are case-sensitive. Case-insensitive
195 matching is not currently provided due to the significant performance hit
196 when doing case-insensitive key matching. In the future, case-insensitive
197 object key matching may be provided via an option to the generator.
198
199* easyjson makes use of `unsafe`, which simplifies the code and
200 provides significant performance benefits by allowing no-copy
201 conversion from `[]byte` to `string`. That said, `unsafe` is used
202 only when unmarshaling and parsing JSON, and any `unsafe` operations
203 / memory allocations done will be safely deallocated by
204 easyjson. Set the build tag `easyjson_nounsafe` to compile it
205 without `unsafe`.
206
207* easyjson is compatible with Google App Engine. The `appengine` build
208 tag (set by App Engine's environment) will automatically disable the
209 use of `unsafe`, which is not allowed in App Engine's Standard
210 Environment. Note that the use with App Engine is still experimental.
211
212* Floats are formatted using the default precision from Go's `strconv` package.
213 As such, easyjson will not correctly handle high precision floats when
214 marshaling/unmarshaling JSON. Note, however, that there are very few/limited
215 uses where this behavior is not sufficient for general use. That said, a
216 different package may be needed if precise marshaling/unmarshaling of high
217 precision floats to/from JSON is required.
218
219* While unmarshaling, the JSON parser does the minimal amount of work needed to
220 skip over unmatching parens, and as such full validation is not done for the
221 entire JSON value being unmarshaled/parsed.
222
223* Currently there is no true streaming support for encoding/decoding as
224 typically for many uses/protocols the final, marshaled length of the JSON
225 needs to be known prior to sending the data. Currently this is not possible
226 with easyjson's architecture.
227
228* easyjson parser and codegen based on reflection, so it won't work on `package main`
229 files, because they cant be imported by parser.
230
231## Benchmarks
232
233Most benchmarks were done using the example
234[13kB example JSON](https://dev.twitter.com/rest/reference/get/search/tweets)
235(9k after eliminating whitespace). This example is similar to real-world data,
236is well-structured, and contains a healthy variety of different types, making
237it ideal for JSON serialization benchmarks.
238
239Note:
240
241* For small request benchmarks, an 80 byte portion of the above example was
242 used.
243
244* For large request marshaling benchmarks, a struct containing 50 regular
245 samples was used, making a ~500kB output JSON.
246
247* Benchmarks are showing the results of easyjson's default behaviour,
248 which makes use of `unsafe`.
249
250Benchmarks are available in the repository and can be run by invoking `make`.
251
252### easyjson vs. encoding/json
253
254easyjson is roughly 5-6 times faster than the standard `encoding/json` for
255unmarshaling, and 3-4 times faster for non-concurrent marshaling. Concurrent
256marshaling is 6-7x faster if marshaling to a writer.
257
258### easyjson vs. ffjson
259
260easyjson uses the same approach for JSON marshaling as
261[ffjson](https://github.com/pquerna/ffjson), but takes a significantly
262different approach to lexing and parsing JSON during unmarshaling. This means
263easyjson is roughly 2-3x faster for unmarshaling and 1.5-2x faster for
264non-concurrent unmarshaling.
265
266As of this writing, `ffjson` seems to have issues when used concurrently:
267specifically, large request pooling hurts `ffjson`'s performance and causes
268scalability issues. These issues with `ffjson` can likely be fixed, but as of
269writing remain outstanding/known issues with `ffjson`.
270
271easyjson and `ffjson` have similar performance for small requests, however
272easyjson outperforms `ffjson` by roughly 2-5x times for large requests when
273used with a writer.
274
275### easyjson vs. go/codec
276
277[go/codec](https://github.com/ugorji/go) provides
278compile-time helpers for JSON generation. In this case, helpers do not work
279like marshalers as they are encoding-independent.
280
281easyjson is generally 2x faster than `go/codec` for non-concurrent benchmarks
282and about 3x faster for concurrent encoding (without marshaling to a writer).
283
284In an attempt to measure marshaling performance of `go/codec` (as opposed to
285allocations/memcpy/writer interface invocations), a benchmark was done with
286resetting length of a byte slice rather than resetting the whole slice to nil.
287However, the optimization in this exact form may not be applicable in practice,
288since the memory is not freed between marshaling operations.
289
290### easyjson vs 'ujson' python module
291
292[ujson](https://github.com/esnme/ultrajson) is using C code for parsing, so it
293is interesting to see how plain golang compares to that. It is important to note
294that the resulting object for python is slower to access, since the library
295parses JSON object into dictionaries.
296
297easyjson is slightly faster for unmarshaling and 2-3x faster than `ujson` for
298marshaling.
299
300### Benchmark Results
301
302`ffjson` results are from February 4th, 2016, using the latest `ffjson` and go1.6.
303`go/codec` results are from March 4th, 2016, using the latest `go/codec` and go1.6.
304
305#### Unmarshaling
306
307| lib | json size | MB/s | allocs/op | B/op |
308|:---------|:----------|-----:|----------:|------:|
309| standard | regular | 22 | 218 | 10229 |
310| standard | small | 9.7 | 14 | 720 |
311| | | | | |
312| easyjson | regular | 125 | 128 | 9794 |
313| easyjson | small | 67 | 3 | 128 |
314| | | | | |
315| ffjson | regular | 66 | 141 | 9985 |
316| ffjson | small | 17.6 | 10 | 488 |
317| | | | | |
318| codec | regular | 55 | 434 | 19299 |
319| codec | small | 29 | 7 | 336 |
320| | | | | |
321| ujson | regular | 103 | N/A | N/A |
322
323#### Marshaling, one goroutine.
324
325| lib | json size | MB/s | allocs/op | B/op |
326|:----------|:----------|-----:|----------:|------:|
327| standard | regular | 75 | 9 | 23256 |
328| standard | small | 32 | 3 | 328 |
329| standard | large | 80 | 17 | 1.2M |
330| | | | | |
331| easyjson | regular | 213 | 9 | 10260 |
332| easyjson* | regular | 263 | 8 | 742 |
333| easyjson | small | 125 | 1 | 128 |
334| easyjson | large | 212 | 33 | 490k |
335| easyjson* | large | 262 | 25 | 2879 |
336| | | | | |
337| ffjson | regular | 122 | 153 | 21340 |
338| ffjson** | regular | 146 | 152 | 4897 |
339| ffjson | small | 36 | 5 | 384 |
340| ffjson** | small | 64 | 4 | 128 |
341| ffjson | large | 134 | 7317 | 818k |
342| ffjson** | large | 125 | 7320 | 827k |
343| | | | | |
344| codec | regular | 80 | 17 | 33601 |
345| codec*** | regular | 108 | 9 | 1153 |
346| codec | small | 42 | 3 | 304 |
347| codec*** | small | 56 | 1 | 48 |
348| codec | large | 73 | 483 | 2.5M |
349| codec*** | large | 103 | 451 | 66007 |
350| | | | | |
351| ujson | regular | 92 | N/A | N/A |
352
353\* marshaling to a writer,
354\*\* using `ffjson.Pool()`,
355\*\*\* reusing output slice instead of resetting it to nil
356
357#### Marshaling, concurrent.
358
359| lib | json size | MB/s | allocs/op | B/op |
360|:----------|:----------|-----:|----------:|------:|
361| standard | regular | 252 | 9 | 23257 |
362| standard | small | 124 | 3 | 328 |
363| standard | large | 289 | 17 | 1.2M |
364| | | | | |
365| easyjson | regular | 792 | 9 | 10597 |
366| easyjson* | regular | 1748 | 8 | 779 |
367| easyjson | small | 333 | 1 | 128 |
368| easyjson | large | 718 | 36 | 548k |
369| easyjson* | large | 2134 | 25 | 4957 |
370| | | | | |
371| ffjson | regular | 301 | 153 | 21629 |
372| ffjson** | regular | 707 | 152 | 5148 |
373| ffjson | small | 62 | 5 | 384 |
374| ffjson** | small | 282 | 4 | 128 |
375| ffjson | large | 438 | 7330 | 1.0M |
376| ffjson** | large | 131 | 7319 | 820k |
377| | | | | |
378| codec | regular | 183 | 17 | 33603 |
379| codec*** | regular | 671 | 9 | 1157 |
380| codec | small | 147 | 3 | 304 |
381| codec*** | small | 299 | 1 | 48 |
382| codec | large | 190 | 483 | 2.5M |
383| codec*** | large | 752 | 451 | 77574 |
384
385\* marshaling to a writer,
386\*\* using `ffjson.Pool()`,
387\*\*\* reusing output slice instead of resetting it to nil
View as plain text