1FlexBuffers {#flexbuffers}
2==========
3
4FlatBuffers was designed around schemas, because when you want maximum
5performance and data consistency, strong typing is helpful.
6
7There are however times when you want to store data that doesn't fit a
8schema, because you can't know ahead of time what all needs to be stored.
9
10For this, FlatBuffers has a dedicated format, called FlexBuffers.
11This is a binary format that can be used in conjunction
12with FlatBuffers (by storing a part of a buffer in FlexBuffers
13format), or also as its own independent serialization format.
14
15While it loses the strong typing, you retain the most unique advantage
16FlatBuffers has over other serialization formats (schema-based or not):
17FlexBuffers can also be accessed without parsing / copying / object allocation.
18This is a huge win in efficiency / memory friendly-ness, and allows unique
19use cases such as mmap-ing large amounts of free-form data.
20
21FlexBuffers' design and implementation allows for a very compact encoding,
22combining automatic pooling of strings with automatic sizing of containers to
23their smallest possible representation (8/16/32/64 bits). Many values and
24offsets can be encoded in just 8 bits. While a schema-less representation is
25usually more bulky because of the need to be self-descriptive, FlexBuffers
26generates smaller binaries for many cases than regular FlatBuffers.
27
28FlexBuffers is still slower than regular FlatBuffers though, so we recommend to
29only use it if you need it.
30
31
32# Usage in C++
33
34Include the header `flexbuffers.h`, which in turn depends on `flatbuffers.h`
35and `util.h`.
36
37To create a buffer:
38
39~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp}
40flexbuffers::Builder fbb;
41fbb.Int(13);
42fbb.Finish();
43~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
44
45You create any value, followed by `Finish`. Unlike FlatBuffers which requires
46the root value to be a table, here any value can be the root, including a lonely
47int value.
48
49You can now access the `std::vector<uint8_t>` that contains the encoded value
50as `fbb.GetBuffer()`. Write it, send it, or store it in a parent FlatBuffer. In
51this case, the buffer is just 3 bytes in size.
52
53To read this value back, you could just say:
54
55~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp}
56auto root = flexbuffers::GetRoot(my_buffer);
57int64_t i = root.AsInt64();
58~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
59
60FlexBuffers stores ints only as big as needed, so it doesn't differentiate
61between different sizes of ints. You can ask for the 64 bit version,
62regardless of what you put in. In fact, since you demand to read the root
63as an int, if you supply a buffer that actually contains a float, or a
64string with numbers in it, it will convert it for you on the fly as well,
65or return 0 if it can't. If instead you actually want to know what is inside
66the buffer before you access it, you can call `root.GetType()` or `root.IsInt()`
67etc.
68
69Here's a slightly more complex value you could write instead of `fbb.Int` above:
70
71~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp}
72fbb.Map([&]() {
73 fbb.Vector("vec", [&]() {
74 fbb.Int(-100);
75 fbb.String("Fred");
76 fbb.IndirectFloat(4.0f);
77 });
78 fbb.UInt("foo", 100);
79});
80~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
81
82This stores the equivalent of the JSON value
83`{ vec: [ -100, "Fred", 4.0 ], foo: 100 }`. The root is a dictionary that has
84just two key-value pairs, with keys `vec` and `foo`. Unlike FlatBuffers, it
85actually has to store these keys in the buffer (which it does only once if
86you store multiple such objects, by pooling key values), but also unlike
87FlatBuffers it has no restriction on the keys (fields) that you use.
88
89The map constructor uses a C++11 Lambda to group its children, but you can
90also use more conventional start/end calls if you prefer.
91
92The first value in the map is a vector. You'll notice that unlike FlatBuffers,
93you can use mixed types. There is also a `TypedVector` variant that only
94allows a single type, and uses a bit less memory.
95
96`IndirectFloat` is an interesting feature that allows you to store values
97by offset rather than inline. Though that doesn't make any visible change
98to the user, the consequence is that large values (especially doubles or
9964 bit ints) that occur more than once can be shared (see ReuseValue).
100Another use case is inside of vectors, where the largest element makes
101up the size of all elements (e.g. a single double forces all elements to
10264bit), so storing a lot of small integers together with a double is more efficient if the double is indirect.
103
104Accessing it:
105
106~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp}
107auto map = flexbuffers::GetRoot(my_buffer).AsMap();
108map.size(); // 2
109auto vec = map["vec"].AsVector();
110vec.size(); // 3
111vec[0].AsInt64(); // -100;
112vec[1].AsString().c_str(); // "Fred";
113vec[1].AsInt64(); // 0 (Number parsing failed).
114vec[2].AsDouble(); // 4.0
115vec[2].AsString().IsTheEmptyString(); // true (Wrong Type).
116vec[2].AsString().c_str(); // "" (This still works though).
117vec[2].ToString().c_str(); // "4" (Or have it converted).
118map["foo"].AsUInt8(); // 100
119map["unknown"].IsNull(); // true
120~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
121
122
123# Usage in Java
124
125Java implementation follows the C++ one, closely.
126
127For creating the equivalent of the same JSON `{ vec: [ -100, "Fred", 4.0 ], foo: 100 }`,
128one could use the following code:
129
130~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java}
131FlexBuffersBuilder builder = new FlexBuffersBuilder(ByteBuffer.allocate(512),
132 FlexBuffersBuilder.BUILDER_FLAG_SHARE_KEYS_AND_STRINGS);
133int smap = builder.startMap();
134int svec = builder.startVector();
135builder.putInt(-100);
136builder.putString("Fred");
137builder.putFloat(4.0);
138builder.endVector("vec", svec, false, false);
139builder.putInt("foo", 100);
140builder.endMap(null, smap);
141ByteBuffer bb = builder.finish();
142~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
143
144Similarly, to read the data, just:
145
146~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java}
147FlexBuffers.Map map = FlexBuffers.getRoot(bb).asMap();
148map.size(); // 2
149FlexBuffers.Vector vec = map.get("vec").asVector();
150vec.size(); // 3
151vec.get(0).asLong(); // -100;
152vec.get(1).asString(); // "Fred";
153vec.get(1).asLong(); // 0 (Number parsing failed).
154vec.get(2).asFloat(); // 4.0
155vec.get(2).asString().isEmpty(); // true (Wrong Type).
156vec.get(2).asString(); // "" (This still works though).
157vec.get(2).toString(); // "4.0" (Or have it converted).
158map.get("foo").asUInt(); // 100
159map.get("unknown").isNull(); // true
160~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
161
162
163# Binary encoding
164
165A description of how FlexBuffers are encoded is in the
166[internals](@ref flatbuffers_internals) document.
167
168
169# Nesting inside a FlatBuffer
170
171You can mark a field as containing a FlexBuffer, e.g.
172
173 a:[ubyte] (flexbuffer);
174
175A special accessor will be generated that allows you to access the root value
176directly, e.g. `a_flexbuffer_root().AsInt64()`.
177
178
179# Efficiency tips
180
181* Vectors generally are a lot more efficient than maps, so prefer them over maps
182 when possible for small objects. Instead of a map with keys `x`, `y` and `z`,
183 use a vector. Better yet, use a typed vector. Or even better, use a fixed
184 size typed vector.
185* Maps are backwards compatible with vectors, and can be iterated as such.
186 You can iterate either just the values (`map.Values()`), or in parallel with
187 the keys vector (`map.Keys()`). If you intend
188 to access most or all elements, this is faster than looking up each element
189 by key, since that involves a binary search of the key vector.
190* When possible, don't mix values that require a big bit width (such as double)
191 in a large vector of smaller values, since all elements will take on this
192 width. Use `IndirectDouble` when this is a possibility. Note that
193 integers automatically use the smallest width possible, i.e. if you ask
194 to serialize an int64_t whose value is actually small, you will use less
195 bits. Doubles are represented as floats whenever possible losslessly, but
196 this is only possible for few values.
197 Since nested vectors/maps are stored over offsets, they typically don't
198 affect the vector width.
199* To store large arrays of byte data, use a blob. If you'd use a typed
200 vector, the bit width of the size field may make it use more space than
201 expected, and may not be compatible with `memcpy`.
202 Similarly, large arrays of (u)int16_t may be better off stored as a
203 binary blob if their size could exceed 64k elements.
204 Construction and use are otherwise similar to strings.
View as plain text