1bbolt
2=====
3
4[](https://goreportcard.com/report/github.com/etcd-io/bbolt)
5[](https://codecov.io/gh/etcd-io/bbolt)
6[](https://travis-ci.com/etcd-io/bbolt)
7[](https://godoc.org/github.com/etcd-io/bbolt)
8[](https://github.com/etcd-io/bbolt/releases)
9[](https://github.com/etcd-io/bbolt/blob/master/LICENSE)
10
11bbolt is a fork of [Ben Johnson's][gh_ben] [Bolt][bolt] key/value
12store. The purpose of this fork is to provide the Go community with an active
13maintenance and development target for Bolt; the goal is improved reliability
14and stability. bbolt includes bug fixes, performance enhancements, and features
15not found in Bolt while preserving backwards compatibility with the Bolt API.
16
17Bolt is a pure Go key/value store inspired by [Howard Chu's][hyc_symas]
18[LMDB project][lmdb]. The goal of the project is to provide a simple,
19fast, and reliable database for projects that don't require a full database
20server such as Postgres or MySQL.
21
22Since Bolt is meant to be used as such a low-level piece of functionality,
23simplicity is key. The API will be small and only focus on getting values
24and setting values. That's it.
25
26[gh_ben]: https://github.com/benbjohnson
27[bolt]: https://github.com/boltdb/bolt
28[hyc_symas]: https://twitter.com/hyc_symas
29[lmdb]: https://www.symas.com/symas-embedded-database-lmdb
30
31## Project Status
32
33Bolt is stable, the API is fixed, and the file format is fixed. Full unit
34test coverage and randomized black box testing are used to ensure database
35consistency and thread safety. Bolt is currently used in high-load production
36environments serving databases as large as 1TB. Many companies such as
37Shopify and Heroku use Bolt-backed services every day.
38
39## Project versioning
40
41bbolt uses [semantic versioning](http://semver.org).
42API should not change between patch and minor releases.
43New minor versions may add additional features to the API.
44
45## Table of Contents
46
47 - [Getting Started](#getting-started)
48 - [Installing](#installing)
49 - [Opening a database](#opening-a-database)
50 - [Transactions](#transactions)
51 - [Read-write transactions](#read-write-transactions)
52 - [Read-only transactions](#read-only-transactions)
53 - [Batch read-write transactions](#batch-read-write-transactions)
54 - [Managing transactions manually](#managing-transactions-manually)
55 - [Using buckets](#using-buckets)
56 - [Using key/value pairs](#using-keyvalue-pairs)
57 - [Autoincrementing integer for the bucket](#autoincrementing-integer-for-the-bucket)
58 - [Iterating over keys](#iterating-over-keys)
59 - [Prefix scans](#prefix-scans)
60 - [Range scans](#range-scans)
61 - [ForEach()](#foreach)
62 - [Nested buckets](#nested-buckets)
63 - [Database backups](#database-backups)
64 - [Statistics](#statistics)
65 - [Read-Only Mode](#read-only-mode)
66 - [Mobile Use (iOS/Android)](#mobile-use-iosandroid)
67 - [Resources](#resources)
68 - [Comparison with other databases](#comparison-with-other-databases)
69 - [Postgres, MySQL, & other relational databases](#postgres-mysql--other-relational-databases)
70 - [LevelDB, RocksDB](#leveldb-rocksdb)
71 - [LMDB](#lmdb)
72 - [Caveats & Limitations](#caveats--limitations)
73 - [Reading the Source](#reading-the-source)
74 - [Other Projects Using Bolt](#other-projects-using-bolt)
75
76## Getting Started
77
78### Installing
79
80To start using Bolt, install Go and run `go get`:
81```sh
82$ go get go.etcd.io/bbolt@latest
83```
84
85This will retrieve the library and update your `go.mod` and `go.sum` files.
86
87To run the command line utility, execute:
88```sh
89$ go run go.etcd.io/bbolt/cmd/bbolt@latest
90```
91
92Run `go install` to install the `bbolt` command line utility into
93your `$GOBIN` path, which defaults to `$GOPATH/bin` or `$HOME/go/bin` if the
94`GOPATH` environment variable is not set.
95```sh
96$ go install go.etcd.io/bbolt/cmd/bbolt@latest
97```
98
99### Importing bbolt
100
101To use bbolt as an embedded key-value store, import as:
102
103```go
104import bolt "go.etcd.io/bbolt"
105
106db, err := bolt.Open(path, 0666, nil)
107if err != nil {
108 return err
109}
110defer db.Close()
111```
112
113
114### Opening a database
115
116The top-level object in Bolt is a `DB`. It is represented as a single file on
117your disk and represents a consistent snapshot of your data.
118
119To open your database, simply use the `bolt.Open()` function:
120
121```go
122package main
123
124import (
125 "log"
126
127 bolt "go.etcd.io/bbolt"
128)
129
130func main() {
131 // Open the my.db data file in your current directory.
132 // It will be created if it doesn't exist.
133 db, err := bolt.Open("my.db", 0600, nil)
134 if err != nil {
135 log.Fatal(err)
136 }
137 defer db.Close()
138
139 ...
140}
141```
142
143Please note that Bolt obtains a file lock on the data file so multiple processes
144cannot open the same database at the same time. Opening an already open Bolt
145database will cause it to hang until the other process closes it. To prevent
146an indefinite wait you can pass a timeout option to the `Open()` function:
147
148```go
149db, err := bolt.Open("my.db", 0600, &bolt.Options{Timeout: 1 * time.Second})
150```
151
152
153### Transactions
154
155Bolt allows only one read-write transaction at a time but allows as many
156read-only transactions as you want at a time. Each transaction has a consistent
157view of the data as it existed when the transaction started.
158
159Individual transactions and all objects created from them (e.g. buckets, keys)
160are not thread safe. To work with data in multiple goroutines you must start
161a transaction for each one or use locking to ensure only one goroutine accesses
162a transaction at a time. Creating transaction from the `DB` is thread safe.
163
164Transactions should not depend on one another and generally shouldn't be opened
165simultaneously in the same goroutine. This can cause a deadlock as the read-write
166transaction needs to periodically re-map the data file but it cannot do so while
167any read-only transaction is open. Even a nested read-only transaction can cause
168a deadlock, as the child transaction can block the parent transaction from releasing
169its resources.
170
171#### Read-write transactions
172
173To start a read-write transaction, you can use the `DB.Update()` function:
174
175```go
176err := db.Update(func(tx *bolt.Tx) error {
177 ...
178 return nil
179})
180```
181
182Inside the closure, you have a consistent view of the database. You commit the
183transaction by returning `nil` at the end. You can also rollback the transaction
184at any point by returning an error. All database operations are allowed inside
185a read-write transaction.
186
187Always check the return error as it will report any disk failures that can cause
188your transaction to not complete. If you return an error within your closure
189it will be passed through.
190
191
192#### Read-only transactions
193
194To start a read-only transaction, you can use the `DB.View()` function:
195
196```go
197err := db.View(func(tx *bolt.Tx) error {
198 ...
199 return nil
200})
201```
202
203You also get a consistent view of the database within this closure, however,
204no mutating operations are allowed within a read-only transaction. You can only
205retrieve buckets, retrieve values, and copy the database within a read-only
206transaction.
207
208
209#### Batch read-write transactions
210
211Each `DB.Update()` waits for disk to commit the writes. This overhead
212can be minimized by combining multiple updates with the `DB.Batch()`
213function:
214
215```go
216err := db.Batch(func(tx *bolt.Tx) error {
217 ...
218 return nil
219})
220```
221
222Concurrent Batch calls are opportunistically combined into larger
223transactions. Batch is only useful when there are multiple goroutines
224calling it.
225
226The trade-off is that `Batch` can call the given
227function multiple times, if parts of the transaction fail. The
228function must be idempotent and side effects must take effect only
229after a successful return from `DB.Batch()`.
230
231For example: don't display messages from inside the function, instead
232set variables in the enclosing scope:
233
234```go
235var id uint64
236err := db.Batch(func(tx *bolt.Tx) error {
237 // Find last key in bucket, decode as bigendian uint64, increment
238 // by one, encode back to []byte, and add new key.
239 ...
240 id = newValue
241 return nil
242})
243if err != nil {
244 return ...
245}
246fmt.Println("Allocated ID %d", id)
247```
248
249
250#### Managing transactions manually
251
252The `DB.View()` and `DB.Update()` functions are wrappers around the `DB.Begin()`
253function. These helper functions will start the transaction, execute a function,
254and then safely close your transaction if an error is returned. This is the
255recommended way to use Bolt transactions.
256
257However, sometimes you may want to manually start and end your transactions.
258You can use the `DB.Begin()` function directly but **please** be sure to close
259the transaction.
260
261```go
262// Start a writable transaction.
263tx, err := db.Begin(true)
264if err != nil {
265 return err
266}
267defer tx.Rollback()
268
269// Use the transaction...
270_, err := tx.CreateBucket([]byte("MyBucket"))
271if err != nil {
272 return err
273}
274
275// Commit the transaction and check for error.
276if err := tx.Commit(); err != nil {
277 return err
278}
279```
280
281The first argument to `DB.Begin()` is a boolean stating if the transaction
282should be writable.
283
284
285### Using buckets
286
287Buckets are collections of key/value pairs within the database. All keys in a
288bucket must be unique. You can create a bucket using the `Tx.CreateBucket()`
289function:
290
291```go
292db.Update(func(tx *bolt.Tx) error {
293 b, err := tx.CreateBucket([]byte("MyBucket"))
294 if err != nil {
295 return fmt.Errorf("create bucket: %s", err)
296 }
297 return nil
298})
299```
300
301You can also create a bucket only if it doesn't exist by using the
302`Tx.CreateBucketIfNotExists()` function. It's a common pattern to call this
303function for all your top-level buckets after you open your database so you can
304guarantee that they exist for future transactions.
305
306To delete a bucket, simply call the `Tx.DeleteBucket()` function.
307
308
309### Using key/value pairs
310
311To save a key/value pair to a bucket, use the `Bucket.Put()` function:
312
313```go
314db.Update(func(tx *bolt.Tx) error {
315 b := tx.Bucket([]byte("MyBucket"))
316 err := b.Put([]byte("answer"), []byte("42"))
317 return err
318})
319```
320
321This will set the value of the `"answer"` key to `"42"` in the `MyBucket`
322bucket. To retrieve this value, we can use the `Bucket.Get()` function:
323
324```go
325db.View(func(tx *bolt.Tx) error {
326 b := tx.Bucket([]byte("MyBucket"))
327 v := b.Get([]byte("answer"))
328 fmt.Printf("The answer is: %s\n", v)
329 return nil
330})
331```
332
333The `Get()` function does not return an error because its operation is
334guaranteed to work (unless there is some kind of system failure). If the key
335exists then it will return its byte slice value. If it doesn't exist then it
336will return `nil`. It's important to note that you can have a zero-length value
337set to a key which is different than the key not existing.
338
339Use the `Bucket.Delete()` function to delete a key from the bucket.
340
341Please note that values returned from `Get()` are only valid while the
342transaction is open. If you need to use a value outside of the transaction
343then you must use `copy()` to copy it to another byte slice.
344
345
346### Autoincrementing integer for the bucket
347By using the `NextSequence()` function, you can let Bolt determine a sequence
348which can be used as the unique identifier for your key/value pairs. See the
349example below.
350
351```go
352// CreateUser saves u to the store. The new user ID is set on u once the data is persisted.
353func (s *Store) CreateUser(u *User) error {
354 return s.db.Update(func(tx *bolt.Tx) error {
355 // Retrieve the users bucket.
356 // This should be created when the DB is first opened.
357 b := tx.Bucket([]byte("users"))
358
359 // Generate ID for the user.
360 // This returns an error only if the Tx is closed or not writeable.
361 // That can't happen in an Update() call so I ignore the error check.
362 id, _ := b.NextSequence()
363 u.ID = int(id)
364
365 // Marshal user data into bytes.
366 buf, err := json.Marshal(u)
367 if err != nil {
368 return err
369 }
370
371 // Persist bytes to users bucket.
372 return b.Put(itob(u.ID), buf)
373 })
374}
375
376// itob returns an 8-byte big endian representation of v.
377func itob(v int) []byte {
378 b := make([]byte, 8)
379 binary.BigEndian.PutUint64(b, uint64(v))
380 return b
381}
382
383type User struct {
384 ID int
385 ...
386}
387```
388
389### Iterating over keys
390
391Bolt stores its keys in byte-sorted order within a bucket. This makes sequential
392iteration over these keys extremely fast. To iterate over keys we'll use a
393`Cursor`:
394
395```go
396db.View(func(tx *bolt.Tx) error {
397 // Assume bucket exists and has keys
398 b := tx.Bucket([]byte("MyBucket"))
399
400 c := b.Cursor()
401
402 for k, v := c.First(); k != nil; k, v = c.Next() {
403 fmt.Printf("key=%s, value=%s\n", k, v)
404 }
405
406 return nil
407})
408```
409
410The cursor allows you to move to a specific point in the list of keys and move
411forward or backward through the keys one at a time.
412
413The following functions are available on the cursor:
414
415```
416First() Move to the first key.
417Last() Move to the last key.
418Seek() Move to a specific key.
419Next() Move to the next key.
420Prev() Move to the previous key.
421```
422
423Each of those functions has a return signature of `(key []byte, value []byte)`.
424When you have iterated to the end of the cursor then `Next()` will return a
425`nil` key. You must seek to a position using `First()`, `Last()`, or `Seek()`
426before calling `Next()` or `Prev()`. If you do not seek to a position then
427these functions will return a `nil` key.
428
429During iteration, if the key is non-`nil` but the value is `nil`, that means
430the key refers to a bucket rather than a value. Use `Bucket.Bucket()` to
431access the sub-bucket.
432
433
434#### Prefix scans
435
436To iterate over a key prefix, you can combine `Seek()` and `bytes.HasPrefix()`:
437
438```go
439db.View(func(tx *bolt.Tx) error {
440 // Assume bucket exists and has keys
441 c := tx.Bucket([]byte("MyBucket")).Cursor()
442
443 prefix := []byte("1234")
444 for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
445 fmt.Printf("key=%s, value=%s\n", k, v)
446 }
447
448 return nil
449})
450```
451
452#### Range scans
453
454Another common use case is scanning over a range such as a time range. If you
455use a sortable time encoding such as RFC3339 then you can query a specific
456date range like this:
457
458```go
459db.View(func(tx *bolt.Tx) error {
460 // Assume our events bucket exists and has RFC3339 encoded time keys.
461 c := tx.Bucket([]byte("Events")).Cursor()
462
463 // Our time range spans the 90's decade.
464 min := []byte("1990-01-01T00:00:00Z")
465 max := []byte("2000-01-01T00:00:00Z")
466
467 // Iterate over the 90's.
468 for k, v := c.Seek(min); k != nil && bytes.Compare(k, max) <= 0; k, v = c.Next() {
469 fmt.Printf("%s: %s\n", k, v)
470 }
471
472 return nil
473})
474```
475
476Note that, while RFC3339 is sortable, the Golang implementation of RFC3339Nano does not use a fixed number of digits after the decimal point and is therefore not sortable.
477
478
479#### ForEach()
480
481You can also use the function `ForEach()` if you know you'll be iterating over
482all the keys in a bucket:
483
484```go
485db.View(func(tx *bolt.Tx) error {
486 // Assume bucket exists and has keys
487 b := tx.Bucket([]byte("MyBucket"))
488
489 b.ForEach(func(k, v []byte) error {
490 fmt.Printf("key=%s, value=%s\n", k, v)
491 return nil
492 })
493 return nil
494})
495```
496
497Please note that keys and values in `ForEach()` are only valid while
498the transaction is open. If you need to use a key or value outside of
499the transaction, you must use `copy()` to copy it to another byte
500slice.
501
502### Nested buckets
503
504You can also store a bucket in a key to create nested buckets. The API is the
505same as the bucket management API on the `DB` object:
506
507```go
508func (*Bucket) CreateBucket(key []byte) (*Bucket, error)
509func (*Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error)
510func (*Bucket) DeleteBucket(key []byte) error
511```
512
513Say you had a multi-tenant application where the root level bucket was the account bucket. Inside of this bucket was a sequence of accounts which themselves are buckets. And inside the sequence bucket you could have many buckets pertaining to the Account itself (Users, Notes, etc) isolating the information into logical groupings.
514
515```go
516
517// createUser creates a new user in the given account.
518func createUser(accountID int, u *User) error {
519 // Start the transaction.
520 tx, err := db.Begin(true)
521 if err != nil {
522 return err
523 }
524 defer tx.Rollback()
525
526 // Retrieve the root bucket for the account.
527 // Assume this has already been created when the account was set up.
528 root := tx.Bucket([]byte(strconv.FormatUint(accountID, 10)))
529
530 // Setup the users bucket.
531 bkt, err := root.CreateBucketIfNotExists([]byte("USERS"))
532 if err != nil {
533 return err
534 }
535
536 // Generate an ID for the new user.
537 userID, err := bkt.NextSequence()
538 if err != nil {
539 return err
540 }
541 u.ID = userID
542
543 // Marshal and save the encoded user.
544 if buf, err := json.Marshal(u); err != nil {
545 return err
546 } else if err := bkt.Put([]byte(strconv.FormatUint(u.ID, 10)), buf); err != nil {
547 return err
548 }
549
550 // Commit the transaction.
551 if err := tx.Commit(); err != nil {
552 return err
553 }
554
555 return nil
556}
557
558```
559
560
561
562
563### Database backups
564
565Bolt is a single file so it's easy to backup. You can use the `Tx.WriteTo()`
566function to write a consistent view of the database to a writer. If you call
567this from a read-only transaction, it will perform a hot backup and not block
568your other database reads and writes.
569
570By default, it will use a regular file handle which will utilize the operating
571system's page cache. See the [`Tx`](https://godoc.org/go.etcd.io/bbolt#Tx)
572documentation for information about optimizing for larger-than-RAM datasets.
573
574One common use case is to backup over HTTP so you can use tools like `cURL` to
575do database backups:
576
577```go
578func BackupHandleFunc(w http.ResponseWriter, req *http.Request) {
579 err := db.View(func(tx *bolt.Tx) error {
580 w.Header().Set("Content-Type", "application/octet-stream")
581 w.Header().Set("Content-Disposition", `attachment; filename="my.db"`)
582 w.Header().Set("Content-Length", strconv.Itoa(int(tx.Size())))
583 _, err := tx.WriteTo(w)
584 return err
585 })
586 if err != nil {
587 http.Error(w, err.Error(), http.StatusInternalServerError)
588 }
589}
590```
591
592Then you can backup using this command:
593
594```sh
595$ curl http://localhost/backup > my.db
596```
597
598Or you can open your browser to `http://localhost/backup` and it will download
599automatically.
600
601If you want to backup to another file you can use the `Tx.CopyFile()` helper
602function.
603
604
605### Statistics
606
607The database keeps a running count of many of the internal operations it
608performs so you can better understand what's going on. By grabbing a snapshot
609of these stats at two points in time we can see what operations were performed
610in that time range.
611
612For example, we could start a goroutine to log stats every 10 seconds:
613
614```go
615go func() {
616 // Grab the initial stats.
617 prev := db.Stats()
618
619 for {
620 // Wait for 10s.
621 time.Sleep(10 * time.Second)
622
623 // Grab the current stats and diff them.
624 stats := db.Stats()
625 diff := stats.Sub(&prev)
626
627 // Encode stats to JSON and print to STDERR.
628 json.NewEncoder(os.Stderr).Encode(diff)
629
630 // Save stats for the next loop.
631 prev = stats
632 }
633}()
634```
635
636It's also useful to pipe these stats to a service such as statsd for monitoring
637or to provide an HTTP endpoint that will perform a fixed-length sample.
638
639
640### Read-Only Mode
641
642Sometimes it is useful to create a shared, read-only Bolt database. To this,
643set the `Options.ReadOnly` flag when opening your database. Read-only mode
644uses a shared lock to allow multiple processes to read from the database but
645it will block any processes from opening the database in read-write mode.
646
647```go
648db, err := bolt.Open("my.db", 0666, &bolt.Options{ReadOnly: true})
649if err != nil {
650 log.Fatal(err)
651}
652```
653
654### Mobile Use (iOS/Android)
655
656Bolt is able to run on mobile devices by leveraging the binding feature of the
657[gomobile](https://github.com/golang/mobile) tool. Create a struct that will
658contain your database logic and a reference to a `*bolt.DB` with a initializing
659constructor that takes in a filepath where the database file will be stored.
660Neither Android nor iOS require extra permissions or cleanup from using this method.
661
662```go
663func NewBoltDB(filepath string) *BoltDB {
664 db, err := bolt.Open(filepath+"/demo.db", 0600, nil)
665 if err != nil {
666 log.Fatal(err)
667 }
668
669 return &BoltDB{db}
670}
671
672type BoltDB struct {
673 db *bolt.DB
674 ...
675}
676
677func (b *BoltDB) Path() string {
678 return b.db.Path()
679}
680
681func (b *BoltDB) Close() {
682 b.db.Close()
683}
684```
685
686Database logic should be defined as methods on this wrapper struct.
687
688To initialize this struct from the native language (both platforms now sync
689their local storage to the cloud. These snippets disable that functionality for the
690database file):
691
692#### Android
693
694```java
695String path;
696if (android.os.Build.VERSION.SDK_INT >=android.os.Build.VERSION_CODES.LOLLIPOP){
697 path = getNoBackupFilesDir().getAbsolutePath();
698} else{
699 path = getFilesDir().getAbsolutePath();
700}
701Boltmobiledemo.BoltDB boltDB = Boltmobiledemo.NewBoltDB(path)
702```
703
704#### iOS
705
706```objc
707- (void)demo {
708 NSString* path = [NSSearchPathForDirectoriesInDomains(NSLibraryDirectory,
709 NSUserDomainMask,
710 YES) objectAtIndex:0];
711 GoBoltmobiledemoBoltDB * demo = GoBoltmobiledemoNewBoltDB(path);
712 [self addSkipBackupAttributeToItemAtPath:demo.path];
713 //Some DB Logic would go here
714 [demo close];
715}
716
717- (BOOL)addSkipBackupAttributeToItemAtPath:(NSString *) filePathString
718{
719 NSURL* URL= [NSURL fileURLWithPath: filePathString];
720 assert([[NSFileManager defaultManager] fileExistsAtPath: [URL path]]);
721
722 NSError *error = nil;
723 BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES]
724 forKey: NSURLIsExcludedFromBackupKey error: &error];
725 if(!success){
726 NSLog(@"Error excluding %@ from backup %@", [URL lastPathComponent], error);
727 }
728 return success;
729}
730
731```
732
733## Resources
734
735For more information on getting started with Bolt, check out the following articles:
736
737* [Intro to BoltDB: Painless Performant Persistence](http://npf.io/2014/07/intro-to-boltdb-painless-performant-persistence/) by [Nate Finch](https://github.com/natefinch).
738* [Bolt -- an embedded key/value database for Go](https://www.progville.com/go/bolt-embedded-db-golang/) by Progville
739
740
741## Comparison with other databases
742
743### Postgres, MySQL, & other relational databases
744
745Relational databases structure data into rows and are only accessible through
746the use of SQL. This approach provides flexibility in how you store and query
747your data but also incurs overhead in parsing and planning SQL statements. Bolt
748accesses all data by a byte slice key. This makes Bolt fast to read and write
749data by key but provides no built-in support for joining values together.
750
751Most relational databases (with the exception of SQLite) are standalone servers
752that run separately from your application. This gives your systems
753flexibility to connect multiple application servers to a single database
754server but also adds overhead in serializing and transporting data over the
755network. Bolt runs as a library included in your application so all data access
756has to go through your application's process. This brings data closer to your
757application but limits multi-process access to the data.
758
759
760### LevelDB, RocksDB
761
762LevelDB and its derivatives (RocksDB, HyperLevelDB) are similar to Bolt in that
763they are libraries bundled into the application, however, their underlying
764structure is a log-structured merge-tree (LSM tree). An LSM tree optimizes
765random writes by using a write ahead log and multi-tiered, sorted files called
766SSTables. Bolt uses a B+tree internally and only a single file. Both approaches
767have trade-offs.
768
769If you require a high random write throughput (>10,000 w/sec) or you need to use
770spinning disks then LevelDB could be a good choice. If your application is
771read-heavy or does a lot of range scans then Bolt could be a good choice.
772
773One other important consideration is that LevelDB does not have transactions.
774It supports batch writing of key/values pairs and it supports read snapshots
775but it will not give you the ability to do a compare-and-swap operation safely.
776Bolt supports fully serializable ACID transactions.
777
778
779### LMDB
780
781Bolt was originally a port of LMDB so it is architecturally similar. Both use
782a B+tree, have ACID semantics with fully serializable transactions, and support
783lock-free MVCC using a single writer and multiple readers.
784
785The two projects have somewhat diverged. LMDB heavily focuses on raw performance
786while Bolt has focused on simplicity and ease of use. For example, LMDB allows
787several unsafe actions such as direct writes for the sake of performance. Bolt
788opts to disallow actions which can leave the database in a corrupted state. The
789only exception to this in Bolt is `DB.NoSync`.
790
791There are also a few differences in API. LMDB requires a maximum mmap size when
792opening an `mdb_env` whereas Bolt will handle incremental mmap resizing
793automatically. LMDB overloads the getter and setter functions with multiple
794flags whereas Bolt splits these specialized cases into their own functions.
795
796
797## Caveats & Limitations
798
799It's important to pick the right tool for the job and Bolt is no exception.
800Here are a few things to note when evaluating and using Bolt:
801
802* Bolt is good for read intensive workloads. Sequential write performance is
803 also fast but random writes can be slow. You can use `DB.Batch()` or add a
804 write-ahead log to help mitigate this issue.
805
806* Bolt uses a B+tree internally so there can be a lot of random page access.
807 SSDs provide a significant performance boost over spinning disks.
808
809* Try to avoid long running read transactions. Bolt uses copy-on-write so
810 old pages cannot be reclaimed while an old transaction is using them.
811
812* Byte slices returned from Bolt are only valid during a transaction. Once the
813 transaction has been committed or rolled back then the memory they point to
814 can be reused by a new page or can be unmapped from virtual memory and you'll
815 see an `unexpected fault address` panic when accessing it.
816
817* Bolt uses an exclusive write lock on the database file so it cannot be
818 shared by multiple processes.
819
820* Be careful when using `Bucket.FillPercent`. Setting a high fill percent for
821 buckets that have random inserts will cause your database to have very poor
822 page utilization.
823
824* Use larger buckets in general. Smaller buckets causes poor page utilization
825 once they become larger than the page size (typically 4KB).
826
827* Bulk loading a lot of random writes into a new bucket can be slow as the
828 page will not split until the transaction is committed. Randomly inserting
829 more than 100,000 key/value pairs into a single new bucket in a single
830 transaction is not advised.
831
832* Bolt uses a memory-mapped file so the underlying operating system handles the
833 caching of the data. Typically, the OS will cache as much of the file as it
834 can in memory and will release memory as needed to other processes. This means
835 that Bolt can show very high memory usage when working with large databases.
836 However, this is expected and the OS will release memory as needed. Bolt can
837 handle databases much larger than the available physical RAM, provided its
838 memory-map fits in the process virtual address space. It may be problematic
839 on 32-bits systems.
840
841* The data structures in the Bolt database are memory mapped so the data file
842 will be endian specific. This means that you cannot copy a Bolt file from a
843 little endian machine to a big endian machine and have it work. For most
844 users this is not a concern since most modern CPUs are little endian.
845
846* Because of the way pages are laid out on disk, Bolt cannot truncate data files
847 and return free pages back to the disk. Instead, Bolt maintains a free list
848 of unused pages within its data file. These free pages can be reused by later
849 transactions. This works well for many use cases as databases generally tend
850 to grow. However, it's important to note that deleting large chunks of data
851 will not allow you to reclaim that space on disk.
852
853 For more information on page allocation, [see this comment][page-allocation].
854
855[page-allocation]: https://github.com/boltdb/bolt/issues/308#issuecomment-74811638
856
857
858## Reading the Source
859
860Bolt is a relatively small code base (<5KLOC) for an embedded, serializable,
861transactional key/value database so it can be a good starting point for people
862interested in how databases work.
863
864The best places to start are the main entry points into Bolt:
865
866- `Open()` - Initializes the reference to the database. It's responsible for
867 creating the database if it doesn't exist, obtaining an exclusive lock on the
868 file, reading the meta pages, & memory-mapping the file.
869
870- `DB.Begin()` - Starts a read-only or read-write transaction depending on the
871 value of the `writable` argument. This requires briefly obtaining the "meta"
872 lock to keep track of open transactions. Only one read-write transaction can
873 exist at a time so the "rwlock" is acquired during the life of a read-write
874 transaction.
875
876- `Bucket.Put()` - Writes a key/value pair into a bucket. After validating the
877 arguments, a cursor is used to traverse the B+tree to the page and position
878 where they key & value will be written. Once the position is found, the bucket
879 materializes the underlying page and the page's parent pages into memory as
880 "nodes". These nodes are where mutations occur during read-write transactions.
881 These changes get flushed to disk during commit.
882
883- `Bucket.Get()` - Retrieves a key/value pair from a bucket. This uses a cursor
884 to move to the page & position of a key/value pair. During a read-only
885 transaction, the key and value data is returned as a direct reference to the
886 underlying mmap file so there's no allocation overhead. For read-write
887 transactions, this data may reference the mmap file or one of the in-memory
888 node values.
889
890- `Cursor` - This object is simply for traversing the B+tree of on-disk pages
891 or in-memory nodes. It can seek to a specific key, move to the first or last
892 value, or it can move forward or backward. The cursor handles the movement up
893 and down the B+tree transparently to the end user.
894
895- `Tx.Commit()` - Converts the in-memory dirty nodes and the list of free pages
896 into pages to be written to disk. Writing to disk then occurs in two phases.
897 First, the dirty pages are written to disk and an `fsync()` occurs. Second, a
898 new meta page with an incremented transaction ID is written and another
899 `fsync()` occurs. This two phase write ensures that partially written data
900 pages are ignored in the event of a crash since the meta page pointing to them
901 is never written. Partially written meta pages are invalidated because they
902 are written with a checksum.
903
904If you have additional notes that could be helpful for others, please submit
905them via pull request.
906
907
908## Other Projects Using Bolt
909
910Below is a list of public, open source projects that use Bolt:
911
912* [Algernon](https://github.com/xyproto/algernon) - A HTTP/2 web server with built-in support for Lua. Uses BoltDB as the default database backend.
913* [Bazil](https://bazil.org/) - A file system that lets your data reside where it is most convenient for it to reside.
914* [bolter](https://github.com/hasit/bolter) - Command-line app for viewing BoltDB file in your terminal.
915* [boltcli](https://github.com/spacewander/boltcli) - the redis-cli for boltdb with Lua script support.
916* [BoltHold](https://github.com/timshannon/bolthold) - An embeddable NoSQL store for Go types built on BoltDB
917* [BoltStore](https://github.com/yosssi/boltstore) - Session store using Bolt.
918* [Boltdb Boilerplate](https://github.com/bobintornado/boltdb-boilerplate) - Boilerplate wrapper around bolt aiming to make simple calls one-liners.
919* [BoltDbWeb](https://github.com/evnix/boltdbweb) - A web based GUI for BoltDB files.
920* [BoltDB Viewer](https://github.com/zc310/rich_boltdb) - A BoltDB Viewer Can run on Windows、Linux、Android system.
921* [bleve](http://www.blevesearch.com/) - A pure Go search engine similar to ElasticSearch that uses Bolt as the default storage backend.
922* [btcwallet](https://github.com/btcsuite/btcwallet) - A bitcoin wallet.
923* [buckets](https://github.com/joyrexus/buckets) - a bolt wrapper streamlining
924 simple tx and key scans.
925* [cayley](https://github.com/google/cayley) - Cayley is an open-source graph database using Bolt as optional backend.
926* [ChainStore](https://github.com/pressly/chainstore) - Simple key-value interface to a variety of storage engines organized as a chain of operations.
927* [🌰 Chestnut](https://github.com/jrapoport/chestnut) - Chestnut is encrypted storage for Go.
928* [Consul](https://github.com/hashicorp/consul) - Consul is service discovery and configuration made easy. Distributed, highly available, and datacenter-aware.
929* [DVID](https://github.com/janelia-flyem/dvid) - Added Bolt as optional storage engine and testing it against Basho-tuned leveldb.
930* [dcrwallet](https://github.com/decred/dcrwallet) - A wallet for the Decred cryptocurrency.
931* [drive](https://github.com/odeke-em/drive) - drive is an unofficial Google Drive command line client for \*NIX operating systems.
932* [event-shuttle](https://github.com/sclasen/event-shuttle) - A Unix system service to collect and reliably deliver messages to Kafka.
933* [Freehold](http://tshannon.bitbucket.org/freehold/) - An open, secure, and lightweight platform for your files and data.
934* [Go Report Card](https://goreportcard.com/) - Go code quality report cards as a (free and open source) service.
935* [GoWebApp](https://github.com/josephspurrier/gowebapp) - A basic MVC web application in Go using BoltDB.
936* [GoShort](https://github.com/pankajkhairnar/goShort) - GoShort is a URL shortener written in Golang and BoltDB for persistent key/value storage and for routing it's using high performent HTTPRouter.
937* [gopherpit](https://github.com/gopherpit/gopherpit) - A web service to manage Go remote import paths with custom domains
938* [gokv](https://github.com/philippgille/gokv) - Simple key-value store abstraction and implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more)
939* [Gitchain](https://github.com/gitchain/gitchain) - Decentralized, peer-to-peer Git repositories aka "Git meets Bitcoin".
940* [InfluxDB](https://influxdata.com) - Scalable datastore for metrics, events, and real-time analytics.
941* [ipLocator](https://github.com/AndreasBriese/ipLocator) - A fast ip-geo-location-server using bolt with bloom filters.
942* [ipxed](https://github.com/kelseyhightower/ipxed) - Web interface and api for ipxed.
943* [Ironsmith](https://github.com/timshannon/ironsmith) - A simple, script-driven continuous integration (build - > test -> release) tool, with no external dependencies
944* [Kala](https://github.com/ajvb/kala) - Kala is a modern job scheduler optimized to run on a single node. It is persistent, JSON over HTTP API, ISO 8601 duration notation, and dependent jobs.
945* [Key Value Access Language (KVAL)](https://github.com/kval-access-language) - A proposed grammar for key-value datastores offering a bbolt binding.
946* [LedisDB](https://github.com/siddontang/ledisdb) - A high performance NoSQL, using Bolt as optional storage.
947* [lru](https://github.com/crowdriff/lru) - Easy to use Bolt-backed Least-Recently-Used (LRU) read-through cache with chainable remote stores.
948* [mbuckets](https://github.com/abhigupta912/mbuckets) - A Bolt wrapper that allows easy operations on multi level (nested) buckets.
949* [MetricBase](https://github.com/msiebuhr/MetricBase) - Single-binary version of Graphite.
950* [MuLiFS](https://github.com/dankomiocevic/mulifs) - Music Library Filesystem creates a filesystem to organise your music files.
951* [NATS](https://github.com/nats-io/nats-streaming-server) - NATS Streaming uses bbolt for message and metadata storage.
952* [Prometheus Annotation Server](https://github.com/oliver006/prom_annotation_server) - Annotation server for PromDash & Prometheus service monitoring system.
953* [Rain](https://github.com/cenkalti/rain) - BitTorrent client and library.
954* [reef-pi](https://github.com/reef-pi/reef-pi) - reef-pi is an award winning, modular, DIY reef tank controller using easy to learn electronics based on a Raspberry Pi.
955* [Request Baskets](https://github.com/darklynx/request-baskets) - A web service to collect arbitrary HTTP requests and inspect them via REST API or simple web UI, similar to [RequestBin](http://requestb.in/) service
956* [Seaweed File System](https://github.com/chrislusf/seaweedfs) - Highly scalable distributed key~file system with O(1) disk read.
957* [stow](https://github.com/djherbis/stow) - a persistence manager for objects
958 backed by boltdb.
959* [Storm](https://github.com/asdine/storm) - Simple and powerful ORM for BoltDB.
960* [SimpleBolt](https://github.com/xyproto/simplebolt) - A simple way to use BoltDB. Deals mainly with strings.
961* [Skybox Analytics](https://github.com/skybox/skybox) - A standalone funnel analysis tool for web analytics.
962* [Scuttlebutt](https://github.com/benbjohnson/scuttlebutt) - Uses Bolt to store and process all Twitter mentions of GitHub projects.
963* [tentacool](https://github.com/optiflows/tentacool) - REST api server to manage system stuff (IP, DNS, Gateway...) on a linux server.
964* [torrent](https://github.com/anacrolix/torrent) - Full-featured BitTorrent client package and utilities in Go. BoltDB is a storage backend in development.
965* [Wiki](https://github.com/peterhellberg/wiki) - A tiny wiki using Goji, BoltDB and Blackfriday.
966
967If you are using Bolt in a project please send a pull request to add it to the list.
View as plain text