1<!---
2 Licensed to the Apache Software Foundation (ASF) under one
3 or more contributor license agreements. See the NOTICE file
4 distributed with this work for additional information
5 regarding copyright ownership. The ASF licenses this file
6 to you under the Apache License, Version 2.0 (the
7 "License"); you may not use this file except in compliance
8 with the License. You may obtain a copy of the License at
9
10 http://www.apache.org/licenses/LICENSE-2.0
11
12 Unless required by applicable law or agreed to in writing,
13 software distributed under the License is distributed on an
14 "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15 KIND, either express or implied. See the License for the
16 specific language governing permissions and limitations
17 under the License.
18-->
19
20Apache Arrow for Go
21===================
22
23[](https://pkg.go.dev/github.com/apache/arrow/go/v15)
24
25[Apache Arrow][arrow] is a cross-language development platform for in-memory
26data. It specifies a standardized language-independent columnar memory format
27for flat and hierarchical data, organized for efficient analytic operations on
28modern hardware. It also provides computational libraries and zero-copy
29streaming messaging and inter-process communication.
30
31### A note about FlightSQL drivers
32
33Go FlightSQL drivers live in the
34[ADBC repository](https://github.com/apache/arrow-adbc/tree/main/go/adbc).
35In particular, to use the Golang `database/sql` interface:
36```golang
37import (
38 "database/sql"
39 _ "github.com/apache/arrow-adbc/go/adbc/sqldriver/flightsql"
40)
41
42func main() {
43 dsn := "uri=grpc://localhost:12345;username=mickeymouse;password=p@55w0RD"
44 db, err := sql.Open("flightsql", dsn)
45 ...
46}
47```
48
49DSN option keys are expressed as `k=v`, delimited with `;`.
50Some options keys are defined in ADBC, others are defined in the FlightSQL ADBC driver.
51- Arrow ADBC [developer doc](https://arrow.apache.org/adbc/main/driver/go/flight_sql.html#client-options)
52- ADBC [source code](https://github.com/apache/arrow-adbc/blob/3d12fad1bae21029a8ff25604d6e65760c3f65bd/go/adbc/adbc.go#L149-L158)
53- FlightSQL driver option keys [source code](https://github.com/apache/arrow-adbc/blob/3d12fad1bae21029a8ff25604d6e65760c3f65bd/go/adbc/driver/flightsql/flightsql_adbc.go#L70-L81)
54
55Reference Counting
56------------------
57
58The library makes use of reference counting so that it can track when memory
59buffers are no longer used. This allows Arrow to update resource accounting,
60pool memory such and track overall memory usage as objects are created and
61released. Types expose two methods to deal with this pattern. The `Retain`
62method will increase the reference count by 1 and `Release` method will reduce
63the count by 1. Once the reference count of an object is zero, any associated
64object will be freed. `Retain` and `Release` are safe to call from multiple
65goroutines.
66
67### When to call `Retain` / `Release`?
68
69* If you are passed an object and wish to take ownership of it, you must call
70 `Retain`. You must later pair this with a call to `Release` when you no
71 longer need the object. "Taking ownership" typically means you wish to
72 access the object outside the scope of the current function call.
73
74* You own any object you create via functions whose name begins with `New` or
75 `Copy` or when receiving an object over a channel. Therefore you must call
76 `Release` once you no longer need the object.
77
78* If you send an object over a channel, you must call `Retain` before sending
79 it as the receiver is assumed to own the object and will later call `Release`
80 when it no longer needs the object.
81
82Performance
83-----------
84
85The arrow package makes extensive use of [c2goasm][] to leverage LLVM's
86advanced optimizer and generate PLAN9 assembly functions from C/C++ code. The
87arrow package can be compiled without these optimizations using the `noasm`
88build tag. Alternatively, by configuring an environment variable, it is
89possible to dynamically configure which architecture optimizations are used at
90runtime. See the `cpu` package [README](arrow/internal/cpu/README.md) for a
91description of this environment variable.
92
93### Example Usage
94
95The following benchmarks demonstrate summing an array of 8192 values using
96various optimizations.
97
98Disable no architecture optimizations (thus using AVX2):
99
100```sh
101$ INTEL_DISABLE_EXT=NONE go test -bench=8192 -run=. ./math
102goos: darwin
103goarch: amd64
104pkg: github.com/apache/arrow/go/arrow/math
105BenchmarkFloat64Funcs_Sum_8192-8 2000000 687 ns/op 95375.41 MB/s
106BenchmarkInt64Funcs_Sum_8192-8 2000000 719 ns/op 91061.06 MB/s
107BenchmarkUint64Funcs_Sum_8192-8 2000000 691 ns/op 94797.29 MB/s
108PASS
109ok github.com/apache/arrow/go/arrow/math 6.444s
110```
111
112**NOTE:** `NONE` is simply ignored, thus enabling optimizations for AVX2 and SSE4
113
114----
115
116Disable AVX2 architecture optimizations:
117
118```sh
119$ INTEL_DISABLE_EXT=AVX2 go test -bench=8192 -run=. ./math
120goos: darwin
121goarch: amd64
122pkg: github.com/apache/arrow/go/arrow/math
123BenchmarkFloat64Funcs_Sum_8192-8 1000000 1912 ns/op 34263.63 MB/s
124BenchmarkInt64Funcs_Sum_8192-8 1000000 1392 ns/op 47065.57 MB/s
125BenchmarkUint64Funcs_Sum_8192-8 1000000 1405 ns/op 46636.41 MB/s
126PASS
127ok github.com/apache/arrow/go/arrow/math 4.786s
128```
129
130----
131
132Disable ALL architecture optimizations, thus using pure Go implementation:
133
134```sh
135$ INTEL_DISABLE_EXT=ALL go test -bench=8192 -run=. ./math
136goos: darwin
137goarch: amd64
138pkg: github.com/apache/arrow/go/arrow/math
139BenchmarkFloat64Funcs_Sum_8192-8 200000 10285 ns/op 6371.41 MB/s
140BenchmarkInt64Funcs_Sum_8192-8 500000 3892 ns/op 16837.37 MB/s
141BenchmarkUint64Funcs_Sum_8192-8 500000 3929 ns/op 16680.00 MB/s
142PASS
143ok github.com/apache/arrow/go/arrow/math 6.179s
144```
145
146[arrow]: https://arrow.apache.org
147[c2goasm]: https://github.com/minio/c2goasm
View as plain text