1# Logging, Debugging and Telemetry
2
3**Warning: The OpenCensus project is obsolete and was archived on July 31st,
42023.** This means that any security vulnerabilities that are found will not be
5patched. We recommend that you begin migrating to OpenCensus tracing to
6OpenTelemetry, the successor project. See [OpenCensus](#opencensus) below for
7details.
8
9Logging, debugging and telemetry all capture data that can be used for
10troubleshooting. Logging records specific events and transactions. Debugging
11exposes values for immediate analysis. Telemetry is suitable for production use
12and can serve both logging and monitoring purposes. Telemetry tracing follows
13requests through a system to provide a view of component interactions. Telemetry
14metrics collects data for significant performance indicators, offering insights
15into a system's health.
16
17## Logging and debugging
18
19While working with the Go Client Libraries you may run into some situations
20where you need a deeper level of understanding about what is going on in order
21to solve your problem. Here are some tips and tricks that you can use in these
22cases. *Note* that many of the tips in this section will have a performance
23impact and are therefore not recommended for sustained production use. Use these
24tips locally or in production for a *limited time* to help get a better
25understanding of what is going on.
26
27### HTTP based clients
28
29All of our auto-generated clients have a constructor to create a client that
30uses HTTP/JSON instead of gRPC. Additionally a couple of our hand-written
31clients like Storage and Bigquery are also HTTP based. Here are some tips for
32debugging these clients.
33
34#### Try setting Go's HTTP debug variable
35
36Try setting the following environment variable for verbose Go HTTP logging:
37GODEBUG=http2debug=1. To read more about this feature please see the godoc for
38[net/http](https://pkg.go.dev/net/http).
39
40*WARNING*: Enabling this debug variable will log headers and payloads which may
41contain private information.
42
43#### Add in your own logging with an HTTP middleware
44
45You may want to add in your own logging around HTTP requests. One way to do this
46is to register a custom HTTP client with a logging transport built in. Here is
47an example of how you would do this with the storage client.
48
49*WARNING*: Adding this middleware will log headers and payloads which may
50contain private information.
51
52```go
53package main
54
55import (
56 "context"
57 "fmt"
58 "log"
59 "net/http"
60 "net/http/httputil"
61
62 "cloud.google.com/go/storage"
63 "google.golang.org/api/iterator"
64 "google.golang.org/api/option"
65 htransport "google.golang.org/api/transport/http"
66)
67
68type loggingRoundTripper struct {
69 rt http.RoundTripper
70}
71
72func (d loggingRoundTripper) RoundTrip(r *http.Request) (*http.Response, error) {
73 // Will create a dump of the request and body.
74 dump, err := httputil.DumpRequest(r, true)
75 if err != nil {
76 log.Println("error dumping request")
77 }
78 log.Printf("%s", dump)
79 return d.rt.RoundTrip(r)
80}
81
82func main() {
83 ctx := context.Background()
84
85 // Create a transport with authentication built-in detected with
86 // [ADC](https://google.aip.dev/auth/4110). Note you will have to pass any
87 // required scoped for the client you are using.
88 trans, err := htransport.NewTransport(ctx,
89 http.DefaultTransport,
90 option.WithScopes(storage.ScopeFullControl),
91 )
92 if err != nil {
93 log.Fatal(err)
94 }
95
96 // Embed customized transport into an HTTP client.
97 hc := &http.Client{
98 Transport: loggingRoundTripper{rt: trans},
99 }
100
101 // Supply custom HTTP client for use by the library.
102 client, err := storage.NewClient(ctx, option.WithHTTPClient(hc))
103 if err != nil {
104 log.Fatal(err)
105 }
106 defer client.Close()
107 // Use the client
108}
109```
110
111### gRPC based clients
112
113#### Try setting grpc-go's debug variables
114
115Try setting the following environment variables for grpc-go:
116`GRPC_GO_LOG_VERBOSITY_LEVEL=99` `GRPC_GO_LOG_SEVERITY_LEVEL=info`. These are
117good for diagnosing connection level failures. For more information please see
118[grpc-go's debug documentation](https://pkg.go.dev/google.golang.org/grpc/examples/features/debugging#section-readme).
119
120#### Add in your own logging with a gRPC interceptors
121
122You may want to add in your own logging around gRPC requests. One way to do this
123is to register a custom interceptor that adds logging. Here is
124an example of how you would do this with the secretmanager client. Note this
125example registers a UnaryClientInterceptor but you may want/need to register
126a StreamClientInterceptor instead-of/as-well depending on what kinds of
127RPCs you are calling.
128
129*WARNING*: Adding this interceptor will log metadata and payloads which may
130contain private information.
131
132```go
133package main
134
135import (
136 "context"
137 "log"
138
139 secretmanager "cloud.google.com/go/secretmanager/apiv1"
140 "google.golang.org/api/option"
141 "google.golang.org/grpc"
142 "google.golang.org/grpc/metadata"
143 "google.golang.org/protobuf/encoding/protojson"
144 "google.golang.org/protobuf/reflect/protoreflect"
145)
146
147func loggingUnaryInterceptor() grpc.UnaryClientInterceptor {
148 return func(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
149 err := invoker(ctx, method, req, reply, cc, opts...)
150 log.Printf("Invoked method: %v", method)
151 md, ok := metadata.FromOutgoingContext(ctx)
152 if ok {
153 log.Println("Metadata:")
154 for k, v := range md {
155 log.Printf("Key: %v, Value: %v", k, v)
156 }
157 }
158 reqb, merr := protojson.Marshal(req.(protoreflect.ProtoMessage))
159 if merr == nil {
160 log.Printf("Request: %s", reqb)
161 }
162 return err
163 }
164}
165
166func main() {
167 ctx := context.Background()
168 // Supply custom gRPC interceptor for use by the client.
169 client, err := secretmanager.NewClient(ctx,
170 option.WithGRPCDialOption(grpc.WithUnaryInterceptor(loggingUnaryInterceptor())),
171 )
172 if err != nil {
173 log.Fatal(err)
174 }
175 defer client.Close()
176 // Use the client
177}
178```
179
180## Telemetry
181
182**Warning: The OpenCensus project is obsolete and was archived on July 31st,
1832023.** This means that any security vulnerabilities that are found will not be
184patched. We recommend that you begin migrating to OpenCensus tracing to
185OpenTelemetry, the successor project. See [OpenCensus](#opencensus) below for
186details.
187
188The Google Cloud client libraries for Go still use the OpenCensus project by
189default. However, opt-in support for
190[OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) is now
191available. The transition from OpenCensus to OpenTelemetry is covered in the
192following sections.
193
194### Tracing (experimental)
195
196Apart from spans created by underlying libraries such as gRPC, Google Cloud Go
197generated clients do not create spans. Only the spans created by following
198hand-written clients are in scope for the discussion in this section:
199
200* [cloud.google.com/go/bigquery](https://pkg.go.dev/cloud.google.com/go/bigquery)
201* [cloud.google.com/go/bigtable](https://pkg.go.dev/cloud.google.com/go/bigtable)
202* [cloud.google.com/go/datastore](https://pkg.go.dev/cloud.google.com/go/datastore)
203* [cloud.google.com/go/firestore](https://pkg.go.dev/cloud.google.com/go/firestore)
204* [cloud.google.com/go/spanner](https://pkg.go.dev/cloud.google.com/go/spanner)
205* [cloud.google.com/go/storage](https://pkg.go.dev/cloud.google.com/go/storage)
206
207Currently, the spans created by these clients are for OpenCensus. However,
208OpenCensus users are urged to transition to OpenTelemetry as soon as possible,
209as explained in the next section. OpenTelemetry users can opt-in to experimental
210OpenTelemetry support via an environment variable, as described below.
211
212#### OpenCensus
213
214**Warning: The OpenCensus project is obsolete and was archived on July 31st,
2152023.** This means that any security vulnerabilities that are found will not be
216patched. We recommend that you begin migrating to OpenCensus tracing to
217OpenTelemetry, the successor project.
218
219Using the [OpenTelemetry-Go - OpenCensus Bridge](https://pkg.go.dev/go.opentelemetry.io/otel/bridge/opencensus), you can immediately begin exporting your traces with OpenTelemetry, even while
220dependencies of your application remain instrumented with OpenCensus. If you do
221not use the bridge, you will need to migrate your entire application and all of
222its instrumented dependencies at once. For simple applications, this may be
223possible, but we expect the bridge to be helpful if multiple libraries with
224instrumentation are used.
225
226On May 29, 2024, six months after the
227[release](https://github.com/googleapis/google-cloud-go/releases/tag/v0.111.0)
228of experimental, opt-in support for OpenTelemetry tracing, the default tracing
229support in the clients above will change from OpenCensus to OpenTelemetry, and
230the experimental OpenCensus support will be marked as deprecated. To continue
231using the OpenCensus support after this change, set the environment variable
232`GOOGLE_API_GO_EXPERIMENTAL_TELEMETRY_PLATFORM_TRACING` to the case-insensitive
233value `opencensus` before loading the client library.
234
235```sh
236export GOOGLE_API_GO_EXPERIMENTAL_TELEMETRY_PLATFORM_TRACING=opencensus
237```
238
239On December 2nd, 2024, one year after the release of OpenTelemetry support, the
240experimental and deprecated support for OpenCensus tracing will be removed.
241
242Please note that all Google Cloud Go clients currently provide experimental
243support for the propagation of both OpenCensus and OpenTelemetry trace context
244to their receiving endpoints. The experimental support for OpenCensus trace
245context propagation will be removed at the same time as the experimental
246OpenCensus tracing support.
247
248Please refer to the following resources:
249
250* [Sunsetting OpenCensus](https://opentelemetry.io/blog/2023/sunsetting-opencensus/)
251* [OpenTelemetry-Go - OpenCensus Bridge](https://pkg.go.dev/go.opentelemetry.io/otel/bridge/opencensus)
252
253#### OpenTelemetry
254
255**Warning: OpenTelemetry-Go ensures
256[compatibility](https://github.com/open-telemetry/opentelemetry-go/tree/main?tab=readme-ov-file#compatibility)
257with ONLY the current supported versions of the [Go
258language](https://go.dev/doc/devel/release#policy). This support may be narrower
259than the support that has been offered historically by the Go Client Libraries.
260Ensure that your Go runtime version is supported by the OpenTelemetry-Go
261[compatibility](https://github.com/open-telemetry/opentelemetry-go/tree/main?tab=readme-ov-file#compatibility)
262policy before enabling OpenTelemetry instrumentation.**
263
264To opt-in to experimental OpenTelemetry tracing currently available in the
265clients listed above, set the environment variable
266`GOOGLE_API_GO_EXPERIMENTAL_TELEMETRY_PLATFORM_TRACING` to the case-insensitive
267value `opentelemetry` before loading the client library.
268
269```sh
270export GOOGLE_API_GO_EXPERIMENTAL_TELEMETRY_PLATFORM_TRACING=opentelemetry
271```
272
273On May 29, 2024, the default tracing support will change from OpenCensus to
274OpenTelemetry, and this environment variable will no longer be needed.
275
276Please refer to the following resources:
277
278* [What is OpenTelemetry?](https://opentelemetry.io/docs/what-is-opentelemetry/)
279* [Cloud Trace - Go and OpenTelemetry](https://cloud.google.com/trace/docs/setup/go-ot)
280* On GCE, [use Ops Agent and OpenTelemetry](https://cloud.google.com/trace/docs/otlp)
281
282##### Configuring the OpenTelemetry-Go - OpenCensus Bridge
283
284To configure the OpenCensus bridge with OpenTelemetry and Cloud Trace:
285
286```go
287import (
288 "context"
289 "log"
290 "os"
291 texporter "github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/trace"
292 octrace "go.opencensus.io/trace"
293 "go.opentelemetry.io/contrib/detectors/gcp"
294 "go.opentelemetry.io/otel"
295 "go.opentelemetry.io/otel/bridge/opencensus"
296 "go.opentelemetry.io/otel/sdk/resource"
297 sdktrace "go.opentelemetry.io/otel/sdk/trace"
298 semconv "go.opentelemetry.io/otel/semconv/v1.7.0"
299)
300
301func main() {
302 // Create exporter.
303 ctx := context.Background()
304 projectID := os.Getenv("GOOGLE_CLOUD_PROJECT")
305 exporter, err := texporter.New(texporter.WithProjectID(projectID))
306 if err != nil {
307 log.Fatalf("texporter.New: %v", err)
308 }
309 // Identify your application using resource detection
310 res, err := resource.New(ctx,
311 // Use the GCP resource detector to detect information about the GCP platform
312 resource.WithDetectors(gcp.NewDetector()),
313 // Keep the default detectors
314 resource.WithTelemetrySDK(),
315 // Add your own custom attributes to identify your application
316 resource.WithAttributes(
317 semconv.ServiceNameKey.String("my-application"),
318 ),
319 )
320 if err != nil {
321 log.Fatalf("resource.New: %v", err)
322 }
323 // Create trace provider with the exporter.
324 //
325 // By default it uses AlwaysSample() which samples all traces.
326 // In a production environment or high QPS setup please use
327 // probabilistic sampling.
328 // Example:
329 // tp := sdktrace.NewTracerProvider(sdktrace.WithSampler(sdktrace.TraceIDRatioBased(0.0001)), ...)
330 tp := sdktrace.NewTracerProvider(
331 sdktrace.WithBatcher(exporter),
332 sdktrace.WithResource(res),
333 )
334 defer tp.Shutdown(ctx) // flushes any pending spans, and closes connections.
335 otel.SetTracerProvider(tp)
336 tracer := otel.GetTracerProvider().Tracer("example.com/trace")
337 // Configure the OpenCensus tracer to use the bridge.
338 octrace.DefaultTracer = opencensus.NewTracer(tracer)
339 // Use otel tracer to create spans...
340}
341
342```
343
344
345##### Configuring context propagation
346
347In order to pass options to OpenTelemetry trace context propagation, follow the
348appropriate example for the client's underlying transport.
349
350###### Passing options in HTTP-based clients
351
352```go
353ctx := context.Background()
354trans, err := htransport.NewTransport(ctx,
355 http.DefaultTransport,
356 option.WithScopes(storage.ScopeFullControl),
357)
358if err != nil {
359 log.Fatal(err)
360}
361// An example of passing options to the otelhttp.Transport.
362otelOpts := otelhttp.WithFilter(func(r *http.Request) bool {
363 return r.URL.Path != "/ping"
364})
365hc := &http.Client{
366 Transport: otelhttp.NewTransport(trans, otelOpts),
367}
368client, err := storage.NewClient(ctx, option.WithHTTPClient(hc))
369```
370
371Note that scopes must be set manually in this user-configured solution.
372
373###### Passing options in gRPC-based clients
374
375```go
376projectID := "..."
377ctx := context.Background()
378
379// An example of passing options to grpc.WithStatsHandler.
380otelOpts := otelgrpc.WithMessageEvents(otelgrpc.ReceivedEvents)
381dialOpts := grpc.WithStatsHandler(otelgrpc.NewClientHandler(otelOpts))
382
383ctx := context.Background()
384c, err := datastore.NewClient(ctx, projectID, option.WithGRPCDialOption(dialOpts))
385if err != nil {
386 log.Fatal(err)
387}
388defer c.Close()
389```
390
391
392### Metrics (experimental)
393
394The generated clients do not create metrics. Only the following hand-written
395clients create experimental OpenCensus metrics:
396
397* [cloud.google.com/go/bigquery](https://pkg.go.dev/cloud.google.com/go/bigquery)
398* [cloud.google.com/go/pubsub](https://pkg.go.dev/cloud.google.com/go/pubsub)
399* [cloud.google.com/go/spanner](https://pkg.go.dev/cloud.google.com/go/spanner)
400
401#### OpenTelemetry
402
403The transition of the experimental metrics in the clients above from OpenCensus
404to OpenTelemetry is still TBD.View as plain text