Start using OpenTelemetry with Go Fiber

How to instrument a Go web server using Fiber and export traces to Honeycomb.io

Emanuele Fumagalli

Published in

Stackademic

9 min readAug 17, 2023

Introduction

This is a tutorial on how to instrument Go code with OpenTelemetry.
This particular article is a bit more opinionated and it focuses on how to instrument an HTTP webserver written in Go and using Fiber (if you are using Gin there’s an adaptation of this article here), and run HTTP and gRPC requests from there.
It also shows how to export the traces to Honeycomb.io, a popular observability solution that offers a free plan with unlimited time and doesn’t require a credit card for registering.
On the accompanying repo README there’s a one minute video that shows how to set up everything and start seeing the spans.

I tried to collect here the most common questions when starting to instrument a Go app with OpenTelemetry, .
The only signal discussed here are the traces, I’m not going to present the OTeL metrics or logs and I’ll probably write another article related to them.

This is definitely not a guide on OpenTelemetry or Observability but a very practical way to start seeing distributed tracing on a popular observability solution.

Disclaimer: The code is for showing how to do the instrumentation and not an example on how to write clean code. There are no unit tests and no abstractions.

The repo is available on GitHub:

GitHub - emanuelef/go-fiber-honeycomb: Example of instrumenting a Go Fiber app and export to…

Example of instrumenting a Go Fiber app and export to Honeycomb.io - GitHub - emanuelef/go-fiber-honeycomb: Example of…

github.com

To run the example code on your local machine the requirements are:

Go (> 1.19)
Docker (if you want to run Docker compose or build the images)

TL;TR

Just create an account in Honeycomb.io, clone the repo emanuelef/go-fiber-honeycomb (or run from GitHub Codespaces, docker compose works there as well) create the .env files with the API key (manually or running set_token.sh) and from the root folder run:

./set_token.sh

docker compose up

Call the GET endpoints and watch the traces appear in Honeycomb.io.

If you are using Postman there’s a collection in the repo.
Or you can just run the script that calls all of the endpoints:

./run_http_requests.sh

Create an account on Honeycomb.io

Any observability solution could be used with the code in the repo, and the only changes needed to export the traces to a service different than Honeycomb.io will be in the env variables.

Honeycomb.io, as of today, offers a very generous free plan with no need to add a credit card to create the account.
All is needed is an email from where you will get the activation link.

Sign up here, confirm the email and create a new team (any team name is fine and all the steps for the creation of the team are presented automatically after signing up).

The home page will contain instructions on how to do the instrumentation and the API key that will be needed in the code.

You can retrieve the API key even later by going into Account (at the bottom on the left hand side) and Settings.

The Home page will start showing the traces after instrumenting and running the code in the repo.

Instrumentation

Preparation

Get the code in the repo emanuelef/go-fiber-honeycomb, either clone it or get the zip.

Alternatively you can use the GitHub Codespaces (I tried that for this example and it works perfectly).

Run set_token script and paste the API Key from the Honeycomb website.

./set_token.sh
Honeycomb API Key: *********

This will just create the .env files needed for every app (they contain the API keys so better not put them in the repos), should that not work just copy the .env.example files in each folder (you might just want to start with the one in the root folder, /secondary and/grpc-server are needed later) to .env and replace your_key_here with the API key.

At this point we can check if everything is set up correctly and the traces are exported.
In a terminal from the root folder run :

go run main.go

 ┌───────────────────────────────────────────────────┐ 
 │                   Fiber v2.48.0                   │ 
 │               http://127.0.0.1:8080               │ 
 │                                                   │ 
 │ Handlers ............ 16  Processes ........... 1 │ 
 │ Prefork ....... Disabled  PID ............. 84354 │ 
 └───────────────────────────────────────────────────┘

From another terminal:

curl http://127.0.0.1:8080/hello

Go to the Honeycomb.io website and after a short time you should see the exported trace.

Click on the button on the left and you will see the trace with the single span.

Set up trace provider and exporter

First step is to create a trace provider with an exporter.

In this article and in the repo the traces are exported directly to the public endpoint provided by Honeycomb and specified in the OTEL_EXPORTER_OTLP_ENDPOINT (https://api.honeycomb.io:443) env variable.
Using a Collector is probably more likely to happen in production.

In opentelemetry_setup.go the function InitializeGlobalTracerProvider creates a Trace Provider and an Exporter. The Go OpenTelemetry SDK will use the env variables defined in the .env file, the only ones needed for this examples are:

OTEL_SERVICE_NAME=GoFiberExample
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io:443
OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=your_key_here

During the creation of the Trace Provider is possible to add custom resources that will be added to each span as attributes, resources can be added with the OTEL_RESOURCE_ATTRIBUTES env variable as well.

Check the Semantic Conventions to see if the kind of resources you want to add are already defined.

This is where is possible to add the resources in code:

resource, rErr := resource.Merge(
 resource.Default(),
 resource.NewWithAttributes(
  semconv.SchemaURL,
  attribute.String("environment", "test"),
  attribute.String("stack", "001"),
 ),
)

Instrument Fiber webserver with Otelfiber

Fiber can be instrumented with Otelfiber middleware.

app := fiber.New()

app.Use(otelfiber.Middleware())

app.Get("/hello", func(c *fiber.Ctx) error {
 return c.Send(nil)
})

By adding the otelfiber Middleware a span is created automatically for every HTTP request received.

Should you want for some reason to exclude some endpoints from generating spans you can use the otelfiber.WithNext option, if the passed function returns true the otelfiber Middleware will skip the OTeL instrumentation.

 app.Use(otelfiber.Middleware(otelfiber.WithNext(func(c *fiber.Ctx) bool {
  return c.Path() == "/health"
 })))

As showed in the Preparation a call to GET /hello will create the span.

Add child span

Quite likely the work done by the HTTP request will involve additional spans.
In order to create a child span, or a new root span, you need to have a tracer, there will probably be one for the entire app or one for every package.

var tracer trace.Tracer

func init() {
 // this seems to work even if the init happens before setting up the trace provider
 tracer = otel.Tracer("github.com/emanuelef/go-fiber-honeycomb")
}

Then in the code inside a fiber handler

  ctx, childSpan := tracer.Start(c.UserContext(), "custom-child-span")
  time.Sleep(10 * time.Millisecond) // simulate some work
  childSpan.End()

c.UserContext() retrieves the context from Fiber Context with the opentelemetry infos (trace id, span id).

GET /hello-child will generate the following spans:

If the ctx passed to the tracer.Start contains a span then the newly-created span will be a child of that span, otherwise it will be a root span.
tracer.Start also accepts SpanStartOption so it is possible to add attributes or the span kind if needed.

Create a new span

If a new root span is needed (not child of any other) the process is similar to create a child span but passing a context that doesn’t contain a span already.

ctx, span := tracer.Start(context.Background(), "timed-operation")
// Do something
span.End()

In the repo main app code there’s a time tick that creates a new root span every minute.

Instrument outgoing HTTP requests

I’ll present three ways to add the spans for each outgoing HTTP request (and also propagate the trace to the web server called).
In the end they all build the client with the standard http.Client.

Otelhttp package

The simplest way is to use the package otelhttp defined in go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp, this wraps the http requests adding a span around.

The methods currently implemented there are:

Get
Head
Post
PostForm

Inside a Fiber handler the context to pass is c.UserContext(), otherwise the HTTP request span will be a root one.

externalURL := "https://pokeapi.co/api/v2/pokemon/ditto"
resp, err := otelhttp.Get(c.UserContext(), externalURL)
_, _ = io.ReadAll(resp.Body) // This is needed to close the span

A little gotcha here, the io.ReadAll is actually required to close the span, even if the response is not needed.

Otelhttp will also take care of the Transport injecting the span context into the outbound request headers.

In the repo a call to GET /hello-otelhttp will create the following spans, this will require the secondary app to run and is used to demonstrate that the traceparent (a single HTTP header containing trace_id and span_id, among version and trace_flags) is carried in the HTTP header allowing the distributed tracing across multiple apps.

http.Client

To have more control (and use methods not implemented in otelhttp yet, like PATCH) you can use the http.Client directly.

externalURL := "https://pokeapi.co/api/v2/pokemon/ditto"
client := http.Client{
 Transport: otelhttp.NewTransport(
 http.DefaultTransport,
   ),
 }

req, err := http.NewRequestWithContext(c.UserContext(), "GET", externalURL, nil)
if err != nil {
 return err
}

resp, _ := client.Do(req)
_, _ = io.ReadAll(resp.Body)

Passing the context with an existing span to http.NewRequestWithContext (if inside a Fiber handler will be c.UserContext()) will make the outgoing HTTP call to be a child span of the incoming HTTP call served by Fiber.

By adding the otelhttp transport (using otelhttp.NewTransport) to the http.Client the outgoing request will have injected in the HTTP header (traceparent) all the information needed for enabling the distributed tracing.
Alternatively, the traceparent header can be injected manually in the http.Request with:

otel.GetTextMapPropagator().Inject(c.UserContext(), propagation.HeaderCarrier(req.Header))

To make it more interesting in the GET /hello-http-client implemented in the accompaning repo the Transport has an option to show the details of the HTTP request (htttp.getconn, htt.headers…)

  client := http.Client{
   Transport: otelhttp.NewTransport(
    http.DefaultTransport,
    otelhttp.WithClientTrace(func(ctx context.Context) *httptrace.ClientTrace {
     return otelhttptrace.NewClientTrace(ctx)
    })),
  }

Resty

As the last example on how to have outgoing HTTP calls intrumented we will use the Resty popular library.
As of today Resty doesn’t support OTeL instrumentation directly but it is possible by passing the http.Client in the resty.NewWithClient

externalURL := "https://pokeapi.co/api/v2/pokemon/ditto"

client := resty.NewWithClient(
 &http.Client{
  Transport: otelhttp.NewTransport(http.DefaultTransport,
   otelhttp.WithClientTrace(func(ctx context.Context) *httptrace.ClientTrace {
    return otelhttptrace.NewClientTrace(ctx)
   })),
 },
)

restyReq := client.R()
restyReq.SetContext(c.UserContext())

resp, _ := restyReq.Get(externalURL)

Make sure to set the context with the existing span (if you want to create a child span)

As for the previous example if you don’t want to set the Transport, or not having the opportunity in another library to pass an http.Client with the transport, you can inject the traceparent header directly with:

otel.GetTextMapPropagator().Inject(c.UserContext(), propagation.HeaderCarrier(restyReq.Header))

the GET /hello-resty in the repo will produce something like:

Add span attributes

To add additional attributes to an existing span:

span := trace.SpanFromContext(c.UserContext())
span.SetAttributes(attribute.Bool("isTrue", true), attribute.String("test.text", "Ciao"))

Add span events

Events can be added to a span to define something relevant happening in a point in time inside the span, they can have attributes as well.
Events can also be used as a form of log doing something like:

// simple event
span.AddEvent("Done second fake long running task")

// event with attributes to have a log linked to the span
span.AddEvent("log", trace.WithAttributes(
 attribute.String("log.severity", "warning"),
 attribute.String("log.message", "Example log"),
))

Instrument gRPC calls

In the go-fiber-honeycomb there is also an example to see traces when using gRPC.

In order to instrument the gRPC calls.

On the client:

conn, err := grpc.Dial(grpcTarget,
 grpc.WithTransportCredentials(insecure.NewCredentials()),
 grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()))

defer conn.Close()
cli := protos.NewGreeterClient(conn)
r, err := cli.SayHello(c.UserContext(), &protos.HelloRequest{Greeting: "ciao"})

On the server:

grpcServer := grpc.NewServer(grpc.UnaryInterceptor(otelgrpc.UnaryServerInterceptor()))

call GET http://localhost:8080/hello-grpc to see it in action.

Conclusion

I hope this article and the related repo will help in easily starting to instrument your Go code with OpenTelemetry.
We have been using OTeL in production and having the distributed tracing (and all the other amazing features for analyzing all the OTeL signals) has helped understand our platform performance and reduce troubleshooting time.

Thank you for reading until the end. Please consider following the writer and this publication. Visit Stackademic to find out more about how we are democratizing free programming education around the world.