Distributed Tracing with Grafana Tempo

Proliferation of microservices, complicated architectures and the struggle to provide a better visibility and troubleshooting mechanism in this landscape. Without a doubt, distributed tracing has become an indispensable tool in enterprise ecosystem. Zipkin and Jaeger have been used extensively for long as the protocols, ElasticSearch and Cassandra for storage backends and their respective UI components provide an end-to-end distributed tracing toolset.

Below, you can see a reference architecture created with Jaeger and storage backend. In this scenery, DB can be either ElasticSearch, Cassandra or a custom solution as some users opt for.

Jaeger reference architecture (Courtesy of https://www.jaegertracing.io)

However, both storage backends do not provide a perfect solution for trace data persistence. With the enhancing flow of data and desire to keep trace data for longer periods of time, maintenance burden of the storage backends has increased a lot (e.g indexing and sharding on ElasticSearch). This has been reflected in the community such that, as of the writing date of this story, more than one third of the open issues in Jaeger’s GitHub repository is somewhat related with storage backends. Another thing about the conventional backends is that they need block storage to operate which is an expensive resource on public cloud compared to object storage.

More than one third of the issues in Jaeger’s GitHub repository is somewhat related with storage backends.

In this story, I’ll try to go over Tempo which is a distributed tracing storage backend developed by Grafana. On top of being a persistence solution, it offers unique features such as querying capabilities with TraceQL and metric-trace linkage with exemplars/metrics-generator. Let’s get started!

Disclaimer: I’m not affiliated with Grafana Labs in any means. Points below are my subjective opinions. Beware that architectural decisions are no one-size-fits-all. Please DYOR before deciding on a tool and its deployment. Moreover, I, professionally, haven’t used Tempo in production.

Installing Grafana Tempo

Grafana docs offered multiple options for installing Tempo. The one I chose is the distributed deployment with a helm chart on K8S. On its core, it provides the components given below.

The helm chart’s deployment is done with Flux with customized values given in this repository. Please ignore the Flux and other infrastructure components inside, they are out of the scope of this story. The values given below can be used with vanilla helm deployment method as well (helm install my-release grafana/tempo -f values.yaml).

apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
  name: tempo-test
  namespace: tempo-test
spec:
  interval: 60m
  chart:
    spec:
      chart: tempo-distributed
      version: "*"
      sourceRef:
        kind: HelmRepository
        name: tempo-test
      interval: 12h
  # https://github.com/grafana/helm-charts/blob/main/charts/tempo-distributed/values.yaml
  values:
    storage:
      trace:
        backend: s3
        s3:
          access_key: 'grafana-tempo'
          secret_key: 'supersecret'
          bucket: 'tempo-traces'
          endpoint: 'tempo-test-minio:9000'
          insecure: true
        pool:
          max_workers: 400
          queue_depth: 20000
    #MinIO storage configuration
    minio:
      enabled: true
      mode: standalone
      rootUser: grafana-tempo
      rootPassword: supersecret
      buckets:
        # Default Tempo storage bucket
        - name: tempo-traces
          policy: none
          purge: false
      persistence:
        size: 2Gi
    traces:
      otlp:
        grpc:
          enabled: true
        http:
          enabled: true
      zipkin:
        enabled: false
      jaeger:
        thriftHttp:
          enabled: false
      opencensus:
        enabled: false
    metricsGenerator:
    # -- Specifies whether a metrics-generator should be deployed
      enabled: true
      registry:
        external_labels:
          source: tempo
      config: 
        storage:
          remote_write: 
          - url: "http://kube-prometheus-kube-prome-prometheus.monitoring.svc.cluster.local:9090/api/v1/write"
            send_exemplars: true
    global_overrides:
      metrics_generator_processors: [service-graphs, span-metrics]

The resulting topology is as given below. The official docs neatly summarized what each component is for, therefore I’ll not repeat the effort here. Instead, I’ll briefly go over the customized values for each component.

Distibuted Grafana Tempo deployment and its components.

Let’s start with the storage backend. As I stated previously, Grafana devs wisely realized the need for object storage compatibility for trace data persistence. The choice of data format is determined as Apache Parquet. For a more deep dive info about this, find their discussion on the rationale of this decision here. For a production setup, you’d want to consume a more robust object storage backend such as AWS S3. However, for demo purposes, a minio backend does the trick.

storage:
      trace:
        backend: s3
        s3:
          access_key: 'grafana-tempo'
          secret_key: 'supersecret'
          bucket: 'tempo-traces'
          endpoint: 'tempo-test-minio:9000'
          insecure: true
        pool:
          max_workers: 400
          queue_depth: 20000
minio:
      enabled: true
      mode: standalone
      rootUser: grafana-tempo
      rootPassword: supersecret
      buckets:
        - name: tempo-traces
          policy: none
          purge: false
      persistence:
        size: 2Gi

The next important section is supported protocols. Here you can find all the major ones such as Open Telemetry, Jaeger and Zipkin. As it can be seen below, I chose only to support OTLP with http and grpc.

traces:
      otlp:
        grpc:
          enabled: true
        http:
          enabled: true
      zipkin:
        enabled: false
      jaeger:
        thriftHttp:
          enabled: false
      opencensus:
        enabled: false

The final part is my favorite, Grafana is trying to achieve something new in the field of observability such that they thrive to connect all three dimensions (logs, traces, metrics). To be able to do this, they create Prometheus metrics from trace and span metadata. A linking component which they call exemplars provide a bridge between traces and metrics in this case. Metrics-generator is the name of the component and it remote-writes the produced metrics to an Prometheus endpoint. If you don’t get the grip now, no worries, it was the same for me at the first look 🙂. I’ll try to visualize it with a demo in the end.

metricsGenerator:
      enabled: true
      registry:
        external_labels:
          source: tempo
      config: 
        storage:
          remote_write: 
          - url: "http://kube-prometheus-kube-prome-prometheus.monitoring.svc.cluster.local:9090/api/v1/write"
            send_exemplars: true
    global_overrides:
      metrics_generator_processors: [service-graphs, span-metrics]

For demo purposes, Grafana provided a sample application which exports traces to Grafana agent. The instruction for its installation is provided here. Examples below will utilize it.

What is the added value that Tempo brings?

Provision of object storage compatibility, using less disk space with Parquet format etc. are valuable contributions. However, in my opinion, Tempo’s added value lies with these two points.

First, its TraceQL enables querying historical trace data. Comparing to the conventional method of searching a trace within a time frame or looking up with a trace ID, I could name it as a revolution. You can see a very simple example on demo application in the below screencast. Even with this one, the power of ability to execute queries is evident. TraceQL is paving the way to get rid of “looking for a needle in a haystack” approach in trace data examination.

Querying trace on Grafana with TraceQL.

Second, by utilizing metrics generator and exemplars, you can investigate traces with much higher precision. In below example, first, a grafana dashboard is created using a generated metric “traces_spanmetrics_latency_bucket”. Then, its value is filtered to only include samples with “status_code_error”. And finally, by clicking on a sample, you can directly go to the related Tempo trace&span view to investigate the root cause of a problematic request.

Investigating traces through generated metrics and exemplars.

Conclusion

That’s it for this story. I tried to summarize the deployment, main use cases and added values of Grafana Tempo. Hope you find it useful. To sum up, it is well worth a production PoC with these cool features and innovation in the field of distributed tracing. Please add your experiences as comments below if you have its application in an enterprise setting.

PS: If you liked the article, please support it with claps to reach more people. Cheers!

Stackademic 🎓

Thank you for reading until the end. Before you go:

• Please consider clapping and following the writer! 👏
• Follow us X | LinkedIn | YouTube | Discord
• Visit our other platforms: In Plain English | CoFeed | Venture | Cubed
• More content at Stackademic.com