In this blog post, I want to share how I built nunutravelguide.com over the Christmas break. One of my first AI use cases was actually a personal one: travel. Whenever I visit a new place, I love diving into the local culture and history — I want to learn the stories behind the city, not just see the sights.

While visiting Madrid in 2024, I started using Gemini to generate my own audio guides. The museums were packed, and I could never get close enough to read the descriptions. Instead, I took photos of the exhibits and asked Gemini to explain them to me. It worked incredibly well! The best part was the ability to ask follow-up questions and dig deeper into specific historical concepts.

A year later, AI coding tools have advanced so much that I decided to turn this workflow into a dedicated app. As a year-end challenge for 2025, I built a custom audio travel guide to solve this exact problem.

Start with the UI

Since I had zero experience with front-end development or building mobile apps, I knew I needed a plan. My biggest challenge was figuring out how to automate or generate the bulk of the code.

I had heard that Figma recently launched an AI integration called Figma Make. Since I wasn’t familiar with many other tools, I dove right in. Figma Make worked incredibly well — after about 150 iterations and prompts, I finally landed on a design I loved. Best of all, it generated a mobile-responsive site that served as the perfect foundation for my app!

From the website to the mobile app

I had made great progress — within a few hours, I had a working website. But wait… a website? I needed a mobile app! My goal was to create a cross-platform app for both Android and iOS. The next step was to scaffold a React Native project and begin the transformation.

I discovered the react-native-template-obytes, cloned it, set up the Android tooling, and successfully deployed it to an emulator. Success! However, I still had no idea how to actually port the website code to the mobile app, and I definitely didn’t want to do it manually.

I initially tried using a Figma MCP (Model Context Protocol), but the workflow was clunky. I had to take screenshots of my Figma Make designs and paste them into a design kit just so the AI agent could “see” them. Even then, static images couldn’t capture the button logic that was already working on my website.

After some brainstorming, I tried a different shortcut: I downloaded the website’s source code and asked my AI agent to rewrite the entire thing in React Native. Within minutes, I had a functional mobile app. Amazing!

AI, Text-to-Speech (TTS), and Interactive Maps

My next challenge was using Generative AI to create the guide text and feeding it into a Text-to-Speech (TTS) API to generate the audio. While this was relatively simple with the Firebase API, I eventually transitioned to using OpenAI with a Google Cloud Run proxy to protect my API keys.

Setting up the Cloud Run function was incredibly straightforward. I even discovered that it’s built on top of the open-source Knative project, which I had used in the past!

For the mapping component, I chose Mapbox for its impressive 3D map view. To populate the guide, I pulled landmarks and points of interest directly from the Mapbox API and Wikidata.

RevenueCat and In-App Purchases

While the initial prototype took roughly one weekend, I spent several more days refining the app, adding features, and creating production builds. Integrating RevenueCat and configuring in-app purchases took about a full day of work, especially since it was my first time setting them up.

Antigravity and the AI Tech Stack

The app was built using the Antigravity AI IDE and Gemini 3.0/3.1 models. The IDE is incredibly intuitive, particularly its Agent Manager and Plan Mode for complex tasks. I really enjoyed the workflow of editing the plan by adding comments — it felt very similar to a GitHub code review.

Conclusion

Indeed, 2025 has truly transformed the way we work. With LLMs, we can now create production-ready software from scratch in a matter of days — even without prior experience in a specific stack. As a backend developer specialized in Golang and Java, I had zero UI or mobile development experience before this. For this project, I intentionally didn’t write a single line of code myself just to see what was possible.

I have to admit, it wasn’t always smooth sailing. The AI made plenty of mistakes and failed miserably at times. In one instance, it even tried to convince me there was a “known bug” in the iOS audio player, when the issue was actually in my own backend service! I frequently had to revert the code and start over from a previous Git checkout.

Despite those hurdles, it was an incredible learning experience. Testing the boundaries of current AI capabilities has been eye-opening, and I’m genuinely excited to see what we’ll be able to build in the future.

In this article, I would like to introduce OpenTelemetry Operator’s new feature that significantly simplifies instrumenting workloads deployed on Kubernetes.

Instrumentation is the most tedious process when deploying an observability solution. There are multiple approaches how to instrument and application:

• manual/explicit: source code is explicitly instrumented (e.g. by using OpenTelemetry API) or by using pre-built instrumentation libraries that are linked at compile time.
• automatic: application is instrumented without any code modifications and recompilation of the application is not needed.

The automatic instrumentation was for a long time only available as a proprietary technology offered by various APM/observability vendors. OpenTelemetry changed this paradigm and made this technology available in open-source. Users get vendor-neutral instrumentation to avoid vendor lock-in for the most crucial part of the observability integration.

However, deploying auto-instrumentation at scale or validating proof of value can still be a tedious problem. Modern applications are packaged in immutable container images, hence adding instrumentation to already built images requires container rebuild. Rebuilding a large number of application images requires a major investment. Let’s have a look at how this problem can be solved on Kubernetes.

Instrumentation CR in OpenTelemetry operator

The OpenTelemetry Operator 0.38.0 introduced Instrumentation custom resource (CR) which defines the configuration for OpenTelemetry SDK and instrumentation. The instrumentation is enabled when an Instrumentation CR is present in the cluster and a namespace or workload is annotated:

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://otel-collector:4317
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
EOF

At the moment the instrumentation is supported for Java, NodeJS, and Python languages. The instrumentation is enabled when the following annotation is applied to a workload or a namespace.

• instrumentation.opentelemetry.io/inject-java: "true" — for Java
• instrumentation.opentelemetry.io/inject-nodejs: "true" — for NodeJS
• instrumentation.opentelemetry.io/inject-python: "true" — for Python

After the annotation is applied the operator injects OpenTelemetry auto-instrumentation libraries into the application container and configures the instrumentation to export the data to an endpoint defined in the Instrumentation CR.

Java example with Spring Petclinic

Now let’s deploy the Spring Petclinic Java application that will be instrumented and report data to an OpenTelemetry collector. The collector will log spans to the standard output.

Create the following deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-petclinic
spec:
  selector:
    matchLabels:
      app: spring-petclinic
  replicas: 1
  template:
    metadata:
      labels:
        app: spring-petclinic
      annotations:
        sidecar.opentelemetry.io/inject: "true"
        instrumentation.opentelemetry.io/inject-java: "true"
    spec:
      containers:
      - name: app
        image: ghcr.io/pavolloffay/spring-petclinic:latest

and apply the instrumentation annotation:

kubectl patch deployment.apps/spring-petclinic -p '{"spec": {"template": {"metadata": {"annotations": {"instrumentation.opentelemetry.io/inject-java": "true"}}}}}'

After the annotation is applied the spring-petclinic pod will restart and the newly started pod will be instrumented with OpenTelemetry Java auto-instrumentation. The data will be reported in OTLP format to the collector at the address http://otel-collector:4317. The following code snippet deploys an OpenTelemetry collector that logs traces to the standard output.

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:

    exporters:
      logging:

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [logging]
EOF

Now we can port-forward the application HTTP port via kubectl port-forward deployment.apps/spring-petclinic 8080:8080 and explore the application in WEB browser. The spans should be reported to the OpenTelemetry collector. The reported spans can be viewed via kubectl logs deployment.apps/otel-collector.

Auto-instrumentation is rather a complicated piece of software, the implementation depends on the language. In Java, the auto-instrumentation is called Javaagent and it does bytecode manipulation that injects instrumentation points to specific code paths. Then the instrumentation points create telemetry data (e.g. traces) when the code is executed. The auto-instrumentation is always supported for a defined set of frameworks or APIs. This is the list of supported frameworks and application servers for Java.

How is the injection logic implemented?

You might wonder how the injection of auto-instrumentation is implemented and especially how the operator can configure the application to use it. Let me explain it for Java, however, similar concepts are used for other runtimes as well.

The OpenTelemetry operator implements mutating admission webhook that is invoked when the Pod object is created or updated. The webhook modifies the Pod object to inject auto-instrumentation libraries into the application container, it configures OpenTelemetry SDK and runtime, in this case, Java virtual machine (JVM) to use the auto-instrumentation.

The auto-instrumentation library (Javaagent) is injected into the application container via an init container that copies the Javaagent into a volume that is mounted to the application container. The SDK configuration is done by injecting environment variables into the application container.

Now the final step is to configure the JVM to use the Javaagent. This is done by configuring the environment variable JAVA_TOOL_OPTIONS to use the Javaagent.

Conclusion

We have seen that using OpenTelemetry auto-instrumentation on Kubernetes workloads can be simplified with the operator pattern. No change to application code or container images is required. This approach brings huge value for quickly validating telemetry solutions. At the moment only Java, Python, and NodeJS runtimes are supported, however support for other languages is coming. The statically compiled languages like Golang or C++ can still use this feature to dynamically configure SDK. This essentially means the Instrumentation CR defines a control plane for OpenTelemtry SDK and instrumentations.

References

• OpenTelemetry Operator: https://github.com/open-telemetry/opentelemetry-operator#opentelemetry-auto-instrumentation-injection
• Instrumentation CR docs: https://github.com/open-telemetry/opentelemetry-operator/blob/main/docs/api.md#instrumentation
• Supported frameworks in Java auto-instrumentation: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md#supported-libraries-frameworks-application-servers-and-jvms
• Spring Pet Clinic: https://github.com/spring-projects/spring-petclinic
• Kubernetes admission webhook: https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/
• Java tools options: https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars002.html

Using OpenTelemetry auto-instrumentation/agents in Kubernetes was originally published in OpenTelemetry on Medium, where people are continuing the conversation by highlighting and responding to this story.

First let’s have a look at the timeline when these projects were open-sourced:

Note that the timeline and article does not contain all OSS distributed tracing projects and all the content represents my subjective view on this space. There is missing OpenCensus, Expedia Haystack and many more. The first pull request to OpenCensus dates in 2016, but the project was based on existing Google’s instrumentation-java project.

2016: The beginning with Hawkular and Zipkin

For me it all started in June 2016 when I submitted my first pull request to the Hawkular APM project. Many of you probably don’t know the project so let me give you a quick introduction. Hawkular APM is a performance monitoring tool. It is a “lightweight” Java EE application deployed on Wildfly and it heavily uses JMS, Infinispan and Elasticsearch as a storage backend. The major differentiator between Zipkin and other similar tools is that the project focuses more on data aggregation. For instance it provides service metrics visualisations, flow diagrams, ability to compare two services (e.g. canary deployment) and much more.

The project also provided native and OpenTracing based instrumentation libraries. Building instrumentation clients is an enormous effort, there are a lot of languages, frameworks and application runtimes to support. Hence we have started looking at existing solutions, more about this in the Zipkin section.

For me the Hawkular project was quite a short party and the project was sunset in mid 2017 for several reasons. To make a long story short, we had decided to join forces with Jaeger Tracing project which seemed like a better fit for the Kubernetes ecosystem that we were pursuing.

2016: Zipkin

In 2016 Zipkin was the only robust OSS tracing tool with a solid and proven ecosystem. The project was open-sourced in 2012 by Twitter. The original version was written in Scala and later in 2016 rewritten to Java (Spring Boot). The design of the system was inspired by Google’s Dapper paper released in 2010.

The ecosystem was robust, it had a lot of instrumentation libraries, just to name a few: brave, zipkin-js, zipkin-ruby, zipkin-go and many more. The beauty of the Zipkin project was its technical excellence in Java, simple and stable data model, its own B3 context propagation (it stands for Big Brother Bird, the original Zipkin name at Twitter) and great API level documentation. All these aspects made it clear that Zipkin was set to create a rich ecosystem. This was all great and in the Hawkular project we had decided to expose Zipkin ingestion endpoint that would allow existing Zipkin users to try out Hawkular and vice-versa Hawkular users leverage Zipkin instrumentation libraries.

I did a proof of concept — a polyglot application written in Java, Ruby, Javascript and Python to test out instrumentation interoperability. I was very surprised to find out that some instrumentations did not propagate or decode the context properly. This was a huge problem resulting in broken traces. The other issue was that the instrumentation libraries were not using tags coherently. For instance, some libraries were recording full URLs and others only HTTP path with query parameters. The data quality and consistency is very important for an observability platform to work properly.

Ultimately these issues were fixed over time, they only stress how important it is to standardize interoperability APIs like context-propagation and data specification.

2016: OpenTracing

OpenTracing was created at the end of 2015 and it started gaining significant traction at the end of 2016 as it joined Cloud Native Computing Foundation. The promise was great: a vendor neutral instrumentation API and specification that defines data collection. Clearly a huge win on paper because instrumentation is the hardest part when deploying a distributed tracing system. The project also included a contrib organization with instrumentation libraries for many frameworks.

One of the main goals of the project was to incorporate OpenTracing API directly into RPC frameworks. This again sounds great but it didn’t pan out how we wanted. The RPC framework authors are very sensitive to using 3rd party dependencies. Some frameworks (e.g. Vertx) decided to create their own tracing API and only provide an OpenTracing bridge. The other significant issue was that application authors (e.g. Envoy) didn’t know which OpenTracing implementation should be included in the final compiled distribution. Including multiple implementations created unnecessary maintenance overhead and it complicated the final distribution.

Despite many issues, OpenTracing achieved great things and moved OSS distributed tracing forward into a better place. It paved the way for OpenTelemetry to be even more successful. Personally, I would like to summarize the outcome of the project.

Pros:

• specification for data collection
• it helped to promote distributed tracing

Cons:

• API only, no standard implementation. It was a huge confusion for end-users and duplicated effort for vendors to create implementations.
• when switching the implementation need to recompile and redeploy applications
• no reporting format definition
• no context-propagation format

2017: Open-sourced Jaeger

In early 2017 Uber engineering open-sourced Jaeger. The first version was very similar to Zipkin and with a couple of key differences:

• written in Golang
• native support for full OpenTracing API. For instance, Zipkin for a long time didn’t support span logs
• first-class support for OpenTracing implementations in multiple languages. All the clients were hosted directly in the Jaeger organization and properly tested.
• support for endpoint-based sampling and remotely controlled sampling

Over the years the Jaeger project evolved and many features were added. We have always made sure the tool is super easy to use, no matter if directly as a compiled binary or deployed on Kubernetes. When building an OSS ecosystem it’s important to not to say no and rather find a generic solution that works for everybody.

When looking to the future, Jaeger will embrace OpenTelemetry and the V2 version will be based on the OpenTelemetry collector. I would like to also highlight that the Jaeger project expands with new “APM” capabilities like metrics support.

2019: OpenTelemetry

The OpenTelemetry project was announced in 2019 as a merger of OpenTracing and OpenCensus projects. The project promised to combine the best from both of these projects:

• API and official standard implementation (SDK)
• data collection specification
• mandatory support for W3C-Trace Context
• coming support for metrics and logs
• standardized data model (OTLP)
• a local collector that accepts the most popular tracing formats (OTLP, Zipkin, Jaeger) and exports data to vendor systems
• auto instrumentation technology for various languages

The features mentioned above make it clear that OpenTelemetry promises to deliver robust end-to-end data collection solution. The major differentiator over OpenTracing is the standardized data model, official SDKs and collector. All these combined means that users do not have to touch their workloads when changing observability vendors. This is a huge win as it avoids vendor lock-in!

I am pretty confident that OpenTelemetry will be a successful project and it will take OSS distributed tracing to the next level. Note that the project is the second most active CNCF project after Kubernetes. A lot of active members and vendors has its toll as well. The project velocity is impacted and getting things done takes more time.

2020/2021: Hypertrace, Grafana Tempo and SigNoz

The year 2020 and 2021 was a really interesting one. In 2020 the API observability project Hypertrace was open-sourced. The major differentiator to Jaeger is that it makes APIs and databases/backends first-class entities in the system and provides a powerful slice and dice explorer. For instance, a user can only by a few clicks see the performance of database calls or 3rd party service. The project also includes OpenTelemetry compatible agents that extend the upstream project by capturing 100% of traffic (e.g. payloads and headers). The clients or agents have a pluggable filter mechanism that allows users to run arbitrary code to analyze requests, which is pretty cool! The clients are licensed under Apache 2.0 and the server has a hybrid license model with Apache 2.0 and proprietary Traceable Community License 1.0.

In 2021 a similar tool called SigNoz was born. Similarly to Hypertrace, its goal is to be a full observability system with metrics, logs and traces. The platform also provides rich aggregates on filtered traces which is possible due to the OLAP database. The project uses OSS Apache 2 license.

Another important project that was open-sourced in 2020 is Grafana Tempo. Although it is not a full observability system it brings nice innovation to the ecosystem. It’s a local trace data store highly inspired by Prometheus. Its biggest downside is the query capabilities. It only allows returning trace by ID.

Conclusion and peek into the {bright} future

As we have seen open-source tracing projects made a big step forward since 2016. At the beginning Zipkin was the only major OSS tracing system that developers could use. It definitely paved the way and had a great design like a stable data model and context propagation.

Now, developers have a choice to choose from multiple OSS tracing systems, use robust libraries to collect data from their applications by using OpenTelemetry and use standardized context propagation W3C Trace-Context.

So what is next? I wish I had the answer, but I can certainly express what I think. The OpenTelemetry project will be around and it will become more robust and provide well-tested instrumentation libraries to collect various signals including metrics, logs and traces. We will also see more instrumentation for capturing profiling data. The tracing systems will be adding more APM-like capabilities — e.g. better metrics support and profiling. The cloud vendors will integrate with OpenTelemetry and will embed instrumentation capabilities directly in their platforms. Last but not least we will see runtime security projects being integrated into observability.

References

• Hawkular APM: https://hawkular.gitbooks.io/hawkular-apm-user-guide/content/
• Jaeger Tracing: https://www.jaegertracing.io/
• Zipkin: https://zipkin.io/
• OpenTelemetry: https://opentelemetry.io/
• OpenCensus: https://opencensus.io/
• Hypertrace: https://www.hypertrace.org/
• SigNoz: https://signoz.io/
• Apache SkyWalking: https://skywalking.apache.org/
• W3C Trace-Context: https://www.w3.org/TR/trace-context/
• Grafana Tempo: https://github.com/grafana/tempo

In this article, we are going to have a look at using Jaeger clients with W3C Trace-Context propagation format. The standardized context propagation format assures interoperability between different tracing systems and instrumentation libraries. In this regard we are going to explore two use cases. First how to use OpenTelemetry SDKs in Jaeger instrumented environment. This scenario simulates interoperability between different tracing systems but also a migration path from Jaeger clients to OpenTelemetry SDK. In the second use case we are going to configure Jaeger native clients to use W3C Trace-Context.

W3C Trace-Context

Before we deep dive into the main topic let’s have a look at W3C Trace-Context. W3C Trace-Context is a standardized propagation format for trace identifiers and metadata. It contains two headers:

• traceparent — describes the position of the incoming request in its trace graph. It encodes trace ID, parent span ID and set of flags.
• tracestate — extends traceparent with vendor-specific data represented by a set of name/value pairs. This header is optional and can, for instance, encode tenant name.

The standardized inter-process context propagation format solves these problems:

• exchanging trace context between different vendors
• correlation of traces collected by different tracing vendors
• vendor-specific metadata might be dropped by intermediaries, cloud providers and service providers if there is no standard to follow.

Now let’s have a look at OpenTelemetry Java SDK configuration in a mixed environment with Jaeger native clients.

Configure OpenTelemetry SDK with Jaeger context propagation format

In this section, we are going to have a look at the configuration of OpenTelemetry Java SDK in an environment that is instrumented with Jaeger native clients. This use case is important when we want to introduce new services instrumented with OpenTelemetry and it also showcases interoperability between different tracing instrumentations. In a mixed environment, we have to make sure that all instrumentation libraries understand the same propagation protocol.

Since existing services cannot be recompiled and redeployed to use the new context propagation format, the new services have to be configured to understand the new and old propagation formats at the same time. Follows a simple diagram showing three microservices A, B and X. Services A and B are instrumented with Jaeger and X with OpenTelemetry. Only the service X is using both propagation protocols — Jaeger and W3C Trace-Context. This ensures that “legacy” Jaeger services are able to continue the trace started by service X and vice versa.

Now let’s have a look at the code. We are going to configure OpenTelemetry Java SDK to use the default W3C Trace-Context and Jaeger formats simultaneously. First, let’s add OpenTelemetry maven artifacts for SDK and Jaeger context propagation format:

<dependency>
 <groupId>io.opentelemetry</groupId>
 <artifactId>opentelemetry-sdk</artifactId>
 <version>0.4.1</version>
</dependency>
<dependency>
 <groupId>io.opentelemetry</groupId>
 <artifactId>opentelemetry-contrib-trace-propagators</artifactId>
 <version>0.4.1</version>
</dependency>

and the tracer configuration:

import io.opentelemetry.OpenTelemetry;
import io.opentelemetry.trace.Span;
import io.opentelemetry.trace.Tracer;
import io.opentelemetry.contrib.trace.propagation.JaegerPropagator; import io.opentelemetry.trace.propagation.HttpTraceContext;
import io.opentelemetry.context.propagation.ContextPropagators;
import io.opentelemetry.context.propagation
         .DefaultContextPropagators;

ContextPropagators contextPropagators = DefaultContextPropagators.builder()
    .addHttpTextFormat(new JaegerPropagator())
    .addHttpTextFormat(new HttpTraceContext())
    .build();
OpenTelemetry.setPropagators(contextPropagators);

Tracer tracer = OpenTelemetry.getTracerProvider().get("io.example");
Span span = tracer.spanBuilder("operation").startSpan();
...

Now the tracer is configured to use both W3C Trace-Context and Jaeger propagation formats simultaneously for context injection and extraction. The legacy Jaeger instrumentations will recognize the Jaeger propagation format and drop the W3C Trace-Context.

Configure Jaeger clients with W3C Trace-Context

Jaeger native clients can be configured to use custom context propagation formats. Support for W3C Trace-Context is usually provided directly in the core artifacts. It can be configured programmatically or via an environment variable JAEGER_PROPAGATION=w3c.

The following code snippet shows the programmatic configuration that configures both Jaeger and W3C Trace-Context formats at the same time.

import io.jaegertracing.Configuration;
import io.jaegertracing.Configuration.Propagation;

Configuration configuration = Configuration.fromEnv("io.example");
configuration.getCodec()
    .withPropagation(Propagation.W3C)
    .withPropagation(Propagation.JAEGER);
JaegerTracer jaegerTracerWith = configuration.getTracer();

Conclusion

In this article we have shown how OpenTelemetry SDKs can be used in a heterogeneous environment with Jaeger native clients and also how W3C Trace-Context can be configured in Jaeger clients.

References

• W3C Trace-Context: https://www.w3.org/TR/trace-context/
• Jaeger clients documentation: https://www.jaegertracing.io/docs/latest/client-features/
• OpenTelemetry documentation: https://opentelemetry.io/

Jaeger clients and W3C Trace-Context was originally published in JaegerTracing on Medium, where people are continuing the conversation by highlighting and responding to this story.

OpenTelemetry collector is a vendor-agnostic service for receiving, processing and exporting telemetry data. In the Jaeger project we have decided to deprecate the Jaeger collector and migrate its functionality to an implementation based on OpenTelemetry collector. This has several benefits:

• forward compatibility with OpenTelemetry native data model
• tail-based sampling
• attribute processors
• standardized collection pipeline
• less code to maintain

Roadmap

In the long term we would like to base the Jaeger collector, agent and ingester components on OpenTelemetry collector. These new components will be separate distributions with new image and binary names.

Our goal is to provide a smooth migration from existing binaries by supporting legacy Jaeger configuration (flags, env. vars, Jaeger configuration file). However, there will be a couple of breaking changes:

• a different set of metrics exposed by these components
• different health check and metric endpoints
• not all flags will be supported in the new component e.g. --metrics-backed, --collector.queue-size

Follow this milestone on Github to see what is missing for the first stable release.

In the meantime, you can use collector image with thelatest tag jaegertracing/jaeger-opentelemetry-collector:latest.

Configuration & give it a try!

Upstream OpenTelemetry collector is configurable via a configuration file that is provided as a flag at application startup--config-file=config.yaml. If the configuration file is missing the collector will not start because it does not know what components (receivers, processors, exporters) should be used.

Jaeger’s build of the OpenTelemetry collector is opinionated about the configuration and it always uses a set of default components: Jaeger receiver, processors, and exporter. The exporter is one of Jaeger’s supported storage backends: Elasticsearch, Cassandra, Kafka (buffer). The configuration provided in the file is merged with the default configuration.

Jaeger specific components can be configured by the same flags that were exposed by the Jaeger collector e.g. --es.server-urls. The configuration provided in OpenTelemetry config has higher precedence.

The configuration precedence from the lowest to the highest is as follows:

1. Jaeger default values <Jaeger config file< environmental variables < flags (Viper’s default precedence order)
2. OpenTelemetry configuration file

Let’s have a look at an example that configures Jaeger OpenTelemetry collector:

• Enables Elasticsearch backend with URL http://elasticsearch:9200, 3 primary shards (default is 5) and 2 replica shards (default 1)
• Disables batch processor (enabled by default). It’s disabled because it is not specified in the service.pipelines.traces.processors array.
• Enables attribute processor (disabled by default). Note that new components have to be explicitly added to the pipeline.

docker run --rm -it -v ${PWD}:/config \
    -e SPAN_STORAGE_TYPE=elasticsearch \
    jaegertracing/jaeger-opentelemetry-collector \
    --config-file=/config/config.yaml \
    --es.server.urls=http://localhost:9200 \
    --es.num-shards=3

The content of config.yaml:

exporters:
  jaeger_elasticsearch:
    es:
      server-urls: http://elasticsearch:9200
      num-replicas: 2

processors:
  attributes:
    actions:
      - key: user
        action: delete

service:
  pipelines:
    traces: 
      processors: [attributes]

The storage exporter can be configured via the same environmental variable SPAN_STORAGE_TYPE as Jaeger collector or it can be specified in service.pipelines.traces.exporters.

Jaeger Operator

Because the new components can be considered as almost drop-in replacements for existing Jaeger binaries we will be able to directly use them in the Jaeger Operator, by explicitly providing the image name in the CR. The required change in the Jaeger Operator is to expose the OpenTelemetry configuration in the CR. At the moment this is just a design proposal that is being discussed in the issue jaeger-operator/issues/1004.

Following the same configuration as in the previous section, the OpenTelemetry configuration is directly embedded into Jaeger collector node:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  strategy: production
  collector:
    config: |
      exporters:
        jaeger_elasticsearch:
          es:
            server-urls: http://elasticsearch:9200
            num-replicas: 2
      processors:
        attributes:
          actions:
            - key: user
              action: delete
      service:
        pipelines:
          traces:
            processors: [attributes]
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://localhost:9200
        num-shards: 3

Conclusion

We have explained how the Jaeger project integrates with OpenTelemetry collector and what are the key differences between these components. Share your feedback with us and try our new collector based on OpenTelemetry.

References

• Jaeger OpenTelemetry collector source code: https://github.com/jaegertracing/jaeger/tree/master/cmd/opentelemetry-collector
• Jaeger OpenTelemetry collector docker image: https://hub.docker.com/r/jaegertracing/jaeger-opentelemetry-collector
• OpenTelemetry collector: https://github.com/open-telemetry/opentelemetry-collector

Jaeger embraces OpenTelemetry collector was originally published in JaegerTracing on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the previous blog post Data analytics with Jaeger aka traces tell us more! we have introduced our data science initiative and platform. The ultimate goal is to develop new functionality within the Jaeger project based on AI/ML that will provide new insights into our applications. This type of functionality is also referred to as AI operations (AIOps).

Jupyter notebooks

Jupyter notebooks provide a simple user interface for experimenting with data. There are two main use cases that we want to accomplish with the notebooks:

• an interface for data scientists to experiment with tracing data
• on-demand incident investigation

The first use case is self-explanatory. It lowers the bar for non-infrastructure people to connect to the Jaeger server to consume and analyze the data. The second use case is more tricky. Imagine an incident that requires us to analyze a trace or set of traces with hundreds or thousands of spans. Such an analysis might not be feasible in the user interface. Instead, we could write code to verify our hypothesis. For this purpose we have developed Trace DSL based on graph query language Gremlin to simplify filtering and feature extraction of tracing data. Hence a Jaeger user would be able to spin up a Jupyter notebook on demand with the Trace DSL and write a query and analysis.

Jupyter notebook example with Jaeger

In this section we are going to deploy Jupyter notebook with Jaeger Trace DSL and write a simple query against the Jaeger server.

Let’s deploy Jaeger, HotROD example and Jupyter notebook with Trace DSL:

docker run --rm -it -p 16686:16686 --name=jaeger jaegertracing/all-in-one:1.17

docker run --rm -it -p 8080:8080 --link=jaeger -e JAEGER_ENDPOINT=http://jaeger:14268/api/traces jaegertracing/example-hotrod:1.17

docker run --rm -it -p 8888:8888 -p 4041:4040 -p 9001:9001 --link=jaeger -e JUPYTER_ENABLE_LAB=yes quay.io/jaegertracing/jaeger-analytics-java:latest

Add -v ${PWD}:/home/jovyan/work to Jupyter notebook if you want to open the notebooks from your current directory. The notebooks are hosted in jaegertracing/jaeger-analytics-java repository.

Now open Jaeger UI at http://localhost:16686, HotROD example at http://localhost:8080 and Jupyter notebook at http://localhost:8888/lab. The token for the Jupyter lab is written in the Jupyter console logs.

For the analysis we have to generate some data, so in the HotROD UI click on the blue boxes to order a car that generates nice traces. To verify that the trace reached Jaeger open Jaeger UI and search for traces from thefrontend service. The trace should look like this:

Once we know that the data is stored in Jaeger we can move to Jupyter notebook and load the trace there. Jaeger notebooks are stored in thejupyter directory. This directory can either be opened from the project root directory or from work in case the notebooks from host filesystem are injected into the docker container.

Before running the analysis we have to load dependencies into the notebook’s classpath. Just click on the dependencies cell to make it active and then on the play icon in the top navigation menu.

Before running the code we have to update the variable traceIdStr to point into one of the traces we have generated earlier in the HotROD app.

The results are written below the code cell. In this case the trace has the height 3 and there is one calculated network latency between frontend and server service of 0.00102 ms . The latency is small because all services are run as part of the same process and there is no real network overhead.

The last code cell shows the direct use of Trace DSL with Apache Gremlin. Gremlin is a graph traversal language and class TraceTraversalSource.class it extends and adds methods for trace filtering and feature extraction. For instance, it adds a method like hasName(String name) to filter spans by operation name. In our example the query verifies whether two spans with given operation names are directly or indirectly connected, or in other words one is a descendant of the other.

Conclusion

We have seen how easy it is to deploy Jupyter notebook and write a simple query against Jaeger server using gRPC generated stubs. This feature provides a powerful interface to write a custom query and hypothesis analysis against traces retrieved from the Jaeger server. The example can easily be extended to collect a stream of traces from Kafka and experiment on the live data.

Any feedback is welcome! Get in touch with us on our Jaeger Gitter channel or simply open an issue and share your feedback or ideas.

References

• Jaeger Java analytics: https://github.com/jaegertracing/jaeger-analytics-java
• Data analytics with Jaeger blog post: https://medium.com/jaegertracing/data-analytics-with-jaeger-aka-traces-tell-us-more-973669e6f848
• Apache Gremlin documentation: http://tinkerpop.apache.org/docs/current/reference/
• Jaeger HotROD example application demo https://medium.com/opentracing/take-opentracing-for-a-hotrod-ride-f6e3141f7941

Jaeger data analytics with Jupyter notebooks was originally published in JaegerTracing on Medium, where people are continuing the conversation by highlighting and responding to this story.

So first things first, GraalVM is a polyglot virtual machine. It can run any JVM based language but also other mainstream languages like NodeJS, Python and Ruby… It also supports LLVM languages like C and C++. The sweet spot is that the languages can be combined together in a single application. Another very interesting feature is that GraalVM can produce a native executable for JVM languages. This feature is used by modern cloud-native Java runtimes like Quarkus and Micronaut.

Styles of instrumentation

Before we deep dive into the demo application let’s have a look at different instrumentation techniques. There are several instrumentation styles that can be applied to code fragments or the whole application. At a high level we can talk about black-box and white-box instrumentation. The black-box technique does not require any modifications to the application. The monitoring system, in this case, uses only events produced by the application. These events can be logs or any data that the application produces.

In this article we are going to focus on the white-box technique. This technique assumes that the code of the application is known and can be inspected. There are several methods of how the instrumentation points can be added to the application:

• Non-intrusive agent-like instrumentation: in this case the instrumentation is added to the application dynamically. In some languages (Python, Ruby, JavaScript) this technique is called monkey-patching where modules or classes are dynamically replaced at runtime. In other languages less dynamic techniques are allowed, for instance -javaagent in Java.
• Explicit instrumentation: this instrumentation is applied directly to the code. However it does not mean that the application developer has to instrument all the code. The instrumentation code can be embedded directly into (RPC) frameworks or external integration. In some languages and runtime frameworks it is easier to initialize the instrumentation like in other languages. For instance in Java it’s enough just to drop an instrumentation JAR into the classpath, whereas in statically linked languages like Golang the initialization has to be done explicitly.

The application

The application is a simple NodeJS server with one handler. The full demo code with instructions can be found in pavolloffay/graalvm-t-trace. Here is the server.js file:

const port = 3000;

const http = require("http");
const srv = http.createServer((req, res) => {
  console.log(`server: obtained request ${res.id}`);
  setTimeout(() => {
    res.write(`OK# ${res.id}`);
    console.log(`server: replied to request ${res.id}`);
    res.end();
  }, 5);
});
srv.listen(port);

As you can see there is no tracing related in the application. The application prints request ids at the beginning and at the end of the request. The res.id is set in the agent script — this is just for the demo purpose to demonstrate that the agent script can pass objects to the main application.

Now let’s have a look at the instrumentation script jaegernode.js. There are two functions: initializeJaeger and initializeAgent. The first function creates Jaeger tracer instance with HTTP sender to the collector port 14268 and calls the second one at the end:

let initializeJaeger = function (require) {
    let http = require("http");    console.log(`${typeof     
    http.createServer} http.createServer is available to the agent`);     
    var initTracer = require('jaeger-client').initTracer;       
    console.log('server: Jaeger tracer obtained');

// See schema https://github.com/jaegertracing/jaeger-client-node/blob/master/src/configuration.js#L37
    var config = {
        serviceName: 't-trace-demo',
        reporter: {
            // Provide the traces endpoint; this forces the client to connect directly to the Collector and send
            // spans over HTTP
            collectorEndpoint: 'http://localhost:14268/api/traces',
            // logSpans: true
        },
        sampler: {
            type : 'const',
            param : 1
        }
    };
    var options = {
        tags: {
            't-trace-demo.version': '1.1.2',
        },
        //  metrics: metrics,
        logger: console,
        sampler: {
            type : 'const',
            param : 1
        }
    };

    var tracer = initTracer(config, options);
    initializeAgent(tracer);
};

let waitForRequire = function (event) {
    if (typeof process === 'object' && process.mainModule &&  
                 process.mainModule.require) {
        agent.off('source', waitForRequire); 
        initializeJaeger(  
             process.mainModule.require.bind(process.mainModule));
    }
};

agent.on('source', waitForRequire, { roots: true });

The second function is initializeAgent which is instrumenting the application code:

let initializeAgent = function(tracer) {
    agent.on('enter', function(ctx, frame) {
        const args = frame.args;
        if ('request' !== frame.type || args.length !== 2 || typeof args[0] !== 'object' || typeof args[1] !== 'object') {
            return;
        }
        const req = args[0];
        const res = args[1];
        const span = tracer.startSpan(req.method);
        span.setTag("span.kind", "server");
        span.setTag("http.url", req.url);
        span.setTag("http.method", req.method);
        res.id = span.context().spanIdStr;
        res.span = span;
        console.log(`agent: handling #${res.id} request for ${req.url}`);
    }, {
        roots: true,
        rootNameFilter: name => name === 'emit',
        sourceFilter: src => src.name === 'events.js'
    });

    agent.on('return', function(ctx, frame) {
        var res = frame['this'];
        if (res.span) {
            res.span.setTag("http.status_code", res.statusCode);
            if (res.statusCode >= 400) {
                res.span.setTag("error", "true");
            }
            res.span.finish();
            console.log(`agent: finished #${res.id} request`);
        } else {
            // OK, caused for example by Tracer itself connecting to Jaeger server
        }
    }, {
        roots: true,
        rootNameFilter: name => name === 'end',
        sourceFilter: src => src.name === '_http_outgoing.js'
    });
    console.log('agent: ready');
};

The instrumentation is done via agent.on('enter', fn) and agent.on('return', fn). The first instrumentation point is invoked when any function from the application is invoked and the second when a function returns execution the caller. The agent.on function has access to the frame variables and also to the method arguments. The arguments are used in a condition to check whether the function is HTTP handler. You can also note that the span object is injected into the response.

Now let’s run the application and Jaeger server:

docker run --rm -it --net=host jaegertracing/all-in-one:1.16.0
$GRAALVM_HOME/bin/npm install jaeger-client@3.17.2
$GRAALVM_HOME/bin/node --experimental-options --js.print=true  --agentscript=jaegernode.js server.js
curl http://localhost:3000

Conclusion

We have seen a NodeJS hello-world example of T-Trace with Jaeger and OpenTracing. The demo shows how agent-like instrumentation can be applied to the NodeJS application without monkey-patching.

There are many improvements that can be made to the code. For instance we could extend it by not instrumenting the code in the agent script but rather reusing OpenTracing instrumentation for NodeJS and just install it in the agent script. Another possible improvement we could do is to support dynamic loading of the script. In this case the application will be started with the agent script that exposes REST API for loading and disabling scripts. This is a very powerful feature that enables changing the granularity of the instrumentation on the fly without recompiling and redeploying the application.

T-Trace is also able to mix the instrumentation language with the main language. For example, the agent script can be written in JavaScript to trace Ruby or C++ application.

References

• Demo code: https://github.com/pavolloffay/graalvm-t-trace
• T-Trace: https://github.com/oracle/graal/blob/master/tools/docs/T-Trace.md
• T-Trace, polyglot tracing: https://github.com/oracle/graal/blob/master/tools/docs/T-Trace-Manual.md#trully-polyglot---t-trace-any-language
• T-Trace embedded tracing: https://github.com/oracle/graal/blob/master/tools/docs/T-Trace-Embedding.md

T-Trace: agent-like instrumentation for GraalVM was originally published in JaegerTracing on Medium, where people are continuing the conversation by highlighting and responding to this story.

In Jaeger we want to address this problem and provide a platform for data scientists and operators to verify a hypothesis and ultimately answer the question of what caused an outage or why the system behaves in a certain way. Apart from on-demand incident investigation, the goal is to also derive insights from all traces Jaeger collects as part of the standard Jaeger deployment. So let’s have a look at some use cases and then at the technical details.

Metrics

Before we deep dive into the platform overview I would like to talk about what metrics can be derived from traces. Certainly different post-processing jobs will produce results of different structures, but for now let’s only have a look at metrics as these can be directly consumed by metric systems so the solution does not require any custom storage and presentation layer.

Network latency

A trace contains end-to-end information about the request/transaction. By doing some minimal calculations we are able to derive network latency between client and server calls. The results can be exported as a histogram and partitioned by client and server service labels. Example results in Prometheus metrics:

network_latency_seconds_bucket{client="frontend",server="driver",le="0.005",} 8.0
network_latency_seconds_bucket{client="frontend",server="driver",le="0.01",} 9.0 network_latency_seconds_bucket{client="frontend",server="driver",le="0.025",} 12.0 network_latency_seconds_bucket{client="frontend",server="driver",le="0.05",} 15.0 network_latency_seconds_bucket{client="frontend",server="driver",le="0.075",} 20.0

A further improvement would be to use host as a label, as a service may be load balanced across multiple hosts, with different network characteristic.

Another variation of this metric might be a duration between a consumer and a producer in messaging systems.

Trace and service depth

Sometimes it is important to validate the structure of the call graph in our microservice architecture. For instance, we might be interested to know what is the maximal depth of our call graph which can be used to find outlier traces with unusual deep structures.

Service depth is the maximum network hops between root span and leaf spans.

Service dependencies

Another metric that falls into a trace structure category are:

• a number of dependencies.
• a number of dependent services of a service.

Trace quality

There is no doubt that proper tracing instrumentation is the most difficult part of rolling out tracing infrastructure in an organization. Therefore it is fundamental to measure how well applications are instrumented to evaluate tracing adoption. These metrics can be used:

• jaeger_client_version an appropriate Jaeger client version is used in an application.
• server_span and client_span — a trace contains right combination of server and client spans. For instance, if there is a client span there should be an appropriate server span. The same metric could be used for messaging spans — producer and consumer, to find out whether both ends of the channel are instrumented.
• unique_span_id — spans within a trace contain unique span id.

trace_quality_minimum_client_version_total{pass="false",service="route",version="Go-2.21.1",} 100.0 trace_quality_minimum_client_version_total{pass="false",service="redis",version="Go-2.21.1",} 119.0 trace_quality_minimum_client_version_total{pass="false",service="customer",version="Go-2.21.1",} 10.0 trace_quality_minimum_client_version_total{pass="false",service="driver",version="Go-2.21.1",} 10.0 trace_quality_server_tag_total{pass="false",service="mysql",} 10.0 trace_quality_server_tag_total{pass="true",service="customer",} 8.0 trace_quality_client_tag_total{pass="true",service="frontend",} 67.0 trace_quality_client_tag_total{pass="false",service="driver-client",} 2.0

These quality metrics were initially open-sourced in Jaeger’s Flink repository by Uber. The solution calculates metrics and stores the results in a Cassandra table. The results are ultimately just counters, therefore we can export them to any metric system. However, the former solution also provides links to traces that fail a certain quality indicator. Which was proven to be very useful. To fully support this feature we have to wait until OSS metric systems support trace exemplars. However, even trace exemplars might not be sufficient if hundreds of traces will have to be linked to a single metric data point.

Trace DSL

Okay, we have talked about some use cases and we have defined our goal to provide a platform where these use cases could be easily implemented and executed as part of the standard Jaeger deployment. To make it dead easy to write aggregation jobs, filtering, navigation within a trace and feature extraction we should also provide an API and a library to process a trace or set of traces.

A trace is a directed acyclic graph, therefore it makes a lot of sense to represent it as a graph. We have decided to reuse an existing graph API and query/traversal language Gremlin from the Apache TinkerPop project. The project also provides an in-memory database TinkerGraph which we use once we load traces from the storage (Kafka, Jaeger-query).

Let’s have a look at some examples of our Trace DSL. The first example answers the question “Is there a client span with duration 120 microseconds?”

TraceTraversal<Vertex, Vertex> traversal =      
  graph.traversal(TraceTraversalSource.class)
    .hasTag(Tags.SPAN_KIND.getKey(), Tags.SPAN_KIND_CLIENT)
    .duration(P.gt(120));

As you might have noticed the query uses two methods from Trace DSL: hasTag and duration . These methods were added to Gremlin core API via TraceTraversalSource.class. The result is a list of vertices/spans that satisfy this query. From the vertices/spans we can navigate to other parts of the trace.

The second example answers the question: “Is span with a name child2 a direct descendant of root span?”.

TraceTraversal<Vertex, Vertex> traversal = 
  graph.traversal(TraceTraversalSource.class)
    .hasName("root")
    .repeat(__.out())
    .until(__.hasName("child2"));

This query is more complicated and it uses core Gremlin API calls like repeat(__.out()) to traverse to outgoing edges. It also makes sense to provide this query as an extension to Gremlin API if it becomes commonly used. I acknowledge that writing Gremlin queries is not trivial, hence feature full Trace DSL should simplify the work.

Architecture

The following diagram depicts Jaeger streaming architecture with data analytics integration.

There are two parts of the analytics platform: Spark streaming for all incoming data and on-demand Jupyter notebook.

The Spark streaming connects to the same Kafka topic used by the Jaeger collection pipeline. It consumes and analyzes the data, exposing the results as Prometheus metrics or writing them to the storage.

The second integration path is done via Jupyter notebooks. The notebook can connect to Kafka to get the stream of data or get the historical data from Jaeger query. Then do the analysis and show results in the notebook or publish to Prometheus or the storage.

Conclusion

We have talked about the reasons and use cases for data analytics platform for Jaeger. Our goal is to build a community for this effort to help develop the models but also validate them on real deployments. Ultimately, analytics features should provide us more insights into our application behavior and a power-user interface for incident analysis. The project is still in early development and we would like to hear your feedback! Do not hesitate to reach out to us directly or create a feature request in the repository.

References

• Jaeger Java analytics: https://github.com/jaegertracing/jaeger-analytics-java
• Jaeger analytics flink: https://github.com/jaegertracing/jaeger-analytics-flink
• Apache TinkerPop Gremlin: https://tinkerpop.apache.org/gremlin.html
• Apache TinkerGraph docs: http://tinkerpop.apache.org/docs/current/reference/#tinkergraph-gremlin

Data analytics with Jaeger aka traces tell us more! was originally published in JaegerTracing on Medium, where people are continuing the conversation by highlighting and responding to this story.

To get started with Jaeger operator refer to our operator blog post or documentation.

What is the operator?

Kubernetes operator is a software component that simplifies deployment and operations of an application running in a Kubernetes cluster — in our case Jaeger and Elasticsearch. Think of it as software that knows how to deploy, monitor, upgrade and scale an application.

Behind the scenes, the operator uses Kubernetes custom resource (CR) to expose application configuration. The operator watches the resource and accordingly it makes changes to the application configuration or deployment based on the options provided in the CR.

To make life even easier with operators there is one additional component — Operator Lifecycle Manager (OLM). This component helps users install, update, and manage the lifecycle of all operators and their associated services running across their clusters.

To learn more about operators refer to the Kubernetes documentation, OpenShift or coreos documentation.

Elasticsearch operator on Kubernetes

In this section, we are going to use elastic.co Elasticsearch operator as backend storage for Jaeger. Refer to the Elasticsearch documentation on how to deploy the Elasticsearch operator. Jaeger configuration in this section can also be used for any external Elasticsearch cluster.

Create an Elasticsearch CR that will be used by the operator to deploy a single node Elasticsearch instance:

cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.5.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
EOF

Before deploying Jaeger we have to get the password from Elasticsearch secret and create a new secret for Jaeger with username and password. Alternatively, we could specify these two options directly in storage options in Jaeger CR.

PASSWORD=$(kubectl get secret quickstart-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode)

kubectl create secret generic jaeger-secret --from-literal=ES_PASSWORD=${PASSWORD} --from-literal=ES_USERNAME=elastic

Now the secret jaeger-secretwith username and password for Elasticsearch has been created and we can specify it in spec.storage.secretName within the Jaeger CR. We have to also specify Elasticsearch URL and injected CA certificate as a volume from quickstart-es-http-certs-public secret.

cat <<EOF | kubectl apply -f -
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: https://quickstart-es-http:9200
        tls:
          ca: /es/certificates/ca.crt
        num-shards: 1
        num-replicas: 0
    secretName: jaeger-secret
  volumeMounts:
    - name: certificates
      mountPath: /es/certificates/
      readOnly: true
  volumes:
    - name: certificates
      secret:
        secretName: quickstart-es-http-certs-public
EOF

The above configuration works since Jaeger 1.16.0 (#PR1918). Older Jaeger versions should deploy, however the cron job to clean old data will fail on TLS related errors. Note that spark-dependencies also does not support TLS (#ISSUE83).

The successful deployment can be verified by:

kubectl get deployment                                                                                                                                                                                                                                                                               
NAME                    READY     UP-TO-DATE   AVAILABLE   AGE
simple-prod-collector   1/1       1            1           4m12s
simple-prod-query       1/1       1            1           4m12s

Now let’s have a look at deploying Jaeger on OpenShift using self-provisioning of Elasticsearch cluster.

Openshift 4

Within an OpenShift environment Jaeger can make use of cluster logging and automatically provision an Elasticsearch cluster based on the configuration in Jaeger CR. OpenShift 4 also comes by default with OLM and user interface, therefore Jaeger provisioning can be done directly in the user interface. We will show how to deploy via OLM’s user interface. The same could be accomplished by directly deploying operators without OLM, however we will use benefits of “click on a button” updates and other management features OLM provides.

First we have to deploy Jaeger and Elasticsearch operator via OLM. Open OpenShift UI and navigate to Operators and OperatorHub menu and search for Elasticsearch and Jaeger.

Then we can go to Installed Operators menu to verify operators were deployed. There we click on Jaeger and create an instance.

We can also create an instance in the command line:

cat <<EOF | kubectl apply -f -
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  strategy: production
  storage:
    type: elasticsearch
    elasticsearch:
      nodeCount: 1
      resources:
        requests:
          cpu: 200m
          memory: 1Gi
        limits:
          memory: 1Gi
EOF

This CR is significantly simpler than the one for external Elasticsearch. First we only have to create a single CR for Jaeger and storage, additionally, we no longer have to inject certificates, user credentials or the number of primary and replica shards. All the configuration is done automatically by the operator. For instance, the operator autonomously chooses data replication factor based on the nodes in the cluster. Behind the scenes, Jaeger operator creates Elasticsearch CR (for cluster logging Elasticsearch) which triggers Elasticsearch deployment for the given configuration.

Persistent storage can be specified in.elasticsearch.storage by providing storage class name and size. Based on this configuration operator provisions persistent volume claim (PVC) and persistent volume (PV). The persistent volume is bound to the Elasticsearch instance by its name and it should be reused if the instance with the same name is re-created.

Created Jaeger instance will show up in Jaeger tab. Once we click on the instance we will see what Kubernetes objects belong to it, and we can also modify the configuration if needed.

Let’s also have a look at created deployments by operators:

kubectl get deployment                                                                   
NAME                                                    READY deployment.apps/elasticsearch-cdm-defaultsimpleprod-1   1/1       
deployment.apps/simple-prod-collector                   1/1     
deployment.apps/simple-prod-query                       1/1    

Deployment on OpenShift also provides simplified and secure access to Jaeger console. Console URL can be obtained from the networking menu in UI or via oc get routes. To login to Jaeger we use our OpenShift credentials.

Conclusion

We have deployed Jaeger with Elasticsearch in two different ways. First we have used vanilla Kubernetes and upstream Elasticsearch operator. This required more configuration work and the creation of multiple Kubernetes objects. In the second scenario we have used OpenShift 4 with Operator Lifecycle Manager (OLM) user interface to provision Jaeger instance with Elasticsearch storage. This didn’t require any storage and security configuration. Last but not least OLM provides automatic updates for operators and their instances.

References

• Operator Hub https://operatorhub.io/
• Jaeger operator documentation https://www.jaegertracing.io/docs/1.15/operator/
• Elasticsearch operator https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html
• Self-provisioned Elasticsearch https://www.jaegertracing.io/docs/1.15/operator/#self-provisioned
• Elasticsearch cluster logging operator https://docs.openshift.com/container-platform/4.1/logging/config/efk-logging-elasticsearch.html

Jaeger with Elasticsearch on Kubernetes using Operators was originally published in JaegerTracing on Medium, where people are continuing the conversation by highlighting and responding to this story.

Jaeger uses index-per-day pattern to store its data to Elasticsearch. It creates a new index for each day based on span’s timestamp. These indices have to be periodically removed by jaeger-es-index-cleaner cron job. Typically users keep data from one week up to one month which results in 7 or 30 indices only for spans.

Index-per-day pattern might not be effective if data or resources are not evenly distributed. For example indices which do not contain any data still allocate shards or an index might contain significantly more data than the others. These use-cases result in sub-optimal use of resources.

How rollover works

In contrast to index-per-day management rollover uses an index alias which can rollover to a new index based on configured conditions. In practice, this means there are two aliases one for reading and another for writing. The read alias points to a set of read-only indices and write alias to one write index.

These are the conditions used for rolling over to a new index:

• max_age — the maximum age of the index
• max_docs — the maximum number of documents the index should contain
• max_size — the maximum estimated size of primary shards (since Elasticsearch 6.x)

The rollover REST API has to be called periodically, the conditions are evaluated only during the REST call and they are not stored in Elasticsearch. Therefore a cron job has to be configured to make use of this feature.

Configuration

The Jaeger configuration of rollover consists of several steps.

1. Initial configuration

Before deploying Jaeger it is mandatory to initialize the feature by running jaeger-es-rollover docker image (assuming Elasticsearch runs on localhost). This command creates read/write aliases and write indices.

docker run -it --rm --net=host jaegertracing/jaeger-es-rollover:latest init http://localhost:9200

Run the command again with-e ARCHIVE=true if you want to use this feature also with the archive index.

Once this is done Jaeger can be deployed with rollover enabled --es.use-aliases=true and --es.archive.use-aliases=true (for archive storage):

docker run -it --rm --net=host -e SPAN_STORAGE_TYPE=elasticsearch jaegertracing/all-in-one:latest --es.use-aliases=true --es-archive.enabled=true --es-archive.use-aliases=true

2. Rolling over to a new index

Jaeger has been deployed and now it writes data to a write alias. The next step is to periodically execute rollover API which rolls the write alias to a new index based on supplied conditions. The command also adds a new index to read alias to make new data available for search. The following command rolls the alias over to a new index if the age of the current write index is older than 1 second. We use this short time interval just for demo purposes to make sure rollover takes place.

docker run -it --rm --net=host -e CONDITIONS='{"max_age": "1s"}' jaegertracing/jaeger-es-rollover:latest rollover  http://localhost:9200

Run the same command with -e ARCHIVE=true for the archive storage.

The next step is to remove old indices from read aliases. It means that old data will not be available for search. This imitates the behavior of --es.max-span-age flag used in default index-per-day deployment. This step could be optional and old indices could be simply removed by index cleaner in the next step.

docker run -it --rm --net=host -e UNIT=seconds -e UNIT_COUNT=1 jaegertracing/jaeger-es-rollover:latest lookback  http://localhost:9200

Run the same command with -e ARCHIVE=true for the archive storage.

3. Removing old data

The old data from Elasticsearch has to be periodically removed by deleting old indices. In this case we are running index cleaner with the parameter0 which removes all indices including the ones created today.

docker run -it --rm --net=host -e ROLLOVER=true jaegertracing/jaeger-es-index-cleaner:latest 0 http://localhost:9200

Run the same command with -e ARCHIVE=true for the archive storage.

Note that this functionality is broken at the moment and it will be available in the next release (1.13.1/1.14.0). The issue is tracked in #1681.

Rollover in Jaeger Operator

Rollover feature can be enabled in jaeger-operator be specifying --es.use-aliases=true in storage flags. The operator will automatically initialize the aliases and configure rollover cron jobs. The rollover conditions can be specified in theesRollover section under storage. Please refer to the operator documentation for more details.

Migration from daily indices to rollover

Migration from default index-per-day deployment is possible by manually adding old indices to read alias.

curl -ivX POST -H "Content-Type: application/json" localhost:9200/_aliases -d '{
    "actions" : [
        { "add" : { "index" : "jaeger-span-*-*-*", "alias" : "jaeger-span-read" } },
 { "add" : { "index" : "jaeger-service-*-*-*", "alias" : "jaeger-service-read" } }
    ]
}'

And similarly for archive index:

curl -ivX POST -H "Content-Type: application/json" localhost:9200/_aliases -d '{
    "actions" : [
        { "add" : { "index" : "jaeger-span-archive", "alias" : "jaeger-span-archive-read" } }
    ]
}'

Useful links

• Elasticsearch rollover: https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-rollover-index.html
• Jaeger operator for Kubernetes https://github.com/jaegertracing/jaeger-operator

Using Elasticsearch Rollover to manage indices was originally published in JaegerTracing on Medium, where people are continuing the conversation by highlighting and responding to this story.