Distributed tracing with otel4s works best when it is built into the program rather than an afterthought layered on top. The API encourages clear ownership and disciplined context flow, which matches how real systems behave under load.

In this article, we looked at how traces are identified, how spans are scoped and propagated, and how otel4s translates OpenTelemetry concepts onto functional code.

The practical takeaway is simple. Instrument the operations that explain system behavior, keep spans aligned with meaningful work, and let context propagation do the rest. When tracing is consistent, the resulting traces align with the work and give you answers rather than more noise.

In the next part of the series, we will apply the same mindset to metrics and examine how to measure system behavior over time using otel4s.

Practical observability: foundations with OpenTelemetry

2025-12-14T10:00:00+00:00

In this series, we explore how to bring observability into a Scala application using OpenTelemetry</a> - the open standard for telemetry data - and otel4s</a>, a purely functional OpenTelemetry library for the Typelevel (Cats Effect</a>) ecosystem.

Series roadmap </div>

Part 1 (this post): Foundations with OpenTelemetry.</li>

Part 2</a>: Distributed tracing with otel4s.</li>
Part 3 (coming soon): Implementing metrics with otel4s.</li>

Part 4 (coming eventually): Advanced configuration and operational tips.</li> </ul> </div> </div>

This first post focuses on the conceptual foundation: what observability means, how telemetry signals work, and how they stay connected in distributed systems.

1. Introduction</h1>

There are countless variables that can affect how your system behaves in production.

Maybe a new marketing collaboration with a popular tech blogger triples your traffic overnight. A minor slowdown, such as a third-party API call or slow database query, can cascade through your services: threads pile up, external calls time out, queues overflow, and soon unrelated components start failing. Generally, such things are just accidents without a clear trail. We may know that something is wrong, somewhere, however, logs alone rarely tell the full story.

Why did latency spike last night?</li>
Why does one instance behave differently under the same load?</li>
Where does the time actually go between the request entering the system and the response leaving?</li> </ul>
This is where observability comes into the picture.
Observability isn't about collecting more data. It's about understanding your system through the data it produces. The key is capturing structured, correlated signals that let us reason about why something happens, not just what happened.
Traditionally, monitoring told us whether a service was healthy. Observability extends that idea: instead of asking Is it up?</code>, we ask Why is it behaving this way?</code> even when we didn't anticipate the failure mode.
To achieve that, we rely on telemetry - data emitted from our applications in the form of signals: traces, metrics, and logs. Together, these signals create a complete picture of how our system behaves.
2. From monitoring to observability</h1> </div> What problem are we actually trying to solve? </div> </div> Monitoring is great when we know what to expect. We define a few metrics, set thresholds, and get alerts when they are crossed. Unfortunately, in modern distributed systems, a single user request can cross dozens of services and queues. Once systems become that complex, we can't predict every possible point of failure. Monitoring is like a dashboard warning light - it catches a symptom and tells us something is wrong, for example, Latency has increased</code>. Observability is a diagnostic toolkit - it helps uncover the cause and understand why something is wrong, for example, Database connection pool is exhausted in one region</code>. Both are essential, but observability gives us the context to reason about system behavior, not just detect anomalies. As Scala engineers, we often work with concurrent and effectful systems built on asynchronous computations across multiple services. In such environments, failures often have multiple reasons. Monitoring might tell us that a request is slower than usual, but observability helps us pinpoint the source of the slowdown in the execution chain, whether it's a downstream dependency, a blocked fiber, or an overloaded queue. Observability solves the problem of having disconnected telemetry signals that, on their own, fail to tell a coherent story about what happened inside the system. By correlating traces, metrics, and logs around the execution of a single request, observability turns isolated symptoms into an explanation we can reason about. 3. Understanding telemetry signals</h1> Think of your system like a body that ate a questionable gas-station sandwich: when discomfort hits later, you need different vital signs to explain what happened. Traces, metrics, and logs are those vital signs: Traces show what the system did that led to an outcome: "Stopped for gas" -> "bought a sandwich" -> "ate it" -> "drove to work" -> "digestive regret".</li> Metrics describe continuous measurements, like a smartwatch reporting a pulse. You might see the moment when the heart rate spiked after lunch, but the metric alone doesn't explain why.</li> Logs capture the context in the moment: "This sandwich looks fine. What bad could happen?", a few hours later: "Regret. So much regret". Logs record real-time perception and events that help us understand why this happened.</li> </ul> Each signal offers a different perspective - traces supply the sequence, metrics capture the impact over time, and logs preserve moment-to-moment context. Together they tell us not only that something went wrong, but what happened, when, and why. That's observability: connecting separate telemetry pieces into a cohesive, meaningful view of system behavior. 3.1. Traces</h2> </div> What happened, step by step. </div> </div> Traces capture the storyline of a request: a chronological view of how work propagates through a system. Each trace is made up of spans. A span is a building block of a trace - it represents a specific operation and the time it takes to complete, such as handling an HTTP request or querying a database. A span records things like: When the operation started and ended</li> How it relates to other spans (parent/child relationships)</li> Whether it succeeded or failed</li> </ul> A trace then answers broader questions: What did the request actually do?</li> Which components were involved?</li> Where was time spent?</li> Where did errors occur?</li> </ul> Let's dive into an example. A user requests a weather forecast: The gateway-service</code> receives an HTTP GET /weather?location=Kyiv</code> request</li> The gateway-service</code> calls weather-service</code> to fetch the forecast</li> The gateway-service</code> also publishes a Kafka event with request details to the warehouse-service</code> (for analytics to consume later)</li> The weather-service</code> retrieves the forecast and returns it</li> </ul> When we visualize this trace, we can see the entire journey of the request: how it moves through services, how long each step took, and where problems occurred. gantt title Trace timeline: GET /weather?location=Kyiv dateFormat HH:mm:ss axisFormat %H:%M:%S section gateway-service HTTP request handling :active, 10:00:00, 10s Send request to weather-service : 10:00:02, 6s Publish Kafka event : 10:00:02, 3s section weather-service Fetch forecast : 10:00:03, 4s section warehouse-service Consume Kafka event : 10:00:11, 2s Persist analytics record : 10:00:12, 1s </pre> Traces reveal what happened, step by step, allowing us to follow a request across fibers, services, and boundaries. 3.2. Metrics</h2> </div> How the system behaves over time. </div> </div> If traces tell stories, metrics show patterns and trends. Metrics are the numbers a system emits continuously, such as counts, durations, and resource usages, that show how the system performs over time. They let us spot changes at scale: traffic spikes, regressions, or slow degradation that might not be visible from individual traces or logs. Metrics are also essential for alerting and automation. They can detect outages, trigger alerts, or even drive auto-scaling when demand suddenly increases. Metrics answer questions like: How fast are requests being processed on average?</li> What's the 95th percentile latency?</li> How many errors occurred in the past 5 minutes?</li> How much CPU or memory is being used right now?</li> </ul> Here are the most common types of metrics: Type</th> What it does</th> Example</th></tr></thead> Counter</td> Monotonically increasing total; no decrements, reset only on restart.</td> Count successful HTTP requests or emitted events.</td></tr> UpDownCounter</td> Value that increments when work starts and decrements when it ends.</td> Track in-flight HTTP requests or queued tasks by incrementing on start, decrementing on finish.</td></tr> Gauge/Observable</td> Snapshot gathered by a callback or poll.</td> Report CPU utilization, heap usage, or queue depth every collection cycle.</td></tr> Histogram</td> Distribution recorder that powers tail percentiles.</td> Emit latency and payload size metrics to alert on p95 and p99 regressions.</td></tr> </tbody></table> </div> Back to our over-engineered weather service, it might expose metrics like: http.server.active_requests</code> - an UpDownCounter tracking the number of active HTTP requests</li> http.server.request.duration</code> - a Histogram recording the distribution of request latencies</li> system.cpu.utilization</code> - a Gauge reflecting the current CPU usage</li> </ul> --- config: themeVariables: xyChart: titleColor: "#ff0000" --- xychart title "http.server.request.duration p95 latency" x-axis ["12:00", "14:00", "16:00", "18:00", "20:00", "22:00"] y-axis "Latency (ms)" 80 --> 260 bar "active requests" [120, 130, 140, 150, 135, 110] line "p95 duration" [110, 125, 140, 240, 160, 120] </pre> The chart shows how the weather-service histogram of request durations changes throughout a stormy afternoon. A spike around 18:00 signals downstream slowness despite traffic volume remaining roughly the same, precisely the kind of trend metrics excel at detecting early. Metrics reveal how a system is performing over time, helping us observe long-term trends, spot regressions early, and detect anomalies. 3.3. Logs</h2> </div> The context around what happened. </div> </div> If traces tell stories and metrics show trends, logs are the day-to-day journal of your system - recording events one at a time. Logs capture what's happening inside your code: messages, events, and context. They're timestamped, detailed, and usually the first place to look when something goes wrong. Logs answer questions such as: What specific event occurred?</li> What values or state were involved?</li> What errors or exceptions were thrown?</li> </ul> For example, our trusty weather service might log something like: 2025-10-03 15:14:18.123 INFO trace_id=4bf92f3577b34da6a3ce929d0e0e4736 span_id=00f067aa0ba902b7 Checking forecast for location [Kyiv]</code></pre> Here, the trace_id</code> and span_id</code> link this log entry to a specific span within a trace, allowing us to see all related operations and understand them in context. When logs are tied to traces, they stop being isolated lines of text and instead help explain what the system was doing when something happened. 3.4. Baggage</h2> </div> A global context along the path. </div> </div> There's also baggage - not a separate telemetry signal, but metadata that rides inside the same context payload as the trace identifiers. Baggage is a lightweight key-value map that propagates across services and threads, so every instrumentation can read the same request-level hints, such as tenant, plan, experiment flag, and so on. Think of it as a sticky note attached to the trace context: invisible to users yet available to each instrument. Because it travels with every outbound call, baggage should stay small and focused on attributes that truly help explain or partition the work. 4. Context propagation</h1> </div> The invisible glue that makes distributed tracing possible. </div> </div> Now that we've looked at what, when, and why of telemetry signals, one question remains: How do these signals stay connected? The answer is context propagation - the invisible glue that makes distributed tracing possible. The context carries metadata, such as trace and span IDs, that identify the current span and link it to the overall trace. With context propagation, spans can be correlated and assembled into a single trace, regardless of where each span originated. Modern applications are highly asynchronous and distributed. A single HTTP request might trigger a Kafka message, spawn a background fiber, and make a gRPC call before completing. Without context propagation, each of those operations would start a new trace, making it impossible to reconstruct the timeline of an individual request. In practice, context propagation means passing along the trace context: serializing it before an outbound call and deserializing it on the other side. Depending on the protocol, the serialized context travels in different carriers: HTTP - passed via request headers</li> gRPC - passed via metadata entries (the gRPC equivalent of headers)</li> Messaging (Kafka, RabbitMQ) - attached to message headers</li> </ul> For example, the W3C Trace Context</a> standard defines the traceparent</code> header that carries tracing information in HTTP: traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01</code></pre> where 4bf92f3577b34da6a3ce929d0e0e4736</code> is the trace ID, 00f067aa0ba902b7</code> is the span ID, and 01</code> is the trace flags. When a downstream service receives a request with this header, it extracts the context, restores the trace state, and starts a new child span linked to its parent, continuing the same trace seamlessly across service boundaries. sequenceDiagram autonumber actor User participant G as gateway participant WS as weather-service participant WH as warehouse User->>G: HTTP GET /weather?location=Kyiv activate G G->>WS: gRPC: Fetch forecast (location=Kyiv) activate WS WS-->>WS: Retrieve forecast WS-->>G: gRPC: OK + Forecast deactivate WS G-->>WH: Publish event to Kafka (request details) #Note over WH: Consumed for analytics later G-->>User: 200 OK + Forecast deactivate G </pre> In this example, the gateway service starts a new root trace and creates a child span for the gRPC request. When the weather-service handles a gRPC call, it extracts the trace context from the gRPC metadata and restores the trace state so the new span is linked to the parent. Warehouse consumes the Kafka event, extracts the context from the message headers, and either continues the parent or links to it depending on the ingestion style. That way, we can track the full journey of the request from start to finish. 5. OpenTelemetry</h1> </div> The standard behind it all. </div> </div> Obligatory XKCD</a> comics At this point, we've explored the what, when, why, and how of observability signals. Now it's time to look at the standard that ties all these ideas together. OpenTelemetry</a> is an open-source, vendor-neutral framework for collecting, processing, and exporting telemetry data (traces, metrics, logs) from applications. It defines how telemetry data is structured, correlated, and exchanged between services so that your observability setup doesn't depend on any single vendor or tool. Before OpenTelemetry, each observability vendor provided its own SDKs, APIs, and data formats. For application developers, that often meant choosing one monitoring stack and sticking with it. For library authors, it was more problematic: you'd have to provide instrumentation for every possible backend or force users into one ecosystem. OpenTelemetry provides a set of standards and conventions that all languages and vendors can follow. Each language ships an API (the abstraction instrumentation code targets) and an SDK (the concrete implementation that manages exporters, processors, and context propagation). With that split, application code depends only on the stable API while the SDK can evolve or be swapped without touching business logic. On top of the API/SDK split, the project defines surrounding components that keep telemetry portable: Common APIs and data models - consistent interfaces and data structures for generating telemetry signals in any language.</li> Context propagation standards</a> - a unified approach (e.g., the W3C Trace Context</a>) to share trace context across service boundaries.</li> OpenTelemetry Collector</a> - a standalone service that receives, processes (transforms/filters), and routes telemetry data between your applications and observability backends.</li> Semantic conventions</a> - standardized naming and attributes for common operations, such as HTTP requests, database calls, messaging systems, and more.</li> OTLP (OpenTelemetry Protocol)</a> - a standard wire protocol for transmitting telemetry data between SDKs, collectors, and backend systems.</li> </ul> Together, these parts form an ecosystem that keeps your telemetry consistent and portable, regardless of language, framework, or vendor. For example, you might collect traces in your services and send them to a tracing backend like Grafana Tempo</a>, export metrics to Prometheus</a>, and forward logs to Grafana Loki</a>. The OpenTelemetry Collector can route and export data to each of these systems using the standard OTLP format. Concepts stick once you see the API surface. The following sections show the same tiny span/metric examples first with the OpenTelemetry Java SDK, then with otel4s, so you can see the trade-offs between thread-local and effect-aware approaches. 6. OpenTelemetry Java overview</h1> The official OpenTelemetry Java</a> comes with instrumentation for many popular frameworks (e.g., Spring) and libraries (e.g., JDBC connection pools, HTTP clients). That works well for traditional Java or Kotlin applications. However, it doesn't play nicely with purely functional Scala apps built on top of Cats Effect</a>. The reason is that the Java SDK relies on thread-local context propagation, which doesn't align with how Cats Effect manages fiber-based concurrency and context. Below, we can peek at the API and compare it with the otel4s counterparts later. 6.1. Tracing API</h2> The OpenTelemetry Java API keeps instrumentation explicit: you build spans manually, push them into the current Scope</code>, run the work, and close the span yourself. Thread-local context propagation links the span to the current thread, so Span.current()</code> retrieves whatever you placed in scope. static void doWorkWithSpan(Tracer tracer) { Span span = tracer.spanBuilder("do-work").setAttribute("example.attribute", "hello").startSpan(); try (Scope scope = span.makeCurrent()) { doWork(); span.setStatus(StatusCode.OK); } catch (Throwable e) { span.setStatus(StatusCode.ERROR); span.recordException(e); span.setAttribute("error.type", e.getClass().getName()); } finally { span.end(); } } static void doWork() { Span current = Span.current(); System.out.println("Doing some work: " + current.getSpanContext().toString()); }</code></pre>6.2. Metrics API</h2> Metrics follow the same imperative pattern. You measure latency by hand and ensure you don't forget to record failures. static void doWorkWithMetrics(LongCounter counter, DoubleHistogram latencyMs) { Attributes attributes = Attributes.builder().put("example.attribute", "hello").build(); long startNanos = System.nanoTime(); try { doWork(); double elapsedMs = (System.nanoTime() - startNanos) / 1_000_000.0; latencyMs.record(elapsedMs, attributes); counter.add(1, attributes); } catch (Throwable e) { double elapsedMs = (System.nanoTime() - startNanos) / 1_000_000.0; latencyMs.record(elapsedMs, attributes.toBuilder.put("error.type", e.getClass.getName).build()); } } static void doWork() { System.out.println("Doing some work"); }</code></pre>7. otel4s overview</h1> otel4s</a> brings the OpenTelemetry specification to Scala in a purely functional form. It provides effect-aware context propagation and composable, idiomatic APIs for tracing, metrics, and logging. The API integrates seamlessly with Cats Effect applications, without resorting to thread-locals or unsafe side effects. While otel4s works perfectly with the official OpenTelemetry Java SDK, it also offers an alternative SDK implementation written in Scala. This allows you to use the same observability model across different platforms: JVM, Scala.js, and Scala Native. In other words, you can instrument Scala applications on the JVM or Scala Native using the same OpenTelemetry concepts, all within the Typelevel ecosystem. The library utilizes metaprogramming techniques to reduce runtime costs and allocations. Near-zero overhead when telemetry is disabled, ensuring production performance is unaffected when tracing or metrics collection is not required. --- config: look: handDrawn layout: elk --- graph BT otel-spec["OpenTelemetry Specification"] otel4s-core["otel4s-core Defines interfaces, such as Tracer, Meter, and Logger"] --> otel-spec otel4s-sdk["otel4s-sdk An independent implementation of the OpenTelemetry specification in Scala"] --> otel4s-core otel4s-oteljava["otel4s-oteljava Uses <a href="https://github.com/open-telemetry/opentelemetry-java">OpenTelemetry Java SDK</a> under the hood"] --> otel4s-core </pre> An oversimplified otel4s architecture A modular architecture</a> allows including only the required components and makes it easier for library authors to implement instrumentation for their tools. Now, let's get back to the API examples. 7.1. Tracing API</h2> otel4s can automatically manage the lifecycle of spans. This means you don't have to manually start and end them. If the effect fails, the error.type</code> attribute is automatically set to the full-qualified name of the exception class, and the span is marked as errored. Propagation relies on cats.mtl.Local</a> instead of thread-locals, which keeps the context in sync across fibers and async boundaries. def doWorkWithSpan(using Tracer[IO]): IO[Unit] = Tracer[IO].span("do-work", Attribute("example.attribute", "hello")).use { span => doWork() *> span.setStatus(StatusCode.Ok) } def doWork()(using Tracer[IO]): IO[Unit] = for { span <- Tracer[IO].currentSpanOrThrow _ <- IO.println(s"Doing some work: ${span.context}") } yield ()</code></pre> Nevertheless, you can still have full control over the span lifecycle by using a low-level API. 7.2. Metrics API</h2> Meters mirror that philosophy. Instruments such as Histogram</code> expose helpers like recordDuration</code> that wrap an effect and capture timing without manually using timers. def doWorkWithMetrics(counter: Counter[IO, Long], latencyMs: Histogram[IO, Double]): IO[Unit] = { val attributes = Attributes(Attribute("example.attribute", "hello")) val histogramAttributes: Resource.ExitCase => Attributes = { case Resource.ExitCase.Succeeded => attributes case Resource.ExitCase.Errored(e) => attributes + Attribute("error.type", e.getClass.getName) case Resource.ExitCase.Canceled => attributes + Attribute("error.type", "canceled") } latencyMs.recordDuration(TimeUnit.MILLISECONDS, histogramAttributes).surround { doWork() *> counter.add(1L, attributes) } } def doWork(): IO[Unit] = IO.println("Doing some work")</code></pre> As with the tracing API, you can still use the low-level API if you need more control. 8. Conclusion</h1> Observability gives us a way to understand systems through the signals they produce: traces, metrics, and logs. We've looked at how these signals complement each other, how context propagation keeps them connected, and how OpenTelemetry provides a shared foundation for collecting and exchanging them. For Scala developers, otel4s brings these ideas into the functional world, allowing telemetry to flow naturally through effectful code while staying consistent with OpenTelemetry standards. This post focused on the concepts behind observability: what the signals mean, how they relate, and why they matter. In the next part of the series, we'll move from theory to practice and see how to use otel4s to instrument the application. Publishing test traces to Grafana using otel4s and weaver 2024-08-02T10:00:00+00:00 Let's explore how we can publish test traces to Grafana Cloud</a> using otel4s</a> and weaver</a> while running tests on GitHub Action. Even though CI machines often differ from production environments, and recorded spans might be volatile, we can still collect enough evidence to analyze system components' performance (latency). So why bother with test traces? There are a few benefits: Identify potential bottlenecks</li> Track performance degradation</li> Check the structure of spans</li> </ol> The complete example is available in the GitHub repository</a>. While this example uses Grafana Cloud, it will work with any OTLP-compatible setup: Jaeger, Honeycomb, etc. 1. Why use weaver and otel4s?</h1> Weaver</a> is an excellent choice for testing due to its seamless integration with Cats Effect</a>, parallelism out of the box, and flexible API. otel4s</a> is an OpenTelemetry</a> implementation for Scala built on top of Cats Effect. The library is available for all platforms: JVM, Scala.js, Scala Native. 2. otel4s befriends weaver</h1> Settings up the project</h2> To set up your project include the following dependencies in your build.sbt</code> file: libraryDependencies ++= Seq( "org.typelevel" %% "otel4s-core-trace" % "0.8.1", "org.typelevel" %% "otel4s-sdk-exporter-trace" % "0.8.1" % Test, "com.disneystreaming" %% "weaver-cats" % "0.8.4" % Test, "com.disneystreaming" %% "weaver-scalacheck" % "0.8.4" % Test )</code></pre>How weaver handles test results</h2> Weaver handles test results slightly differently from traditional testing frameworks by leveraging the Expectations</code> as the outcome of a test. Instead of failing tests with exceptions, tests return a value of type Expectations</code>, where Expectations</code> is defined as ValidatedNel[AssertionException, Unit]</code>. Extending weaver</h2> First of all, we need to instrument the tests with the traces. Let's start with the utility object: object TraceUtils { def trace[F[_]: MonadCancelThrow: Tracer]( suiteName: String, name: TestName )(fa: F[Expectations]): F[Expectations] = { val testName = name.name def attributes = Attributes( // <1> Attribute("test.name", testName), Attribute("test.suite", suiteName), Attribute("test.source", name.location.fileRelativePath + ":" + name.location.line), Attribute("test.tags", name.tags.toSeq) ) Tracer[F] .spanBuilder(s"$suiteName - $testName") .addAttributes(attributes) .withFinalizationStrategy(SpanFinalizer.Strategy.empty) // <2> .build .use(span => fa.guaranteeCase(recordOutcome(span))) } def recordOutcome[F[_]: Monad](span: Span[F]): Outcome[F, Throwable, Expectations] => F[Unit] = ??? }</code></pre> Configure span attributes: the test name, suite name, source location, and any tags associated with the test.</li> Disable default finalization strategy: we'll handle the statuses manually by recording the outcome of the test execution.</li> </ol> To ensure proper span finalization based on test results, we need to implement the recordOutcome</code> function: private val AnsiColorRegex = "\u001b\\[([;\\d]*)m".r def recordOutcome[F[_]: Monad](span: Span[F]): Outcome[F, Throwable, Expectations] => F[Unit] = { case Outcome.Succeeded(fa) => fa.flatMap { expectations => expectations.run.fold( // <1> e => span.setStatus(StatusCode.Error) >> span.recordException(removeAnsiColors(e.head)), _ => span.setStatus(StatusCode.Ok) ) } case Outcome.Canceled() => span.setStatus(StatusCode.Error, "canceled") // <2> case Outcome.Errored(error) => // <3> span.setStatus(StatusCode.Error) >> span.recordException(removeAnsiColors(error)) } private def removeAnsiColors(throwable: Throwable): Throwable = throwable match { case a: AssertionException => a.copy(message = AnsiColorRegex.replaceAllIn(a.message, "")) case other => other }</code></pre> If there's an error, we set the span status to Error</code> and record the exception. If the expectations are met, we set the span status to Ok</code>.</li> Set the span status to Error</code> with the description 'canceled', indicating that the test was aborted.</li> Set the span status to Error</code> and record the exception.</li> </ol> Additionally, the removeAnsiColors</code> function cleans up ANSI escape codes from the exception messages to ensure they appear clearly in Grafana or Jaeger. Extending the SimpleIOSuite</code> and IOSuite</code></h3> Weaver provides two types of test suites: SimpleIOSuite</code> for tests without shared resources</li> IOSuite</code> for tests with shared resources</li> </ul> We can add a tracedTest</code> utility method to the TracedSimpleIOSuite</code> trait: trait TracedSimpleIOSuite extends SimpleIOSuite { def tracedTest(name: TestName)(run: Tracer[IO] => IO[Expectations]): Unit = registerTest(name)(_ => Test(name.name, traced(name)(run))) private def traced(testName: TestName)(body: Tracer[IO] => IO[Expectations]): IO[Expectations] = SdkTraces .autoConfigured[IO](_.addExporterConfigurer(OtlpSpanExporterAutoConfigure[IO])) // <1> .evalMap(_.tracerProvider.get(getClass.getName.stripSuffix("$"))) .use(implicit tracer => TraceUtils.trace(name, testName)(body(tracer))) // <2> }</code></pre> Create an auto-configured Tracer[IO]</code>.</li> Use TraceUtils.trace</code> to wrap the test execution in a tracing span and record the outcome.</li> </ol> For tests that use shared resources, such as a Tracer[IO]</code>, the TracedIOSuite</code> provides a way to integrate tracing into IOSuite</code>: trait TracedIOSuite extends IOSuite { self => type Res = Tracer[IO] def sharedResource: Resource[IO, Tracer[IO]] = // <1> SdkTraces .autoConfigured[IO](_.addExporterConfigurer(OtlpSpanExporterAutoConfigure[IO])) .evalMap(_.tracerProvider.get(getClass.getName.stripSuffix("$"))) override def test(name: TestName): PartiallyAppliedTest = // <2> new TracedPartiallyAppliedTest(name) private final class TracedPartiallyAppliedTest( testName: TestName ) extends PartiallyAppliedTest(testName) { override def apply(run: Res => IO[Expectations]): Unit = registerTest(testName) { tracer => Test(testName.name, (log: Log[IO]) => traced(log, tracer)((res, _) => run(res))) } override def apply(run: (Res, Log[IO]) => IO[Expectations]): Unit = registerTest(testName) { tracer => Test(testName.name, (log: Log[IO]) => traced(log, tracer)(run)) } private def traced( log: Log[IO], tracer: Tracer[IO] )(run: (Tracer[IO], Log[IO]) => IO[Expectations]): IO[Expectations] = TraceUtils.trace(name, testName)(run(tracer, log))(implicitly, tracer) } }</code></pre> The sharedResource</code> method sets up a shared Tracer[IO]</code> instance that is created once per suite, so all tests within the suite use the same Tracer</code>.</li> The test</code> method is overridden to incorporate tracing into the test execution.</li> </ol> Bonus point. Instrumenting ScalaCheck</h3> ScalaCheck</a> is a powerful property-based testing framework that allows you to test your code against pseudo-randomly generated properties. We can wrap each property check invocation into its own span to make traces more meaningful and granular: trait TracedIOCheckers extends Checkers { self: MutableIOSuite => // must be 'def', so for every instantiation there will be a fresh counter implicit protected def unwrapWithTrace(implicit T: Tracer[F]): PropF[F[Expectations]] = new Checkers.Prop[F, F[Expectations]] { private val counter = Ref.unsafe[IO, Int](0) def lift(a: F[Expectations]): F[Expectations] = for { count <- counter.getAndUpdate(_ + 1) result <- Tracer[F] .spanBuilder(s"forall#$count") .withFinalizationStrategy(SpanFinalizer.Strategy.empty) .build .use(span => a.guaranteeCase(TraceUtils.recordOutcome(span))) } yield result } }</code></pre> The unwrapWithTrace</code> method creates a new tracing span for each property check invocation. The span name includes a counter to differentiate between multiple property checks within the same test. 3. Write tests</h1> With the tracing utilities in place, we can finally write tests for the Service</code> class that calculates the sum of two numbers: class Service[F[_]: Temporal: Tracer: Random] { def sum(x: Long, y: Long): F[Long] = Tracer[F].span("sum", Attribute("x", x), Attribute("y", y)).surround { for { delay <- Random[F].nextIntBounded(50) _ <- Temporal[F].sleep(delay.millis) // simulates computation time } yield x + y } } object Service { def create[F[_]: Async: Tracer]: F[Service[F]] = Random.scalaUtilRandom[F].map(implicit random => new Service[F]) }</code></pre>Tests with TracedSimpleIOSuite</code></h3> object ServiceSuite extends TracedSimpleIOSuite with TracedIOCheckers { tracedTest("calculate a sum of numbers (fixed)") { implicit tracer => for { service <- Service.create[IO] result <- service.sum(1L, 2L) } yield expect(result == 3L) } tracedTest("calculate a sum of numbers") { implicit tracer => forall { (a: Long, b: Long) => for { service <- Service.create[IO] result <- service.sum(a, b) } yield expect(result == (a + b)) } } }</code></pre>Tests with TracedIOSuite</code></h3> object ServiceSharedTracerSuite extends TracedIOSuite with TracedIOCheckers { test("calculate a sum of numbers (fixed)") { implicit tracer => for { service <- Service.create[IO] result <- service.sum(1L, 2L) } yield expect(result == 3L) } test("calculate a sum of numbers") { implicit tracer => forall { (a: Long, b: Long) => for { service <- Service.create[IO] result <- service.sum(a, b) } yield expect(result == (a + b)) } } }</code></pre>4. Collect traces</h1> Now, our tests are instrumented and can export proper traces. It's time to collect them while running tests on GitHub Actions. Prepare build.sbt</code></h2> I suggest using sbt-github-actions</a> plugin to generate the GitHub Actions workflow. First of all, we need to bind OpenTelemetry variables to configure the exporter: ThisBuild / githubWorkflowEnv ++= Map( "OTEL_EXPORTER_OTLP_PROTOCOL" -> "${{ secrets.OTEL_EXPORTER_OTLP_PROTOCOL }}", // <1> "OTEL_EXPORTER_OTLP_ENDPOINT" -> "${{ secrets.OTEL_EXPORTER_OTLP_ENDPOINT }}", // <2> "OTEL_EXPORTER_OTLP_HEADERS" -> "${{ secrets.OTEL_EXPORTER_OTLP_HEADERS }}", // <3> "OTEL_SERVICE_NAME" -> "${{ github.ref }}", // <4> "OTEL_RESOURCE_ATTRIBUTES" -> "revision=${{ github.sha }}", // <5> "OTEL_SDK_DISABLED" -> "${{ !startsWith(github.ref, 'refs/pull') }}" // <6> )</code></pre> Define the protocol for exporting traces (e.g., grpc, http/protobuf)</li> The OTLP collector endpoint where spans should be sent</li> Security headers for authentication with the collector</li> Dynamically name the service based on the GitHub reference</li> Include additional attributes (like commit SHA) for filtering traces in Grafana</li> Disable tracing for non-PR branches, ensuring traces are only published for pull requests</li> </ol> Configure Cloud Grafana</h2> There are a few mandatory steps before we can start sending traces to the Grafana: Sign-in</a> into Grafana Cloud and navigate to your org:</h3> Enable OpenTelemetry OTLP collector</h3> Create a new security token</h3> Save the secrets</h3> Configure GitHub secrets and variables</h2> Navigate to your GitHub repository, go to Settings > Secrets and variables > Actions</code>, and add the secrets obtained in the previous step. The secrets will be picked up by sbt while running tests. Configure GitHub secrets</h3> Configure GitHub variables</h3> Bonus point. Create a comment on a PR with a link to traces</h2> Update the build.sbt</code> file: ThisBuild / githubWorkflowBuildPreamble += // <1> WorkflowStep.ComputeVar("tests_start_time", "date -d \"$(date +'%Y-%m-%d') 00:00:00\" +%s%3N") ThisBuild / githubWorkflowBuildPostamble += // <2> WorkflowStep.ComputeVar("tests_end_time", "date -d \"$(date +'%Y-%m-%d') 23:59:59\" +%s%3N") ThisBuild / githubWorkflowBuildPostamble += { // <3> def body = { val panes = """{"7uq": { | "datasource":"grafanacloud-traces", | "queries":[ | { | "refId":"A", | "datasource":{"type":"tempo","uid":"grafanacloud-traces"}, | "queryType":"traceql", | "limit":100, | "query":"{resource.revision=\"${{ github.sha }}\"}" | } | ], | "range":{"from":"${{ env.tests_start_time }}","to":"${{ env.tests_end_time }}"} |}}""".stripMargin.replace(" ", "").replace("\n", "") val link = s"https://$${{ vars.GRAFANA_HOST }}/explore?panes=$panes&schemaVersion=1&orgId=1" s"The traces can be reviewed here - $link." } WorkflowStep.Use( UseRef.Public("peter-evans", "create-or-update-comment", "v4"), name = Some("Publish 'Grafana traces' comment"), cond = Some("startsWith(github.ref, 'refs/pull')"), params = Map( "issue-number" -> "${{ github.event.pull_request.number }}", "body" -> body ) ) }</code></pre> Register tests_start_time</code> before tests start - start of the day</li> Register tests_end_time</code> once tests are executed - end of the day</li> Add a comment to the pull request</li> </ol> It creates a comment on your pull request with a link to Grafana, where you can view traces for the specific commit and time range. Note that Grafana currently does not support timezone parameters in the URL, so the time range is set to a whole day of the test execution time. To make comments work, you must enable token's write permissions in the Settings > Actions > General > Read and write permissions</code>: 5. Check the result</h1> Push your changes, create a PR, check the result. The generated link automatically shows all spans associated with the current pipeline run. Successful traces: An errored trace: Scala 2: boilerplate-free type class definition 2020-01-09T00:00:00+00:00 The development of an application requires design decisions. Fairly often a functional application is being designed around ad-hoc polymorphism. Ad-hoc polymorphism refers to when a value is able to adopt any one of several types because it, or a value it uses, has been given a separate definition for each of those types. Haskell wiki</a> </blockquote> Type class is an approach to ad-hoc polymorphism in Scala. Functional libraries widely use this concept: EntityEncoder</code> in http4s</a>, Functor</code> in cats</a>, Encoder</code> in circe</a>, and so on. Since you need an instance of a type class for a specific type it can be either created manually or derived by a library (shapeless</a>, magnolia</a>). In common, an instance can be defined in a companion object or being derived at a call site. This is the point where the problem starts. TL;DR</h2> scalaz-deriving</a> is a zero dependency library that provides a boilerplate-free way of defining a type class instance: @scalaz.annotation.deriving(Eq, Hash, Show, Codec) final case class User(firstName: String, lastName: String)</code></pre> The detailed configuration described at the end of the article. What's wrong with automated derivation</h1> I guess everyone is familiar with wildcard imports that provide a generic derivation of a type class, e.g. import io.circe.generic.auto._</code> , import ShowMagnolia._</code> , and so on. This approach has several disadvantages: 1. A type class is being generated at every call site It’s not a problem until your application is small or you don’t care about the compilation speed. Example: import io.circe.generic.auto._ final case class User(firstName: String, lastName: String) def stringifyNoSpaces(value: User): String = io.circe.Encoder[User].apply(value).noSpaces // Encoder instance #1 def stringify(value: User): String = io.circe.Encoder[User].apply(value).toString // Encoder instance #2 def decode(input: String): Either[io.circe.Error, String] = io.circe.parser.decode[User](input) // Decoder instance #1</code></pre> 2. The output can be different The problem arises from the previous point. Magnolia can choose a different instance for the underlying type based on the imports. Example: import ShowMagnolia._ final case class User(firstName: String, lastName: String) def stringify1(value: User): String = { import Domain.toUpperCaseStringShow Show[User].show(value) // Instance #1 } def stringify2(value: User): String = Show[User].show(value) // Instance #2 val user = User("John", "Smith") stringify1(user) // output: User(firstName = JOHN, lastName = SMITH) stringify2(user) // output: User(firstName = John, lastName = Smith)</code></pre> The example above is slightly unreal in the real world. But you can get a different behavior due to an accidental wildcard import import Domain._</code>. Cached instances</h1> An instance can be defined in a companion object and the compiler will reuse it in all necessary places: import io.circe.Codec import io.circe.generic.semiauto final case class User(firstName: String, lastName: String) object User { implicit val userCodec: Codec[User] = semiauto.deriveCodec }</code></pre> Looks good, isn’t it? The compiler will automatically pick up an instance of the type class from the companion object. The most important thing, that the instance is being created only once, which is good for the compilation speed. But the further you dive into the development process, the more type classes your model requires: cats.Hash</code>, cats.Eq</code>, cats.Show</code>, and so on. import io.circe.generic.semiauto final case class User(firstName: String, lastName: String) object User { implicit val userCodec: Codec[User] = semiauto.deriveCodec implicit val userEq: Eq[User] = Eq.fromUniversalEquals implicit val userHash: Hash[User] = Hash.fromUniversalHashCode implicit val userShow: Show[User] = ShowMagnolia.derive }</code></pre> Well, this one looks much more polluted. And now imagine that you have dozens of different models. And it quickly turns into a mess. Looking at the example above we can trace the pattern: the creation of an instance does not require any custom logic. An instance is being created by a utility method and since the implementation depends only on the type, the boilerplate part can be generated by a compiler plugin. scalaz-deriving plugin</h1> Apart from the name, the compiler plugin does not have any dependency on scalaz. The plugin creates corresponding instances using derivation methods described at deriving.conf</code>. The plugin logic is pretty straightforward: Collect instances that should be derived;</li> Find a derivation method in the resources/deriving.conf</code> file;</li> Generate (if it does not exist) a companion object and append necessary instances.</li> </ol> Configuration</h2> Add the compiler plugin and macro library to the build.sbt</code></li> </ol> libraryDependencies += "org.scalaz" %% "deriving-macro" % "2.0.0-M7" addCompilerPlugin("org.scalaz" %% "deriving-plugin" % "2.0.0-M7" cross CrossVersion.full)</code></pre> If you want to use methods for derivation defined in the same compilation unit (e.g. Magnolia), add to the build.sbt</code>: Compile / managedClasspath := { val res = (Compile / resourceDirectory).value val old = (Compile / managedClasspath).value Attributed.blank(res) +: old }</code></pre> Describe derivation rules in the resources/deriving.conf</code> file</li> </ol> The compiler plugin takes the rules from the deriving.conf</code> file, which must be in the resources</code> folder. Rules example: cats.kernel.Eq = cats.kernel.Eq.fromUniversalEquals cats.Show = example.ShowMagnolia.derive cats.kernel.Hash = cats.kernel.Hash.fromUniversalHashCode io.circe.Codec = io.circe.generic.semiauto.deriveCodec</code></pre> Annotate your domain models</li> </ol> @scalaz.annotation.deriving(Eq, Hash, Show, Codec) final case class User(firstName: String, lastName: String)</code></pre> The difference is significant, isn’t it? The model looks much cleaner now. The first thing you can pay attention to is the lack of a companion object. It’s being generated along with the type classes. Moreover, the plugin supports generic classes as well: @scalaz.annotation.deriving(Show) final case class GenericClass[A, B, C](a: A, b: B, c: C)</code></pre> The generated code: final case class GenericClass[A, B, C](a: A, b: B, c: C) object GenericClass { implicit def _deriving_show[A, B, C](implicit evidence$$A: Show[A], evidence$$B: Show[B], evidence$$C: Show[C] ): Show[GenericClass[A, B, C]] = example.ShowMagnolia.derive }</code></pre> As you can see, scalaz-deriving</code> provides a simple way of keeping the models clean. Instead of a conclusion</h1> Not only scalaz-deriving</code> can help us with a boilerplate. The library derevo</a> provides a similar syntax, but it does not support custom types yet. Also, Scala 3 received a fancy derivation mechanism. But in Scala 2 world, we can rely either on scalaz-deriving</code> or derevo</code>. Further reading</h1> Magnolia</a> — shapeless-free type class derivation. Scala 3 Derivation</a> — Scala 3 finally received a fancy derivation mechanism. Implicits, type classes, and extension methods</a> — explained Scala 2 implicits.

Practical observability: foundations with OpenTelemetry

Publishing test traces to Grafana using otel4s and weaver

4. Collect traces</h1>
Now, our tests are instrumented and can export proper traces. It's time to collect them while running tests on GitHub Actions.</p>

Configure Cloud Grafana</h2>
There are a few mandatory steps before we can start sending traces to the Grafana:</p>

`5. Check the result</h1> Push your changes, create a PR, check the result. The generated link automatically shows all spans associated with the current pipeline run.</p> Successful traces: </p> An errored trace: </p>`

Scala 2: boilerplate-free type class definition

`Further reading</strong></h1> Magnolia</a> — shapeless-free type class derivation. Scala 3 Derivation</a> — Scala 3 finally received a fancy derivation mechanism. Implicits, type classes, and extension methods</a> — explained Scala 2 implicits.</p>`

Maksym Ochenashko

Practical observability: distributed tracing with otel4s

Maksym Ochenashko

Practical observability: distributed tracing with otel4s

7.5. Seeing traces in Grafana</h2> To validate the traces, we run the service locally, send a real request, and inspect the resulting trace in Grafana using a local OpenTelemetry stack.</p>

Practical observability: foundations with OpenTelemetry

Publishing test traces to Grafana using otel4s and weaver

2. otel4s befriends weaver</h1> Settings up the project</h2> To set up your project include the following dependencies in your build.sbt</code> file:</p>

Settings up the project</h2> To set up your project include the following dependencies in your build.sbt</code> file:</p>

4. Collect traces</h1> Now, our tests are instrumented and can export proper traces. It's time to collect them while running tests on GitHub Actions.</p>

Configure Cloud Grafana</h2> There are a few mandatory steps before we can start sending traces to the Grafana:</p>

Enable OpenTelemetry OTLP collector</h3> </p>

Create a new security token</h3> </p>

Save the secrets</h3> </p>

Configure GitHub secrets and variables</h2> Navigate to your GitHub repository, go to Settings > Secrets and variables > Actions</code>, and add the secrets obtained in the previous step. The secrets will be picked up by sbt while running tests.</p>

Configure GitHub variables</h3> </p>

5. Check the result</h1> Push your changes, create a PR, check the result. The generated link automatically shows all spans associated with the current pipeline run.</p> Successful traces: </p> An errored trace: </p>

Scala 2: boilerplate-free type class definition

4. Collect traces</h1>
Now, our tests are instrumented and can export proper traces. It's time to collect them while running tests on GitHub Actions.</p>

Configure Cloud Grafana</h2>
There are a few mandatory steps before we can start sending traces to the Grafana:</p>

Enable OpenTelemetry OTLP collector</h3>
</p>

Create a new security token</h3>
</p>

Save the secrets</h3>
</p>

Configure GitHub secrets and variables</h2>
Navigate to your GitHub repository, go to `Settings > Secrets and variables > Actions</code>, and add the secrets obtained in the previous step. The secrets will be picked up by sbt while running tests.</p>`

Configure GitHub variables</h3>
</p>

`5. Check the result</h1> Push your changes, create a PR, check the result. The generated link automatically shows all spans associated with the current pipeline run.</p> Successful traces: </p> An errored trace: </p>`