Frameworks

High-volume REST APIs can easily become bottlenecked by database access, leading to high latency and poor throughput. Even after optimizing SQL queries and adding indexes, a database call might take hundreds of milliseconds, still far slower than a competitor’s 50 ms response that leverages caching. In-memory caching offers orders of magnitude faster data access. Traditional databases measure response times in milliseconds, while Redis operations complete in microseconds. By storing frequently accessed data in memory, APIs can handle dramatically more requests per second with much lower latency. As an example, one test showed that using Redis cut an expensive request’s response time from over 10 seconds down to under 1 second. Setting Up Redis Caching in Spring Boot Before diving into patterns, let’s ensure the basic setup is in place. We assume you have a local Redis server running. In your Spring Boot project, include the necessary dependencies for caching and Redis integration. For example, add the following to your Maven pom.xml: XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-cache</artifactId> <version>3.1.5</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-redis</artifactId> <version>3.1.5</version> </dependency> These bring in Spring’s generic caching support and the Redis connector. Next, enable caching in your application by annotating a configuration or main class with @EnableCaching. Spring Boot will auto-configure a RedisCacheManager if it finds Redis on the classpath. You can then define cache settings via configuration. For example, you might set a default time to live for cache entries in application.properties or via a RedisCacheConfiguration bean. A simple property-based configuration for a local Redis could be: Properties files spring.cache.type=redis spring.redis.host=localhost spring.redis.port=6379 spring.cache.redis.time-to-live=600000 # 600000 ms = 10 minutes TTL Now we have a basic cache setup. Let’s explore caching patterns and how to implement them in Spring Boot. Write-Through and Write-Behind Caching Caching isn’t just for reads; we also need a strategy for writes. Write-through and write-behind are patterns to handle data modifications in a cached system: Write-Through On every data write, the application synchronously writes to the database and the cache. This ensures the cache is always up-to-date with the latest data. In practice, a write-through approach might perform the database operation, then immediately update the Redis cache with the new value. Spring’s caching abstraction can support this via annotations like @CachePut or by combining a normal save method with a manual cache update. For example, in a product service, we might do: Java @CachePut(value = "products", key = "#product.id") public Product updateProduct(Product product) { // Save to DB first Product saved = repo.save(product); return saved; // Spring will put this return value into "products::[id]" cache } This method will update the database and also put the new product data into the cache under the given key. The next read for that product can be served from cache immediately, with no stale data. If we delete an item, we can use @CacheEvict to remove it from the cache at the same time as removing it from the DB, preventing ghost entries. Write-Behind (Write-Back) In this less common strategy, the application writes to the cache first and defers the database write till later. The idea is to batch or coalesce many writes to reduce DB pressure. Avoiding Cache Stampede (Thundering Herd) When caching for high-volume traffic, cache stampedes are a serious concern. A stampede occurs when a cache entry expires or is missing, and many concurrent requests attempt to fetch the same data from the database at once. In a high QPS system, this can overwhelm the database and essentially negate the benefit of caching. We need strategies to prevent dozens or hundreds of threads from piling onto the DB when a popular item cache invalidates. One common solution is to use locking or synchronization around cache misses. The idea is to ensure only one thread does the expensive database fetch and populates the cache, while the others wait or get served a stale value. In a single-instance application, you might synchronize on a Java lock per key. In a distributed environment, you’ll want a distributed lock. Redis itself can be used to implement this. For our Spring Boot application, we could integrate Redisson and use it in the service method. For instance: Java RLock lock = redissonClient.getLock("lock:product:" + productId); boolean acquired = lock.tryLock(5, 10, TimeUnit.SECONDS); // wait up to 5s to acquire, auto-release after 10s if (acquired) { try { // Double-check cache after acquiring lock Product cached = redisTemplate.opsForValue().get(cacheKey); if (cached != null) { return cached; } // Cache still empty, fetch from DB and update cache Product dbData = repo.findById(productId); redisTemplate.opsForValue().set(cacheKey, dbData, Duration.ofMinutes(10)); return dbData; } finally { lock.unlock(); } } else { // Could not acquire lock (timed out) – fallback to a stale cache or return an error ... } In the above pseudocode, multiple threads hitting a missing cache key will attempt to tryLock. One will succeed and do the DB query, while others wait up to 5 seconds. Once the first thread populates the cache and releases the lock, the others will find the data in the cache and avoid hitting the DB. This approach effectively serializes the cache miss for a given key, preventing a herd of concurrent DB calls. It’s a bit heavy, so you might not use it for every key; typically, you'll use it for very hot items or expensive queries that you know could trigger stampedes. Simpler techniques can also mitigate stampedes, like cache early recomputation or using slightly randomized TTLs so not everything expires at the same time. Load Testing the Impact of Caching With JMeter After implementing Redis caching, it’s critical to verify the performance improvements under realistic load. Apache JMeter is a popular tool for simulating concurrent users and measuring response times and throughput of your API. We can use JMeter to compare the API’s behavior with and without cache and ensure that our caching does indeed handle high volume as expected. For example, suppose we want to test an endpoint /products/{id} which we’ve optimized with caching. We can create a JMeter test plan with a Thread Group of, say, 100 threads and loop them to send requests for various product IDs. JMeter will report metrics like average response time, throughput, error rate, etc. In a baseline test, you might observe higher latencies and lower throughput. Then, in a test with the cache warmed (most requests hitting the cache), you should see a dramatic reduction in response time and the ability to handle more requests per second. In one real-world inspired demo, using Redis caching improved latency from 10 seconds on a cold miss to under 1 second on subsequent hits. Another way to look at it: memory caching can serve data so fast that your throughput might be an order of magnitude higher than relying solely on the DB. This aligns with the earlier statement that no amount of DB tuning beats data served from an in-memory cache. Using JMeter Set up JMeter (you can run it in GUI mode to design the test plan, and then use non-GUI mode for the actual high-load run for better accuracy). Configure an HTTP Request sampler pointing at your API (e.g., GET http://localhost:8080/products/1234). Use a Thread Group to simulate the desired number of concurrent users and iterations. You can add a Timer if you want a delay between requests, or just hammer the API as fast as possible to find its max throughput. Add listeners like Summary Report or Aggregate Report to gather results. To automate performance testing, you can even integrate JMeter with your build. A Maven plugin exists to run JMeter tests as part of a build pipeline. JMeter Configuration Snippet Suppose we want to quickly run a load test from the command line (non-GUI). We could use a command like: Shell jmeter -n -t path/to/testplan.jmx -l results.jtl -Jthreads=100 -Jduration=60 This would run the JMeter test plan for 60 seconds with 100 threads, logging results to results.jtl. Make sure to monitor your system while testing, especially if everything is on the same machine; the load test could itself become a bottleneck or interfere with results if not planned carefully. As a quick check, you can also use Spring Boot Actuator metrics or Redis monitoring to see cache hit rates. A healthy caching layer under load should show a high cache hit percentage, which correlates with lower DB usage and faster responses. Conclusion Optimizing a high-volume REST API often requires rethinking data access patterns, and Redis caching is a powerful technique to achieve massive performance gains. By using the cache-aside pattern, we serve most reads from fast in-memory storage, drastically reducing latency and database load. With write-through strategies and careful cache invalidation, we keep cached data consistent with the source of truth. It’s equally important to anticipate real-world issues like cache stampedes using locks or other techniques to prevent cache misses from overwhelming your database in a traffic surge. Finally, always test under load. Use tools like JMeter to simulate concurrent access and measure the impact of your caching. You should observe significant improvements in throughput and response times, validating that the cache is doing its job. If the results aren’t as expected, that’s an indication to refine your caching strategy or investigate bottlenecks.

I’ve just released Spring CRUD Generator v1.1.0 — an open-source generator that helps you bootstrap a Spring Boot CRUD backend from a single YAML specification. If you’ve built more than a couple of CRUD-heavy services, you’ve probably experienced the same pain points: repeating the same layers (entity, repository, service, controller), keeping consistent naming and structure across modules, and constantly adjusting boilerplate when requirements change. Spring CRUD Generator aims to reduce that overhead by letting you define your data model and project options once (in YAML) and generate a consistent project structure around it. This release adds field-level validation, improves Redis caching, and fixes compatibility issues so the generator works reliably with Spring Boot 3 and Spring Boot 4. It also improves behavior when Open Session In View (OSIV) is disabled by adding EntityGraph support in generated resources. What’s New in v1.1.0 1. Field Validation (fields.validation) The headline feature in v1.1.0 is a new optional “validation” section inside each entity field definition. Instead of sprinkling validation rules manually throughout your DTOs and controllers after generation, you can now describe typical constraints directly in the YAML config and have the generator produce the appropriate validation-aware output. This is useful for teams that want a single source of truth for model constraints. A YAML-driven validation model also makes it easier to review and evolve constraints alongside schema changes. For example, you can express “this field must be required” or “string length must be within a range” directly where the field is defined. A notable addition in this release is support for regex-based validation via “pattern.” That’s a practical constraint for fields like passwords, identifiers, or custom-formatted strings. It’s worth mentioning that the validation section is optional: if you don’t need it, you don’t have to add it, and your existing specs remain valid. 2. Cache and Redis Improvements Caching is a common performance layer in CRUD systems, but it becomes tricky when you combine Redis serialization and Hibernate’s lazy-loaded associations. This release includes two important caching-related improvements: The generator previously produced incorrect values for @Cacheable(value=...) in certain cases. That has been fixed, ensuring that cache names/values are generated consistently and correctly.Cache configuration has been updated to include HibernateLazyNullModule. This improves Redis caching behavior when Hibernate lazy-loaded entities are involved. In practice, it reduces the likelihood of serialization issues (or unexpected failures) when caching objects that contain lazily-loaded properties. If your generator output is used in services that rely on Redis for caching, this update should make caching more stable and predictable. 3. Spring Boot 3 and 4 Compatibility Another big part of this release is compatibility. The generator is now fully compatible with both Spring Boot 3 and Spring Boot 4. In ecosystems like Spring, small changes between major versions can break builds, plugins, or code generation assumptions. Ensuring compatibility across versions is essential for teams that want to upgrade gradually (or maintain multiple services on different baselines). v1.1.0 addresses compatibility issues so you can use the generator reliably on either Spring Boot line. 4. OSIV Control + EntityGraph Support Open Session In View (OSIV) is often considered an anti-pattern because it can hide N+1 query problems and produce unpredictable lazy-loading behavior in higher layers. Many teams disable it intentionally. In v1.1.0, the generator introduces a new configuration entry: additionalProperties.spring.jpa.open-in-view (default: false) With OSIV disabled by default, the generated resources now include EntityGraph support to handle lazy relations more safely. The goal here is practical: avoid surprising LazyInitializationException scenarios without relying on OSIV. This approach also nudges generated projects toward better data-fetching discipline. If your project still relies on OSIV, you can explicitly set it to true in the configuration. But the default aligns with a more defensive and production-friendly setup. 5. Improvements and Stability Beyond features, v1.1.0 includes a set of improvements that make the project easier to maintain and more robust: Documentation has been updated to reflect the new fields.validation structure.Tests have been updated and fixed so they work correctly with the new validation model.The generator internals were refactored to improve readability and maintainability.A validation edge case was fixed: when min/max are not provided, the generator no longer throws a NullPointerException (NPE). This prevents configuration mistakes from turning into runtime failures during generation. There is also an updated “full CRUD spec YAML” reference example in the repository. It existed before, but it has now been refreshed to include the new fields.validation configuration and the additionalProperties.spring.jpa.open-in-view setting. If you want to see the complete configuration surface area (not just a short snippet), that reference YAML is the best place to start. Updated CRUD Spec YAML (Short Example) The repository includes an updated full CRUD spec YAML reference example. Below is a shortened snippet that highlights the new and most important parts (validation + OSIV): YAML configuration: database: postgresql javaVersion: 21 springBootVersion: 4 cache: enabled: true type: REDIS expiration: 5 openApi: apiSpec: true additionalProperties: rest.basePath: /api/v1 spring.jpa.open-in-view: false entities: - name: ProductModel storageName: product_table fields: - name: id type: Long id: strategy: IDENTITY - name: name type: String column: nullable: false unique: true length: 10000 validation: required: true notBlank: true minLength: 10 maxLength: 10000 - name: price type: Integer column: nullable: false validation: required: true min: 1 max: 100 - name: users type: UserEntity relation: type: OneToMany fetch: LAZY joinColumn: product_id validation: required: true minItems: 1 maxItems: 10 - name: UserEntity storageName: user_table fields: - name: id type: Long id: strategy: IDENTITY - name: email type: String validation: required: true email: true - name: password type: String validation: required: true pattern: "^(?=.*[A-Za-z])(?=.*\\d)[A-Za-z\\d]{8,}$" Tip: Check the repository’s full CRUD spec YAML example to see the complete supported configuration surface Upgrade Notes fields.validation is optional — add it only where needed.spring.jpa.open-in-view defaults to false. If your project relies on OSIV behavior, explicitly set it to true. If you build Spring Boot services frequently and want to reduce repetitive CRUD boilerplate, feel free to try the generator and share feedback. If you find the project useful, I’d really appreciate a star on the repository — it helps a lot and keeps the momentum going. Repository: https://github.com/mzivkovicdev/spring-crud-generator. Development continues, and more improvements are on the way. Thanks for the support!

In this article, we will dive deep into actors, nonisolated methods, @MainActor and @GlobalActors, and the concept of actor reentrancy. We will also explore what happens behind the scenes in the Swift concurrency runtime, including jobs, executors, workers, and schedulers, so you can understand not just how to use these tools, but why they work the way they do. Whether you’re already using Swift’s async/await features or just starting to explore concurrency, this guide will give you a solid understanding of the mechanisms that keep your concurrent code safe and efficient. Actors and Isolation in Swift Concurrency If you’ve spent years working with Grand Central Dispatch (GCD), you already know the core problem: shared mutable state. When multiple threads can read and write the same data at the same time, you risk data races: inconsistent reads, lost updates, or crashes that only appear under heavy load. With GCD, we relied on discipline using serial queues or locks. But discipline fails. One forgotten .sync call and your correctness vanishes. Swift concurrency introduces Actors to make data-race freedom a language-level guarantee. Class vs. Struct vs. Actor Type Semantics Thread Safety Mutation Model Struct Value By-copy safe Explicit mutating Class Reference Unsafe by default Shared mutable state Actor Reference Data-race safe Serialized access Actors sit exactly where classes used to be, but with correctness guarantees. Actor Basics An actor is a reference type that protects its mutable state through isolation. Unlike a class, you cannot accidentally touch an actor’s internal state from multiple threads. Swift actor BankStore { private var balance: Int = 0 func deposit(_ amount: Int) { balance += amount } func withdraw(_ amount: Int) -> Bool { guard balance >= amount else { return false } balance -= amount return true } Key properties of actors: Reference semanticsOnly one task at a time can access actor-isolated stateExternal access requires await nonisolated: Opting Out of Isolation Sometimes you need functionality that doesn’t touch the actor’s state or needs to be callable synchronously. Use the nonisolated keyword for these “pure” utilities. Swift actor ImageCache { nonisolated static let maxItems = 100 nonisolated func cacheKey(for url: URL) -> String { url.absoluteString } } Rule of thumb: if it reads or writes actor state - it should not be nonisolated. The Actor Model: The Mailbox Mental Model Think of an actor as having a mailbox: Each actor has a queue of pending work.Messages (calls) are enqueued as tasks.The actor processes these one at a time. When you write await store.deposit(50), you aren’t calling a function in the traditional sense. You are sending a message to the actor and suspending your current thread until the actor finishes processing that message. This is why await is mandatory: the actor might be busy with someone else’s request. Working With @MainActor and Other @GlobalActors When building scalable iOS applications, managing shared state across isolated domains like UI components, network layers, and local caches becomes a complex puzzle. Swift simplifies this with @GlobalActor. A global actor is essentially a singleton actor. It allows you to isolate state and operations globally without needing to pass an actor reference around your entire dependency graph. The most famous of these is, of course, the @MainActor. The @MainActor is uniquely tied to the main thread. Anything marked with this attribute is guaranteed to execute on the main thread, making it the bedrock for all UI updates. Swift @MainActor final class FlashcardViewModel: ObservableObject { @Published var currentCard: Card? func loadNextCard() async { // Safe to update UI state directly; we are isolated to the MainActor. self.currentCard = await fetchCard() } } However, the power of global actors isn’t limited to the main thread. You can define your own global actors to serialize access to highly contested shared resources, such as a centralized local database or an aggressive retry policy manager. Swift @globalActor public actor SyncActor { public static let shared = SyncActor() } @SyncActor final class OfflineSyncManager { var pendingMutations: [Mutation] = [] func queue(mutation: Mutation) { pendingMutations.append(mutation) } } By annotating OfflineSyncManager with @SyncActor, you guarantee that all accesses to pendingMutations are serialized on that specific actor’s executor, completely eliminating data races from different parts of your app trying to queue offline changes simultaneously. Actor Reentrancy Explained If you’re coming from the world of Grand Central Dispatch (GCD) and DispatchQueue, actors require a fundamental mental shift. A serial dispatch queue executes tasks strictly one after another. If a task is running, nothing else can run on that queue until it finishes. Swift actors are different: they are reentrant. Reentrancy means that while an actor guarantees mutual exclusion for synchronous code execution (only one thread can be inside the actor at a time), it explicitly allows other tasks to interleave at suspension points. When an actor encounters an await, it suspends the current task. Crucially, it also gives up its lock on the executor. During this suspension, the actor is completely free to pick up and execute other pending tasks. Once the awaited operation finishes, the original task is scheduled to resume on the actor when it’s free again. This design prevents deadlocks. If actors weren’t reentrant, two actors awaiting each other would instantly freeze your application. However, reentrancy introduces its own subtle class of concurrency bugs. The Hidden Risks of Suspending Inside Actor Methods Because the actor unblocks during an await, the state of your actor before the await might not match the state after the await. This is the single biggest trap engineers fall into when adopting Swift concurrency. Imagine implementing a session manager that fetches a fresh authentication token. If multiple requests fail and trigger a token refresh simultaneously, you might accidentally fire off multiple network requests if you don’t account for reentrancy. Swift actor SessionManager { private var cachedToken: String? func getValidToken() async throws -> String { // 1. Check local state if let token = cachedToken { return token } // 2. Suspend! The actor is now free to process other calls to `getValidToken()` let freshToken = try await performNetworkRefresh() // 3. State mutation. // DANGER: If another task interleaved during step 2, we might overwrite a valid token, // or we just unnecessarily performed multiple network requests. self.cachedToken = freshToken return freshToken } } To protect against this, you must rethink how you handle in-flight asynchronous operations. Instead of caching just the result, you often need to cache the Task itself. Swift actor SessionManager { private var cachedToken: String? private var refreshTask: Task<String, Error>? func getValidToken() async throws -> String { if let token = cachedToken { return token } // Return the in-flight task if one exists if let existingTask = refreshTask { return try await existingTask.value } // Otherwise, create a new task and cache IT immediately let task = Task { let freshToken = try await performNetworkRefresh() self.cachedToken = freshToken self.refreshTask = nil // Clean up return freshToken } self.refreshTask = task return try await task.value } } Always remember: across an await, your actor’s state is completely unguarded. Inside the Swift Concurrency Runtime To truly master structured concurrency, we need to step out of the syntax and into the engine room. Swift’s concurrency model isn’t just syntactic sugar over GCD; it is a completely bespoke, highly optimized runtime built around a cooperative thread pool. Understanding Jobs In the Swift runtime, a Job is the fundamental unit of schedulable work. When you write an async function, the compiler breaks your function down into partial tasks or “continuations” split at every await keyword. Each of these segments is wrapped into a Job. When a task suspends, the current Job finishes. When the awaited result is ready, a new Job is enqueued to resume the remainder of the function. Jobs are lightweight, heavily optimized, and managed entirely by the Swift runtime. How Executors Work If Jobs are the work, Executors are the environments where the work is allowed to happen. An executor defines the execution semantics for a set of Jobs. Every actor has a serial executor. This executor acts as a funnel, ensuring that only one Job associated with that actor runs at any given microsecond. When you call an actor method, you are submitting a Job to that actor’s executor. Custom Serial Executors (Actor Level) In the first example, we create a MainQueueExecutor conforming to SerialExecutor. This is particularly useful when you have a legacy codebase heavily dependent on a specific DispatchQueue and you want to wrap that logic into a modern Actor. Swift final class MainQueueExecutor: SerialExecutor { func enqueue(_ job: consuming ExecutorJob) { let unownedJob = UnownedJob(job) let unownedExecutor = asUnownedSerialExecutor() DispatchQueue.main.async { unownedJob.runSynchronously(on: unownedExecutor) } } func asUnownedSerialExecutor() -> UnownedSerialExecutor { UnownedSerialExecutor(ordinary: self) } } @globalActor actor CustomGlobalActor: GlobalActor { static let sharedUnownedExecutor = MainQueueExecutor() static let shared = CustomGlobalActor() nonisolated var unownedExecutor: UnownedSerialExecutor { Self.sharedUnownedExecutor.asUnownedSerialExecutor() } } Task Executors (Task Level) While a SerialExecutor protects an actor’s state, a TaskExecutor influences the “ambient” environment where a task and its children run. It doesn’t provide serial isolation; it provides a preferred execution location. Swift final class MainQueueExecutor: TaskExecutor { func enqueue(_ job: consuming ExecutorJob) { let unownedJob = UnownedJob(job) self.enqueue(unownedJob) } func enqueue(_ job: UnownedJob) { let unownedExecutor = asUnownedTaskExecutor() DispatchQueue.main.async { job.runSynchronously(on: unownedExecutor) } } func asUnownedTaskExecutor() -> UnownedTaskExecutor { UnownedTaskExecutor(ordinary: self) } } let executor = MainQueueExecutor() Task.detached(executorPreference: executor) { // TODO: Perform an async operation } What Workers Do Executors don’t magically run code; they need CPU threads. This is where Workers come in. In Swift concurrency, there is a global, cooperative thread pool. The threads inside this pool are the “workers.” Unlike GCD, which can spawn hundreds of threads, leading to thread explosion and massive memory overhead, the Swift thread pool is strictly limited, generally to the number of active CPU cores. However, this isn’t a hard-and-fast rule; there are specific cases where the pool may spawn more threads. We took a deep dive into this behavior in the article Swift Concurrency: Part 1. Workers ask executors for Jobs. When a worker thread picks up a Job from an executor, it executes it until completion or suspension. Because the number of workers is limited, Swift enforces a strict rule: you must never use blocking APIs (like semaphores or synchronous network calls) inside an async context. If you block a worker thread, you are permanently stealing a core from the concurrency runtime. The Role of Schedulers The Scheduler is the invisible conductor orchestrating this entire process. It decides which Jobs sit on which Executors, and which Workers get assigned to process them. The scheduler is highly priority-aware. When you spawn a Task(priority: .userInitiated), the scheduler ensures the resulting job jumps ahead of background jobs in the queue. It handles the complex logic of priority inversion avoidance, waking up worker threads, and balancing the load across the CPU. Types of Executors and How They’re Chosen Swift utilizes different types of executors depending on the context of your code: The global concurrent executor: If your code is not isolated to any actor (e.g., a detached task or a standalone async function), it runs on the default global concurrent executor. This executor distributes Jobs freely across all available workers in the cooperative thread pool.The main actor executor: This is a specialized serial executor permanently bound to the application’s main thread. The scheduler ensures that any Job submitted here is handed off to the main runloop.Default serial executors: Every standard actor you create gets its own default serial executor. The runtime dynamically maps this executor to any available worker thread in the pool as needed.Custom executors (Swift 5.9+): Advanced use cases might require overriding how an actor executes its jobs. By implementing the SerialExecutor protocol, you can create custom executors, for instance, to force an actor to run its jobs on a specific, legacy DispatchQueue to interoperate with older C++ or Objective-C codebases seamlessly. How the Runtime Chooses an Executor Understanding that executors exist is one thing; predicting exactly where your code will run is another. When a Job is ready to execute, the Swift runtime evaluates a precise decision tree to route that workload. Here is the exact algorithm the runtime uses to select an executor: Is the method isolated? (i.e., is it bound to a specific actor?) No (Non-isolated): Is there a preferred Task executor? Yes: The task executes on the Preferred Task Executor.No: The task executes on the standard Global Concurrent Executor.Yes (Actor-isolated): Does the actor provide its own custom executor? Yes: The task executes strictly on the Actor’s Custom Executor.No: Does the current Task have a preferred executor? Yes: The task executes on the Preferred Task Executor (while still strictly upholding the actor’s serial isolation).No: The task executes on the Default Actor Executor. This cascading logic ensures that actors maintain their state safety while allowing developers to influence the underlying execution environment when necessary. Inspecting Your Context: The #isolation Macro When dealing with deep call stacks and complex async boundaries, you might lose track of your current execution context. Swift 5.10 introduced a brilliant diagnostic tool to solve this: the #isolation macro. This macro evaluates at compile time to capture the actor isolation of the current context. It returns an any Actor? representing the actor you are currently isolated to, or nil if you are executing concurrently. Swift func debugCurrentContext() { // Prints the instance of the actor (like MainActor), or "no isolation" print(#isolation ?? "no isolation") } Sprinkling this into your logging infrastructure is invaluable when debugging data races or verifying that a heavy computation isn’t accidentally blocking the @MainActor. Task Executors vs. Actor Executors With recent advancements in Swift Evolution (specifically SE-0417 and SE-0392), developers now have the unprecedented ability to provide custom executors. However, to wield this power safely, you must deeply understand the difference between the two primary executor protocols: TaskExecutor and ActorExecutor (via SerialExecutor). What is a Task Executor? A Task Executor governs the execution environment for a specific Task hierarchy. Crucially, a Task Executor is inherently concurrent. It represents a thread pool or a concurrent queue where multiple jobs can be processed simultaneously. When you assign a preferred Task Executor, you are telling the runtime, “Unless an actor says otherwise, run the asynchronous work for this task pool over here.” What is an Actor Executor? An Actor Executor (which conforms to the SerialExecutor protocol) governs the execution environment for a specific actor instance. Unlike a Task Executor, an Actor Executor is strictly serial. It processes one job at a time, enforcing the mutual exclusion that makes actors safe from data races. The Danger of Custom Implementations Understanding the concurrent nature of Task Executors and the serial nature of Actor Executors is not just trivia, it is a strict runtime invariant. If you decide to write a custom executor (for example, wrapping an old C++ thread pool or a specific Grand Central Dispatch queue), you carry the burden of upholding these invariants: If you implement a SerialExecutor for an actor, but your underlying implementation accidentally allows concurrent execution, you will break the actor’s state isolation and introduce impossible-to-reproduce data races.Conversely, if you implement a TaskExecutor but back it with a serial queue, you risk starving the cooperative thread pool and introducing unexpected deadlocks across your async task hierarchies. The compiler trusts you to maintain these semantic guarantees. If you break them, the concurrency model shatters. Conclusion Swift concurrency is more than syntactic sugar for asynchronous code. It is a carefully designed execution model that formalizes how work is scheduled, isolated, and resumed. Actors provide safety guarantees, but understanding reentrancy and executor behavior is what allows engineers to reason about concurrency with confidence. By understanding these low-level mechanics when an actor temporarily releases isolation and how the runtime schedules jobs across worker threads, you can build iOS applications that are not only performant but also resilient to the subtle concurrency bugs that once plagued asynchronous systems.

Image classification is now a key part of many applications. Whether you’re automating photo organization, filtering uploaded content, or enriching product catalogs with visual tags, knowing what’s in an image can be just as important as knowing what a user typed. For Java developers, the challenge is familiar: most computer vision examples live in Python notebooks, while the systems that actually need image classification run on the JVM. Bridging that gap usually means standing up a separate Python microservice, managing REST calls, and dealing with serialization overhead. That’s a lot of ceremony for what should be a single processing step. This tutorial will show you how to build an image classification pipeline in pure Java with Apache Camel and the Deep Java Library (DJL). We’ll cover watching folders for new images, running classification with a pre-trained ResNet model, tidying up the predictions into clean reports, and routing results to output files, all while leaning on those trusty Enterprise Integration Patterns you’re probably already familiar with. What You'll Learn By the time you’re done here, you’ll be comfortable with: Develop a file-based image classification pipeline using Apache Camel.Use a pre-trained ResNet image classification model via Camel’s DJL component.Understand the djl: URI syntax and model configuration for computer vision tasks in Apache Camel.Structure routes with content-based routing and multiple formatter beans.Run image classification locally using Java and Apache Camel, without external APIs or Python services. Frameworks Used Apache Camel Apache Camel is an awesome open-source integration framework built on Enterprise Integration Patterns. It has great components for connecting systems, moving data, and orchestrating workflows using declarative routes. In this project, we look at file ingestion, message transformation, content-based routing, bean integration, error handling, and output persistence. Deep Java Library (DJL) DJL is a deep learning framework for Java that is engine-agnostic. It provides a high-level API for inference, training, and serving deep learning models right on the JVM. We use the Camel-DJL component to load a pre-trained ResNet model from the DJL Model Zoo, run image classification inference inside the JVM, and return structured classification results. ResNet for Image Classification Residual Network (ResNet) is a deep convolutional neural network architecture that introduced skip connections to solve the vanishing gradient problem. The model we use here is pre-trained on the ImageNet dataset, which covers 1,000+ categories — animals, vehicles, everyday objects, food items, you name it. It strikes a nice balance between accuracy and inference speed for CPU-based classification. Project Structure Let's look at the project structure below: reStructuredText camel-image-classifier/ ├── src/main/java/com/example/imageclassifier/ │ ├── MainApp.java # Application entry point │ ├── routes/ │ │ └── ImageClassificationRoutes.java # Camel route for image processing │ └── processor/ │ ├── ClassificationsFormatter.java # Formats DJL Classifications output │ ├── MapResultsFormatter.java # Formats Map-based results │ └── FallbackFormatter.java # Handles unexpected outputs ├── src/main/resources/ │ └── application.properties # Camel configuration ├── data/ │ ├── input/ # Drop JPEG images here │ ├── output/ # Classification results (text files) │ └── classified/ # Processed images archive ├── gradle/wrapper/ # Gradle wrapper files ├── build.gradle # Project dependencies ├── settings.gradle # Gradle settings ├── gradlew.bat # Gradle wrapper script ├── README.md # Main documentation Gradle Dependencies build.gradle Groovy plugins { id 'java' id 'application' } group = 'com.example' version = '1.0.0' description = 'Image Classification with Apache Camel and DJL' java { toolchain { languageVersion = JavaLanguageVersion.of(21) } } application { mainClass = 'com.example.imageclassifier.MainApp' } repositories { mavenCentral() } dependencies { // Apache Camel implementation 'org.apache.camel:camel-core:4.4.0' implementation 'org.apache.camel:camel-main:4.4.0' implementation 'org.apache.camel:camel-file:4.4.0' implementation 'org.apache.camel:camel-djl:4.4.0' // DJL (Deep Java Library) for image classification implementation platform('ai.djl:bom:0.28.0') implementation 'ai.djl:api' // MXNet engine for image classification (used by Camel DJL component) implementation 'ai.djl.mxnet:mxnet-engine' implementation 'ai.djl.mxnet:mxnet-model-zoo' // Use CPU-only MXNet runtime for Windows runtimeOnly 'ai.djl.mxnet:mxnet-native-mkl:1.9.1:win-x86_64' // Logging implementation 'org.slf4j:slf4j-simple:2.0.9' } A few things to note here compared to a typical NLP setup. For image classification, we use the MXNet engine instead of PyTorch. MXNet’s model zoo ships with a well-tested ResNet model optimized for image classification, and the mxnet-native-mkl dependency gives you CPU-optimized native libraries via Intel MKL. The DJL BOM makes sure the versions are consistent across engines and models. Application Entry Point The application starts up using the MainApp class and starts Camel using Main: Java package com.example.imageclassifier; import com.example.imageclassifier.routes.ImageClassificationRoutes; import org.apache.camel.main.Main; public class MainApp { public static void main(String[] args) throws Exception { System.out.println("================================================="); System.out.println("Image Classification with Apache Camel and DJL"); System.out.println("================================================="); // Create and configure Camel Main Main main = new Main(); // Add routes main.configure().addRoutesBuilder(new ImageClassificationRoutes()); // Start Camel System.out.println("\nStarting Apache Camel..."); System.out.println("Watching folder: data/input"); System.out.println("Output folder: data/output"); System.out.println("Press Ctrl+C to stop\n"); main.run(); } } Image Classification Route The ImageClassificationRoutes.java is where the core logic is implemented using the Camel DJL component’s URI. It uses the “from” component for image ingestion (watches for JPEG files, processes them one at a time to extract the raw bytes, archives them with a timestamp), and uses a single “to” URI endpoint to run image classification using the DJL component URI. The route then dispatches to the right formatter using Camel’s content-based routing. ImageClassificationRoutes.java Java package com.example.imageclassifier.routes; import com.example.imageclassifier.processor.ClassificationsFormatter; import com.example.imageclassifier.processor.FallbackFormatter; import com.example.imageclassifier.processor.MapResultsFormatter; import org.apache.camel.builder.RouteBuilder; import java.io.File; import java.nio.file.Files; /** * Apache Camel routes for image classification. */ public class ImageClassificationRoutes extends RouteBuilder { @Override public void configure() throws Exception { // Route to process JPEG images from input folder from("file:data/input?include=.*\\.(jpg|jpeg|JPG|JPEG)&noop=false&move=../classified/${date:now:yyyyMMdd-HHmmss}-${file:name}") .routeId("image-classification-route") .log("Processing image: ${file:name}") // Read file into bytes so the DJL component can create an Image internally. .process(exchange -> { File imageFile = exchange.getIn().getBody(File.class); exchange.getIn().setBody(Files.readAllBytes(imageFile.toPath())); }) // Run inference via Camel DJL component. .to("djl:cv/image_classification?artifactId=ai.djl.mxnet:resnet:0.0.1") // Convert output to a text report using Camel choice/bean components. .choice() .when(body().isInstanceOf(ai.djl.modality.Classifications.class)) .bean(new ClassificationsFormatter(), "format") .when(body().isInstanceOf(java.util.Map.class)) .bean(new MapResultsFormatter(), "format") .otherwise() .bean(new FallbackFormatter(), "format") .end() .log("Inference done for ${file:name}") // Write results to output folder .to("file:data/output?fileName=${date:now:yyyyMMdd-HHmmss}-${file:name.noext}.txt") .log("Results saved to output folder"); } } Let’s break this down: Stage 1: File Ingestion Java from("file:data/input?include=.*\\.(jpg|jpeg|JPG|JPEG)&noop=false&move=../classified/${date:now:yyyyMMdd-HHmmss}-${file:name}") The from component watches the data/input/ folder for JPEG files. The regex pattern include=.*\\.(jpg|jpeg|JPG|JPEG) makes sure only image files get picked up. Once processed, each image is moved to data/classified/ with a timestamp prefix, which prevents reprocessing and provides a clean audit trail. Setting noop=false means the file is consumed (moved), not left in place. Stage 2: Image to Bytes Java .process(exchange -> { File imageFile = exchange.getIn().getBody(File.class); exchange.getIn().setBody(Files.readAllBytes(imageFile.toPath())); }) The DJL component expects the image as a byte[] so it can construct a DJL Image object internally. This inline processor reads the file into a byte array and replaces the message body with it. It’s a small but essential step; without it, the DJL component would receive a File reference instead of raw pixel data. Stage 3: DJL Inference Java .to("djl:cv/image_classification?artifactId=ai.djl.mxnet:resnet:0.0.1") This single line is the heart of the pipeline. Let’s unpack the URI: djl – The Camel DJL componentcv/image_classification – The computer vision task type (as opposed to nlp/sentiment_analysis used in NLP tasks)artifactId=ai.djl.mxnet:resnet:0.0.1 – Identifies the pre-trained ResNet model from DJL’s MXNet Model Zoo This single line replaces what would otherwise be hundreds of lines of model loading, image preprocessing, tensor conversion, and inference code. Stage 4: Content-Based Routing Java .choice() .when(body().isInstanceOf(ai.djl.modality.Classifications.class)) .bean(new ClassificationsFormatter(), "format") .when(body().isInstanceOf(java.util.Map.class)) .bean(new MapResultsFormatter(), "format") .otherwise() .bean(new FallbackFormatter(), "format") .end() Here’s something you’ll run into with image classification that you won’t see in the sentiment analysis setup: the DJL component can return different types depending on the engine and model version. Most of the time, you get a Classifications object, but some MXNet model variants hand back a Map<String, Float> instead. Rather than assuming one type and risking a ClassCastException in production, we use Camel’s Content-Based Router pattern to dispatch to the right formatter bean. The FallbackFormatter catches anything unexpected — so the pipeline never crashes silently. This is a classic Enterprise Integration Pattern, and it’s one of the biggest advantages of using Camel for ML pipelines. The routing logic is declarative, testable, and easy to extend. Formatter Beans ClassificationsFormatter.java This is the primary formatter, handling the standard Classifications output from DJL: Java package com.example.imageclassifier.processor; import ai.djl.modality.Classifications; import org.apache.camel.Exchange; import java.util.List; /** * Bean to format DJL Classifications object into a text report. */ public class ClassificationsFormatter { public String format(Classifications classifications, Exchange exchange) { StringBuilder sb = new StringBuilder(); String fileName = exchange.getIn().getHeader("CamelFileName", String.class); sb.append("Image: ").append(fileName).append('\n'); List<Classifications.Classification> topK = classifications.topK(5); if (!topK.isEmpty()) { Classifications.Classification top = topK.get(0); sb.append("Top Prediction: ").append(top.getClassName()) .append(" (Confidence: ").append(String.format("%.2f%%", top.getProbability() * 100)) .append(")\n\n"); } sb.append("Top 5 predictions:\n"); for (int i = 0; i < topK.size(); i++) { Classifications.Classification c = topK.get(i); sb.append(String.format("%d. %s: %.2f%%\n", i + 1, c.getClassName(), c.getProbability() * 100)); } return sb.toString(); } } The topK(5) call extracts the five most confident predictions. Each classification carries a class name (e.g., “golden retriever”) and a probability score. The formatter produces a clean, human-readable report with the top prediction highlighted and all five ranked below it. MapResultsFormatter.java Some MXNet model variants return results as a Map<String, Float> instead of a Classifications object. This formatter handles that case: Java package com.example.imageclassifier.processor; import org.apache.camel.Exchange; import java.util.ArrayList; import java.util.List; import java.util.Map; /** * Bean to format Map-based classification results into a text report. * Handles HashMap output from MXNet models. */ public class MapResultsFormatter { public String format(Map<String, Float> results, Exchange exchange) { StringBuilder sb = new StringBuilder(); String fileName = exchange.getIn().getHeader("CamelFileName", String.class); sb.append("Image: ").append(fileName).append('\n'); // Convert to sorted list by probability (descending) List<Map.Entry<String, Float>> sortedResults = new ArrayList<>(results.entrySet()); sortedResults.sort((a, b) -> Float.compare(b.getValue(), a.getValue())); // Get top 5 List<Map.Entry<String, Float>> top5 = sortedResults.subList(0, Math.min(5, sortedResults.size())); if (!top5.isEmpty()) { Map.Entry<String, Float> topEntry = top5.get(0); sb.append("Top Prediction: ").append(topEntry.getKey()) .append(" (Confidence: ").append(String.format("%.2f%%", topEntry.getValue() * 100)) .append(")\n\n"); } sb.append("Top 5 predictions:\n"); for (int i = 0; i < top5.size(); i++) { Map.Entry<String, Float> entry = top5.get(i); sb.append(String.format("%d. %s: %.2f%%\n", i + 1, entry.getKey(), entry.getValue() * 100)); } return sb.toString(); } } Since a Map has no inherent ordering, we sort the entries by value in descending order before pulling out the top 5. The output format mirrors ClassificationsFormatter exactly, so downstream consumers don’t need to care which formatter produced the report. FallbackFormatter.java In case of an unexpected output type, the FallbackFormatter makes sure the pipeline keeps producing meaningful output rather than crashing. This follows a critical production pattern - fail softly: Java package com.example.imageclassifier.processor; import org.apache.camel.Exchange; /** * Bean to format unexpected result types into a text report. */ public class FallbackFormatter { public String format(Object result, Exchange exchange) { StringBuilder sb = new StringBuilder(); String fileName = exchange.getIn().getHeader("CamelFileName", String.class); sb.append("Image: ").append(fileName).append('\n'); sb.append("Raw result type: ").append(result == null ? "null" : result.getClass().getName()).append('\n'); sb.append("Result:\n").append(String.valueOf(result)).append('\n'); return sb.toString(); } } How to Run the Application Build and run using Gradle: gradlew clean run. Then drop JPEG images into data/input/. For example, place a photo of a dog. The classification result is written to data/output/, and the original image is archived to data/classified/ with a timestamp. Example output: Plain Text Image: golden_retriever.jpg Top Prediction: golden retriever (Confidence: 95.67%) Top 5 predictions: 1. golden retriever: 95.67% 2. Labrador retriever: 2.34% 3. tennis ball: 1.12% 4. cocker spaniel: 0.45% 5. Irish setter: 0.23% The model recognizes 1,000+ ImageNet categories - animals, vehicles, everyday objects, food items, plants, and more. Sentiment Analysis vs. Image Classification: Side by Side If you read my previous article on building a sentiment analysis pipeline with Camel and DJL, you’ll notice a deliberate symmetry between the two projects. The table below highlights the key differences: Aspect Sentiment Analysis Image Classification DJL Task Type nlp/sentiment_analysis cv/image_classification Model DistilBERT (PyTorch) ResNet (MXNet) Input Text files (.txt) JPEG images (.jpg, .jpeg) Input Preprocessing Files.readString() → String Files.readAllBytes() → byte[] DJL Engine PyTorch MXNet Output Positive/Negative with confidence Top 5 category predictions Formatter Count 2 (Classifications + Fallback) 3 (Classifications + Map + Fallback) The core Camel route structure — file ingestion, DJL inference, content-based routing, and formatted output — is identical. That’s the power of the Camel + DJL integration: switching from NLP to computer vision is essentially a URI change and a different set of dependencies. The integration pattern stays the same. DJL Behind the Scenes On first execution, the ResNet model (~100MB) is downloaded automatically from the DJL Model Zoo, and MXNet native libraries are initialized. The model is cached locally under ~/.djl.ai/, so subsequent runs load from cache, making startup significantly faster. The DJL component handles all the heavy lifting internally: image decoding, resizing to the model’s expected input dimensions, tensor conversion, forward pass through the neural network, and softmax normalization of the output probabilities. You don’t write any of this code - the Camel DJL component abstracts it away entirely. Production Considerations For performance, always warm up the model on startup if latency is a concern. The first inference call triggers model loading and JIT compilation, which can take several seconds. Allocate sufficient JVM heap: image classification models are memory-intensive and typically require 500MB–1GB. Scale horizontally with multiple Camel instances watching different input directories, or vertically using GPU-enabled DJL engines. MXNet supports CUDA out of the box— swap the mxnet-native-mkl dependency for mxnet-native-cu* to enable GPU acceleration. The content-based router with a fallback formatter makes sure the pipeline doesn’t crash on unexpected model output. For production deployments, consider adding Camel’s onException handler for retries and dead-letter routing. And Camel’s built-in metrics and JMX support give you visibility into processing rates, error counts, and route performance, critical for production ML pipelines. Conclusion This tutorial demonstrates that computer vision doesn’t need to be a separate system. With Apache Camel and DJL, image classification becomes just another step in your integration flow — composable, observable, and production-ready. There’s no per-request API cost, image data stays on-premise, and you have full control over routing and error handling. Compared to calling external vision APIs (Google Vision, AWS Rekognition, Azure Computer Vision), you get zero network latency for inference, no data leaving your infrastructure, and predictable cost regardless of volume. Compared to standing up a Python Flask service with TensorFlow or PyTorch, you get native integration with enterprise Java systems and first-class support for Enterprise Integration Patterns. If you already use Camel, adding computer vision capabilities is no longer a leap. It’s a small, well-structured step.

In microservices, you’ve likely broken a cold sweat more than once when a request suddenly 'vanishes' the moment it hits a Database or a Message Broker. It is a true operational nightmare. However, with the release of Spring Boot 4 in early 2026, building a comprehensive Observability system has become easier than ever, thanks to the 'all-in' support from micrometer tracing. The Problem: "Anonymous" Queries When your database starts lagging (slow queries), you check the processlist in MySQL only to find a vague line: SELECT * FROM orders WHERE status = 'PENDING' ... At this point, the ultimate head-scratcher arises: "Who triggered this? Which API is executing this statement?" Without a Trace ID embedded directly into the query, you are guaranteed to spend hours digging through logs just to piece the two ends together. The Solution: "Pinning" Trace IDs Directly into SQL Comments With Spring Boot 4, we no longer need complex third-party libraries or clunky, "home-brewed" workarounds. Everything is now handled seamlessly through Spring Boot Actuator and Hibernate StatementInspector. The concept is simple: we attach the Trace ID directly to the SQL statement as a comment. When looking at the Database logs, you will know exactly where that request originated. Project Setup Let’s start by initializing a Spring Boot 4.0.2 project with the following structure: File: build.gradle To unlock the power of Observability, you will need to include these key dependencies in your configuration file: Groovy plugins { id 'java' id 'org.springframework.boot' version '4.0.2' id 'io.spring.dependency-management' version '1.1.7' } group = 'org.example' version = '0.0.1-SNAPSHOT' description = 'demo-trace' java { toolchain { languageVersion = JavaLanguageVersion.of(17) } } repositories { mavenCentral() } dependencies { implementation 'org.springframework.boot:spring-boot-starter-data-jpa' implementation 'org.springframework.boot:spring-boot-starter-web' implementation 'org.springframework.boot:spring-boot-starter-actuator' implementation 'io.micrometer:micrometer-tracing-bridge-otel' implementation 'com.mysql:mysql-connector-j' compileOnly 'org.projectlombok:lombok' annotationProcessor 'org.projectlombok:lombok' } tasks.named('test') { useJUnitPlatform() } Implementing the SQL Inspector Now, we will create a class that acts as a "gatekeeper" to intercept and modify every SQL statement just before it is sent to the Database. File: SqlCommentStatementInspector.java Here is how we use Hibernate's StatementInspector to automatically inject the Trace ID into your queries: Java package org.example.demotrace; import lombok.extern.slf4j.Slf4j; import org.hibernate.resource.jdbc.spi.StatementInspector; import org.slf4j.MDC; import java.net.InetAddress; @Slf4j public class SqlCommentStatementInspector implements StatementInspector { private static String HOST_NAME; static { try { HOST_NAME = InetAddress.getLocalHost().getHostName(); } catch (Exception e) { log.error("Cannot get local host name", e); HOST_NAME = "unknown-host"; } } @Override public String inspect(String sql) { // Elastic APM Agent auto add traceId vào MDC with key "traceId" String traceId = MDC.get("traceId"); if (traceId == null) traceId = "no-trace"; return sql + " /* host: " + HOST_NAME + "; traceId: " + traceId + " */"; } } To complete the process, we need a "bridge" to ensure the Trace ID is always available within the context of each request. Below is how we set up a Filter to manage this. Linking the Trace ID to MDC (Mapped Diagnostic Context) For the SqlCommentStatementInspector to accurately retrieve the Trace ID, we must ensure this information is pushed into the MDC. We will implement a standard Servlet Filter to handle this "identification" process the moment a request hits the system. File: TraceIdFilter.java This code snippet synchronizes the Trace ID from Micrometer into the Log context, ensuring that both your log files and SQL comments are "aligned under a single source of truth": Java package org.example.demotrace; import jakarta.servlet.*; import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletResponse; import org.slf4j.MDC; import org.springframework.stereotype.Component; import java.io.IOException; import java.util.UUID; @Component public class TraceIdFilter implements Filter { private static final String TRACE_ID_HEADER = "X-Trace-Id"; private static final String TRACE_ID_MDC_KEY = "traceId"; @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { HttpServletRequest httpRequest = (HttpServletRequest) request; HttpServletResponse httpResponse = (HttpServletResponse) response; // get trace from header or create String traceId = httpRequest.getHeader(TRACE_ID_HEADER); if (traceId == null || traceId.isEmpty()) { traceId = UUID.randomUUID().toString(); } MDC.put(TRACE_ID_MDC_KEY, traceId); httpResponse.setHeader(TRACE_ID_HEADER, traceId); try { chain.doFilter(request, response); } finally { // remove trace after done MDC.remove(TRACE_ID_MDC_KEY); } } } Hibernate Configuration To let Spring Boot know it should use the SqlCommentStatementInspector for every database transaction, you only need to declare a single line in your configuration file. File: application.properties Add the following line to your configuration file: Properties files spring.application.name=demo-trace spring.datasource.url=jdbc:mysql://mysql:3306/tracing_db?createDatabaseIfNotExist=true spring.datasource.username=root spring.datasource.password=root spring.jpa.hibernate.ddl-auto=update # Register statement_inspector spring.jpa.properties.hibernate.session_factory.statement_inspector=org.example.demotrace.SqlCommentStatementInspector spring.jpa.show-sql=true management.tracing.sampling.probability=1.0 logging.pattern.level=%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}] Test Run: Create a Data Query API We will create a UserController to simulate a real user request. When this API is called, Spring Boot 4 will automatically generate a Trace ID, pass it through the filter, attach it to the MDC, and finally embed it into the SQL query. File: UserController.java Java package org.example.demotrace.controller; import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; import org.example.demotrace.entity.User; import org.example.demotrace.repository.UserRepository; import org.springframework.web.bind.annotation.*; import java.util.List; @Slf4j @RestController @RequestMapping("/api/users") @RequiredArgsConstructor public class UserController { private final UserRepository userRepository; @PostMapping public User createUser(@RequestBody User user) { log.info("Request Success!"); User rs = userRepository.save(user); userRepository.findUserSlowly(rs.getId()); return rs; } @GetMapping public List<User> getAllUsers() { return userRepository.findAll(); } } Entity: User.java This is the structure of the data table we will be querying. You can use Lombok to keep the code clean and concise as shown below: Java package org.example.demotrace.entity; import jakarta.persistence.*; import lombok.Data; @Entity @Table(name = "users") @Data public class User { @Id @GeneratedValue(strategy = GenerationType.IDENTITY) private Long id; private String name; private String email; } Repository: UserRepository.java Implementing a simulated slow query to test tracing at the MySQL database layer. Java package org.example.demotrace.repository; import org.example.demotrace.entity.User; import org.springframework.data.jpa.repository.JpaRepository; import org.springframework.data.jpa.repository.Query; import org.springframework.data.repository.query.Param; import java.util.Optional; public interface UserRepository extends JpaRepository<User, Long> { @Query(value = "SELECT u.*, SLEEP(50000) FROM users u WHERE u.id = :id", nativeQuery = true) Optional<User> findUserSlowly(@Param("id") Long id); } Docker Compose and Dockerfile for Kibana APM Integration Below are the Docker Compose and Dockerfile configurations required to run the application and visualize tracing data within Kibana APM. File: docker-compose.yml YAML services: mysql: image: mysql:8.0 environment: MYSQL_ROOT_PASSWORD: root volumes: # Map file init vào container - ./init.sql:/docker-entrypoint-initdb.d/init.sql ports: - "3306:3306" healthcheck: test: ["CMD", "mysqladmin" ,"ping", "-h", "localhost"] timeout: 20s retries: 10 elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0 environment: - discovery.type=single-node - "ES_JAVA_OPTS=-Xms512m -Xmx512m" ports: - "9200:9200" apm-server: image: docker.elastic.co/apm/apm-server:7.17.0 depends_on: [elasticsearch] ports: ["8200:8200"] command: > apm-server -e -E output.elasticsearch.hosts=["elasticsearch:9200"] -E apm-server.host="0.0.0.0:8200" kibana: image: docker.elastic.co/kibana/kibana:7.17.0 depends_on: [elasticsearch] ports: ["5601:5601"] app: build: . dns: - 8.8.8.8 - 8.8.4.4 depends_on: mysql: condition: service_healthy apm-server: condition: service_started ports: - "8080:8080" Dockerfile: YAML # Stage 2: run (Runtime) FROM eclipse-temurin:17-jre-jammy WORKDIR /app # Copy file jar # (need build app from gradle local or ide) COPY build/libs/demo-trace-0.0.1-SNAPSHOT.jar app.jar # download agent apm ADD https://repo1.maven.org/maven2/co/elastic/apm/elastic-apm-agent/1.43.0/elastic-apm-agent-1.43.0.jar elastic-apm-agent.jar ENTRYPOINT ["java", \ "-javaagent:/app/elastic-apm-agent.jar", \ "-Delastic.apm.service_name=demo-trace-service", \ "-Delastic.apm.server_urls=http://apm-server:8200", \ "-Delastic.apm.application_packages=org.example.demotrace", \ "-Delastic.apm.enable_log_correlation=true", \ "-jar", "app.jar"] Monitoring and "Crushing" Slow Queries Now that the coding is finished, let's deploy the environment to verify our results. We will use Docker to simulate a complete, production-ready system. Deployment with Docker First, build your project (ensure you have JDK 17+ installed): ./gradlew clean build. Next, spin up the technology stack (including the App, MySQL, and Observability tools): docker compose up -d. "Tracing" in Action Imagine you receive an alert that the Database is hanging. You log into MySQL and run the command to inspect the currently executing processes: MySQL SELECT ID,USER,HOST,DB,COMMAND,TIME,STATE,INFO FROM information_schema.processlist WHERE COMMAND != 'Sleep' AND INFO IS NOT NULL ORDER BY TIME DESC; The result will look like this: Why Is This a "Lifesaver"? Identify the culprit: Looking at the Info column, you can immediately see the traceId=6794d2e1b....Backtrace with ease: Simply copy this Trace ID and paste it into your log management system (such as Grafana Loki or ELK). Instantly, you’ll uncover the request's entire journey: where it started, which user triggered it, and exactly why it’s lagging.Decisive action: If this query is hanging the system, you can confidently execute KILL 12 (the process ID) because you know exactly which feature it belongs to and what the impact of killing it will be. Lightning-Fast Backtracing This is the "money shot" — the most valuable part of the entire process. Once you’ve identified a "culprit" query in the database, finding its origin takes only a few seconds: Extract the trace: Copy the traceId from the INFO column in the MySQL SHOW PROCESSLIST output.Search on Kibana: Navigate to your Kibana dashboard (typically at http://localhost:5601).Paste and search: Paste the traceId into the search bar.The big reveal: Kibana will instantly display every log entry associated with that ID. You will discover: Which user was performing the action.Which service sent the request.The input parameters provided to that specific API.And even the preceding processing steps and how much time each one consumed. Application logs from the service environment: Every trace now provides end-to-end visibility, spanning from the initial user request, cutting through the application layer, and reaching down to the deepest database level. Leveling Up: Tracing Through CDC and Kafka Real-world systems don't just stop at the database. When you need to synchronize data across other services via change data capture (CDC) and Kafka, the Trace ID acts as a "Golden Thread" connecting every link in the chain. CDC (e.g., Debezium): When scanning the Database Binlog, the CDC capture process picks up the SQL content — including the comments containing the Trace ID we embedded. You can then extract this ID and include it in the Event Metadata.Kafka headers: Spring Boot 4 provides native support for context propagation. When Service A sends a message to Kafka, this identifier is automatically "injected" into the Kafka Header.Scalability: Service B (the Consumer) will automatically restore the context from that Header, continuing to log activities under the same unique Trace ID. Summary The synergy between Spring Boot 4, SQL Comment Tracing, and Kafka CDC creates an incredibly robust monitoring ecosystem: Transparency: You gain a crystal-clear understanding of the "origin story" behind every single database query.Loose coupling: You can freely scale and expand your services without the fear of requests "vanishing" or losing their trail.Performance: You can fully leverage Kafka's asynchronous processing power while maintaining comprehensive, end-to-end observability.

Consider the following two requests: JSON {} and { "email": null } They look similar, but they are fundamentally different. In the first case, nothing is said about email; in the second case, email is explicitly set to null. So, what should the system do? And how is that behavior defined today? Most systems cannot answer this precisely. They rely on conventions, mapping frameworks, or implicit behavior. This is not a limitation of JSON or JPA. It is the absence of a defined write model. The Core Problem A system that cannot distinguish between a field what was: Not sent Sent with null Sent with a value does not have a well-defined way to update data. Without this distinction, the system must interpret the request. And interpretation introduces ambiguity. A missing field may: Be ignoredOverwrite existing dataBe treated differently depending on the mapping layer None of this is visible in the API. The behavior exists, but it is not defined. A Real Failure Scenario Assume the current state is: name = "Anna"email = "[email protected]" A client sends: {"name":"Anna"} A common implementation looks like this: Java public void updateUser(UserDTO dto, User entity) { entity.setName(dto.getName()); entity.setEmail(dto.getEmail()); } What happens to email? If the mapper sets null → email is overwritten If the mapper ignores null → email is preserved If behavior changes later → the result changes The API does not define this. The framework does. A refactoring, a new mapper, or a configuration change may alter the outcome without changing the request. This is not a client error; it is undefined behavior. Common Approaches Several approaches try to address partial updates: Full updates (PUT): Require complete objects Partial updates (PATCH): Allow partial structures JSON merge patch: Merges values into existing state Mapping frameworks: Copy values into entities All of these operate on structure; none of them define intent. They describe what data looks like, not what the client meant. A Simple Principle To achieve correctness, the system must follow a simple rule: Only explicit intent may change state. If the client does not express intent, the system must not infer it. This leads to three cases: Field not present → no intent → no change Field present with value → updateField present with null → clear The distinction is not in the value itself, but in whether the field was present. Making Intent Explicit To implement this, the system must track presence per field. A minimal example: Java public class UserDTO { private String email; private boolean emailPresent; public void setEmail(String email) { this.email = email; this.emailPresent = true; } public boolean isEmailPresent() { return emailPresent; } public String getEmail() { return email; } } Now the backend can apply changes explicitly: Java public void updateUser(UserDTO dto, User entity) { if (dto.isEmailPresent()) { entity.setEmail(dto.getEmail()); } } The behavior is now defined: Not present → no changePresent null → clearPresent value → update No interpretation is required. Where It Breaks First Booleans: {"active":false} Is false an explicit value, or just a default? Without presence tracking, the system cannot tell. Collections: {"roles":[]} Does this mean: Clear all rolesOr ignore roles With explicit presence: Java if (dto.isRolesPresent()) { entity.getRoles().clear(); entity.getRoles().addAll(dto.getRoles()); } The meaning is clear: omitted → no change[] → clear[x,y] → replace These are not edge cases; they are normal updates. Structural Consequence The backend does not detect client mistakes. It cannot know whether a field was omitted intentionally or by accident. What reaches the backend is the only expressed intent. The system can: Validate valuesEnforce invariantsReject invalid states But it must not guess missing intent. A system that treats missing data as instructions is making decisions on behalf of the client. This is where incorrect writes begin. What This Solves With explicit intent: Missing fields never overwrite existing data Null values are consistently interpreted as clear Collections behave predictablyUpdates are stable across refactoring and framework changes Most importantly: The system no longer interprets requests; it executes them. What This Does NOT Solve This approach does not address: Business validationConcurrency controlClient-side errorsDefault values Those concerns remain in the domain and persistence layers. This pattern addresses only one problem: ambiguity in write intent. Trade-Offs This approach introduces additional structure. It requires: Tracking presence per field or segment Explicit update logic in the backend More code than direct mapping In other words, it adds boilerplate. But the trade-off is clear: No implicit behaviorNo ambiguityNo unintended data loss Conclusion There are multiple ways to interpret partial updates. A system can: Ignore missing fieldsReject incomplete requestsApply defaultsAttempt to merge state Each choice has consequences. This approach makes one decision explicit: The system does not guess what the client meant; it only acts on what is explicitly sent. If a field is missing, no intent was expressed, and without intent, nothing should change. The question is not how your API handles updates; the question is whether that behavior is defined or left to chance.

Monitoring Spring Boot Applications with Prometheus and Grafana Spring Boot’s Actuator and Micrometer provide rich metrics that can be scraped by Prometheus and visualized in Grafana. This guide covers configuring a Spring Boot application to expose Prometheus-formatted metrics, writing custom metrics, and setting up Prometheus and Grafana for monitoring. We cover installing Prometheus, writing a configuration to scrape your application, importing Grafana dashboards, and crafting PromQL queries and alerting rules. We also discuss Prometheus best practices, including metric naming conventions, label cardinality, and retention settings. Security considerations, troubleshooting tips, and the performance impact of metrics collection are also included. The diagram below illustrates a typical monitoring architecture. Figure: Data flow for Spring Boot monitoring. Prometheus scrapes metrics from the Spring Boot /actuator/prometheus endpoint and stores time-series data. Grafana queries Prometheus for visualization, while Alertmanager handles notifications. Spring Boot Actuator and Micrometer Setup Spring Boot auto-configures Micrometer when the micrometer-registry-prometheus dependency is on the classpath. Include these dependencies in your build: Maven (pom.xml) XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> </dependency> Gradle (build.gradle) Groovy implementation 'org.springframework.boot:spring-boot-starter-actuator' implementation 'io.micrometer:micrometer-registry-prometheus' These enable Actuator and the Prometheus registry. Next, configure Actuator to expose the Prometheus endpoint. By default, only the health endpoint is exposed. To enable others: Properties Plain Text management.endpoint.prometheus.enabled=true management.endpoints.web.exposure.include=prometheus The first property enables the /actuator/prometheus endpoint; the second exposes it over HTTP. Be cautious: exposing all endpoints can leak sensitive information. For production: Properties Plain Text management.endpoints.web.exposure.include=health,prometheus management.endpoints.web.exposure.exclude=env,beans Also secure /actuator via Spring Security or network ACLs. Verify metrics by visiting: Plain Text http://<host>:<port>/actuator/prometheus You should see Prometheus-formatted output like: Plain Text jvm_memory_used_bytes{area="heap",id="PS Eden Space"} 1.2345E7 http_server_requests_seconds_count{exception="None",method="GET",status="200",uri="/hello"} 42.0 These include JVM and HTTP metrics. Sample Spring Boot Metrics Code Below is a simple Spring Boot application with a REST endpoint and custom Micrometer metrics: Java @SpringBootApplication public class DemoApplication { public static void main(String[] args) { SpringApplication.run(DemoApplication.class, args); } } @RestController public class HelloController { private final Counter helloCounter; public HelloController(MeterRegistry registry) { // Define a custom counter with tags this.helloCounter = Counter.builder("custom_hello_requests_total") .description("Total number of /hello requests") .tags("env", "demo") .register(registry); } @GetMapping("/hello") public String hello() { // Increment counter on each request helloCounter.increment(); return "Hello from Spring Boot!"; } } Micrometer automatically exposes these metrics, along with built-in ones like jvm_memory_* and http_server_requests_*. Prometheus Setup Installing Prometheus Native: Download from the official site, extract, and run ./prometheus (default port: 9090).Docker: Bash Plain Text docker pull prom/prometheus docker run --name prometheus -d -p 9090:9090 prom/Prometheus To use a custom configuration: Bash YAML docker run --name prometheus -d -p 9090:9090 \ -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml \ prom/prometheus prometheus.yml Configuration YAML global: scrape_interval: 15s scrape_configs: - job_name: 'springboot-app' metrics_path: '/actuator/prometheus' scrape_interval: 15s static_configs: - targets: ['localhost:8080'] In this example Prometheus scrapes the /actuator/prometheus endpoint on localhost:8080 every 15 seconds. Adjust targets for your actual hostnames/IPs. For containerized or Kubernetes environments Prometheus can use service discovery. Grafana Setup Grafana reads data from Prometheus and renders dashboards. Install Grafana: Download from grafana.com and runAdd Prometheus Data Source: In Grafana UI go to Configuration -Data Sources-Add data source, choose Prometheus and set URL to http://localhost:9090 . Click Save & Test to confirm connection.Import a Dashboard: Grafana can import community dashboards. For Spring Boot, use the Spring Boot Statistics dashboard. In Grafana, go Dashboards- Import then paste the dashboard ID or JSON. The steps are click Dashboards – Import upload JSON or paste URL/ID. Choose your Prometheus data source when prompted.Create Custom Panels: we can also build your own dashboards. Common panels are JVM memory: jvm_memory_used_bytes{area="heap"} / jvm_memory_max_bytes{area="heap"} Threads: jvm_threads_live GC Activity: rate(jvm_gc_pause_seconds) Metric Naming and Label Best Practices Prometheus and Micrometer follow a naming convention to ensure clarity. Metric names should: Use snake_case describe one thing, and include a suffix for units. For example, http_request_duration_seconds, process_cpu_seconds_total, data_pipeline_last_record_processed_timestamp_seconds.Include an application or domain prefix when useful though commonly used metrics (like JVM or process metrics) often lack a custom prefix. Prometheus and Grafana Best Practices Cardinality: Reduce label explosion. For example avoid including full URLs with query strings or user specific labels on metrics. Keep label sets small and enumerable.Retention: Prometheus by default retains 15 days of data. You can change with flags: e.g. storage.tsdb.retention.time=30d. Alternatively use storage.tsdb.retention.size to cap disk usage. Security Considerations Protect Actuator: Never expose /actuator/prometheus or other management endpoints to the public internet without security. Restrict to a monitoring subnet or secure with Spring SecurityHTTPS: Use TLS for Grafana and Prometheus web UIs if reachable over the network. For example run Grafana behind an HTTPS proxy or enable its built in TLS support.Authentication: Enable authentication on Grafana. For Prometheus you may run it behind an authenticated proxy or use basic authNetwork: Consider placing monitoring tools in a VPN or private network segment. Do not open 9090/3000 publiclyAlerting secrets: In Alertmanager configs, protect credentials. Performance Implications Instrumenting with micrometer has minimal overhead on the application side. Prometheus scraping and storage is where costs accrue Summary of Key Information Dependencies & Config-Add spring boot starter actuator and micrometer registry prometheus enable management endpoint prometheus and expose it.Prometheus setup run prometheus and configure prometheus.yml to scrape your app at /actuator/prometheusGrafana dashboards-Add Prometheus as a data source. Import or create dashboardsMetric naming: Use descriptive names with units avoid redundant labels and avoid high cardinality labels.Metric types: Counters, Gauges, Histograms/Summaries. Prefer histograms for cluster quantiles.Alerts: Define Prometheus alerting rules for conditions like instance down or high error rate. Example expressions: rate(http_server_requests_seconds_count{status=~"5.."}[5m]) Conclusion By combining Spring Boot Actuator, Micrometer with Prometheus and Grafana. We will gain end to end monitoring of your Java applications. You expose rich application metrics via the /actuator/prometheus endpoint configure Prometheus to scrape them and build Grafana dashboards to visualize system health. This setup supports alerting so you can be notified of issues like high error rates or downtime. Careful attention to metric naming label cardinality and data retention ensures scalable high performance monitoring. Following best practices and using community dashboards will help your monitoring solution be effective and maintainable.

We had hundreds of microservices. Thousands of enterprise customers. And alerts firing constantly — CPU at 80%, memory at 75%, disk at 60%. Engineers were drowning in noise, and still, every few weeks, a customer would open a ticket before we knew anything was wrong. The problem wasn't a lack of monitoring. It was a lack of structure. After years of running large-scale cloud platforms, I built a top-down, five-layer monitoring framework that changed how my team operated. This article walks through how it works, why it works, and how you can start adopting it without a big-bang overhaul. The Core Problem With Most Observability Setups Here's the typical pattern I see: teams instrument what's easy — CPU, memory, disk, request count — and then wonder why they're constantly chasing false alarms while real customer issues go undetected. The root cause is that there's no hierarchy. Your infrastructure metrics don't know about your business SLOs. Your service health dashboards don't connect to your capacity model. Everything is siloed, and when something breaks, engineers manually trace across six dashboards to find the actual problem. What's missing is explicit traceability — a clear chain from customer pain all the way down to infrastructure, so any engineer at any layer can navigate up and down without guesswork. The Five-Layer Framework The framework organizes monitoring into five explicit layers, each with a defined scope and clear connections to the layers above and below it. Plain Text Layer 1: Business Transactions ← What customers actually experience Layer 2: Service Health ← How your services are performing Layer 3: Pod Behavior ← How individual containers are behaving Layer 4: Data Service Performance ← How your databases and caches are doing Layer 5: Capacity Planning ← Are you running out of headroom? The key design principle: alerts fire at Layer 1. Investigation flows downward. You start from customer pain, not from infrastructure noise. Layer 1: Business Transactions — The Source of Truth This is the most important layer, and the most commonly missing one. Layer 1 metrics answer one question: Are customers being affected right now? Examples: Transaction error rate by workflow typeSession availability percentageP99 latency for top customer-facing operationsBusiness-critical operation success rate Why alert here and not on CPU? A CPU alert at 80% fires constantly in a healthy system under normal load. A transaction error rate alert at 1% fires only when customers are actually affected. One of these matters, the other creates on-call fatigue. SQL # Error rate by workflow label — fires when customers are hurting sum(rate(http_requests_total{status=~"5..", workflow!=""}[5m])) by (workflow) / sum(rate(http_requests_total{workflow!=""}[5m])) by (workflow) The workflow label here is critical. It groups requests by business function — not by service, not by pod, but by what the customer is actually trying to do. This is what makes cross-service error aggregation possible. Layer 2: Service Health — Where Investigation Starts When a Layer 1 alert fires, the first question is: which service is responsible? Layer 2 gives you the answer. This layer tracks the health of each individual service using the RED method (Rate, Errors, Duration): Request rate: Is traffic normal?Error rate: Is this service returning errors?Duration: Is this service slow? SQL # Service-level error rate sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m])) by (service) / sum(rate(http_server_requests_seconds_count[5m])) by (service) Layer 2 is also the best place to start adoption (more on this below). It gives you immediate operational value — you know which services are unhealthy — without requiring full instrumentation of all five layers. Layer 3: Pod Behavior — When the Service Looks Fine but Isn't Sometimes a service reports healthy aggregate metrics, but individual pods are struggling. This layer catches those cases. Layer 3 applies the USE method (Utilization, Saturation, Errors) at the pod level: Utilization: Is this pod close to its resource limits?Saturation: Is it queuing work it can't handle?Errors: Are individual pod errors being masked by healthy pods? Saturation metrics are often better early-warning signals than utilization: SQL # Thread pool queue depth — saturation indicator tomcat_threads_busy_threads / tomcat_threads_config_max_threads A pod at 75% CPU utilization might be fine. A pod with its thread pool queue at 90% capacity is about to drop requests, and you want to know that before it happens. Layer 4: Data Service Performance — The Hidden Bottleneck In most distributed systems, database and cache performance is where latency problems actually live. This layer monitors your databases, caches, and message queues using the same USE methodology. Key signals: Connection pool exhaustion (saturation)Query latency by operation typeCache hit rateGC pause time (often the most undermonitored metric) SQL # Connection pool saturation hikaricp_connections_active / hikaricp_connections_max GC pause time deserves special attention. Long GC pauses cause latency spikes that look like application slowness but are actually JVM behavior. Without Layer 4, you'll spend hours debugging your application code when the fix is a heap size adjustment. Layer 5: Capacity Planning — Getting Ahead of Problems This layer is about the future. While Layers 1–4 tell you what's happening now, Layer 5 tells you what's coming. The key insight: business metrics drive capacity needs. If your Layer 1 metrics show that customer transaction volume is growing 15% month-over-month, you can project when your current infrastructure will saturate — before it does. Layer 5 connects business growth metrics to infrastructure headroom: SQL # Days until connection pool exhaustion at current growth rate (hikaricp_connections_max - hikaricp_connections_active) / deriv(hikaricp_connections_active[7d]) / 86400 This kind of metric transforms capacity planning from a reactive scramble into a scheduled, predictable activity. The High-Cardinality Problem You Need to Avoid One of the most common Prometheus mistakes I've seen: putting user IDs, session tokens, or dynamic URL paths into metric labels. SQL # DO NOT DO THIS http_requests_total{user_id="usr_12345", url="/api/v1/query/abc123"} High-cardinality labels cause Prometheus performance to degrade severely — each unique label combination creates a separate time series. With millions of users or dynamic URLs, you'll bring your Prometheus instance to its knees. The rule: high-cardinality analysis belongs in your logging layer, not your metrics layer. Keep metric labels to a small, bounded set of values — service name, workflow type, environment, status code. If you need to debug a specific user session, go to your logs. How to Adopt This Without Starting Over You don't need to instrument all five layers at once. Here's the sequence that delivers value at each step: Step 1: Start With Layer 2 (Service Health) This gives you immediate value — you know which services are healthy and which aren't. Most teams already have some of this instrumentation; structure what you have into a consistent RED dashboard. Step 2: Add Layer 1 (Business Transactions) Define your customer-facing workflows and instrument them. Move your primary alerts here. This is when on-call noise drops dramatically. Step 3: Build Downward (Layers 3–5) Add pod behavior monitoring, then data service monitoring, then capacity planning. Each layer makes the one above it easier to debug. The framework delivers operational value at each step — you're not waiting for a big-bang implementation before anything is useful. What This Looks Like in Practice Here's a real incident pattern this framework resolved: Symptom: Layer 1: alert fires — transaction error rate for the report-generation workflow exceeds 2%.Layer 2: report-service shows elevated error rate. Other services healthy.Layer 3: Two of five report-service pods show thread pool saturation above 90%. The other three look fine.Layer 4: Database connection pool for the reporting DB is at 95% capacity. Root cause: A new query introduced in last week's release had a higher connection hold time than expected. Under normal load, the connection pool held. Under peak load, two pods exhausted their connections, causing errors that surfaced as customer-visible failures. Fix: Increase connection pool size, optimize query connection hold time, add saturation alert at Layer 4. Without the framework, this would have been a multi-hour investigation across disconnected dashboards. With it, the trace from customer pain to root cause took under 20 minutes. Key Takeaways Structure your monitoring into explicit layers — business transactions, service health, pod behavior, data service performance, and capacity planning. Each layer has a defined scope and connects to the layers above and below it.Alert on Layer 1 customer pain metrics, not infrastructure thresholds. CPU at 80% is noise. Transaction error rate at 1% is signal.Apply the USE method consistently across Layers 3 and 4 — utilization, saturation, and errors give you a shared vocabulary that makes cross-team debugging faster.Keep metric labels low-cardinality. High-cardinality labels like user IDs and dynamic URLs belong in logs, not metrics.Start with Layer 2, then Layer 1, then build downward. You don't need all five layers on day one — each step delivers value on its own.

Successful HTTP requests have become a deceptively comforting metric in modern web systems. Dashboards show low latency, the network tab fills with green entries and the backend reports clean 2xx rates, yet users experience empty screens, contradictory state, stuck workflows or data that appears to randomly revert. This failure mode is common in Angular applications because the transport layer can succeed while the application layer has already violated a business contract and Angular’s default HTTP and reactive ergonomics are optimized around HTTP-level success versus domain-level correctness. How Angular Treats 200 as Success Angular’s HTTP layer is intentionally aligned with HTTP semantics a request is represented as an Observable and failures in the HTTP layer are emitted on the Observable error channel. Angular documents three broad categories of request failure network/connection failure, timeout and backend error responses and states that HttpClient captures these errors as an HttpErrorResponse returned through the Observable’s error channel. When an API responds with a non success HTTP status, the error channel is used and HttpErrorResponse provides the HTTP layer context. This design becomes a trap when a backend returns 200 for a domain failure by embedding an error in the payload. In that scenario, Angular observes no HTTP failure, so the Observable emits on the success path. Any code that assumes failures arrive only as HttpErrorResponse or that relies on catchError placed near the HTTP call to absorb failures will miss the problem entirely because nothing in the HTTP layer is wrong. Angular’s interceptor model is the correct leverage point for addressing this mismatch because interceptors can transform the response stream and can implement cross-cutting policies over requests and responses. Angular describes interceptors as functions that form a chain and can influence the overall flow of requests and responses, including customizing response parsing, caching behavior, measuring response times, and driving UI state such as loading indicators. This is relevant because domain validity is effectively custom parsing of the response body, it is an interpretation step that belongs at the boundary. Converting Semantic Failure into a Real Error Signal Eliminating “200-with-error-body” at the source is the most robust fix. Guidance on REST error behavior stresses using HTTP status codes and mapping errors cleanly to standards based codes so clients can consume and act on outcomes consistently. Standardized error payloads reduce ambiguity further. RFCs published through the standards process of the Internet Engineering Task Force define Problem Details for HTTP APIs, a machine readable format intended to avoid bespoke error response formats and provide consistent error information. In many environments, changing backend status-code behavior is slow, and Angular must handle the reality of mixed semantics during migrations. A practical client-side approach is to normalize responses into one internal contract and throw domain errors when the payload indicates failure, even if the HTTP status is 200. This can be expressed without introducing boilerplate classes by using a narrow envelope type and validating it at the edge: TypeScript type ApiEnvelope<T> = | { ok: true; data: T } | { ok: false; error: { code: string; message: string } }; function unwrapOrThrow<T>(raw: unknown): T { const env = raw as Partial<ApiEnvelope<T>>; if (env && env.ok === true && 'data' in env) return env.data as T; const err = (env as any)?.error; const code = typeof err?.code === 'string' ? err.code : 'UNKNOWN'; const message = typeof err?.message === 'string' ? err.message : 'Domain failure with HTTP 200'; throw new Error(`${code}: ${message}`); } The key is that the exception is thrown inside the reactive pipeline. RxJS treats a thrown exception from an operator such as map as an error notification, making semantic failure indistinguishable from other failures to downstream logic. The catchError operator is explicitly defined to listen to the error channel and map errors to a new observable, making it a suitable mechanism for converting such failures into fallback UI state, retries or telemetry. This normalization can be applied centrally through an interceptor so individual services do not replicate the same checks. Angular’s interceptor documentation shows response interception by inspecting response events in the stream and acting on them. A domain-validation interceptor can keep the HTTP transport intact while enforcing business meaning: TypeScript export function domainEnvelopeInterceptor(req, next) { return next(req).pipe( map((event) => { if (event.type !== HttpEventType.Response) return event; const body = event.body; if (body && body.ok === false) { const code = body.error?.code ?? 'UNKNOWN'; const message = body.error?.message ?? 'Domain failure with HTTP 200'; throw new Error(`${code}: ${message}`); } return event; }) ); } This approach preserves the ergonomics of HTTP-based error handling while acknowledging that HTTP 200 does not communicate domain success. It also creates a single place to migrate behavior toward standards-based error responses, including RFC-style problem details, once backend endpoints evolve. Preventing RxJS state corruption from “successful” bad data Angular applications frequently compose HTTP Observables into longer-lived streams that back components, route resolvers and shared state stores. The most expensive failures in this space are not exceptions, they are stable-looking streams that carry incorrect state. A 200 response with silent contract drift can populate application state with values that satisfy TypeScript’s compile-time types but violate runtime invariants. Angular’s own HTTP guidance emphasizes inspecting the response to identify the error cause and using RxJS operators such as catchError and retry operators to manage failures. That guidance becomes more effective when failure includes semantic violations, not only non-2xx outcomes. A service method can defensively validate invariants in-stream and downgrade failures to an explicit UI state rather than allowing partial data to poison downstream logic: TypeScript loadAccountSummary(accountId: string) { return this.http.get(`/api/accounts/${accountId}/summary`).pipe( map(unwrapOrThrow), map((summary) => { if (summary.balance == null || Number.isNaN(summary.balance)) { throw new Error('INVALID_SUMMARY: balance missing or not numeric'); } return summary; }), catchError((err) => of({ state: 'error', reason: String(err?.message ?? err) })) ); } This approach ensures that downstream consumers receive either validated data or an explicit error state, rather than receiving a successful emission that forces templates and components to implicitly handle undefined behavior. The grounding here is RxJS’s contract catchError maps error notifications to a replacement observable and forwards other events unchanged so throwing in map produces a consistent and catchable failure signal. Caching amplifies semantic failures. In Angular, shareReplay is often used to memoize HTTP results so multiple subscribers do not trigger multiple network calls. The operator’s own implementation documentation states that a successfully completed source will stay cached in the shareReplayed observable forever and further describes reference counting behavior, including that the default configuration does not unsubscribe the source when the reference count drops to zero. HTTP calls complete after a single response so a single successful but invalid payload can become a permanent cached truth for the session. For that reason, validation must occur before caching, and caching configuration must be deliberate: TypeScript this.summary$ = this.http.get('/api/summary').pipe( map(unwrapOrThrow), map((v) => { if (!v.timestamp) throw new Error('INVALID_SUMMARY: missing timestamp'); return v; }), retry({ count: 2 }), shareReplay({ bufferSize: 1, refCount: true }) ); The validation ensures that only semantically valid summaries are ever eligible for being replayed and enabling refCount aligns with the operator’s documented behavior where dropping subscribers can lead to a new subscription and a new cache when a later subscriber arrives. The retry operator is mentioned in Angular’s own HTTP guidance as a strategy for transient failures and becomes equally relevant after semantic failures are modeled as errors in the stream. Making semantic failure visible to operations When semantic failures are treated as successful HTTP outcomes, observability systems that key off HTTP status codes and backend exception rates will remain green. Angular’s interceptor guidance explicitly calls out response-time measurement and logging as canonical interceptor use cases, reinforcing the principle that cross-cutting telemetry belongs at the HTTP boundary. Once semantic validation is expressed as actual stream errors, it can be logged, counted and traced with the same primitives used for network failures. Client-side telemetry is increasingly implemented through OpenTelemetry. The OpenTelemetry JavaScript documentation describes generating and collecting telemetry data such as metrics, logs and traces in both Node.js and the browser while also warning that browser client instrumentation is experimental and still evolving. Its browser getting-started documentation shows the use of a zone-based context manager (@opentelemetry/context-zone) for asynchronous context propagation, matching the execution model common in Angular applications. A pragmatic pattern is to record a custom event or span annotation when domain validation fails, keyed by endpoint, contract version and error code, while still surfacing an appropriate UI fallback. This can be performed inside the interceptor that throws the error, ensuring every domain failure is observable even if it is later recovered through catchError to keep the UI responsive. The end result is that operational dashboards stop equating “no 5xx” with “no user impact” and begin tracking contract violations as a first-class signal. Conclusion HTTP 200 confirms that a message was successfully carried across the network and processed at the transport layer but it says nothing about whether the payload preserves domain meaning, user intent or application invariants and it is even heuristically cacheable in ways that can preserve incorrect state. Angular’s HttpClient and its Observable-based error channel correctly model HTTP-layer failures, but semantic failures returned inside 200 responses bypass that channel and therefore bypass conventional error handling unless domain validation is explicitly introduced. The reliable remedy is to treat response bodies as untrusted until validated, convert domain failures into real stream errors through centralized interceptors and runtime checks, validate before caching with shareReplay and instrument semantic failures so observability tracks user-impacting correctness rather than only transport success.

Building real-time applications means balancing user responsiveness with heavy backend processing. A proven solution is to decouple heavy workloads using events and asynchronous processing. In this approach, a Spring Boot application quickly publishes events to Kafka instead of processing requests inline. Then Kafka consumers (with AI/ML logic) handle the data in the background, and the results are pushed to clients in real time via WebSockets. This article highlights three key patterns enabling this architecture: Event Production with Spring Boot and KafkaAI-Driven Processing in Kafka ConsumersReal-Time WebSocket Delivery to the Frontend Event Production with Spring Boot and Kafka The first step is capturing an event and publishing it to Kafka. By offloading work to Kafka the application can respond immediately to the user without waiting for processing. Spring Boot’s integration with Apache Kafka provides a KafkaTemplate to send messages to topics. A Spring Boot REST controller might receive a request create an Event object from the payload and use an EventProducer service to send it to a Kafka topic. The controller then returns an HTTP 200 response while the event is queued for processing. Plain Text @Service public class EventProducer { private final KafkaTemplate<String, Event> kafkaTemplate; @Value("${app.topic.name}") private String topicName; public void sendEvent(Event event) { kafkaTemplate.send(topicName, event); } } Here Event is a custom payload class carrying the request data. Publishing to Kafka instead of handling logic immediately achieves loose coupling. The producer does not need to know who will consume the event or how it will be processed. AI-Driven Processing in Kafka Consumers Once events are in Kafka consumer service can process them asynchronously. This is where we introduce AI-driven analysis. Keeping ML logic out of the request thread ensures we don’t slow down user interactions. Instead a consumer pulls events from Kafka and performs inference, enrichment or anomaly detection on each event. Plain Text @Service public class AiConsumerService { private final AIService aiService; private final UpdateSocketHandler updateHandler; // constructor omitted @KafkaListener(topics = "${app.topic.name}", groupId = "consumers") public void handleEvent(Event event) { AnalysisResult analysis = aiService.analyze(event.getData()); ResultEvent result = new ResultEvent(event.getId(), analysis); updateHandler.sendUpdate(result); } } Here AIService encapsulates the ML logic calling a model to get a prediction or insight from event.getData(). After computing an AnalysisResult we wrap it in a ResultEvent and immediately push it out. In this case, we use a WebSocket handler to send the result to clients as soon as it's ready. Using a Kafka consumer for AI processing offers several benefits: Async processing: The AI work happens in the background. Scalability: Multiple ConsumerService instances can share the load allowing throughput to grow with demand. Fault isolation: If AI processing fails or lags, it doesn’t break the user request flow. The event remains in Kafka for a retry or dead-letter handling, and the main app continues running. Real-Time WebSocket Delivery to the Frontend After events are processed and results are generated the final step is delivering updates to users in real time. Instead of clients polling for updates, webSockets let the server push data to browsers instantly for a live-updating experience. Spring Boot’s WebSocket support makes it straightforward to broadcast messages. We can create a handler to manage client connections and send out updates: Plain Text @Component public class UpdateSocketHandler extends TextWebSocketHandler { private WebSocketSession clientSession; private final ObjectMapper jsonMapper = new ObjectMapper(); @Override public void afterConnectionEstablished(WebSocketSession session) { this.clientSession = session; } @Override public void afterConnectionClosed(WebSocketSession session, CloseStatus status) { this.clientSession = null; } public void sendUpdate(ResultEvent result) throws IOException { if (clientSession != null && clientSession.isOpen()) { String json = jsonMapper.writeValueAsString(result); clientSession.sendMessage(new TextMessage(json)); } } } This handler stores the client session when a connection is established. The sendUpdate method converts a ResultEvent into JSON and pushes it to the client if the connection is open. On the frontend webSocket client would listen for these messages to update the UI. Finally, we register this handler to expose a WebSocket endpoint . A web client can connect to ws://<server>/updates and start receiving ResultEvent messages. Now whenever our backend calls updateHandler.sendUpdate(result) the data is immediately pushed to the client. The user interface updates without any page refresh or polling. Why WebSockets? They enable low-latency, server-push updates. As soon as an AI result is available the user sees it. This pattern is ideal for live dashboards, notifications or any real-time monitoring scenario providing a smooth user experience with up-to-the-second information. Conclusion Combining event-driven architecture with AI processing and real-time WebSocket delivery yields a powerful yet decoupled system design. Spring Boot and Kafka let us offload and buffer work the front-end/API layer remains responsive while the back end performs intensive AI computations asynchronously. WebSockets close the loop by instantly pushing results to users ensuring they always have the latest data. These three patterns Kafka-based event production, AI-augmented consumption and WebSocket-based client updates work in tandem to create a system that is scalable, flexible and intelligent. Each layer is modular and can be scaled or updated independently. In practice this architecture can power anything from fraud detection to IoT analytics . By leveraging Kafka as the backbone, Spring Boot for rapid development and WebSockets for live updates you deliver instant feedback and smart features to users while keeping the solution loosely coupled and maintainable.