Coding

Picture the scene: One of the services in your backend is a mature Django app that no one has the resources, time, or, frankly, the will to rewrite. The ORM, the admin panel, and the broader ecosystem all earn their keep. But you’re looking for the best way to describe your API, and FastAPI catches your eye. It looks like a great fit: native typing, pydantic-based validation, OpenAPI out of the box, and of course the support for async endpoints. That's the situation our team found itself in - we decided to use both frameworks and take from each what suited us best. Not everything went smoothly — this post is what we built, what broke afterward, and what we learned. The First Win So we wired it up, and it works. FastAPI runs as the ASGI application, and the existing Django app plugs into it. Python # asgi.py import os os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings") from django.core.asgi import get_asgi_application from fastapi import FastAPI app = FastAPI() django_app = get_asgi_application() app.mount("/legacy", django_app) Great! Now: Both Django and FastAPI endpoints live side by side, with no pressure to refactor everything in a single day — that was important for us.In the new parts of the app, Django steps back into a single role: communicating with the database through its models.Endpoints can be either sync or async. That was the win. But there was the other side also. Pitfall 1: Async Endpoints Started Running One at a Time When you reach out to external services, chances are you also want to enrich the request with something from your database, or save the result back to it (we did). Here's a tiny example: A single async handler that fetches data about Order from the database (we use Postgres) and forwards it to an external payment provider. Python from asgiref.sync import sync_to_async from fastapi import FastAPI app = FastAPI() @app.post("/orders/{order_id}/dispatch") async def dispatch_order(order_id: int) -> OrderDTO: order = await sync_to_async(get_order)(order_id) # fetch from DB await client.send_order(order) # call external service return order # code that uses a Django model def get_order(order_id: int) -> OrderDTO: order = Order.objects.get(id=order_id) return OrderDTO(id=order.id, amount=order.amount) Inside an async function, you can’t call the Django ORM synchronously. The documented approach is sync_to_async, which moves the synchronous call to a separate thread so it doesn’t block the event loop. Now let's see what happens under concurrent load. Drop a three-second sleep into get_order: Python from django.db import connection def get_order(order_id: int) -> OrderDTO: order = Order.objects.get(id=order_id) with connection.cursor() as cursor: cursor.execute("SELECT pg_sleep(3);") return OrderDTO(id=order.id, amount=order.amount) And fire three requests in parallel: Shell URL="http://localhost:8000/orders/1/dispatch" curl -s -o /dev/null -X POST -w "%{time_total}s\n" "$URL" & curl -s -o /dev/null -X POST -w "%{time_total}s\n" "$URL" & curl -s -o /dev/null -X POST -w "%{time_total}s\n" "$URL" >> 3.012s >> 6.024s >> 9.037s We expected ~3 seconds and got nine. The handlers ran one after another, not concurrently. And if you log the thread and database connection IDs from inside get_order, all three requests print the same values. Why? By default sync_to_async(get_order) runs with thread_sensitive=True, which means the function runs in the same thread as all other thread_sensitive functions. A standalone Django ASGI app does extra work here: it opens a fresh context per request, so requests run in parallel. The benchmark suggests that in our setup FastAPI doesn't: all three sync_to_async calls land on the same thread and line up one behind another. The event loop itself stays free, by the way: a purely async route keeps responding while the three /dispatch requests wait in that queue. But three async handlers with ORM calls queue up on the same thread, sharing the same connection. For a moment we hoped Order.objects.aget(...) or other Django async ORM helpers would save us here. They won't: for now under the hood they call the same sync_to_async. Can we just flip to sync_to_async(..., thread_sensitive=False)? Probably not - it is not a safe default. Django carries a lot of per-request state in thread-locals: the current DB connection, transaction.atomic(), etc. The Django docs say: "a lot of existing Django code assumes it all runs in the same thread." What to Do About It No silver bullet, but two approaches hold up: Split handlers by what they touch. Reserve async def for endpoints that genuinely don't touch the ORM — async-native HTTP calls, cache reads, etc. For ORM-bound endpoints, declare them as plain sync routes. FastAPI runs sync routes on its thread pool, so they actually run in parallel, and each thread gets its own Django connection. As long as these endpoints don't make many slow external calls, this can work.Move the work out of the handler entirely. If your project already runs with a message broker, the possible answer to "external API + DB write inside a handler" is to stop doing it inside a handler at all. Drop an event on the bus, let consumers handle the side effects, return immediately. The catch: this only makes sense when an event-driven flow already fits your system — because it is, of course, no small refactor. Pitfall 2: Tests That Can't See Their Own Data Now let's write a test for get_order — a sync endpoint that reads an order from the DB. The test runs with pytest-django: we create an order in the database and call the handler. Python # app.py import pytest from fastapi import FastAPI, HTTPException from fastapi.testclient import TestClient app = FastAPI() @app.get("/orders/{order_id}") def get_order(order_id: int) -> OrderDTO: try: order = Order.objects.get(id=order_id) except Order.DoesNotExist: raise HTTPException(status_code=404) return OrderDTO(id=order.id, amount=order.amount) @pytest.mark.django_db def test_get_order(): Order.objects.create(id=1) response = TestClient(app).get("/orders/1") assert response.status_code == 200 # and we'll have 404 You get 404 Not Found. The handler ran, looked at the database, and the order was nowhere to be found. Four facts conspire here: Pytest runs your test's data setup in one thread; when the FastAPI test client calls the endpoint, the handler runs in another.pytest-django wraps every test in an open transaction and rolls it back at the end. That's how the suite stays fast and isolated. The transaction lives on a single database connection.Django opens a database connection per thread.Postgres defaults to READ COMMITTED isolation: one connection cannot see another connection's uncommitted writes. So: the test body runs in the pytest thread. Its Order.objects.create(...) uses connection 1, inside pytest-django's open transaction. When TestClient hits the endpoint, FastAPI dispatches the handler to a worker thread from its thread pool, on another thread with its own connection 2. Connection 2 looks at the database and sees no order, because connection 1 hasn't committed, so connection 1's write is effectively invisible to everyone else. Again — What to Do? Test in layers. Unit-test the endpoint contract with the ORM mocked - those tests don't cross thread or connection boundaries, so the visibility problem simply can't appear. Test business logic and data access in their own tests, without going through TestClient. For cases when the full end-to-end test is still needed - the commonly suggested fix is @pytest.mark.django_db(transaction=True). This switches the test to a mode where writes actually commit, so other connections can see them. But it has its cost: pytest-django now does a database flush after every test, and the suite gets noticeably slower. On a large suite, for us "noticeably" meant minutes - too much on every run, so we use it only for exceptional cases. The Recap FastAPI brings obvious wins — OpenAPI docs, clean endpoint code, typing all the way through; Django gives you a greatly tested ORM and admin. Putting them in the same process gives us both — and a thread-and-connection model that doesn't behave the way we'd expect. Budget for the architecture work before you budget for the migration. Was it worth it? Yes — we got the clean, typed API we were after, and we kept Django's ORM instead of porting the whole data layer to another framework. Would we do it again? Not sure. The trade-offs of this integration may outweigh its benefits for us, so other combinations might be a better fit. If you’ve run into the same solution and found an approach with better trade-offs, please share; the comments are open. Reproduce it yourself. An example with a benchmark and failing tests is in https://github.com/evchibisova/fastapi-over-django-test.

My first attempt to deploy a Spring Boot microservice on AWS Fargate didn’t fail loudly. It failed quietly — in a loop. ECS kept launching tasks, the Application Load Balancer kept marking them unhealthy, and the service never stabilized. The logs looked fine, the container looked fine, but the ALB replaced every task within seconds. The root cause was painfully simple: Spring Boot needed 45 seconds to start, and my ALB health‑check timeout was 5 seconds. The tasks never had a chance. That night changed how I build and deploy microservices. It forced me to rethink startup behavior, JVM sizing, networking, task definitions, and the entire CI/CD pipeline. This article is the guide I wish I had before that incident — a practitioner’s walkthrough of deploying a production‑ready Spring Boot service on AWS Fargate, with real artifacts and the details that matter when things go wrong. The Architecture That Finally Worked Once the health‑check issue was fixed, the architecture settled into a predictable, cloud‑native flow: Developers push code to GitHubGitHub Actions builds the JARDocker image is built and pushed to Amazon ECRECS service runs AWS Fargate tasksTraffic enters through an Application Load BalancerTasks run in private subnetsConfiguration comes from Parameter Store and Secrets ManagerLogs and metrics flow to CloudWatch It’s the standard modern microservice pipeline — but the difference between “standard” and “production‑ready” is in the details. The Spring Boot Service The microservice itself was simple — a REST API with a few endpoints. The real complexity wasn’t the controller logic; it was everything around it: startup time, health checks, configuration management, and container behavior under load. A Dockerfile Built for Production My first Dockerfile looked like the one many tutorials start with: a single‑stage build running as root with no JVM tuning. It worked locally but failed under real load. Fargate tasks with default JVM heap sizing inside a 2GB container are a classic OOM story. Here’s the hardened version that finally stabilized deployments: Dockerfile FROM eclipse-temurin:21-jre # Create non-root user RUN useradd -u 1001 springuser WORKDIR /app # Layer extraction for faster builds COPY target/*.jar app.jar # JVM tuning for Fargate ENV JAVA_OPTS="\ -XX:MaxRAMPercentage=75 \ -XX:+UseContainerSupport \ -XX:+ExitOnOutOfMemoryError \ " USER springuser ENTRYPOINT ["sh", "-c", "java $JAVA_OPTS -jar app.jar"] This eliminated the OOMKilled events I saw on 2GB tasks and made startup time predictable. Pushing to Amazon ECR With Real Commands The first time I wrote down my ECR commands, they were placeholders. In production, they need to be exact: C aws ecr get-login-password --region us-east-1 \ | docker login --username AWS --password-stdin <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com docker build -t employee-service:1.0.3 . docker tag employee-service:1.0.3 \ <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/employee-service:1.0.3 docker push \ <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/employee-service:1.0.3 Immutable semantic version tags make rollbacks predictable and prevent “latest‑tag roulette.” The ECS Task Definition That Actually Runs in Production A real Fargate deployment lives or dies by its task definition. Here’s the JSON I use today — including secrets pulled from Parameter Store and Secrets Manager: JSON { "family": "employee-service", "networkMode": "awsvpc", "requiresCompatibilities": ["FARGATE"], "cpu": "512", "memory": "1024", "executionRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/ecsTaskExecutionRole", "taskRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/employeeServiceRole", "containerDefinitions": [ { "name": "employee-service", "image": "<ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/employee-service:1.0.3", "portMappings": [ { "containerPort": 8080, "protocol": "tcp" } ], "secrets": [ { "name": "DB_PASSWORD", "valueFrom": "arn:aws:ssm:us-east-1:<ACCOUNT_ID>:parameter/db/password" }, { "name": "API_KEY", "valueFrom": "arn:aws:secretsmanager:us-east-1:<ACCOUNT_ID>:secret:thirdparty/api" } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/employee-service", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "ecs" } } } ] } The ALB Health Check That Stopped the Outage My outage happened because the ALB was impatient. Here’s the configuration that finally stabilized deployments: settingvalue Path /actuator/health Interval 20 seconds Timeout 10 seconds Healthy threshold 3 Unhealthy threshold 3 Spring Boot startup time + ALB patience = stable deployments. Why Fargate Tasks Belong in Private Subnets Early on, I deployed tasks in public subnets because it felt simpler. It wasn’t. Public IPs meant the containers were directly reachable from the internet — port scans, bot traffic, and noisy logs. Moving tasks to private subnets solved several problems at once: Reduced Attack Surface No public IPs. No direct inbound traffic. Only the ALB can reach the tasks. A Single Secure Entry Point The ALB handles TLS termination, redirects HTTP→HTTPS, performs health checks, and integrates with WAF. Clients never bypass it. Cleaner Security Groups ALB SG: inbound 443 from the internetTask SG: inbound only from ALB SG Nothing else touches the containers. Compliance Alignment PCI, SOC 2, HIPAA — all prefer minimizing public exposure. Controlled Outbound Access Tasks use a NAT Gateway for outbound calls (updates, third‑party APIs) without exposing themselves. Better Scalability ALB target groups automatically track tasks across AZs as ECS scales. The architecture becomes simple and predictable: Internet → ALB (public subnets) → Fargate tasks (private subnets) It’s quieter, safer, and easier to operate. The GitHub Actions Workflow That Deploys Automatically Here’s the pipeline that builds, tests, pushes, and deploys the service: YAML name: Deploy to Fargate on: push: branches: ["main"] jobs: build-deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up JDK uses: actions/setup-java@v4 with: java-version: "21" - name: Build JAR run: mvn -B clean package - name: Login to ECR uses: aws-actions/amazon-ecr-login@v2 - name: Build and Push Image run: | docker build -t employee-service:1.0.3 . docker tag employee-service:1.0.3 ${{ env.ECR_REGISTRY }/employee-service:1.0.3 docker push ${{ env.ECR_REGISTRY }/employee-service:1.0.3 - name: Deploy ECS Service uses: aws-actions/amazon-ecs-deploy-task-definition@v2 with: task-definition: ecs-task.json service: employee-service cluster: prod-cluster Auto Scaling With Real Target Tracking JSON Target tracking is the simplest and most reliable scaling strategy for Fargate: JSON { "TargetValue": 50.0, "PredefinedMetricSpecification": { "PredefinedMetricType": "ECSServiceAverageCPUUtilization" }, "ScaleOutCooldown": 30, "ScaleInCooldown": 60 } I use 50% as the target because it balances cost and responsiveness. What I Learned Every failure taught me something: ALB timeouts taught me to respect startup timeOOMKilled tasks taught me to tune the JVMPublic subnets taught me to isolate workloadsManual deployments taught me to automate everything AWS Fargate really does deliver on its promise — no servers to manage, automatic scaling, and clean integration with ECS — but only after you learn the hard parts. If you’re deploying Spring Boot on Fargate, I hope you learn those lessons from this article instead of from your own outage.

Arrays in Java are fundamental data structures used to store elements of the same type sequentially in memory. They provide a convenient way to manage collections of data where each element is accessed by its index. Basics of Arrays An array in Java is a fixed-size container that holds a specific number of elements of the same data type. This means all elements in an array must be of the same type such as integers (int), floating-point numbers (double), characters (char) or objects (Object). Declaring and Initializing Arrays To declare an array in Java we specify the type of elements followed by square brackets [] and the array name: Java dataType[] arrayName; For example, to declare an integer array named numbers: Java int[] numbers; Arrays in Java are objects and like all objects they must be instantiated with the new keyword before they can be used: Java arrayName = new dataType[arraySize]; For instance, to create an integer array numbers with a size of 5: Java int[] numbers = new int[5]; This initializes an array numbers that can hold 5 integers with indices ranging from 0 to 4. Accessing Elements in Arrays Array elements are accessed using their index, which starts at 0 for the first element and goes up to arraySize - 1 for the last element. For example, to access and modify elements of the numbers array: Java int[] numbers = {10, 20, 30, 40, 50}; int firstElement = numbers[0]; // Retrieves the first element (10) int thirdElement = numbers[2]; // Retrieves the third element (30) numbers[1] = 25; // Modifies the second element to 25 Array Length The length of an array in Java, which is the number of elements it can hold can be obtained using the length property: Java int arrayLength = numbers.length; // Returns 5 for the 'numbers' array The length property is a final variable defined in the array object itself and it cannot be changed after the array is created. Iterating Through Arrays Arrays can be traversed using loops such as for or foreach to access and manipulate each element sequentially: Java int[] numbers = {10, 20, 30, 40, 50}; for (int i = 0; i < numbers.length; i++) { System.out.println("Element at index " + i + ": " + numbers[i]); } Alternatively, Java provides an enhanced for-each loop also known as the enhanced for loop to iterate through elements of an array without explicitly using an index: Java for (int number : numbers) { System.out.println(number); } Multidimensional Arrays Java supports multidimensional arrays which are arrays of arrays. we can declare and initialize them as follows: Java dataType[][] arrayName = new dataType[rows][columns]; For example, to create a 2D integer array matrix with 3 rows and 3 columns: Java int[][] matrix = new int[3][3]; Accessing elements in a 2D array requires specifying both row and column indices: Java int element = matrix[1][2]; // Retrieves element at row 1, column 2 Arrays Class Methods The Arrays class in Java provides utility methods for working with arrays such as sorting, searching and comparing arrays: Java import java.util.Arrays; int[] numbers = {5, 3, 8, 2, 7}; Arrays.sort(numbers); // Sorts the 'numbers' array in ascending order int index = Arrays.binarySearch(numbers, 8); // Searches for '8' in the sorted array Other useful methods include copyOf(), fill() and equals(). Common Operations on Arrays Sorting: Arrays can be sorted using Arrays.sort().Searching: Use Arrays.binarySearch() to search for an element in a sorted array.Copying: Arrays can be copied using Arrays.copyOf() or System.arraycopy().Filling: Arrays can be filled with a specific value using Arrays.fill(). Applications of Arrays Arrays are used extensively in various applications, such as: Storing and manipulating collections of data in algorithms and applications.Implementing data structures like lists, queues, and matrices.Handling input/output operations in Java programs.Passing arrays as parameters to methods for processing and manipulation. Conclusion Arrays are fundamental data structures in Java that provide efficient storage and access mechanisms for homogeneous collections of data. They play a crucial role in Java programming offering versatility and performance in managing and manipulating data elements. FAQs 1. What is an array in Java? An array in Java is a fixed-size collection of elements of the same type stored sequentially in memory.2. How do you declare an array in Java? You declare an array in Java by specifying the type of elements followed by square brackets [] and the array name, like int[] numbers;.3. Can arrays in Java store elements of different data types? No, arrays in Java can only store elements of the same data type. Once declared the data type of an array is fixed.4. What is the difference between length and length() in arrays? length is a final variable in arrays that denotes the number of elements it can hold. length() is a method used with the strings and other objects to get the number of characters or elements.5. How do you initialize an array in Java? You can initialize an array in Java using the new keyword followed by the array type and size like int[] numbers = new int[5];.6. What are multidimensional arrays in Java? Multidimensional arrays in Java are arrays of arrays. They allow you to store data in multiple dimensions such as rows and columns in a matrix.7. How can you iterate through an array in Java? we can iterate through an array in Java using a for loop or an enhanced for-each loop to access each element sequentially.8. Can you resize an array in Java once it's created? No, once an array is created with a specific size its size cannot be changed. we would need to create a new array with the desired size and copy elements if resizing is needed.9. What are the common operations you can perform on arrays in Java? Common operations include sorting arrays (Arrays.sort()) searching for elements (Arrays.binarySearch()), copying arrays (System.arraycopy()) and filling arrays (Arrays.fill()).10. What are the applications of arrays in Java? Arrays are used for implementing data structures like lists and queues storing data in algorithms handling input/output operations and passing data to methods efficiently.

Not long ago, I broke a backtest without changing a single line of code. I moved the script to a different machine—same OS, supposedly the same Python version — and the equity curve suddenly told a completely different story. Nothing in the logic had changed. The environment was the only obvious difference. That was the day I stopped treating the runtime environment as an afterthought and started treating reproducibility as part of the experiment itself. What I learned is this: a backtest isn’t truly reproducible just because it ran once on your laptop. A result you can’t regenerate reliably isn’t a research finding — it’s a coincidence. And when that coincidence eventually meets real money, it can get expensive fast. What follows is a minimal but complete pipeline for containerizing and automatically testing a Python backtesting system. No over-engineering — just Docker, GitHub Actions, and a few habits that make your results trustworthy. The Real Goal We aren’t building a live trading platform. We’re building a workflow that ensures every change is verified in a controlled environment: Plain Text Code change → automated tests → versioned Docker image build The system must do four things: Produce deterministic results, within an acceptable numerical tolerance, from the same versioned inputs in the same containerized environment.Automatically test every code change before it can be merged.Halt the pipeline when a test fails, with no exceptions.Tag every image build so we can trace it back to the exact source-code version. If a workflow can’t do that, it’s just a script living on someone’s laptop. Project Structure Before containerizing the application, let’s organize the repository clearly: Python backtesting-system/ ├── app/ │ ├── engine.py │ └── main.py ├── tests/ │ └── test_engine.py ├── data/ │ └── sample.csv ├── requirements.txt ├── requirements-dev.txt ├── Dockerfile └── .github/ └── workflows/ └── ci.yml The app/ directory holds the strategy logic, while tests/ remains separate. The data/ directory contains a small, frozen sample dataset that never changes — our gold standard. There are no absolute paths or machine-specific assumptions. Everything that might vary, including the data path, starting capital, and fee rate, comes from environment variables or a configuration file. Containerizing With Docker The classic “works on my machine” problem usually indicates an environment mismatch. Docker reduces this drift by packaging the application and its runtime dependencies into a versioned image. Here is the Dockerfile: Dockerfile FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY app ./app COPY data ./data CMD ["python", "-m", "app.main"] A few decisions matter here. We pin the Python 3.12 image series and can use an image digest when stricter reproducibility is required. Dependencies are pinned to exact versions in requirements.txt, since version ranges can silently introduce changes. We also use slim to keep the image small. Crucially, we never copy local keys, cached results, or temporary files into the image. The container doesn’t guarantee that the logic is correct. It helps ensure that sound logic runs in a controlled and substantially more consistent environment. Adding Tests That Matter Automation without tests simply automates mistakes. Our tests don’t attempt to prove that a strategy is profitable. They prove that the program behaves consistently. We check for: Clear errors when input files are empty or missing.Deterministic results when the same data and seed are used.Correct fee calculations.Rejection of malformed rows rather than silent processing.Required fields in every output. Here is one example, including the necessary imports: Python import pytest from app.engine import run_backtest def test_backtest_is_reproducible(): first = run_backtest("data/sample.csv", seed=42) second = run_backtest("data/sample.csv", seed=42) assert first["trades"] == second["trades"] assert first["final_equity"] == pytest.approx( second["final_equity"], rel=1e-9 ) This test establishes a simple contract: given the same starting conditions, the system will not drift beyond an acceptable margin. The direct comparison of trades works here because we assume that every trade entry uses standardized types such as integers and strings. If the trade records contain floating-point prices, those values should be checked individually with an appropriate tolerance. Building the GitHub Actions Pipeline Now we automate the workflow. It runs on pushes, pull requests, and manual triggers through workflow_dispatch: YAML name: Backtest CI on: push: pull_request: workflow_dispatch: jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" - run: pip install -r requirements-dev.txt - run: pytest - run: docker build -t backtest:${{ github.sha } . The steps are straightforward: check out the code, set up Python, install the development dependencies, run the tests, and build the Docker image. The requirements-dev.txt file includes both the production dependencies and a pinned version of pytest: Python -r requirements.txt pytest==8.3.4 If pytest detects a failure, the job stops immediately. The broken change never reaches the image-build step. The resulting image is tagged with the Git commit hash, creating a clear link between the source code and the image built during that workflow run. Managing Configuration and Secrets Environment-specific configuration and secrets should never be baked into the image. Environment variables can control data paths and run modes. If the system is later connected to a live data source, API credentials should be stored in GitHub Secrets or a cloud secrets manager—never in the source code or Dockerfile. Logs must not expose keys or sensitive headers. Even if today’s “production” environment is only a scheduled test run, development and production should use separate configuration sets. Treat every secret as sensitive, and keep environment-specific configuration outside the image. Lightweight Monitoring and a Path to Rollback Once the pipeline runs regularly, monitoring must go beyond asking whether the process is still alive. Useful questions include: Did the latest job complete, or did it hang?Has execution time increased dramatically?Is the input data intact?Were the output files generated, and are they non-empty?Which image version produced the latest results? If images are later pushed to a container registry, retaining the last few stable versions provides a straightforward rollback path. For scheduled backtest runs, we should also archive the data snapshot, parameters, and results. That allows us to return a month later and answer a very specific question: “What exactly did we test on July 24?” These practices aren’t unique to systems we build ourselves. Commercial grid-trading interfaces make automated execution accessible without revealing every part of their internal deployment pipelines. BYDFi is one example I encounter in my work. Because I work with the platform, this is a disclosed reference rather than an independent recommendation. The comparison is conceptual: understanding reproducibility, automated checks, and configuration management helps developers evaluate any automated tool more thoughtfully. The Experiment Isn’t Finished Until It’s Verified We started with a broken backtest and a frustrating realization. Now we have a different mindset. Docker reduces environment drift. Automated tests guard program behavior. GitHub Actions ensures every change passes through the same gate. Monitoring and versioning give us a clear path to detect problems and support rollback as the pipeline evolves. In a reliable backtesting system, reproducibility and verification are not tasks that come after the experiment. They are part of the experiment itself. The moment we treat them that way, our results stop being anecdotes and start becoming evidence. And when the decisions involved can carry real financial weight, evidence is the only thing worth building.

The Problem With "Just Add More Workers" Most Spark performance issues on Databricks aren't solved by scaling the cluster — they're caused by shuffle and skew, and no amount of extra nodes fixes a badly partitioned join. This post builds a realistic pipeline (order events joined against a small dimension table, aggregated, and written to Delta Lake) from the ground up, and uses it to work through: How Spark's shuffle actually behaves during a wide transformationDiagnosing and fixing data skew with salting and adaptive query execution (AQE)Laying out the resulting Delta table with Z-Ordering so downstream queries skip irrelevant filesGoverning access to the whole pipeline with Unity Catalog Architecture Overview Pipeline shape – a batch job reading raw events, joining against a dimension table, aggregating, and writing to a governed Delta table: What happens inside a shuffle stage – this is the part most tutorials skip, and it's the key to understanding why skew hurts: Step 1: Set Up Governed Tables in Unity Catalog Everything downstream depends on tables being registered under Unity Catalog, which gives you centralized access control and lineage instead of per-workspace table grants. SQL -- setup.sql, run in a Databricks SQL or notebook cell CREATE CATALOG IF NOT EXISTS retail_analytics; CREATE SCHEMA IF NOT EXISTS retail_analytics.events; CREATE TABLE IF NOT EXISTS retail_analytics.events.raw_orders ( order_id STRING, customer_id STRING, product_id STRING, quantity INT, event_ts TIMESTAMP ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/raw_orders'; CREATE TABLE IF NOT EXISTS retail_analytics.events.dim_products ( product_id STRING, category STRING, unit_cost DOUBLE ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/dim_products'; GRANT SELECT ON TABLE retail_analytics.events.raw_orders TO `analysts`; Step 2: Read and Force a Broadcast Join for the Small Dimension Table dim_products is small, so let Spark broadcast it rather than shuffle both sides of the join. Python # pipeline.py from pyspark.sql import functions as F orders = spark.table("retail_analytics.events.raw_orders") products = spark.table("retail_analytics.events.dim_products") Without the explicit broadcast hint, Spark's cost-based optimizer usually picks a broadcast join automatically for small tables, but being explicit avoids surprises when the dimension table grows past spark.sql.autoBroadcastJoinThreshold (default 10MB) without anyone noticing. Step 3: The Aggregation That Triggers a Shuffle groupBy on customer_id is a wide transformation — Spark must shuffle rows so all records for a given key land on the same reducer. Python agg = ( joined .groupBy("customer_id", "category") .agg( F.sum(F.col("quantity") * F.col("unit_cost")).alias("total_spend"), F.count("order_id").alias("order_count"), ) ) If one customer_id (say, a test account or a large B2B buyer) accounts for a disproportionate share of rows, this is where skew shows up: one reducer task runs for minutes while the rest of the stage finishes in seconds. You'll see this in the Spark UI as a single long-running task in an otherwise short stage. Step 4: Fixing Skew With Salting Adaptive Query Execution (AQE) handles a lot of skew automatically in modern Databricks Runtime, but for known hot keys, explicit salting is still the most predictable fix. Python from pyspark.sql import functions as F import random SALT_BUCKETS = 20 # Add a salt column to spread the hot key across multiple reducers salted = joined.withColumn("salt", (F.rand() * SALT_BUCKETS).cast("int")) partial_agg = ( salted .groupBy("customer_id", "category", "salt") .agg( F.sum(F.col("quantity") * F.col("unit_cost")).alias("total_spend"), F.count("order_id").alias("order_count"), ) ) # Second pass: combine the salted partial aggregates into the final result final_agg = ( partial_agg .groupBy("customer_id", "category") .agg( F.sum("total_spend").alias("total_spend"), F.sum("order_count").alias("order_count"), ) ) This two-phase pattern — pre-aggregate on a salted key, then combine — is the same trick used inside combiners in older MapReduce systems. It trades a bit of extra shuffle for eliminating the single-reducer bottleneck. Also worth setting explicitly rather than relying on the default of 200: Python spark.conf.set("spark.sql.shuffle.partitions", "auto") # let AQE size it dynamically spark.conf.set("spark.sql.adaptive.enabled", "true") spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true") Step 5: Write to Delta Lake With Optimized Writes Python ( final_agg.write .format("delta") .mode("overwrite") .option("delta.autoOptimize.optimizeWrite", "true") .saveAsTable("retail_analytics.events.customer_spend_summary") ) Optimized writes shuffle data before writing so you get fewer, better-sized files instead of many small ones — this costs some write-time latency in exchange for faster reads later. Step 6: Z-Order the Table for the Queries That Matter If most downstream queries filter on customer_id, co-locate related rows physically so Spark can skip files that can't match the filter. Python OPTIMIZE retail_analytics.events.customer_spend_summary ZORDER BY (customer_id); Z-ordering aims to produce evenly balanced data files by row count rather than raw size, and its effectiveness depends on the column having reasonably high cardinality — Z-ordering by a low-cardinality column like category alone gives little benefit. On newer Delta Lake / Databricks Runtime versions, Liquid Clustering is generally the preferred choice for new tables since it adapts as query patterns change, while ZORDER remains relevant mainly for existing tables not yet migrated. SQL -- Preferred on new tables (Databricks Runtime supporting Liquid Clustering): CREATE TABLE retail_analytics.events.customer_spend_summary CLUSTER BY (customer_id); Comparing Shuffle-Mitigation Techniques TechniqueFixesCostWhen to useBroadcast joinShuffle on the large side of a joinExtra memory on executorsSmall (<~10MB by default) dimension/lookup tablesAQE skew join handlingAutomatic detection of skewed partitionsMinor planning overheadDefault-on; good general safety netManual saltingKnown, severe hot keysExtra shuffle for the two-phase aggregateHot keys that AQE doesn't fully resolveRepartition by keyUneven task distribution before a shuffle stageOne extra shufflePre-shaping data before multiple downstream joinsZ-OrderingSlow reads due to unnecessary file scansShuffle + rewrite during OPTIMIZEExisting tables, high-cardinality filter columnsLiquid ClusteringSame as Z-Order, plus evolving query patternsOngoing background clusteringNew tables on supporting runtime versions Governance: Tying It Back to Unity Catalog Because every table above was created under retail_analytics, access control, audit logging, and lineage are handled centrally rather than per-cluster: SQL -- Restrict PII-adjacent columns without duplicating the table CREATE VIEW retail_analytics.events.customer_spend_summary_masked AS SELECT customer_id, category, total_spend, order_count FROM retail_analytics.events.customer_spend_summary; GRANT SELECT ON VIEW retail_analytics.events.customer_spend_summary_masked TO `bi_readers`; Production Considerations Diagnose before tuning. Check the Spark UI's stage view for one long-running task among many short ones — that's the signature of skew, not just "the job is slow."Predictive optimization can run OPTIMIZE and ANALYZE automatically on Unity Catalog-managed tables, which reduces the need for scheduled maintenance jobs for many workloads.Don't Z-Order everything. It requires shuffling and rewriting the table (or partition), so reserve it for columns that are actually filtered on frequently downstream. References Best practices: Delta Lake — Azure Databricks / Microsoft LearnData skipping (Z-ordering) — Azure Databricks / Microsoft LearnOptimizations — Delta Lake documentationDelta Lake Under the Hood: What Every Data Engineer Should Know — Databricks CommunityMastering Delta Lake Performance: Z-Ordering vs Liquid Clustering — Medium

By Jubin Soni, FBCS CORE

Language models become much more useful when they can answer questions about information they were never trained on, including your internal documentation, product manuals, policies, and other proprietary data. Prompting alone cannot solve this, because the model simply does not have access to that knowledge. Retrieval-Augmented Generation, or RAG, is the most common way to bridge that gap. Spring AI comes with solid support for building RAG systems. It has been almost three years since Spring AI showed up, and in that time it has grown from an experimental member of the Spring portfolio into a mature layer over chat models, embedding models, vector stores, and the plumbing that sits between them, which happen to be exactly the pieces a RAG system needs. In this article, we build a small but complete RAG service with Spring AI 2.0. The application reads a set of documents into a PostgreSQL vector store, retrieves the fragments that are relevant to a user question, and lets Anthropic's Claude put together the answer based on those fragments. Everything runs from a standard Spring Boot project, and every step can be reproduced on macOS, Windows, or Linux. The full project is available on GitHub. If you just want to see the finished result, or you would rather skip the step-by-step build below, you can clone the repository and run it as it is. Everyone else can follow along and generate this project from scratch. The prompts themselves are kept deliberately simple. You can tune retrieval and prompts forever; here we care about the architecture and how the pieces fit together in Spring. Approach RAG is not really a single feature. It is more of a small pipeline, and the code below makes a lot more sense once its parts have names. Embedding: a vector of numbers that captures the meaning of a piece of text. Texts that mean similar things end up with vectors that are close to each other.Embedding model: the model that computes these embeddings. It is a different model from the chat model, and it has a different job.Vector store: a database that keeps text fragments together with their embeddings and can answer the question, "which stored fragments are closest in meaning to this query?"Chunking: documents are too large to embed and retrieve as a whole, so we split them into smaller fragments (chunks) before storing them.Similarity search: we embed the user question and fetch the top-k closest chunks from the store.Augmentation: we append the retrieved chunks to the user question before sending it to the chat model, so the model answers from the context we provided instead of from its training data. One thing here is worth calling out, because it shapes the whole setup of the project: the LLM model used in chat and the embedding model are two separate choices. As of today, Anthropic offers LLM models but no embedding API, so a Claude-based RAG system always has to pair Claude with an embedding model from somewhere else. Rather than bringing in a second cloud provider and a second API key, this project computes embeddings locally (inside the JVM), using Spring AI's ONNX transformers module and the well-known all-MiniLM-L6-v2 sentence transformer. It is free and fast enough for this, and it keeps everything on one API key. In our scenario, the service is an internal assistant for a fictional company called Nimbusfield Systems, and it answers employee questions based on the company handbook. The company and the handbook are fictional on purpose. Claude cannot possibly know about it, which makes it easy to verify that the answers really come from our documents and not from the model's own memory. We build this in three steps: Expose a /ask endpoint backed by Claude, with no retrieval, and show that the model cannot answer handbook questions.Ingest the handbook into PGvector at application startup: read, chunk, embed, and store.Attach Spring AI's QuestionAnswerAdvisor to the same ChatClient and ask again. Prerequisites Java 21Maven 3.9.x (the Maven wrapper included in generated projects works too)Spring Boot 4.0.xSpring AI 2.0.0Docker Desktop (macOS/Windows) or Docker Engine (Linux), used only to run PostgreSQL. A project skeleton can be generated at start.spring.io by selecting Web, Anthropic Claude, PGvector Vector Store, and Docker Compose Support. The remaining Spring AI modules are added manually below. The Claude API Key Sign in (or sign up) at the Anthropic Console, open Settings, then API Keys, and create a new key. New accounts may need a small prepaid credit before the API accepts requests, but the runs in this article cost only a few cents. The key is shown only once, so store it right away as an environment variable. If you would rather not spend anything at all, you can still follow along and read through the steps without running the calls yourself. macOS/Linux: export ANTHROPIC_API_KEY=sk-ant-... Windows (PowerShell, persists across sessions after reopening the terminal): setx ANTHROPIC_API_KEY "sk-ant-..." Solution Dependencies With the Spring AI BOM in place, there is no need to repeat versions on the individual artifacts. Initializr expresses the BOM's own version as a property rather than a hardcoded literal, so there is a single place to bump it later: XML <properties> <java.version>21</java.version> <spring-ai.version>2.0.0</spring-ai.version> </properties> <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-bom</artifactId> <version>${spring-ai.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> A common source of confusion is that start.spring.io has no dependency literally named "Spring AI." Each provider- or store-specific starter (Anthropic Claude, PGvector Vector Database, and so on) is itself a Spring AI module, and picking one transitively pulls in the framework's core classes. (like ChatClient, VectorStore, etc.) Selecting any one of them is also what makes Initializr add the spring-ai-bom as shown above to the generated pom.xml for you. The BOM itself is never a separate item you tick on the Initializr dependency screen. The application needs six Spring AI modules on top of the web starter, each one with a single responsibility. XML <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webmvc</artifactId> </dependency> <!-- Chat model: Anthropic Claude --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-anthropic</artifactId> </dependency> <!-- Embedding model: local ONNX sentence transformer --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-transformers</artifactId> </dependency> <!-- Vector store: PostgreSQL + pgvector --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-vector-store-pgvector</artifactId> </dependency> <!-- RAG advisor --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-vector-store-advisor</artifactId> </dependency> <!-- Document reading (PDF, Word, Markdown, HTML, and more) --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-tika-document-reader</artifactId> </dependency> <!-- Starts the database container on application startup --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-docker-compose</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> <!-- Docker Compose service connections for Spring AI vector stores --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-spring-boot-docker-compose</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> </dependencies> Two models are referenced from the code here. One is the chat model, Claude, which is served from the Anthropic API. The other is the embedding model, which runs locally, right inside the application. We will look at that local embedding model in the next section. The Embedding Model By default, the transformers starter fetches tokenizer.json and model.onnx from Spring AI's own GitHub repository the first time the application starts and then caches them locally. In practice, this default setup is a bit fragile. raw.githubusercontent.com may rate-limit unauthenticated requests, and model.onnx (which is roughly 90 MB) is stored via Git LFS, whose bandwidth quota can run out independently of the ordinary rate limit. When that happens, the endpoint serves the small LFS pointer stub instead of the binary, with a normal-looking HTTP 200, and the failure only shows up later as a cryptic ONNX Runtime protobuf-parsing error rather than a clear download error. The fix is to bundle both files with the application instead of fetching them at startup. So we download them once: Shell mkdir -p src/main/resources/onnx/all-MiniLM-L6-v2 curl -fL -o src/main/resources/onnx/all-MiniLM-L6-v2/tokenizer.json \ https://raw.githubusercontent.com/spring-projects/spring-ai/main/models/spring-ai-transformers/src/main/resources/onnx/all-MiniLM-L6-v2/tokenizer.json curl -fL --http1.1 -o src/main/resources/onnx/all-MiniLM-L6-v2/model.onnx \ https://media.githubusercontent.com/media/spring-projects/spring-ai/main/models/spring-ai-transformers/src/main/resources/onnx/all-MiniLM-L6-v2/model.onnx Then we point the embedding model at these local files in our application.properties, overriding the GitHub-backed defaults: Properties files spring.ai.embedding.transformer.onnx.model-uri=classpath:/onnx/all-MiniLM-L6-v2/model.onnx spring.ai.embedding.transformer.tokenizer.uri=classpath:/onnx/all-MiniLM-L6-v2/tokenizer.json With these two properties set, the application never touches the network for the embedding model, neither on the first run nor on any run after it. The Database The pgvector team publishes a PostgreSQL image with the extension already installed. A compose.yaml in the project root is all we need: YAML services: pgvector: image: "pgvector/pgvector:pg17" environment: - "POSTGRES_DB=nimbusfield" - "POSTGRES_USER=nimbusfield" - "POSTGRES_PASSWORD=nimbusfield" labels: - "org.springframework.boot.service-connection=postgres" ports: - "5432" The labels entry is important. Spring Boot's Docker Compose support auto-detects connection details by matching the image name against a list of well-known images. Plain Postgres is on that list, but pgvector is not, since it is a third-party image. The label tells Spring Boot to treat this container as if it were the official Postgres image, and that is what actually makes the automatic connection wiring work. If we omit it, the container still starts, but Spring Boot never creates a ConnectionDetails bean for it, so the run fails with a connection error rather than falling back gracefully. Because spring-boot-docker-compose is on the classpath, running the application starts the container automatically and injects the connection details. This works the same way on macOS and Windows, as long as Docker Desktop is running. Anyone who prefers to manage the container manually can run the same image with docker run -p 5432:5432 .. and set the datasource properties explicitly. Configuration The complete application.properties, now including the embedding model overrides shown earlier: Properties files spring.ai.anthropic.api-key=${ANTHROPIC_API_KEY} spring.ai.anthropic.chat.model=claude-sonnet-5 spring.ai.anthropic.chat.max-tokens=1024 spring.ai.embedding.transformer.onnx.model-uri=classpath:/onnx/all-MiniLM-L6-v2/model.onnx spring.ai.embedding.transformer.tokenizer.uri=classpath:/onnx/all-MiniLM-L6-v2/tokenizer.json spring.ai.vectorstore.pgvector.initialize-schema=true spring.ai.vectorstore.pgvector.dimensions=384 spring.ai.vectorstore.pgvector.index-type=HNSW spring.ai.vectorstore.pgvector.distance-type=COSINE_DISTANCE logging.level.org.springframework.ai.chat.client.advisor=DEBUG Four details matter here. First, max-tokens is mandatory for the Anthropic API, which caps every response explicitly. Spring AI provides a default, but it is better stated than left implied. Second, the two spring.ai.embedding.transformer.* properties point the embedding model at the local files we bundled in the previous section, instead of Spring AI's own GitHub-backed defaults. See "The Embedding Model" above for why this matters. Third, initialize-schema=true enables the automatic creation of the vector-store table and the required extensions. (Since Spring AI 1.0, this no longer happens silently by default.) Fourth, dimensions=384 must match the embedding model. all-MiniLM-L6-v2 produces 384-dimensional vectors. If the embedding model changes later, the table has to be recreated, because the column type is vector(384). The Documents Two short Markdown files under src/main/resources/docs play the role of the company handbook. remote-work-policy.md Markdown # Nimbusfield Systems Remote Work Policy Employees may work remotely up to three days per week. Remote days must be registered in the portal by Thursday of the preceding week. Working from abroad is permitted for a maximum of 30 calendar days per year and requires prior approval from both the line manager and the People team. travel-expenses.md: Markdown # Nimbusfield Systems Travel and Expenses The daily meal allowance for business trips is 65 EUR in Europe and 80 USD elsewhere. Taxi rides are reimbursed only between airports, hotels, and client sites. Flights longer than six hours may be booked in premium economy. All expense reports are due within 15 working days after the trip via the portal. Thanks to the Tika reader used below, dropping PDFs or Word documents into the same folder works without any code changes. Step 1: Chat Without Retrieval We start with a service that wraps a ChatClient, built once from the auto-configured builder: Java @Service public class AssistantService { private final ChatClient chatClient; public AssistantService(ChatClient.Builder builder) { this.chatClient = builder .defaultSystem(""" You are the internal assistant of Nimbusfield Systems. Answer employee questions precisely and briefly. If you do not know the answer, say so. """) .build(); } public String ask(String question) { return chatClient.prompt() .user(question) .call() .content(); } } And a controller associated with it: Java @RestController public class AssistantController { private final AssistantService assistantService; public AssistantController(AssistantService assistantService) { this.assistantService = assistantService; } @GetMapping("/ask") public ResponseEntity<String> ask(@RequestParam("question") String question) { return ResponseEntity.ok(assistantService.ask(question)); } } Start the application (./mvnw spring-boot:run on macOS/Linux, mvnw.cmd spring-boot:run on Windows) and ask it a handbook question: http://localhost:8080/ask?question=What is the daily meal allowance for business trips in Europe? The response, as we might expect, is: I don't have that information in my available knowledge base. Nimbusfield Systems' specific travel and expense policy—including per diem rates for European business trips—isn't something I can confirm accurately. To get the correct figure, please check: The company's Travel & Expense Policy document (likely on the intranet/HR portal)Your Finance or HR department directlyYour manager, if travel budgets are pre-approved per trip Would you like help with anything else I can assist with more reliably? This gives us a baseline. The model behaves correctly given what it knows, which is nothing at all about this company. Step 2: The Ingestion Pipeline Ingestion follows Spring AI's extract, transform, load structure: a DocumentReader extracts the text, a TextSplitter chunks it, and the VectorStore embeds and stores the chunks. The embedding call happens implicitly inside vectorStore.add() call. The auto-configured TransformersEmbeddingModel is wired into the PgVectorStore and each chunk is embedded into the table. Java @Component public class HandbookIngestion implements ApplicationRunner { private static final Logger log = LoggerFactory.getLogger(HandbookIngestion.class); private final VectorStore vectorStore; private final JdbcTemplate jdbcTemplate; private final Resource[] handbook; public HandbookIngestion(VectorStore vectorStore, JdbcTemplate jdbcTemplate, @Value("classpath:docs/*.md") Resource[] handbook) { this.vectorStore = vectorStore; this.jdbcTemplate = jdbcTemplate; this.handbook = handbook; } @Override public void run(ApplicationArguments args) { Integer count = jdbcTemplate.queryForObject( "select count(*) from vector_store", Integer.class); if (count != null && count > 0) { log.info("Vector store already contains {} chunks, skipping ingestion", count); return; } TokenTextSplitter splitter = TokenTextSplitter.builder() .withChunkSize(300) .build(); for (Resource resource : handbook) { List<Document> documents = new TikaDocumentReader(resource).get(); documents.forEach(doc -> doc.getMetadata().put("source", resource.getFilename())); List<Document> chunks = splitter.apply(documents); vectorStore.add(chunks); log.info("Ingested {} chunks from {}", chunks.size(), resource.getFilename()); } } } The count check makes ingestion idempotent, so restarting the application does not duplicate every chunk. And the source metadata attached to each chunk enables filtered searches later, for instance restricting retrieval to a single document. That same idempotency check has a practical downside worth pointing out. Once the vector store has data, restarting the application will not pick up edits to the handbook files, since the count check short-circuits before the splitter ever runs. To force a clean re-ingestion, for instance after changing a handbook document, tear down the container together with its data volume, not just the container: docker compose down -v The chunk size of 300 tokens is generous for documents this small. The splitter's default of 800 is aimed at larger, real-world content. Chunking is the least exciting and yet the most important knob in a RAG system: chunks that are too large dilute the similarity signals, while chunks that are too small lose their context. It is worth experimenting here: try a few different chunk sizes and see how the system behaves. Just remember to run docker compose down -v between runs, so the vector store is rebuilt from scratch each time. Step 3: Attaching the Retrieval Advisor Now we come back to the plain AssistantService from Step 1 and upgrade it, rather than writing something new. The ChatClient wiring we built earlier stays and what changes is what gets attached to it. Spring AI models the cross-cutting concerns around a chat call as "advisors", which are conceptually close to interceptors. The QuestionAnswerAdvisor embeds the incoming user question, runs a similarity search against the vector store, and appends the retrieved chunks to the prompt before it reaches Claude. Enabling RAG is therefore a change to how the ChatClient is constructed, not to how the request is handled: Java public AssistantService(ChatClient.Builder builder, VectorStore vectorStore) { this.chatClient = builder .defaultSystem(""" You are the internal assistant of Nimbusfield Systems. Answer employee questions precisely and briefly. If you do not know the answer, say so. """) .defaultAdvisors( QuestionAnswerAdvisor.builder(vectorStore) .searchRequest(SearchRequest.builder() .topK(4) .similarityThreshold(0.5) .build()) .build(), new SimpleLoggerAdvisor()) .build(); } topK(4) retrieves at most four chunks per question, and similarityThreshold(0.5) discards weak matches, so an entirely unrelated question augments the prompt with nothing rather than with noise. The SimpleLoggerAdvisor, combined with the DEBUG logging property we set earlier, prints the fully augmented prompt. This is the single most useful debugging tool while tuning retrieval, because it shows exactly what Claude was given. We restart and repeat the same request: http://localhost:8080/ask?question=What is the daily meal allowance for business trips in Europe? The daily meal allowance for business trips in Europe is 65 EUR. Same model, same question, and this time a precise answer grounded in the retrieved handbook chunk instead of a generic deflection. The debug log confirms what is going on behind the scenes: the user question arrives at Claude wrapped in a prompt that contains the retrieved handbook fragments as context. Going Further The default behavior of QuestionAnswerAdvisor is usable, but there are two refinements worth implementing if you want to take this pattern further. The first one concerns grounding. Even with retrieved context, the model may fall back on its general knowledge when the context does not actually contain the answer. The advisor accepts a custom PromptTemplate that controls how the question and the context are merged, and this is the place to enforce stricter behavior. The template must contain the query and question_answer_context placeholders: Java PromptTemplate strictTemplate = PromptTemplate.builder() .template(""" {query} Answer strictly based on the context below. If the context does not contain the answer, reply exactly: "This is not covered by the handbook." --------------------- {question_answer_context} --------------------- """) .build(); QuestionAnswerAdvisor advisor = QuestionAnswerAdvisor.builder(vectorStore) .promptTemplate(strictTemplate) .build(); Asking about, say, the parental leave policy (which is absent from our two files) now produces the fixed refusal instead of an invention. If people are going to rely on it, you want this on. The second refinement could be structured output, and it composes cleanly with retrieval. Declaring a record and calling .entity() instead of .content() gives back a typed object, with Spring AI instructing the model to respond in the matching JSON schema: Java public record HandbookAnswer(String answer, String sourceHint, boolean coveredByHandbook) { } public HandbookAnswer askStructured(String question) { return chatClient.prompt() .user(question) .call() .entity(HandbookAnswer.class); } A last note on the embedding choice. A local MiniLM model is not the strongest embedding model available, and for a large multilingual corpus a hosted embedding API or a bigger ONNX model would retrieve better. This choice is easy to reverse: EmbeddingModel is an interface, swapping the implementation is a matter of a dependency and a property, and the only hard constraint is the one mentioned earlier: the vector dimensions in PGvector have to match whatever the embedding model produces. Conclusion In this article, we built the RAG flow step by step. We started with a plain chat endpoint that could not answer anything about the Nimbusfield handbook, because Claude had never seen it. We then ingested that handbook into PGvector, embedding each chunk locally, and attached Spring AI's QuestionAnswerAdvisor to the same client. That single change was enough to turn a generic model into a service that answers from your own documents. After that, we talked about how we can tighten the grounding, so the model says it does not know when the context has no answer, and pulled the response straight into a typed Java record. If you want to take it further, clone the project, point it at your own documents, apply further the techniques we discussed in the Going Further section, play with different chunk sizes, retrieval settings, and prompts to see how the answers change. The Spring AI documentation goes deeper into advisors, vector stores, and retrieval configuration. The complete, runnable project is available on GitHub.

By Murat Balkan CORE

Building a single AI agent is not usually the hard part. You send a prompt to a model, get a response back, and wire it into your app. Done. The hard part starts when that agent becomes one step in a larger system. A real AI workflow might need to ingest a file, extract text, chunk it, generate embeddings, call an LLM, write results to a database, sync to an external API, and notify a user. Those steps do not behave the same. Text extraction might finish in seconds. An LLM call might take minutes. A sync job might fail because some external API is having a bad day. That is where a lot of "agent" systems stop looking magical and start looking like regular distributed systems. I have seen this fail in boring ways: The same job gets processed twice.A worker writes to the database, then crashes before marking the job complete.A model call runs longer than expected and the message gets picked up again.A retried tool call creates duplicate external writes.Failed jobs sit in processing until someone manually checks the database. None of this is new. AI agents do not magically avoid old infrastructure problems. They still need queues, retries, idempotency, durable state, and monitoring. AWS SQS is a good fit for that middle layer. It is not a full workflow engine. I would not use it for every orchestration problem. But if you need a durable queue between independent agent stages, SQS is simple, reliable, and usually enough. The Coordination Problem A basic multi-stage AI workflow often looks like this: Plain Text Input source -> ingestion -> processing -> generation -> sync The first version is usually a database table with a status column. That works for a while. Then concurrency shows up. Two workers read the same pending row. A process crashes and leaves a job stuck in processing. Someone adds sleep(30) because the previous step "usually finishes by then." That last one is the kind of fix that works just long enough to become a production bug. A queue gives each stage a cleaner boundary. One stage publishes work. Another stage consumes it. If the next stage slows down, the queue absorbs the backlog instead of forcing the whole pipeline to wait. Plain Text Input Source -> ingest_queue -> Ingestion Worker -> chunk_queue -> Chunking Worker -> embedding_queue -> Embedding Worker -> summary_queue -> Summary Worker -> sync_queue -> Sync Worker Now ingestion can scale separately from summarization. If LLM generation is slow, messages pile up in summary_queue. That is not automatically a failure. That is what the queue is there for. A failed summary worker does not corrupt the whole workflow. The message can be retried. If it keeps failing, it moves to a dead letter queue. Standard Queues vs. FIFO Queues SQS gives you two main queue types: standard queues and FIFO queues. Standard Queues Standard queues give at-least-once delivery and best-effort ordering. A message can be delivered more than once. Messages may not arrive in the exact order sent. That sounds scary, but most background AI work should already handle this. Use standard queues for work like document processing, embedding generation, batch classification, independent user requests, and webhook processing. For these jobs, throughput matters more than strict ordering. FIFO Queues FIFO queues preserve ordering within a MessageGroupId and support deduplication. Use when sequence actually matters: conversation turns, per-user workflows, ordered state transitions. Python response = sqs.send_message( QueueUrl=queue_url, MessageBody=json.dumps(payload), MessageGroupId=payload["user_id"], MessageDeduplicationId=payload["task_id"] ) Be careful with the group ID. If every message uses the same MessageGroupId, you have serialized the whole queue by accident. Give each conversation, user, or workflow its own group ID so you preserve ordering per entity while allowing parallelism across different ones. My default rule: start with standard queues unless ordering is clearly required. Then make the handler idempotent. That matters more than the queue type. Ensuring Idempotency in Your Agent Flow Idempotency means the same task can run more than once without creating duplicate or incorrect side effects. This is the part I would not skip. SQS standard queues use at-least-once delivery, so duplicates are part of the contract. But this matters even more with AI workloads because model calls are expensive and outputs can be non-deterministic. Retrying the same prompt may cost money and return a different answer. Retrying the same tool call may send a duplicate email or write a second database row. The basic pseudo workflow: Plain Text receive message check if task already completed if completed, delete message and exit if not completed, process task store result delete message Simple version: Python def handle_message(message, store, sqs, queue_url): payload = json.loads(message["Body"]) task_id = payload["task_id"] if store.already_completed(task_id): sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=message["ReceiptHandle"]) return {"status": "skipped", "task_id": task_id} result = run_agent_logic(payload) store.mark_completed(task_id, result) sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=message["ReceiptHandle"]) return {"status": "completed", "task_id": task_id} The store can be Postgres, DynamoDB, Redis, or anything durable with atomic writes. For Postgres, a unique constraint saves you: SQL CREATE TABLE agent_task_results ( task_id TEXT PRIMARY KEY, status TEXT NOT NULL, result JSONB ); INSERT INTO agent_task_results (task_id, status) VALUES ($1, 'processing') ON CONFLICT (task_id) DO NOTHING; If the insert succeeds, this worker owns the task. If it does nothing, another worker already claimed or completed it. The Failure Case I Designed Around Plain Text summary_queue -> Summary Worker -> Postgres -> sync_queue The summary worker receives a message, calls an LLM, writes the summary to Postgres, then deletes the SQS message. Now suppose the worker writes to Postgres but crashes before deleting the SQS message. From SQS's point of view, the job never finished. After the visibility timeout expires, another worker receives the same message and runs the task again. Without idempotency, that retry may call the LLM again, generate a slightly different summary, and write a second result. A safer handler checks whether model output already exists before calling the model: Python def summary_handler(payload, store): task_id = payload["task_id"] existing = store.get(task_id) if existing and existing.get("model_output"): summary = existing["model_output"] else: text = load_text(payload["input"]["text_uri"]) summary = call_llm(text) store.save_model_output(task_id, summary) store.save_final_result(task_id, {"summary": summary}) return {"next_stage": "sync", "next_input": {"summary_task_id": task_id} That avoids repeating the expensive part if the first attempt already got that far. Visibility Timeout When a worker receives a message, SQS hides it from other workers for the visibility timeout. If the worker finishes, it deletes the message. If the worker crashes, the message becomes visible again after the timeout expires. Too short: another worker receives the same message while the first is still running. Duplicate execution. Too long: failed jobs take too long to retry. Plain Text visibility_timeout = 2x to 5x expected processing time Reference: Metadata validation: 30-60 secondsEmbedding generation: 1-5 minutesLLM-heavy summary: 5-15 minutesLong document analysis: 15+ minutes with heartbeat For long-running tasks, extend visibility: Python sqs.change_message_visibility( QueueUrl=queue_url, ReceiptHandle=receipt_handle, VisibilityTimeout=extension_seconds ) The message should describe the work, not carry the workload. Bad: JSON {"task_id": "123", "full_pdf_text": "... thousands of lines ..."} Better: JSON { "task_id": "123", "stage": "summarize", "input": {"document_uri": "s3://bucket/docs/input.pdf"}, "metadata": {"user_id": "789", "priority": "normal"} } Store large files in S3. Send references through SQS. Do not let the queue become your storage layer. Dead Letter Queues A DLQ captures messages that fail repeatedly. Without one, poison messages cycle forever. Python sqs.set_queue_attributes( QueueUrl=main_queue_url, Attributes={ "RedrivePolicy": json.dumps({ "deadLetterTargetArn": dlq_arn, "maxReceiveCount": 5 }) } ) Use 3-5 as a starting point. A DLQ is not a trash bin - it's an alert. AI-Agent-Specific Failure Modes Duplicate LLM calls: Bigger bill, possibly different answer. Use task_id as idempotency key.Non-deterministic outputs: Store first successful output.Tool-call side effects: Make idempotent.Long-running inference: Use visibility heartbeat. What to Monitor MetricWhyApproximateAgeOfOldestMessageUser-facing delayApproximateNumberOfMessagesVisibleBacklogDLQ message countRepeated failures Two alerts: Oldest message exceeds latency targetDLQ has messages When SQS Is Not the Right Tool RequirementBetter fitSimple async tasksSQSVisual multi-step workflowStep FunctionsComplex event routingEventBridgeHuman approvalsStep Functions I have seen teams burn hours building multi-agent systems with database polling and sleep timers. It works at demo scale. It usually does not survive production traffic. SQS gives you durable message delivery primitives. But the app still needs idempotent handlers, visibility timeout tuning, and DLQ monitoring. Default architecture: One queue between major stagesStandard queues unless ordering requiredEvery handler idempotentLarge payloads outside the queueVisibility timeouts based on real processing timeDead letter queues for failures The difference between an AI demo and a reliable AI system is rarely the prompt. It is the infrastructure around the prompt. Build that layer intentionally.

If you've spent time in enterprise integration, you know the pattern: your platform needs to talk to dozens (sometimes hundreds) of external partner systems, and none of them agree on how they want to be talked to. Some expect SOAP envelopes. Others have moved to REST/JSON. Some want Basic Auth, others OAuth, others a bespoke token scheme. Multiply that by data formats that differ subtly — different XML schemas, different field names, different nesting — and you have a classic integration headache. Back in 2019, I inherited a service in exactly this position. It was a .NET-based SOAP web service acting as a middleware layer: a caller would hit our service, our service would reach out to a customer's web service, retrieve the data, and hand a response back to the original caller. Request and response payloads were transformed using XSLT, with a distinct transformation mapped to each customer's expected schema. It worked — as long as every customer on the other end was also SOAP-based. The problem was that fewer and fewer of them were. New customers were arriving with REST/JSON-only APIs, and the existing architecture had no way to talk to them without either forking the service or building a parallel one from scratch. Neither option scales well when you're maintaining integrations for a large, growing customer base. The Goal: One Service, Protocol-Agnostic Rather than duplicate the service or maintain two codebases, the objective was to make the existing service protocol-agnostic — capable of speaking SOAP to SOAP customers and REST to REST customers, from a single deployable unit, with the protocol decision made dynamically rather than hardcoded per environment or build. That last point matters. This wasn't a matter of standing up REST and SOAP versions side by side. It was one service where the protocol used to talk to any given downstream customer was determined by a database configuration record tied to that customer. Add a new customer, flip a config flag, and the service knows how to reach them — no redeploy, no branching codebase. High-Level Flow Original caller sends a request into the service (as XML/SOAP).The service looks up the target customer's configuration in the database.Based on that configuration: SOAP path: request is transformed via XSLT into the customer's expected SOAP/XML schema and sent as-is.REST path: request XML is transformed via XSLT, then serialized into JSON, and sent as a REST call.The customer's response comes back in whatever format they use (XML or JSON).If REST/JSON: the response is deserialized and converted back into XML.The final XML response — normalized regardless of which protocol was used under the hood — is transformed (again via XSLT) and returned to the original caller. The key design principle: the original caller never needs to know or care what protocol the downstream customer speaks. From their perspective, they send XML and get XML back. All protocol and format negotiation happens inside the service, driven entirely by configuration. Why XSLT Stayed at the Core It might seem odd to keep XSLT as the backbone of a service that's now also fluent in JSON, but there's a good reason: XSLT was already doing the heavy lifting of per-customer schema mapping for the SOAP path, and that logic didn't need to be thrown away when REST support was added — it needed to be extended. For REST customers, the pipeline became: Plain Text Internal XML → XSLT transform (customer-specific schema) → JSON serialization → REST call And on the way back: Plain Text JSON response → XML conversion → XSLT transform (normalize to caller's expected schema) → Response to caller This meant the substantial investment in customer-specific XSLT mappings carried forward. Instead of writing all-new transformation logic for every REST customer, the same schema-mapping approach was reused, with a JSON conversion step bolted onto either end. It also meant that if a customer migrated from SOAP to REST on their side (which happened more than once), the mapping logic didn't need to be re-engineered from scratch — only the transport and serialization layer changed. Designing the Auth Layer Protocol wasn't the only thing that varied by customer — so did authentication. Some customers were still on Basic Authentication. Others required OAuth token flows. A few had proprietary token-based schemes that didn't fit neatly into either category. Rather than hardcode auth logic per customer (which would have recreated the same maintenance problem the protocol switch was meant to solve), the auth layer was built as a pluggable component, selected — like the protocol — via configuration: Basic authentication – credentials stored securely and attached to outbound requests per customer config.OAuth – token acquisition and refresh handled transparently before the outbound call, with tokens cached and renewed as needed.Token-based auth – support for customer-issued tokens that didn't follow standard OAuth flows. The auth layer was designed to sit orthogonally to the protocol layer. A customer's auth scheme and their transport protocol were independent configuration dimensions — a SOAP customer could use OAuth, a REST customer could use Basic Auth, and so on, in any combination. This separation of concerns turned out to be important: protocol and auth requirements rarely change in lockstep when a customer updates their infrastructure, so keeping them decoupled avoided a lot of "well, we changed one thing, but now we have to change three things" maintenance pain. What This Bought Us A few concrete benefits came out of this design: Onboarding speed. Adding a new customer — regardless of whether they were SOAP or REST, and regardless of their auth scheme — became a configuration exercise plus a customer-specific XSLT mapping, rather than a new development effort.Single codebase, single deployment. No fork-and-maintain-two-versions problem. Bug fixes, performance improvements, and security patches applied once, benefited every customer.Future-proofing. As more customers migrated from SOAP to REST over time (which, unsurprisingly, kept happening), the service didn't need architectural rework — just configuration changes and new mappings.Consistent caller experience. The original caller's contract never changed. Internal complexity was fully absorbed by the service; external consumers were shielded from it entirely. Lessons for Anyone Building Similar Middleware If you're facing a similar integration sprawl problem, a few things I'd emphasize: Push protocol and format decisions into configuration, not code. The moment you're writing if (customerX) { ... } else if (customerY) { ... } for protocol handling, you've built something that won't scale past a handful of customers.Don't throw away working transformation logic when you add a new protocol. In this case, the existing XSLT investment for SOAP customers extended cleanly to REST customers with a serialization step added — no need to rebuild schema mapping from scratch.Decouple auth from transport. They're separate concerns, and customers will mix and match schemes in ways your first design probably didn't anticipate.Design for the direction things are moving. In this case, that was SOAP-to-REST migration. Building flexibility in ahead of that trend, rather than reacting to each customer's migration individually, saved a lot of one-off engineering work down the line. The result was a service that started as a single-protocol SOAP integration point and evolved, without a rewrite, into a durable piece of infrastructure that's been reused across multiple products and customer bases well beyond its original scope — which, in hindsight, is the real test of whether an integration architecture was designed well: not whether it solves today's problem, but whether it absorbs tomorrow's without a rewrite.

If you've spent more than a year building enterprise Java apps, you've probably felt this specific kind of pain: a product manager asks for a new search filter, and you open your repository file to find it already has 18 methods. You write number 19, then 20, and somewhere around method 25 you start wondering if there's a better way. There is. It's called Spring Data JPA Specifications, and it's been sitting quietly in the framework the whole time. The Problem With Hard-Coded Query Methods Spring Data JPA's derived query methods are great for simple lookups. findByEmail is clean, readable, and requires zero SQL. But enterprise search rarely stays simple. Your CRM users want to filter customers by name and status. Then by date range. Then by city. Then by a keyword that could match name or email. Before long, you're maintaining a repository that looks like this: Java findByNameAndStatus(...) findByNameAndStatusAndCreatedDateBetween(...) findByNameOrEmailAndStatus(...) findByNameContainingIgnoreCaseAndStatusAndCreatedDateBetween(...) Each new requirement means a new method. The repository becomes a dumping ground. Testing it becomes a chore. Onboarding someone new becomes a conversation about which of the 30 methods to use. Specifications solve this by letting you define small, composable query predicates and combine them at runtime based on what filters the user actually provided. What a Specification Actually Is Under the hood, a Specification wraps the JPA Criteria API, the programmatic, type-safe way to build queries without writing raw SQL or JPQL. The Criteria API is powerful but verbose and tricky to read. Specifications give you that power with a cleaner surface area. Each Specification is just a lambda that produces a predicate: Java (root, query, criteriaBuilder) -> criteriaBuilder.equal(root.get("status"), "ACTIVE") That's it. One condition, one method, composable with anything else. Building It: A Customer Search Example Let's make this concrete. Imagine a Customer entity with name, email, status, and createdDate. Users can filter by any combination of these or none at all. The Entity Java @Entity public class Customer { @Id @GeneratedValue(strategy = GenerationType.IDENTITY) private Long id; private String name; private String email; private String status; private LocalDate createdDate; } A Specifications Utility Class Rather than scattering predicates across services, I keep them in a dedicated class: Java public class CustomerSpecifications { public static Specification<Customer> nameContains(String name) { return (root, query, cb) -> name == null ? null : cb.like(cb.lower(root.get("name")), "%" + name.toLowerCase() + "%"); } public static Specification<Customer> emailContains(String email) { return (root, query, cb) -> email == null ? null : cb.like(cb.lower(root.get("email")), "%" + email.toLowerCase() + "%"); } public static Specification<Customer> statusEquals(String status) { return (root, query, cb) -> status == null ? null : cb.equal(root.get("status"), status); } public static Specification<Customer> createdBetween(LocalDate start, LocalDate end) { return (root, query, cb) -> { if (start == null || end == null) return null; return cb.between(root.get("createdDate"), start, end); }; } } The null returns are intentional; Spring Data JPA ignores null predicates, which means you get automatic "skip this filter if not provided" behavior for free. The Repository Your repository needs to extend JpaSpecificationExecutor: Java public interface CustomerRepository extends JpaRepository<Customer, Long>, JpaSpecificationExecutor<Customer> { } Wiring It Together in the Service Java public List<Customer> searchCustomers(CustomerSearchRequest request) { Specification<Customer> spec = Specification .where(CustomerSpecifications.nameContains(request.getName())) .and(CustomerSpecifications.emailContains(request.getEmail())) .and(CustomerSpecifications.statusEquals(request.getStatus())) .and(CustomerSpecifications.createdBetween(request.getStartDate(), request.getEndDate())); return customerRepository.findAll(spec); } That single findAll call dynamically adapts to whatever combination of filters the caller provides. No branching logic, no 20 repository methods. When product asks for a fifth filter next sprint, you add one method to CustomerSpecifications and one .and() line in the service. Done. Going Further: OR Conditions, Joins, and Pagination OR Conditions The .or() combinator works exactly as you'd expect. A global search bar that checks name or email: Java Specification<Customer> spec = Specification .where(CustomerSpecifications.nameContains(keyword)) .or(CustomerSpecifications.emailContains(keyword)); Filtering Across Joins If your Customer has a nested Address, you can reach into it without any joins in your service layer: Java public static Specification<Customer> cityEquals(String city) { return (root, query, cb) -> city == null ? null : cb.equal(root.join("address").get("city"), city); } The join happens inside the Specification. Your service code stays clean. Pagination Because JpaSpecificationExecutor exposes a findAll(Specification, Pageable) overload, adding pagination is one line: Java Page<Customer> page = customerRepository.findAll(spec, PageRequest.of(0, 20, Sort.by("name"))); Mistakes I've Seen in the Wild Returning non-null predicates for null filters: This is the most common gotcha. If you forget the null check and return a valid predicate anyway, you'll silently filter out data that should be returned. Always guard at the top of the lambda. Mixing business logic into Specifications: A Specification should do one thing: produce a predicate. I've seen Specifications that log, that call services, that check permissions. Don't. Keep them pure. Creating a single "God Specification" that handles all filters: This trades the bloated repository problem for a bloated Specification problem. Small, single-purpose Specifications stay testable and reusable. A statusEquals Specification can serve your search screen, your reporting module, and your admin dashboard without any of them knowing about each other. Skipping case normalization for string searches: cb.like(root.get("name"), "%dzone%") won't match "DZone" or "DZONE." Always normalize: cb.lower(root.get("name")) paired with a lowercased input. Why This Pays Off Over Time The real dividend from Specifications shows up six months after you introduce them, when requirements change, and they always do. Adding a filter? One new static method, one .and(). Removing a filter? Delete the method and the combinator line. Reusing a filter across two features? Import the same Specification class. Unit testing a filter? Instantiate the Specification, pass a mock CriteriaBuilder, assert the predicate. No Spring context required. In complex enterprise codebases, the kind with multiple development teams, evolving product requirements, and a long maintenance tail, that kind of modularity is worth a lot more than it sounds at first. Final Thought Specifications aren't exotic. They're part of the Spring Data JPA standard library; they work with everything you already have, and they solve a problem that every team with a search screen eventually hits. If your repository is starting to look like an alphabetized index of every filter combination your users have ever requested, it's a good time to make the switch.

Artificial intelligence is rapidly transforming software testing by enabling QA engineers to generate test cases and test plans, automate browser interactions, analyze and debug failures, and execute complex testing workflows using simple natural-language prompts. While cloud-based AI assistants offer impressive capabilities, they often require subscriptions and sharing potentially sensitive application data with third-party services. Running an AI-powered testing assistant locally addresses these concerns by providing better privacy, lower operating costs, and complete control over the testing environment. In this tutorial, we’ll learn how to build our own local AI QA engineer using Docker, Ollama, Qwen3:8b, LibreChat, and Playwright MCP. It will allow us to perform browser automation and interact with web applications using natural language, all without relying on cloud-based AI services. Understanding the Architecture Every interaction begins with the user. For example, a user enters a prompt in LibreChat, such as “Open the Playwright website and click the ‘Get Started’ button.” LibreChat serves as the conversational interface through which users interact with the AI assistant. Rather than processing the request itself, it forwards the prompt to a locally hosted large language model, Qwen3:8b, running via Ollama. After receiving the prompt, Qwen3:8b interprets the user’s intent and generates a step-by-step execution plan. Instead of interacting with the browser directly, the model determines which tools are required and communicates those instructions using the Model Context Protocol (MCP). These MCP requests are handled by the Playwright MCP Server, which acts as the bridge between the language model and the browser. It translates the AI-generated instructions into executable Playwright commands. The Playwright MCP Server then launches a Chrome browser and performs the requested actions. Depending on the prompt, it can navigate to websites, click buttons, complete forms, extract text from web pages, capture screenshots, and execute a wide range of browser automation tasks. Once the browser completes the requested operations, the execution results are returned to Qwen3:8b. The language model analyzes the browser output and transforms the technical details into a clear, human-readable response. LibreChat then presents this response to the user. Instead of displaying raw Playwright logs, it provides a concise summary such as: “Navigation completed successfully. The Playwright website was opened, and the Get Started button was clicked successfully.” This architecture enables browser automation through natural language while ensuring that every component runs locally. As a result, we benefit from enhanced privacy, greater security, and complete control over the entire AI-powered automation workflow. Prerequisites Before getting started, ensure that the following software is installed on your machine: DockerNode.js 20 or higher versionGitOllama We’ll use Docker Desktop to run LibreChat, Node.js to install and run the Playwright MCP Server, Git to clone the required repositories, and Ollama to download and serve the local large language model. Having these tools installed beforehand will make the setup process smooth and straightforward. System Requirements Running a local AI-powered browser automation stack requires a reasonably capable machine. A system with 16 GB of RAM or more is recommended to run Docker containers and the language model efficiently. We’ll also need 20–25 GB of available disk space, preferably on an SSD, to accommodate Docker images and downloaded models. While a dedicated GPU can significantly improve model inference speed, it is entirely optional, and the setup works well on modern CPUs. For this tutorial, I’m using the following configuration: Operating system: macOS (M2 Pro)Memory: 16 GB RAM We can have the same setup on Windows and Linux, with only minor platform-specific differences in the installation steps. Setting Up the Environment for the Local AI QA Engineer Docker, Node.js, and Git are widely used development tools, and detailed installation guides for each are readily available online. Installing Ollama To install Ollama, either download the installer from the official website or use the installation command provided for your operating system. For macOS, it can also be installed using the following Homebrew command: Plain Text brew install ollama Once the installation is complete, it can be verified by running the following command in the terminal: Plain Text ollama --version Installing Qwen3:8b Qwen3:8b is chosen for this setup because it offers a strong balance of reasoning, code generation, and performance, making it ideal for Playwright TypeScript test generation, AI agents, MCP integration, and modern QA automation workflows while running efficiently on a local machine. However, other higher models can also be chosen if you know a better one. Another factor in choosing this model was the available system memory. Since my machine has 16 GB of RAM, some memory also needs to be reserved for other tools used in this setup, such as Docker, LibreChat, and Playwright. We need to start Ollama first by running the following command from the terminal. (It should be kept running in the background): Plain Text ollama serve Open a new terminal and run the following command to pull the Qwen3:8b model: Plain Text ollama pull qwen3:8b It should take some time to complete the pull, as the model is around 5.2GB. Once the download completes, we can check the model by running the command: Plain Text ollama list It should list the model downloaded. Next, we can quickly verify by running the model using the command: Plain Text ollama run qwen3:8b Once the model starts, it will prompt you to enter a query. To verify that everything is working correctly, try a simple prompt such as “What is 2 + 2?”. Observe how the model processes the request and generates its response. If the setup is successful, it should return the correct answer, 4, confirming that the model has been downloaded, installed, and is functioning properly. To stop the model, type “/bye” in the prompt, and it should exit. Qwen3:8b provides a good balance between performance and resource usage, making it a suitable choice for this hardware configuration. If more RAM is available, you can opt for larger LLMs that offer stronger reasoning and coding capabilities. Installing LibreChat With Docker LibreChat is an open-source AI platform that provides a unified and customizable interface for interacting with multiple AI models. It enables us to manage all our AI conversations from a single application while supporting features such as AI agents, Model Context Protocol (MCP) servers, custom tools, and integrations with both local and cloud-based LLMs. LibreChat acts as the front-end chat interface that communicates with the locally running Qwen3:8b model through Ollama. It allows us to execute AI-powered browser automation workflows entirely on our local machine. Follow the steps below to install LibreChat: Step 1: Clone the LibreChat GitHub Repository The repository can be cloned by running the following command: Plain Text git clone https://github.com/danny-avila/LibreChat After cloning the repository, navigate to the LibreChat folder, copy the .env.example file, and create a new .env file from it. Plain Text cd LibreChat cp .env.example .env Let's keep the .env file as it is, using the default values. Step 2: Connect Ollama to LibreChat Ollama can be connected to LibreChat by updating its configuration in the “librechat.yaml” file. The example file is already available in the cloned repo. Run the following command to copy librechat.example.yaml and create librechat.yaml. Plain Text cp librechat.example.yaml librechat.yaml Update the following configuration in the file to connect Ollama to LibreChat: YAML endpoints: custom: - name: "Ollama" apiKey: "ollama" baseURL: "http://host.docker.internal:11434/v1" models: default: - "qwen3:8b" fetch: true titleConvo: true titleModel: "current_model" summarize: false summaryModel: "current_model" modelDisplayLabel: "Ollama" Make sure that this configuration is added to the “custom” block, which falls under the “endpoints” block. This configuration adds Ollama as a custom AI endpoint in LibreChat. The baseURL tells LibreChat where to connect to the Ollama API, while the default model specifies that Qwen3:8b should be used by default. Since LibreChat is running inside a Docker container while Ollama is running directly on the host machine, we use http://host.docker.internal:11434/v1 instead of localhost. The special hostname host.docker.internal allows the Docker container to access services running on the host system, enabling LibreChat to connect to the locally running Qwen3:8b model through Ollama. Setting fetch: true allows LibreChat to automatically detect and display all models available in Ollama. The remaining options configure the user interface by generating conversation titles using the current model, disabling conversation summarization, and displaying the endpoint with the label Ollama in the LibreChat interface. Step 3: Mount the Configuration in the docker-compose-override.yml The docker-compose-override.yml can be copied and created in the same way as we did “librechat.example.yaml”. Plain Text cp docker-compose.override.yml.example docker-compose.override.yml The following block should be updated in the docker-compose.override.yml file. YAML services: api: volumes: - ./librechat.yaml:/app/librechat.yaml This file mounts the custom “librechat.yaml” configuration file into the LibreChat container. By mapping ./librechat.yaml to /app/librechat.yaml, Docker ensures that LibreChat uses the custom configuration each time the container starts. This approach allows us to modify settings, such as custom endpoints and AI models, without rebuilding the Docker image. Step 4: Start the LibreChat Application Using Docker Compose The LibreChat application can be started using the following command: Plain Text docker compose up -d It will take some time for the Docker images to download, and containers will start. Run the following command from the terminal to check the Container status: Plain Text docker ps -a This command displays the status of all Docker containers. If any container is unhealthy or encounters an issue, its status will be clearly indicated in the output. In case any container is unhealthy or encounters an issue, the following command can be run to check its logs: Plain Text docker logs <container name> Once all the containers are started successfully, open a new browser and navigate to http://localhost:3080 to start LibreChat. Since we are accessing LibreChat for the first time, we will be prompted to register and create a new user account. After completing the registration process, we can sign in and start using the application. Step 5: Selecting Ollama > Qwen3:8b Model By default, the gpt-5.5 model is selected. To select the Qwen3:8b model: Click on the gpt-5.5 modelSelect Ollama > Qwen3:8b Once the Qwen3:8b model is selected, we can verify if it is working by sending a simple prompt such as “What is 2+2?” Make sure the command “ollama serve” is already running in the terminal in the background, else the model Qwen3:8b won't work on LibreChat. Once we receive a successful response from the model, we can confirm that the Qwen3:8b model has been configured and integrated successfully with LibreChat. Install Playwright MCP Server Playwright MCP can be installed by running the following command in the terminal: Plain Text npx @playwright/mcp@latest \ --host 0.0.0.0 \ --allowed-hosts "*" \ --port 8931 \ By default, Playwright MCP listens only on localhost, which means applications running inside Docker (like LibreChat) cannot connect to it. Using --host 0.0.0.0 makes the server accessible from Docker containers, while --allowed-hosys "*" allows requests from host.docker.internal instead of restricting access to localhost. Once the Playwright MCP server is started, we can leave it running in the terminal. After the Playwright MCP server starts, it shows the following message at the bottom: “For legacy SSE transport support, you can use the /sse endpoint instead”. We will configure the Playwright MCP server using the SSE (Server-Sent Events) transport. Although Playwright MCP also supports the Streamable HTTP transport, LibreChat currently does not support connecting to it via the /mcp endpoint. Therefore, the SSE transport is used to establish a reliable connection between LibreChat and the Playwright MCP server. Configure Playwright MCP Server in LibreChat Playwright MCP server can be added to LibreChat by updating the following configuration in the “librechat.yaml” file. YAML mcpServers: playwright: type: sse url: http://host.docker.internal:8931/sse timeout: 120000 This configuration registers the Playwright MCP server with LibreChat. The type: sse setting specifies that the connection uses the Server-Sent Events (SSE) transport, while the url points to the Playwright MCP server running on the host machine. The hostname host.docker.internal allows the LibreChat Docker container to communicate with services running outside the container. The timeout: 120000 sets the request timeout to 120 seconds, giving the AI agent sufficient time to complete browser automation tasks before the connection expires. However, the timeout can be extended to 15–20 minutes or more, as there is no harm in doing that. YAML mcpSettings: allowedDomains: - 'host.docker.internal:8931' - 'localhost:8931' The mcpSettings configuration also needs to be added under the ‘actions’ block in the “librechat.yaml” file. The mcpSettings.allowedDomains section defines the list of trusted MCP server endpoints that LibreChat is allowed to connect to. By including both host.docker.internal:8931 and localhost:8931, LibreChat can establish a secure connection to the Playwright MCP server, whether it is accessed from within the Docker container (host.docker.internal) or directly from the host machine (localhost). Any MCP server not included in this list will be blocked, providing an additional layer of security. Restart the LibreChat app so it reads the newly configured Playwright MCP server: Plain Text docker compose restart That, or we can also shut down the already running LibreChat and start it again by using the commands below: 1. To shut down LibreChat: Plain Text docker compose down 2. To start it again: Plain Text docker compose up -d After restarting LibreChat, log in and navigate to the home page, and follow the steps below: Click on the MCP Settings menu on the left-hand menu panel.In the MCP Settings window, click on the “+” button to add MCP. Fill in the details for adding the Playwright MCP server; make sure to add the following settings: MCP server URL: http://host.docker.internal:8931/sseTransport: SSEAuthentication: NoneTick the “I trust this application” checkbox. Click on the “Create” button to save the details. Make sure that the Playwright MCP server is started and running on the terminal as discussed in the earlier section Click Connect for the newly created MCP server to establish the connection and begin using it. If everything is fine, a message should be displayed on successful connection. Understanding Model Context Protocol (MCP) By itself, a large language model (LLM) is limited to generating text. It can answer questions, explain concepts, write code, or summarize information, but it cannot directly interact with external systems or perform real-world actions. Model Context Protocol (MCP) changes this by enabling AI models to communicate with external tools and services through a standardized interface. Instead of simply providing suggestions, an AI model can execute tasks such as interacting with browsers, reading files, querying databases, or creating pull requests. Think of MCP as USB for AI A simple way to understand MCP is by comparing it to the USB standard. Before USB became the universal standard, every hardware manufacturer used its own proprietary connector. Printers, keyboards, cameras, and other peripherals all required different cables and custom software integrations. This made connecting devices unnecessarily complicated. USB solved this problem by introducing a common communication standard. Once both the computer and the device supported USB, they could communicate regardless of the device type. Whether you connected a keyboard, webcam, microphone, or external hard drive, the same protocol handled the communication. MCP brings the same level of standardization to AI systems. Without MCP, every AI application requires building and maintaining custom integrations for every external tool it wants to use. If we switch to a different AI application, those integrations often need to be recreated from scratch, resulting in duplicated effort and increased maintenance. A collection of awesome servers for the Model Context Protocol can be found at mcpservers.org. With MCP, tools expose a common interface that any MCP-compatible AI application can use. The AI model only needs to understand the MCP protocol, while the implementation details are handled by the individual MCP servers. Why MCP Matters for QA Automation For QA Automation Engineers, MCP unlocks the ability to automate complete testing workflows rather than isolated tasks. Consider the following request: “Read the Jira story, generate Playwright tests, execute them, analyze any failures, and create a GitHub pull request.” With MCP, the AI agent can coordinate multiple tools to complete the entire workflow. For example, it can: Read the user story from JiraAccess the application’s source code from GitHubGenerate Playwright TypeScript testsExecute the tests in a real browserCapture screenshots, logs, and execution reportsCommit the generated tests to GitHubUpdate the Jira ticket with the test results Each of these actions may be handled by a different MCP server, such as a Jira MCP server, GitHub MCP server, and Playwright MCP server. From the AI model’s perspective, however, every server is accessed using the same standardized MCP protocol. This standardization is what makes MCP so powerful. Rather than building custom integrations for every tool, AI systems communicate through a single, consistent protocol. As a result, MCP servers for Playwright, GitHub, databases, and many other services can be integrated and used in a uniform, scalable manner, significantly simplifying the development of AI-powered automation workflows. Creating an AI Agent With Playwright MCP Server in LibreChat for Automation Testing Let’s create a new AI Agent for browser automation testing with Playwright MCP using the steps below: Step 1: Click on the Agent Builder menu on the left-hand menu panel. Step 2: Enter the following mandatory details to create a new agent: Name: Provide a meaningful name to the agent.Category: Provide a category to the agent.Model: Select Qwen3:8bMCP Servers: Click on the Add MCP Server Tools button > Select the Playwright MCP Server that we created in the earlier section.Click on the Save button. Step 3: Update the model parameters. Clicking on the Model field, which has Qwen3:8b selected, should open the Model Parameters page. The following parameters can be set using this page: Provider: OllamaModel: Qwen3:8bTemperature: 0.2Top P: 0.85Frequency Penalty: 0.00Presence Penalty: 0.00Reasoning Effort: MediumReasoning Summary: Auto Click on the Save button to set the parameters. Step 4: Setting the instructions for the AI agent. The Following instructions can be pasted into the Instructions field in the Agent Builder window, or a “SKILL.MD” file can be created and uploaded using the Skills section of this agent. Markdown # Skills for the Local AI Agent for automation testing You are an expert QA Automation Engineer controlling a browser through Playwright MCP. Your goal is to execute browser actions safely and reliably. ## Tool Usage Rules - Do not run all MCP tools at the same time - Use only one Playwright MCP tool at a time. - Wait for the result of each tool before deciding the next action. - Never assume the page state. - Inspect the current page before interacting. - Do not start the next MCP tool unless the first one is complete ## Navigation Rules Treat the following actions as navigation-triggering actions: - Clicking Login, Submit, Continue, Save, Next, Checkout, etc. - Clicking any hyperlink. - Form submission. - Any action that changes the URL or reloads the page. - Wait until the page is fully loaded before making another tool call. After any navigation-triggering action: 1. Do not call any DOM inspection tool immediately. 2. Wait until the page has completely loaded. 3. Wait for the URL to stabilize if it changes. 5. Continue only after the new page is available. 6. Never inspect the previous page after navigation. ## Rules for locating web elements - Take a fresh snapshot to inspect the current page - Do not use XPath locator strategy - Use the same field name to locate elements, do not hallucinate and add prefix or suffix to field names - Use Semantic locator strategy: getByRole, getByText, getByLabel, getByPlaceHolder, getByAltText, getByTitle, getByTestId - Never use brittle CSS selectors such as .btn-primary, .container > div:nth-child(2), #content div span, or auto-generated classes. - Avoid nth() unless there is no unique locator. ## Interaction Rules - Verify and confirm that an element exists before interacting. ## Error Recovery If any Playwright tool fails: - Stop issuing new actions. - Inspect the current page. - Check Interaction Rules - Determine whether navigation has occurred. - Retry only if the page state confirms it is safe. - Do not repeat the same action more than once without confirming that the page state has not changed. Never repeat the same click more than once without checking the current page. ## Important If a click causes navigation, always assume the previous execution context has been destroyed. Do not read the DOM until the new page has fully loaded and a fresh snapshot has been obtained. Show a summary of test execution with the step count and pass or fail status - Run only the steps that are provided; do not hallucinate - Any deviation from these rules is not acceptable - Do not generate any additional steps - Always prioritize stability over speed. Providing instructions to an AI agent helps define its behavior, responsibilities, and the boundaries within which it should operate. These instructions act as persistent guidance, ensuring the agent follows consistent practices every time it performs a task instead of relying solely on the user’s prompt. For detailed setup instructions and troubleshooting guidance, refer to the GitHub repository. With these steps, the local AI agent is now ready to take commands. Running the AI Agent for Browser Automation To start using the AI Agent, click on New Chat.Click on the model name dropdown and select My Agents > The name of the agent that you created. Let’s use the following simple prompt and see how it works. Plain Text open http://playwright.dev verify the page title Once the prompt is submitted, we can observe the browser as the AI agent begins executing the task. The agent invokes the Playwright MCP server, which automatically launches a browser and performs the requested actions to navigate to the website and interact with the page. After the task is completed, Qwen3:8b analyzes the outcome and returns the results directly in the LibreChat conversation, demonstrating browser automation powered by Playwright MCP and Qwen3:8b. Let’s run another prompt for a login test scenario: Plain Text Navigate to https://parabank.parasoft.com/parabank/index.htm Locate "Username" field using "name=username" Enter "john" into the "Username" field. Locate "Password" field using "name=password" Enter "demo" into the "Password" field. Locator "Log In" button using "input[type="submit"] Click on the "Log In" button Verify that the "Accounts Overview" page is displayed This prompt also takes some time to understand the request before execution begins. It is important to note that the clearer and more specific the prompt, the more efficiently the AI agent can interpret and execute it. Well-structured prompts reduce ambiguity, minimize the chances of hallucinations, and typically result in faster execution and more accurate outcomes. As a best practice, break complex tasks into clear, sequential instructions whenever possible to improve the agent’s reliability and overall performance. As shown in the screenshot above, the AI agent invoked five tools from the Playwright MCP server to interact with the application and complete the requested workflow. It navigated to the website, located the username and password fields, entered the provided credentials, and submitted the login form. Finally, it verified that the login was successful by confirming that the “Accounts Overview” page was displayed. Since this setup runs entirely on a local machine, the AI agent takes approximately one minute to begin execution and around 4–5 minutes to complete a simple scenario. For more complex scenarios involving multiple steps, validations, or integrations, the AI agent is expected to take longer to analyze the request and complete the execution. But Execution time can be significantly reduced by running the setup on a machine with more powerful hardware, such as additional RAM, a faster CPU, or a dedicated GPU. Watch the step-by-step YouTube tutorial for Building your Local AI QA Engineer. Final Words Building a local AI QA engineer with Docker, Ollama, LibreChat, and Playwright MCP is an excellent way to explore the future of AI-powered software testing while keeping complete control over the data and infrastructure. By running everything locally, we eliminate recurring API costs, improve data privacy, and create a flexible environment for experimenting with AI-assisted browser automation using natural language. This setup is only the beginning of what’s possible. As we become more familiar with MCP and AI agents, the local QA assistant can be extended by integrating tools such as GitHub, Jira, databases, or custom MCP servers to automate even more of the testing workflow. Happy AI-powered testing!!

By Faisal Khatri CORE