DZone Spotlight

High concurrency in Databricks means many jobs or queries running in parallel, accessing the same data. Delta Lake provides ACID transactions and snapshot isolation, but without care, concurrent writes can conflict and waste compute. Optimizing the Delta table layout and Databricks' settings lets engineers keep performance stable under load. Key strategies include: Layout tables: Use partitions or clustering keys to isolate parallel writes.Enable row-level concurrency: Turn on liquid clustering so concurrent writes rarely conflict.Cache and skip: Use Databricks' disk cache for hot data and rely on Delta’s data skipping (min/max column stats) to prune reads.Merge small files: Regularly run OPTIMIZE or enable auto compaction to coalesce files and maintain query speed. Understanding Databricks Concurrency and Delta ACID On Databricks, parallel workloads often compete for the same tables. Delta Lake’s optimistic concurrency control lets each writer take a snapshot and commit atomically. If two writers modify overlapping data, one will abort. Two concurrent streams updating the same partition will conflict and cause a retry, adding latency. Snapshot isolation means readers aren’t blocked by writers, but excessive write retries can degrade throughput. Data Layout: Partitioning vs. Clustering Fast queries begin with data skipping, but physical file layout is critical for high-concurrency, low-latency performance. Partitioning and clustering determine how data is physically stored, which affects both write isolation and read efficiency. Partitioning organizes data into folders and allows Delta to prune by key. Choose moderate cardinality columns if partitions are too fine or there are many tiny files; query performance degrades. Also note that partition columns are fixed; you cannot change them without rewriting data. For example, writing a DataFrame to a date-partitioned Delta table: Python df_orders.write.partitionBy("sale_date") \ .format("delta") \ .save("/mnt/delta/sales_data") This creates one folder per date, which helps isolate concurrent writes and filter pruning. Liquid clustering replaces manual partitioning/ZORDER. By using CLUSTER BY (col) on table creation or write, Databricks continuously sorts data by that column. Liquid clustering adapts to changing query patterns and works for streaming tables. It is especially useful for high cardinality filters or skewed data. For example, write a Delta table clustered by customer_id: Python df_orders.write.clusterBy("customer_id") \ .format("delta") \ .mode("overwrite") \ .saveAsTable("customer_orders") This ensures new data files are organized by customer_id. Databricks recommends letting liquid clustering manage layout, as it isn’t compatible with manual ZORDER on the same columns. Databricks also offers auto liquid clustering and predictive optimization as a hands-off approach. It uses AI to analyze query patterns and automatically adjust clustering keys, continuously reorganizing data for optimal layout. This set-it-and-forget-it mode ensures data remains efficiently organized as workloads evolve. Row-Level Concurrency With Liquid Clustering Multiple jobs or streams writing to the same Delta table can conflict under the old partition level model. Databricks ' row-level concurrency detects conflicts at the row level instead of the partition level. In Databricks Runtime, tables created or converted with CLUSTER BY automatically get this behavior. This means two concurrent writers targeting different customer_id values will both succeed without one aborting. Enabling liquid clustering on an existing table upgrades it so that independent writers effectively just work without manual retry loops. Python spark.sql("ALTER TABLE customer_orders CLUSTER BY (customer_id)") Optimizing Table Writes: Compaction and Auto-Optimize Under heavy write loads, Delta tables often produce many small files. Small files slow down downstream scans. Use OPTIMIZE to bin-pack files and improve read throughput. For example: Python from delta.tables import DeltaTable delta_table = DeltaTable.forName(spark, "customer_orders") delta_table.optimize().executeCompaction() This merges small files into larger ones. You can also optimize a partition range via SQL: OPTIMIZE customer_orders WHERE order_date >= '2025-01-01'. Because Delta uses snapshot isolation, running OPTIMIZE does not block active queries or streams. Automate compaction by enabling Delta’s auto-optimize features. For instance: SQL ALTER TABLE customer_orders SET TBLPROPERTIES ( 'delta.autoOptimize.autoCompact' = true, 'delta.autoOptimize.optimizeWrite' = true ); These settings make every write attempt compact data, preventing the creation of excessively small files without extra jobs. You can also set the same properties in Spark config: Python spark.conf.set("spark.databricks.delta.autoOptimize.autoCompact", "true") spark.conf.set("spark.databricks.delta.autoOptimize.optimizeWrite", "true") Additionally, schedule VACUUM operations to remove old file versions. If you set delta.logRetentionDuration='7 days', you can run VACUUM daily to drop any files older than 7 days. This keeps the transaction log lean and metadata lookups fast. Speeding Up Reads: Caching and Data Skipping For read-heavy workloads under concurrency, caching and intelligent pruning are vital. Databricks' disk cache (local SSD cache) can drastically speed up repeated reads. When enabled, Delta’s Parquet files are stored locally after the first read, so subsequent queries are served from fast storage. For example: Python spark.conf.set("spark.databricks.io.cache.enabled", "true") Use cache-optimized instance types and configure spark.databricks.io.cache.* if needed. Note that disk cache stores data on disk, not in memory, so it doesn’t consume the executor heap. The cache automatically detects file changes and invalidates stale blocks, so you don’t need manual cache management. Delta also collects min/max stats on columns automatically, enabling data skipping. Queries filtering on those columns will skip irrelevant files entirely. To amplify skipping, sort or cluster data by common filter columns. In older runtimes, you could run OPTIMIZE <table> ZORDER BY (col) to improve multi-column pruning. With liquid clustering, the system manages this automatically. Overall, caching plus effective skipping keeps concurrent query latency low. Structured Streaming Best Practices Delta optimizations apply equally to streaming pipelines. In structured streaming, you can use clusterBy in writeStream to apply liquid clustering on streaming sinks. For example: Python (spark.readStream.table("orders_stream") .withWatermark("timestamp", "5 minutes") .groupBy("customer_id").count() .writeStream .format("delta") .outputMode("update") .option("checkpointLocation", "/mnt/checkpoints/orders") .clusterBy("customer_id") .table("customer_order_counts")) This streaming query writes to a table clustered by customer_id. The combination of clusterBy and auto-optimize means each micro batch will compact its output, keeping file counts low. Also, tune stream triggers and watermarks to match your data rate. For example, use maxOffsetsPerTrigger or availableNow triggers to control batch size, and ensure your cluster has enough resources so streams don’t queue. Summary of Best Practices Use optimized clusters: Choose compute-optimized instances and enable autoscaling. These nodes have NVMe SSDs, so file operations can scale across workers.Partition/cluster wisely: Choose moderate cardinality partition keys and prefer liquid clustering for automated, evolving layout.Enable row-level concurrency: With liquid clustering or deletion vectors, concurrent writers succeed at the row level without conflict retries.Merge files proactively: Regularly OPTIMIZE or turn on auto-compaction so file sizes stay large and IO per query stays low.Cache and skip: Leverage Databricks' SSD cache for hot data and rely on Delta’s skip indexes to reduce I/O for frequent queries.Maintain and tune: Run VACUUM to purge old files and tune streaming triggers so micro-batches keep up under load.Tune Delta log: Set delta.checkpointInterval=100 to reduce log-file overhead, creating fewer checkpoints. Databricks notes that efficient file layout is critical for high-concurrency, low-latency performance. These techniques yield near-linear throughput under concurrency. Teams bake defaults (partitioning, clustering, auto-optimize) into pipeline templates so every new Delta table is optimized by default. Design choices pay off at scale.

Six months ago, your recommendation model looked perfect. It hit 95% accuracy on the test set, passed cross-validation with strong scores, and the A/B test showed a 3% lift in engagement. The team celebrated and deployed with confidence. Today, that model is failing. Click-through rates have declined steadily. Users are complaining. The monitoring dashboards show no errors or crashes, but something has broken. The model that performed so well during development is struggling in production, and the decline was unexpected. I’ve seen this pattern repeatedly while working on recommendation systems at Meta, particularly on Instagram Reels, one of the highest-traffic machine learning surfaces globally. When models fail after deployment, it’s rarely because the model itself is flawed. The problem is that production environments differ fundamentally from the training environment. Production systems are dynamic. Your model doesn’t just make predictions. It influences what users see, which shapes what they click, which generates tomorrow’s training data, which trains future versions of the model. This creates feedback loops that produce failure modes invisible to offline testing, regardless of how thorough your evaluation process is. The Problem With Offline Metrics Offline evaluation assumes a static environment. You split your data, train on one portion, test on another, and use those metrics to predict production performance. This works well for certain applications like spam filters or image classifiers, where predictions don’t significantly affect future inputs. But recommendation systems, ranking algorithms, and decision-making models operate differently. These systems actively intervene in the world. Offline evaluation answers one question: how well does this model reproduce patterns from historical data? Production asks a different question: how well will this model perform when its predictions actively shape user behavior? These questions require different evaluation approaches. In your test set, the data is fixed. Users already behave in specific ways, and your model’s predictions cannot change that. But in production, the model and user behavior interact continuously. The model makes predictions, users respond, their responses generate data, and this data influences future predictions. If your system recommends cooking videos because they showed high engagement, users will engage with cooking videos partly because that’s what you’re showing them. The model interprets this as validation and increases those recommendations, even if users might prefer different content if given the option. Offline metrics also struggle with temporal changes. You might test on February data to simulate March deployment, but your model could run for six months before retraining. During that time, user preferences shift, competitor products change behavior, and new content types emerge. Your offline metrics only simulated one month ahead, not six. Perhaps most importantly, offline evaluation misses long-term consequences. When you optimize for immediate clicks, your metrics reward predictions that maximize short-term engagement. If those predictions damage user trust over months, leading to eventual churn, your test set cannot detect this trade-off. The negative effects appear long after deployment. Offline evaluation remains essential for comparing models and catching obvious problems. The issue is treating strong offline metrics as sufficient proof that a model will succeed in production. Five Production Failure Modes 1. Covariate Drift: Input Distributions Change Covariate drift occurs when input features change their statistical properties while the underlying relationship between features and outcomes stays stable. When Instagram Reels launched in India, the feature distribution shifted substantially. Average video length changed from 15 seconds to 30 seconds. Music genre preferences were completely different. Engagement patterns shifted to different times of day. The model’s learned patterns still applied. Videos matching user preferences still performed well. But the model was now operating in regions of the feature space it rarely encountered during training. You can detect covariate drift when feature statistics diverge from training baselines. Out-of-vocabulary features increase. Feature importance remains stable, but the actual feature values shift. Model predictions often cluster in narrower confidence ranges. Address this through continuous monitoring of input distributions using measures like KL divergence or Wasserstein distance. Use rolling statistics for feature normalization instead of fixed training values. Retrain regularly with recent data. Python # Covariate drift detection def monitor_feature_drift(train_features, prod_features, feature_name): """ Track distribution shifts in input features using KL divergence Returns: drift_score, alert_threshold_exceeded """ from scipy.stats import entropy # Calculate distributions train_hist, bins = np.histogram(train_features[feature_name], bins=50, density=True) prod_hist, _ = np.histogram(prod_features[feature_name], bins=bins, density=True) # KL divergence (add small epsilon to avoid log(0)) kl_div = entropy(train_hist + 1e-10, prod_hist + 1e-10) # Alert if drift exceeds threshold alert = kl_div > KL_THRESHOLD return kl_div, alert 2. Concept Drift: Relationships Change Concept drift happens when the relationship between inputs and outcomes evolves. Six months ago, users engaged heavily with 15-second quick-cut videos. The model learned this pattern. Today, users prefer longer storytelling content. The videos still have the same features (15 seconds, quick cuts), but the relationship between those features and engagement has changed. The model continues recommending quick-cut videos because that’s what training taught it. Users now skip this content. The features look identical, but what they mean has shifted. This appears as declining model performance despite stable input distributions. Feature importance changes dramatically. Calibration breaks down, with predicted probabilities drifting from actual rates. The model makes confident predictions that turn out wrong on recent data. Solutions include time-weighted training where recent examples receive more weight, sliding window retraining that removes outdated patterns, and online learning approaches that continuously adapt. Python # Concept drift detection via prediction calibration def monitor_concept_drift(predictions, actuals, timestamps): """ Detect concept drift by tracking prediction calibration over time Returns: calibration_error, drift_detected """ import pandas as pd from datetime import datetime, timedelta df = pd.DataFrame({’pred’: predictions, ’actual’: actuals, ’timestamp’: timestamps}) # Compare recent week to previous week recent = df[df[’timestamp’] > (datetime.now() - timedelta(days=7))] older = df[(df[’timestamp’] > (datetime.now() - timedelta(days=14))) & (df[’timestamp’] <= (datetime.now() - timedelta(days=7)))] def calibration_error(pred, actual): bins = np.linspace(0, 1, 11) bin_indices = np.digitize(pred, bins) calibration_gaps = [] for i in range(1, len(bins)): mask = bin_indices == i if mask.sum() > 0: predicted_prob = pred[mask].mean() actual_rate = actual[mask].mean() calibration_gaps.append(abs(predicted_prob - actual_rate)) return np.mean(calibration_gaps) recent_error = calibration_error(recent[’pred’], recent[’actual’]) older_error = calibration_error(older[’pred’], older[’actual’]) # Drift if calibration degraded significantly drift_detected = recent_error > (older_error * 1.5) return recent_error, drift_detected 3. Feedback Loops: Models Influence Their Training Data Your ranking model surfaces certain content types. Users engage with them because that’s what you showed. Your logging records this as high engagement. You retrain on this data. The model learns to surface more of that content. The catalog narrows. Diversity decreases. I’ve observed that this reduces content diversity rapidly. Content that starts with low exposure gets few clicks, and the model learns to deprioritize it further. Meanwhile, a few content types get amplified in every recommendation. Warning signs include decreasing diversity in recommendations, increasing concentration in top items, and entire categories dropping to zero exposure despite potential quality. Combat this by forcing exploration. Use strategies like epsilon-greedy or Thompson sampling. Add explicit diversity constraints to ranking. Log propensity scores for debiasing future training. Some teams run separate exploration and exploitation models. 4. Metric Misalignment: Optimizing the Wrong Objective When a measure becomes a target, it often ceases to be a good measure. Optimize for click-through rate, and you might surface clickbait content. Optimize for watch time, and you might prioritize addictive over valuable content. The metric improves while user satisfaction declines. I’ve watched teams celebrate rising engagement metrics while user satisfaction scores fell. The proxy metric was improving while the actual business goal deteriorated. Modern production systems address this through multi-task architectures. Instead of optimizing a single metric, predict multiple signals: immediate engagement, satisfaction ratings, and long-term retention. Combine these through learned reward models or constrained optimization. This teaches the model to balance competing objectives rather than maximizing one proxy. Run A/B tests for weeks, not days. Delayed effects matter substantially. 5. Delayed Effects: Consequences Appear Later Show users low-quality viral content today, boost engagement metrics now, lose their trust over three months as they realize the platform wastes their time. By the time they leave, you’ve retrained the model multiple times on data that said the content was performing well. This is the challenge of decisions that appear positive immediately but cause damage outside your observation window. It shows up in cohort analysis when long-term user value declines despite short-term wins. The solution requires extending evaluation windows to 30, 60, or 90 days. Use survival analysis for churn prediction. Maintain holdout groups for months instead of weeks. This is more expensive and slower, but necessary to catch these effects. Building Resilient Systems Understanding these failure modes enables better system design. Monitor your system, not just your model. Track data distributions, diversity metrics, and concentration ratios alongside prediction accuracy. Set up alerts for drift. Design for feedback from the beginning. Build exploration into ranking. Log information needed for debiasing future training data. Use counterfactual evaluation during development. Align metrics with actual goals. In large-scale systems, predict multiple signals and combine them rather than optimizing a single proxy. Measure effects over realistic time periods. Treat deployment as an intervention. Your model will change user behavior. Run extended A/B tests. Monitor indirect effects. Establish rollback criteria based on meaningful long-term metrics. Build continuous learning into your architecture. Set retraining schedules that match your domain’s pace: daily or weekly for fast-moving systems. Automate drift detection. Keep human review for significant distribution changes. Practical Considerations Think of your model as one component in a dynamic system where inputs, outputs, and the model itself all change together. Offline evaluation measures how well your model fits historical data. Production requires knowing how well it shapes future outcomes. These need different evaluation strategies. Whatever you optimize will improve. Whatever you don’t measure will likely degrade. Choose metrics knowing this. Pre-Deployment Checklist Before deploying, verify these points. Can you detect when production data diverges from training data? Have you identified potential feedback loops and built exploration mechanisms? Are your optimization metrics aligned with long-term goals? Are you measuring effects over sufficient time windows? Did you correct for selection and position bias in training data? What triggers automatic rollback? What’s your retraining schedule? Models fail in production not because of poor design. They fail because production environments differ fundamentally from training environments. The gap between offline success and production performance is structural, not a bug to fix. It requires system-level thinking, feedback-aware design, and continuous adaptation. Build systems that expect feedback, monitor for drift, optimize for long-term goals, and adapt continuously. In production, your model isn’t just predicting. It’s also changing what it will predict next.

Most organizations have poured heavy capital into endpoint automation. That investment has yielded partial results at best. IT teams frequently find themselves trapped maintaining the very scripts designed to save them time. Recent data from the Automox 2026 State of Endpoint Management report reveals that only 6% of organizations consider themselves fully automated. Meanwhile, 57% operate as partially automated using custom workflows. This setup still depends too heavily on people stepping in and undermines the whole point of automation in the first place. That’s why the industry is moving toward autonomous endpoint management systems that can enforce policies, catch configuration drift, and fix issues on their own without someone having to manually kick things off. The Partial Automation Trap Current automation efforts fall short of enterprise requirements. Traditional endpoint tools fail to match the pace of hybrid work and escalating compliance demands. When environments change, hardcoded scripts break. When key staff resign, organizations lose the undocumented knowledge required to maintain those workflows. Rigid systems cannot adapt to novel conditions. Teams still rely heavily on scripts and manual work, with patching and visibility tools seen as the biggest automation wins. Data highlights this maturity plateau. While 50% of IT teams automate OS patching in some capacity, this targeted approach ignores visibility gaps across diverse platforms. The Automox report shows 57% of teams rely heavily on custom scripts for recurring tasks. These act as helpful stopgaps but struggle to scale. Another 37% execute manual procedures based on written documentation. Only 23% have fully automated their recurring software deployments, leaving the vast majority exposed. Partial automation is merely a temporary plateau. It reduces manual entry but proves insufficient for closing exposure windows across distributed IT infrastructures. The Trust Barrier to Scaling Automation Even when organizations recognize the necessity of scaling their capabilities, deep-seated hesitation stalls progress. The barrier is not a failure to understand the value. The issue is risk amplification. "It's one thing to be wrong. It's a whole other thing to be wrong at scale," notes Jason Kikta, Chief Technology Officer at Automox. "If I'm wrong on an individual computer, that's a problem. If I'm wrong on the entire network, I might get fired. If I'm wrong for a day on a backup, that's not good. If I'm wrong for three months, that might end the company. And so that's where people's fears take them." This fear is entirely rational. Automation applied across thousands of assets amplifies both operational benefits and potential errors. The Automox report quantifies these concerns regarding autonomous adoption. Data privacy and security implications worry 46% of IT leaders. The risk of incorrect or unauthorized system changes holds back another 44%. Decision-makers also cite limited trust in AI-driven recommendations (36%). One of the biggest operational challenges, according to them, is not being able to clearly see what automated systems are doing in real time (36%). Another issue is seen in having to rely on algorithmic decisions that often feel like a black box (34%). Organizations need to provide solutions to these issues. They must show their IT teams that automated changes will remain controlled, transparent, and not be allowed to run unchecked. Guardrails Enable Scale Organizations overcome adoption hesitation by implementing strict operational boundaries. Guardrails act as the primary enabler for scale — not an obstacle to speed. Industry best practices from Datto emphasize testing patches before deployment. Datto also recommends using phased rollouts and maintaining rollback capabilities. With these mechanisms, organizations can expand automation confidently because they know they can intervene, verify, and recover immediately. IT leaders demand these safeguards before ceding control. Automox’s data shows that requested protections include automatic rollback (43%), the ability to pause or override anytime (42%), role-based access controls and audit logs (42%), and approval workflows for critical assets (41%). Control over when agent updates apply is highly important to 74% of respondents. But another 46% expressed strong concern regarding unauthorized device actions. The operating philosophy shifts to a pragmatic baseline: trust but verify. Even when automation works perfectly, you check in on it. What Autonomous Endpoint Management Actually Delivers Autonomous endpoint management (AEM) represents the convergence of visibility, policy enforcement, and adaptive response. Rather than replacing human judgment, it removes technicians from repetitive decision loops where raw speed dictates security outcomes. AEM platforms deliver continuous monitoring, AI-assisted insight, and integrated operations workflows that translate telemetry into timely decisions. These systems monitor environments around the clock. A simple way to think about it is as a self-healing endpoint defense layer for your organization. The platform identifies vulnerabilities and pushes out the required fixes automatically so IT teams don’t have to manually trigger every response. Policy-driven automation doesn't sideline human oversight; it actually gives IT personnel the speed to make decisive moves. Automox asked teams which single task they would automate today. Patch installation led the pack at 39%, followed by automating rollbacks (21%) and managing approvals (20%). AEM delivers these exact capabilities seamlessly. The Automation Ceiling Is Real, Autonomy Breaks Through It Partial automation serves as a temporary stopping point rather than a permanent end state. Organizations stuck at the script-and-schedule level face the same exposure risks as those with zero automation in place. They simply manage a higher degree of infrastructure complexity. AEM represents the definitive next stage of maturity for IT operations. These policy-driven systems continuously maintain the desired security state across distributed assets without requiring constant human oversight, transforming reactive defense into sustainable operational resilience.

By Anton Lucanus CORE

It's not a theoretical scenario. The cluster health checks all come back "green." Node status shows Ready across the board. Your monitoring stack reports nominal CPU and memory utilization. And somewhere in a utilities namespace, a container has restarted 24,069 times over the past 68 days — every five minutes, quietly, without triggering a single critical alert. That number — 24,069 restarts — came from a real non-production cluster scan run last week, an open-source Kubernetes scanner that operates with read-only permissions — it can see the state of the cluster, but it cannot and did not change a single thing. The failures we found were entirely of the cluster's own making. The namespace it lived in showed green in every dashboard the team monitored. No alert had fired. No ticket had been created. The workload had essentially been broken for over two months, and the cluster's observability layer had communicated exactly nothing about it. This is not a tooling failure. It is an architectural characteristic of how Kubernetes surfaces health information — and understanding that characteristic is what separates reactive incident response from operational awareness. The Illusion of Cluster Health Kubernetes communicates health through a layered abstraction. At the top of that abstraction — the level most teams observe — are node status, pod phase, and deployment availability. These signals are accurate and fast. They answer one question well: Is the cluster currently able to run workloads? What they do not answer is whether the workloads running on it are actually functioning. A pod in CrashLoopBackOff is, from Kubernetes' perspective, operating normally. The controller is doing exactly what it was designed to do: restarting the failed container on an exponential backoff schedule. The pod exists. The namespace exists. The deployment reports its desired replica count. If your alerting threshold for restart counts is set to a reasonable number — say, 50 or 100 restarts — a workload that has been failing continuously for months will eventually coast past that threshold and simply become background noise. This is not an edge case. In the scan that produced the 24,069-restart finding, there were fourteen additional containers in CrashLoopBackOff state across multiple namespaces, with restart counts ranging from 817 to 23,990. All of them were in a non-production environment. All of them had been failing for between three and sixty-eight days. The cluster health summary: nominal. Why Control Plane Signals Lag Runtime Reality The control plane knows what state it has requested. It reconciles against that desired state continuously. What it cannot observe — by design — is whether the application inside a running container is doing what it is supposed to do. This creates a specific and predictable gap. Kubernetes will tell you a pod is Running. It will not tell you that the running pod is connected to a database that stopped accepting connections six hours ago. It will tell you that a container restarted 24,000 times. It will not tell you whether that matters to anyone, or whether the failure has been silently swallowing work since December. The second failure type from the same scan illustrates a different dimension of this gap: A networking component — unschedulable for four days. The control plane recorded the scheduling failure accurately. The cluster health dashboard showed the node pool as healthy because the nodes themselves were healthy. The pod simply could not land on any of them. Whether the existing running replica of this component was operating at reduced capacity, or whether the failure to schedule a replacement had any operational consequence, was not surfaced anywhere in the standard observability layer. (Diagram: Control Plane Signal Timeline — from failure event to alert visibility across CrashLoopBackOff, OOMKill, and Unschedulable scenarios) The OOMKill Signal You Almost Miss Among the fifteen critical findings in the scan was a single OOMKill event in a system namespace: Shell kube-system]/[security-monitoring-pod] └─ Status: OOMKilled | Restarts: 1 | Age: 10h └─ Container killed due to out of memory One restart. Ten hours old. Easy to overlook next to containers with five-digit restart counts. But the significance is different: this is a system-level component — a security monitoring agent — that was killed because it ran out of memory. One restart means it recovered. It also means there was a period, however brief, during which security event collection from those nodes was interrupted. In a compliance-sensitive environment, that gap matters. Not because the sky fell, but because the gap exists and is not logged anywhere that post-incident reviewers would typically look. The restart count is 1. The container is Running. The audit trail of what happened in those nodes during the gap is incomplete. This is precisely why OOMKill events deserve separate attention from CrashLoopBackOff events in incident analysis. The failure mode is different, the cause is different, and the window of exposure is bounded and often short, which makes it easy to dismiss and hard to account for later. The Resource Allocation Gap The resource picture from the same cluster adds a different dimension to the health illusion. The cluster reports 237 CPU cores and 1,877 GB of memory available. Requested allocation sits at 63% of CPU and 15% of memory. Plain Text Cluster Capacity: 237.1 CPU cores 1877.5 GB memory Total Requested: 149.6 CPU cores 293.7 GB memory (63.1%) (15.6%) The memory figure is the more interesting one. 15.6% of available memory is requested across the entire cluster, while multiple namespaces carry an OVER-PROV flag. The over-provisioned namespaces are not requesting too little — they are requesting CPU allocations that suggest the workloads were sized for a traffic profile that no longer exists, or never existed. The scheduler sees requests as the unit of resource accounting. A pod requesting 2.1 CPU cores holds 2.1 cores of schedulable capacity regardless of whether it is actually using 0.3. This matters during incidents specifically because resource headroom feels like a safety margin. A cluster at 63% CPU requested feels like it has room to absorb load spikes. But if the workloads consuming that 63% are predominantly over-provisioned, the actual utilization is substantially lower, and the resource accounting is misleading when you are trying to understand whether a performance problem is capacity-related or configuration-related. (Diagram: Requested vs Actual Resource Utilization — showing the gap between scheduled reservation and real consumption, and how that gap obscures diagnosis during load incidents) What This Breaks in Post-Incident Analysis The consequences of these observability gaps are most visible after incidents, not during them. When a post-incident review asks "how long was this broken?", the answer depends on what signals were recorded and when. A container that has restarted 24,069 times over 68 days was broken on a specific day. Identifying that day requires correlating restart count history, deployment event timestamps, and application logs — none of which are surfaced in standard cluster health views. The cluster remembers the current state. It does not easily tell you when the current state began. For teams using AI-assisted or automated remediation, this gap becomes a reliability problem. Automated systems that trigger on pod status or restart thresholds will respond to symptoms rather than causes. A restart count of 24,069 looks the same to an automation rule as a restart count of 50. The automation cannot distinguish between a container that has been in a known-broken state for months and one that just started failing. Acting on the high-restart pod without understanding its history risks masking a dependency failure, triggering unnecessary rollbacks, or creating the appearance of remediation without actually fixing anything. The deeper issue is causal history. Kubernetes convergence is stateless in a useful sense: the system drives toward the desired state without preserving a record of how it got there. That property is what makes Kubernetes resilient. It is also what makes it difficult to reconstruct a failure timeline after the fact. The cluster that auto-recovered from an OOMKill ten hours ago left no evidence trail that most teams would find without specifically looking for it. What Platform Teams Should Institutionalize The gap described here is not closeable by any single tool. It is a structural property of how cluster health is defined and communicated. But it is manageable if teams build the right habits around it. Restart count history needs a retention policy and a query pattern. A container at 24,069 restarts did not arrive there overnight. Most teams have the data in their metrics store — they simply do not have a standing query or alert that surfaces sustained CrashLoopBackOff conditions as distinct from transient ones. An alert that fires at 100 restarts and resolves when the pod recovers is different from a signal that tracks cumulative restart velocity over a 24-hour window. OOMKill events in system namespaces warrant dedicated alerting. A security agent being OOMKilled is not the same severity event as an application container being OOMKilled, but it is not ignorable. System namespace OOMKills should route to a different channel than application health alerts. Resource allocation audits should be treated as operational hygiene, not optimization exercises. The 63%/15% split between CPU and memory requests on this cluster is not a cost problem — it is a diagnostic problem. When requests do not reflect actual usage, resource-based reasoning during incidents becomes unreliable. Finally, the question "how long has this been broken?" should have a fast answer. If it takes more than five minutes to determine when a CrashLoopBackOff condition started, the observability tooling is not configured to support incident response effectively. That question should be answerable from a single dashboard panel or query without log archaeology. The Honest Question for Your Cluster Every cluster of meaningful age and complexity carries some version of what this scan revealed. The combination of sustained crash loops, scheduling failures, and request/utilization gaps is not unusual — it is the natural state of a cluster that has been operated without systematic health archaeology. The question worth asking of your own environment is not whether these conditions exist. They almost certainly do. The question is whether your current observability layer would surface them before they became incident preconditions — or whether you would find them the same way they were found here: by looking specifically and deliberately, rather than by being alerted. If the answer is the latter, that is where the work is — and it starts with picking a namespace and looking deliberately. The 24,069-restart container in your cluster is waiting to be found. The scan data in this article was collected from a real non-production Azure Kubernetes Service cluster. All namespace and resource names have been anonymized. Findings were produced using opscart-k8s-watcher, a read-only open-source Kubernetes scanner that observes cluster state without making changes. No cluster state was modified during the investigation. Connect: Blog: https://opscart.comGitHub: https://github.com/opscartLinkedIn: linkedin.com/in/shamsherkhan

By Shamsher Khan CORE

An Architect's Guide to 100GB+ Heaps in the Era of Agency In the "Chat Phase" of AI, we could afford a few seconds of lag while a model hallucinated a response. But as we transition into the Integration Renaissance — an era defined by autonomous agents that must Plan -> Execute -> Reflect — latency is no longer just a performance metric; it is a governance failure. When your autonomous agent mesh is responsible for settling a €5M intercompany invoice or triggering a supply chain move, a multi-second "Stop-the-World" (STW) garbage collection (GC) pause doesn't just slow down the application; it breaks the deterministic orchestration required for enterprise trust. For an integrator operating on modern Java virtual machines (JVMs), the challenge is clear: how do we manage mountains of data without the latency spikes that torpedo agentic workflows? The answer lies in the current triumvirate of advanced OpenJDK garbage collectors: G1, Shenandoah, and ZGC. The Stop-the-World Crisis: Why Throughput Isn't Enough Garbage collection is the process of automatically reclaiming memory, but as our heaps grow beyond 50 GB to handle AI inference pipelines and massive event streams, traditional collectors can cause devastating latency spikes. In high-stakes environments, the predictability of pause times is just as critical as raw throughput. To achieve sub-millisecond or single-digit millisecond pauses on terabyte-scale heaps, we have moved beyond the "one-size-fits-all" approach. 1. G1: The Balanced Heavyweight (The Reliable Default) The Garbage-First (G1) collector, introduced in Java 7, was designed to handle large heaps with more predictability than its predecessors. It is now the default for most Hotspot-based JVMs because it self-tunes remarkably well for both stable and dynamic workloads. Architectural Mechanics Region-based heap: Instead of a single monolithic space, G1 divides the heap into fixed-size regions (typically 1 MB to 32 MB). These regions are logically categorized into Young, Old, and Humongous regions (for objects exceeding 50% of the region size).Garbage-first priority: G1 identifies regions with the most reclaimable "garbage" and collects them first, using a cost-benefit analysis to meet user-defined pause-time goals (set via -XX:MaxGCPauseMillis).Incremental compaction: By compacting memory incrementally during "mixed collections," G1 reduces the memory fragmentation that leads to catastrophic Full GC events. Best for: Most enterprise applications that require a balance of good throughput and predictable, manageable pause times. 2. Shenandoah: The Ultra-Low Pause Specialist When single-digit millisecond latency is the non-negotiable requirement, Shenandoah is the surgical tool of choice. Its primary differentiator is that it performs heap compaction concurrently with your application threads, unlike traditional collectors that pause the application to move objects. Architectural Mechanics Forwarding pointers and barriers: Shenandoah uses "forwarding pointers" to redirect object references to their new memory locations while they are being moved. It relies on specialized read and write barriers to intercept memory access and ensure the application always sees the correct location of an object.Concurrent evacuation: Most GCs pause the world to "evacuate" live objects from a region being reclaimed. Shenandoah performs this evacuation while the application is still running, keeping pauses typically under 10 milliseconds regardless of heap size.No generational model: Traditionally, Shenandoah treated the heap as a single space without dividing it into young and old generations, which simplifies implementation and avoids generational GC complexities. Best for: Near-real-time systems where a 100ms pause is a "service down" event. 3. ZGC: Taming Terabytes at Hyperscale The Z Garbage Collector (ZGC) is the "deep iron" solution for the most massive IT estates. It is engineered to handle heaps up to 16 TB while maintaining pause times under 1 millisecond. Architectural Mechanics Pointer coloring: ZGC uses 64-bit object pointers to encode metadata directly into the pointer itself. This metadata includes the Marking State (tracking live objects), Relocation State (tracking moved objects), and Generational State (identifying object age in JDK 21+).ZPages: The heap is divided into memory regions called ZPages, which come in three sizes: small (2 MB) for regular objects, medium (32 MB) for larger allocations, and large (1 GB) for humongous objects. This allows ZGC to manage memory with extreme efficiency at scale.Load barriers: Every memory read is intercepted by a "load barrier" that checks the "colored pointer" to ensure the application interacts only with valid, up-to-date references.Generational ZGC (JDK 21+): The latest evolution partitions the heap into young and old generations, optimizing reclamation for short-lived objects and significantly improving overall throughput. Best For: Hyperscale applications and AI orchestration layers that require sub-millisecond latency on massive datasets. The Architect’s Decision Matrix CollectorMax Heap SupportTypical Pause GoalKey StrategyG164 GB+200ms - 500msRegion-based, incremental compaction.Shenandoah100 GB+< 10msConcurrent evacuation using forwarding pointers.ZGCUp to 16 TB< 1msPointer coloring and concurrent compaction. The "Agentic Strangler" Pattern and Memory Management As an integrator, I often advocate for the Agentic Strangler Fig strategy: wrapping legacy monoliths in AI agents using the Model Context Protocol rather than attempting a "Big Bang" rewrite. However, this "facade" approach creates a new performance bottleneck. If your "Agent Facade" is running on a JVM with untuned garbage collection, the latency of your modernization layer will exceed the latency of the legacy system it is trying to strangle. Using ZGC or Shenandoah in your integration layer ensures that your modern "facade" remains invisible to the user, providing the low-latency "Doing" engine required for the Integration Renaissance. Tuning for the Real World: The "Player-Coach" Playbook As someone who has resolved critical production outages for Global 50 logistics providers through JVM heap dump analysis and GC tuning, I can tell you: the default settings are rarely enough for mission-critical loads. Fix your heap size. Resizing a heap is a high-latency operation. Set your initial heap size (-Xms) equal to your maximum heap size (-Xmx) to ensure predictable allocation from the start.Monitor distributions, not averages. Averages are a lie. A "10ms average" can hide a 2-second spike that kills your API gateway. Track frequency histograms and maximum pause times to understand the true "tail latency" of your system.Use realistic workloads. Synthetic benchmarks are "security theater" for performance. Test your GC strategy under real-world application pressure, accounting for the messy, unoptimized event streams that characterize the Integration Renaissance.Hardware-rooted trust. In high-security environments, remember that identity is the perimeter. Ensure your GC strategy isn't creating side-channel vulnerabilities. Leverage Hardware Roots of Trust (like IBM z16) to ensure your memory-intensive AI agents are governed in a secure "Citadel." Conclusion We can no longer treat garbage collection as a "set-and-forget" background task. In the era of autonomous agents and the Integration Renaissance, your choice of GC defines the reliability of your entire digital workforce. Whether you are balancing throughput with G1, chasing ultra-low latency with Shenandoah, or scaling to the stars with ZGC, the goal is the same: move from systems that merely "Show Me" data to systems that can reliably "Do It For Me" across mission-critical enterprise systems.

As autonomous agents evolve from simple chatbots into complex workflow orchestrators, the “context window” has become the most significant bottleneck in AI engineering. While models like GPT-4o or Claude 3.5 Sonnet offer massive context windows, relying solely on short-term memory is computationally expensive and architecturally fragile. To build truly intelligent systems, we must decouple memory from the model, creating a persistent, streaming state layer. This article explores the architecture of streaming long-term memory (SLTM) using Amazon Kinesis. We will dive deep into how to transform transient agent interactions into a permanent, queryable knowledge base using real-time streaming, vector embeddings, and serverless processing. The Memory Challenge in Agentic Workflows Standard large language models (LLMs) are stateless. Every request is a clean slate. While large context windows (LCW) allow us to pass thousands of previous tokens, they suffer from two major flaws: Recall degradation: Often referred to as “Lost in the Middle,” LLMs tend to forget information buried in the center of a massive context window.Linear cost scaling: Costs scale linearly (or worse) with context length. Passing 100k tokens for a simple follow-up question is economically unfeasible at scale. Long-term memory solves this by using retrieval-augmented generation (RAG). However, traditional RAG is often “pull-based” or batch-processed. For an agent that needs to learn from its current conversation and apply those lessons immediately in the next step, we need a push-based, streaming architecture. Architecture Overview: The Streaming Memory Pipeline To implement streaming memory, we treat every agent interaction — input, output, and tool call — as a data event. These events are pushed to Amazon Kinesis, processed in real-time, and indexed into a vector database. System Interaction Flow The following sequence diagram illustrates how an agent interaction is captured and persisted without blocking the user response. Why Amazon Kinesis for Agent Memory? Amazon Kinesis Data Streams serves as the nervous system of this architecture. Unlike a standard message queue (like SQS), Kinesis allows for multiple consumers to read the same data stream, enabling us to build complex memory ecosystems where one consumer handles vector indexing, another handles audit logging, and a third performs real-time sentiment analysis. Kinesis vs. Traditional Approaches FeatureKinesis Data StreamsStandard SQSBatch Processing (S3+Glue)OrderingGuaranteed per Partition KeyBest Effort (except FIFO)Not applicableLatencySub-second (Real-time)MillisecondsMinutes to HoursPersistenceUp to 365 daysDeleted after consumptionPermanent (S3)ThroughputProvisioned/On-demand ShardsVirtually UnlimitedHigh throughput (Batch)ConcurrencyMultiple concurrent consumersSingle consumer per messageDistributed processing Deep Dive: Implementing the Producer The “Producer” is your Agent application (running on AWS Lambda, Fargate, or EC2). It must capture the raw interaction and a set of metadata (session ID, user ID, timestamp) to ensure the memory remains contextual. Partition Key Strategy In Kinesis, the partition key determines which shard a record is sent to. For agent memory, the SessionID or AgentID is the ideal partition key. This ensures that all interactions for a specific user session are processed in strict chronological order, which is vital when updating a state machine or a conversation summary. Python Implementation (Boto3) Here is how you push an interaction to the stream using Python: Python import json import boto3 from datetime import datetime kinesis_client = boto3.client('kinesis', region_name='us-east-1') def stream_agent_interaction(session_id, user_query, agent_response): # Prepare the payload payload = { 'session_id': session_id, 'timestamp': datetime.utcnow().isoformat(), 'interaction': { 'user': user_query, 'assistant': agent_response }, 'metadata': { 'version': '1.0', 'type': 'conversation_step' } } try: response = kinesis_client.put_record( StreamName='AgentMemoryStream', Data=json.dumps(payload), PartitionKey=session_id # Ensures ordering for this session ) return response['SequenceNumber'] except Exception as e: print(f"Error streaming to Kinesis: {e}") raise e The Memory Consumer: Transforming Data into Knowledge The consumer is where the “learning” happens. Simply storing raw text isn’t enough; we need to perform memory consolidation. This involves: Cleaning: Removing noise, sensitive PII, or redundant system prompts.Summarization: Condensing long dialogues into key facts.Embedding: Converting the summary into a high-dimensional vector. The Lambda Consumer Pattern Using AWS Lambda with Kinesis allows for seamless scaling. When the volume of agent interactions spikes, Kinesis increases the number of active shards (if in On-Demand mode), and Lambda scales its concurrent executions to match. Python import json import base64 import boto3 from opensearchpy import OpenSearch, RequestsHttpConnection # Clients bedrock = boto3.client('bedrock-runtime') def lambda_handler(event, context): for record in event['Records']: # Kinesis data is base64 encoded raw_data = base64.b64decode(record['kinesis']['data']) data = json.loads(raw_data) text_to_embed = f"User: {data['interaction']['user']} Assistant: {data['interaction']['assistant']}" # 1. Generate Embedding using Amazon Bedrock (Titan G1 - Text) body = json.dumps({"inputText": text_to_embed}) response = bedrock.invoke_model( body=body, modelId='amazon.titan-embed-text-v1', accept='application/json', contentType='application/json' ) embedding = json.loads(response.get('body').read())['embedding'] # 2. Store in OpenSearch Serverless (Vector Store) # (Logic to upsert into your vector index goes here) index_memory(data['session_id'], embedding, text_to_embed, data['timestamp']) return {'statusCode': 200, 'body': 'Successfully processed records.'} Managing Memory State: The Lifecycle Memory isn’t binary (present vs. absent). Effective agents use a tiered approach similar to human cognition: working memory, short-term memory, and long-term memory. Tiered Memory Logic Working memory: The current conversation turn (stored in-memory or in Redis).Short-term memory: The last 5-10 interactions, retrieved from a fast cache.Long-term memory: Semantic history retrieved from the Vector Database using Kinesis-driven updates. Advanced Concept: Real-Time Summarization Sharding A common issue with long-term memory is vector drift. Over thousands of interactions, the vector space becomes crowded, and retrieval accuracy drops (O(n) search time, though optimized by HNSW/ANN algorithms, still suffers from noise). To solve this, use a "Summarizer Consumer" on the same Kinesis stream. This consumer aggregates interactions within a window (e.g., every 50 messages) and creates a "Consolidated Memory" record. This reduces the number of vectors the agent must search through while preserving high-level context. Comparative Analysis: Memory Storage Strategies StrategyStorage EngineBest ForComplexityFlat Vector RAGOpenSearch ServerlessGeneral semantic searchLowGraph-Linked MemoryAmazon NeptuneRelationship and entity mappingHighTime-Decayed MemoryPinecone / Redis VLRecency-biased retrievalMediumHierarchical SummaryDynamoDB + S3Large-scale longitudinal historyMediumHybrid (Search + Graph)OpenSearch + NeptuneContext-aware, relational agentsVery High Handling Scale and Backpressure When building a streaming memory system, you must design for failures. Kinesis provides a robust platform, but you must handle your consumers gracefully. Dead letter queues (DLQ): If the Lambda consumer fails to embed a record (e.g., Bedrock API timeout), send the record to an SQS DLQ. This prevents the Kinesis shard from blocking.Batch size optimization: In your Lambda trigger, set a BatchSize. A batch size of 100 is often the sweet spot between latency and cost-efficiency.Checkpointing: Kinesis tracks which records have been processed. If your consumer crashes, it resumes from the last successful sequence number, ensuring no memory loss. Data Flow Logic: The Consolidation Algorithm How do we decide what is worth remembering? Not every "Hello" needs to be vectorized. We can implement a filtering logic in our Kinesis consumer. Performance and Scaling Considerations When calculating the performance of your memory system, focus on the Time-to-Consistency (TTC). This is the duration between an agent finishing a sentence and that knowledge being available for retrieval in the next turn. With Kinesis and Lambda, the TTC typically looks like this: Kinesis ingestion: 20-50msLambda trigger overhead: 10-100msBedrock embedding (Titan): 200-400msOpenSearch indexing: 50-150ms Total TTC: ~300ms to 700ms. Since human users typically take 1–2 seconds to read a response and type a follow-up, a TTC of sub-700ms is effectively "instant" for the next turn in the conversation. Complexity Metrics In terms of search complexity, vector retrieval typically operates at O(log n) using Hierarchical Navigable Small World (HNSW) graphs. By streaming data into these structures in real-time, we maintain high performance even as the memory grows to millions of records. Security and Privacy in Streaming Memory Streaming agent memory involves sensitive data. You must implement the following: Encryption at rest: Enable KMS encryption on the Kinesis stream and the OpenSearch index.Identity isolation: Use AWS IAM roles with the principle of least privilege. The agent should only have kinesis:PutRecord permissions, while the consumer has kinesis:GetRecords and bedrock:InvokeModel permissions.PII redaction: Integrate Amazon Comprehend into your Kinesis consumer to automatically mask Personally Identifiable Information before it reaches the long-term vector store. Conclusion Building a long-term memory system with Amazon Kinesis transforms your AI agents from simple stateless functions into intelligent entities with a persistent "life history." By decoupling memory from the LLM and treating it as a real-time data stream, you achieve a system that is scalable, cost-effective, and deeply contextual. This architecture isn't just about storage; it's about building a foundation for agents that can truly learn and adapt over time, providing a superior user experience and unlocking new use cases in enterprise automation. Further Reading and Resources Amazon Kinesis Data Streams Developer GuideBuilding Vector Search Applications on AWSAmazon Bedrock DocumentationDesign Patterns for LLM-Based AgentsScaling Laws for Neural Language Models

By Jubin Abhishek Soni CORE

I have been writing and building in the AI space for a while now. From writing about MCP when Anthropic first announced it in late 2024 to publishing a three-part series on AI infrastructure for agents and LLMs on DZone, one question keeps coming up in comments, DMs, and community calls: What is the right tool for the job when building with AI? For a long time, the answer felt obvious. You pick an agent framework, write some Python, and ship it. But the ecosystem has moved fast. We now have MCP servers connecting AI to the real world, Skills encoding domain know-how as simple markdown files, and agent scripts that can orchestrate entire workflows end to end. The options are better than ever. The confusion around them is too. I have seen teams spend weeks building a full agent setup for something a 50-line SKILL.md would have solved in an afternoon. I have also seen people reach for Skills when their agent actually needed live data from a real system. And I have watched MCP get used where a plain API call would have been simpler and faster. The problem is not a lack of options. The problem is that most content out there treats MCP, Skills, and Agent scripts as competing choices. They are not. They are different layers of the same stack, and knowing when to use each one is what separates a good AI system from a messy one. In this article, I want to give you a clear, practical breakdown of all three. Not theory. Not slides. Just the kind of thinking you need to make the right call on your next build. Quick Definitions First MCP (Model Context Protocol) is an open protocol that lets AI models connect to external tools, APIs, and data sources through a common interface. Think of it as USB-C for AI. Plug anything in, and the model knows how to use it. Skills are reusable instruction files (usually markdown) that tell an AI agent how to handle a specific type of task. SKILL.md files, best practices, step-by-step guides. They are not code. They are context. Agents with Scripts are programs (Python, Node.js, Bash) where the AI drives a loop: think, act, observe, repeat. The script owns the execution from start to finish. Comparison of MCP, Skills, and Agent with script The Big Comparison DimensionMCPSkillsAgents with ScriptsPrimary purposeTool and API connectivityTask-specific guidanceEnd-to-end task executionWho defines the logicServer builderPrompt engineerDeveloper + LLMHow it runsProtocol-driven callsContext injectionAgent loopState managementStatelessStatelessCan hold stateSetup effortMedium (needs a server)Low (just a markdown file)High (code + infra)ReusabilityHighHighLow to MediumDebuggingNetwork and protocol levelPrompt inspectionCode + LLM tracesBest forLive external dataConsistent output qualityComplex, multi-step work When to Use MCP Use MCP when your AI needs to talk to something outside itself: a database, an API, a file system, a calendar, a CRM. When Anthropic announced MCP in December 2024, the pitch was simple: instead of every team writing custom connectors for every data source, you build one server that speaks a common protocol, and any model can plug into it. That framing still holds. Good fit when: You need live data that is not in the model's training or contextMultiple AI clients need to hit the same tool in the same wayYou want access control at the protocol levelYou are building a platform where tools need to be easy to swap out Real example: You are building a support agent that pulls order status from Shopify and creates tickets in Jira. Set both up as MCP servers. Your agent calls the tool and gets back what it needs. It does not need to know anything about the underlying APIs. Pros: Common, vendor-neutral interfaceKeeps AI logic separate from API integration codeWorks across different models and clients Cons: You need a running MCP server, which means infra to manageAdds a bit of latency on every tool callToo much for simple or one-off integrations When to Use Skills Use Skills when the problem is not about getting data. It is about getting good output. Skills carry the know-how. They are like a senior teammate sitting next to the model, saying: "For this task, here is how we do it." Good fit when: You want repeatable, consistent results across many sessionsThe task has nuance that basic prompting does not catchTeams need outputs that all follow the same structure and toneYou want to document a process without writing any code Real example: Your team writes Word docs all the time: proposals, reports, SOWs. Without a Skill, every output looks different. Add a SKILL.md that defines the structure, tone, and formatting rules, and suddenly every doc comes out clean and consistent. Pros: No infra needed, just a markdown fileEasy to read, version, and updateWorks in the background without extra setup Cons: Cannot actually do things, it only guides how they are doneOnly as useful as the instructions inside itNo replacement for real tool access When to Use Agents With Scripts Use Agents with Scripts when the task has multiple steps and needs real decisions along the way. These are your power tools. I have written before about AI agent architectures on DZone, and the recurring theme is the same: the think-act-observe loop is powerful, but it needs structure, or it gets expensive and hard to debug fast. Scripts give you that structure. You control the flow. The LLM handles the reasoning inside each step. Good fit when: The workflow has if/else logic based on what happens at runtimeYou need to chain multiple tools in a specific orderThe job runs in the background without anyone watchingYou need retry logic, error handling, or a way to track progress Real example: A nightly agent that pulls your GitHub PRs, runs a quick check, posts a summary to Slack, updates a Notion tracker, and sends an alert email for anything critical. That is not a single tool call. It is a full workflow. Skills guide it. MCP servers feed it data. The agent script ties it all together. If you have built process monitoring scripts before, this loop will feel familiar. The same principles from Linux process monitoring apply here: watch what is running, handle failures, and log everything. Pros: Total flexibilityHandles complex if/else logic wellCan hold state, retry failed steps, and recover from errorsCan use Skills and MCP together as building blocks Cons: Most effort to set up and maintainLLM responses mid-loop can be unpredictableHarder to debug (you are tracking prompts, code, and external services at once)Token costs add up fast in long loops Pick your tool They Work Together, Not Against Each Other Here is the thing most people miss: MCP, Skills, and Agents are not competing options. They are layers in a stack. The layers in a stack A solid AI system uses all three together: MCP connects to tools and data sourcesSkills tell the model how to use them wellAgent scripts run the whole show Simple way to think about it: MCP is the hands, Skills are the know-how, and the Agent script is what decides what to do next. Effort vs. Power at a Glance SkillsMCPAgentsSetup effortLowMediumHighOutput controlMediumMediumHighInfra neededNoneServerFull stackAutonomyGuidedTool-drivenFully autonomousLearning curveEasyModerateSteep Conclusion If there is one thing I have learned from years of building and writing about AI systems, it is that complexity is easy to add and hard to remove. Every team I talk to wants to jump straight to agents. That makes sense. Agents feel like the real thing. But many of the problems they are trying to solve do not require an agent. They need better guidance baked into the model, or a clean interface to a tool. Start with Skills. They cost nothing, take an hour to write, and make your AI smarter right away. Then bring in MCP when your agent needs to reach outside itself and connect to real systems. Use Agent scripts when you have a genuine multi-step workflow that needs to run on its own and handle failures gracefully. This is not a new idea either. Look at how automation has evolved in infrastructure work. In my recent piece on how IaC evolved to power AI workloads, the pattern is identical: you start simple, layer in tooling as complexity grows, and resist the urge to over-engineer from day one. The same thinking applies here. If you want to go deeper on the tooling side, my complete guide to modern AI developer tools covers the broader ecosystem, and the AI infrastructure series on DZone goes into how all of these layers fit together at scale. The AI tooling space is moving fast, but the principles are stable. Pick the right layer for the job. Keep your stack simple until it needs to grow. And build things you can actually maintain six months from now.

By Vidyasagar (Sarath Chandra) Machupalli FBCS CORE

Hello, our dearest DZone Community! Last year, we asked you for your thoughts on emerging and evolving software development trends, your day-to-day as devs, and workflows that work best — all to shape our 2026 Community Research Report. The goal is simple: to better understand our community and provide the right content and resources developers need to support their career journeys. After crunching some numbers and piecing the puzzle together, alas, it is in (and we have to warn you, it's quite a handful)! This report summarizes the survey responses we collected from December 9, 2025, to January 27 of this year, and includes an overview of the DZone community, the stacks developers are currently using, the rising trend in AI adoption, year-over-year highlights, and so much more. Here are a few takeaways worth mentioning: AI use climbs this year, with 67.3% of readers now adopting it in their workflows.While most use multiple languages in their developer stacks, Python takes the top spot.Readers visit DZone primarily for practical learning and problem-solving. These are just a small glimpse of what's waiting in our report, made possible by you. You can read the rest of it below. 2026 Community Research ReportRead the Free Report We really appreciate you lending your time to help us improve your experience and nourish DZone into a better go-to resource every day. Here's to new learnings and even newer ideas! — Your DZone Content and Community team

Many enterprises operating with a large legacy application landscape struggle with fragmented master data. Core entities such as country, location, product, broker, or security are often duplicated across multiple application databases. Over time, this results in data inconsistencies, redundant implementations, and high maintenance costs. This article outlines Master Data Hub (MDH) architecture, inspired by real-world enterprise transformation programs, and explains how to centralize master data using canonical schemas, API-first access, and strong governance. Fragmented Master Data In a typical legacy environment, the applications manage the master data differently: Each application owns and maintains its own master tablesMaster definitions diverge over timeFunctional logic is reimplemented repeatedlyChanges require coordination across multiple systems The absence of a single source of truth increases operational risk and slows down innovation. Centralized Master Data Hub The proposed solution is to establish a centralized master data hub that acts as the single source of truth for enterprise-level master data. Below are the key principles for a centralized master data hub: Central ownership of master dataCanonical, version-controlled schemasAPI-only access to master dataStrong governance and auditability Below are the benefits of a centralized master data hub: Single, authoritative source of master dataReduced duplication and inconsistenciesLower maintenance and implementation costsImproved data quality and governanceMedium-risk migration strategy Below are the risks and challenges identified while building a centralized master data hub. These risks can be mitigated through caching, high availability design, and phased adoption. Synchronization between local and centralized mastersIncreased dependency on hub availabilityCareful change management requiredPerformance considerations for high-volume consumers Commonly, creating a scheme from fragmented data would be a challenge (migration from a multi-schema to a single-schema model). To address schema standardization challenges, the MDH is implemented as a centralized database platform with two logical tracks. Track 1: Canonical Schema Creation This track hosts enterprise-wide common masters, such as country, state, currency, branch, location, etc., with the characteristics of: Canonical, normalized schemaExposed via entity-level APIsLightweight master data UIBulk upload support These masters are designed for reuse across the enterprise. Track 2: As-Is Configurable Schemas This track supports application-specific masters that cannot be immediately standardized with the characteristics of: Per-application schemasMinimal or no schema changes initiallyExisting stored procedures can continueGradual migration to API-based access This dual‑track approach minimizes migration risk while supporting long‑term standardization. Master data migration plays a critical role in transitioning from fragmented data sources to a centralized master data hub. A well‑defined, structured migration process enables enterprise teams to establish the hub effectively and ensures seamless adoption by consuming applications. The migration flow outlines the high‑level activities involved in consolidating and standardizing master data. Once the MDH is established, enterprise‑wide access to master data becomes essential. APIs built on top of canonical master entities are consumed by applications across the organization. All consumers access master data exclusively through APIs, eliminating direct database access. This model decouples applications from underlying data structures and enables controlled, scalable evolution of master data. For any enterprise transformation initiative, strong governance is essential to manage master data effectively. The following governance use cases and processes help ensure consistency, control, and long‑term success of the MDH. Change in Data: Adding a new country/location in master data The process steps help the teams to adopt: Raise a service request or ticketReview and approval by the MDH teamUpdate via master data UIAutomatic synchronization to consuming applications Changes in Schema: Adding a new attribute to an existing master The process steps below help the teams to adopt: Change request initiationImpact analysis by MDH administratorsStakeholder review and approvalSchema, API, and UI enhancementsTesting (MDH and impacted systems)Deployment and closure Change in API: Modifying an existing master API, process mirrors schema changes, with additional focus on backward compatibility and consumer impact. Governance is managed by the two approaches: Conclusion Implementing a centralized master data hub based on canonical schemas, API‑driven access, and strong governance provides a scalable and maintainable approach to enterprise master data consistency. When paired with a pragmatic migration strategy, this model enables modernization without disrupting existing application ecosystems. The approach effectively balances standardization and flexibility, making it suitable for complex enterprise environments with multiple consuming applications

By Ravi Kiran Mallidi CORE

APIs are at the heart of almost every application, and even small issues can have a big impact. Data-driven API testing with JSON files using REST Assured and TestNG makes it easier to validate multiple scenarios without rewriting the same tests again and again. By separating test logic from test data, we can build cleaner, flexible, and more scalable automation suites. In this article, we’ll walk through a practical, beginner-friendly approach to writing API automation tests with REST Assured and TestNG using JSON files as the data provider. Data-Driven API Testing With JSON files and TestNG’s @DataProvider The setup and configuration remain the same as discussed in the earlier tutorial. Additionally, the following dependency for the Google-gson library should be added to the pom.xml to handle the JSON files. XML <dependency> <groupId>com.google.code.gson</groupId> <artifactId>gson</artifactId> <version>2.13.2</version> <scope>compile</scope> </dependency> For this demonstration, we will use the POST /addOrder API from the RESTful e-commerce demo application. The API schema is shown below for reference: JSON [ { "user_id": "string", "product_id": "string", "product_name": "string", "product_amount": 0, "qty": 0, "tax_amt": 0, "total_amt": 0 } ] The following are two approaches for handling the JSON file as a data provider: POJO-based (Object-Mapping) approachMap-based (Dynamic Parsing) approach POJO-Based (Object-Mapping) Approach In the POJO-based approach, JSON data is mapped directly to custom Java classes that represent the structure of the API request or response. Each field in the JSON corresponds to a variable in the POJO, making the data easy to read, access, and maintain. This approach is useful for stable APIs where the data format does not change frequently. Creating the POJO Class The following POJO class should be created to map the JSON file fields to the data provider: Java @Getter @Setter @AllArgsConstructor @ToString public class Order { private String user_id; private String product_id; private String product_name; private double product_amount; private int qty; private double tax_amt; private double total_amt; } The Order class maps each field of the JSON file to the request body of the POST /addOrder API. The annotations @Getter and @Setter provided by the Lombok library automatically generate getter and setter methods for all fields at compile time, helping in reducing boilerplate code. The @AllArgsConstructor annotation generates a constructor that accepts all class fields as parameters, making it easy to create a fully initialized order object. Each variable in the class corresponds to a field in the JSON data, such as user_id, product_id, product_name, product_amount, and so on. The JSON data can be automatically mapped to this class using the Google Gson library. The @ToString annotation automatically generates a toString() method. This is required so that the values provided in the Order object are printed correctly after test execution. Creating a Utility to Read JSON Files We need to create a utility method that reads and parses the JSON file, and finally returns the required data for testing. Java public class JsonReader { public static List<Order> getOrderData (String fileName) { InputStream inputStream = JsonReader.class.getClassLoader () .getResourceAsStream (fileName); if (inputStream == null) { throw new RuntimeException ("File not found: " + fileName); } try ( Reader reader = new InputStreamReader (inputStream)) { Type listType = new TypeToken<List<Order>> () { }.getType (); return new Gson ().fromJson (reader, listType); } catch (IOException e) { throw new RuntimeException ("File not found: " + fileName); } } Code Walkthrough The getOrderData() is a utility method that accepts the filename as a parameter. It searches for the specified file in the src\test\resources folder. If the file is not found, it throws a RunTimeException with the human-readable message “File not found.” The file is initially loaded as an InputStream and then converted into a Reader using try-with-resources to read the data. The try-with-resources ensures that the Reader is automatically closed after use. The Google Gson library needs type information to convert JSON into generic objects. It is done using the TypeToken class that tells Google Gson that the target type is List<Order>. Finally, the fromJson() method reads the JSON data from the file, converts it into a List<Order>, and returns it. Creating a DataProvider Method The following data provider method returns the test data from the JSON file as Iterator<Object[]>, which is further consumed by the test. Java @DataProvider (name = "orderData") public Iterator<Object[]> getOrderData () { List<Order> orderList = JsonReader.getOrderData ("orders_data.json"); List<Object[]> data = new ArrayList<> (); for (Order order : orderList) { data.add (new Object[] { order }); } return data.iterator (); } Code Walkthrough A TestNG @DataProvider named “orderData” is defined using this code that supplies test data to test methods. It reads a list of Order objects from the “orders_data.json” file using the JsonReader.getOrderData() method. Each Order is wrapped inside an Object[] and added to a list. Finally, it returns an Iterator<Object[]> so that each test execution receives one Order object at a time. JSON File With Test Data The following JSON file is used for testing the POST /addOrder API: JSON [ { "user_id": "1", "product_id": "1", "product_name": "iPhone", "product_amount": 500.00, "qty": 1, "tax_amt": 5.99, "total_amt": 505.99 }, { "user_id": "1", "product_id": "2", "product_name": "iPad", "product_amount": 699.00, "qty": 1, "tax_amt": 7.99, "total_amt": 706.99 }, { "user_id": "2", "product_id": "2", "product_name": "iPhone 15 PRO", "product_amount": 999.00, "qty": 2, "tax_amt": 9.99, "total_amt": 1088.99 }, { "user_id": "3", "product_id": "3", "product_name": "Samsung S24 Ultra", "product_amount": 4300.00, "qty": 1, "tax_amt": 5.99, "total_amt": 4305.99 } ] Writing the API Automation Test Let’s write the test for the POST /addOrder API that creates orders using the test data supplied from the JSON files using the data provider: Java @Test (dataProvider = "orderData") public void testCreateOrder (Order order) { List<Order> orderData = List.of (order); given ().contentType (ContentType.JSON) .when () .log () .all () .body (orderData) .post ("http://localhost:3004/addOrder") .then () .log () .all () .statusCode (201) .assertThat () .body ("message", equalTo ("Orders added successfully!")); } Code Walkthrough The testCreateOrder() method uses the orderData DataProvider to run the test repeatedly, using a different Order object from the JSON file each time. Before sending the POST request, each order is wrapped in a list with List.of(Order) because the POST /addOrder API expects a list of orders in the request body. The test then checks the response by ensuring the status code is 201 and that the success message “Orders added successfully” is returned. Test Execution When the test runs, TestNG automatically runs the testCreateOrder() method multiple times, each time using a different set of data pulled from the JSON file via the orderData DataProvider. Java Map-Based (Dynamic Parsing) Approach The POJO-based approach is good when the JSON is stable and well-defined. However, it requires continuous updates and maintenance whenever the JSON structure changes, which increases maintenance time and effort. This makes it less suitable for dynamic or frequently evolving JSON files, where even minor changes can break parsing and tests. In such situations, the Map-based approach comes in handy, where we do not need to maintain POJOs for the JSON. It can handle changing or unknown fields dynamically without requiring code changes. Creating the JSON Reader Utility With Java Map Let’s create a new utility method to parse the JSON files dynamically using a Java Map. Java public static List<Map<String, Object>> getOrderData (String fileName) { InputStream inputStream = JsonReader.class.getClassLoader () .getResourceAsStream (fileName); if (inputStream == null) { throw new RuntimeException ("File not found: " + fileName); } try ( Reader reader = new InputStreamReader (inputStream)) { Type listType = new TypeToken<List<Map<String, Object>>> () { }.getType (); return new Gson ().fromJson (reader, listType); } catch (IOException e) { throw new RuntimeException ("Error reading the file: " + fileName); } } Code Walkthrough The getOrderData() method reads a JSON file and converts it to a list of maps using the Google Gson library. Return type: It returns a List of Map<String, Object>, where: Each Map represents one JSON object.Keys are JSON field names.Values are their corresponding values. Loading the JSON file: The file is read from the src\test\resources folder and returns an InputStream. If the file is not found, the inputStream object will be null. In that case, the program throws a RuntimeException with the message “File not found.” Parsing the JSON file: A try-with-resources block is used to safely read the JSON file using a Java Reader, ensuring the stream is closed automatically. It defines the target type as List<Map<String, Object>> using TypeToken and then uses the fromJson() method of the Google Gson library to convert the JSON data into this dynamic structure. If any file-reading error occurs, it throws a runtime exception with a message “Error reading the file” with the file name. JSON File With Test Data The following JSON file is used for testing the POST /addOrder API: JSON [ { "user_id": "1", "product_id": "1", "product_name": "iPhone", "product_amount": 500.00, "qty": 1, "tax_amt": 5.99, "total_amt": 505.99 }, { "user_id": "1", "product_id": "2", "product_name": "iPad", "product_amount": 699.00, "qty": 1, "tax_amt": 7.99, "total_amt": 706.99 }, { "user_id": "2", "product_id": "2", "product_name": "iPhone 15 PRO", "product_amount": 999.00, "qty": 2, "tax_amt": 9.99, "total_amt": 1088.99 }, { "user_id": "3", "product_id": "3", "product_name": "Samsung S24 Ultra", "product_amount": 4300.00, "qty": 1, "tax_amt": 5.99, "total_amt": 4305.99 } ] Creating a DataProvider Method The following data provider method retrieves test data from the JSON file in Iterator <Object[]> format, which is then used by the test method for execution. Java @DataProvider (name = "orderData") public Iterator<Object[]> getOrderData () { List<Map<String, Object>> orderList = JsonReader.getOrderData ("orders_data.json"); List<Object[]> data = new ArrayList<> (); for (Map<String, Object> order : orderList) { data.add (new Object[] { order }); } return data.iterator (); } Code Walkthrough The getOrderData() DataProvider method reads order data from a JSON file and stores it as a List<Map<String, Object>>. It then converts each map into an Object[] and adds it to a list, which is returned as an iterator. This allows TestNG to run the test multiple times using a different set of order data supplied from the JSON file. Writing the API Automation Test Let’s write a test for the POST /addOrder API that creates orders using the test data from the JSON files through the data provider. Java @Test (dataProvider = "orderData") public void testCreateOrder (Map<String, Object> order) { List<Map<String, Object>> orderData = List.of (order); given ().contentType (ContentType.JSON) .when () .log () .all () .body (orderData) .post ("http://localhost:3004/addOrder") .then () .log () .all () .statusCode (201) .assertThat () .body ("message", equalTo ("Orders added successfully!")); } Code Walkthrough The Map<String, Object> order parameter to the testCreateOrder() method represents a single order read dynamically from the JSON file. It is wrapped inside a List<Map<String, Object>> as the API expects an array of orders in the request body, not just a single object. This approach allows the test to stay flexible and work with dynamic JSON data without relying on fixed POJO classes. The test then logs the request and response, verifies that the status code is 201, and verifies the response message confirming that the order was created successfully. Test Execution The following is a screenshot of the test executed using IntelliJ IDE. It shows that the same test was run multiple times using the test data from a JSON file. It can be noted that when we ran the tests using a POJO-based approach, the test data appeared with the POJO name: testCreateOrder[Order(userId…)}. However, using the Map-based dynamic approach, the test data appears directly with the field names as provided in the JSON file. Summary Data-driven API testing with JSON files, REST Assured, and TestNG allows running the same test multiple times using JSON files as input, making tests more reusable and comprehensive. When parsing JSON, POJO-based approaches provide type safety and clear structure but require frequent updates whenever the JSON changes, making them less flexible. In contrast, Map-based (dynamic) parsing is more flexible and low-maintenance, as it can handle unknown or changing fields without modifying code, though it offers less type safety. Choosing between them depends on the API’s stability: use POJOs for fixed structures and Maps for dynamic or evolving JSON data. Happy testing!

By Faisal Khatri CORE