The Problem: Redis Was Getting Expensive, and Nobody Had Noticed

It didn’t start as a Redis problem. I was investigating a high-latency endpoint. One of those tasks where you pull on a thread expecting to find a slow query or an N+1, and end up somewhere completely different. While profiling the request path, I pulled up our Redis instance metrics to check cache behaviour. That’s when it caught my eye.

We were running a large Redis instance, significantly over-provisioned relative to what we actually needed. Nobody had raised an alarm because the cache worked. Hit rates were healthy, latency was acceptable, and the infrastructure cost had grown gradually enough that it never triggered a conversation. Classic boiling frog.

Digging deeper, the root cause was straightforward: we were inserting raw JSON directly into Redis: verbose, deeply nested, full of repetitive field names. Across hundreds of thousands of keys with a 24-hour TTL, that adds up fast. We were consuming close to ~40 GB of memory for data that, fundamentally, didn’t need to be that large.

The high-latency investigation got parked. This was the bigger problem.

The options on the table were:

1. Increase instance size: easy, but a band-aid.
2. Reduce TTLs: would hurt cache hit rates.
3. Prune cached fields: risky without a full audit of all consumers.
4. Compress the values: low risk, high upside, no application changes required.

We went with option 4.

Choosing the Right Compression Algorithm

Not all compression is the same. Before writing a single line of code, we spent time understanding the trade-off space. Picking the wrong algorithm for a cache layer can make things worse, not better.

The Contenders

Gzip (DEFLATE): The default choice for a lot of engineers because it’s everywhere. gzip combines LZ77 with Huffman coding to achieve high compression ratios. The problem is speed. gzip is optimised for ratio, not throughput. Compression can take several milliseconds on a moderately sized JSON payload. For files or HTTP transfer, that’s fine. For a hot cache path executing thousands of times per second, it’s a tax you can’t afford.

Zstd (Zstandard): Meta’s algorithm, designed specifically to close the gap between ratio and speed. At its default level, zstd gets close to gzip’s compression ratio at 5–10x the speed. It also supports dictionary-based compression, which can push ratios even higher for data with shared structure. zstd was a serious candidate for us.

Snappy: Google’s algorithm, built for speed over ratio. Snappy explicitly doesn’t aim for maximum compression. It targets very fast compression with reasonable results and is widely used in systems like LevelDB, Cassandra, and Hadoop. The downside: compression ratios are lower than LZ4 on JSON data, and Python ecosystem support is thinner.

LZ4: Pure speed. LZ4 is designed to saturate memory bandwidth rather than CPU cycles. It can compress and decompress at multiple GB/s on modern hardware. Compression ratio is in the same ballpark as Snappy but generally better on structured text data like JSON.

Why We Chose LZ4

• Decompression speed matters more than compression speed. Every cache read triggers a decompression. Every cache write triggers a compress. But in a read-heavy cache, decompression runs far more often. LZ4’s decompression at ~4 GB/s is essentially free at our scale, adding single-digit microseconds per operation.
• The ratio difference from zstd doesn’t justify the overhead. zstd would have given us perhaps 5–8% better compression on our payloads. That’s real, but not meaningful when LZ4 already gets us into the 60–70% reduction range. We’d be adding CPU overhead for diminishing returns.
• JSON compresses well regardless of the algorithm. Deeply nested JSON with repetitive field names and predictable structure doesn’t need a sophisticated algorithm to shrink significantly. Even a fast one like LZ4 tears through it. You don’t need to reach for gzip unless you’re optimising for absolute minimum storage with no latency constraints.

LZ4 hits the sweet spot: 55–70% size reduction on JSON payloads with compression and decompression times in the microsecond range.

How We Implemented It

The core idea is simple: compress before SET, decompress after GET. The goal was to make this completely invisible to all call sites. No application code should need to know compression exists.

Our codebase already had an BaseCacheHandler abstraction sitting over Redis, providing a consistent interface for every cache key. That was the clean insertion point.

We created a CompressedCacheHandler that extended the base class and wrapped every read and write with LZ4 compression. We paired LZ4 orjson instead of the standard json module. orjson is a Rust-backed JSON serializer that outputs bytes directly, which is exactly what LZ4 expects as input. One fewer allocation, faster serialisation, and cleaner code.

def _compress(data: Any) -> bytes:
    return lz4.frame.compress(orjson.dumps(data))

def _decompress(data: bytes) -> Any:
    try:
        return orjson.loads(lz4.frame.decompress(data))
    except (RuntimeError, TypeError, orjson.JSONDecodeError):
        return data  # legacy uncompressed key, return as-is

The fallback in _decompress handles backward compatibility. Since LZ4 frame bytes can never be mistaken for valid JSON, the failure is deterministic. Old uncompressed keys return safely, and as they expired through their natural 24-hour TTL, the mixed state resolved itself with zero intervention.

Sharding

We also run a multi-Redis-instance setup with consistent hashing. Rather than entangle sharding logic with compression logic, we composed them via multiple inheritance:

class ShardedCompressedCacheHandler(CompressedCacheHandler, RendezvousSharding):
    def __init__(self, key: str, timeout: int, alias: list | str):
        shard = self._get_shard(alias, key) if isinstance(alias, list) else alias
        super().__init__(key, timeout, shard)

Rendezvous hashing ensures that when a Redis node is added or removed, only the minimum number of keys are remapped. Keeping compression and sharding as separate layers meant both stayed independently testable.

The Results

We rolled out over the weekend. New writes came in compressed, and old keys expired through their TTL. By the end of the window, Redis memory had dropped by roughly 70% with no measurable impact on latency or cache hit rates.

What to Watch Out For

Small values don’t always compress well. LZ4 has frame overhead for very small inputs. If your cached values are under roughly 200 bytes, compression can actually increase size. Worth adding a minimum size threshold before compressing.

The exception-based fallback is intentional, not lazy. Our approach of catching decompression failures to handle legacy keys is safe, specifically because LZ4 frame bytes can’t be mistaken for valid JSON. If you have existing keys in a format that could look like compressed data, use an explicit version prefix instead.

Already-compressed or binary values need special handling. If you cache anything that isn’t plain JSON — protobuf, msgpack, base64-encoded blobs — the exception fallback will silently return raw bytes. Make sure downstream consumers can handle that, or add type detection before compressing.

Monitor compression ratio per key namespace. Some data compresses dramatically. Other data barely shrinks. It’s worth logging the ratio per key prefix in staging to identify where you’re getting the most benefit and where compression is a waste of effort.

Conclusion

What started as a routine latency investigation turned into one of the highest-impact changes we shipped that quarter. We didn’t rewrite anything, didn’t change how data was structured, and didn’t touch a single call site. We just stopped storing data in an unnecessarily large format.

LZ4 compression on JSON-heavy Redis values is one of those rare optimisations where the risk is low, the effort is minimal, and the upside is significant. If you’re storing raw JSON in Redis at scale and haven’t looked at compression yet, it’s worth a few hours of your time.

If you’ve done something similar or hit edge cases I didn’t cover, I’d love to hear about it.

The State of AI Agent Memory Today

In the past year, companies everywhere have been racing to deploy AI agents. Sales qualification, customer support, onboarding, and follow-up tasks that used to require a trained human are getting handed off to bots. And then the metrics don’t move!!

The ROI doesn’t show up. Conversion stays flat. Support deflection looks fine on paper, but customer satisfaction quietly drops. Teams that were proud of what they built six months ago are now quietly wondering what went wrong.
The usual diagnosis: the AI can’t handle objections. It can’t reason through pushback. It can’t convince people the way a trained human rep can.

That’s true, but it’s the wrong diagnosis.

A great human sales/support agent isn’t better because they’re human. They’re better because they have context. They remember that this person called in three weeks ago, frustrated about pricing. They know the company just changed plans. They caught a detail in a prior conversation that tells them exactly how to frame this one. They’re not smarter, they’re informed.

Your AI agent isn’t losing because it’s a machine. It’s losing because it starts every single conversation knowing nothing.

Every interaction is session zero. The user who explained their situation last Tuesday? Never met them. The person who was almost ready to convert two weeks ago? Stranger. The customer who mentioned a very specific constraint that would’ve changed your entire pitch? Gone.

This isn’t a failure of the LLM. It’s a failure of infrastructure. And right now, most teams are solving it badly, or not solving it at all.

The Context Window Is Not Memory

The first thing teams try is history injection, pulling the last N conversations, stuffing them into the system prompt, and letting the model work it out.

This is understandable. It sort of works. And it creates four problems you don’t notice until you’re in production.

• Cost: Every conversation now costs as much as all previous conversations combined. Ten sessions in, you’re passing ten sessions of text on every single request.
• Latency: More tokens mean a slower response. At scale, users feel this.
• Noise over signal: A raw transcript is mostly filler, pleasantries, false starts, and repetition. The facts buried in it, like who this person is, what they need, and what’s changed, don’t surface. You’re asking the model to read a document when it needs a briefing.
• No cross-channel coherence: A user who spoke to your voice agent on Monday and opened a chat on Thursday is two separate people in your system. Their histories don’t connect. There’s no concept of identity, just sessions!!

This is what we learned from our experience. The context window is a working memory tool. It was never meant to be long-term memory.

Vector Search Alone Doesn’t Solve It Either

The next step teams take is RAG, store conversation history in a vector database, retrieve relevant chunks at query time, and inject them as context. Better. But still incomplete.

• You’re storing the wrong thing: Embedding raw transcripts means you’re searching through noise. The semantically relevant unit is not a paragraph of conversation; it’s an extracted fact. “User prefers async communication” is a better memory than three hundred words of a transcript that implies it.
• Retrieval is probabilistic: You might get the relevant memory. You might not. Whether an agent remembers something about a user depends on whether the right embedding surfaces at the right moment. That’s fragile.
• No rollup: After fifty conversations with a user, you have fifty chunks in a vector store. Every time you need context, you’re re-synthesising who this person is from scratch. There’s no maintained, up-to-date summary, no single answer to “what do we know about this person today?”
• No identity anchor: The vector store has no concept of a person. It has documents. Memories are scoped to whatever key you happened to use, often a session ID, not a stable identity.

There’s No Standard Memory Infrastructure

The deeper issue is that memory for AI agents is treated as an application concern, not an infrastructure concern. Every team builds their own version.

A Redis key per user with a JSON blob that gets appended. A table of past summaries with no deduplication. A prompt template that injects “what we know” from a hand-maintained database.

These work in demos. They break in different ways in production — duplication, drift, orphaned records, no idempotency when events replay, and no way to support multiple conversation sources without forking the logic. There’s no agreed-upon model for what memory is, how it gets updated, and how agents consume it.

Here’s How We Solved It: Building Cerebro

We built Cerebro, a memory microservice for AI agents. Named after Professor X’s machine from X-Men, similar to the movie, our service does the same; it reaches into every past conversation and surfaces what an AI agent needs to know about a person, right now.

The goal was to solve each of these gaps with explicit design decisions, not workarounds.

Anchor Memory to a Person, Not a Session

The foundational shift: memory belongs to a person, not a session.

Every ingest starts with resolving an Identity, a stable record keyed by phone number, email, or account ID. All memories attach to that identity. It doesn’t matter if the conversation came from voice, chat, or email. It doesn’t matter how many sessions there have been. The identity is the anchor.

When an agent needs context, it asks: “What do we know about this person?” and not “What happened in these sessions?”

This one change makes cross-channel memory coherent by design.

Store Facts, Not Transcripts

We used mem0 for memory extraction. When a conversation ends, and the transcript arrives, mem0’s LLM pass extracts atomic facts from it:

“Prefers remote work. Has 3 years of Python experience. Looking for a role in fintech.”

These facts are embedded and stored in a vector database. Not the transcript, the facts. The vector store contains a signal, not noise.

But we don’t rely on retrieval alone. We maintain two main types of insight per identity:

• EPISODE: What we learned from this specific conversation. Tied to that interaction, immutable. A record of what happened.
• CURRENT STATE: A continuously updated rollup of everything we know about this person. One row. Always current. When a user’s situation changes, this row changes. When an agent needs context, it reads this row first.

The current state solves the re-synthesis problem. You never re-derive who a person is from fifty episodes. You maintain the answer.

One Core, Many Sources

Different conversation types need different extraction logic. A support chat and a sales call should produce different kinds of facts. The prompts, the field mappings, the interaction types — all different.

We solved this with YAML pipeline configs. Each source gets a file that declares what kind of interaction it represents and what extraction prompt to use. Cerebro routes each ingest through the right pipeline automatically.

Adding a new conversation source is adding a YAML file. The processing core doesn’t change.

What Changed

Before Cerebro, agents started every conversation cold. No context. Users repeated themselves. Agents couldn’t reference prior interactions, couldn’t build on past conversations, couldn’t detect when something had changed.

After Cerebro, every agent interaction starts with a rich, distilled profile of the person. The agent knows what has been discussed, what has changed, and what matters. Conversations feel like continuations, not restarts.

The shift isn’t just in user experience. It’s in what agents can do. An agent with persistent memory can detect change over time. It can flag when a user’s situation has changed since the last contact. It can personalise responses based on accumulated context, not just the current turn.

Memory is what turns a reactive agent into one that actually knows the people it’s working with.

The Principles That Held

If you’re building something similar, these are the decisions that mattered most.

• Anchor memory to identity, not sessions. Sessions end. People don’t.
• Store facts, not transcripts. The value is in extraction, not retention.
• Maintain a current state. Don’t re-synthesise on every request; maintain the answer.
• Build idempotency in from day one. Duplicate events are not an edge case.
• Separate the pipeline from the core. New sources should be config changes, not code changes.

The human rep who outperforms your agent isn’t smarter. They just remember. Build agents that do the same.

Introduction

In today’s data-first landscape, organizations are under increasing pressure to make decisions in real time. While traditional batch ETL systems have served us well, they come with inherent limitations — data latency, operational complexity, and resource inefficiency. We faced this challenge head-on: a legacy ETL system with fixed batch windows, stale data, and brittle pipelines.

This post outlines our journey of replacing an aging batch ETL pipeline with Apache Flink, a powerful stream processing engine that enabled real-time data availability, lowered costs, and simplified our architecture.

The Bottlenecks of Batch ETL

We began with a conventional batch architecture that processed various datasets through scheduled jobs executed every 24 hours. Over time, this setup became a bottleneck due to several critical issues:

• Data Latency: Business decisions were based on data that was already stale by the time it was available.
• Resource Spikes: Heavy jobs ran in parallel during narrow time windows, leading to massive load peaks and under-utilized resources the rest of the day.
• Maintenance Overhead: Each pipeline had its own logic, scheduling, monitoring, and alerting mechanisms.
• Poor Scalability: Scaling batch pipelines meant increasing compute power, which only delayed job runtimes further.
• Data Drift and Inconsistency: Syncing logic across multiple pipelines was error-prone and fragile.

We realized this architecture could no longer support the dynamic, high-volume, low-latency data demands of our organization.

Why Apache Flink?

We evaluated multiple stream processing frameworks, including Apache Spark Streaming, Kafka Streams, and Flink. Apache Flink stood out due to its architectural elegance and maturity in stream-first processing.

• Stream-First Architecture: Unlike micro-batch engines, Flink operates on true streams. Every event is processed as it arrives, unlocking low-latency computation.
• Exactly-Once Semantics: Flink’s stateful stream processing and checkpointing give strong consistency guarantees.
• Backpressure Management: Flink automatically detects slow downstream operators and propagates backpressure to throttle the source, preventing overload.
• Rich SQL & CEP Support: Flink offers powerful abstractions for developers and analysts, including SQL, Table API, and Complex Event Processing.
• Unified Batch + Stream API: Flink handles batch as a special case of streaming, reducing cognitive load and allowing us to gradually migrate legacy jobs.

New Architecture: Event-Driven and Stream-Native

Our reimagined data platform includes three key components:

• Kafka: Acts as the event buffer, decoupling producers and consumers. All incoming data (logs, transactions, events) is published to Kafka topics.
• Apache Flink: The heart of the system. We built multiple Flink jobs, each consuming one or more topics, performing enrichment, validation, filtering, aggregations, and joining across streams.
• PostgreSQL: Used as the final sink for storing processed and transformed data, which is then used by downstream analytics tools and internal dashboards.

Each Flink job runs continuously and reacts to data as it arrives, providing near-instant availability of information.

How Flink Solved Our Pain Points

The shift to Apache Flink addressed our major pain points:

1. Latency Dropped from Hours to Seconds: Business metrics and KPIs are now updated in real-time, enabling operational decisions based on the current state.
2. Smarter Resource Utilization: Instead of concentrating all compute in a nightly window, Flink distributes compute needs evenly throughout the day.
3. Unified Pipelines: We now manage significantly fewer jobs, each built on a shared abstraction with common monitoring and alerting.
4. Scalability by Design: Adding a new stream or increasing throughput is as simple as scaling Flink clusters horizontally.
5. Improved Data Quality: With built-in stateful processing, we implemented deduplication, out-of-order handling, and anomaly detection within the stream jobs.

Impact Metrics

The impact of the migration was measurable and immediate:

• Latency: Reduced from 24 hours to sub-10 seconds
• Throughput: 5x increase in events handled per second
• Infrastructure Cost: 30% reduction in compute costs due to even load distribution
• Operational Load: 50% fewer on-call alerts and data quality incidents

Conclusion

Transitioning from batch ETL to Apache Flink was not just a technical upgrade — it was a paradigm shift. We moved from passive, delayed insights to a proactive, real-time data ecosystem.

Flink enabled us to build a system that is not only faster and cheaper but also fundamentally better suited to the dynamic nature of modern data.

If you’re operating with stale data and struggling with complex batch jobs, it might be time to consider the real-time revolution. We did — and we’re not looking back.