Jens Willmer

Data Cleaning for RAG Search and Response

Tue, 23 Jun 2026 08:00:00 +0000

In a previous post, I covered what Retrieval-Augmented Generation is and how to prepare data for ingestion. A companion post on the ingest pipeline walks through the data cleaning techniques that get content into the vector store. This post picks up where retrieval begins.

Ingesting documents into a vector database is only half the problem. The other half is what happens when someone types a question: understanding the query, ranking results, validating citations, and handling failures along the way.

Query handling

User input can contain control characters, excessively long text, or prompt injection attempts. Before the query reaches any LLM or embedding model, three layers of sanitization run:

Control character removal: strip everything except newlines and tabs.
Prompt injection mitigation: regex-based detection of patterns like “ignore previous instructions” or “disregard system prompt.” Matched patterns are stripped before the query reaches any model.
Length truncation: queries are capped at 500 characters for classification and topic extraction (4000 for full content extraction).

Even though the LLM system prompts include their own guardrails, sanitizing at the boundary prevents entire categories of attacks from reaching the model.

Intent classification

Once the query is clean, the next question is: does it even need the RAG pipeline?

A four-way intent classifier runs before any retrieval:

Intent	Action	Example
`factual_query`	Full RAG pipeline	“What happened to vessel X in January?”
`exploratory`	Skip RAG, offer clarification	“I’m wondering about maintenance schedules…”
`off_topic`	Skip RAG, scope reminder	“What’s the weather in Athens?”
`greeting`	Canned response, no API calls	“Hi”, “Thanks”

A regex fast-path catches common greetings before the LLM classifier runs. The classifier uses temperature=0.0 for deterministic routing, so identical queries always get the same intent. If the classifier fails entirely, the system defaults to factual_query so search is never blocked by a transient error.

In typical usage, 15-25% of messages are greetings or acknowledgments. Routing them correctly saves 200-400ms and $0.001-0.01 per message by skipping embedding generation, vector search, and reranking.

Search and retrieval

With a confirmed factual_query, the system needs to find relevant content. This happens in several coordinated steps.

Topic-based filtering

Searching the entire vector space for every query produces noisy results. A question about cargo damage should not return chunks about engine maintenance just because they share vocabulary.

Topics are extracted from the query and matched against the corpus topic ontology using cosine similarity. The default mode is embedding-only: the query embedding is matched directly against topic embeddings, skipping an LLM extraction step and saving ~300ms per query.

Because the topic ontology is fragmented (median chunk count per topic is 1, concepts span ~7 synonymous topic rows), the matcher returns top-K closest topics per name rather than a single winner, pooling fragments into effective clusters.

If topic-filtered results return fewer chunks than a threshold, the filter retries with a looser similarity threshold. If still too sparse, the topic filter is dropped entirely while structural filters (like vessel metadata) are preserved.

Vessel metadata filtering

Fleet operators frequently need results for a specific vessel, type, or class. These are structural metadata filters applied at the database level before vector similarity: exact, not probabilistic. When an operator asks about a specific vessel, they get results only from that vessel.

Vessel names are resolved with fuzzy matching against a cached vessel list, and whole-word regex auto-detects vessel types and classes mentioned in the query text. Active filters persist per conversation, so follow-up questions inherit vessel context.

Hybrid vector + BM25 search

Pure vector similarity misses exact keyword matches (part numbers, IMO numbers, incident codes). Pure keyword search misses semantic meaning. The system combines both:

Vector search via HNSW index (pgvector) with cosine similarity. hnsw.ef_search = 100 (up from the default 40) trades a few milliseconds of latency for 5-10% recall improvement.
Optional BM25 scoring boosts results that match query keywords exactly.
A similarity threshold (default 0.3) discards low-confidence results before reranking.

This catches both semantic matches (“propulsion system failure” matching “engine breakdown”) and exact matches (“IMO 9876543” matching only that specific number).

Thread digest retrieval

Broad queries like “any engine issues last month?” need to match content spanning an entire email thread. Individual message chunks may contain only the problem or only the resolution.

During ingest, multi-message threads get a synthesized digest chunk (200-400 words) that captures the full problem-investigation-resolution arc. At search time, this digest competes alongside regular message chunks in vector similarity. A query like “how was the cylinder crack resolved?” might match the resolution message chunk poorly (it mentions “replaced liner” without context), but the digest chunk matches well because it contains the full narrative.

Reranking

After initial vector retrieval returns ~40 candidates, a cross-encoder model (Cohere rerank) scores each candidate against the query. Cross-encoders process the query and document together rather than encoding them separately, and are 20-35% more accurate than bi-encoder similarity alone.

Before reranking, each document is prefixed with its email subject and source title. This helps the reranker distinguish between chunks from different incidents that have similar technical content.

Results below a rerank score of 0.2 are filtered out, but at least one result is always returned to prevent empty responses. If the reranking API fails, the pipeline falls back to unranked vector results with a warning log.

In practice, reranking made the biggest difference to answer quality. The LLM attends most to the first chunks in its context, so getting the order right matters more than retrieving a few extra candidates.

Building the LLM context

At ingest time, each attachment is classified into one of three embedding modes (full, summary, or metadata_only). This classification carries through to how context is built for the LLM.

In full mode, a context summary is prepended to the chunk text, separated by ---. The LLM sees both document-level context and the specific passage. In summary mode, only the context summary is emitted, since the chunk content is already a synthesized summary and repeating it wastes context window. metadata_only emits just the filename and basic metadata.

Each search result also gets a structured citation header:

[CITE:chunk_id | title:"Cargo Report..." | subject:"RE: Vessel Inspection..." | date:2024-01-15 | from:captain@... | page:5]

Only the chunk ID is exposed as an identifier, no doc_id or source_id to confuse the model. Title is capped at 50 characters, subject at 40.

When multiple results come from the same email thread, they are grouped by root email with an incident summary. The LLM sees “3 incidents found, 12 relevant passages total” rather than a flat list, so it can piece together the timeline of an incident from multiple chunks.

Citations and trust

The LLM generates inline [C:chunk_id] markers, which then go through a validation pipeline:

Regex extraction captures citation markers (tolerating whitespace variations).
ID validation confirms each chunk ID is exactly 12 hex characters.
Deduplication collapses repeated citations of the same chunk.
Archive verification concurrently checks that each referenced file exists in storage, running in parallel so verification takes max(times) instead of sum.
If more than 50% of citations are invalid, the response is flagged. Invalid markers are stripped from the response text.

On the frontend, citations render as numbered badges with hover previews. When the data-citations SSE frame arrives mid-stream, raw [C:hex] markers are replaced with full HTML tags containing metadata, source title, page number, and download URI. A 3-second fallback ensures citations still render even if the validation pipeline is slow.

Users can click a citation badge to see the original email, the highlighted passage, and download the original PDF. Citation detail endpoints validate chunk IDs at the API boundary (12-char hex format check) before any database query runs.

For stored conversations, the citation payload is persisted alongside each assistant message. When a user returns days later, the same rendering function produces identical results: working links, accurate source references, same visual treatment.

Reliability

The search pipeline depends on several external services: embedding API, vector database, reranking API, LLM provider. Any of them can fail transiently. Every stage has a fallback:

Stage	On failure	Fallback
Intent classifier	LLM error	Default to `factual_query`
Topic extraction	LLM error	Broad search (no topic filter)
Topic filter	Too few results	Retry with loose threshold, then drop filter
Reranking	API error	Return unranked vector results
Citation validation	Archive check fails	Mark as unverified (still renders)
Citation SSE frame	Network timeout	Frontend renders from raw markers

Error messages that reach the user are sanitized: Bearer tokens, API keys, and database connection strings are stripped before display.

LangGraph agent checkpoints can also contain interrupted tool calls (the LLM requested a search, but the response was never received). Before each LLM call, the message history is scanned and orphaned calls are filtered out. Without this, a single interrupted request would make the entire conversation unusable.

Prompt caching reduces latency on follow-up messages by 40-60%. The system prompt is marked as a stable prefix for caching; dynamic content (vessel context, intent-specific instructions) is placed after conversation history so it does not invalidate the cache.

Observability

Without visibility into search quality, degradation goes unnoticed. Bad results do not produce errors; they produce silent user dissatisfaction.

Every search pipeline step is traced with Langfuse: session ID, user ID, intent classification result, topic filter decisions, retrieval count, rerank scores, generation output, and citation validation outcome. Users can submit thumbs-up/thumbs-down on each response, recorded as a Langfuse score linked to the specific trace.

Low-rated responses can be investigated by examining the full pipeline trace: which topics were extracted, which chunks were retrieved, what the reranker promoted, and what the LLM generated. If you do not have this kind of tracing, quality issues stay invisible until someone complains.

Citation detail endpoints validate chunk IDs before querying the database. Archive file serving rejects path traversal attempts. All endpoints have per-user rate limits (30/min for search, 100/min for citations) to protect upstream API budgets. Download URLs use pre-signed links with 300-second expiry rather than open storage access.

What held up

Reranking was the biggest quality improvement: a cross-encoder examining query and document together was 20-35% more accurate than embedding similarity alone. Topic filtering and metadata filters improved precision without sacrificing recall, because the loosening fallback catches edge cases.

I did not expect citation validation to matter as much as it did. A few dead links in an otherwise good answer made users distrust the whole system. Verifying every citation against the source archive before it reaches the user is worth the added latency.

The ingest pipeline gets data in. The search pipeline gets the right data out. Both need the same care around data quality, just at different points, and on the search side the user is watching in real time.

Data Cleaning for a RAG Ingest Pipeline

Tue, 16 Jun 2026 08:00:00 +0000

If your RAG pipeline ingests dirty data, the answers will be wrong. The embedding model and prompt chain cannot fix what was broken before indexing.

I built this pipeline for a maritime email corpus: thousands of .eml files with PDF attachments, Office documents, images, and ZIP archives, turned into a searchable knowledge base. The examples here are maritime, but the patterns apply to any industry where you ingest unstructured documents into a RAG system. Corporate email, support tickets, compliance archives, internal wikis: the same cleaning problems show up everywhere.

This post covers the techniques that actually mattered, based on a corpus of 6,000+ emails with tens of thousands of attachments.

Cleaning the raw email content

Corporate email is one of the noisiest data sources you can feed into a RAG system. A single email thread might contain the incident report you actually care about, buried under forwarded headers, legal disclaimers, satellite communication blocks, mailto: links, and five previous replies where someone wrote “OK, noted.”

The first step is thread splitting. Regex patterns detect message boundaries across Gmail, Outlook, and generic separators, splitting a forwarded thread into individual messages so each can be cleaned independently. Over 20 boilerplate regex patterns then strip <mailto:> links, [cid:] image references, horizontal separator lines, phone/fax contact blocks, satellite communication headers (common in maritime), address blocks, legal disclaimers, and unsubscribe footers.

For the edges that regex cannot handle cleanly, an LLM-based boundary detector identifies the first and last meaningful words in the email body, trimming headers and signatures with fuzzy matching that tolerates formatting drift between email clients.

Messages under 20 words get discarded entirely. Auto-replies and bare acknowledgments like “Noted, thanks” produce low-value chunks that score artificially high once context enrichment is applied, pushing the actual incident descriptions out of search results.

Without this cleanup, embeddings cluster around disclaimer text instead of the actual content. A query about “main engine cylinder crack” should not return ten results whose top match is a legal footer that appears in every email.

Encoding, HTML, and character corruption

Emails arrive from systems worldwide using different character encodings. A single encoding error can cascade: a garbled MIME header prevents attachment extraction, and you lose an entire document from the index.

The pipeline uses a cascading charset fallback that tries encodings in priority order: declared charset, UTF-8, UTF-8-sig (with BOM), ISO-8859-1, Windows-1252, CP1252. Only as a last resort does it use UTF-8 with replacement characters. UTF-8 BOM bytes are stripped before parsing, and CSV files from legacy systems are truncated at the first NUL byte with line endings normalized.

HTML-to-text conversion strips <script> and <style> tags, converts <a> tags to markdown links, replaces block elements with newlines, and unescapes HTML entities. Plain text MIME parts are preferred when available, because clean plain text produces better embeddings than HTML-tagged content.

Parsing: routing by complexity

Using the wrong parser for a document is either wasteful or produces empty output.

PDFs are classified as simple or complex before parsing. Simple PDFs (clean text layer, no form fields) go to PyMuPDF4LLM: free, local, milliseconds. Complex PDFs (scanned pages, form fields, image-only content) route to Gemini 2.5 Flash via OpenRouter at roughly $0.0025 per page, with page-range pagination and adaptive batch halving when the model truncates its output. Oversized PDFs above a configurable page ceiling get a local peek of the first three pages for classification, then route to summary mode if the content is noise.

Office documents are rendered locally. DOCX paragraphs become plain text, tables become GitHub Flavored Markdown with proper delimiters. XLSX sheets become headed sections with GFM tables. CSV files go through the charset fallback chain before table rendering. Only legacy binary formats (.doc, .xls, .ppt) still require a cloud parser.

One thing worth noting: when a local parser opens a file but extracts zero text (an image-only DOCX, for instance), a specific EmptyContentError triggers fallback to the cloud parser. Generic errors from corrupt files do not trigger the fallback, so you don’t waste API calls on unrecoverable files.

Tiered routing cuts parsing costs by 28-42% compared to sending everything through cloud parsers, saving roughly $63-94 per full ingest of the production corpus.

Filtering images and stripping parser artifacts

Emails contain many non-informative images: logos, banners, tracking pixels, signature graphics. Each Vision API call costs roughly $0.01, and in a typical maritime email corpus, 60-80% of embedded images are logos and banners.

A two-stage filter handles this. Heuristic rules hard-skip filenames matching patterns like logo, banner, signature, or icon. Files under 15KB (tracking pixels) or with dimensions under 100px are skipped. Images shorter than 50px with a width-to-height ratio above 3:1 are classified as email separator strips. Only images that pass these checks reach the Vision API, which classifies them as meaningful or decorative. The heuristic filter alone eliminates 30-50% of Vision API calls, saving $19-32 per full ingest.

Parser artifacts are another source of noise. LLM-based parsers inject placeholder image references like ![](page_3_image_1.jpg) that point to non-existent files. Regex-based stripping removes these while preserving any alt text that carries semantic value. A maintenance command can retroactively apply improved stripping patterns to all archived markdown, so bumping the regex fixes older archives without re-parsing.

Deciding what to embed

Sensor logs, repetitive tables, and empty PDFs waste embedding API calls and pollute the vector space. There is no reason to chunk and embed all of them the same way.

An embedding mode classifier routes each attachment into one of three modes:

full: standard chunking with context and embeddings. Default for prose content.
summary: one LLM-synthesized summary chunk. Used for bulk numeric or tabular content (digit ratio above 30%, table content above 40%, high repetition scores).
metadata_only: one stub chunk from the filename, no LLM calls. For empty or near-empty attachments (under 50 tokens, or very low prose ratio with no headings).

The decision tree uses computed content shape signals: digit ratio, prose ratio, table character percentage, repetition score, short line ratio, heading count. For documents in a medium-confidence band, a single LLM triage call makes the final classification. The triage prompt includes an anti-injection preamble, and the response parser reads only the first character, so crafted attachments cannot steer their own classification.

In the production corpus, 15-25% of attachments are sensor logs or repetitive tables. Summary mode reduces their embedding footprint by 95% while keeping them searchable.

Chunking and context enrichment

Once content is clean and classified, it needs to be split into chunks that respect the embedding model’s token limits and the document’s logical structure.

Token-aware splitting uses tiktoken (matching the embedding model’s tokenizer) to keep chunks within the context window. Markdown content uses a MarkdownTextSplitter that respects heading boundaries; plain text uses a recursive splitter with separators ordered from paragraph breaks down to character level. Splits under 5 words are discarded, since these are typically PDF pagination artifacts like lone page numbers.

Each chunk records its character offset and line numbers in the original document, so the chat UI can highlight the exact source passage when citing a chunk. Heading hierarchy extraction tracks the structural path (e.g., ## Safety > ### Fire Prevention) for additional context.

Contextual embedding had more effect on retrieval quality than anything else I tried. An LLM produces a 2-3 sentence summary of each document (type, subject, key entities), which is prepended to every chunk before embedding. A chunk like “The valve was replaced on Tuesday” becomes retrievable for queries about “MV Atlantic valve replacement” because the prepended context supplies the vessel name and subject. Research on contextual retrieval shows this improves retrieval accuracy by 35-67% over naive chunking.

For email threads with multiple messages, a separate thread digest captures the full conversation in a 200-400 word summary chunk. When a user asks “how was the pump failure fixed?”, the digest matches because it connects the resolution in message 5 to the original problem in message 1.

Metadata extraction and inheritance

Chunks from attachments have no context on their own. A PDF inspection report does not know it was attached to an email about “Main Engine Cylinder 3 Crack on MV ATLANTIC.” Without the parent email’s context, attachment chunks produce generic embeddings that miss filtered searches.

Vessel mention extraction uses regex patterns for maritime naming conventions (M/V [NAME], MV [NAME]) with a curated noise filter for non-vessel tokens (TANKERS, SHIP, CORP). Extracted vessel IDs go on every chunk, which makes per-vessel filtering possible during search.

Attachment chunks inherit their parent email’s vessel IDs and topic IDs. A cargo damage report PDF shows up in “Cargo Damage” topic-filtered searches even if the PDF itself never mentions the topic by name. Both the direct-attachment and ZIP-member code paths use the same shared helper to prevent drift.

Topic extraction uses an LLM to assign 1-5 topics per email, with aggressive name normalization and semantic deduplication via cosine similarity (threshold 0.92) to prevent the same concept from fragmenting across entries like “Cargo Damage”, “cargo damage”, and “Damaged Cargo.”

Storage integrity and deduplication

Re-ingesting emails after a parser upgrade, bug fix, or crash recovery must not create duplicate documents or chunks.

Content-addressable IDs make this deterministic. Document IDs are SHA256(source_id + file_hash), chunk IDs derive from the document ID plus character offsets. Same content always produces the same IDs. The database enforces UNIQUE(chunk_id) with INSERT OR REPLACE, making chunk insertion idempotent. Foreign key cascading deletes prevent orphaned chunks when a document is re-processed. The progress tracker uses file content hash (not file path) to decide what needs processing, so renaming a file does not trigger re-ingest.

Validation and targeted repair

Bugs will slip through regardless. The last line of defense is a validation suite with 31 automated checks covering referential integrity (orphaned chunks, broken parent references), embedding completeness (missing vectors, NaN values, zero vectors), content quality (empty chunks, missing context summaries), archive consistency (broken URIs, orphaned folders), schema drift, mode invariants, and topic health.

Every step stamps a trail in the archive metadata: which parser, which LLM model, which embedding model produced each artifact. When a check fails, the trail tells you exactly what to fix. “This PDF was parsed by Gemini 2.5 Flash with a 25-page batch” narrows the scope instantly.

The system supports targeted repair over full re-ingest. A regex improvement needs only clean-archive-md (zero API cost). A parser bug on specific emails uses mark-failed plus retry-failed (targeted cost). A chunking strategy change uses reindex-chunks (moderate cost, no re-parsing). Full re-ingest is the last resort, since it costs real money and hours of wall-clock time at production scale.

What held up

Looking back, the techniques that moved retrieval accuracy the most were mundane: stripping boilerplate, fixing encodings, filtering junk images, making sure every chunk carries enough context to be found.

Contextual embedding (prepending a document-level summary to every chunk) improved retrieval accuracy by 35-67%. Tiered parsing and embedding mode classification saved 28-42% on parsing and 95% on embedding costs for non-prose content. Content-addressable IDs and idempotent inserts meant I could iterate on the pipeline without worrying about corrupting production data or burning through API budgets. Automated validation after every ingest caught the data-dependent edge cases that unit tests never will.

None of this is exciting work. But in a RAG system, it is the work that decides whether your users get the right answer.

Things to think about

A few concerns that fall outside the core data cleaning flow but are worth keeping in mind:

Error message sanitization. API errors can contain Bearer tokens, API keys, or connection strings in their stack traces. If you store error messages (for debugging or admin UIs), strip credentials before persisting them.
ZIP security. Email attachments can include ZIP archives with path traversal attacks (../ in filenames), ZIP bombs (high compression ratios that exhaust memory), or deeply nested archives. Configurable limits on extraction depth, file count, and total size prevent a single malicious attachment from stalling the pipeline. Office formats (.docx, .xlsx, .pptx) are technically ZIP files and need to be detected and routed to their proper parsers instead of being extracted.

Gear Ratio Advanced: Velocity, Torque & Efficiency

Fri, 27 Jun 2025 21:21:00 +0000

This post builds on Gear Ratio Basics and explains how gear ratios affect speed (velocity), torque, and efficiency. Understanding these concepts helps you design gear systems that work the way you need them to.

🔄 Gear Ratio and Velocity

The gear ratio controls how fast the output gear turns compared to the input gear. The formula for calculating this is shown below:

\[\text{Gear Ratio} = \frac{\text{Input Speed}}{\text{Output Speed}}\]

For example, if your input gear rotates at 200 revolutions per minute (RPM) and you use a 2:1 gear ratio, you can calculate the output speed like this:

\[\text{Output Speed} = \frac{200\, \text{RPM}}{2} = 100\, \text{RPM}\]

This means the output gear turns at 100 RPM. A higher gear ratio slows down the output gear because the driven gear rotates fewer times for each turn of the driver gear. That’s how gear reductions let you decrease speed when needed, like in a robot’s drivetrain or a power tool’s gearbox.

💪 Gear Ratio and Torque

Gear ratios also affect torque, which is the twisting force transmitted through the gears. The relationship between gear ratio and torque is:

\[\text{Output Torque} = \text{Input Torque} \times \text{Gear Ratio}\]

For example, if your motor produces 1 newton-meter (Nm) of torque and you use a 3:1 gear ratio, you calculate the output torque like this:

\[\text{Output Torque} = 1\, \text{Nm} \times 3 = 3\, \text{Nm}\]

This means you get 3 Nm of torque at the output. By slowing the rotation, the gear system lets the same power apply more force. In practical terms, a gear reduction gives you more torque at the cost of slower speed, which is essential when you need to move something heavy.

⚠️ Direction Reminder

Whenever gears mesh, they change the direction of rotation. Every time you add a gear between the driver and the driven gear, the direction reverses again. For example, if you connect your driver gear to an idler gear, the idler reverses the direction, and the next gear will spin the opposite way compared to the driver. An idler gear does not change the gear ratio, but it flips the rotation direction once more. An even number of gears in the train means the output spins in the same direction as the input. An odd number of gears means the output spins in the opposite direction.

⚙️ Gear Efficiency

No gear system is perfectly efficient. Energy is lost through friction, heat, and flexing of materials, so it is important to understand gear efficiency. Spur gears usually operate at 95 to 98 percent efficiency for each gear mesh. Worm gears have much lower efficiency, often below 70 percent, because of the sliding friction that happens between the worm and worm wheel.

To calculate the total efficiency of a gear train with multiple stages, multiply the efficiency of each gear pair together. For example, if you have two spur gear pairs with 97 percent efficiency each, you calculate the total efficiency like this:

\[\text{Total Efficiency} = 0.97 \times 0.97 = 0.9409 \quad (\text{or } 94.1\%)\]

This means your gear train will deliver about 94.1 percent of the input power to the output. Lower efficiency means more heat and less useful power, which can become a problem in systems where energy savings or heat management are important.

✅ Summary

A higher gear ratio reduces the output speed but increases the output torque. Gears reverse rotation direction with each mesh, and the total number of gears in a train determines whether the output spins in the same or opposite direction as the input. Efficiency losses add up with every gear pair, so always include them in your calculations when designing a gear system.

What is Retrieval-Augmented Generation (RAG)

Tue, 20 May 2025 06:20:00 +0000

Large language models are impressive, but they’re limited by what they were trained on. They can’t access your internal documentation, stay current with new data, or reliably distinguish fact from fiction.

Retrieval-Augmented Generation (RAG) addresses this gap. It augments a language model by giving it access to external data at runtime. When a question is asked, the system first retrieves relevant information from a knowledge base—usually a vector database of semantically indexed chunks. Only then does the model generate a response, grounded in this retrieved context.

This enables more accurate, domain-aware, and verifiable answers without retraining the model. RAG effectively gives language models a dynamic memory—on your terms.

When to Use RAG

Retrieval-Augmented Generation is ideal when your application needs accurate, current, and domain-specific responses—but you don’t want to (or can’t) retrain the model.

Use RAG when:

Your data changes frequently. Traditional fine-tuning locks knowledge at training time. RAG lets you update answers by simply changing the underlying documents.
You need traceability. With RAG, every response is backed by retrievable content. Users (or auditors) can trace outputs to their original source.
Your knowledge is proprietary. Whether it’s internal policies, customer reports, or technical documentation, RAG can surface private data securely at inference time.
You want modular updates. By storing and referencing chunks with unique IDs, you can update individual pieces of content without retraining or reindexing everything.

RAG is especially useful for:

Internal support agents
Developer or product documentation assistants
Compliance and legal tools
Systems needing multilingual or version-aware responses

If your app needs dynamic answers with real references, RAG is the right foundation. It’s not just a theoretical model—real systems are using it to solve hard problems today.

Real-world systems already use this architecture to great effect. For instance, Mem0 implements a memory layer built on RAG. It retrieves semantically indexed memory entries—rather than relying on fragile prompt chains—enabling consistent, personalized responses over time.

At the infrastructure level, vector search engines like Qdrant power these retrieval systems. Qdrant supports hybrid filtering, payload scoring, and fast nearest neighbor search, making it ideal for large-scale, production-grade RAG systems.

Preparing Data for Ingestion

To make RAG work reliably, your content must be structured for retrieval and usable by a language model. This isn’t about dumping documents into a vector database—it’s about shaping the content so the model can reason over it effectively.

Start by extracting clean content from your sources. Remove layout artifacts, navigation elements, and anything irrelevant to the actual information. You want concise, plain-language text that reflects what a human would read to understand the topic.

Next, normalize the language. Rewrite content — especially code snippets, config files, or logs — into complete, natural sentences. Instead of embedding a raw string like user_limit: 250, convert it to “The maximum number of users allowed is 250.” The goal is natural language that the model can easily process and use in a response.

Every chunk should include descriptive metadata. This includes values like the source URL, section title, date, author, product or version, and any tags or classifications you use internally. Metadata can be stored separately or embedded into the text depending on your system design, but it must be consistent and queryable.

Finally, and critically, assign a unique and stable ID to every chunk or document. This lets you update or delete specific entries later without affecting the rest of your dataset. It’s essential for keeping your index maintainable over time.

RAG is only as good as the data it retrieves—so preparing your content with care is the foundation for everything that follows.

Improving Retrieval with Context

Once your data is clean, natural, and enriched with metadata, the next step is making it retrievable in a meaningful way. This is where context comes in. A single paragraph or sentence often lacks enough information on its own to match a user’s query effectively. By embedding context into each chunk, you improve both recall and precision during retrieval.

Inspired by Anthropic’s contextual retrieval approach, one method is to prepend a short description that explains what the chunk is about. For example, instead of storing:

“The system will reject login attempts after 5 failed tries.”

You might store:

“From the user authentication section of the security policy: The system will reject login attempts after 5 failed tries.”

This extra framing helps the embedding model encode why this text matters and where it belongs. It also improves match quality for more abstract or high-level questions like “What are our login security rules?”

Context can come from:

Section headings or document structure
File paths or category tags
Summaries or topic labels
Manual annotations (if scale allows)

In addition to prepending context to text, you can enrich your vector index with structured metadata. Many systems support hybrid search—combining vector similarity with keyword filters. For example, you can restrict results by audience (developer), language (de), or document type (release_notes).

The key idea is: make the meaning explicit. Give the system as much information as possible, up front, to help it retrieve the right content later.

Conclusion: Why RAG Matters

RAG brings structure, memory, and accountability to generative systems. It bridges the gap between static models and real-world knowledge — without the overhead of retraining.

To recap:

Use RAG when your data changes often, needs to stay private, or must be cited.
Prepare your data with clean, readable language and real metadata.
Track unique IDs for each entry so your dataset stays maintainable.
Add contextual information to each chunk to improve retrieval precision.

Done right, RAG systems are more flexible than fine-tuning, more trustworthy than standalone LLMs, and more adaptable to your evolving needs.

If you’re building AI systems that need to be smart and reliable, RAG isn’t just an option—it’s the standard.

Docker Networking Pitfalls

Sun, 02 Feb 2025 21:15:00 +0000

Docker networking is powerful but can be confusing, especially when dealing with communication between the host machine and Docker containers. Many developers assume that network behavior inside a Docker network works the same as on the host, leading to unexpected issues. In this post, we’ll explore common pitfalls when running services with Docker Compose and how to handle them correctly.

The Basics: Host vs. Docker Network

Docker Compose creates an isolated network for services to communicate. Each container gets a unique DNS name corresponding to its service name in docker-compose.yml. However, the way networking works inside Docker is different from how it works on the host machine.

Host Perspective (External Access)

When accessing a service from the host machine (outside Docker), you must use localhost and the mapped port:

curl http://localhost:8080  # Accessing a container-bound service via a mapped port

You cannot use Docker-internal DNS names (such as service_name) or host.docker.internal from the host machine.

Docker Container Perspective (Internal Communication)

Containers within the same Docker network can refer to each other by service name:

curl http://app-store:5000  # Works inside Docker

But if a container needs to communicate with a service running on the host machine, it must use host.docker.internal:

curl http://host.docker.internal:3000  # Works inside Docker to reach host

Common Pitfalls and How to Solve Them

Pitfall 1: Trying to Access a Container Using Service Names from the Host

❌ Incorrect:

curl http://app-store:5000  # Won't work from the host machine

The app-store service name is only resolvable inside Docker’s internal network.

✅ Correct:

curl http://localhost:5000  # Use localhost + mapped port from the host

This works if the port is correctly mapped in docker-compose.yml:

docker-compose.yml:
services:
  app-store:
    ports:
      - "5000:5000"

Pitfall 2: Using `localhost` Inside a Container to Reach Another Container

❌ Incorrect:

curl http://localhost:5000  # Won't work inside a container

Inside a container, localhost refers to itself, not other services in the Docker network.

✅ Correct:

curl http://app-store:5000  # Use the service name inside Docker

Pitfall 3: A Container Trying to Access a Host Service Without `host.docker.internal`

❌ Incorrect:

curl http://localhost:3306  # Won't work inside Docker

Since localhost inside a container refers to the container itself, this won’t connect to a host service.

✅ Correct:

curl http://host.docker.internal:3306  # Correct way inside Docker

Pitfall 4: Web Applications Generating Incorrect URLs

Web applications often generate links for users dynamically based on their environment. If the application is running inside a Docker container, it may generate links using internal Docker service names, which are not accessible to users.

❌ Incorrect:

<a href="http://web-service:8080">Click here</a>  <!-- Won't work for the user -->

✅ Correct:

Ensure the application differentiates between:

Internal URLs (used by services within Docker, such as web-service:8080)
External URLs (used by users, such as localhost:8080 or a domain name)

Understanding `host.docker.internal` Availability

host.docker.internal is a special hostname that resolves to the host machine’s IP address from within a Docker container. However, its availability depends on the operating system:

Platform	Availability	Notes
Windows	✅ Available	Works out of the box
Mac (Intel & M1/M2)	✅ Available	Built-in since Docker Desktop 18.03
Linux	❌ Not Available	Requires manual setup via `extra_hosts`
WSL 2	✅ Available	Works with Docker Desktop
iOS	❌ Not Available	No official support

For Linux users, a workaround is required:

docker-compose.yml:
services:
  my-service:
    extra_hosts:
      - "host.docker.internal:host-gateway"

This maps host.docker.internal to the host gateway.

Visualizing the Networking Model

Source	Destination	Works?	Solution
Host → Container	`localhost:port`	✅	Use mapped port in `docker-compose.yml`
Host → Container	`service_name`	❌	Won’t resolve
Container → Container	`service_name`	✅	Works within the same Docker network
Container → Host	`localhost`	❌	Refers to itself
Container → Host	`host.docker.internal:port`	✅	Works correctly
Web App → User Link	`service_name:port`	❌	Won’t work for users
Web App → User Link	`localhost:port`	✅	Use proper externally accessible URLs

Key Takeaways

Overview of the communication flow

Use localhost:port to access Docker services from the host.
Use service names for inter-container communication inside Docker.
Use host.docker.internal for Docker-to-host communication (if supported).
Linux requires manual configuration for host.docker.internal.
Mapped ports are crucial for external access.
Ensure web applications generate URLs users can actually reach.

By keeping these rules in mind, you can avoid common networking pitfalls when using Docker Compose. Share this guide with your colleagues to clarify how Docker networking works and improve debugging efficiency!

Interactive Motion Extraction with JavaScript

Sun, 13 Oct 2024 19:15:00 +0000

Motion Extraction in Action: Real-time Video Processing with JavaScript

In this blog post, we’ll explore a motion extraction technique inspired by an approach presented in this YouTube video by Steve of CodeParade. Steve’s work showcases impressive concepts in image processing, and this implementation applies similar ideas using JavaScript to manipulate video frames in real time. You can interact with the code embedded below or visit the GitHub gist for the complete snippet.

How Motion Extraction Works

Motion extraction refers to isolating differences between frames in a video or stream. This technique can be useful in scenarios like surveillance, where detecting motion helps trigger specific events, or for analyzing movements in sports or other activities.

This example utilizes the following core steps:

Frame Capture: We take snapshots of a video feed (from the camera or an uploaded file).
Color Inversion: Each frame is inverted to enhance the contrast between motion and background.
Motion Detection: By comparing consecutive frames, we cancel out parts that remain static, highlighting only the regions that exhibit change (motion).
Display: The processed frame is displayed, and the effect is repeated in intervals to create continuous motion detection.

Users can adjust parameters like resolution and delay to experiment with the results.

Key Features of the Code

Live or Uploaded Video: You can toggle between your camera feed and an uploaded video file.
Freeze Frame: This feature freezes one frame so that you will see all motion changes in respect to the frozen frame.
Inverted Color Blending: The code uses color inversion and blending between consecutive frames to cancel out static areas, highlighting only moving objects.

Freeze Frame Example

Imagine you activate the freeze-frame option, and at that moment, a grey background is captured, like in the image below:

Now, when you remove an object from the scene—such as a highlighting pen—the motion extraction process will clearly detect what has changed. In the following image, you can see the pen, which was removed, now highlighted by the motion extraction algorithm:

This technique is highly effective in identifying what has been removed or changed in the scene by comparing the captured freeze-frame with subsequent frames.

Try it Out

Here is the live version of the code, which you can interact with to test motion extraction:

For those who want to dive deeper into the code, you can check out the GitHub Gist.

Personal Docker Cheat Sheet

Thu, 19 Sep 2024 13:35:00 +0000

This is a personal reference for Docker commands that are not used often but frequently need to be looked up. Instead of searching every time, this list provides direct access to those less common yet essential commands.

Remove All Stopped Containers

docker rm $(docker ps -a -q)

Explanation: This command removes all stopped containers.
docker ps -a -q: Lists the IDs of all containers (stopped and running).
docker rm: Removes the listed containers.

Remove All Docker Images Not In Use

docker rmi $(docker images -q)

Explanation: This command removes all Docker images.
docker images -q: Lists the IDs of all images.
docker rmi: Removes the listed images.

Check Docker Disk Usage

docker system df -v

Explanation: Shows detailed information about Docker disk usage, including volumes, images, containers, and how much space they occupy.

Remove All Unused Volumes Except Specified Ones

Test Command (preview volumes to be deleted):

docker volume ls -qf dangling=true | grep -vE 'volume1|volume2'

Actual Removal Command:

docker volume ls -qf dangling=true | grep -vE 'volume1|volume2' | xargs -r docker volume rm

Explanation: This command removes all unused (dangling) Docker volumes except the specified ones.
docker volume ls -qf dangling=true: Lists all unused volumes.
- -q: Shows only the volume names.
- -f dangling=true: Filters to list only the volumes that are dangling (unused).
grep -vE 'volume1|volume2': Filters out the volumes you want to keep (replace volume1, volume2, etc.).
xargs -r docker volume rm: Passes the remaining volume names to docker volume rm and removes them. The -r option ensures the command is only run if there are volumes to delete.

Zero Downtime Deployment with Docker Rollout

Fri, 13 Sep 2024 14:40:00 +0000

This guide demonstrates a Zero Downtime Deployment using Docker Rollout, featuring an intentional failure, rollback scenario, and a successful rollout. Additional I demonstrate the use of a Traefik reverse proxy to expose the service ports.

Prerequisites:

Ensure you have:

Docker and Docker Compose installed.
Docker Rollout installed:

# Create directory for Docker cli plugins
mkdir -p ~/.docker/cli-plugins

# Download docker-rollout script to Docker cli plugins directory
curl https://raw.githubusercontent.com/wowu/docker-rollout/master/docker-rollout -o ~/.docker/cli-plugins/docker-rollout

# Make the script executable
chmod +x ~/.docker/cli-plugins/docker-rollout

Directory Setup

Create the following files in your working directory:

Dockerfile-1 (working version):

# Use the official NGINX base image
FROM nginx:latest

# Create a custom text file with "Hello World V1"
RUN echo "Hello World V1" > /usr/share/nginx/html/index.html

# Expose port 80 (the default NGINX port)
EXPOSE 80

Dockerfile-2 (intentional failure):

# Use the official NGINX base image
FROM nginx:latest

# Create a custom text file with "Hello World V2"
RUN echo "Hello World V2" > /usr/share/nginx/html/index.html

# Expose port 8080 (causing health check failure due to mismatch)
EXPOSE 8080

Note: Version 2 exposes port 8080, which will cause the health check to fail initially as it is configured to check port 80.

docker-compose.yml:

services:
  web:
    image: static-site:1
    restart: always
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:80 || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 2
      start_period: 5s
    networks:
      - my_network

  test:
    image: alpine
    container_name: test
    command: sh -c "apk add --no-cache wget && tail -f /dev/null"
    networks:
      - my_network

networks:
  my_network:
    driver: bridge

Build Docker Images

Build version 1:

docker build -t static-site:1 -f Dockerfile-1 .

Build version 2 (the failing one):

docker build -t static-site:2 -f Dockerfile-2 .

Run the Initial Version

Start the services using Docker Compose:

docker-compose up -d

This will start version 1 (static-site:1) of the web service.

Test the Initial Deployment

Verify that version 1 is running by executing wget inside the test container:

docker exec -it test wget -qO- http://web

You should see:

Hello World V1

Update the Web Service Version for Rollout

Before running Docker Rollout, update the image of the web service to version 2 in your docker-compose.yml file:

web:
  image: static-site:2
  # rest of the configuration remains the same

Important: Updating the version in the docker-compose.yml file is necessary before executing the rollout command to initiate the update.

Perform Zero Downtime Deployment with Docker Rollout

Run Docker Rollout to perform the zero downtime deployment:

docker rollout web --file docker-compose.yml

Observations During Rollout:

No downtime for the web service: The web service remains fully operational during the update process.
One container becomes unhealthy: Since version 2 exposes a different port (8080), the new version’s container will fail the health check, causing the deployment to fail.
Automatic rollback: Docker Rollout will automatically roll back to version 1, ensuring the service remains healthy.

Update the Health Check for a Successful Rollout

To observe a successful rollout, update the health check in the docker-compose.yml file to match port 8080, which version 2 is using:

healthcheck:
  test: ["CMD-SHELL", "curl -f http://localhost:8080 || exit 1"]
  interval: 10s
  timeout: 5s
  retries: 2
  start_period: 5s

Update the docker-compose.yml file with the health check for port 8080.
Run the rollout again with:

docker rollout web --file docker-compose.yml

Observations During Successful Rollout:

The service will now update successfully from version 1 to version 2, as the health check matches the correct port.
You should now see the new version output by running:

docker exec -it test wget -qO- http://web

Output:

Hello World V2

Setting up Traefik Reverse Proxy for HTTP Access

You can use Traefik to expose your services externally without needing to define specific ports or container names in your Docker Compose file. This is especially useful when working with the Docker Rollout plugin, as it imposes limitations similar to Docker Swarm, where ports or container names cannot be explicitly defined. Traefik simplifies this by handling service discovery and routing dynamically, allowing you to maintain external accessibility while supporting zero downtime deployments.

By routing traffic based on service labels instead of container specifics, Traefik enables seamless updates and rollbacks without requiring manual changes to the container’s network settings.

docker-compose.yml with Traefik added:

version: '3'
services:
  web:
    image: static-site:1
    restart: always
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:80 || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 2
      start_period: 5s
    networks:
      - my_network
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.web.rule=PathPrefix(`/`)"
      - "traefik.http.services.web.loadbalancer.server.port=80"

  traefik:
    image: traefik:v2.4
    command:
      - "--providers.docker=true"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
    networks:
      - my_network

  test:
    image: alpine
    container_name: test
    command: sh -c "apk add --no-cache wget && tail -f /dev/null"
    networks:
      - my_network

networks:
  my_network:
    driver: bridge

Summary

The Docker Rollout plugin enables near zero downtime deployments by updating service instances one at a time, ensuring that previous versions remain active until the new ones pass health checks. While the plugin strives to eliminate downtime, there is still a small possibility of downtime during the transition, as noted in Issue #21, which is currently being worked on and may be resolved in the future.

Additionally, using the restart: always policy is discouraged in favor of restart: unless-stopped, as outlined in Issue #25. This prevents conflicts with Docker Rollout’s management of container lifecycles during updates.

Key features of Docker Rollout:

Sequential updates: New instances are spun up one at a time, with health checks ensuring each instance is functioning properly before proceeding.
Health check validation: Only instances that pass their health checks are considered live, reducing the risk of introducing faulty versions.
Automatic rollback: In the event of a failure, Docker Rollout triggers a rollback to the last known good version, maintaining service stability.
Zero downtime goal: Though there may be brief downtimes in specific scenarios, Docker Rollout aims to keep services uninterrupted.
Flexible configuration: It supports custom timeouts, wait times, and health check parameters for diverse deployment needs.

Securely Expose Local Ports with Tailscale Funnel

Tue, 03 Sep 2024 15:35:00 +0000

When developing applications that need to interact with external services such as OAuth providers or webhooks, it’s often necessary to expose your local environment to the internet. Tailscale Funnel provides a quick, secure, and hassle-free method to do this, allowing any port on your local machine to be accessible over the internet with minimal configuration. This guide will walk you through setting up Tailscale Funnel to expose your application’s port, making it ideal for developers who need a temporary public endpoint for testing.

What is Tailscale Funnel?

Tailscale Funnel allows you to expose a local service running on your machine to the internet via HTTPS, leveraging Tailscale’s secure VPN. This feature is especially useful for scenarios where you need a public endpoint, such as testing OAuth callbacks, receiving webhooks, or sharing a development environment temporarily.

Key Benefits of Tailscale Funnel

Quick Setup: The process is straightforward and quick, making it perfect for rapid testing and debugging.
Secure and Easily Disabled: Public access can be enabled or disabled with a single command, ensuring your local environment remains secure when not in use.
Automatic HTTPS and MagicDNS: Tailscale handles HTTPS provisioning and DNS management, simplifying the setup.

Prerequisites

To use Tailscale Funnel, you need the following:

Tailscale account: Sign up at Tailscale.
Tailscale client: Installed and running on your development machine.
An application running locally: Your application should be running on a specific port (e.g., localhost:5000).

Step-by-Step Guide to Set Up Tailscale Funnel

1. Install and Configure Tailscale

Make sure Tailscale is installed and configured on your machine:

Download and install the Tailscale client from Tailscale Downloads.
Log in to your Tailscale account using the Tailscale client.
Authorize your device in the Tailscale admin console.

2. Provision a Certificate for Your Device

To ensure your service is accessible over HTTPS, you need to provision a certificate for your device using Tailscale:

Open the DNS page in the Tailscale admin console.
Enable MagicDNS if it is not already enabled for your tailnet.
Under HTTPS Certificates, select Enable HTTPS.
To obtain a certificate on your machine run the following command in the terminal:

tailscale cert

This step is crucial as it enables secure connections to your public endpoint through HTTPS.

3. Configure Access Controls

To enable Funnel, you need to adjust your Tailscale network’s access control list (ACL) settings:

Open the Tailscale admin console and go to Access Controls.
Modify your ACL configuration to include the Funnel attribute:

{
  "nodeAttrs": [
    // Adds the "funnel" attribute to all devices in your network
    { "target": ["autogroup:member"], "attr": ["funnel"] }
  ],
  "acls": [
    // Allow all connections.
    { "action": "accept", "src": ["*"], "dst": ["*:*"] },
  ]
}

4. Enable Tailscale Funnel for Your Application

Now, enable Funnel to expose your application:

tailscale funnel <port>

This command will make your application accessible via a public URL over HTTPS.

4. Obtain and Test Your Public URL

Once Funnel is enabled, Tailscale generates a public URL for your service. It will be in the format:

https://<device-name>.ts.net:<port>

For instance, if your device name is my-laptop and your application is running on port 5000, the URL will be:

https://my-laptop.ts.net:5000

7. Use the URL for External Integrations

With your application now accessible online, you can:

Test OAuth callbacks: Configure your OAuth provider’s redirect URI to your Tailscale Funnel URL (e.g., https://my-laptop.ts.net:5000/callback).
Receive webhooks: Set the Funnel URL as the endpoint for services needing to send data to your application.
Collaborate easily: Share your development environment securely for team testing or demonstrations.

8. Disable Funnel When Done

After testing, you can easily disable Funnel to secure your environment:

Open your terminal or command prompt.
Run the command to disable Funnel:

tailscale funnel disable <port>

In addition to exposing your local development environment, Tailscale Funnel can be used to share files quickly over the internet. This is particularly useful when you need to send files directly from your device without using an external file-sharing service. Here’s how to use Funnel for file sharing:

tailscale funnel /tmp/public

This command will activate file sharing mode, and Tailscale will automatically generate a public URL for you.

UXG-Lite router and Vigor-167 modem configuration

Wed, 10 Jan 2024 20:10:00 +0000

I’m using the UniFi UXG-Lite as a router in my home network. It connects to my DrayTek Vigor 167 modem (previously Vigor 130). In this post I publish the settings to configure the two devices and describe how you can reach the modem once the setup is running.

Modem configuration

My configuration is Telekom specific. Especially the VDSL2 Tag 7 can be different depending on the provider. I configured it on the modem, but it can also be configured on the router. It does not make a difference. The Modem has a static IPv4 assigned: 192.168.0.1

MPoA Settings

PPPoE Settings

IPv6 Settings

WAN Settings

Physical Connection Overview

Router Configuration

The internet configuration on the router is a no brainer. Open the Internet settings and if you haven’t configured the VLAN ID on the modem you have to select it in the router. In the IPv4 configuration section you have to select PPPoE and input username and password that was provided by your ISP (Telekom).

Telekom username

The construction of the username is very cumbersome. Below is the format you have to generate:

Anschlusskennung: 111111111111
Teilnehmernummer: 222222222222 (12 digits) or 33333333333 (11 digits)
Mitbenutzerkennung: 0001

Teilnehmernummer with 12 digits example: 1111111111112222222222220001@t-online.de
Teilnehmernummer with 11 digits example: 11111111111133333333333#0001@t-online.de
                                         11111111111133333333333\#0001@t-online.de

Connecting to the Modem

Once everything is configured and in place the modem is not reachable anymore. On a UXG Gateway you can create the following configuration to make it work:

{
    "interfaces": {
        "pseudo-ethernet": {
            "peth0": {
                "address": ["192.168.0.2/24"],
                "description": "Access to Modem",
                "link": ["eth1"]
            }
        }
    },
    "service": {
        "nat": {
            "rule": {
                "5000": {
                    "description": "MASQ Modem access",
                    "destination": {
                        "address": ["192.168.0.1"]
                    },
                    "outbound-interface": ["peth0"],
                    "type": "masquerade"
                }
            }
        }
    }
}

However, the UXG-Lite does not have the option to configure it permanently. What you can do instead is to login to the UXG-Lite via SSH and add the routing for eth1 manually:

# Assign a Static IP to eth1
sudo ip addr add 192.168.0.2/24 dev eth1

# Bring the Interface Up
sudo ip link set eth1 up

# Add a static route
sudo ip route add 192.168.0.0/24 dev eth1

After that you can create a SSH tunnel to your UXG-Lite and setup a SOCKS5 proxy in your browser in order to reach the Modem.

Putty tunnel

Firefox SOCKS5 proxy

Revert the changes to the routes

This is not strictly necessary. After a reboot of the device the settings are lost. I also noticed that any modification to the WAN settings on the UXG-Lite will remove the route. However, if you like to manually remove it after you are done you can execute the following command:

sudo ip route del 192.168.0.0/24

Jens Willmer

Data Cleaning for RAG Search and Response

Query handling

Intent classification

Search and retrieval

Topic-based filtering

Vessel metadata filtering

Hybrid vector + BM25 search

Thread digest retrieval

Reranking

Building the LLM context

Citations and trust

Reliability

Observability

What held up

Data Cleaning for a RAG Ingest Pipeline

Cleaning the raw email content

Encoding, HTML, and character corruption

Parsing: routing by complexity

Filtering images and stripping parser artifacts

Deciding what to embed

Chunking and context enrichment

Metadata extraction and inheritance

Storage integrity and deduplication

Validation and targeted repair

What held up

Things to think about

Gear Ratio Advanced: Velocity, Torque & Efficiency

🔄 Gear Ratio and Velocity

💪 Gear Ratio and Torque

⚠️ Direction Reminder

⚙️ Gear Efficiency

✅ Summary

What is Retrieval-Augmented Generation (RAG)

When to Use RAG

Preparing Data for Ingestion

Improving Retrieval with Context

Conclusion: Why RAG Matters

Docker Networking Pitfalls

The Basics: Host vs. Docker Network

Host Perspective (External Access)

Docker Container Perspective (Internal Communication)

Common Pitfalls and How to Solve Them

Pitfall 1: Trying to Access a Container Using Service Names from the Host

❌ Incorrect:

✅ Correct:

Pitfall 2: Using localhost Inside a Container to Reach Another Container

❌ Incorrect:

✅ Correct:

Pitfall 3: A Container Trying to Access a Host Service Without host.docker.internal

❌ Incorrect:

✅ Correct:

Pitfall 4: Web Applications Generating Incorrect URLs

❌ Incorrect:

✅ Correct:

Understanding host.docker.internal Availability

Visualizing the Networking Model

Key Takeaways

Interactive Motion Extraction with JavaScript

Motion Extraction in Action: Real-time Video Processing with JavaScript

How Motion Extraction Works

Key Features of the Code

Freeze Frame Example

Try it Out

Personal Docker Cheat Sheet

Remove All Stopped Containers

Remove All Docker Images Not In Use

Check Docker Disk Usage

Remove All Unused Volumes Except Specified Ones

Zero Downtime Deployment with Docker Rollout

Prerequisites:

Directory Setup

Build Docker Images

Run the Initial Version

Test the Initial Deployment

Update the Web Service Version for Rollout

Perform Zero Downtime Deployment with Docker Rollout

Observations During Rollout:

Update the Health Check for a Successful Rollout

Observations During Successful Rollout:

Pitfall 2: Using `localhost` Inside a Container to Reach Another Container

Pitfall 3: A Container Trying to Access a Host Service Without `host.docker.internal`

Understanding `host.docker.internal` Availability