<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.9.5">Jekyll</generator><link href="https://blogs.buildwithmanish.com/feed.xml" rel="self" type="application/atom+xml" /><link href="https://blogs.buildwithmanish.com/" rel="alternate" type="text/html" /><updated>2026-01-16T17:54:49+00:00</updated><id>https://blogs.buildwithmanish.com/feed.xml</id><title type="html">BuildWithManish</title><subtitle>Exploring code, architecture, and technology. Technical deep dives, tutorials, and insights from a software engineer.</subtitle><author><name>Manish Tiwari</name></author><entry><title type="html">Top 5 Programming Languages to learn in 2026</title><link href="https://blogs.buildwithmanish.com/deep-dive/career-growth/software-engineering/system-design/tech-trends/2025/12/31/top-5-programming-languages-to-learn-in-2026.html" rel="alternate" type="text/html" title="Top 5 Programming Languages to learn in 2026" /><published>2025-12-31T00:00:00+00:00</published><updated>2025-12-31T00:00:00+00:00</updated><id>https://blogs.buildwithmanish.com/deep-dive/career-growth/software-engineering/system-design/tech-trends/2025/12/31/top-5-programming-languages-to-learn-in-2026</id><content type="html" xml:base="https://blogs.buildwithmanish.com/deep-dive/career-growth/software-engineering/system-design/tech-trends/2025/12/31/top-5-programming-languages-to-learn-in-2026.html"><![CDATA[<p><img src="https://blogs.buildwithmanish.com/assets/images/hero-top-5-programming-languages-to-learn-in-2026-0-2025-12-31-69086.png" alt="Top 5 Programming Languages Hero" /></p>

<h1 id="introduction-the-confession-of-a-senior-engineer">Introduction: The Confession of a Senior Engineer</h1>

<p>I have a confession to make. For years, I was wrong.</p>

<p>I was wrong because I fell into the trap that many content creators and tech leads fall into: the trap of the linear roadmap. I used to make content that prescribed a rigid path: “Learn Language A first, then Language B, then Framework C.” It was neat, it was clickable, and it was fundamentally flawed.</p>

<p>In the real world of software engineering, specifically as we approach 2026, blindly listing programming languages in decreasing order of preference is a disservice to the craft. No technology is inherently “useless.” Every tool in our stack exists because it solved a specific problem at a specific time better than its predecessors.</p>

<p>If you are reading a blog or watching a video that lists languages at Number 1, 2, and 3, you must ask: <strong>What is the basis for this ranking?</strong></p>

<p>The reality is that the “best” language depends entirely on your context:</p>
<ul>
  <li><strong>Are you a student?</strong> You need foundational strength (Memory management, DSA).</li>
  <li><strong>Are you a working professional?</strong> You need scalability and ecosystem support.</li>
  <li><strong>Are you pivoting from non-tech?</strong> You need accessibility and rapid feedback loops.</li>
</ul>

<p>If I simply gave you a list-1, 2, 3, 4, 5-without this context, this article would be meaningless.</p>

<h2 id="why-this-deep-dive-exists">Why This Deep Dive Exists</h2>

<p>So, why am I writing this extensively now? The reason is the overwhelming volume of requests regarding the roadmap for 2026. “Harry Bhai, what should I learn to stay relevant?”</p>

<p>My job is to guide you, not just to feed you syntax. In this deep dive, the order I share is not a ranking of quality, but a ranking of <strong>strategic opportunity</strong> based on where you are in your life. Time is your scarcest resource. You can earn unlimited money as a developer, but you cannot reclaim the time spent learning a dying stack or a language that doesn’t align with your career goals.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/concept_illustration-top-5-programming-languages-to-learn-in-2026-1-2025-12-31-79866.png" alt="Concept Illustration" /></p>

<h2 id="the-methodology-data-driven-decisions">The Methodology: Data-Driven Decisions</h2>

<p>We are engineers; we trust data, not hype. To compile this analysis for 2026, I didn’t rely on gut feeling. I aggregated insights from three primary pillars of truth in our industry:</p>

<ol>
  <li><strong>The Stack Overflow Developer Survey:</strong> For understanding developer sentiment and daily usage.</li>
  <li><strong>The GitHub Octoverse Data:</strong> For analyzing actual code being pushed, PRs merged, and open-source velocity.</li>
  <li><strong>The TIOBE Index:</strong> For long-term historical trends and industry stability.</li>
</ol>

<p><img src="https://blogs.buildwithmanish.com/assets/images/comparison_chart-top-5-programming-languages-to-learn-in-2026-2-2025-12-31-20190.png" alt="Comparison Chart" /></p>

<p>Let’s dive into the technical architectures, trade-offs, and code-level realities of the top 5 languages for 2026.</p>

<hr />

<h1 id="1-python-the-api-of-the-ai-revolution">1. Python: The API of the AI Revolution</h1>

<h3 id="the-context">The Context</h3>
<p>Python is the first language on this list, and for 2026, it is arguably the most critical for career velocity.</p>

<p><strong>Who is this for?</strong></p>
<ul>
  <li><strong>Beginners:</strong> It reads like English, abstracting away memory management.</li>
  <li><strong>Intermediates/Career Switchers:</strong> If you are moving into Data Analytics or Data Science, this is non-negotiable.</li>
  <li><strong>Experts:</strong> It is the glue code of the internet and the interface for high-performance computing (HPC) via C-extensions.</li>
</ul>

<h3 id="the-data">The Data</h3>
<ul>
  <li><strong>TIOBE Index (Nov 2025):</strong> #1. Python has held the “Language of the Year” title repeatedly (2007, 2010, 2018, 2020, 2021, 2024).</li>
  <li><strong>Stack Overflow (2025):</strong> 57.9% of respondents use Python, a massive 7% jump from 2024, driven entirely by the AI boom.</li>
  <li><strong>GitHub Octoverse:</strong> In 2024, Python officially overtook JavaScript as the most popular language on the platform.</li>
</ul>

<h3 id="technical-deep-dive-why-python">Technical Deep Dive: Why Python?</h3>

<p>Many critics argue that Python is “slow” because of the Global Interpreter Lock (GIL) and its interpreted nature. While technically true for pure Python code, this argument misses the architectural reality of modern Python.</p>

<p>Python is rarely used for raw computation in production; it is used as an <strong>interface</strong>. When you run a heavy workload in Python (like training an LLM or processing a Petabyte of data), Python is merely dispatching instructions to highly optimized C/C++ or CUDA kernels.</p>

<h4 id="architecture-the-c-extension-ecosystem">Architecture: The C-Extension Ecosystem</h4>
<p>Libraries like NumPy, PyTorch, and TensorFlow are written in C and C++. Python provides the high-level syntax to manipulate these low-level memory structures.</p>

<p>Consider the recent updates to my Data Science course. We are running OpenAI’s Whisper model. You aren’t writing the matrix multiplication algorithms in Python loops; you are calling a compiled binary.</p>

<p><strong>Code Example: Running Whisper Inference</strong>
Here is how simple it is to leverage State-of-the-Art (SOTA) AI, hiding massive complexity behind a clean Python API.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">torch</span>
<span class="kn">from</span> <span class="nn">transformers</span> <span class="kn">import</span> <span class="n">AutoModelForSpeechSeq2Seq</span><span class="p">,</span> <span class="n">AutoProcessor</span><span class="p">,</span> <span class="n">pipeline</span>

<span class="c1"># 1. Device Agnosticism: Check for GPU (CUDA), Mac (MPS), or CPU
</span><span class="n">device</span> <span class="o">=</span> <span class="s">"cuda:0"</span> <span class="k">if</span> <span class="n">torch</span><span class="p">.</span><span class="n">cuda</span><span class="p">.</span><span class="n">is_available</span><span class="p">()</span> <span class="k">else</span> <span class="s">"mps"</span> <span class="k">if</span> <span class="n">torch</span><span class="p">.</span><span class="n">backends</span><span class="p">.</span><span class="n">mps</span><span class="p">.</span><span class="n">is_available</span><span class="p">()</span> <span class="k">else</span> <span class="s">"cpu"</span>
<span class="n">torch_dtype</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">float16</span> <span class="k">if</span> <span class="n">torch</span><span class="p">.</span><span class="n">cuda</span><span class="p">.</span><span class="n">is_available</span><span class="p">()</span> <span class="k">else</span> <span class="n">torch</span><span class="p">.</span><span class="n">float32</span>

<span class="n">model_id</span> <span class="o">=</span> <span class="s">"openai/whisper-large-v3"</span>

<span class="c1"># 2. Load the Pre-trained Model (Optimized C++ backend)
</span><span class="n">model</span> <span class="o">=</span> <span class="n">AutoModelForSpeechSeq2Seq</span><span class="p">.</span><span class="n">from_pretrained</span><span class="p">(</span>
    <span class="n">model_id</span><span class="p">,</span> <span class="n">torch_dtype</span><span class="o">=</span><span class="n">torch_dtype</span><span class="p">,</span> <span class="n">low_cpu_mem_usage</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">use_safetensors</span><span class="o">=</span><span class="bp">True</span>
<span class="p">)</span>
<span class="n">model</span><span class="p">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span>

<span class="n">processor</span> <span class="o">=</span> <span class="n">AutoProcessor</span><span class="p">.</span><span class="n">from_pretrained</span><span class="p">(</span><span class="n">model_id</span><span class="p">)</span>

<span class="c1"># 3. Create the Inference Pipeline
</span><span class="n">pipe</span> <span class="o">=</span> <span class="n">pipeline</span><span class="p">(</span>
    <span class="s">"automatic-speech-recognition"</span><span class="p">,</span>
    <span class="n">model</span><span class="o">=</span><span class="n">model</span><span class="p">,</span>
    <span class="n">tokenizer</span><span class="o">=</span><span class="n">processor</span><span class="p">.</span><span class="n">tokenizer</span><span class="p">,</span>
    <span class="n">feature_extractor</span><span class="o">=</span><span class="n">processor</span><span class="p">.</span><span class="n">feature_extractor</span><span class="p">,</span>
    <span class="n">max_new_tokens</span><span class="o">=</span><span class="mi">128</span><span class="p">,</span>
    <span class="n">chunk_length_s</span><span class="o">=</span><span class="mi">30</span><span class="p">,</span>
    <span class="n">batch_size</span><span class="o">=</span><span class="mi">16</span><span class="p">,</span>
    <span class="n">torch_dtype</span><span class="o">=</span><span class="n">torch_dtype</span><span class="p">,</span>
    <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">,</span>
<span class="p">)</span>

<span class="c1"># 4. Execute
# This single line triggers massive parallel computation on the GPU
</span><span class="n">result</span> <span class="o">=</span> <span class="n">pipe</span><span class="p">(</span><span class="s">"audio_sample.mp3"</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Transcribed Text: </span><span class="si">{</span><span class="n">result</span><span class="p">[</span><span class="s">'text'</span><span class="p">]</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
</code></pre></div></div>

<p><img src="https://blogs.buildwithmanish.com/assets/images/technical_diagram-top-5-programming-languages-to-learn-in-2026-3-2025-12-31-89405.png" alt="Python Technical Diagram" /></p>

<h3 id="career--salary">Career &amp; Salary</h3>
<p>In India, Python developers in the Data Science/AI domain command between <strong>13 LPA to 26 LPA</strong> on average.</p>

<p><strong>The Warning:</strong> If you are a first-year student and your curriculum focuses on C/C++, <strong>do not ignore it for Python.</strong> Python makes you lazy with memory. C++ teaches you how the computer actually works. Learn Python on the side, but master your foundations first.</p>

<hr />

<h1 id="2-typescript-the-savior-of-scale">2. TypeScript: The Savior of Scale</h1>

<h3 id="the-context-1">The Context</h3>
<p>At number two, we have TypeScript.</p>

<p><strong>Who is this for?</strong></p>
<ul>
  <li><strong>Web Developers:</strong> If you know JavaScript, you <em>must</em> migrate to TypeScript.</li>
  <li><strong>Full Stack Engineers:</strong> Essential for Next.js, React, and Node.js environments.</li>
</ul>

<h3 id="the-data-1">The Data</h3>
<ul>
  <li><strong>GitHub Octoverse 2025:</strong> TypeScript is #1 in terms of growth context.</li>
  <li><strong>Growth Rate:</strong> 66% Year-on-Year growth vs Python’s 48%.</li>
  <li><strong>Adoption:</strong> It has become the “Recruiter’s Choice.”</li>
</ul>

<h3 id="technical-deep-dive-static-typing-in-a-dynamic-world">Technical Deep Dive: Static Typing in a Dynamic World</h3>

<p>JavaScript is loosely typed. This is great for prototyping but catastrophic for large-scale applications. A variable <code class="language-plaintext highlighter-rouge">user</code> can be an object today, a string tomorrow, and <code class="language-plaintext highlighter-rouge">undefined</code> on Friday evening production deploys.</p>

<p>TypeScript introduces <strong>Structural Typing</strong>. Unlike Java’s nominal typing (where the name of the class matters), TypeScript cares about the <em>shape</em> of the object. It compiles down to JavaScript, meaning it adds zero runtime overhead-all checks happen at build time.</p>

<h4 id="the-problem-undefined-is-not-a-function">The Problem: “undefined is not a function”</h4>
<p>In 2026, shipping code without type safety is considered professional negligence in top-tier tech companies.</p>

<p><strong>Code Example: Interfaces and Generics</strong>
Here is how TypeScript prevents an entire class of bugs that plague JavaScript developers.</p>

<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Defining the Shape of Data</span>
<span class="kr">interface</span> <span class="nx">User</span> <span class="p">{</span>
  <span class="nl">id</span><span class="p">:</span> <span class="kr">number</span><span class="p">;</span>
  <span class="nl">username</span><span class="p">:</span> <span class="kr">string</span><span class="p">;</span>
  <span class="nl">email</span><span class="p">:</span> <span class="kr">string</span><span class="p">;</span>
  <span class="nl">role</span><span class="p">:</span> <span class="dl">'</span><span class="s1">admin</span><span class="dl">'</span> <span class="o">|</span> <span class="dl">'</span><span class="s1">editor</span><span class="dl">'</span> <span class="o">|</span> <span class="dl">'</span><span class="s1">viewer</span><span class="dl">'</span><span class="p">;</span> <span class="c1">// Union types</span>
  <span class="nl">meta</span><span class="p">?:</span> <span class="nb">Record</span><span class="o">&lt;</span><span class="kr">string</span><span class="p">,</span> <span class="nx">unknown</span><span class="o">&gt;</span><span class="p">;</span> <span class="c1">// Optional metadata</span>
<span class="p">}</span>

<span class="c1">// Generic Response Wrapper</span>
<span class="kr">interface</span> <span class="nx">ApiResponse</span><span class="o">&lt;</span><span class="nx">T</span><span class="o">&gt;</span> <span class="p">{</span>
  <span class="na">data</span><span class="p">:</span> <span class="nx">T</span><span class="p">;</span>
  <span class="nl">status</span><span class="p">:</span> <span class="kr">number</span><span class="p">;</span>
  <span class="nl">timestamp</span><span class="p">:</span> <span class="kr">string</span><span class="p">;</span>
<span class="p">}</span>

<span class="c1">// Function with strict type enforcement</span>
<span class="k">async</span> <span class="kd">function</span> <span class="nx">fetchUser</span><span class="p">(</span><span class="nx">userId</span><span class="p">:</span> <span class="kr">number</span><span class="p">):</span> <span class="nb">Promise</span><span class="o">&lt;</span><span class="nx">ApiResponse</span><span class="o">&lt;</span><span class="nx">User</span><span class="o">&gt;&gt;</span> <span class="p">{</span>
  <span class="c1">// Simulate DB call</span>
  <span class="kd">const</span> <span class="nx">user</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">database</span><span class="p">.</span><span class="nx">users</span><span class="p">.</span><span class="nx">findUnique</span><span class="p">({</span> <span class="na">where</span><span class="p">:</span> <span class="p">{</span> <span class="na">id</span><span class="p">:</span> <span class="nx">userId</span> <span class="p">}</span> <span class="p">});</span>

  <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">user</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span><span class="p">(</span><span class="dl">"</span><span class="s2">User not found</span><span class="dl">"</span><span class="p">);</span>
  <span class="p">}</span>

  <span class="c1">// TypeScript will error here if 'user' doesn't match the 'User' interface</span>
  <span class="k">return</span> <span class="p">{</span>
    <span class="na">data</span><span class="p">:</span> <span class="nx">user</span><span class="p">,</span>
    <span class="na">status</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
    <span class="na">timestamp</span><span class="p">:</span> <span class="k">new</span> <span class="nb">Date</span><span class="p">().</span><span class="nx">toISOString</span><span class="p">()</span>
  <span class="p">};</span>
<span class="p">}</span>

<span class="c1">// Usage</span>
<span class="kd">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">fetchUser</span><span class="p">(</span><span class="mi">101</span><span class="p">);</span>

<span class="c1">// SAFTEY: TypeScript knows 'result.data.role' exists.</span>
<span class="c1">// It also knows 'result.data.password' DOES NOT exist.</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">result</span><span class="p">.</span><span class="nx">data</span><span class="p">.</span><span class="nx">role</span> <span class="o">===</span> <span class="dl">'</span><span class="s1">admin</span><span class="dl">'</span><span class="p">)</span> <span class="p">{</span>
    <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">"</span><span class="s2">Access Granted</span><span class="dl">"</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>

<p><img src="https://blogs.buildwithmanish.com/assets/images/technical_diagram-top-5-programming-languages-to-learn-in-2026-4-2025-12-31-1775.png" alt="TypeScript Diagram" /></p>

<h3 id="career--salary-1">Career &amp; Salary</h3>
<p>For React/Next.js developers in India, the average salary ranges from <strong>10 LPA to 30 LPA</strong>. Senior developers architecting large frontend systems with strict TypeScript configurations can easily surpass <strong>40 LPA</strong>.</p>

<hr />

<h1 id="3-c-and-c-the-timeless-foundation">3. C and C++: The Timeless Foundation</h1>

<h3 id="the-context-2">The Context</h3>
<p>At number three, we return to the metal.</p>

<p><strong>Who is this for?</strong></p>
<ul>
  <li><strong>Students:</strong> This is non-negotiable for building a mental model of computing.</li>
  <li><strong>System Programmers:</strong> Game engines, High-Frequency Trading (HFT), Embedded Systems.</li>
</ul>

<h3 id="the-data-2">The Data</h3>
<ul>
  <li><strong>TIOBE Index:</strong> Consistently #2 and #3.</li>
  <li><strong>GitHub Octoverse:</strong> C grew 20% YoY; C++ grew 11.8%.</li>
  <li><strong>Use Case:</strong> The AI libraries mentioned in the Python section (NumPy, TensorFlow) are all maintained by C++ engineers.</li>
</ul>

<h3 id="technical-deep-dive-manual-memory-mastery">Technical Deep Dive: Manual Memory Mastery</h3>

<p>Why is C++ still growing in 2026? <strong>Performance per watt and predictability.</strong>
In languages like Java or Python, a Garbage Collector (GC) runs periodically to clean up memory. This causes “stop-the-world” pauses. In HFT or Real-Time systems (like self-driving cars), a 50ms GC pause causes a crash or a financial loss.</p>

<p>C++ gives you RAII (Resource Acquisition Is Initialization) and Smart Pointers, allowing for deterministic memory management without the GC overhead.</p>

<p><strong>Code Example: Modern C++ (C++20) &amp; Memory Management</strong>
Notice how we use <code class="language-plaintext highlighter-rouge">std::unique_ptr</code> to handle memory automatically (when it goes out of scope) without a garbage collector.</p>

<div class="language-cpp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include</span> <span class="cpf">&lt;iostream&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;vector&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;memory&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;algorithm&gt;</span><span class="cp">
</span>
<span class="k">class</span> <span class="nc">MatrixEngine</span> <span class="p">{</span>
    <span class="kt">int</span> <span class="n">rows</span><span class="p">,</span> <span class="n">cols</span><span class="p">;</span>
    <span class="n">std</span><span class="o">::</span><span class="n">vector</span><span class="o">&lt;</span><span class="kt">double</span><span class="o">&gt;</span> <span class="n">data</span><span class="p">;</span>

<span class="nl">public:</span>
    <span class="n">MatrixEngine</span><span class="p">(</span><span class="kt">int</span> <span class="n">r</span><span class="p">,</span> <span class="kt">int</span> <span class="n">c</span><span class="p">)</span> <span class="o">:</span> <span class="n">rows</span><span class="p">(</span><span class="n">r</span><span class="p">),</span> <span class="n">cols</span><span class="p">(</span><span class="n">c</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">data</span><span class="p">.</span><span class="n">resize</span><span class="p">(</span><span class="n">r</span> <span class="o">*</span> <span class="n">c</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">);</span>
        <span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o">&lt;&lt;</span> <span class="s">"Matrix initialized on Heap"</span> <span class="o">&lt;&lt;</span> <span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span>
    <span class="p">}</span>

    <span class="o">~</span><span class="n">MatrixEngine</span><span class="p">()</span> <span class="p">{</span>
        <span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o">&lt;&lt;</span> <span class="s">"Matrix destroyed (Memory Freed)"</span> <span class="o">&lt;&lt;</span> <span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span>
    <span class="p">}</span>

    <span class="kt">void</span> <span class="n">compute</span><span class="p">()</span> <span class="p">{</span>
        <span class="c1">// Simulating heavy computation</span>
        <span class="n">std</span><span class="o">::</span><span class="n">transform</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">begin</span><span class="p">(),</span> <span class="n">data</span><span class="p">.</span><span class="n">end</span><span class="p">(),</span> <span class="n">data</span><span class="p">.</span><span class="n">begin</span><span class="p">(),</span> <span class="p">[](</span><span class="kt">double</span> <span class="n">d</span><span class="p">)</span> <span class="p">{</span>
            <span class="k">return</span> <span class="n">d</span> <span class="o">+</span> <span class="mf">1.0</span><span class="p">;</span>
        <span class="p">});</span>
    <span class="p">}</span>
<span class="p">};</span>

<span class="kt">int</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span>
    <span class="c1">// Smart Pointer: Automatically manages lifecycle</span>
    <span class="c1">// No 'new' or 'delete' keywords needed here.</span>
    <span class="n">std</span><span class="o">::</span><span class="n">unique_ptr</span><span class="o">&lt;</span><span class="n">MatrixEngine</span><span class="o">&gt;</span> <span class="n">engine</span> <span class="o">=</span> <span class="n">std</span><span class="o">::</span><span class="n">make_unique</span><span class="o">&lt;</span><span class="n">MatrixEngine</span><span class="o">&gt;</span><span class="p">(</span><span class="mi">1000</span><span class="p">,</span> <span class="mi">1000</span><span class="p">);</span>

    <span class="n">engine</span><span class="o">-&gt;</span><span class="n">compute</span><span class="p">();</span>

    <span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o">&lt;&lt;</span> <span class="s">"Computation Complete"</span> <span class="o">&lt;&lt;</span> <span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span>
    
    <span class="c1">// 'engine' goes out of scope here. </span>
    <span class="c1">// The destructor is called immediately and deterministically.</span>
    <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p><img src="https://blogs.buildwithmanish.com/assets/images/concept_illustration-top-5-programming-languages-to-learn-in-2026-5-2025-12-31-46471.png" alt="C++ Concept Illustration" /></p>

<h3 id="advice">Advice</h3>
<p>If you are in college, obsess over C++. If you understand Pointers, Stack vs. Heap, and Pass-by-Reference, learning any other language (Java, JS, Python) becomes trivial.</p>

<hr />

<h1 id="4-java-the-enterprise-monarch">4. Java: The Enterprise Monarch</h1>

<h3 id="the-context-3">The Context</h3>
<p>At number four is Java. People have been predicting the death of Java for a decade, yet it remains the backbone of the Fortune 500.</p>

<p><strong>Who is this for?</strong></p>
<ul>
  <li><strong>Backend Engineers:</strong> Building massive, scalable microservices.</li>
  <li><strong>Enterprise Aspirants:</strong> Targeting banks (JP Morgan, Wells Fargo), Uber, Netflix, or Amazon.</li>
</ul>

<h3 id="the-data-3">The Data</h3>
<ul>
  <li><strong>Market Dominance:</strong> While Python rules AI, Java rules <strong>Business Logic</strong>.</li>
  <li><strong>Ecosystem:</strong> The Spring Boot ecosystem is unrivaled for rapid enterprise development.</li>
</ul>

<h3 id="technical-deep-dive-the-jvm-and-modern-concurrency">Technical Deep Dive: The JVM and Modern Concurrency</h3>

<p>Java runs on the Java Virtual Machine (JVM), a marvel of engineering. The Just-In-Time (JIT) compiler optimizes code as it runs, often making long-running Java applications faster than compiled C++ code in specific server scenarios due to runtime profiling.</p>

<p><strong>The Evolution:</strong>
Java isn’t stagnant. Recent versions introduced <strong>Virtual Threads (Project Loom)</strong>. Previously, one Java thread equaled one OS thread (expensive). Now, Java can handle millions of lightweight virtual threads, rivaling Go’s goroutines and Node.js’s event loop for throughput.</p>

<p><strong>Code Example: Spring Boot 3 &amp; Virtual Threads</strong>
This is what modern backend code looks like-declarative and highly scalable.</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">package</span> <span class="nn">com.example.demo</span><span class="o">;</span>

<span class="kn">import</span> <span class="nn">org.springframework.boot.SpringApplication</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.springframework.boot.autoconfigure.SpringBootApplication</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.springframework.web.bind.annotation.GetMapping</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.springframework.web.bind.annotation.RestController</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.concurrent.Executors</span><span class="o">;</span>

<span class="nd">@SpringBootApplication</span>
<span class="nd">@RestController</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">BankingService</span> <span class="o">{</span>

    <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
        <span class="nc">SpringApplication</span><span class="o">.</span><span class="na">run</span><span class="o">(</span><span class="nc">BankingService</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">args</span><span class="o">);</span>
    <span class="o">}</span>

    <span class="nd">@GetMapping</span><span class="o">(</span><span class="s">"/process-transaction"</span><span class="o">)</span>
    <span class="kd">public</span> <span class="nc">String</span> <span class="nf">process</span><span class="o">()</span> <span class="o">{</span>
        <span class="c1">// In Java 21+, we can use Virtual Threads for high-throughput I/O</span>
        <span class="k">try</span> <span class="o">(</span><span class="kt">var</span> <span class="n">executor</span> <span class="o">=</span> <span class="nc">Executors</span><span class="o">.</span><span class="na">newVirtualThreadPerTaskExecutor</span><span class="o">())</span> <span class="o">{</span>
            <span class="n">executor</span><span class="o">.</span><span class="na">submit</span><span class="o">(()</span> <span class="o">-&gt;</span> <span class="o">{</span>
                <span class="c1">// Simulate Database Latency</span>
                <span class="nc">Thread</span><span class="o">.</span><span class="na">sleep</span><span class="o">(</span><span class="mi">100</span><span class="o">);</span> 
                <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Transaction processed by: "</span> <span class="o">+</span> <span class="nc">Thread</span><span class="o">.</span><span class="na">currentThread</span><span class="o">());</span>
            <span class="o">});</span>
        <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
            <span class="k">return</span> <span class="s">"Failed"</span><span class="o">;</span>
        <span class="o">}</span>
        <span class="k">return</span> <span class="s">"Transaction Queued"</span><span class="o">;</span>
    <span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>

<p><img src="https://blogs.buildwithmanish.com/assets/images/architecture_diagram-top-5-programming-languages-to-learn-in-2026-6-2025-12-31-45787.png" alt="Java Architecture" /></p>

<h3 id="career--salary-2">Career &amp; Salary</h3>
<p>Jobs in Java are massive in volume. If you know Spring Boot and Microservices architecture, you will never be unemployed. In India, salaries are comparable to Python but often offer more stability in traditional banking and finance sectors.</p>

<hr />

<h1 id="5-rust-the-future-of-systems-engineering">5. Rust: The Future of Systems Engineering</h1>

<h3 id="the-context-4">The Context</h3>
<p>At number five, we have the challenger: Rust.</p>

<p><strong>Who is this for?</strong></p>
<ul>
  <li><strong>Intermediate/Senior Developers:</strong> (4-7 years experience).</li>
  <li><strong>Performance Enthusiasts:</strong> Those who want C++ speed with memory safety guarantees.</li>
</ul>

<h3 id="the-data-4">The Data</h3>
<ul>
  <li><strong>Adoption:</strong> Used in the Linux Kernel, Windows Kernel, and core tools like <code class="language-plaintext highlighter-rouge">uv</code> (Python package manager) and <code class="language-plaintext highlighter-rouge">swc</code> (JS compiler).</li>
  <li><strong>Salary:</strong> While job volume is lower than Java/Python, the <strong>salary per job is significantly higher</strong>.</li>
</ul>

<h3 id="technical-deep-dive-the-borrow-checker">Technical Deep Dive: The Borrow Checker</h3>

<p>Rust solves the biggest problem in C++: Memory Safety. In C++, you can accidentally access memory that has been freed (Use-After-Free), leading to crashes or security vulnerabilities.</p>

<p>Rust prevents this at <strong>compile time</strong> using the <strong>Borrow Checker</strong>. It introduces the concept of <strong>Ownership</strong>.</p>
<ol>
  <li>Each value in Rust has a variable that’s called its owner.</li>
  <li>There can only be one owner at a time.</li>
  <li>When the owner goes out of scope, the value will be dropped.</li>
</ol>

<p><strong>Code Example: Fighting (and winning) against the Borrow Checker</strong></p>

<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="n">thread</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="nn">sync</span><span class="p">::{</span><span class="nb">Arc</span><span class="p">,</span> <span class="n">Mutex</span><span class="p">};</span>

<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
    <span class="c1">// Arc (Atomic Reference Counting) allows shared ownership across threads</span>
    <span class="c1">// Mutex allows safe mutability</span>
    <span class="k">let</span> <span class="n">counter</span> <span class="o">=</span> <span class="nn">Arc</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="nn">Mutex</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="mi">0</span><span class="p">));</span>
    <span class="k">let</span> <span class="k">mut</span> <span class="n">handles</span> <span class="o">=</span> <span class="nd">vec!</span><span class="p">[];</span>

    <span class="k">for</span> <span class="n">_</span> <span class="k">in</span> <span class="mi">0</span><span class="o">..</span><span class="mi">10</span> <span class="p">{</span>
        <span class="k">let</span> <span class="n">counter</span> <span class="o">=</span> <span class="nn">Arc</span><span class="p">::</span><span class="nf">clone</span><span class="p">(</span><span class="o">&amp;</span><span class="n">counter</span><span class="p">);</span>
        <span class="k">let</span> <span class="n">handle</span> <span class="o">=</span> <span class="nn">thread</span><span class="p">::</span><span class="nf">spawn</span><span class="p">(</span><span class="k">move</span> <span class="p">||</span> <span class="p">{</span>
            <span class="c1">// Lock the mutex to get access to the data</span>
            <span class="k">let</span> <span class="k">mut</span> <span class="n">num</span> <span class="o">=</span> <span class="n">counter</span><span class="nf">.lock</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">();</span>
            <span class="o">*</span><span class="n">num</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">;</span>
        <span class="p">});</span>
        <span class="n">handles</span><span class="nf">.push</span><span class="p">(</span><span class="n">handle</span><span class="p">);</span>
    <span class="p">}</span>

    <span class="k">for</span> <span class="n">handle</span> <span class="k">in</span> <span class="n">handles</span> <span class="p">{</span>
        <span class="n">handle</span><span class="nf">.join</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">();</span>
    <span class="p">}</span>

    <span class="c1">// In C++, doing this without race conditions is incredibly hard.</span>
    <span class="c1">// In Rust, the compiler REFUSES to build if you do it wrong.</span>
    <span class="nd">println!</span><span class="p">(</span><span class="s">"Result: {}"</span><span class="p">,</span> <span class="o">*</span><span class="n">counter</span><span class="nf">.lock</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">());</span>
<span class="p">}</span>
</code></pre></div></div>

<p><img src="https://blogs.buildwithmanish.com/assets/images/concept_illustration-top-5-programming-languages-to-learn-in-2026-7-2025-12-31-3931.png" alt="Rust Concept Illustration" /></p>

<h3 id="career--salary-3">Career &amp; Salary</h3>
<p>Rust developers are rare. Supply is low, demand is high (especially in Silicon Valley and remote roles). If you master Rust, you are positioning yourself for top-tier engineering roles globally.</p>

<hr />

<h1 id="honorable-mentions--conclusion">Honorable Mentions &amp; Conclusion</h1>

<p>While the top 5 are clear, we cannot ignore:</p>
<ul>
  <li><strong>C# (.NET):</strong> Still the king of Game Development (Unity) and strong in enterprise.</li>
  <li><strong>Go (Golang):</strong> The language of the Cloud. Kubernetes and Docker are written in Go.</li>
  <li><strong>Kotlin:</strong> The default for native Android development.</li>
</ul>

<h3 id="final-verdict-context-is-everything">Final Verdict: Context is Everything</h3>

<p>I will end this deep dive where I started. There is no “Best Language.”</p>

<ul>
  <li>If you are building an AI startup: <strong>Python</strong>.</li>
  <li>If you are building a SaaS frontend: <strong>TypeScript</strong>.</li>
  <li>If you are writing a Game Engine: <strong>C++</strong>.</li>
  <li>If you are building a Banking API: <strong>Java</strong>.</li>
  <li>If you are building the next Browser Engine: <strong>Rust</strong>.</li>
</ul>

<p>Real-world engineering limits your choices. Sometimes you join a company and <em>have</em> to write Java. Sometimes you inherit a legacy codebase in PHP. That is the reality.</p>

<p>However, if you have the luxury of time-if you are in your 20s, entering college, or looking to upskill-choose the tool that aligns with the problem you want to solve.</p>

<p>I hope this technical breakdown gives you the clarity to navigate 2026. Code is just a tool; what matters is what you build with it.</p>

<p><em>See you in the next commit.</em></p>]]></content><author><name>Manish Tiwari</name></author><category term="deep-dive" /><category term="career-growth" /><category term="software-engineering" /><category term="system-design" /><category term="tech-trends" /><summary type="html"><![CDATA[]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" /><media:content medium="image" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Relational Database Management System: An Engineering Deep-Dive</title><link href="https://blogs.buildwithmanish.com/deep-dive/database-systems/sql/data-modeling/backend-engineering/2025/12/30/relational-database-management-system.html" rel="alternate" type="text/html" title="Relational Database Management System: An Engineering Deep-Dive" /><published>2025-12-30T00:00:00+00:00</published><updated>2025-12-30T00:00:00+00:00</updated><id>https://blogs.buildwithmanish.com/deep-dive/database-systems/sql/data-modeling/backend-engineering/2025/12/30/relational-database-management-system</id><content type="html" xml:base="https://blogs.buildwithmanish.com/deep-dive/database-systems/sql/data-modeling/backend-engineering/2025/12/30/relational-database-management-system.html"><![CDATA[<p><img src="https://blogs.buildwithmanish.com/assets/images/hero-relational-database-management-system-0-2025-12-30-77512.png" alt="Relational Database Management System Hero" /></p>

<p>Despite the meteoric rise of NoSQL, time-series, and vector databases in the last decade, the Relational Database Management System (RDBMS) remains the immutable backbone of global enterprise software. Whether it’s financial ledgers, inventory management, or user identity systems, the relational model’s promise of ACID (Atomicity, Consistency, Isolation, Durability) compliance and mathematical rigor is irreplaceable.</p>

<p>This article is not a high-level overview. We are going to deconstruct the RDBMS from the ground up-from the theoretical underpinnings of tuple calculus and storage architecture to the pragmatic realities of SQL optimization and schema normalization.</p>

<hr />

<h1 id="unit-1-the-foundation--architecture--the-relational-model">Unit 1: The Foundation – Architecture &amp; The Relational Model</h1>

<p>To engineer robust systems, one must understand what happens <em>before</em> a query is executed. An RDBMS is not merely a data bucket; it is a complex engine designed to bridge the gap between logical data representation and physical storage.</p>

<h2 id="purpose-and-view-of-data">Purpose and View of Data</h2>
<p>Legacy file processing systems suffered from data redundancy, isolation, and lack of atomicity. If a file transfer crashed halfway, data was corrupted. RDBMS solves this via <strong>Data Abstraction</strong>:</p>

<ol>
  <li><strong>Physical Level:</strong> How data is actually stored (block sizes, B-Trees, hashing).</li>
  <li><strong>Logical Level:</strong> What data is stored and the relationships (Tables, Columns).</li>
  <li><strong>View Level:</strong> Virtual tables (Views) that hide complexity or sensitive data from specific users.</li>
</ol>

<h2 id="database-architecture">Database Architecture</h2>
<p>The architecture of a database determines its throughput and reliability. It is generally split into the <strong>Query Processor</strong> and the <strong>Storage Manager</strong>.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/architecture_diagram-relational-database-management-system-1-2025-12-30-90537.png" alt="Database Architecture Diagram" /></p>

<h3 id="1-the-storage-manager">1. The Storage Manager</h3>
<p>This is the interface between the low-level data stored in the OS file system and the application.</p>
<ul>
  <li><strong>Buffer Manager:</strong> Critical for performance. It manages caching data blocks in RAM to minimize expensive disk I/O. If your DB is slow, check your buffer pool hit ratio.</li>
  <li><strong>File Manager:</strong> Manages allocation of space on disk structures.</li>
  <li><strong>Transaction Manager:</strong> Ensures the system remains in a consistent state despite failures (Power loss, crashes).</li>
</ul>

<h3 id="2-the-query-processor">2. The Query Processor</h3>
<p>This is the “brain” of the operation.</p>
<ul>
  <li><strong>DDL Interpreter:</strong> Interprets schema definitions.</li>
  <li><strong>DML Compiler:</strong> Translates SQL into an evaluation plan (relational algebra).</li>
  <li><strong>Query Evaluation Engine:</strong> Executes the low-level instructions generated by the compiler.</li>
</ul>

<h2 id="the-relational-model">The Relational Model</h2>
<p>Proposed by E.F. Codd, the relational model is based on set theory.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/technical_diagram-relational-database-management-system-2-2025-12-30-92972.png" alt="Relational Model Diagram" /></p>

<ul>
  <li><strong>Relation (Table):</strong> A set of tuples.</li>
  <li><strong>Tuple (Row):</strong> An ordered list of values.</li>
  <li><strong>Attribute (Column):</strong> A named property of the relation.</li>
  <li><strong>Domain:</strong> The set of permitted values for an attribute (e.g., Integer, Varchar).</li>
</ul>

<h3 id="keys-the-guardians-of-integrity">Keys: The Guardians of Integrity</h3>
<p>Keys are not just for indexing; they strictly define identity.</p>

<ol>
  <li><strong>Super Key:</strong> A set of one or more attributes that uniquely identify a tuple.</li>
  <li><strong>Candidate Key:</strong> A minimal super key (no extraneous attributes).</li>
  <li><strong>Primary Key (PK):</strong> The candidate key chosen by the database designer as the principal means of identification.</li>
  <li><strong>Foreign Key (FK):</strong> An attribute in one table that references the PK of another. This enforces <strong>Referential Integrity</strong>.</li>
</ol>

<p><strong>Scenario:</strong> If you have an <code class="language-plaintext highlighter-rouge">Orders</code> table referencing a <code class="language-plaintext highlighter-rouge">Users</code> table via <code class="language-plaintext highlighter-rouge">user_id</code>, the database engine (via FK constraints) physically prevents you from deleting a User who has active orders, preventing orphaned data.</p>

<hr />

<h1 id="unit-2-database-design--normalization">Unit 2: Database Design &amp; Normalization</h1>

<p>A poorly designed schema is a technical debt that gathers interest at a compound rate. Database design is the process of converting real-world requirements into a mathematical model that minimizes redundancy.</p>

<h2 id="entity-relationship-e-r-model">Entity-Relationship (E-R) Model</h2>
<p>Before writing SQL, we model. The E-R model represents the world in terms of <strong>Entities</strong> (objects) and <strong>Relationships</strong> (associations).</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/technical_diagram-relational-database-management-system-3-2025-12-30-80972.png" alt="E-R Model Diagram" /></p>

<h3 id="core-components">Core Components</h3>
<ul>
  <li><strong>Entity Sets:</strong> E.g., <code class="language-plaintext highlighter-rouge">Employee</code>.</li>
  <li><strong>Attributes:</strong> Properties like <code class="language-plaintext highlighter-rouge">SSN</code>, <code class="language-plaintext highlighter-rouge">Name</code>.
    <ul>
      <li><em>Derived Attributes:</em> Calculated from others (e.g., <code class="language-plaintext highlighter-rouge">Age</code> from <code class="language-plaintext highlighter-rouge">DOB</code>).</li>
      <li><em>Multivalued Attributes:</em> E.g., <code class="language-plaintext highlighter-rouge">Phone_Numbers</code>.</li>
    </ul>
  </li>
  <li><strong>Relationships:</strong> The association between entities (e.g., <code class="language-plaintext highlighter-rouge">Works_For</code>).</li>
</ul>

<h3 id="constraints--cardinality">Constraints &amp; Cardinality</h3>
<ul>
  <li><strong>One-to-One (1:1):</strong> CEO manages one Company.</li>
  <li><strong>One-to-Many (1:N):</strong> Department employs many Employees.</li>
  <li><strong>Many-to-Many (M:N):</strong> Students enroll in many Courses.</li>
</ul>

<p><strong>Design Issue - Reduction to Relational Schemas:</strong>
When converting E-R to Tables:</p>
<ol>
  <li><strong>Strong Entity Sets</strong> become their own tables.</li>
  <li><strong>Weak Entity Sets</strong> (depend on a strong entity for existence) include the strong entity’s PK as a Foreign Key.</li>
  <li><strong>M:N Relationships</strong> require a <strong>Join Table</strong> (associative entity) containing the PKs of both participating entities.</li>
</ol>

<h2 id="relational-database-design-normalization">Relational Database Design: Normalization</h2>

<p>Normalization is the process of organizing data to reduce redundancy and improve data integrity. It relies heavily on <strong>Functional Dependencies (FDs)</strong>.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/concept_illustration-relational-database-management-system-4-2025-12-30-90188.png" alt="Relational Database Design Concept" /></p>

<h3 id="functional-dependency-theory">Functional Dependency Theory</h3>
<p>An FD is denoted as $X \rightarrow Y$. It means if two tuples agree on attribute $X$, they must agree on attribute $Y$.</p>
<ul>
  <li><strong>Axioms of Armstrong:</strong> Reflexivity, Augmentation, Transitivity. These allow us to infer new dependencies.</li>
</ul>

<h3 id="the-normal-forms">The Normal Forms</h3>
<p>We decompose tables to eliminate anomalies (Update, Insert, Delete anomalies).</p>

<h4 id="1-first-normal-form-1nf">1. First Normal Form (1NF)</h4>
<p><strong>Rule:</strong> Attributes must be atomic. No repeating groups or arrays.</p>
<ul>
  <li><em>Bad:</em> <code class="language-plaintext highlighter-rouge">User(ID, Name, Phones)</code> where Phones is “555-0101, 555-0102”.</li>
  <li><em>Good:</em> Move phones to a separate table <code class="language-plaintext highlighter-rouge">UserPhones(UserID, Phone)</code>.</li>
</ul>

<h4 id="2-second-normal-form-2nf">2. Second Normal Form (2NF)</h4>
<p><strong>Rule:</strong> Must be in 1NF AND no <strong>Partial Dependencies</strong>.</p>
<ul>
  <li><em>Context:</em> Applies when the Primary Key is composite.</li>
  <li><em>Violation:</em> <code class="language-plaintext highlighter-rouge">OrderItems(OrderID, ProductID, ProductName, Quantity)</code>. PK is <code class="language-plaintext highlighter-rouge">(OrderID, ProductID)</code>. <code class="language-plaintext highlighter-rouge">ProductName</code> depends only on <code class="language-plaintext highlighter-rouge">ProductID</code>, not the whole key.</li>
  <li><em>Fix:</em> Split into <code class="language-plaintext highlighter-rouge">Products(ProductID, ProductName)</code> and <code class="language-plaintext highlighter-rouge">OrderItems(OrderID, ProductID, Quantity)</code>.</li>
</ul>

<h4 id="3-third-normal-form-3nf">3. Third Normal Form (3NF)</h4>
<p><strong>Rule:</strong> Must be in 2NF AND no <strong>Transitive Dependencies</strong>.</p>
<ul>
  <li><em>Violation:</em> <code class="language-plaintext highlighter-rouge">Employee(ID, ZipCode, City)</code>. <code class="language-plaintext highlighter-rouge">ID -&gt; ZipCode</code>, but <code class="language-plaintext highlighter-rouge">ZipCode -&gt; City</code>.</li>
  <li><em>Fix:</em> <code class="language-plaintext highlighter-rouge">Employee(ID, ZipCode)</code> and <code class="language-plaintext highlighter-rouge">Locations(ZipCode, City)</code>.</li>
</ul>

<h4 id="4-bcnf-boyce-codd-normal-form">4. BCNF (Boyce-Codd Normal Form)</h4>
<p>A stricter version of 3NF. For every FD $X \rightarrow Y$, $X$ must be a super key. This handles edge cases where a prime attribute depends on a non-prime attribute.</p>

<p><strong>Decomposition Algorithm:</strong>
When decomposing tables, two properties must be preserved:</p>
<ol>
  <li><strong>Lossless Join:</strong> Can we join the tables back to get the original data without creating spurious rows?</li>
  <li><strong>Dependency Preservation:</strong> Can we enforce constraints without joining tables?</li>
</ol>

<hr />

<h1 id="unit-3-sql-basics--definition--control">Unit 3: SQL Basics – Definition &amp; Control</h1>

<p>SQL (Structured Query Language) is the implementation of the relational model. It is declarative: you tell the DB <em>what</em> you want, not <em>how</em> to get it.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/comparison_chart-relational-database-management-system-5-2025-12-30-31040.png" alt="SQL Basics Comparison Chart" /></p>

<h2 id="data-definition-language-ddl">Data Definition Language (DDL)</h2>
<p>DDL defines the structure. In a production environment, DDL is dangerous. Locking a table to add a column can bring down a high-traffic service.</p>

<h3 id="creating-tables-with-constraints">Creating Tables with Constraints</h3>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">Employees</span> <span class="p">(</span>
    <span class="n">emp_id</span> <span class="nb">INT</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span>
    <span class="n">email</span> <span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">255</span><span class="p">)</span> <span class="k">UNIQUE</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
    <span class="n">salary</span> <span class="nb">DECIMAL</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="k">CHECK</span> <span class="p">(</span><span class="n">salary</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">),</span>
    <span class="n">dept_id</span> <span class="nb">INT</span><span class="p">,</span>
    <span class="n">hire_date</span> <span class="nb">DATE</span> <span class="k">DEFAULT</span> <span class="k">CURRENT_DATE</span><span class="p">,</span>
    <span class="k">CONSTRAINT</span> <span class="n">fk_dept</span> 
        <span class="k">FOREIGN</span> <span class="k">KEY</span> <span class="p">(</span><span class="n">dept_id</span><span class="p">)</span> 
        <span class="k">REFERENCES</span> <span class="n">Departments</span><span class="p">(</span><span class="n">dept_id</span><span class="p">)</span>
        <span class="k">ON</span> <span class="k">DELETE</span> <span class="k">SET</span> <span class="k">NULL</span>
<span class="p">);</span>
</code></pre></div></div>
<ul>
  <li><strong>Constraint Naming:</strong> Always name your constraints (<code class="language-plaintext highlighter-rouge">fk_dept</code>). It makes debugging error logs significantly easier.</li>
  <li><strong>Types:</strong> <code class="language-plaintext highlighter-rouge">INT</code>, <code class="language-plaintext highlighter-rouge">VARCHAR</code>, <code class="language-plaintext highlighter-rouge">DECIMAL</code> (for money-never use FLOAT), <code class="language-plaintext highlighter-rouge">DATE</code>.</li>
</ul>

<h3 id="altering-and-dropping">Altering and Dropping</h3>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">-- Adding a column</span>
<span class="k">ALTER</span> <span class="k">TABLE</span> <span class="n">Employees</span> <span class="k">ADD</span> <span class="k">COLUMN</span> <span class="n">phone_number</span> <span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">20</span><span class="p">);</span>

<span class="c1">-- Renaming (Database specific, standard SQL uses RENAME TO)</span>
<span class="k">ALTER</span> <span class="k">TABLE</span> <span class="n">Employees</span> <span class="k">RENAME</span> <span class="k">TO</span> <span class="n">Staff</span><span class="p">;</span>

<span class="c1">-- Truncate vs Delete</span>
<span class="k">TRUNCATE</span> <span class="k">TABLE</span> <span class="n">Staff</span><span class="p">;</span> <span class="c1">-- Fast, resets High Water Mark, no undo (DDL)</span>
<span class="k">DELETE</span> <span class="k">FROM</span> <span class="n">Staff</span><span class="p">;</span>    <span class="c1">-- Slower, logs every row, transactional (DML)</span>
</code></pre></div></div>

<h2 id="dcl-data-control-language">DCL (Data Control Language)</h2>
<p>Security is paramount.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">GRANT</span> <span class="k">SELECT</span><span class="p">,</span> <span class="k">INSERT</span> <span class="k">ON</span> <span class="n">Employees</span> <span class="k">TO</span> <span class="n">app_user</span><span class="p">;</span>
<span class="k">REVOKE</span> <span class="k">DELETE</span> <span class="k">ON</span> <span class="n">Employees</span> <span class="k">FROM</span> <span class="n">app_user</span><span class="p">;</span>
</code></pre></div></div>

<h2 id="error-codes--spooling">Error Codes &amp; Spooling</h2>
<ul>
  <li><strong>Spooling:</strong> In CLI tools (like SQL*Plus), spooling directs query output to a file. Useful for generating CSV reports via cron jobs.</li>
  <li><strong>Error Codes:</strong> Applications should trap SQLSTATE codes. For example, a duplicate key violation usually returns specific codes (e.g., 23505 in Postgres) allowing the app to fail gracefully.</li>
</ul>

<hr />

<h1 id="unit-4-data-management-retrieval--functions">Unit 4: Data Management, Retrieval &amp; Functions</h1>

<p>This is where the rubber meets the road. Writing efficient DML (Data Manipulation Language) is the primary skill of a backend engineer.</p>

<h2 id="dml-managing-data">DML: Managing Data</h2>
<p>The <code class="language-plaintext highlighter-rouge">INSERT</code>, <code class="language-plaintext highlighter-rouge">UPDATE</code>, and <code class="language-plaintext highlighter-rouge">DELETE</code> statements are straightforward but powerful.</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">-- Multi-row insert</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">Products</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">price</span><span class="p">)</span> <span class="k">VALUES</span> 
<span class="p">(</span><span class="s1">'Widget A'</span><span class="p">,</span> <span class="mi">10</span><span class="p">.</span><span class="mi">00</span><span class="p">),</span>
<span class="p">(</span><span class="s1">'Widget B'</span><span class="p">,</span> <span class="mi">15</span><span class="p">.</span><span class="mi">50</span><span class="p">);</span>

<span class="c1">-- Update with logic</span>
<span class="k">UPDATE</span> <span class="n">Employees</span> 
<span class="k">SET</span> <span class="n">salary</span> <span class="o">=</span> <span class="n">salary</span> <span class="o">*</span> <span class="mi">1</span><span class="p">.</span><span class="mi">05</span> 
<span class="k">WHERE</span> <span class="n">performance_rating</span> <span class="o">&gt;</span> <span class="mi">4</span><span class="p">;</span>

<span class="c1">-- Substitution Variables (Tool specific, e.g., SQL*Plus)</span>
<span class="c1">-- DEFINE department_id = 10;</span>
<span class="c1">-- SELECT * FROM Employees WHERE dept_id = &amp;department_id;</span>
</code></pre></div></div>

<h2 id="advanced-retrieval-the-select-anatomy">Advanced Retrieval: The SELECT Anatomy</h2>
<p>The execution order of a SQL query is different from the writing order:</p>
<ol>
  <li><strong>FROM / JOIN</strong> (Gather data)</li>
  <li><strong>WHERE</strong> (Filter rows)</li>
  <li><strong>GROUP BY</strong> (Aggregate)</li>
  <li><strong>HAVING</strong> (Filter groups)</li>
  <li><strong>SELECT</strong> (Return columns)</li>
  <li><strong>ORDER BY</strong> (Sort)</li>
</ol>

<h3 id="conditional-logic-case">Conditional Logic: CASE</h3>
<p>The <code class="language-plaintext highlighter-rouge">CASE</code> statement allows for if-then logic directly in the projection.</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> 
    <span class="n">name</span><span class="p">,</span>
    <span class="n">salary</span><span class="p">,</span>
    <span class="k">CASE</span> 
        <span class="k">WHEN</span> <span class="n">salary</span> <span class="o">&lt;</span> <span class="mi">50000</span> <span class="k">THEN</span> <span class="s1">'Junior'</span>
        <span class="k">WHEN</span> <span class="n">salary</span> <span class="k">BETWEEN</span> <span class="mi">50000</span> <span class="k">AND</span> <span class="mi">100000</span> <span class="k">THEN</span> <span class="s1">'Mid'</span>
        <span class="k">ELSE</span> <span class="s1">'Senior'</span>
    <span class="k">END</span> <span class="k">as</span> <span class="n">seniority_level</span>
<span class="k">FROM</span> <span class="n">Employees</span><span class="p">;</span>
</code></pre></div></div>

<h2 id="multiple-tables-joins">Multiple Tables: Joins</h2>
<p>Joins are the physical realization of the Cartesian product filtered by a predicate.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/technical_diagram-relational-database-management-system-6-2025-12-30-17942.png" alt="Joins Diagram" /></p>

<ol>
  <li><strong>Inner Join:</strong> Returns rows when there is a match in both tables.</li>
  <li><strong>Left (Outer) Join:</strong> All rows from the left table, and matched rows from the right table (NULL if no match).</li>
  <li><strong>Right (Outer) Join:</strong> Inverse of Left Join.</li>
  <li><strong>Full Join:</strong> All rows from both tables.</li>
  <li><strong>Self Join:</strong> Joining a table to itself (e.g., Employee hierarchy where Manager is also an Employee).</li>
</ol>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> 
    <span class="n">e</span><span class="p">.</span><span class="n">name</span> <span class="k">as</span> <span class="n">Employee</span><span class="p">,</span> 
    <span class="n">m</span><span class="p">.</span><span class="n">name</span> <span class="k">as</span> <span class="n">Manager</span>
<span class="k">FROM</span> <span class="n">Employees</span> <span class="n">e</span>
<span class="k">LEFT</span> <span class="k">JOIN</span> <span class="n">Employees</span> <span class="n">m</span> <span class="k">ON</span> <span class="n">e</span><span class="p">.</span><span class="n">manager_id</span> <span class="o">=</span> <span class="n">m</span><span class="p">.</span><span class="n">emp_id</span><span class="p">;</span>
</code></pre></div></div>

<h2 id="functions-and-grouping">Functions and Grouping</h2>
<p>Aggregation reduces multiple rows into a single summary value.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/code_flow-relational-database-management-system-7-2025-12-30-6284.png" alt="Functions and Grouping Flow" /></p>

<h3 id="built-in-functions">Built-in Functions</h3>
<ul>
  <li><strong>Numeric:</strong> <code class="language-plaintext highlighter-rouge">ROUND()</code>, <code class="language-plaintext highlighter-rouge">TRUNC()</code>, <code class="language-plaintext highlighter-rouge">MOD()</code>.</li>
  <li><strong>String:</strong> <code class="language-plaintext highlighter-rouge">UPPER()</code>, <code class="language-plaintext highlighter-rouge">SUBSTR()</code>, <code class="language-plaintext highlighter-rouge">CONCAT()</code>.</li>
  <li><strong>Date:</strong> <code class="language-plaintext highlighter-rouge">ADD_MONTHS()</code>, <code class="language-plaintext highlighter-rouge">EXTRACT(YEAR FROM date)</code>.</li>
</ul>

<h3 id="grouping-and-having">Grouping and HAVING</h3>
<p>The <code class="language-plaintext highlighter-rouge">HAVING</code> clause is essential because the <code class="language-plaintext highlighter-rouge">WHERE</code> clause cannot filter on aggregate functions (because <code class="language-plaintext highlighter-rouge">WHERE</code> runs <em>before</em> aggregation).</p>

<p><strong>Scenario:</strong> Find departments where the average salary is greater than $80,000.</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> 
    <span class="n">dept_id</span><span class="p">,</span> 
    <span class="k">AVG</span><span class="p">(</span><span class="n">salary</span><span class="p">)</span> <span class="k">as</span> <span class="n">avg_sal</span>
<span class="k">FROM</span> <span class="n">Employees</span>
<span class="k">WHERE</span> <span class="n">status</span> <span class="o">=</span> <span class="s1">'Active'</span>  <span class="c1">-- Filter raw rows first</span>
<span class="k">GROUP</span> <span class="k">BY</span> <span class="n">dept_id</span>         <span class="c1">-- Aggregate</span>
<span class="k">HAVING</span> <span class="k">AVG</span><span class="p">(</span><span class="n">salary</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">80000</span><span class="p">;</span> <span class="c1">-- Filter the aggregates</span>
</code></pre></div></div>

<h3 id="set-operations">Set Operations</h3>
<p>These combine the results of two separate queries.</p>
<ul>
  <li><strong>UNION:</strong> Combines results, removes duplicates.</li>
  <li><strong>UNION ALL:</strong> Combines results, keeps duplicates (Faster).</li>
  <li><strong>INTERSECT:</strong> Returns only rows present in both sets.</li>
  <li><strong>MINUS / EXCEPT:</strong> Returns rows in the first set but not the second.</li>
</ul>

<hr />

<h1 id="conclusion">Conclusion</h1>

<p>The Relational Database Management System is a triumph of computer science, blending rigid mathematical theory with practical software engineering. From the low-level <strong>Storage Manager</strong> optimizing disk access to the high-level <strong>SQL</strong> logic handling complex joins and aggregations, the RDBMS is designed for data integrity and reliability.</p>

<p>As you design systems, remember:</p>
<ol>
  <li><strong>Normalize</strong> to ensure consistency, but be willing to <strong>Denormalize</strong> judiciously for read performance.</li>
  <li><strong>Constraints</strong> are your friends; let the database enforce the rules, not just the application code.</li>
  <li>Understand the <strong>Execution Order</strong> of SQL to write performant queries.</li>
</ol>

<p>Mastering these units provides the capability to handle not just a few thousand records, but to architect systems that scale to terabytes of mission-critical data.</p>]]></content><author><name>Manish Tiwari</name></author><category term="deep-dive" /><category term="database-systems" /><category term="sql" /><category term="data-modeling" /><category term="backend-engineering" /><summary type="html"><![CDATA[]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" /><media:content medium="image" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Textbook for Excel Exam</title><link href="https://blogs.buildwithmanish.com/deep-dive/excel-mastery/data-analysis/spreadsheet-engineering/certification/2025/12/29/textbook-for-excel-exam.html" rel="alternate" type="text/html" title="Textbook for Excel Exam" /><published>2025-12-29T00:00:00+00:00</published><updated>2025-12-29T00:00:00+00:00</updated><id>https://blogs.buildwithmanish.com/deep-dive/excel-mastery/data-analysis/spreadsheet-engineering/certification/2025/12/29/textbook-for-excel-exam</id><content type="html" xml:base="https://blogs.buildwithmanish.com/deep-dive/excel-mastery/data-analysis/spreadsheet-engineering/certification/2025/12/29/textbook-for-excel-exam.html"><![CDATA[<p><img src="https://blogs.buildwithmanish.com/assets/images/hero-textbook-for-excel-exam-0-2025-12-29-23091.png" alt="Hero Image" /></p>

<h1 id="introduction">Introduction</h1>

<p>In the world of data engineering and business intelligence, Microsoft Excel remains the ubiquitous “first draft” of data analysis. While Python, SQL, and Tableau dominate big data pipelines, Excel is where the business world lives. It is effectively a functional programming environment coupled with a two-dimensional visual database.</p>

<p>This guide serves as a comprehensive “textbook” designed specifically for the <strong>Basics of MS Excel Course End Semester Examination</strong>. We are not just covering button clicks; we are deconstructing the architecture of Excel to ensure you can pass the <strong>MCQ Test (40 Marks)</strong> and the <strong>Workbook Submission (40 Marks)</strong> with distinction.</p>

<p>We will traverse the four mandatory units, treating the spreadsheet not as a grid of boxes, but as a development environment.</p>

<hr />

<h1 id="unit-i-the-excel-environment--architecture">UNIT-I: The Excel Environment &amp; Architecture</h1>

<p>Before executing complex logic, one must master the IDE (Integrated Development Environment). In Excel, the interface is your IDE.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/technical_diagram-textbook-for-excel-exam-1-2025-12-29-51456.png" alt="Technical Diagram" /></p>

<h2 id="1-anatomy-of-the-interface">1. Anatomy of the Interface</h2>

<h3 id="the-ribbon--quick-access-toolbar-qat">The Ribbon &amp; Quick Access Toolbar (QAT)</h3>
<p>The <strong>Ribbon</strong> is organized into Tabs (Home, Insert, Data, View). Think of these as namespaces containing specific functions.</p>
<ul>
  <li><strong>Ribbon Display Options:</strong> Located in the top-right, allows you to Auto-hide, Show Tabs, or Show Tabs and Commands. Critical for maximizing screen real estate during complex modeling.</li>
  <li><strong>QAT:</strong> The only customizable toolbar that persists across tabs. <em>Pro Tip: Add “Paste Special” and “Filter” here for O(1) access.</em></li>
</ul>

<h3 id="the-formula-bar--name-box">The Formula Bar &amp; Name Box</h3>
<ul>
  <li><strong>Name Box:</strong> Located to the left of the formula bar. It usually displays the cell address (e.g., <code class="language-plaintext highlighter-rouge">A1</code>). However, it is primarily used for <strong>Named Ranges</strong>. Naming a cell <code class="language-plaintext highlighter-rouge">TaxRate</code> allows you to refer to it globally, eliminating “magic numbers” in your formulas.</li>
  <li><strong>Formula Bar:</strong> Your code editor. It handles formula input and debugging.</li>
</ul>

<h3 id="views--zoom">Views &amp; Zoom</h3>
<ul>
  <li><strong>Normal View:</strong> The default grid.</li>
  <li><strong>Page Layout View:</strong> Visualizes printed output (headers/footers visible).</li>
  <li><strong>Page Break Preview:</strong> Essential for defining print areas.</li>
</ul>

<h2 id="2-the-cell-the-atomic-unit">2. The Cell: The Atomic Unit</h2>
<p>A cell is an object that holds properties: Value, Formula, and Format.</p>

<h3 id="formatting--data-types">Formatting &amp; Data Types</h3>
<p>Excel attempts to infer data types (General), but explicit typing is required for data integrity.</p>
<ul>
  <li><strong>Number Formatting:</strong>
    <ul>
      <li><strong>Currency vs. Accounting:</strong> Currency allows custom placement of the symbol. Accounting aligns symbols to the left and decimals to the right for easier scanning.</li>
      <li><strong>Dates:</strong> Stored internally as serial numbers (1 = Jan 1, 1900). This allows mathematical operations on dates.</li>
    </ul>
  </li>
</ul>

<h3 id="managing-structure-rows-columns-and-worksheets">Managing Structure: Rows, Columns, and Worksheets</h3>
<ul>
  <li><strong>Worksheets:</strong> The individual pages. Copying a worksheet (<code class="language-plaintext highlighter-rouge">Ctrl + Drag</code>) creates a complete fork of the data and logic.</li>
  <li><strong>Resizing:</strong> Double-clicking the boundary between column headers performs an “AutoFit,” adjusting width to the longest data string.</li>
</ul>

<h2 id="3-keyboard-shortcuts-speed-engineering">3. Keyboard Shortcuts: Speed Engineering</h2>
<p>To pass the practical exam within the time limit, the mouse is your enemy.</p>

<table>
  <thead>
    <tr>
      <th style="text-align: left">Shortcut</th>
      <th style="text-align: left">Function</th>
      <th style="text-align: left">Context</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">Ctrl + Arrow</code></td>
      <td style="text-align: left">Navigation</td>
      <td style="text-align: left">Jumps to the edge of data regions.</td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">Ctrl + Shift + Arrow</code></td>
      <td style="text-align: left">Selection</td>
      <td style="text-align: left">Selects data from active cell to edge.</td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">Alt + =</code></td>
      <td style="text-align: left">AutoSum</td>
      <td style="text-align: left">Inserts <code class="language-plaintext highlighter-rouge">SUM()</code> function automatically.</td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">Ctrl + 1</code></td>
      <td style="text-align: left">Format Cells</td>
      <td style="text-align: left">Opens the detailed formatting dialog.</td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">F4</code></td>
      <td style="text-align: left">Repeat/Lock</td>
      <td style="text-align: left">Repeats last action OR cycles absolute references (<code class="language-plaintext highlighter-rouge">$A$1</code>).</td>
    </tr>
  </tbody>
</table>

<hr />

<h1 id="unit-ii-formulas-functions-and-tables">UNIT-II: Formulas, Functions, and Tables</h1>

<p>This unit moves from static data to dynamic computation.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/code_flow-textbook-for-excel-exam-2-2025-12-29-27532.png" alt="Code Flow" /></p>

<h2 id="1-formula-syntax-and-cell-referencing">1. Formula Syntax and Cell Referencing</h2>
<p>A formula always begins with <code class="language-plaintext highlighter-rouge">=</code>. The most critical concept here is <strong>Relative vs. Absolute Referencing</strong>.</p>

<ul>
  <li><strong>Relative (<code class="language-plaintext highlighter-rouge">A1</code>):</strong> Updates when copied. Used for iterating rows.</li>
  <li><strong>Absolute (<code class="language-plaintext highlighter-rouge">$A$1</code>):</strong> Locked constant. Used for parameters (e.g., Tax Rate).</li>
  <li><strong>Mixed (<code class="language-plaintext highlighter-rouge">$A1</code> or <code class="language-plaintext highlighter-rouge">A$1</code>):</strong> Locks only the column or row.</li>
</ul>

<h2 id="2-library-of-functions">2. Library of Functions</h2>

<h3 id="mathematical--statistical">Mathematical &amp; Statistical</h3>
<pre><code class="language-excel">=SUM(A1:A10)        -- Aggregate total
=AVERAGE(B1:B20)    -- Arithmetic mean
=COUNT(C1:C10)      -- Counts cells with numbers
=COUNTA(C1:C10)     -- Counts non-empty cells (includes text)
</code></pre>

<h3 id="text-manipulation-string-functions">Text Manipulation (String Functions)</h3>
<p>Data often arrives “dirty.” These functions sanitize it.</p>
<ul>
  <li><strong>CONCAT:</strong> Joins strings. <code class="language-plaintext highlighter-rouge">=CONCAT(A2, " ", B2)</code></li>
  <li><strong>LEFT/RIGHT/MID:</strong> Extract substrings.</li>
  <li><strong>TRIM:</strong> Removes leading/trailing whitespace (crucial for database matching).</li>
</ul>

<h3 id="logical-functions">Logical Functions</h3>
<p>The <code class="language-plaintext highlighter-rouge">IF</code> statement is the backbone of Excel logic.
<strong>Syntax:</strong> <code class="language-plaintext highlighter-rouge">=IF(logical_test, value_if_true, value_if_false)</code></p>

<p><strong>Nested Logic (AND/OR):</strong></p>
<pre><code class="language-excel">-- Return "Pass" if Score &gt; 50 AND Attendance &gt; 75%
=IF(AND(A2&gt;50, B2&gt;0.75), "Pass", "Fail")
</code></pre>

<h2 id="3-excel-tables-structured-references">3. Excel Tables (Structured References)</h2>

<p><img src="https://blogs.buildwithmanish.com/assets/images/comparison_chart-textbook-for-excel-exam-3-2025-12-29-23135.png" alt="Comparison Chart" /></p>

<p>Converting a range to a Table (<code class="language-plaintext highlighter-rouge">Ctrl + T</code>) transforms standard cells into a database object.</p>

<p><strong>Why use Tables?</strong></p>
<ol>
  <li><strong>Dynamic Ranges:</strong> Charts and PivotTables built on Tables update automatically when new rows are added.</li>
  <li><strong>Structured References:</strong> Formulas use column names instead of A1 syntax.
    <ul>
      <li><em>Standard:</em> <code class="language-plaintext highlighter-rouge">=C2 * D2</code></li>
      <li><em>Table:</em> <code class="language-plaintext highlighter-rouge">=[@Price] * [@Quantity]</code></li>
    </ul>
  </li>
  <li><strong>Calculated Columns:</strong> Typing a formula in one row automatically propagates it to the entire column.</li>
</ol>

<hr />

<h1 id="unit-iii-filters-sorting-and-visualization">UNIT-III: Filters, Sorting, and Visualization</h1>

<p>Once calculation is complete, data must be explored and presented.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/concept_illustration-textbook-for-excel-exam-4-2025-12-29-85967.png" alt="Concept Illustration" /></p>

<h2 id="1-sorting-and-filtering-the-query-layer">1. Sorting and Filtering (The Query Layer)</h2>

<h3 id="sorting">Sorting</h3>
<ul>
  <li><strong>Single Level:</strong> Basic A-Z.</li>
  <li><strong>Multi-Level Sort:</strong> Sort by Department (A-Z), <em>then</em> by Salary (Largest to Smallest). Accessed via Data &gt; Sort.</li>
  <li><strong>Custom Sort:</strong> Sorting by non-alphabetical logic (e.g., High, Medium, Low). You must define a Custom List in File &gt; Options.</li>
</ul>

<h3 id="filtering">Filtering</h3>
<p>Filtering hides rows that do not match criteria.</p>
<ul>
  <li><strong>Text Filters:</strong> Contains, Begins With.</li>
  <li><strong>Number Filters:</strong> Greater Than, Top 10.</li>
  <li><strong>Date Filters:</strong> Next Month, Year to Date.</li>
  <li><strong>Advanced Filter:</strong> Allows extracting unique records to a new location.</li>
</ul>

<h2 id="2-charts-data-visualization">2. Charts: Data Visualization</h2>
<p>A chart is a view layer on top of your data model.</p>

<h3 id="chart-types">Chart Types</h3>
<ol>
  <li><strong>Column/Bar:</strong> Comparing categories.</li>
  <li><strong>Line:</strong> Trends over time.</li>
  <li><strong>Pie:</strong> Parts of a whole (Use sparingly; humans are bad at judging angles).</li>
  <li><strong>Scatter:</strong> Correlation between two variables.</li>
</ol>

<h3 id="chart-architecture">Chart Architecture</h3>
<ul>
  <li><strong>Data Series:</strong> The actual values being plotted.</li>
  <li><strong>Axes:</strong> The scale (X and Y). Formatting axis bounds (Minimum/Maximum) is crucial for emphasizing differences.</li>
  <li><strong>Legend:</strong> Identifies the series.</li>
  <li><strong>Chart Layouts:</strong> Pre-defined templates for quick styling.</li>
</ul>

<p><strong>Scenario:</strong> <em>You need to plot Sales (Bars) and Profit Margin (Line) on the same chart.</em>
<strong>Solution:</strong> Use a <strong>Combo Chart</strong>. Set Sales to “Clustered Column” on the Primary Axis and Profit Margin to “Line” on the <strong>Secondary Axis</strong>.</p>

<hr />

<h1 id="unit-iv-pivot-tables--advanced-reporting">UNIT-IV: Pivot Tables &amp; Advanced Reporting</h1>

<p>Pivot Tables are the crown jewel of Excel. They function as an OLAP (Online Analytical Processing) tool, allowing you to summarize 100,000 rows of data in seconds without writing a single formula.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/architecture_diagram-textbook-for-excel-exam-5-2025-12-29-26388.png" alt="Architecture Diagram" /></p>

<h2 id="1-creating-a-pivot-table">1. Creating a Pivot Table</h2>
<ol>
  <li>Select Data (preferably a Table).</li>
  <li>Insert &gt; PivotTable.</li>
  <li><strong>The Cache:</strong> Excel takes a snapshot of your data into a memory cache. <em>Note: If source data changes, you must click “Refresh”.</em></li>
</ol>

<h2 id="2-the-four-quadrants">2. The Four Quadrants</h2>
<p>Understanding where to drag fields is key:</p>
<ul>
  <li><strong>Rows:</strong> Group data vertically (e.g., Region).</li>
  <li><strong>Columns:</strong> Group data horizontally (e.g., Year).</li>
  <li><strong>Values:</strong> The aggregation math (Sum, Count, Average).</li>
  <li><strong>Filters:</strong> Global slicer for the report.</li>
</ul>

<h2 id="3-manipulating-values">3. Manipulating Values</h2>
<p>By default, Excel sums numbers and counts text. You can change this behavior:</p>
<ul>
  <li><strong>Summarize Values By:</strong> Change Sum to Average, Max, Min.</li>
  <li><strong>Show Values As:</strong> Change raw numbers to <strong>% of Grand Total</strong> or <strong>Difference From</strong>. This is powerful for market share analysis.</li>
</ul>

<h2 id="4-pivot-charts">4. Pivot Charts</h2>
<p>A Pivot Chart is bound to the Pivot Table. Slicing the table (filtering) automatically updates the chart. This is the basis for creating interactive dashboards.</p>

<hr />

<h1 id="exam-preparation-guide">Exam Preparation Guide</h1>

<h2 id="part-i-mcq-strategy-15-hours">Part I: MCQ Strategy (1.5 Hours)</h2>
<p>Based on the referenced textbooks (Excel 2016 Bible, Excel 2019 All-in-One), expect questions on:</p>
<ol>
  <li><strong>File Extensions:</strong> <code class="language-plaintext highlighter-rouge">.xlsx</code> (standard), <code class="language-plaintext highlighter-rouge">.xlsm</code> (macro-enabled), <code class="language-plaintext highlighter-rouge">.csv</code> (comma separated).</li>
  <li><strong>Error Codes:</strong>
    <ul>
      <li><code class="language-plaintext highlighter-rouge">#DIV/0!</code>: Division by zero.</li>
      <li><code class="language-plaintext highlighter-rouge">#VALUE!</code>: Wrong argument type (adding text to number).</li>
      <li><code class="language-plaintext highlighter-rouge">#REF!</code>: Invalid cell reference (deleted row).</li>
    </ul>
  </li>
  <li><strong>Ribbon Locations:</strong> Which tab contains “Remove Duplicates”? (Answer: Data Tab).</li>
</ol>

<h2 id="part-ii-workbook-submission-strategy-40-marks">Part II: Workbook Submission Strategy (40 Marks)</h2>
<p>You will likely be given a raw dataset (e.g., Sales Data) and asked to perform specific tasks.</p>

<h3 id="sample-problem--solution-workflow">Sample Problem &amp; Solution Workflow</h3>

<p><strong>Problem:</strong> “Calculate the commission for each salesperson. If Sales &gt; 10,000, commission is 10%, otherwise 5%. Create a summary table showing total sales by Region.”</p>

<p><strong>Step-by-Step Implementation:</strong></p>

<ol>
  <li><strong>Data Entry &amp; Formatting:</strong>
    <ul>
      <li>Enter data. Format the ‘Sales’ column as Currency (<code class="language-plaintext highlighter-rouge">$</code>).</li>
      <li>Format the Header row (Bold, Background Color).</li>
    </ul>
  </li>
  <li><strong>Formula Logic (The <code class="language-plaintext highlighter-rouge">IF</code> Function):</strong>
    <ul>
      <li>In cell C2 (Commission), type:
        <pre><code class="language-excel">=IF(B2&gt;10000, B2*0.10, B2*0.05)
</code></pre>
      </li>
      <li>Double-click the fill handle to propagate down.</li>
    </ul>
  </li>
  <li><strong>Pivot Table (The Summary):</strong>
    <ul>
      <li>Select data. Insert PivotTable.</li>
      <li>Drag <code class="language-plaintext highlighter-rouge">Region</code> to <strong>Rows</strong>.</li>
      <li>Drag <code class="language-plaintext highlighter-rouge">Sales</code> to <strong>Values</strong>.</li>
      <li>Format the PivotTable results as Currency.</li>
    </ul>
  </li>
  <li><strong>Printing/Output:</strong>
    <ul>
      <li>Set Print Area.</li>
      <li>Insert Header with your Name/Roll Number.</li>
      <li>Scale to Fit (1 Page Width).</li>
    </ul>
  </li>
</ol>

<p><img src="https://blogs.buildwithmanish.com/assets/images/concept_illustration-textbook-for-excel-exam-6-2025-12-29-26439.png" alt="Concept Illustration" /></p>

<h1 id="conclusion">Conclusion</h1>

<p>Passing the Basics of MS Excel exam requires moving beyond “guessing” which button to click. It requires understanding the object model-how cells relate to formulas, how ranges relate to charts, and how raw data flows into Pivot Tables.</p>

<p><strong>Final Checklist for the Exam:</strong></p>
<ol>
  <li><strong>Sanitize inputs:</strong> Check for extra spaces or stored-as-text numbers before calculating.</li>
  <li><strong>Lock references:</strong> Always ask, “Should this cell reference move when I drag the formula?”</li>
  <li><strong>Label everything:</strong> Charts without titles and axes labels are mathematically meaningless.</li>
  <li><strong>Save frequently:</strong> <code class="language-plaintext highlighter-rouge">Ctrl + S</code> is the most important shortcut of all.</li>
</ol>

<p>Master these four units, and you will not only ace the exam but possess a toolset valuable for any data-driven career.</p>]]></content><author><name>Manish Tiwari</name></author><category term="deep-dive" /><category term="excel-mastery" /><category term="data-analysis" /><category term="spreadsheet-engineering" /><category term="certification" /><summary type="html"><![CDATA[]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" /><media:content medium="image" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">How I built an AI blogger Agent using N8N and Jekyll</title><link href="https://blogs.buildwithmanish.com/deep-dive/n8n/generative-ai/jekyll/automation/2025/12/18/how-i-built-an-ai-blogger-agent-using-n8n-and-jekyll.html" rel="alternate" type="text/html" title="How I built an AI blogger Agent using N8N and Jekyll" /><published>2025-12-18T00:00:00+00:00</published><updated>2025-12-18T00:00:00+00:00</updated><id>https://blogs.buildwithmanish.com/deep-dive/n8n/generative-ai/jekyll/automation/2025/12/18/how-i-built-an-ai-blogger-agent-using-n8n-and-jekyll</id><content type="html" xml:base="https://blogs.buildwithmanish.com/deep-dive/n8n/generative-ai/jekyll/automation/2025/12/18/how-i-built-an-ai-blogger-agent-using-n8n-and-jekyll.html"><![CDATA[<p><img src="https://blogs.buildwithmanish.com/assets/images/hero-how-i-built-an-ai-blogger-agent-using-n8n-and-jekyll-0-2025-12-18-94311.png" alt="Hero Image - AI Blogger Agent Workflow" /></p>

<p>As engineers, we often suffer from the “cobbler’s children” syndrome. We build complex, scalable systems for clients and employers, yet our own personal portfolios and blogs gather dust. The friction of context-switching from coding to writing, finding assets, and managing deployments often kills the momentum before a post is even drafted.</p>

<p>I decided to solve this not by “writing more,” but by engineering a solution. I built an autonomous AI Agent capable of end-to-end technical blogging. This isn’t a simple “text-generation” script; it is a full-stack automated pipeline that orchestrates visual planning, asset generation, content synthesis, version control, and continuous deployment.</p>

<p>This post creates a deep-dive analysis of the architecture, the specific n8n workflows used to manage the backend logic, and the Jekyll frontend that serves the content.</p>

<h2 id="the-architecture-a-birds-eye-view">The Architecture: A Bird’s Eye View</h2>

<p>The system is designed as a decoupled pipeline. The “Brain” (backend) is an n8n instance that orchestrates logic, while the “Face” (frontend) is a static Jekyll site hosted on Railway. The bridge between them is GitHub.</p>

<div id="section_architecture">
  <img src="https://blogs.buildwithmanish.com/assets/images/architecture_diagram-how-i-built-an-ai-blogger-agent-using-n8n-and-jekyll-1-2025-12-18-4974.png" alt="System Architecture Diagram" class="img-fluid" />
</div>

<p>The flow operates on a “Push” model:</p>
<ol>
  <li><strong>Trigger:</strong> A Google Sheet acts as the editorial calendar.</li>
  <li><strong>Orchestration (n8n):</strong> Handles the logic, API calls, and state management.</li>
  <li><strong>Intelligence (Gemini):</strong> Provides the reasoning, visual planning, and content writing.</li>
  <li><strong>Storage (GitHub):</strong> Acts as the CMS (Content Management System) for both Markdown and binary assets.</li>
  <li><strong>Build &amp; Deploy (Railway):</strong> Detects changes in the repository and builds the static site.</li>
</ol>

<p>This architecture was chosen for <strong>resilience</strong> and <strong>cost-efficiency</strong>. By using Git as the source of truth, we eliminate the need for a traditional database, and by using n8n, we can visually debug the complex chain of asynchronous operations required to generate a rich media post.</p>

<hr />

<h2 id="part-1-the-backend-workflow-n8n">Part 1: The Backend Workflow (n8n)</h2>

<p>The core of this system is a complex n8n workflow. It doesn’t just “ask for an article.” It mimics the cognitive process of a human technical writer: Plan visuals first, create assets, then write the text around those assets.</p>

<div id="section_backend_workflow">
  <img src="https://blogs.buildwithmanish.com/assets/images/technical_diagram-how-i-built-an-ai-blogger-agent-using-n8n-and-jekyll-2-2025-12-18-4186.png" alt="Backend Workflow Logic" class="img-fluid" />
</div>

<h3 id="1-the-trigger-and-state-management">1. The Trigger and State Management</h3>
<p>The workflow begins by polling a Google Sheet. While webhooks are faster, polling offers better rate-limit control when dealing with LLMs.</p>

<p><strong>The Filter Logic:</strong>
We fetch all rows but process only one where <code class="language-plaintext highlighter-rouge">Status == "Pending"</code>. This acts as a semaphore, preventing the agent from trying to write the entire backlog simultaneously.</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// n8n Function Item Node: Filter for next assignment</span>
<span class="kd">const</span> <span class="nx">rows</span> <span class="o">=</span> <span class="nx">items</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">json</span><span class="p">.</span><span class="nx">rows</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">pendingTask</span> <span class="o">=</span> <span class="nx">rows</span><span class="p">.</span><span class="nx">find</span><span class="p">(</span><span class="nx">row</span> <span class="o">=&gt;</span> <span class="nx">row</span><span class="p">.</span><span class="nx">status</span> <span class="o">===</span> <span class="dl">'</span><span class="s1">Pending</span><span class="dl">'</span><span class="p">);</span>

<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">pendingTask</span><span class="p">)</span> <span class="p">{</span>
  <span class="k">return</span> <span class="p">[];</span> <span class="c1">// Stop workflow if nothing to do</span>
<span class="p">}</span>

<span class="k">return</span> <span class="p">[{</span> <span class="na">json</span><span class="p">:</span> <span class="nx">pendingTask</span> <span class="p">}];</span>
</code></pre></div></div>

<h3 id="2-the-visual-strategist-gemini">2. The Visual Strategist (Gemini)</h3>
<p>Before writing a single word, the workflow invokes Google’s Gemini LLM. The prompt here is crucial. We don’t ask for the blog post yet. We ask for a <strong>Visual Asset Plan</strong>.</p>

<p><strong>Prompt Engineering:</strong></p>
<blockquote>
  <p>“You are a technical editor. Based on the topic ‘${topic}’, create a plan for 4 visual assets. For each asset, define the ‘Type’ (chart, diagram, hero) and a detailed ‘Image Generation Prompt’.”</p>
</blockquote>

<p>Gemini returns a structured JSON array. This is critical because we need to generate these images <em>before</em> the final writing phase so that we can embed their permanent URLs into the Markdown.</p>

<h3 id="3-image-generation-and-github-commits">3. Image Generation and GitHub Commits</h3>
<p>This is the most technically demanding part of the workflow. We iterate over the JSON plan provided by Gemini. For each item, we call an Image Generation API (e.g., OpenAI DALL-E 3 or Stable Diffusion via API).</p>

<p>The challenge lies in handling the binary data.</p>

<div id="section_image_processing">
  <img src="https://blogs.buildwithmanish.com/assets/images/code_flow-how-i-built-an-ai-blogger-agent-using-n8n-and-jekyll-3-2025-12-18-57086.png" alt="Image Processing Code Flow" class="img-fluid" />
</div>

<p>The API returns a URL or Base64 string. To host this on our own domain (to avoid hotlinking or expiring URLs), we must upload it to GitHub.</p>

<p><strong>The Base64 Challenge:</strong>
The GitHub API requires file content to be Base64 encoded. In n8n, handling binary buffers requires specific node configurations.</p>

<ol>
  <li><strong>HTTP Request:</strong> Download image as Binary.</li>
  <li><strong>Code Node:</strong> Convert Binary Buffer to Base64 string.</li>
</ol>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// n8n Code Node: Buffer to Base64 for GitHub API</span>
<span class="kd">const</span> <span class="nx">binaryData</span> <span class="o">=</span> <span class="nx">items</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">binary</span><span class="p">.</span><span class="nx">data</span><span class="p">;</span> <span class="c1">// The image from previous node</span>
<span class="kd">const</span> <span class="nx">buffer</span> <span class="o">=</span> <span class="nx">Buffer</span><span class="p">.</span><span class="k">from</span><span class="p">(</span><span class="nx">binaryData</span><span class="p">.</span><span class="nx">data</span><span class="p">,</span> <span class="dl">'</span><span class="s1">base64</span><span class="dl">'</span><span class="p">);</span> <span class="c1">// n8n stores internal binary as base64 already, but let's ensure context</span>

<span class="c1">// GitHub requires standard Base64</span>
<span class="kd">const</span> <span class="nx">content</span> <span class="o">=</span> <span class="nx">buffer</span><span class="p">.</span><span class="nx">toString</span><span class="p">(</span><span class="dl">'</span><span class="s1">base64</span><span class="dl">'</span><span class="p">);</span>

<span class="k">return</span> <span class="p">{</span>
  <span class="na">json</span><span class="p">:</span> <span class="p">{</span>
    <span class="na">filename</span><span class="p">:</span> <span class="s2">`assets/images/</span><span class="p">${</span><span class="nx">generated_slug</span><span class="p">}</span><span class="s2">-</span><span class="p">${</span><span class="nx">timestamp</span><span class="p">}</span><span class="s2">.png`</span><span class="p">,</span>
    <span class="na">content</span><span class="p">:</span> <span class="nx">content</span><span class="p">,</span>
    <span class="na">message</span><span class="p">:</span> <span class="s2">`chore: add auto-generated asset for </span><span class="p">${</span><span class="nx">topic</span><span class="p">}</span><span class="s2">`</span>
  <span class="p">}</span>
<span class="p">};</span>
</code></pre></div></div>

<p><strong>The GitHub Commit:</strong>
We then fire a <code class="language-plaintext highlighter-rouge">PUT</code> request to <code class="language-plaintext highlighter-rouge">https://api.github.com/repos/{owner}/{repo}/contents/{path}</code>.</p>

<p>Success here returns the <code class="language-plaintext highlighter-rouge">download_url</code>. We collect these URLs into an array to pass to the next stage.</p>

<h3 id="4-contextual-content-generation">4. Contextual Content Generation</h3>
<p>Now, we have a list of live image URLs. We feed these back into Gemini for the final writing phase.</p>

<div id="section_content_generation">
  <img src="https://blogs.buildwithmanish.com/assets/images/concept_illustration-how-i-built-an-ai-blogger-agent-using-n8n-and-jekyll-4-2025-12-18-15690.png" alt="Concept Illustration: Contextual Generation" class="img-fluid" />
</div>

<p><strong>The “Deep-Dive” Prompt:</strong></p>
<blockquote>
  <p>“Write a technical blog post about ‘${topic}’. Here are the URLs for the images you planned earlier: ${image_url_list}. You MUST embed these images in the Markdown where they contextually fit. Do not simply append them at the end. Use the syntax <code class="language-plaintext highlighter-rouge">![alt text](url)</code>.”</p>
</blockquote>

<p>This ensures that the diagrams appear exactly where the text explains them, mimicking a human author’s layout.</p>

<h3 id="5-the-final-commit-markdown">5. The Final Commit (Markdown)</h3>
<p>Finally, the generated Markdown is cleaned (removing Markdown code block fences usually returned by LLMs) and committed to the <code class="language-plaintext highlighter-rouge">_posts/</code> directory.</p>

<div id="section_github_structure">
  <img src="https://blogs.buildwithmanish.com/assets/images/technical_diagram-how-i-built-an-ai-blogger-agent-using-n8n-and-jekyll-5-2025-12-18-8248.png" alt="GitHub Directory Structure" class="img-fluid" />
</div>

<p>The filename format is strict: <code class="language-plaintext highlighter-rouge">YYYY-MM-DD-title-slug.md</code>. This is required for Jekyll to parse the date correctly.</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Filename generation logic</span>
<span class="kd">const</span> <span class="nx">date</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Date</span><span class="p">().</span><span class="nx">toISOString</span><span class="p">().</span><span class="nx">split</span><span class="p">(</span><span class="dl">'</span><span class="s1">T</span><span class="dl">'</span><span class="p">)[</span><span class="mi">0</span><span class="p">];</span>
<span class="kd">const</span> <span class="nx">slug</span> <span class="o">=</span> <span class="nx">title</span><span class="p">.</span><span class="nx">toLowerCase</span><span class="p">().</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/</span><span class="se">[^</span><span class="sr">a-z0-9</span><span class="se">]</span><span class="sr">+/g</span><span class="p">,</span> <span class="dl">'</span><span class="s1">-</span><span class="dl">'</span><span class="p">).</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/</span><span class="se">(</span><span class="sr">^-|-$</span><span class="se">)</span><span class="sr">+/g</span><span class="p">,</span> <span class="dl">''</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">filename</span> <span class="o">=</span> <span class="s2">`_posts/</span><span class="p">${</span><span class="nx">date</span><span class="p">}</span><span class="s2">-</span><span class="p">${</span><span class="nx">slug</span><span class="p">}</span><span class="s2">.md`</span><span class="p">;</span>
</code></pre></div></div>

<hr />

<h2 id="part-2-the-frontend-jekyll-on-railway">Part 2: The Frontend (Jekyll on Railway)</h2>

<p>While the backend is the brain, the frontend is the experience. I chose Jekyll because it transforms the filesystem into a website. There is no database to hack, no CMS to update, and the “API” is just the Git repository.</p>

<h3 id="deployment-pipeline">Deployment Pipeline</h3>
<p>I use <strong>Railway</strong> for hosting. Railway has a robust integration with GitHub.</p>

<div id="section_frontend_deployment">
  <img src="https://blogs.buildwithmanish.com/assets/images/comparison_chart-how-i-built-an-ai-blogger-agent-using-n8n-and-jekyll-6-2025-12-18-77025.png" alt="Deployment Comparison Chart" class="img-fluid" />
</div>

<ol>
  <li>n8n commits a new <code class="language-plaintext highlighter-rouge">.md</code> file to <code class="language-plaintext highlighter-rouge">_posts</code>.</li>
  <li>GitHub triggers a webhook to Railway.</li>
  <li>Railway pulls the repo.</li>
  <li>Railway runs <code class="language-plaintext highlighter-rouge">bundle exec jekyll build</code>.</li>
  <li>The <code class="language-plaintext highlighter-rouge">_site</code> directory is served via Nginx.</li>
</ol>

<p>This entire process takes about 45 seconds from the moment n8n finishes to the post being live.</p>

<h3 id="custom-frontend-features-vanilla-js">Custom Frontend Features (Vanilla JS)</h3>

<p>A standard Jekyll theme wasn’t enough. I wanted a “Medium-like” experience but without the paywall and tracking. I implemented several features using raw JavaScript to keep the site lightweight.</p>

<h4 id="1-image-lightbox">1. Image Lightbox</h4>
<p>Since the AI generates detailed diagrams, users need to zoom in. I wrote a script that wraps every post image in a modal trigger.</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">document</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="dl">'</span><span class="s1">DOMContentLoaded</span><span class="dl">'</span><span class="p">,</span> <span class="p">()</span> <span class="o">=&gt;</span> <span class="p">{</span>
  <span class="kd">const</span> <span class="nx">images</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">querySelectorAll</span><span class="p">(</span><span class="dl">'</span><span class="s1">.post-content img</span><span class="dl">'</span><span class="p">);</span>
  <span class="kd">const</span> <span class="nx">lightbox</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="dl">'</span><span class="s1">lightbox</span><span class="dl">'</span><span class="p">);</span>
  <span class="kd">const</span> <span class="nx">lightboxImg</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="dl">'</span><span class="s1">lightbox-img</span><span class="dl">'</span><span class="p">);</span>

  <span class="nx">images</span><span class="p">.</span><span class="nx">forEach</span><span class="p">(</span><span class="nx">img</span> <span class="o">=&gt;</span> <span class="p">{</span>
    <span class="nx">img</span><span class="p">.</span><span class="nx">style</span><span class="p">.</span><span class="nx">cursor</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">zoom-in</span><span class="dl">'</span><span class="p">;</span>
    <span class="nx">img</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="dl">'</span><span class="s1">click</span><span class="dl">'</span><span class="p">,</span> <span class="p">()</span> <span class="o">=&gt;</span> <span class="p">{</span>
      <span class="nx">lightboxImg</span><span class="p">.</span><span class="nx">src</span> <span class="o">=</span> <span class="nx">img</span><span class="p">.</span><span class="nx">src</span><span class="p">;</span>
      <span class="nx">lightbox</span><span class="p">.</span><span class="nx">classList</span><span class="p">.</span><span class="nx">add</span><span class="p">(</span><span class="dl">'</span><span class="s1">active</span><span class="dl">'</span><span class="p">);</span>
    <span class="p">});</span>
  <span class="p">});</span>

  <span class="nx">lightbox</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="dl">'</span><span class="s1">click</span><span class="dl">'</span><span class="p">,</span> <span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
    <span class="k">if</span> <span class="p">(</span><span class="nx">e</span><span class="p">.</span><span class="nx">target</span> <span class="o">!==</span> <span class="nx">lightboxImg</span><span class="p">)</span> <span class="p">{</span>
      <span class="nx">lightbox</span><span class="p">.</span><span class="nx">classList</span><span class="p">.</span><span class="nx">remove</span><span class="p">(</span><span class="dl">'</span><span class="s1">active</span><span class="dl">'</span><span class="p">);</span>
    <span class="p">}</span>
  <span class="p">});</span>
<span class="p">});</span>
</code></pre></div></div>

<h4 id="2-multi-tag-filtering">2. Multi-Tag Filtering</h4>
<p>Jekyll handles categories natively, but real-time filtering usually requires a page reload. I implemented a client-side filter. I serialize all post metadata into a JSON object in the DOM, allowing instant filtering.</p>

<p><strong>_layouts/home.html:</strong></p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;script&gt;</span>
  <span class="nb">window</span><span class="p">.</span><span class="nx">posts</span> <span class="o">=</span> <span class="p">[</span>
    
      <span class="p">{</span>
        <span class="dl">"</span><span class="s2">title</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Top 5 Programming Languages to learn in 2026</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">url</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">/deep-dive/career-growth/software-engineering/system-design/tech-trends/2025/12/31/top-5-programming-languages-to-learn-in-2026.html</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">tags</span><span class="dl">"</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">deep-dive</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">career-growth</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">software-engineering</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">system-design</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">tech-trends</span><span class="dl">"</span><span class="p">]</span>
      <span class="p">},</span>
    
      <span class="p">{</span>
        <span class="dl">"</span><span class="s2">title</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Relational Database Management System: An Engineering Deep-Dive</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">url</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">/deep-dive/database-systems/sql/data-modeling/backend-engineering/2025/12/30/relational-database-management-system.html</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">tags</span><span class="dl">"</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">deep-dive</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">database-systems</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">sql</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">data-modeling</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">backend-engineering</span><span class="dl">"</span><span class="p">]</span>
      <span class="p">},</span>
    
      <span class="p">{</span>
        <span class="dl">"</span><span class="s2">title</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Textbook for Excel Exam</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">url</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">/deep-dive/excel-mastery/data-analysis/spreadsheet-engineering/certification/2025/12/29/textbook-for-excel-exam.html</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">tags</span><span class="dl">"</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">deep-dive</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">excel-mastery</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">data-analysis</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">spreadsheet-engineering</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">certification</span><span class="dl">"</span><span class="p">]</span>
      <span class="p">},</span>
    
      <span class="p">{</span>
        <span class="dl">"</span><span class="s2">title</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Reactive Architecture in Flutter: Mastering BLoC for Scalable State Management</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">url</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">/deep-dive/flutter/state-management/bloc/reactive-architecture/2025/12/18/reactive-architecture-in-flutter-mastering-bloc-for-scalable-state-management.html</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">tags</span><span class="dl">"</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">deep-dive</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">flutter</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">state-management</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">bloc</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">reactive-architecture</span><span class="dl">"</span><span class="p">]</span>
      <span class="p">},</span>
    
      <span class="p">{</span>
        <span class="dl">"</span><span class="s2">title</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Implementing Agentic Workflows for Java-to-Node.js Migration</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">url</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">/deep-dive/generative-ai/legacy-migration/java-spring/nodejs-nestjs/2025/12/18/implementing-agentic-workflows-for-javatonodejs-migration.html</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">tags</span><span class="dl">"</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">deep-dive</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">generative-ai</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">legacy-migration</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">java-spring</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">nodejs-nestjs</span><span class="dl">"</span><span class="p">]</span>
      <span class="p">},</span>
    
      <span class="p">{</span>
        <span class="dl">"</span><span class="s2">title</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">How I built an AI blogger Agent using N8N,Github and Jekyll</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">url</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">/deep-dive/n8n/ai-automation/jekyll/system-design/2025/12/18/how-i-built-an-ai-blogger-agent-using-n8ngithub-and-jekyll.html</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">tags</span><span class="dl">"</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">deep-dive</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">n8n</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">ai-automation</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">jekyll</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">system-design</span><span class="dl">"</span><span class="p">]</span>
      <span class="p">},</span>
    
      <span class="p">{</span>
        <span class="dl">"</span><span class="s2">title</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">How I built an AI blogger Agent using N8N and Jekyll</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">url</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">/deep-dive/n8n/generative-ai/jekyll/automation/2025/12/18/how-i-built-an-ai-blogger-agent-using-n8n-and-jekyll.html</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">tags</span><span class="dl">"</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">deep-dive</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">n8n</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">generative-ai</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">jekyll</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">automation</span><span class="dl">"</span><span class="p">]</span>
      <span class="p">},</span>
    
      <span class="p">{</span>
        <span class="dl">"</span><span class="s2">title</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Mastering Razorpay Webhooks: Architecting an Idempotent, Event-Driven Ingestion Engine</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">url</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">/deep-dive/system-design/payments/event-driven-architecture/razorpay/2025/12/17/mastering-razorpay-webhooks-architecting-an-idempotent-eventdriven-ingestion-engine.html</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">tags</span><span class="dl">"</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">deep-dive</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">system-design</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">payments</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">event-driven-architecture</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">razorpay</span><span class="dl">"</span><span class="p">]</span>
      <span class="p">},</span>
    
      <span class="p">{</span>
        <span class="dl">"</span><span class="s2">title</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Java Concurrency at Scale: Migrating from Thread Pools to Virtual Threads</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">url</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">/deep-dive/java/concurrency/virtual-threads/software-architecture/2025/12/17/java-concurrency-at-scale-migrating-from-thread-pools-to-virtual-threads.html</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">tags</span><span class="dl">"</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">deep-dive</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">java</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">concurrency</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">virtual-threads</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">software-architecture</span><span class="dl">"</span><span class="p">]</span>
      <span class="p">},</span>
    
      <span class="p">{</span>
        <span class="dl">"</span><span class="s2">title</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Escaping the if (isAdmin) Trap: Implementing Scalable RBAC and ABAC in Node.js</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">url</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">/deep-dive/nodejs/security/rbac/abac/system-design/2025/12/17/escaping-the-if-isadmin-trap-implementing-scalable-rbac-and-abac-in-nodejs.html</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">tags</span><span class="dl">"</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">deep-dive</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">nodejs</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">security</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">rbac</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">abac</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">system-design</span><span class="dl">"</span><span class="p">]</span>
      <span class="p">},</span>
    
      <span class="p">{</span>
        <span class="dl">"</span><span class="s2">title</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Building a Virtual Data Analyst: Architecting Agentic AI Workflows over Power BI</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">url</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">/deep-dive/gen-ai/power-bi/agentic-workflows/python/2025/12/17/building-a-virtual-data-analyst-architecting-agentic-ai-workflows-over-power-bi.html</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">tags</span><span class="dl">"</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">deep-dive</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">gen-ai</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">power-bi</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">agentic-workflows</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">python</span><span class="dl">"</span><span class="p">]</span>
      <span class="p">},</span>
    
      <span class="p">{</span>
        <span class="dl">"</span><span class="s2">title</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Beyond the Happy Path: Architecting Fault-Tolerant Recurring Payments with Razorpay</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">url</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">/technical/deep-dive/2025/12/17/beyond-the-happy-path-architecting-faulttolerant-recurring-payments-with-razorpay.html</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">tags</span><span class="dl">"</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">technical</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">deep-dive</span><span class="dl">"</span><span class="p">]</span>
      <span class="p">},</span>
    
      <span class="p">{</span>
        <span class="dl">"</span><span class="s2">title</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Beyond GET and SET: Architecting Resilient Distributed Caching Patterns for Microservices</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">url</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">/deep-dive/system-design/redis/distributed-systems/microservices/2025/12/17/beyond-get-and-set-architecting-resilient-distributed-caching-patterns-for-microservices.html</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">tags</span><span class="dl">"</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">deep-dive</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">system-design</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">redis</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">distributed-systems</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">microservices</span><span class="dl">"</span><span class="p">]</span>
      <span class="p">},</span>
    
  <span class="p">];</span>
<span class="nt">&lt;/script&gt;</span>
</code></pre></div></div>

<p>The Javascript then simply toggles <code class="language-plaintext highlighter-rouge">display: none</code> on the post cards based on the selected tag array.</p>

<h4 id="3-dark-mode-with-persistence">3. Dark Mode with Persistence</h4>
<p>Dark mode is implemented via CSS variables and <code class="language-plaintext highlighter-rouge">localStorage</code>. This prevents the “flash of white content” (FOUC) by checking storage in the <code class="language-plaintext highlighter-rouge">&lt;head&gt;</code> before the body renders.</p>

<div class="language-css highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">:root</span> <span class="p">{</span>
  <span class="py">--bg-color</span><span class="p">:</span> <span class="m">#ffffff</span><span class="p">;</span>
  <span class="py">--text-color</span><span class="p">:</span> <span class="m">#1a1a1a</span><span class="p">;</span>
<span class="p">}</span>

<span class="o">[</span><span class="nt">data-theme</span><span class="o">=</span><span class="s1">"dark"</span><span class="o">]</span> <span class="p">{</span>
  <span class="py">--bg-color</span><span class="p">:</span> <span class="m">#0f172a</span><span class="p">;</span>
  <span class="py">--text-color</span><span class="p">:</span> <span class="m">#e2e8f0</span><span class="p">;</span>
<span class="p">}</span>

<span class="nt">body</span> <span class="p">{</span>
  <span class="nl">background-color</span><span class="p">:</span> <span class="n">var</span><span class="p">(</span><span class="n">--bg-color</span><span class="p">);</span>
  <span class="nl">color</span><span class="p">:</span> <span class="n">var</span><span class="p">(</span><span class="n">--text-color</span><span class="p">);</span>
  <span class="nl">transition</span><span class="p">:</span> <span class="n">background-color</span> <span class="m">0.3s</span> <span class="n">ease</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<hr />

<h2 id="advanced-scenarios--trade-offs">Advanced Scenarios &amp; Trade-offs</h2>

<p>Building an autonomous agent isn’t without its edge cases. Here are the production issues I encountered and how I solved them.</p>

<h3 id="1-hallucinations-and-json-validity">1. Hallucinations and JSON Validity</h3>
<p>When asking an LLM for a JSON plan (visual assets), it sometimes wraps the JSON in markdown code blocks (```json … ```) or adds conversational text.</p>

<p><strong>Solution:</strong> In n8n, I use a “Function Item” node specifically to sanitize the LLM output before parsing it.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Regex to extract pure JSON</span>
<span class="kd">const</span> <span class="nx">jsonMatch</span> <span class="o">=</span> <span class="nx">items</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">json</span><span class="p">.</span><span class="nx">content</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="sr">/</span><span class="se">\{[\s\S]</span><span class="sr">*</span><span class="se">\}</span><span class="sr">/</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">jsonMatch</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">return</span> <span class="p">{</span> <span class="na">json</span><span class="p">:</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span><span class="nx">jsonMatch</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="p">};</span>
<span class="p">}</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span><span class="p">(</span><span class="dl">"</span><span class="s2">LLM failed to return valid JSON</span><span class="dl">"</span><span class="p">);</span>
</code></pre></div></div>

<h3 id="2-github-api-race-conditions">2. GitHub API Race Conditions</h3>
<p>If the agent tries to upload 4 images simultaneously (parallel execution in n8n), GitHub might reject requests if the git tree updates haven’t propagated instantly, or we might hit rate limits.</p>

<p><strong>Solution:</strong> I configured the n8n “Split In Batches” node to size <code class="language-plaintext highlighter-rouge">1</code>. This forces sequential uploads. It’s slower, but significantly more reliable. It also simplifies the logic of aggregating the resulting URLs.</p>

<h3 id="3-context-window-limits">3. Context Window Limits</h3>
<p>Passing high-res base64 strings back to an LLM context is expensive and often exceeds token limits.</p>

<p><strong>Solution:</strong> This is why the architecture uploads to GitHub <em>first</em>. We pass the <em>URL</em> (string), not the <em>image data</em>, to the writing agent. This keeps the token count low while allowing the LLM to know exactly what asset it is referencing.</p>

<hr />

<h2 id="conclusion">Conclusion</h2>

<p>This project demonstrates that we have crossed a threshold in Generative AI. We are no longer limited to “chatbots” that answer questions. We can build <strong>Agents</strong>-systems that plan, execute, use tools (APIs), and persist their work.</p>

<p>By combining the low-code orchestration of n8n with the raw power of Gemini and the stability of Jekyll, I’ve created a blogging engine that doesn’t just save time-it enforces a level of visual and structural consistency that is hard to maintain manually.</p>

<p>The next step for this agent is <strong>Self-Correction</strong>. I plan to implement a feedback loop where the agent reads the build logs from Railway. If a build fails (e.g., due to bad Liquid syntax), the agent will trigger a “Fix It” workflow to patch the markdown file automatically.</p>

<p>The future of content isn’t just AI-generated; it’s AI-architected.</p>

<hr />
<p><em>Note: All diagrams in this post were planned and generated by the agent itself as part of the workflow described above.</em></p>]]></content><author><name>Manish Tiwari</name></author><category term="deep-dive" /><category term="n8n" /><category term="generative-ai" /><category term="jekyll" /><category term="automation" /><summary type="html"><![CDATA[]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" /><media:content medium="image" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">How I built an AI blogger Agent using N8N,Github and Jekyll</title><link href="https://blogs.buildwithmanish.com/deep-dive/n8n/ai-automation/jekyll/system-design/2025/12/18/how-i-built-an-ai-blogger-agent-using-n8ngithub-and-jekyll.html" rel="alternate" type="text/html" title="How I built an AI blogger Agent using N8N,Github and Jekyll" /><published>2025-12-18T00:00:00+00:00</published><updated>2025-12-18T00:00:00+00:00</updated><id>https://blogs.buildwithmanish.com/deep-dive/n8n/ai-automation/jekyll/system-design/2025/12/18/how-i-built-an-ai-blogger-agent-using-n8ngithub-and-jekyll</id><content type="html" xml:base="https://blogs.buildwithmanish.com/deep-dive/n8n/ai-automation/jekyll/system-design/2025/12/18/how-i-built-an-ai-blogger-agent-using-n8ngithub-and-jekyll.html"><![CDATA[<p><img src="https://blogs.buildwithmanish.com/assets/images/hero-how-i-built-an-ai-blogger-agent-using-n8ngithub-and-jekyll-0-2025-12-18-22032.png" alt="Hero Image" /></p>

<p>In the current landscape of software engineering, “content creation” is often viewed as a secondary concern-a distraction from shipping code. However, technical documentation and knowledge sharing are critical components of system design. The problem isn’t the lack of knowledge; it’s the friction involved in the pipeline: ideation, drafting, asset creation, formatting, and deployment.</p>

<p>I decided to solve this not by writing more, but by engineering a solution. I built an autonomous AI Agent that functions as a full-stack technical blogger. This isn’t a simple wrapper around ChatGPT that spits out generic text. This is a deterministic, multi-step pipeline that orchestrates planning, visual asset generation, code-aware writing, version control integration, and continuous deployment.</p>

<p>This post creates a deep dive into the architecture of this system, specifically focusing on the orchestration layer (n8n), the storage/versioning layer (GitHub), and the presentation layer (Jekyll on Railway).</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/architecture_diagram-how-i-built-an-ai-blogger-agent-using-n8ngithub-and-jekyll-1-2025-12-18-94439.png" alt="Architecture Diagram" /></p>

<h2 id="the-core-philosophy-agents-vs-generators">The Core Philosophy: Agents vs. Generators</h2>

<p>Before looking at the code, we must distinguish between <strong>Generative AI</strong> and <strong>Agentic AI</strong>.</p>
<ul>
  <li><strong>Generative AI</strong> is input-output. You give a prompt; you get text.</li>
  <li><strong>Agentic AI</strong> involves reasoning, planning, and tool usage.</li>
</ul>

<p>My goal was to build the latter. An agent that looks at a topic, <em>thinks</em> about what diagrams are needed to explain that topic, generates those diagrams using external tools, hosts them, and <em>then</em> writes the article referencing those specific assets.</p>

<h2 id="part-1-the-backend-orchestration-n8n">Part 1: The Backend Orchestration (n8n)</h2>

<p>The brain of this operation is n8n. While I could have written this in Python using LangChain or Swarm, n8n provides a visual interface for flow control which is invaluable when debugging complex asynchronous chains involving file buffers and API calls.</p>

<h3 id="1-the-trigger-and-state-management">1. The Trigger and State Management</h3>

<p>The workflow is state-driven. We don’t want to regenerate old blogs, nor do we want to hallucinate topics. I use Google Sheets as a lightweight database.</p>

<p><strong>The Data Structure:</strong></p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">Topic</code>: The technical subject (e.g., “Implementing Rate Limiting with Redis”).</li>
  <li>
    <table>
      <tbody>
        <tr>
          <td><code class="language-plaintext highlighter-rouge">Status</code>: <code class="language-plaintext highlighter-rouge">Pending</code></td>
          <td><code class="language-plaintext highlighter-rouge">Processing</code></td>
          <td><code class="language-plaintext highlighter-rouge">Done</code>.</td>
        </tr>
      </tbody>
    </table>
  </li>
  <li><code class="language-plaintext highlighter-rouge">Date</code>: Timestamp.</li>
</ul>

<p>The n8n workflow begins with a <strong>Google Sheets Trigger</strong> that fetches rows where <code class="language-plaintext highlighter-rouge">Status == Pending</code>.</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">//</span><span class="w"> </span><span class="err">Sample</span><span class="w"> </span><span class="err">JSON</span><span class="w"> </span><span class="err">Output</span><span class="w"> </span><span class="err">from</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">Trigger</span><span class="w"> </span><span class="err">Node</span><span class="w">
</span><span class="p">[</span><span class="w">
  </span><span class="p">{</span><span class="w">
    </span><span class="nl">"Topic"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Deep Dive into Kubernetes Networking"</span><span class="p">,</span><span class="w">
    </span><span class="nl">"Status"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Pending"</span><span class="p">,</span><span class="w">
    </span><span class="nl">"RowNumber"</span><span class="p">:</span><span class="w"> </span><span class="mi">4</span><span class="w">
  </span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span></code></pre></div></div>

<h3 id="2-the-planner-agent-gemini-pro">2. The Planner Agent (Gemini Pro)</h3>

<p>This is the most critical architectural decision. I do not ask the LLM to write the blog yet. If you ask an LLM to “write a blog with images,” it will hallucinate URLs (e.g., <code class="language-plaintext highlighter-rouge">![img](imgur.com/fake.png)</code>).</p>

<p>Instead, I use a “Planner” node. I feed the topic to Gemini Pro with a system prompt designed to act as a Technical Art Director.</p>

<p><strong>System Prompt Strategy:</strong></p>
<blockquote>
  <p>“You are a Technical Art Director. Analyze the following topic. Outline a plan for 3-5 visual assets required to explain this concept. For each asset, provide a detailed image generation prompt and a specific filename.”</p>
</blockquote>

<p>The output is forced into a structured JSON format (using n8n’s JSON parser):</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
  </span><span class="nl">"assets"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
    </span><span class="p">{</span><span class="w">
      </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"architecture_diagram"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"filename"</span><span class="p">:</span><span class="w"> </span><span class="s2">"k8s-networking-flow.png"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"prompt"</span><span class="p">:</span><span class="w"> </span><span class="s2">"High contrast technical diagram, kubernetes pod communication, flat vector style..."</span><span class="w">
    </span><span class="p">},</span><span class="w">
    </span><span class="p">{</span><span class="w">
      </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hero_image"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"filename"</span><span class="p">:</span><span class="w"> </span><span class="s2">"k8s-hero.png"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"prompt"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Cyberpunk style container ship, digital ocean, isometric 3d render..."</span><span class="w">
    </span><span class="p">}</span><span class="w">
  </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<p><img src="https://blogs.buildwithmanish.com/assets/images/technical_diagram-how-i-built-an-ai-blogger-agent-using-n8ngithub-and-jekyll-2-2025-12-18-21121.png" alt="Technical Diagram" /></p>

<h3 id="3-asset-generation-and-binary-handling">3. Asset Generation and Binary Handling</h3>

<p>Once the plan is generated, n8n iterates over the <code class="language-plaintext highlighter-rouge">assets</code> array. This is where we hit a common hurdle in low-code automation: <strong>Binary Data Handling</strong>.</p>

<p>The workflow makes a call to an Image Generation API (e.g., OpenAI DALL-E 3 or Stability AI). The API returns a URL or a Base64 string. We cannot simply pass a URL to the writer; we need to own the asset to prevent link rot.</p>

<p><strong>The Process:</strong></p>
<ol>
  <li><strong>HTTP Request:</strong> Send prompt to Image Gen API.</li>
  <li><strong>Buffer Handling:</strong> Download the image to n8n’s memory.</li>
  <li><strong>Base64 Conversion:</strong> GitHub’s API requires file content to be Base64 encoded.</li>
</ol>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// n8n Function Node: Prepare for GitHub API</span>
<span class="kd">const</span> <span class="nx">binaryData</span> <span class="o">=</span> <span class="nx">items</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">binary</span><span class="p">.</span><span class="nx">data</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">base64String</span> <span class="o">=</span> <span class="nx">binaryData</span><span class="p">.</span><span class="nx">toString</span><span class="p">(</span><span class="dl">'</span><span class="s1">base64</span><span class="dl">'</span><span class="p">);</span>

<span class="k">return</span> <span class="p">{</span>
  <span class="na">json</span><span class="p">:</span> <span class="p">{</span>
    <span class="na">path</span><span class="p">:</span> <span class="s2">`assets/images/</span><span class="p">${</span><span class="nx">generatedFilename</span><span class="p">}</span><span class="s2">`</span><span class="p">,</span>
    <span class="na">message</span><span class="p">:</span> <span class="s2">`chore: auto-add asset </span><span class="p">${</span><span class="nx">generatedFilename</span><span class="p">}</span><span class="s2">`</span><span class="p">,</span>
    <span class="na">content</span><span class="p">:</span> <span class="nx">base64String</span><span class="p">,</span>
    <span class="na">branch</span><span class="p">:</span> <span class="dl">"</span><span class="s2">main</span><span class="dl">"</span>
  <span class="p">}</span>
<span class="p">};</span>
</code></pre></div></div>

<p><img src="https://blogs.buildwithmanish.com/assets/images/concept_illustration-how-i-built-an-ai-blogger-agent-using-n8ngithub-and-jekyll-3-2025-12-18-85866.png" alt="Concept Illustration" /></p>

<h3 id="4-committing-assets-to-github">4. Committing Assets to GitHub</h3>

<p>I interact directly with the GitHub REST API. This is cleaner than trying to use git commands inside a Docker container within n8n.</p>

<p><strong>Endpoint:</strong> <code class="language-plaintext highlighter-rouge">PUT /repos/{owner}/{repo}/contents/{path}</code></p>

<p>We loop through every generated image and commit it to the <code class="language-plaintext highlighter-rouge">assets/images/</code> directory in the repository. Crucially, after the commit is successful, we construct the <strong>public raw URL</strong> or the <strong>CDN URL</strong> for that image.</p>

<p><strong>The Context Map:</strong>
We aggregate the results into a context map that looks like this:</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
  </span><span class="nl">"hero"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://blogs.buildwithmanish.com/assets/images/k8s-hero.png"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"diagram_1"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://blogs.buildwithmanish.com/assets/images/k8s-networking-flow.png"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<h3 id="5-the-writer-agent-context-injection">5. The Writer Agent (Context Injection)</h3>

<p>Now, and only now, do we invoke the Writer Agent. We pass the original topic <em>and</em> the Context Map of valid image URLs.</p>

<p><strong>The Prompt:</strong></p>
<blockquote>
  <p>“Write a deep-dive technical blog post about .
You MUST use the following image URLs in your markdown.
Place the hero image after the front matter.
Place the architecture diagram in the ‘Network Concepts’ section.
Use strict Markdown syntax.
Include Jekyll Front Matter.”</p>
</blockquote>

<p>This ensures the LLM writes the content <em>around</em> the images we actually possess, rather than making up images that don’t exist.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/code_flow-how-i-built-an-ai-blogger-agent-using-n8ngithub-and-jekyll-4-2025-12-18-33674.png" alt="Code Flow" /></p>

<h3 id="6-final-commit-and-cleanup">6. Final Commit and Cleanup</h3>

<p>The final step in n8n is taking the Markdown output from the Writer Agent and committing it to the <code class="language-plaintext highlighter-rouge">_posts/</code> directory on GitHub. The filename is auto-generated based on the date and title (slugified), e.g., <code class="language-plaintext highlighter-rouge">2025-12-18-k8s-networking.md</code>.</p>

<p>Finally, the Google Sheet row is updated from <code class="language-plaintext highlighter-rouge">Pending</code> to <code class="language-plaintext highlighter-rouge">Done</code>.</p>

<hr />

<h2 id="part-2-the-frontend-jekyll--railway">Part 2: The Frontend (Jekyll &amp; Railway)</h2>

<p>The backend handles the “Supply” side. The frontend handles the “Demand” side. I chose Jekyll because it transforms Markdown into static HTML. It is secure, fast, and requires zero database maintenance on the read path.</p>

<h3 id="1-the-build-pipeline">1. The Build Pipeline</h3>

<p>The deployment architecture is completely event-driven.</p>

<ol>
  <li><strong>Event:</strong> n8n commits a new <code class="language-plaintext highlighter-rouge">.md</code> file to the <code class="language-plaintext highlighter-rouge">main</code> branch on GitHub.</li>
  <li><strong>Trigger:</strong> Railway (my hosting provider) detects the commit via Webhook.</li>
  <li><strong>Build:</strong> Railway pulls the repo and runs the Jekyll build command.</li>
</ol>

<p><img src="https://blogs.buildwithmanish.com/assets/images/architecture_diagram-how-i-built-an-ai-blogger-agent-using-n8ngithub-and-jekyll-5-2025-12-18-28554.png" alt="Architecture Diagram" /></p>

<p><strong>Railway Configuration (<code class="language-plaintext highlighter-rouge">railway.json</code> or Build Command):</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bundle <span class="nb">install</span> <span class="o">&amp;&amp;</span> bundle <span class="nb">exec </span>jekyll build
</code></pre></div></div>

<p>This generates the <code class="language-plaintext highlighter-rouge">_site</code> folder, which Railway serves via NGINX. The entire process from “Commit” to “Live URL” takes approximately 45 seconds.</p>

<h3 id="2-frontend-engineering-beyond-default-themes">2. Frontend Engineering: Beyond Default Themes</h3>

<p>While Jekyll themes exist, I built a custom frontend to support the specific requirements of a technical blog.</p>

<h4 id="a-multi-tag-filtering">A. Multi-Tag Filtering</h4>
<p>Technical posts often cross domains (e.g., “DevOps” and “AWS”). I implemented a filtering system using Vanilla JavaScript.</p>

<p>In the Jekyll template (<code class="language-plaintext highlighter-rouge">home.html</code>), I generate data attributes:</p>

<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;div</span> <span class="na">class=</span><span class="s">"post-card"</span> <span class="na">data-tags=</span><span class="s">"n8n, automation, ai"</span><span class="nt">&gt;</span>
  <span class="c">&lt;!-- Post Content --&gt;</span>
<span class="nt">&lt;/div&gt;</span>
</code></pre></div></div>

<p>The JavaScript simply toggles visibility:</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">filterPosts</span><span class="p">(</span><span class="nx">selectedTag</span><span class="p">)</span> <span class="p">{</span>
  <span class="kd">const</span> <span class="nx">posts</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">querySelectorAll</span><span class="p">(</span><span class="dl">'</span><span class="s1">.post-card</span><span class="dl">'</span><span class="p">);</span>
  <span class="nx">posts</span><span class="p">.</span><span class="nx">forEach</span><span class="p">(</span><span class="nx">post</span> <span class="o">=&gt;</span> <span class="p">{</span>
    <span class="kd">const</span> <span class="nx">tags</span> <span class="o">=</span> <span class="nx">post</span><span class="p">.</span><span class="nx">getAttribute</span><span class="p">(</span><span class="dl">'</span><span class="s1">data-tags</span><span class="dl">'</span><span class="p">).</span><span class="nx">split</span><span class="p">(</span><span class="dl">'</span><span class="s1">, </span><span class="dl">'</span><span class="p">);</span>
    <span class="k">if</span> <span class="p">(</span><span class="nx">tags</span><span class="p">.</span><span class="nx">includes</span><span class="p">(</span><span class="nx">selectedTag</span><span class="p">)</span> <span class="o">||</span> <span class="nx">selectedTag</span> <span class="o">===</span> <span class="dl">'</span><span class="s1">all</span><span class="dl">'</span><span class="p">)</span> <span class="p">{</span>
      <span class="nx">post</span><span class="p">.</span><span class="nx">style</span><span class="p">.</span><span class="nx">display</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">block</span><span class="dl">'</span><span class="p">;</span>
    <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
      <span class="nx">post</span><span class="p">.</span><span class="nx">style</span><span class="p">.</span><span class="nx">display</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">none</span><span class="dl">'</span><span class="p">;</span>
    <span class="p">}</span>
  <span class="p">});</span>
<span class="p">}</span>
</code></pre></div></div>

<h4 id="b-image-lightbox">B. Image Lightbox</h4>
<p>Since the AI generates detailed architecture diagrams, users need to zoom in. I implemented a lightbox feature that intercepts clicks on any image within the <code class="language-plaintext highlighter-rouge">.post-content</code> div.</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">document</span><span class="p">.</span><span class="nx">querySelectorAll</span><span class="p">(</span><span class="dl">'</span><span class="s1">.post-content img</span><span class="dl">'</span><span class="p">).</span><span class="nx">forEach</span><span class="p">(</span><span class="nx">image</span> <span class="o">=&gt;</span> <span class="p">{</span>
  <span class="nx">image</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="dl">'</span><span class="s1">click</span><span class="dl">'</span><span class="p">,</span> <span class="nx">e</span> <span class="o">=&gt;</span> <span class="p">{</span>
    <span class="kd">const</span> <span class="nx">src</span> <span class="o">=</span> <span class="nx">e</span><span class="p">.</span><span class="nx">target</span><span class="p">.</span><span class="nx">src</span><span class="p">;</span>
    <span class="c1">// Logic to open modal overlay with this src</span>
    <span class="nx">openLightbox</span><span class="p">(</span><span class="nx">src</span><span class="p">);</span>
  <span class="p">});</span>
<span class="p">});</span>
</code></pre></div></div>

<h4 id="c-search-functionality">C. Search Functionality</h4>
<p>For a static site, server-side search isn’t an option. I avoided heavy libraries like Algolia for this use case. Instead, I use a build-time generated JSON index.</p>

<p>In <code class="language-plaintext highlighter-rouge">search.json</code> (a Liquid template):</p>
<div class="language-liquid highlighter-rouge"><div class="highlight"><pre class="highlight"><code>---
layout: null
---
[
  
    {
      "title": "Top 5 Programming Languages to learn in 2026",
      "url": "/deep-dive/career-growth/software-engineering/system-design/tech-trends/2025/12/31/top-5-programming-languages-to-learn-in-2026.html",
      "tags": "",
      "content": "![Top 5 Programming Languages Hero](https://blogs.buildwithmanish.com/assets/images/hero-top-5-programming-languages-to-learn-in-2026-0-2025-12-31-69086.png) # Introduction: The Confession of a Senior Engineer I have a confession to make. For years, I was wrong. I was wrong because I fell into the trap that many content creators and tech leads fall into: the trap of the linear roadmap. I used..."
    } ,
  
    {
      "title": "Relational Database Management System: An Engineering Deep-Dive",
      "url": "/deep-dive/database-systems/sql/data-modeling/backend-engineering/2025/12/30/relational-database-management-system.html",
      "tags": "",
      "content": "![Relational Database Management System Hero](https://blogs.buildwithmanish.com/assets/images/hero-relational-database-management-system-0-2025-12-30-77512.png) Despite the meteoric rise of NoSQL, time-series, and vector databases in the last decade, the Relational Database Management System (RDBMS) remains the immutable backbone of global enterprise software. Whether it’s financial ledgers, inventory management, or user identity systems, the relational model&amp;#39;s promise of ACID (Atomicity,..."
    } ,
  
    {
      "title": "Textbook for Excel Exam",
      "url": "/deep-dive/excel-mastery/data-analysis/spreadsheet-engineering/certification/2025/12/29/textbook-for-excel-exam.html",
      "tags": "",
      "content": "![Hero Image](https://blogs.buildwithmanish.com/assets/images/hero-textbook-for-excel-exam-0-2025-12-29-23091.png) # Introduction In the world of data engineering and business intelligence, Microsoft Excel remains the ubiquitous &amp;quot;first draft&amp;quot; of data analysis. While Python, SQL, and Tableau dominate big data pipelines, Excel is where the business world lives. It is effectively a functional programming environment coupled with a two-dimensional..."
    } ,
  
    {
      "title": "Reactive Architecture in Flutter: Mastering BLoC for Scalable State Management",
      "url": "/deep-dive/flutter/state-management/bloc/reactive-architecture/2025/12/18/reactive-architecture-in-flutter-mastering-bloc-for-scalable-state-management.html",
      "tags": "",
      "content": "![Hero Image - Reactive Architecture](https://blogs.buildwithmanish.com/assets/images/hero-reactive-architecture-in-flutter-mastering-bloc-for-scalable-state-management-0-2025-12-18-12168.png) When building scalable mobile applications, the ease of Flutter’s `setState` is a siren song. It works beautifully for a counter app or a simple toggle, but as your application grows into an enterprise-grade solution, coupling business logic tightly with UI widgets creates a codebase that..."
    } ,
  
    {
      "title": "Implementing Agentic Workflows for Java-to-Node.js Migration",
      "url": "/deep-dive/generative-ai/legacy-migration/java-spring/nodejs-nestjs/2025/12/18/implementing-agentic-workflows-for-javatonodejs-migration.html",
      "tags": "",
      "content": "![Hero Image](https://blogs.buildwithmanish.com/assets/images/hero-implementing-agentic-workflows-for-javatonodejs-migration-0-2025-12-18-75248.png) Legacy migration is the &amp;quot;dark matter&amp;quot; of software engineering: we know it makes up a massive portion of the enterprise universe, yet we struggle to observe it directly without getting sucked into a black hole of regression testing and lost business logic. For the last decade, the migration..."
    } ,
  
    {
      "title": "How I built an AI blogger Agent using N8N,Github and Jekyll",
      "url": "/deep-dive/n8n/ai-automation/jekyll/system-design/2025/12/18/how-i-built-an-ai-blogger-agent-using-n8ngithub-and-jekyll.html",
      "tags": "",
      "content": "![Hero Image](https://blogs.buildwithmanish.com/assets/images/hero-how-i-built-an-ai-blogger-agent-using-n8ngithub-and-jekyll-0-2025-12-18-22032.png) In the current landscape of software engineering, &amp;quot;content creation&amp;quot; is often viewed as a secondary concern-a distraction from shipping code. However, technical documentation and knowledge sharing are critical components of system design. The problem isn&amp;#39;t the lack of knowledge; it&amp;#39;s the friction involved in the pipeline: ideation, drafting,..."
    } ,
  
    {
      "title": "How I built an AI blogger Agent using N8N and Jekyll",
      "url": "/deep-dive/n8n/generative-ai/jekyll/automation/2025/12/18/how-i-built-an-ai-blogger-agent-using-n8n-and-jekyll.html",
      "tags": "",
      "content": "As engineers, we often suffer from the “cobbler’s children” syndrome. We build complex, scalable systems for clients and employers, yet our own personal portfolios and blogs gather dust. The friction of context-switching from coding to writing, finding assets, and managing deployments often kills the momentum before a post is even..."
    } ,
  
    {
      "title": "Mastering Razorpay Webhooks: Architecting an Idempotent, Event-Driven Ingestion Engine",
      "url": "/deep-dive/system-design/payments/event-driven-architecture/razorpay/2025/12/17/mastering-razorpay-webhooks-architecting-an-idempotent-eventdriven-ingestion-engine.html",
      "tags": "",
      "content": "Introduction: The Deceptive Simplicity of Webhooks Integrating a payment gateway like Razorpay often starts with a false sense of security. The documentation shows a happy path: user pays, Razorpay calls your webhook, you update the database, and ship the product. In a development environment with one request per minute, this..."
    } ,
  
    {
      "title": "Java Concurrency at Scale: Migrating from Thread Pools to Virtual Threads",
      "url": "/deep-dive/java/concurrency/virtual-threads/software-architecture/2025/12/17/java-concurrency-at-scale-migrating-from-thread-pools-to-virtual-threads.html",
      "tags": "",
      "content": "For the better part of two decades, Java concurrency has been a negotiation with the Operating System. We built high-throughput systems on the back of the “One-Thread-Per-Request” model, eventually hitting the hard ceiling of OS resource limits. We patched this with thread pools, and when that wasn’t enough, we twisted..."
    } ,
  
    {
      "title": "Escaping the if (isAdmin) Trap: Implementing Scalable RBAC and ABAC in Node.js",
      "url": "/deep-dive/nodejs/security/rbac/abac/system-design/2025/12/17/escaping-the-if-isadmin-trap-implementing-scalable-rbac-and-abac-in-nodejs.html",
      "tags": "",
      "content": "We have all written it. The line of code that marks the beginning of the end for a scalable codebase: if (user.isAdmin || (user.role === &amp;#39;manager&amp;#39; &amp;amp;amp;&amp;amp;amp; resource.ownerId === user.id)) { // Allow logic } It starts innocently enough. A boolean flag here, a string check there. But as your..."
    } ,
  
    {
      "title": "Building a Virtual Data Analyst: Architecting Agentic AI Workflows over Power BI",
      "url": "/deep-dive/gen-ai/power-bi/agentic-workflows/python/2025/12/17/building-a-virtual-data-analyst-architecting-agentic-ai-workflows-over-power-bi.html",
      "tags": "",
      "content": "In the rush to adopt Generative AI, organizations have flooded their backlogs with “Chat with your Data” initiatives. Most of these projects follow a predictable, often disappointing, pattern: a text-to-SQL layer slapped on top of a raw data warehouse. While this works for simple questions (“How many widgets did we..."
    } ,
  
    {
      "title": "Beyond the Happy Path: Architecting Fault-Tolerant Recurring Payments with Razorpay",
      "url": "/technical/deep-dive/2025/12/17/beyond-the-happy-path-architecting-faulttolerant-recurring-payments-with-razorpay.html",
      "tags": "",
      "content": "The Illusion of the “Happy Path” In the world of distributed systems, the “happy path” is a sedative. It lulls engineers into a false sense of security where networks never partition, latency is zero, and customers always have sufficient funds in their bank accounts. When building one-off payment flows, you..."
    } ,
  
    {
      "title": "Beyond GET and SET: Architecting Resilient Distributed Caching Patterns for Microservices",
      "url": "/deep-dive/system-design/redis/distributed-systems/microservices/2025/12/17/beyond-get-and-set-architecting-resilient-distributed-caching-patterns-for-microservices.html",
      "tags": "",
      "content": "Introduction In the infancy of a backend application, caching is often treated as a simple key-value store optimization-a sprinkle of “magic dust” to speed up slow SQL queries. Developers implement a basic client.get() and client.set(), push to production, and watch latencies drop. It feels victorious. However, as traffic scales and..."
    } 
  
]
</code></pre></div></div>

<p>The frontend fetches this JSON on page load (or on first focus of the search bar) and performs a client-side filter. It’s incredibly fast for blogs with under 1,000 posts.</p>

<h2 id="advanced-considerations-and-edge-cases">Advanced Considerations and Edge Cases</h2>

<p>Building a “Happy Path” agent is easy. Building one that runs autonomously requires handling failure.</p>

<h3 id="1-hallucination-control">1. Hallucination Control</h3>
<p>Occasionally, the Image Gen API fails or returns a policy violation error.</p>
<ul>
  <li><strong>Mitigation:</strong> The n8n workflow has error paths. If an image fails to generate, the workflow substitutes a generic “placeholder technical pattern” image from a pre-defined bucket. This ensures the blog post doesn’t break due to a missing asset.</li>
</ul>

<h3 id="2-markdown-sanitization">2. Markdown Sanitization</h3>
<p>LLMs sometimes include conversational filler (“Here is your blog post…”) before the Front Matter.</p>
<ul>
  <li><strong>Mitigation:</strong> I use a Regex extraction node in n8n to strip everything before the first <code class="language-plaintext highlighter-rouge">---</code> and after the last character of the content.</li>
</ul>

<h3 id="3-github-api-rate-limits">3. GitHub API Rate Limits</h3>
<p>Committing 7 images + 1 markdown file in rapid succession can trigger secondary rate limits on GitHub if not careful.</p>
<ul>
  <li><strong>Mitigation:</strong> I implemented a <code class="language-plaintext highlighter-rouge">Wait</code> node in n8n, introducing a 2-second delay between commit calls. This also prevents race conditions if multiple workflows trigger simultaneously (though n8n execution mode is set to strictly sequential for this workflow).</li>
</ul>

<h2 id="conclusion-the-roi-of-automation">Conclusion: The ROI of Automation</h2>

<p>This system demonstrates the power of <strong>Composite AI</strong>-combining deterministic code (n8n/GitHub/Jekyll) with probabilistic AI (LLMs).</p>

<p>By decoupling the <em>Planner</em> from the <em>Writer</em> and managing assets programmatically, we achieve a level of consistency and quality that simple “Prompt-to-Blog” tools cannot match.</p>

<p>Here is a comparison of this architecture versus standard AI generation:</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/comparison_chart-how-i-built-an-ai-blogger-agent-using-n8ngithub-and-jekyll-6-2025-12-18-18336.png" alt="Comparison Chart" /></p>

<p>The pipeline is now live. Every time I have an idea, I add a row to Google Sheets. My agent handles the rest, allowing me to focus on high-level architecture rather than CSS tweaks and file management.</p>

<h3 id="next-steps">Next Steps</h3>
<p>The next iteration of this agent will include:</p>
<ol>
  <li><strong>SEO Optimization:</strong> An additional LLM step to analyze keywords and inject them into the metadata.</li>
  <li><strong>Social Media Broadcasting:</strong> Automatically generating LinkedIn and Twitter threads based on the final Markdown and scheduling them via n8n.</li>
  <li><strong>Interactive Components:</strong> Experimenting with generating Mermaid.js code instead of static images for diagrams, allowing for render-time interactivity.</li>
</ol>]]></content><author><name>Manish Tiwari</name></author><category term="deep-dive" /><category term="n8n" /><category term="ai-automation" /><category term="jekyll" /><category term="system-design" /><summary type="html"><![CDATA[]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" /><media:content medium="image" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Implementing Agentic Workflows for Java-to-Node.js Migration</title><link href="https://blogs.buildwithmanish.com/deep-dive/generative-ai/legacy-migration/java-spring/nodejs-nestjs/2025/12/18/implementing-agentic-workflows-for-javatonodejs-migration.html" rel="alternate" type="text/html" title="Implementing Agentic Workflows for Java-to-Node.js Migration" /><published>2025-12-18T00:00:00+00:00</published><updated>2025-12-18T00:00:00+00:00</updated><id>https://blogs.buildwithmanish.com/deep-dive/generative-ai/legacy-migration/java-spring/nodejs-nestjs/2025/12/18/implementing-agentic-workflows-for-javatonodejs-migration</id><content type="html" xml:base="https://blogs.buildwithmanish.com/deep-dive/generative-ai/legacy-migration/java-spring/nodejs-nestjs/2025/12/18/implementing-agentic-workflows-for-javatonodejs-migration.html"><![CDATA[<p><img src="https://blogs.buildwithmanish.com/assets/images/hero-implementing-agentic-workflows-for-javatonodejs-migration-0-2025-12-18-75248.png" alt="Hero Image" /></p>

<p>Legacy migration is the “dark matter” of software engineering: we know it makes up a massive portion of the enterprise universe, yet we struggle to observe it directly without getting sucked into a black hole of regression testing and lost business logic.</p>

<p>For the last decade, the migration of monolithic Java Spring Boot applications to microservices-based Node.js architectures (typically running on Kubernetes) has been a manual, high-friction endeavor. The “Strangler Fig” pattern is reliable, but slow.</p>

<p>However, the emergence of <strong>Agentic Workflows</strong>-specifically Large Language Models (LLMs) integrated into a loop of reasoning, tool execution, and environment feedback-has fundamentally altered the economics of refactoring. We are no longer talking about “code translation” (a simple syntax swap). We are talking about <strong>semantic migration</strong> with self-healing capabilities.</p>

<p>This post details the architecture and implementation of a specialized “Migration Agent” designed to convert strong-typed Spring Boot applications into idiomatic Node.js services (using NestJS as the target due to its structural parity with Spring). We will explore AST parsing, paradigm mapping, and the critical <strong>Plan, Transpile, Verify</strong> feedback loop.</p>

<hr />

<h2 id="architecture-the-migration-agent">Architecture: The Migration Agent</h2>

<p>The flaw in using a standard LLM chat interface for migration is context limits and hallucination. You cannot paste a 50,000-line codebase into a context window and expect a working application.</p>

<p>Instead, we treat the migration as a multi-step agentic workflow. The agent acts as an orchestrator that utilizes specific tools:</p>
<ol>
  <li><strong>File System Readers:</strong> To traverse the repo.</li>
  <li><strong>AST Parsers:</strong> To understand the <em>structure</em> of the code, not just the text.</li>
  <li><strong>Transpilers:</strong> To generate the initial draft.</li>
  <li><strong>Test Runners:</strong> To execute the generated code.</li>
  <li><strong>Debuggers:</strong> To analyze stack traces.</li>
</ol>

<p><img src="https://blogs.buildwithmanish.com/assets/images/architecture_diagram-implementing-agentic-workflows-for-javatonodejs-migration-1-2025-12-18-16851.png" alt="Architecture Diagram" /></p>

<p>The architecture is split into three distinct phases: <strong>Discovery</strong>, <strong>Transpilation</strong>, and <strong>Verification (The Self-Healing Loop)</strong>.</p>

<hr />

<h2 id="phase-1-discovery-via-ast-abstract-syntax-tree">Phase 1: Discovery via AST (Abstract Syntax Tree)</h2>

<p>Regex is insufficient for parsing Java. You cannot reliably identify dependency injection chains or transaction boundaries using string matching. We must construct an Abstract Syntax Tree (AST) of the source Java code to build a dependency graph.</p>

<p>We utilize tools like <code class="language-plaintext highlighter-rouge">JavaParser</code> (wrapped in a Python or Node script) to extract metadata before the LLM even sees the code.</p>

<h3 id="why-ast-matters">Why AST matters</h3>
<p>When migrating a Service class, the Agent needs to know:</p>
<ul>
  <li>What other classes does this inject? (<code class="language-plaintext highlighter-rouge">@Autowired</code>)</li>
  <li>Does it use Aspect-Oriented Programming (AOP)? (<code class="language-plaintext highlighter-rouge">@Transactional</code>)</li>
  <li>What are the return types?</li>
</ul>

<p><img src="https://blogs.buildwithmanish.com/assets/images/concept_illustration-implementing-agentic-workflows-for-javatonodejs-migration-3-2025-12-18-71504.png" alt="Concept Illustration - AST" /></p>

<p>Here is a Python snippet using a wrapper around a Java parser to extract class signatures to feed into the Agent’s “Context Memory”:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">javalang</span>

<span class="k">def</span> <span class="nf">parse_java_structure</span><span class="p">(</span><span class="n">file_content</span><span class="p">):</span>
    <span class="n">tree</span> <span class="o">=</span> <span class="n">javalang</span><span class="p">.</span><span class="n">parse</span><span class="p">.</span><span class="n">parse</span><span class="p">(</span><span class="n">file_content</span><span class="p">)</span>
    <span class="n">structure</span> <span class="o">=</span> <span class="p">{</span>
        <span class="s">"package"</span><span class="p">:</span> <span class="nb">str</span><span class="p">(</span><span class="n">tree</span><span class="p">.</span><span class="n">package</span><span class="p">.</span><span class="n">name</span><span class="p">),</span>
        <span class="s">"imports"</span><span class="p">:</span> <span class="p">[],</span>
        <span class="s">"classes"</span><span class="p">:</span> <span class="p">[]</span>
    <span class="p">}</span>

    <span class="c1"># Extract Imports
</span>    <span class="k">for</span> <span class="n">imp</span> <span class="ow">in</span> <span class="n">tree</span><span class="p">.</span><span class="n">imports</span><span class="p">:</span>
        <span class="n">structure</span><span class="p">[</span><span class="s">"imports"</span><span class="p">].</span><span class="n">append</span><span class="p">(</span><span class="n">imp</span><span class="p">.</span><span class="n">path</span><span class="p">)</span>

    <span class="c1"># Extract Class Metadata
</span>    <span class="k">for</span> <span class="n">type_decl</span> <span class="ow">in</span> <span class="n">tree</span><span class="p">.</span><span class="n">types</span><span class="p">:</span>
        <span class="n">class_info</span> <span class="o">=</span> <span class="p">{</span>
            <span class="s">"name"</span><span class="p">:</span> <span class="n">type_decl</span><span class="p">.</span><span class="n">name</span><span class="p">,</span>
            <span class="s">"annotations"</span><span class="p">:</span> <span class="p">[</span><span class="n">a</span><span class="p">.</span><span class="n">name</span> <span class="k">for</span> <span class="n">a</span> <span class="ow">in</span> <span class="n">type_decl</span><span class="p">.</span><span class="n">annotations</span><span class="p">],</span>
            <span class="s">"methods"</span><span class="p">:</span> <span class="p">[]</span>
        <span class="p">}</span>
        
        <span class="c1"># specific logic to find Dependency Injection
</span>        <span class="k">for</span> <span class="n">field</span> <span class="ow">in</span> <span class="n">type_decl</span><span class="p">.</span><span class="n">fields</span><span class="p">:</span>
            <span class="k">if</span> <span class="s">"Autowired"</span> <span class="ow">in</span> <span class="p">[</span><span class="n">a</span><span class="p">.</span><span class="n">name</span> <span class="k">for</span> <span class="n">a</span> <span class="ow">in</span> <span class="n">field</span><span class="p">.</span><span class="n">annotations</span><span class="p">]:</span>
                <span class="n">class_info</span><span class="p">[</span><span class="s">"dependencies"</span><span class="p">]</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">field</span><span class="p">.</span><span class="nb">type</span><span class="p">.</span><span class="n">name</span><span class="p">)</span>

        <span class="n">structure</span><span class="p">[</span><span class="s">"classes"</span><span class="p">].</span><span class="n">append</span><span class="p">(</span><span class="n">class_info</span><span class="p">)</span>
        
    <span class="k">return</span> <span class="n">structure</span>
</code></pre></div></div>

<p>By running this across the entire Java project, the Agent builds a <strong>Topological Sort</strong> of the application. It knows it must migrate the DTOs and Entities first, then the Repositories, then Services, and finally the Controllers.</p>

<hr />

<h2 id="phase-2-the-core-loop-plan-transpile-verify">Phase 2: The Core Loop (Plan, Transpile, Verify)</h2>

<p>Once the dependency graph is established, the Agent enters its core execution loop. This is not a linear “read-write” operation. It is recursive.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/code_flow-implementing-agentic-workflows-for-javatonodejs-migration-2-2025-12-18-44538.png" alt="Code Flow - Core Loop" /></p>

<h3 id="the-planner-agent">The Planner Agent</h3>
<p>The Planner does not write code. It analyzes the Java file and the AST metadata to create a “Migration Specification.”</p>
<ul>
  <li><strong>Input:</strong> <code class="language-plaintext highlighter-rouge">UserService.java</code></li>
  <li><strong>Output:</strong> JSON Spec
    <ul>
      <li>Target Framework: NestJS</li>
      <li>Dependencies: TypeORM, generic-pool</li>
      <li>Async Strategy: <code class="language-plaintext highlighter-rouge">async/await</code> (replace blocking I/O)</li>
    </ul>
  </li>
</ul>

<h3 id="the-transpiler-paradigm-mapping">The Transpiler (Paradigm Mapping)</h3>

<p>This is where the semantic translation happens. We are targeting <strong>NestJS</strong> because its architecture-Decorators, Modules, and Dependency Injection-maps 1:1 with Spring Boot.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/comparison_chart-implementing-agentic-workflows-for-javatonodejs-migration-4-2025-12-18-63606.png" alt="Comparison Chart" /></p>

<h4 id="1-dependency-injection">1. Dependency Injection</h4>
<p>The Agent must convert field injection (<code class="language-plaintext highlighter-rouge">@Autowired</code>) to constructor injection, which is the standard in TypeScript/NestJS.</p>

<p><strong>Java Source:</strong></p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Service</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">UserService</span> <span class="o">{</span>
    <span class="nd">@Autowired</span>
    <span class="kd">private</span> <span class="nc">UserRepository</span> <span class="n">userRepository</span><span class="o">;</span>

    <span class="kd">public</span> <span class="nc">User</span> <span class="nf">getUser</span><span class="o">(</span><span class="nc">Long</span> <span class="n">id</span><span class="o">)</span> <span class="o">{</span>
        <span class="k">return</span> <span class="n">userRepository</span><span class="o">.</span><span class="na">findById</span><span class="o">(</span><span class="n">id</span><span class="o">).</span><span class="na">orElseThrow</span><span class="o">();</span>
    <span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>

<p><strong>Agent Generated NestJS:</strong></p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">@</span><span class="nd">Injectable</span><span class="p">()</span>
<span class="k">export</span> <span class="kd">class</span> <span class="nx">UserService</span> <span class="p">{</span>
    <span class="c1">// Agent identifies @Autowired and converts to Constructor Injection</span>
    <span class="kd">constructor</span><span class="p">(</span>
        <span class="p">@</span><span class="nd">InjectRepository</span><span class="p">(</span><span class="nx">User</span><span class="p">)</span>
        <span class="k">private</span> <span class="k">readonly</span> <span class="nx">userRepository</span><span class="p">:</span> <span class="nx">Repository</span><span class="o">&lt;</span><span class="nx">User</span><span class="o">&gt;</span><span class="p">,</span>
    <span class="p">)</span> <span class="p">{}</span>

    <span class="k">async</span> <span class="nx">getUser</span><span class="p">(</span><span class="nx">id</span><span class="p">:</span> <span class="kr">number</span><span class="p">):</span> <span class="nb">Promise</span><span class="o">&lt;</span><span class="nx">User</span><span class="o">&gt;</span> <span class="p">{</span>
        <span class="c1">// Agent converts blocking call to async/await</span>
        <span class="kd">const</span> <span class="nx">user</span> <span class="o">=</span> <span class="k">await</span> <span class="k">this</span><span class="p">.</span><span class="nx">userRepository</span><span class="p">.</span><span class="nx">findOneBy</span><span class="p">({</span> <span class="nx">id</span> <span class="p">});</span>
        <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">user</span><span class="p">)</span> <span class="k">throw</span> <span class="k">new</span> <span class="nx">NotFoundException</span><span class="p">();</span>
        <span class="k">return</span> <span class="nx">user</span><span class="p">;</span>
    <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<h4 id="2-managing-dependencies-maven-to-npm">2. Managing Dependencies (Maven to NPM)</h4>
<p>One of the hardest parts of migration is the ecosystem shift. <code class="language-plaintext highlighter-rouge">pom.xml</code> dependencies do not map one-to-one with <code class="language-plaintext highlighter-rouge">package.json</code>.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/code_flow-implementing-agentic-workflows-for-javatonodejs-migration-7-2025-12-18-28413.png" alt="Code Flow - Dependencies" /></p>

<p>The Agent utilizes a lookup dictionary (which it can expand via web search tools) to resolve these:</p>

<table>
  <thead>
    <tr>
      <th style="text-align: left">Java Paradigm</th>
      <th style="text-align: left">Maven Artifact</th>
      <th style="text-align: left">Node.js Equivalent</th>
      <th style="text-align: left">Package</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: left"><strong>Validation</strong></td>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">hibernate-validator</code></td>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">class-validator</code></td>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">class-validator</code>, <code class="language-plaintext highlighter-rouge">class-transformer</code></td>
    </tr>
    <tr>
      <td style="text-align: left"><strong>Utility</strong></td>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">commons-lang3</code></td>
      <td style="text-align: left">Lodash</td>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">lodash</code></td>
    </tr>
    <tr>
      <td style="text-align: left"><strong>HTTP Client</strong></td>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">RestTemplate</code></td>
      <td style="text-align: left">Axios</td>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">@nestjs/axios</code></td>
    </tr>
    <tr>
      <td style="text-align: left"><strong>Logging</strong></td>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">slf4j</code></td>
      <td style="text-align: left">Winston/Pino</td>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">winston</code></td>
    </tr>
  </tbody>
</table>

<p>The Agent must parse the <code class="language-plaintext highlighter-rouge">pom.xml</code>, extract versions, find the compatible Node version, and generate the <code class="language-plaintext highlighter-rouge">package.json</code> before writing application code.</p>

<hr />

<h2 id="phase-3-the-self-healing-feedback-loop">Phase 3: The Self-Healing Feedback Loop</h2>

<p>This is the differentiator between a “Code Generator” and an “Agent.”</p>

<p>Code generated by LLMs will contain bugs. It might use a Java method signature that doesn’t exist in JavaScript (e.g., <code class="language-plaintext highlighter-rouge">string.equals()</code>), or mishandle <code class="language-plaintext highlighter-rouge">Promise</code> resolution.</p>

<p>To solve this, we implement a <strong>Self-Healing Loop</strong>.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/technical_diagram-implementing-agentic-workflows-for-javatonodejs-migration-5-2025-12-18-61781.png" alt="Technical Diagram - Self Healing" /></p>

<h3 id="step-1-test-generation">Step 1: Test Generation</h3>
<p>Immediately after transpiling <code class="language-plaintext highlighter-rouge">UserService.ts</code>, the Agent is tasked to write <code class="language-plaintext highlighter-rouge">UserService.spec.ts</code>. It uses the logic from the original Java <code class="language-plaintext highlighter-rouge">UserServiceTest.java</code> to ensure functional parity.</p>

<h3 id="step-2-execution--capture">Step 2: Execution &amp; Capture</h3>
<p>The Agent executes the test suite using a sandbox runner.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npm <span class="nb">test </span>src/users/user.service.spec.ts
</code></pre></div></div>

<h3 id="step-3-analysis--patching">Step 3: Analysis &amp; Patching</h3>
<p>If the test fails, the Agent does not stop. It captures the <code class="language-plaintext highlighter-rouge">stderr</code> and the stack trace. It feeds this error back into the LLM context along with the source code.</p>

<p><strong>The Prompt Chain:</strong></p>
<ol>
  <li><strong>Context:</strong> “You wrote this Node.js service based on this Java logic.”</li>
  <li><strong>Observation:</strong> “Running the test resulted in <code class="language-plaintext highlighter-rouge">TypeError: Cannot read properties of undefined (reading 'findOne')</code>.”</li>
  <li><strong>Instruction:</strong> “Analyze the stack trace. Reasoning step-by-step, identify why the mock was not injected correctly. Patch the code.”</li>
</ol>

<p><strong>Example Scenario: The “Async” Trap</strong>
A common failure mode is treating asynchronous Node code as synchronous Java code.</p>

<ul>
  <li><em>Agent Draft 1:</em>
    <div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">users</span><span class="p">.</span><span class="nx">forEach</span><span class="p">(</span><span class="nx">user</span> <span class="o">=&gt;</span> <span class="p">{</span>
    <span class="c1">// ERROR: saving inside forEach without Promise.all</span>
    <span class="k">this</span><span class="p">.</span><span class="nx">repo</span><span class="p">.</span><span class="nx">save</span><span class="p">(</span><span class="nx">user</span><span class="p">);</span> 
<span class="p">});</span>
</code></pre></div>    </div>
  </li>
  <li><em>Test Result:</em> Test finishes before data is saved. Assertion fails.</li>
  <li><em>Agent Analysis:</em> “I detected a race condition. <code class="language-plaintext highlighter-rouge">forEach</code> does not await promises. I need to switch to <code class="language-plaintext highlighter-rouge">for...of</code> or <code class="language-plaintext highlighter-rouge">Promise.all</code>.”</li>
  <li><em>Agent Patch:</em>
    <div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">await</span> <span class="nb">Promise</span><span class="p">.</span><span class="nx">all</span><span class="p">(</span><span class="nx">users</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">user</span> <span class="o">=&gt;</span> <span class="k">this</span><span class="p">.</span><span class="nx">repo</span><span class="p">.</span><span class="nx">save</span><span class="p">(</span><span class="nx">user</span><span class="p">)));</span>
</code></pre></div>    </div>
  </li>
</ul>

<p>This loop continues up to a maximum retry count (e.g., 5 attempts) until the tests pass.</p>

<hr />

<h2 id="database-and-entity-migration">Database and Entity Migration</h2>

<p>Migration isn’t just code; it’s data. Moving from Hibernate/JPA to TypeORM or Prisma requires careful handling of relationships.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/concept_illustration-implementing-agentic-workflows-for-javatonodejs-migration-6-2025-12-18-21806.png" alt="Concept Illustration - Database Migration" /></p>

<h3 id="the-lazy-loading-problem">The Lazy Loading Problem</h3>
<p>Java Hibernate heavily relies on Lazy Loading. You fetch a <code class="language-plaintext highlighter-rouge">User</code>, and when you call <code class="language-plaintext highlighter-rouge">user.getOrders()</code>, Hibernate silently queries the DB.</p>

<p>Node.js ORMs generally default to explicit loading to avoid the “N+1 Query Problem.”</p>

<p>The Agent must detect <code class="language-plaintext highlighter-rouge">@OneToMany(fetch = FetchType.LAZY)</code> in the Java Entity and transform the consuming service code to explicitly request relations.</p>

<p><strong>Java (Implicit):</strong></p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Logic inside a Service</span>
<span class="nc">User</span> <span class="n">user</span> <span class="o">=</span> <span class="n">repo</span><span class="o">.</span><span class="na">findById</span><span class="o">(</span><span class="mi">1</span><span class="o">);</span>
<span class="k">return</span> <span class="n">user</span><span class="o">.</span><span class="na">getOrders</span><span class="o">().</span><span class="na">size</span><span class="o">();</span> <span class="c1">// Hibernate triggers query here</span>
</code></pre></div></div>

<p><strong>Node.js (Explicit - Generated by Agent):</strong>
If the Agent simply translates the logic, it will crash because <code class="language-plaintext highlighter-rouge">user.orders</code> is undefined. The Agent must recognize the context and rewrite the query:</p>

<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Agent rewrite</span>
<span class="kd">const</span> <span class="nx">user</span> <span class="o">=</span> <span class="k">await</span> <span class="k">this</span><span class="p">.</span><span class="nx">repo</span><span class="p">.</span><span class="nx">findOne</span><span class="p">({</span>
    <span class="na">where</span><span class="p">:</span> <span class="p">{</span> <span class="na">id</span><span class="p">:</span> <span class="mi">1</span> <span class="p">},</span>
    <span class="na">relations</span><span class="p">:</span> <span class="p">[</span><span class="dl">'</span><span class="s1">orders</span><span class="dl">'</span><span class="p">]</span> <span class="c1">// Explicit join added by Agent</span>
<span class="p">});</span>
<span class="k">return</span> <span class="nx">user</span><span class="p">.</span><span class="nx">orders</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span>
</code></pre></div></div>

<hr />

<h2 id="implementation-guide-the-controller-agent">Implementation Guide: The Controller Agent</h2>

<p>Let’s look at the specific implementation of the “Controller Migration” step using Python and LangChain.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">langchain.chat_models</span> <span class="kn">import</span> <span class="n">ChatOpenAI</span>
<span class="kn">from</span> <span class="nn">langchain.schema</span> <span class="kn">import</span> <span class="n">SystemMessage</span><span class="p">,</span> <span class="n">HumanMessage</span>

<span class="k">def</span> <span class="nf">migrate_controller</span><span class="p">(</span><span class="n">java_code</span><span class="p">,</span> <span class="n">service_interface_ts</span><span class="p">):</span>
    <span class="n">chat</span> <span class="o">=</span> <span class="n">ChatOpenAI</span><span class="p">(</span><span class="n">model</span><span class="o">=</span><span class="s">"gpt-4-turbo"</span><span class="p">,</span> <span class="n">temperature</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
    
    <span class="n">system_prompt</span> <span class="o">=</span> <span class="s">"""
    You are a Senior Backend Engineer migrating Spring Boot to NestJS.
    Rules:
    1. Map @RequestMapping to @Controller.
    2. Map @GetMapping, @PostMapping to NestJS decorators.
    3. Ensure all service calls use 'await'.
    4. Use DTOs defined in the provided context.
    5. Return strict TypeScript types.
    """</span>
    
    <span class="n">user_prompt</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"""
    ### Java Source:
    </span><span class="si">{</span><span class="n">java_code</span><span class="si">}</span><span class="s">
    
    ### Associated TypeScript Service Interface:
    </span><span class="si">{</span><span class="n">service_interface_ts</span><span class="si">}</span><span class="s">
    
    Convert this Controller to NestJS. Output ONLY the code.
    """</span>
    
    <span class="n">response</span> <span class="o">=</span> <span class="n">chat</span><span class="p">([</span>
        <span class="n">SystemMessage</span><span class="p">(</span><span class="n">content</span><span class="o">=</span><span class="n">system_prompt</span><span class="p">),</span>
        <span class="n">HumanMessage</span><span class="p">(</span><span class="n">content</span><span class="o">=</span><span class="n">user_prompt</span><span class="p">)</span>
    <span class="p">])</span>
    
    <span class="k">return</span> <span class="n">response</span><span class="p">.</span><span class="n">content</span>

<span class="k">def</span> <span class="nf">verification_loop</span><span class="p">(</span><span class="n">file_path</span><span class="p">,</span> <span class="n">test_path</span><span class="p">):</span>
    <span class="n">attempts</span> <span class="o">=</span> <span class="mi">0</span>
    <span class="k">while</span> <span class="n">attempts</span> <span class="o">&lt;</span> <span class="mi">5</span><span class="p">:</span>
        <span class="n">result</span> <span class="o">=</span> <span class="n">run_npm_test</span><span class="p">(</span><span class="n">test_path</span><span class="p">)</span>
        <span class="k">if</span> <span class="n">result</span><span class="p">.</span><span class="n">success</span><span class="p">:</span>
            <span class="k">return</span> <span class="bp">True</span>
        
        <span class="c1"># Self-Healing Step
</span>        <span class="n">error_log</span> <span class="o">=</span> <span class="n">result</span><span class="p">.</span><span class="n">stderr</span>
        <span class="n">current_code</span> <span class="o">=</span> <span class="n">read_file</span><span class="p">(</span><span class="n">file_path</span><span class="p">)</span>
        
        <span class="n">patch</span> <span class="o">=</span> <span class="n">agent_fix_code</span><span class="p">(</span><span class="n">current_code</span><span class="p">,</span> <span class="n">error_log</span><span class="p">)</span>
        <span class="n">write_file</span><span class="p">(</span><span class="n">file_path</span><span class="p">,</span> <span class="n">patch</span><span class="p">)</span>
        <span class="n">attempts</span> <span class="o">+=</span> <span class="mi">1</span>
        
    <span class="k">raise</span> <span class="nb">Exception</span><span class="p">(</span><span class="s">"Failed to converge on working code."</span><span class="p">)</span>
</code></pre></div></div>

<p>This Python script represents the orchestrator. It holds the state, manages file I/O, and executes the shell commands that provide the “reality check” for the LLM.</p>

<hr />

<h2 id="advanced-edge-cases">Advanced Edge Cases</h2>

<p>In production migrations, several edge cases typically break simple conversion tools.</p>

<h3 id="1-static-contexts--singletons">1. Static Contexts &amp; Singletons</h3>
<p>Java developers often use <code class="language-plaintext highlighter-rouge">public static</code> methods for utilities or holding state. In a Node.js microservice context, global mutable state is dangerous due to the single-threaded event loop serving all requests. The Agent must detect <code class="language-plaintext highlighter-rouge">static</code> fields that are mutated and refactor them into request-scoped providers or Redis-backed state.</p>

<h3 id="2-threadlocal">2. ThreadLocal</h3>
<p>Spring Security often stores user context in <code class="language-plaintext highlighter-rouge">ThreadLocal</code>.</p>
<ul>
  <li><strong>Java:</strong> <code class="language-plaintext highlighter-rouge">SecurityContextHolder.getContext().getAuthentication()</code></li>
  <li><strong>Node:</strong> The Agent must identify this pattern and introduce <code class="language-plaintext highlighter-rouge">AsyncLocalStorage</code> (ALS) or pass the <code class="language-plaintext highlighter-rouge">request</code> object through the method chain (Context Propagation).</li>
</ul>

<h3 id="3-synchronized-blocks">3. Synchronized Blocks</h3>
<p>Java uses <code class="language-plaintext highlighter-rouge">synchronized</code> for thread safety. Node.js is single-threaded, so standard concurrency issues don’t apply the same way, but race conditions on external resources (DB, File IO) do. The Agent must translate <code class="language-plaintext highlighter-rouge">synchronized</code> blocks into distributed locks (e.g., using Redlock) if the intent was to protect a shared resource across instances, or remove them entirely if they were protecting in-memory state.</p>

<hr />

<h2 id="conclusion">Conclusion</h2>

<p>Migrating from Java to Node.js is no longer a brute-force manual effort. By implementing an <strong>Agentic Workflow</strong>, we move from “Syntax Translation” to “Logic Porting.”</p>

<p>The combination of <strong>AST Analysis</strong> for deep understanding, <strong>Paradigm Mapping</strong> for idiomatic translation, and a <strong>Self-Healing Feedback Loop</strong> for functional verification allows us to automate 70-80% of the migration effort.</p>

<p>The remaining 20%? That’s where you, the Senior Engineer, come in. You review the architecture, optimize the database queries that the Agent mapped too literally, and handle the complex business logic that requires human intuition. But the boilerplate, the DTOs, the basic CRUD, and the test suites? Let the Agents handle that.</p>]]></content><author><name>Manish Tiwari</name></author><category term="deep-dive" /><category term="generative-ai" /><category term="legacy-migration" /><category term="java-spring" /><category term="nodejs-nestjs" /><summary type="html"><![CDATA[]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" /><media:content medium="image" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Reactive Architecture in Flutter: Mastering BLoC for Scalable State Management</title><link href="https://blogs.buildwithmanish.com/deep-dive/flutter/state-management/bloc/reactive-architecture/2025/12/18/reactive-architecture-in-flutter-mastering-bloc-for-scalable-state-management.html" rel="alternate" type="text/html" title="Reactive Architecture in Flutter: Mastering BLoC for Scalable State Management" /><published>2025-12-18T00:00:00+00:00</published><updated>2025-12-18T00:00:00+00:00</updated><id>https://blogs.buildwithmanish.com/deep-dive/flutter/state-management/bloc/reactive-architecture/2025/12/18/reactive-architecture-in-flutter-mastering-bloc-for-scalable-state-management</id><content type="html" xml:base="https://blogs.buildwithmanish.com/deep-dive/flutter/state-management/bloc/reactive-architecture/2025/12/18/reactive-architecture-in-flutter-mastering-bloc-for-scalable-state-management.html"><![CDATA[<p><img src="https://blogs.buildwithmanish.com/assets/images/hero-reactive-architecture-in-flutter-mastering-bloc-for-scalable-state-management-0-2025-12-18-12168.png" alt="Hero Image - Reactive Architecture" /></p>

<p>When building scalable mobile applications, the ease of Flutter’s <code class="language-plaintext highlighter-rouge">setState</code> is a siren song. It works beautifully for a counter app or a simple toggle, but as your application grows into an enterprise-grade solution, coupling business logic tightly with UI widgets creates a codebase that is impossible to test, difficult to refactor, and prone to regression bugs.</p>

<p>To build robust Flutter applications, we must move beyond imperative state manipulation and embrace <strong>Reactive Architecture</strong>.</p>

<p>In this deep dive, we are going to deconstruct the Business Logic Component (BLoC) pattern. We aren’t just looking at how to use the library; we are examining the architectural principles of separating presentation from logic using Dart Streams, enforcing immutability with <code class="language-plaintext highlighter-rouge">freezed</code>, handling complex side effects, and proving correctness through isolated unit testing.</p>

<h2 id="the-imperative-vs-reactive-paradigm">The Imperative vs. Reactive Paradigm</h2>

<p>In imperative programming, you tell the computer <em>how</em> to change the state (e.g., “Change the text variable to ‘Loading’ and redraw”). In reactive programming, you define <em>streams</em> of data. The UI becomes a passive view that simply reacts to the current state of the stream.</p>

<p>The BLoC pattern, created by Google, relies on this concept of <strong>Unidirectional Data Flow (UDF)</strong>.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/architecture_diagram-reactive-architecture-in-flutter-mastering-bloc-for-scalable-state-management-1-2025-12-18-75752.png" alt="Architecture Diagram - UDF" /></p>

<ol>
  <li><strong>Events (Input):</strong> The UI captures user interactions (taps, inputs) and adds an <code class="language-plaintext highlighter-rouge">Event</code> to the sink.</li>
  <li><strong>BLoC (Processor):</strong> The component receives the event, communicates with repositories/services, executes business logic, and emits a new <code class="language-plaintext highlighter-rouge">State</code>.</li>
  <li><strong>States (Output):</strong> The UI listens to the state stream and redraws itself based on the data received.</li>
</ol>

<p>This strict separation means your UI knows <em>nothing</em> about how data is fetched, and your Logic knows <em>nothing</em> about how data is rendered. This is the holy grail of clean architecture.</p>

<h2 id="the-reactive-core-streams-sinks-and-transformers">The Reactive Core: Streams, Sinks, and Transformers</h2>

<p>At the heart of BLoC is the Dart <code class="language-plaintext highlighter-rouge">Stream</code>. If you are coming from a purely imperative background, think of a <code class="language-plaintext highlighter-rouge">Stream</code> as an asynchronous pipe. Data enters one end (via a <code class="language-plaintext highlighter-rouge">StreamController</code> or <code class="language-plaintext highlighter-rouge">Sink</code>) and exits the other end where listeners react to it.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/concept_illustration-reactive-architecture-in-flutter-mastering-bloc-for-scalable-state-management-2-2025-12-18-60940.png" alt="Concept Illustration - Reactive Streams" /></p>

<p>In the context of the <code class="language-plaintext highlighter-rouge">flutter_bloc</code> library, the complexity of managing StreamControllers, subscriptions, and memory leaks (closing streams) is abstracted away, allowing us to focus on the transitions:</p>

<div class="language-dart highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Conceptual Model of a BLoC</span>
<span class="n">Stream</span><span class="p">&lt;</span><span class="n">State</span><span class="p">&gt;</span> <span class="n">mapEventToState</span><span class="p">(</span><span class="n">Event</span> <span class="n">event</span><span class="p">)</span> <span class="kd">async</span><span class="o">*</span> <span class="p">{</span>
  <span class="k">if</span> <span class="p">(</span><span class="n">event</span> <span class="k">is</span> <span class="n">LoadData</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">yield</span> <span class="n">LoadingState</span><span class="p">();</span>
    <span class="k">try</span> <span class="p">{</span>
      <span class="kd">final</span> <span class="n">data</span> <span class="o">=</span> <span class="k">await</span> <span class="n">repository</span><span class="o">.</span><span class="na">fetch</span><span class="p">();</span>
      <span class="k">yield</span> <span class="n">LoadedState</span><span class="p">(</span><span class="n">data</span><span class="p">);</span>
    <span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="n">e</span><span class="p">)</span> <span class="p">{</span>
      <span class="k">yield</span> <span class="n">ErrorState</span><span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="na">toString</span><span class="p">());</span>
    <span class="p">}</span>
  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>While the modern BLoC library uses <code class="language-plaintext highlighter-rouge">on&lt;Event&gt;</code> handlers rather than <code class="language-plaintext highlighter-rouge">mapEventToState</code> generators, the underlying reactive principle remains: <strong>State is a function of time and events.</strong></p>

<h2 id="supercharging-bloc-with-freezed-immutable-unions">Supercharging BLoC with Freezed: Immutable Unions</h2>

<p>One of the biggest pain points in standard BLoC implementation is verbosity and lack of safety when handling states. If you implement states as standard Dart classes, you have to manually override <code class="language-plaintext highlighter-rouge">==</code> and <code class="language-plaintext highlighter-rouge">hashCode</code> to ensure that BLoC knows when a state has actually changed (value equality vs. referential equality). Furthermore, checking states with <code class="language-plaintext highlighter-rouge">if (state is Loading)</code> is brittle.</p>

<p>Enter <strong>Freezed</strong>. This code-generation package allows us to define “Sealed Classes” (Union Types). This gives us two massive advantages:</p>
<ol>
  <li><strong>Immutability out of the box.</strong></li>
  <li><strong>Pattern Matching</strong> (forcing us to handle every possible state).</li>
</ol>

<p><img src="https://blogs.buildwithmanish.com/assets/images/code_flow-reactive-architecture-in-flutter-mastering-bloc-for-scalable-state-management-3-2025-12-18-41841.png" alt="Code Flow - Freezed Implementation" /></p>

<p>Let’s look at a real-world scenario: An <code class="language-plaintext highlighter-rouge">OrderProcessing</code> feature.</p>

<h3 id="1-defining-events-and-states-with-freezed">1. Defining Events and States with Freezed</h3>

<div class="language-dart highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="s">'package:freezed_annotation/freezed_annotation.dart'</span><span class="o">;</span>

<span class="kn">part</span> <span class="s">'order_bloc.freezed.dart'</span><span class="o">;</span>

<span class="c1">// EVENTS: actions the user or system can take</span>
<span class="nd">@freezed</span>
<span class="kd">class</span> <span class="nc">OrderEvent</span> <span class="k">with</span> <span class="n">_$OrderEvent</span> <span class="p">{</span>
  <span class="kd">const</span> <span class="kd">factory</span> <span class="n">OrderEvent</span><span class="o">.</span><span class="na">started</span><span class="p">()</span> <span class="o">=</span> <span class="n">_Started</span><span class="p">;</span>
  <span class="kd">const</span> <span class="kd">factory</span> <span class="n">OrderEvent</span><span class="o">.</span><span class="na">submitOrder</span><span class="p">({</span><span class="kd">required</span> <span class="kt">String</span> <span class="n">orderId</span><span class="p">,</span> <span class="kd">required</span> <span class="kt">double</span> <span class="n">amount</span><span class="p">})</span> <span class="o">=</span> <span class="n">_SubmitOrder</span><span class="p">;</span>
  <span class="kd">const</span> <span class="kd">factory</span> <span class="n">OrderEvent</span><span class="o">.</span><span class="na">cancelOrder</span><span class="p">(</span><span class="kt">String</span> <span class="n">orderId</span><span class="p">)</span> <span class="o">=</span> <span class="n">_CancelOrder</span><span class="p">;</span>
<span class="p">}</span>

<span class="c1">// STATES: The various snapshots of the UI</span>
<span class="nd">@freezed</span>
<span class="kd">class</span> <span class="nc">OrderState</span> <span class="k">with</span> <span class="n">_$OrderState</span> <span class="p">{</span>
  <span class="kd">const</span> <span class="kd">factory</span> <span class="n">OrderState</span><span class="o">.</span><span class="na">initial</span><span class="p">()</span> <span class="o">=</span> <span class="n">_Initial</span><span class="p">;</span>
  <span class="kd">const</span> <span class="kd">factory</span> <span class="n">OrderState</span><span class="o">.</span><span class="na">processing</span><span class="p">()</span> <span class="o">=</span> <span class="n">_Processing</span><span class="p">;</span>
  <span class="kd">const</span> <span class="kd">factory</span> <span class="n">OrderState</span><span class="o">.</span><span class="na">success</span><span class="p">(</span><span class="kt">String</span> <span class="n">confirmationCode</span><span class="p">)</span> <span class="o">=</span> <span class="n">_Success</span><span class="p">;</span>
  <span class="kd">const</span> <span class="kd">factory</span> <span class="n">OrderState</span><span class="o">.</span><span class="na">failure</span><span class="p">(</span><span class="kt">String</span> <span class="n">error</span><span class="p">)</span> <span class="o">=</span> <span class="n">_Failure</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<h3 id="2-implementing-the-business-logic">2. Implementing the Business Logic</h3>

<p>Now we implement the BLoC. Notice how the logic is purely focused on data transformation. It doesn’t care if the UI is a button or a gesture detector.</p>

<div class="language-dart highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="s">'package:flutter_bloc/flutter_bloc.dart'</span><span class="o">;</span>

<span class="kd">class</span> <span class="nc">OrderBloc</span> <span class="kd">extends</span> <span class="n">Bloc</span><span class="p">&lt;</span><span class="n">OrderEvent</span><span class="p">,</span> <span class="n">OrderState</span><span class="p">&gt;</span> <span class="p">{</span>
  <span class="kd">final</span> <span class="n">OrderRepository</span> <span class="n">_repository</span><span class="p">;</span>

  <span class="n">OrderBloc</span><span class="p">(</span><span class="k">this</span><span class="o">.</span><span class="na">_repository</span><span class="p">)</span> <span class="o">:</span> <span class="k">super</span><span class="p">(</span><span class="kd">const</span> <span class="n">OrderState</span><span class="o">.</span><span class="na">initial</span><span class="p">())</span> <span class="p">{</span>
    <span class="kd">on</span><span class="p">&lt;</span><span class="n">_SubmitOrder</span><span class="p">&gt;(</span><span class="n">_onSubmitOrder</span><span class="p">);</span>
    <span class="kd">on</span><span class="p">&lt;</span><span class="n">_CancelOrder</span><span class="p">&gt;(</span><span class="n">_onCancelOrder</span><span class="p">);</span>
  <span class="p">}</span>

  <span class="n">Future</span><span class="p">&lt;</span><span class="kt">void</span><span class="p">&gt;</span> <span class="n">_onSubmitOrder</span><span class="p">(</span>
    <span class="n">_SubmitOrder</span> <span class="n">event</span><span class="p">,</span>
    <span class="n">Emitter</span><span class="p">&lt;</span><span class="n">OrderState</span><span class="p">&gt;</span> <span class="n">emit</span><span class="p">,</span>
  <span class="p">)</span> <span class="kd">async</span> <span class="p">{</span>
    <span class="c1">// 1. Emit loading state</span>
    <span class="n">emit</span><span class="p">(</span><span class="kd">const</span> <span class="n">OrderState</span><span class="o">.</span><span class="na">processing</span><span class="p">());</span>

    <span class="k">try</span> <span class="p">{</span>
      <span class="c1">// 2. Perform async business logic</span>
      <span class="kd">final</span> <span class="n">result</span> <span class="o">=</span> <span class="k">await</span> <span class="n">_repository</span><span class="o">.</span><span class="na">processOrder</span><span class="p">(</span><span class="n">event</span><span class="o">.</span><span class="na">orderId</span><span class="p">,</span> <span class="n">event</span><span class="o">.</span><span class="na">amount</span><span class="p">);</span>
      
      <span class="c1">// 3. Emit success state with data</span>
      <span class="n">emit</span><span class="p">(</span><span class="n">OrderState</span><span class="o">.</span><span class="na">success</span><span class="p">(</span><span class="n">result</span><span class="o">.</span><span class="na">confirmationCode</span><span class="p">));</span>
    <span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="n">e</span><span class="p">)</span> <span class="p">{</span>
      <span class="c1">// 4. Handle errors gracefully</span>
      <span class="n">emit</span><span class="p">(</span><span class="n">OrderState</span><span class="o">.</span><span class="na">failure</span><span class="p">(</span><span class="s">"Failed to process order: </span><span class="si">${e.toString()}</span><span class="s">"</span><span class="p">));</span>
    <span class="p">}</span>
  <span class="p">}</span>

  <span class="n">Future</span><span class="p">&lt;</span><span class="kt">void</span><span class="p">&gt;</span> <span class="n">_onCancelOrder</span><span class="p">(</span>
    <span class="n">_CancelOrder</span> <span class="n">event</span><span class="p">,</span>
    <span class="n">Emitter</span><span class="p">&lt;</span><span class="n">OrderState</span><span class="p">&gt;</span> <span class="n">emit</span><span class="p">,</span>
  <span class="p">)</span> <span class="kd">async</span> <span class="p">{</span>
    <span class="c1">// Logic for cancellation...</span>
  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Because <code class="language-plaintext highlighter-rouge">Freezed</code> overrides <code class="language-plaintext highlighter-rouge">==</code> and <code class="language-plaintext highlighter-rouge">hashCode</code>, if you emit <code class="language-plaintext highlighter-rouge">OrderState.processing()</code> twice in a row, the BLoC library detects the values are identical and will <em>not</em> trigger a UI rebuild, optimizing performance automatically.</p>

<h2 id="dependency-injection-multiblocprovider-and-scope">Dependency Injection: MultiBlocProvider and Scope</h2>

<p>A BLoC needs dependencies (Repositories, API Clients). A Widget needs a BLoC. Connecting these cleanly is vital for a decoupled architecture.</p>

<p>The <code class="language-plaintext highlighter-rouge">flutter_bloc</code> package provides <code class="language-plaintext highlighter-rouge">BlocProvider</code> to inject a BLoC into the widget tree. Since providers rely on <code class="language-plaintext highlighter-rouge">InheritedWidget</code> under the hood, any child widget can access the BLoC instance via <code class="language-plaintext highlighter-rouge">context.read&lt;OrderBloc&gt;()</code>.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/architecture_diagram-reactive-architecture-in-flutter-mastering-bloc-for-scalable-state-management-4-2025-12-18-8002.png" alt="Architecture Diagram - Dependency Injection" /></p>

<h3 id="the-strategic-use-of-multiblocprovider">The Strategic Use of MultiBlocProvider</h3>

<p>In complex apps, you often have global BLoCs (Authentication, Theme, Localization) and scoped BLoCs (OrderProcessing, Cart).</p>

<p><strong>Do not create BLoCs globally.</strong> Create them as close to where they are needed as possible, but high enough to be shared by all widgets that need them.</p>

<div class="language-dart highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
  <span class="n">runApp</span><span class="p">(</span>
    <span class="c1">// Inject Repositories first</span>
    <span class="n">RepositoryProvider</span><span class="p">(</span>
      <span class="nl">create:</span> <span class="p">(</span><span class="n">context</span><span class="p">)</span> <span class="o">=</span><span class="p">&gt;</span> <span class="n">OrderRepository</span><span class="p">(),</span>
      <span class="nl">child:</span> <span class="kd">const</span> <span class="n">MyApp</span><span class="p">(),</span>
    <span class="p">),</span>
  <span class="p">);</span>
<span class="p">}</span>

<span class="kd">class</span> <span class="nc">MyApp</span> <span class="kd">extends</span> <span class="n">StatelessWidget</span> <span class="p">{</span>
  <span class="kd">const</span> <span class="n">MyApp</span><span class="p">({</span><span class="k">super</span><span class="o">.</span><span class="na">key</span><span class="p">});</span>

  <span class="nd">@override</span>
  <span class="n">Widget</span> <span class="n">build</span><span class="p">(</span><span class="n">BuildContext</span> <span class="n">context</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">return</span> <span class="n">MultiBlocProvider</span><span class="p">(</span>
      <span class="nl">providers:</span> <span class="p">[</span>
        <span class="c1">// Global State: Authentication</span>
        <span class="n">BlocProvider</span><span class="p">&lt;</span><span class="n">AuthBloc</span><span class="p">&gt;(</span>
          <span class="nl">create:</span> <span class="p">(</span><span class="n">context</span><span class="p">)</span> <span class="o">=</span><span class="p">&gt;</span> <span class="n">AuthBloc</span><span class="p">().</span><span class="o">.</span><span class="na">add</span><span class="p">(</span><span class="kd">const</span> <span class="n">AuthEvent</span><span class="o">.</span><span class="na">appStarted</span><span class="p">()),</span>
        <span class="p">),</span>
        <span class="c1">// Feature State: Global Shopping Cart</span>
        <span class="n">BlocProvider</span><span class="p">&lt;</span><span class="n">CartBloc</span><span class="p">&gt;(</span>
          <span class="nl">create:</span> <span class="p">(</span><span class="n">context</span><span class="p">)</span> <span class="o">=</span><span class="p">&gt;</span> <span class="n">CartBloc</span><span class="p">(),</span>
        <span class="p">),</span>
      <span class="p">],</span>
      <span class="nl">child:</span> <span class="n">MaterialApp</span><span class="p">(</span>
        <span class="nl">home:</span> <span class="kd">const</span> <span class="n">HomeScreen</span><span class="p">(),</span>
      <span class="p">),</span>
    <span class="p">);</span>
  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>When navigating to a specific feature screen (e.g., Checkout), create the <code class="language-plaintext highlighter-rouge">OrderBloc</code> specifically for that route:</p>

<div class="language-dart highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">Navigator</span><span class="o">.</span><span class="na">of</span><span class="p">(</span><span class="n">context</span><span class="p">)</span><span class="o">.</span><span class="na">push</span><span class="p">(</span>
  <span class="n">MaterialPageRoute</span><span class="p">(</span>
    <span class="nl">builder:</span> <span class="p">(</span><span class="n">context</span><span class="p">)</span> <span class="o">=</span><span class="p">&gt;</span> <span class="n">BlocProvider</span><span class="p">(</span>
      <span class="nl">create:</span> <span class="p">(</span><span class="n">context</span><span class="p">)</span> <span class="o">=</span><span class="p">&gt;</span> <span class="n">OrderBloc</span><span class="p">(</span><span class="n">context</span><span class="o">.</span><span class="na">read</span><span class="p">&lt;</span><span class="n">OrderRepository</span><span class="p">&gt;()),</span>
      <span class="nl">child:</span> <span class="kd">const</span> <span class="n">CheckoutScreen</span><span class="p">(),</span>
    <span class="p">),</span>
  <span class="p">),</span>
<span class="p">);</span>
</code></pre></div></div>

<p>This ensures that when <code class="language-plaintext highlighter-rouge">CheckoutScreen</code> is popped from the navigation stack, the <code class="language-plaintext highlighter-rouge">OrderBloc</code> is automatically closed (disposed), freeing up resources.</p>

<h2 id="handling-side-effects-vs-ui-building">Handling Side Effects vs. UI Building</h2>

<p>This is where 90% of developers get BLoC wrong.</p>

<p><strong>The Problem:</strong> You want to show a <code class="language-plaintext highlighter-rouge">SnackBar</code> or navigate to a new screen when the state changes to <code class="language-plaintext highlighter-rouge">Success</code>.
<strong>The Mistake:</strong> Putting this logic inside the <code class="language-plaintext highlighter-rouge">builder</code> function of a <code class="language-plaintext highlighter-rouge">BlocBuilder</code>.</p>

<p>The <code class="language-plaintext highlighter-rouge">builder</code> function in <code class="language-plaintext highlighter-rouge">BlocBuilder</code> is strictly for returning a Widget. It may be called multiple times by the Flutter framework. Triggering navigation or showing alerts here creates unstable behavior and errors (e.g., “setState() or markNeedsBuild() called during build”).</p>

<h3 id="the-solution-listener-vs-builder">The Solution: Listener vs. Builder</h3>

<ul>
  <li><strong>BlocBuilder:</strong> Used <em>only</em> for rendering UI based on state.</li>
  <li><strong>BlocListener:</strong> Used for side effects that happen <em>once</em> per state change (Navigation, Dialogs, Toasts).</li>
  <li><strong>BlocConsumer:</strong> A helper that combines both.</li>
</ul>

<p><img src="https://blogs.buildwithmanish.com/assets/images/comparison_chart-reactive-architecture-in-flutter-mastering-bloc-for-scalable-state-management-5-2025-12-18-88786.png" alt="Comparison Chart - Side Effects" /></p>

<p>Here is the correct implementation using <code class="language-plaintext highlighter-rouge">freezed</code> pattern matching inside the UI:</p>

<div class="language-dart highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">CheckoutView</span> <span class="kd">extends</span> <span class="n">StatelessWidget</span> <span class="p">{</span>
  <span class="kd">const</span> <span class="n">CheckoutView</span><span class="p">({</span><span class="k">super</span><span class="o">.</span><span class="na">key</span><span class="p">});</span>

  <span class="nd">@override</span>
  <span class="n">Widget</span> <span class="n">build</span><span class="p">(</span><span class="n">BuildContext</span> <span class="n">context</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">return</span> <span class="n">BlocConsumer</span><span class="p">&lt;</span><span class="n">OrderBloc</span><span class="p">,</span> <span class="n">OrderState</span><span class="p">&gt;(</span>
      <span class="c1">// LISTEN: Handle Side Effects</span>
      <span class="nl">listener:</span> <span class="p">(</span><span class="n">context</span><span class="p">,</span> <span class="n">state</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">state</span><span class="o">.</span><span class="na">whenOrNull</span><span class="p">(</span>
          <span class="nl">success:</span> <span class="p">(</span><span class="n">confirmationCode</span><span class="p">)</span> <span class="p">{</span>
            <span class="n">ScaffoldMessenger</span><span class="o">.</span><span class="na">of</span><span class="p">(</span><span class="n">context</span><span class="p">)</span><span class="o">.</span><span class="na">showSnackBar</span><span class="p">(</span>
              <span class="n">SnackBar</span><span class="p">(</span><span class="nl">content:</span> <span class="n">Text</span><span class="p">(</span><span class="s">'Order Placed! Code: </span><span class="si">$confirmationCode</span><span class="s">'</span><span class="p">)),</span>
            <span class="p">);</span>
            <span class="n">Navigator</span><span class="o">.</span><span class="na">pop</span><span class="p">(</span><span class="n">context</span><span class="p">);</span> 
          <span class="p">},</span>
          <span class="nl">failure:</span> <span class="p">(</span><span class="n">error</span><span class="p">)</span> <span class="p">{</span>
            <span class="n">ScaffoldMessenger</span><span class="o">.</span><span class="na">of</span><span class="p">(</span><span class="n">context</span><span class="p">)</span><span class="o">.</span><span class="na">showSnackBar</span><span class="p">(</span>
              <span class="n">SnackBar</span><span class="p">(</span><span class="nl">content:</span> <span class="n">Text</span><span class="p">(</span><span class="n">error</span><span class="p">),</span> <span class="nl">backgroundColor:</span> <span class="n">Colors</span><span class="o">.</span><span class="na">red</span><span class="p">),</span>
            <span class="p">);</span>
          <span class="p">},</span>
        <span class="p">);</span>
      <span class="p">},</span>
      <span class="c1">// BUILD: Render UI</span>
      <span class="nl">builder:</span> <span class="p">(</span><span class="n">context</span><span class="p">,</span> <span class="n">state</span><span class="p">)</span> <span class="p">{</span>
        <span class="k">return</span> <span class="n">state</span><span class="o">.</span><span class="na">maybeWhen</span><span class="p">(</span>
          <span class="nl">processing:</span> <span class="p">()</span> <span class="o">=</span><span class="p">&gt;</span> <span class="kd">const</span> <span class="n">Center</span><span class="p">(</span><span class="nl">child:</span> <span class="n">CircularProgressIndicator</span><span class="p">()),</span>
          <span class="nl">orElse:</span> <span class="p">()</span> <span class="o">=</span><span class="p">&gt;</span> <span class="n">Column</span><span class="p">(</span>
            <span class="nl">children:</span> <span class="p">[</span>
              <span class="kd">const</span> <span class="n">Text</span><span class="p">(</span><span class="s">"Review your order"</span><span class="p">),</span>
              <span class="n">ElevatedButton</span><span class="p">(</span>
                <span class="nl">onPressed:</span> <span class="p">()</span> <span class="p">{</span>
                  <span class="n">context</span><span class="o">.</span><span class="na">read</span><span class="p">&lt;</span><span class="n">OrderBloc</span><span class="p">&gt;()</span><span class="o">.</span><span class="na">add</span><span class="p">(</span>
                    <span class="kd">const</span> <span class="n">OrderEvent</span><span class="o">.</span><span class="na">submitOrder</span><span class="p">(</span><span class="nl">orderId:</span> <span class="s">'123'</span><span class="p">,</span> <span class="nl">amount:</span> <span class="mf">99.99</span><span class="p">)</span>
                  <span class="p">);</span>
                <span class="p">},</span>
                <span class="nl">child:</span> <span class="kd">const</span> <span class="n">Text</span><span class="p">(</span><span class="s">"Pay Now"</span><span class="p">),</span>
              <span class="p">)</span>
            <span class="p">],</span>
          <span class="p">),</span>
        <span class="p">);</span>
      <span class="p">},</span>
    <span class="p">);</span>
  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Notice the use of <code class="language-plaintext highlighter-rouge">state.whenOrNull</code> and <code class="language-plaintext highlighter-rouge">state.maybeWhen</code>. These are generated by Freezed.</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">listener</code>: We only care about Success and Failure.</li>
  <li><code class="language-plaintext highlighter-rouge">builder</code>: We show a spinner for Processing, and the form for everything else.</li>
</ul>

<h2 id="testing-the-ultimate-benefit-of-bloc">Testing: The Ultimate Benefit of BLoC</h2>

<p>If you write logic inside a UI Widget (StatefulWidget), testing it requires <code class="language-plaintext highlighter-rouge">widget_test</code>, which is slower and requires mocking the Flutter engine context.</p>

<p>With BLoC, your business logic is pure Dart code. You can test it in complete isolation using the <code class="language-plaintext highlighter-rouge">bloc_test</code> package. This is unit testing on steroids.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/technical_diagram-reactive-architecture-in-flutter-mastering-bloc-for-scalable-state-management-6-2025-12-18-82639.png" alt="Technical Diagram - Testing Flow" /></p>

<h3 id="writing-the-test">Writing the Test</h3>

<p>We define the “Scenario,” the “Act” (event added), and the “Expectation” (sequence of states).</p>

<div class="language-dart highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="s">'package:bloc_test/bloc_test.dart'</span><span class="o">;</span>
<span class="kn">import</span> <span class="s">'package:flutter_test/flutter_test.dart'</span><span class="o">;</span>
<span class="kn">import</span> <span class="s">'package:mocktail/mocktail.dart'</span><span class="o">;</span>

<span class="c1">// Mock the dependency</span>
<span class="kd">class</span> <span class="nc">MockOrderRepository</span> <span class="kd">extends</span> <span class="n">Mock</span> <span class="kd">implements</span> <span class="n">OrderRepository</span> <span class="p">{}</span>

<span class="kt">void</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
  <span class="kd">late</span> <span class="n">OrderBloc</span> <span class="n">orderBloc</span><span class="p">;</span>
  <span class="kd">late</span> <span class="n">MockOrderRepository</span> <span class="n">mockRepo</span><span class="p">;</span>

  <span class="n">setUp</span><span class="p">(()</span> <span class="p">{</span>
    <span class="n">mockRepo</span> <span class="o">=</span> <span class="n">MockOrderRepository</span><span class="p">();</span>
    <span class="n">orderBloc</span> <span class="o">=</span> <span class="n">OrderBloc</span><span class="p">(</span><span class="n">mockRepo</span><span class="p">);</span>
  <span class="p">});</span>

  <span class="n">tearDown</span><span class="p">(()</span> <span class="p">{</span>
    <span class="n">orderBloc</span><span class="o">.</span><span class="na">close</span><span class="p">();</span>
  <span class="p">});</span>

  <span class="n">group</span><span class="p">(</span><span class="s">'OrderBloc'</span><span class="p">,</span> <span class="p">()</span> <span class="p">{</span>
    <span class="kd">final</span> <span class="n">tOrderId</span> <span class="o">=</span> <span class="s">'123'</span><span class="p">;</span>
    <span class="kd">final</span> <span class="n">tAmount</span> <span class="o">=</span> <span class="mf">50.0</span><span class="p">;</span>
    <span class="kd">final</span> <span class="n">tCode</span> <span class="o">=</span> <span class="s">'CONFIRM-ABC'</span><span class="p">;</span>

    <span class="n">test</span><span class="p">(</span><span class="s">'initial state is OrderState.initial'</span><span class="p">,</span> <span class="p">()</span> <span class="p">{</span>
      <span class="n">expect</span><span class="p">(</span><span class="n">orderBloc</span><span class="o">.</span><span class="na">state</span><span class="p">,</span> <span class="kd">const</span> <span class="n">OrderState</span><span class="o">.</span><span class="na">initial</span><span class="p">());</span>
    <span class="p">});</span>

    <span class="n">blocTest</span><span class="p">&lt;</span><span class="n">OrderBloc</span><span class="p">,</span> <span class="n">OrderState</span><span class="p">&gt;(</span>
      <span class="s">'emits [processing, success] when SubmitOrder is successful'</span><span class="p">,</span>
      <span class="nl">build:</span> <span class="p">()</span> <span class="p">{</span>
        <span class="c1">// Arrange</span>
        <span class="n">when</span><span class="p">(()</span> <span class="o">=</span><span class="p">&gt;</span> <span class="n">mockRepo</span><span class="o">.</span><span class="na">processOrder</span><span class="p">(</span><span class="n">tOrderId</span><span class="p">,</span> <span class="n">tAmount</span><span class="p">))</span>
            <span class="o">.</span><span class="na">thenAnswer</span><span class="p">((</span><span class="n">_</span><span class="p">)</span> <span class="kd">async</span> <span class="o">=</span><span class="p">&gt;</span> <span class="n">OrderResult</span><span class="p">(</span><span class="n">tCode</span><span class="p">));</span>
        <span class="k">return</span> <span class="n">orderBloc</span><span class="p">;</span>
      <span class="p">},</span>
      <span class="nl">act:</span> <span class="p">(</span><span class="n">bloc</span><span class="p">)</span> <span class="o">=</span><span class="p">&gt;</span> <span class="n">bloc</span><span class="o">.</span><span class="na">add</span><span class="p">(</span><span class="n">OrderEvent</span><span class="o">.</span><span class="na">submitOrder</span><span class="p">(</span><span class="nl">orderId:</span> <span class="n">tOrderId</span><span class="p">,</span> <span class="nl">amount:</span> <span class="n">tAmount</span><span class="p">)),</span>
      <span class="nl">expect:</span> <span class="p">()</span> <span class="o">=</span><span class="p">&gt;</span> <span class="p">[</span>
        <span class="kd">const</span> <span class="n">OrderState</span><span class="o">.</span><span class="na">processing</span><span class="p">(),</span>
        <span class="n">OrderState</span><span class="o">.</span><span class="na">success</span><span class="p">(</span><span class="n">tCode</span><span class="p">),</span>
      <span class="p">],</span>
      <span class="nl">verify:</span> <span class="p">(</span><span class="n">_</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">verify</span><span class="p">(()</span> <span class="o">=</span><span class="p">&gt;</span> <span class="n">mockRepo</span><span class="o">.</span><span class="na">processOrder</span><span class="p">(</span><span class="n">tOrderId</span><span class="p">,</span> <span class="n">tAmount</span><span class="p">))</span><span class="o">.</span><span class="na">called</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
      <span class="p">},</span>
    <span class="p">);</span>

    <span class="n">blocTest</span><span class="p">&lt;</span><span class="n">OrderBloc</span><span class="p">,</span> <span class="n">OrderState</span><span class="p">&gt;(</span>
      <span class="s">'emits [processing, failure] when SubmitOrder throws'</span><span class="p">,</span>
      <span class="nl">build:</span> <span class="p">()</span> <span class="p">{</span>
        <span class="n">when</span><span class="p">(()</span> <span class="o">=</span><span class="p">&gt;</span> <span class="n">mockRepo</span><span class="o">.</span><span class="na">processOrder</span><span class="p">(</span><span class="n">any</span><span class="p">(),</span> <span class="n">any</span><span class="p">()))</span>
            <span class="o">.</span><span class="na">thenThrow</span><span class="p">(</span><span class="n">Exception</span><span class="p">(</span><span class="s">'Payment Declined'</span><span class="p">));</span>
        <span class="k">return</span> <span class="n">orderBloc</span><span class="p">;</span>
      <span class="p">},</span>
      <span class="nl">act:</span> <span class="p">(</span><span class="n">bloc</span><span class="p">)</span> <span class="o">=</span><span class="p">&gt;</span> <span class="n">bloc</span><span class="o">.</span><span class="na">add</span><span class="p">(</span><span class="n">OrderEvent</span><span class="o">.</span><span class="na">submitOrder</span><span class="p">(</span><span class="nl">orderId:</span> <span class="n">tOrderId</span><span class="p">,</span> <span class="nl">amount:</span> <span class="n">tAmount</span><span class="p">)),</span>
      <span class="nl">expect:</span> <span class="p">()</span> <span class="o">=</span><span class="p">&gt;</span> <span class="p">[</span>
        <span class="kd">const</span> <span class="n">OrderState</span><span class="o">.</span><span class="na">processing</span><span class="p">(),</span>
        <span class="kd">const</span> <span class="n">OrderState</span><span class="o">.</span><span class="na">failure</span><span class="p">(</span><span class="s">"Failed to process order: Exception: Payment Declined"</span><span class="p">),</span>
      <span class="p">],</span>
    <span class="p">);</span>
  <span class="p">});</span>
<span class="p">}</span>
</code></pre></div></div>

<p>This test suite gives you 100% confidence that your business logic works correctly before you even draw a single pixel on the screen.</p>

<h2 id="advanced-techniques-event-transformers">Advanced Techniques: Event Transformers</h2>

<p>For Senior Engineers, BLoC offers powerful control over <em>how</em> events are processed using <code class="language-plaintext highlighter-rouge">EventTransformer</code>.</p>

<p>Imagine a Search Bar. Users type quickly. You don’t want to hit your API on every keystroke. In an imperative world, you’d manually manage a <code class="language-plaintext highlighter-rouge">Timer</code>. In BLoC, you just use a reactive operator: <code class="language-plaintext highlighter-rouge">debounce</code>.</p>

<div class="language-dart highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="s">'package:rxdart/rxdart.dart'</span><span class="o">;</span>

<span class="n">EventTransformer</span><span class="p">&lt;</span><span class="n">T</span><span class="p">&gt;</span> <span class="n">debounce</span><span class="p">&lt;</span><span class="n">T</span><span class="p">&gt;(</span><span class="n">Duration</span> <span class="n">duration</span><span class="p">)</span> <span class="p">{</span>
  <span class="k">return</span> <span class="p">(</span><span class="n">events</span><span class="p">,</span> <span class="n">mapper</span><span class="p">)</span> <span class="o">=</span><span class="p">&gt;</span> <span class="n">events</span><span class="o">.</span><span class="na">debounceTime</span><span class="p">(</span><span class="n">duration</span><span class="p">)</span><span class="o">.</span><span class="na">flatMap</span><span class="p">(</span><span class="n">mapper</span><span class="p">);</span>
<span class="p">}</span>

<span class="kd">class</span> <span class="nc">SearchBloc</span> <span class="kd">extends</span> <span class="n">Bloc</span><span class="p">&lt;</span><span class="n">SearchEvent</span><span class="p">,</span> <span class="n">SearchState</span><span class="p">&gt;</span> <span class="p">{</span>
  <span class="n">SearchBloc</span><span class="p">(</span><span class="k">this</span><span class="o">.</span><span class="na">_api</span><span class="p">)</span> <span class="o">:</span> <span class="k">super</span><span class="p">(</span><span class="n">SearchInitial</span><span class="p">())</span> <span class="p">{</span>
    <span class="kd">on</span><span class="p">&lt;</span><span class="n">SearchTextChanged</span><span class="p">&gt;(</span>
      <span class="n">_onTextChanged</span><span class="p">,</span>
      <span class="c1">// Apply the transformer</span>
      <span class="nl">transformer:</span> <span class="n">debounce</span><span class="p">(</span><span class="kd">const</span> <span class="n">Duration</span><span class="p">(</span><span class="nl">milliseconds:</span> <span class="mi">300</span><span class="p">)),</span>
    <span class="p">);</span>
  <span class="p">}</span>

  <span class="n">Future</span><span class="p">&lt;</span><span class="kt">void</span><span class="p">&gt;</span> <span class="n">_onTextChanged</span><span class="p">(</span><span class="n">SearchTextChanged</span> <span class="n">event</span><span class="p">,</span> <span class="n">Emitter</span><span class="p">&lt;</span><span class="n">SearchState</span><span class="p">&gt;</span> <span class="n">emit</span><span class="p">)</span> <span class="kd">async</span> <span class="p">{</span>
    <span class="c1">// This logic only runs if the user stops typing for 300ms</span>
    <span class="kd">final</span> <span class="n">results</span> <span class="o">=</span> <span class="k">await</span> <span class="n">_api</span><span class="o">.</span><span class="na">search</span><span class="p">(</span><span class="n">event</span><span class="o">.</span><span class="na">query</span><span class="p">);</span>
    <span class="n">emit</span><span class="p">(</span><span class="n">SearchSuccess</span><span class="p">(</span><span class="n">results</span><span class="p">));</span>
  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>This elegant integration of <code class="language-plaintext highlighter-rouge">rxdart</code> with BLoC transforms a complex concurrency problem into a single line of configuration.</p>

<h2 id="conclusion">Conclusion</h2>

<p>Reactive Architecture with BLoC is not just a library choice; it is a commitment to quality. By strictly separating UI from Logic, implementing immutable state with Freezed, and adhering to the Input/Output stream contract, you create applications that are:</p>

<ol>
  <li><strong>Scalable:</strong> New features can be added as new BLoCs without breaking existing ones.</li>
  <li><strong>Testable:</strong> Logic is verified in isolation.</li>
  <li><strong>Predictable:</strong> Unidirectional flow makes debugging state changes trivial.</li>
</ol>

<p>While the boilerplate may seem higher initially compared to <code class="language-plaintext highlighter-rouge">setState</code> or <code class="language-plaintext highlighter-rouge">ChangeNotifier</code>, the payoff in maintainability for long-term projects is exponential. In the world of Flutter development, BLoC is the bedrock of professional engineering.</p>]]></content><author><name>Manish Tiwari</name></author><category term="deep-dive" /><category term="flutter" /><category term="state-management" /><category term="bloc" /><category term="reactive-architecture" /><summary type="html"><![CDATA[]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" /><media:content medium="image" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Beyond GET and SET: Architecting Resilient Distributed Caching Patterns for Microservices</title><link href="https://blogs.buildwithmanish.com/deep-dive/system-design/redis/distributed-systems/microservices/2025/12/17/beyond-get-and-set-architecting-resilient-distributed-caching-patterns-for-microservices.html" rel="alternate" type="text/html" title="Beyond GET and SET: Architecting Resilient Distributed Caching Patterns for Microservices" /><published>2025-12-17T00:00:00+00:00</published><updated>2025-12-17T00:00:00+00:00</updated><id>https://blogs.buildwithmanish.com/deep-dive/system-design/redis/distributed-systems/microservices/2025/12/17/beyond-get-and-set-architecting-resilient-distributed-caching-patterns-for-microservices</id><content type="html" xml:base="https://blogs.buildwithmanish.com/deep-dive/system-design/redis/distributed-systems/microservices/2025/12/17/beyond-get-and-set-architecting-resilient-distributed-caching-patterns-for-microservices.html"><![CDATA[<p><img src="https://blogs.buildwithmanish.com/assets/images/hero-beyond-get-and-set-architecting-resilient-distributed-caching-patterns-for-microservices-0-2025-12-17-15651.png" alt="Hero Image - Abstract representation of distributed caching" /></p>

<h2 id="introduction">Introduction</h2>

<p>In the infancy of a backend application, caching is often treated as a simple key-value store optimization-a sprinkle of “magic dust” to speed up slow SQL queries. Developers implement a basic <code class="language-plaintext highlighter-rouge">client.get()</code> and <code class="language-plaintext highlighter-rouge">client.set()</code>, push to production, and watch latencies drop. It feels victorious.</p>

<p>However, as traffic scales and the architecture fractures into microservices, this naive approach becomes a ticking time bomb. In distributed systems, a cache is not just a performance booster; it is a critical component of system stability. When used incorrectly, a cache becomes a source of non-deterministic bugs, data inconsistency, and catastrophic cascading failures.</p>

<p>The difference between a junior implementation and a senior architect’s design lies in how the system behaves when things go wrong. What happens when the cache cluster nodes flap? What happens when a marketing push expires 10 million keys at the exact same second? How do you maintain read-your-own-write consistency across distributed nodes?</p>

<p>This post dives deep into the architecture of resilient caching. We will move beyond simple storage and explore robust topologies, mitigation strategies for reliability failures, and advanced sharding techniques.</p>

<hr />

<h2 id="caching-topologies--patterns">Caching Topologies &amp; Patterns</h2>

<p>Choosing the right caching pattern dictates how your application interacts with the data store. There is no “one size fits all”; the choice depends heavily on your read/write ratio and consistency requirements.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/comparison_chart-beyond-get-and-set-architecting-resilient-distributed-caching-patterns-for-microservices-1-2025-12-17-28518.png" alt="Comparison Chart of Caching Patterns" /></p>

<h3 id="1-cache-aside-lazy-loading">1. Cache-Aside (Lazy Loading)</h3>
<p>This is the most common pattern in distributed systems. The application code (the side) is responsible for orchestrating the data flow.</p>

<ul>
  <li><strong>Flow:</strong> The app checks the cache. If it hits, return data. If it misses, the app queries the database, updates the cache, and returns the data.</li>
  <li><strong>Pros:</strong> Resilient to cache failure (the DB is the source of truth); data model in cache can differ from DB.</li>
  <li><strong>Cons:</strong> “Stale data” window between DB update and cache expiry; logic clutter in application code.</li>
</ul>

<h3 id="2-read-through--write-through">2. Read-Through / Write-Through</h3>
<p>Here, the application treats the cache as the main data store. The cache itself is responsible for fetching data from the DB or writing to it.</p>

<ul>
  <li><strong>Read-Through:</strong> On a miss, the cache loader fetches from the DB, caches it, and returns it.</li>
  <li><strong>Write-Through:</strong> Data is written to the cache, which synchronously writes to the DB.</li>
  <li><strong>Pros:</strong> Application logic is cleaner (DRY); strong consistency for reads (since cache is always updated on write).</li>
  <li><strong>Cons:</strong> Higher write latency (two writes must succeed); requires a cache provider that supports custom loader logic (e.g., RedisGears or specific framework abstractions).</li>
</ul>

<h3 id="3-write-back-write-behind">3. Write-Back (Write-Behind)</h3>
<p>The application writes to the cache, and the cache acknowledges immediately. The cache then asynchronously flushes the data to the DB.</p>

<ul>
  <li><strong>Pros:</strong> Incredible write performance; absorbs massive write spikes (e.g., IoT sensor data, clickstreams).</li>
  <li><strong>Cons:</strong> <strong>Data Loss Risk.</strong> If the cache crashes before flushing, data is gone forever.</li>
  <li><strong>Use Case:</strong> Analytics counters, non-critical user interaction logs.</li>
</ul>

<hr />

<h2 id="architecture-sidecar-vs-centralized-cluster">Architecture: Sidecar vs. Centralized Cluster</h2>

<p>Where does the cache live relative to your microservices?</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/architecture_diagram-beyond-get-and-set-architecting-resilient-distributed-caching-patterns-for-microservices-2-2025-12-17-9807.png" alt="Architecture Diagram: Sidecar vs Centralized" /></p>

<h3 id="the-centralized-cluster-redismemcached">The Centralized Cluster (Redis/Memcached)</h3>
<p>The traditional approach. A dedicated fleet of Redis nodes serves all microservices.</p>
<ul>
  <li><strong>Pros:</strong> Shared state across all service instances; independent scaling of compute (app) and memory (cache).</li>
  <li><strong>Cons:</strong> Network latency (serialization + wire time); potential for “Noisy Neighbor” issues if multiple services share the same cluster.</li>
</ul>

<h3 id="the-sidecar-topology">The Sidecar Topology</h3>
<p>With the rise of Service Meshes (Istio, Linkerd) and Dapr, the sidecar pattern places a cache instance (like a small Redis process or Envoy filter) on the same network namespace or pod as the application container.</p>

<ul>
  <li><strong>Pros:</strong> Sub-millisecond latency (loopback interface); isolation (Service A’s cache load doesn’t affect Service B).</li>
  <li><strong>Cons:</strong> Cache fragmentation (low hit rates if requests are load-balanced randomly across pods); difficult to maintain consistency across sidecars; memory overhead per pod.</li>
</ul>

<p><strong>Verdict:</strong> Use a <strong>Centralized Cluster</strong> for shared entity data (User Profiles, Product Catalog). Use <strong>Sidecars</strong> for ephemeral, service-specific configuration or highly transient session state where consistency is less critical than raw speed.</p>

<hr />

<h2 id="the-big-three-reliability-failures">The “Big Three” Reliability Failures</h2>

<p>In high-throughput systems, the way your cache handles expiry and misses determines whether your database survives peak load. We must architecturally defend against the “Big Three.”</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/concept_illustration-beyond-get-and-set-architecting-resilient-distributed-caching-patterns-for-microservices-3-2025-12-17-14034.png" alt="Concept Illustration of Cache Failures" /></p>

<h3 id="1-cache-avalanche">1. Cache Avalanche</h3>
<p><strong>The Scenario:</strong> You cache 10,000 product details, all with a TTL of 60 minutes. At 12:00 PM, you deploy. At 1:00 PM, all 10,000 keys expire simultaneously. Your database is instantly hammered by thousands of reconstruction queries.</p>

<p><strong>The Solution: TTL Jitter.</strong>
Never use a hardcoded TTL. Always add a random variance.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">random</span>

<span class="k">def</span> <span class="nf">set_product_cache</span><span class="p">(</span><span class="n">product_id</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span>
    <span class="n">base_ttl</span> <span class="o">=</span> <span class="mi">3600</span>  <span class="c1"># 60 minutes
</span>    <span class="c1"># Add +/- 10% jitter
</span>    <span class="n">jitter</span> <span class="o">=</span> <span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="o">-</span><span class="mi">360</span><span class="p">,</span> <span class="mi">360</span><span class="p">)</span> 
    <span class="n">final_ttl</span> <span class="o">=</span> <span class="n">base_ttl</span> <span class="o">+</span> <span class="n">jitter</span>
    
    <span class="n">redis_client</span><span class="p">.</span><span class="n">setex</span><span class="p">(</span><span class="sa">f</span><span class="s">"product:</span><span class="si">{</span><span class="n">product_id</span><span class="si">}</span><span class="s">"</span><span class="p">,</span> <span class="n">final_ttl</span><span class="p">,</span> <span class="n">data</span><span class="p">)</span>
</code></pre></div></div>

<p>By spreading the expiry, you smooth out the re-computation load on the database, turning a spike into a manageable curve.</p>

<h3 id="2-cache-penetration">2. Cache Penetration</h3>
<p><strong>The Scenario:</strong> A malicious actor (or a buggy crawler) requests IDs that do not exist in your database (e.g., <code class="language-plaintext highlighter-rouge">id=-1</code> or UUIDs that aren’t real). The cache misses, the DB is queried, returns nothing, and nothing is cached. The attack continues, effectively DoS-ing your database.</p>

<p><strong>The Solution: Bloom Filters and Null Caching.</strong></p>

<p><strong>Strategy A: Cache Nulls.</strong> If the DB returns nothing, cache a “Null Object” with a short TTL (e.g., 5 minutes).</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">get_user</span><span class="p">(</span><span class="n">user_id</span><span class="p">):</span>
    <span class="n">cache_key</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"user:</span><span class="si">{</span><span class="n">user_id</span><span class="si">}</span><span class="s">"</span>
    <span class="n">data</span> <span class="o">=</span> <span class="n">redis_client</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">cache_key</span><span class="p">)</span>
    
    <span class="k">if</span> <span class="n">data</span> <span class="o">==</span> <span class="s">"NULL"</span><span class="p">:</span>
        <span class="k">return</span> <span class="bp">None</span>  <span class="c1"># Hit on a known non-existent key
</span>    <span class="k">if</span> <span class="n">data</span><span class="p">:</span>
        <span class="k">return</span> <span class="n">deserialize</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
        
    <span class="c1"># DB Lookup
</span>    <span class="n">user</span> <span class="o">=</span> <span class="n">db</span><span class="p">.</span><span class="n">find_user</span><span class="p">(</span><span class="n">user_id</span><span class="p">)</span>
    <span class="k">if</span> <span class="ow">not</span> <span class="n">user</span><span class="p">:</span>
        <span class="c1"># Cache the negative result for a short time
</span>        <span class="n">redis_client</span><span class="p">.</span><span class="n">setex</span><span class="p">(</span><span class="n">cache_key</span><span class="p">,</span> <span class="mi">300</span><span class="p">,</span> <span class="s">"NULL"</span><span class="p">)</span>
        <span class="k">return</span> <span class="bp">None</span>
        
    <span class="n">redis_client</span><span class="p">.</span><span class="n">setex</span><span class="p">(</span><span class="n">cache_key</span><span class="p">,</span> <span class="mi">3600</span><span class="p">,</span> <span class="n">serialize</span><span class="p">(</span><span class="n">user</span><span class="p">))</span>
    <span class="k">return</span> <span class="n">user</span>
</code></pre></div></div>

<p><strong>Strategy B: Bloom Filters.</strong> Before even hitting Redis, check a Bloom Filter (a probabilistic data structure). If the Bloom Filter says the key <em>definitely doesn’t exist</em>, reject the request immediately.</p>

<h3 id="3-cache-breakdown-thundering-herd">3. Cache Breakdown (Thundering Herd)</h3>
<p><strong>The Scenario:</strong> A single key (e.g., “Homepage_Top_News”) is extremely hot (10k req/sec). The key expires. Instantly, 10,000 requests hit the cache, miss, and <em>all 10,000</em> rush to the database to calculate the same value.</p>

<p><strong>The Solution: Mutex Locking (Check-Lock-Check).</strong>
Only allow ONE thread/process to rebuild the cache. Everyone else waits or receives stale data.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/code_flow-beyond-get-and-set-architecting-resilient-distributed-caching-patterns-for-microservices-4-2025-12-17-78837.png" alt="Code Flow Diagram for Thundering Herd" /></p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">uuid</span>

<span class="k">def</span> <span class="nf">get_hot_key</span><span class="p">(</span><span class="n">key</span><span class="p">):</span>
    <span class="n">value</span> <span class="o">=</span> <span class="n">redis_client</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
    <span class="k">if</span> <span class="n">value</span><span class="p">:</span>
        <span class="k">return</span> <span class="n">value</span>

    <span class="c1"># Key missed. Acquire distributed lock.
</span>    <span class="n">lock_key</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"lock:</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s">"</span>
    <span class="n">token</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">uuid</span><span class="p">.</span><span class="n">uuid4</span><span class="p">())</span>
    
    <span class="c1"># Try to acquire lock with 10s timeout to prevent deadlocks
</span>    <span class="k">if</span> <span class="n">redis_client</span><span class="p">.</span><span class="nb">set</span><span class="p">(</span><span class="n">lock_key</span><span class="p">,</span> <span class="n">token</span><span class="p">,</span> <span class="n">nx</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">ex</span><span class="o">=</span><span class="mi">10</span><span class="p">):</span>
        <span class="k">try</span><span class="p">:</span>
            <span class="c1"># Recheck cache just in case another thread finished 
</span>            <span class="c1"># while we were waiting for the lock
</span>            <span class="n">value</span> <span class="o">=</span> <span class="n">redis_client</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
            <span class="k">if</span> <span class="n">value</span><span class="p">:</span>
                <span class="k">return</span> <span class="n">value</span>
            
            <span class="c1"># I am the chosen one. Query DB.
</span>            <span class="n">value</span> <span class="o">=</span> <span class="n">db_query</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
            <span class="n">redis_client</span><span class="p">.</span><span class="n">setex</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="mi">3600</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
            <span class="k">return</span> <span class="n">value</span>
        <span class="k">finally</span><span class="p">:</span>
            <span class="c1"># Release lock safely using Lua script (omitted for brevity)
</span>            <span class="k">if</span> <span class="n">redis_client</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">lock_key</span><span class="p">)</span> <span class="o">==</span> <span class="n">token</span><span class="p">:</span>
                <span class="n">redis_client</span><span class="p">.</span><span class="n">delete</span><span class="p">(</span><span class="n">lock_key</span><span class="p">)</span>
    <span class="k">else</span><span class="p">:</span>
        <span class="c1"># Failed to get lock. Sleep and retry (or return stale if architecture allows)
</span>        <span class="n">time</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">get_hot_key</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
</code></pre></div></div>

<hr />

<h2 id="consistency-vs-availability-the-hard-truths">Consistency vs. Availability: The Hard Truths</h2>

<p>There are only two hard things in Computer Science: cache invalidation and naming things. In microservices, Strong Consistency with caching is essentially impossible without sacrificing Availability (CAP theorem).</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/technical_diagram-beyond-get-and-set-architecting-resilient-distributed-caching-patterns-for-microservices-5-2025-12-17-57893.png" alt="Technical Diagram: Consistency Models" /></p>

<h3 id="the-double-delete-strategy">The “Double Delete” Strategy</h3>
<p>When updating data, you have two choices: update the cache or delete the cache. <strong>Deleting is safer</strong> because updating implies you know the exact final state, which might be complex due to serialization.</p>

<p>However, a simple <code class="language-plaintext highlighter-rouge">db.update(); cache.delete();</code> is flawed.</p>
<ol>
  <li>Thread A updates DB.</li>
  <li>Thread A deletes Cache.</li>
  <li>Thread B reads DB (which might still be returning old data due to replication lag).</li>
  <li>Thread B repopulates Cache with <em>old</em> data.</li>
  <li>Cache is now permanently stale.</li>
</ol>

<p><strong>Solution: Delayed Double Delete.</strong></p>
<ol>
  <li>Delete Cache.</li>
  <li>Update Database.</li>
  <li>Sleep (Wait for DB replication lag, e.g., 500ms).</li>
  <li>Delete Cache <em>Again</em>.</li>
</ol>

<p>This ensures that any read that occurred during the race condition is flushed out.</p>

<hr />

<h2 id="advanced-sharding-consistent-hashing">Advanced Sharding: Consistent Hashing</h2>

<p>When you scale Redis beyond a single node, you need to shard keys. The naive approach is Modulo Hashing:
<code class="language-plaintext highlighter-rouge">Node = hash(key) % N</code> (where N is number of nodes).</p>

<p><strong>The Problem:</strong> If you add a node (N becomes N+1), the result of the modulo changes for almost <em>every key</em>. 100% of your cache is invalidated instantly. This is a cache apocalypse.</p>

<p><strong>The Solution: Consistent Hashing (Ring Topology).</strong>
Imagine a circle (Ring) representing the hash space (0 to 2^32).</p>
<ol>
  <li>Place your Server Nodes at points on the ring based on <code class="language-plaintext highlighter-rouge">hash(ServerIP)</code>.</li>
  <li>Place your Keys on the ring based on <code class="language-plaintext highlighter-rouge">hash(Key)</code>.</li>
  <li>To find the node for a key, move <em>clockwise</em> on the ring until you hit a server.</li>
</ol>

<p><img src="https://blogs.buildwithmanish.com/assets/images/technical_diagram-beyond-get-and-set-architecting-resilient-distributed-caching-patterns-for-microservices-6-2025-12-17-4281.png" alt="Technical Diagram: Consistent Hashing Ring" /></p>

<p><strong>Virtual Nodes:</strong>
If you have few nodes, the distribution on the ring might be uneven (Node A gets 80% of data). We create “Virtual Nodes.” Node A exists on the ring as <code class="language-plaintext highlighter-rouge">NodeA_1</code>, <code class="language-plaintext highlighter-rouge">NodeA_2</code>… <code class="language-plaintext highlighter-rouge">NodeA_100</code>. This statistically ensures uniform distribution.</p>

<p>When a node is added or removed, only the keys falling into that specific segment of the ring are affected (roughly <code class="language-plaintext highlighter-rouge">1/N</code> of the keys). This allows elastic scaling without flushing the cache.</p>

<hr />

<h2 id="multi-level-caching-l1l2-architecture">Multi-Level Caching (L1/L2) Architecture</h2>

<p>For extreme performance requirements, a single remote Redis call (1-2ms) is too slow. You need microseconds. This leads to <strong>L1/L2 Caching</strong>.</p>

<ul>
  <li><strong>L1 (Local):</strong> In-memory cache inside the application process (e.g., Caffeine for Java, Ristretto for Go, LRU dictionary for Python). Zero network latency.</li>
  <li><strong>L2 (Remote):</strong> Redis Cluster. Shared state.</li>
</ul>

<p><img src="https://blogs.buildwithmanish.com/assets/images/architecture_diagram-beyond-get-and-set-architecting-resilient-distributed-caching-patterns-for-microservices-7-2025-12-17-79471.png" alt="Architecture Diagram: Multi-Level Caching" /></p>

<h3 id="the-synchronization-problem">The Synchronization Problem</h3>
<p>If Instance A updates a user in DB and invalidates its L1 and the shared L2, Instance B still has the old user in <em>its</em> L1.</p>

<p><strong>The Solution: Pub/Sub Invalidation.</strong>
We use Redis Pub/Sub as a notification channel.</p>

<ol>
  <li><strong>Read Path:</strong> Check L1 -&gt; Check L2 -&gt; Check DB -&gt; Populate L2 -&gt; Populate L1.</li>
  <li><strong>Write Path:</strong>
    <ul>
      <li>Update DB.</li>
      <li>Delete L2 Key.</li>
      <li>Publish message to Redis Channel <code class="language-plaintext highlighter-rouge">cache-invalidation</code>: <code class="language-plaintext highlighter-rouge">{"key": "user:123"}</code>.</li>
      <li>All service instances subscribe to this channel. Upon receiving the message, they evict <code class="language-plaintext highlighter-rouge">user:123</code> from their local L1.</li>
    </ul>
  </li>
</ol>

<p><strong>Implementation Logic (Pseudo-code):</strong></p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Application Startup
</span><span class="k">def</span> <span class="nf">start_subscriber</span><span class="p">():</span>
    <span class="n">pubsub</span> <span class="o">=</span> <span class="n">redis_client</span><span class="p">.</span><span class="n">pubsub</span><span class="p">()</span>
    <span class="n">pubsub</span><span class="p">.</span><span class="n">subscribe</span><span class="p">(</span><span class="s">'cache-invalidation'</span><span class="p">)</span>
    <span class="k">for</span> <span class="n">message</span> <span class="ow">in</span> <span class="n">pubsub</span><span class="p">.</span><span class="n">listen</span><span class="p">():</span>
        <span class="k">if</span> <span class="n">message</span><span class="p">[</span><span class="s">'type'</span><span class="p">]</span> <span class="o">==</span> <span class="s">'message'</span><span class="p">:</span>
            <span class="n">key_to_delete</span> <span class="o">=</span> <span class="n">message</span><span class="p">[</span><span class="s">'data'</span><span class="p">]</span>
            <span class="n">local_cache</span><span class="p">.</span><span class="n">invalidate</span><span class="p">(</span><span class="n">key_to_delete</span><span class="p">)</span>

<span class="c1"># Write Operation
</span><span class="k">def</span> <span class="nf">update_data</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
    <span class="n">db</span><span class="p">.</span><span class="n">update</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
    <span class="n">redis_client</span><span class="p">.</span><span class="n">delete</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="c1"># Clear L2
</span>    <span class="c1"># Notify all L1s to clear
</span>    <span class="n">redis_client</span><span class="p">.</span><span class="n">publish</span><span class="p">(</span><span class="s">'cache-invalidation'</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span> 
</code></pre></div></div>

<p>This architecture provides the speed of local memory with the consistency controls of a distributed system.</p>

<hr />

<h2 id="conclusion">Conclusion</h2>

<p>Architecting distributed caching is an exercise in managing failure modes. It requires shifting your mindset from “Caching is a storage feature” to “Caching is a distributed systems problem.”</p>

<p>To build resilient systems:</p>
<ol>
  <li><strong>Assume failure:</strong> Design for the empty cache scenario (Avalanche protection).</li>
  <li><strong>Defend the DB:</strong> Use Bloom filters and locking to prevent Thundering Herds.</li>
  <li><strong>Respect Physics:</strong> Understand that strong consistency across distributed nodes incurs a latency penalty.</li>
  <li><strong>Layer intelligently:</strong> Use L1/L2 caching when network hops become the bottleneck, but automate the invalidation.</li>
</ol>

<p>The goal isn’t just a high cache hit ratio; it’s a system that degrades gracefully, scales elastically, and remains consistent enough for the business logic to hold true.</p>]]></content><author><name>Manish Tiwari</name></author><category term="deep-dive" /><category term="system-design" /><category term="redis" /><category term="distributed-systems" /><category term="microservices" /><summary type="html"><![CDATA[]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" /><media:content medium="image" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Beyond the Happy Path: Architecting Fault-Tolerant Recurring Payments with Razorpay</title><link href="https://blogs.buildwithmanish.com/technical/deep-dive/2025/12/17/beyond-the-happy-path-architecting-faulttolerant-recurring-payments-with-razorpay.html" rel="alternate" type="text/html" title="Beyond the Happy Path: Architecting Fault-Tolerant Recurring Payments with Razorpay" /><published>2025-12-17T00:00:00+00:00</published><updated>2025-12-17T00:00:00+00:00</updated><id>https://blogs.buildwithmanish.com/technical/deep-dive/2025/12/17/beyond-the-happy-path-architecting-faulttolerant-recurring-payments-with-razorpay</id><content type="html" xml:base="https://blogs.buildwithmanish.com/technical/deep-dive/2025/12/17/beyond-the-happy-path-architecting-faulttolerant-recurring-payments-with-razorpay.html"><![CDATA[<p><img src="https://blogs.buildwithmanish.com/assets/images/hero-beyond-the-happy-path-architecting-faulttolerant-recurring-payments-with-razorpay-0-2025-12-17-80735.png" alt="Hero Image" /></p>

<h2 id="the-illusion-of-the-happy-path">The Illusion of the “Happy Path”</h2>

<p>In the world of distributed systems, the “happy path” is a sedative. It lulls engineers into a false sense of security where networks never partition, latency is zero, and customers always have sufficient funds in their bank accounts. When building one-off payment flows, you might get away with a simple request-response model. But in the domain of <strong>recurring payments and subscriptions</strong>, the happy path is the exception, not the rule.</p>

<p>A subscription system is a long-running process. It is a temporal contract between your architecture and a user’s financial state. Over a 12-month period, cards expire, banks flag transactions as fraudulent, webhooks get dropped, and users change their minds.</p>

<p>If you are architecting a subscription engine on top of Razorpay for a high-scale FinTech or SaaS product, treating payment failure as an edge case is a catastrophic design flaw. It must be a core state of your system.</p>

<p>This article is not a “getting started” guide. We are diving deep into the architectural patterns required to build a fault-tolerant, resilient subscription system. We will cover state machine recovery, idempotent webhook ingestion, complex proration logic, and the “Dunning” architecture that prevents involuntary churn.</p>

<hr />

<h2 id="1-the-subscription-state-machine-purgatory-vs-death">1. The Subscription State Machine: Purgatory vs. Death</h2>

<p>The most common misconception I see in backend implementations is treating the Razorpay subscription status as a binary: <code class="language-plaintext highlighter-rouge">active</code> or <code class="language-plaintext highlighter-rouge">cancelled</code>. The reality is nuanced. The lifecycle of a subscription involves critical intermediate states that dictate whether your system should retry a charge or revoke access.</p>

<p>The two states that cause the most confusion-and revenue loss-are <code class="language-plaintext highlighter-rouge">pending</code> and <code class="language-plaintext highlighter-rouge">halted</code>.</p>

<h3 id="the-critical-distinction">The Critical Distinction</h3>
<ul>
  <li><strong>Active:</strong> The happy path. Payment was successful, access is granted.</li>
  <li><strong>Pending (Purgatory):</strong> Razorpay attempted a charge, but it failed (e.g., insufficient funds). Crucially, <strong>retries are active</strong>. Razorpay’s internal scheduler will attempt to charge the card again based on your retry configuration (usually T+1, T+3 days).
    <ul>
      <li><em>Your Action:</em> Do <strong>not</strong> cancel access immediately. This is often a temporary glitch.</li>
    </ul>
  </li>
  <li><strong>Halted (The Cliff):</strong> All retry attempts have been exhausted. Razorpay has stopped trying to charge the card.
    <ul>
      <li><em>Your Action:</em> This is where most implementations fail. They treat <code class="language-plaintext highlighter-rouge">halted</code> as <code class="language-plaintext highlighter-rouge">cancelled</code>.</li>
    </ul>
  </li>
</ul>

<p><img src="https://blogs.buildwithmanish.com/assets/images/technical_diagram-beyond-the-happy-path-architecting-faulttolerant-recurring-payments-with-razorpay-1-2025-12-17-10252.png" alt="Subscription State Machine Diagram" /></p>

<h3 id="the-rescue-operation-reviving-halted-subscriptions">The “Rescue” Operation: Reviving Halted Subscriptions</h3>

<p>When a subscription hits the <code class="language-plaintext highlighter-rouge">halted</code> state, the intuitive UI flow is to ask the user to “Update Payment Method.” However, purely updating the card doesn’t always immediately trigger a charge or move the state back to <code class="language-plaintext highlighter-rouge">active</code> instantly.</p>

<p>A superior architectural pattern is to <strong>programmatically rescue</strong> the subscription using the Invoice API.</p>

<p>Instead of creating a new subscription (which breaks historical data and metrics), you can issue an ad-hoc invoice linked to the subscription. When this invoice is paid, Razorpay automatically transitions the subscription from <code class="language-plaintext highlighter-rouge">halted</code> back to <code class="language-plaintext highlighter-rouge">active</code>.</p>

<p><strong>The Implementation Strategy:</strong></p>
<ol>
  <li>Listen for the <code class="language-plaintext highlighter-rouge">subscription.halted</code> webhook.</li>
  <li>In your frontend, present a “Pay Outstanding Balance” flow.</li>
  <li>Backend generates a manual invoice for the pending amount.</li>
  <li>User pays the invoice -&gt; Subscription auto-heals.</li>
</ol>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">razorpay</span>
<span class="kn">from</span> <span class="nn">my_app.exceptions</span> <span class="kn">import</span> <span class="n">PaymentError</span>

<span class="n">client</span> <span class="o">=</span> <span class="n">razorpay</span><span class="p">.</span><span class="n">Client</span><span class="p">(</span><span class="n">auth</span><span class="o">=</span><span class="p">(</span><span class="s">"KEY_ID"</span><span class="p">,</span> <span class="s">"KEY_SECRET"</span><span class="p">))</span>

<span class="k">def</span> <span class="nf">rescue_halted_subscription</span><span class="p">(</span><span class="n">subscription_id</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">user_id</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
    <span class="s">"""
    Rescues a halted subscription by creating a manual invoice
    for the failed billing cycle.
    """</span>
    <span class="k">try</span><span class="p">:</span>
        <span class="c1"># 1. Fetch Subscription to get details
</span>        <span class="n">sub</span> <span class="o">=</span> <span class="n">client</span><span class="p">.</span><span class="n">subscription</span><span class="p">.</span><span class="n">fetch</span><span class="p">(</span><span class="n">subscription_id</span><span class="p">)</span>
        
        <span class="k">if</span> <span class="n">sub</span><span class="p">[</span><span class="s">'status'</span><span class="p">]</span> <span class="o">!=</span> <span class="s">'halted'</span><span class="p">:</span>
            <span class="k">raise</span> <span class="n">PaymentError</span><span class="p">(</span><span class="s">"Subscription is not in halted state."</span><span class="p">)</span>

        <span class="c1"># 2. Identify the unpaid amount (usually the plan amount)
</span>        <span class="c1"># In a real scenario, calculate strictly based on outstanding cycles
</span>        <span class="n">amount_due</span> <span class="o">=</span> <span class="n">sub</span><span class="p">[</span><span class="s">'plan_details'</span><span class="p">][</span><span class="s">'amount'</span><span class="p">]</span> 
        
        <span class="c1"># 3. Create an Invoice linked to this subscription
</span>        <span class="c1"># This tells Razorpay: "If this is paid, fix the sub."
</span>        <span class="n">invoice_payload</span> <span class="o">=</span> <span class="p">{</span>
            <span class="s">"type"</span><span class="p">:</span> <span class="s">"invoice"</span><span class="p">,</span>
            <span class="s">"customer_id"</span><span class="p">:</span> <span class="n">sub</span><span class="p">[</span><span class="s">'customer_id'</span><span class="p">],</span>
            <span class="s">"line_items"</span><span class="p">:</span> <span class="p">[</span>
                <span class="p">{</span>
                    <span class="s">"name"</span><span class="p">:</span> <span class="s">"Outstanding Balance Recovery"</span><span class="p">,</span>
                    <span class="s">"description"</span><span class="p">:</span> <span class="s">"Payment for failed renewal"</span><span class="p">,</span>
                    <span class="s">"amount"</span><span class="p">:</span> <span class="n">amount_due</span><span class="p">,</span>
                    <span class="s">"currency"</span><span class="p">:</span> <span class="s">"INR"</span>
                <span class="p">}</span>
            <span class="p">],</span>
            <span class="s">"subscription_id"</span><span class="p">:</span> <span class="n">subscription_id</span><span class="p">,</span> <span class="c1"># CRITICAL LINK
</span>            <span class="s">"sms_notify"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
            <span class="s">"email_notify"</span><span class="p">:</span> <span class="mi">1</span>
        <span class="p">}</span>
        
        <span class="n">invoice</span> <span class="o">=</span> <span class="n">client</span><span class="p">.</span><span class="n">invoice</span><span class="p">.</span><span class="n">create</span><span class="p">(</span><span class="n">invoice_payload</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">invoice</span> <span class="c1"># Return invoice URL to frontend
</span>
    <span class="k">except</span> <span class="nb">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
        <span class="c1"># Log critical failure
</span>        <span class="k">raise</span> <span class="n">PaymentError</span><span class="p">(</span><span class="sa">f</span><span class="s">"Failed to create rescue invoice: </span><span class="si">{</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
</code></pre></div></div>

<p>By linking the <code class="language-plaintext highlighter-rouge">subscription_id</code> in the invoice creation, you maintain the continuity of the subscription entity, preserving the <code class="language-plaintext highlighter-rouge">billing_cycle</code> count and lifetime value (LTV) data.</p>

<hr />

<h2 id="2-webhook-architecture--idempotency">2. Webhook Architecture &amp; Idempotency</h2>

<p>Webhooks are the heartbeat of a subscription system. However, webhooks are asynchronous, unreliable, and prone to duplication. If your architecture assumes that webhooks arrive exactly once and in the correct order, you will eventually double-credit a user or disable an account that just paid.</p>

<h3 id="the-challenges">The Challenges</h3>
<ol>
  <li><strong>Out-of-Order Delivery:</strong> You might receive <code class="language-plaintext highlighter-rouge">subscription.charged</code> before <code class="language-plaintext highlighter-rouge">subscription.pending</code>.</li>
  <li><strong>Duplication:</strong> Razorpay (and most gateways) operates on an “at-least-once” delivery guarantee. If your server times out or responds with a 500 error, they will send the event again.</li>
  <li><strong>Security:</strong> An attacker could spoof a webhook to grant themselves free access.</li>
</ol>

<p><img src="https://blogs.buildwithmanish.com/assets/images/architecture_diagram-beyond-the-happy-path-architecting-faulttolerant-recurring-payments-with-razorpay-2-2025-12-17-98043.png" alt="Webhook Architecture Diagram" /></p>

<h3 id="security-hmac-verification">Security: HMAC Verification</h3>

<p>Before looking at the payload, you <strong>must</strong> verify the signature. This is not optional. It prevents Man-in-the-Middle attacks and spoofing.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">hmac</span>
<span class="kn">import</span> <span class="nn">hashlib</span>
<span class="kn">import</span> <span class="nn">json</span>

<span class="k">def</span> <span class="nf">verify_webhook_signature</span><span class="p">(</span><span class="n">request_body</span><span class="p">:</span> <span class="nb">bytes</span><span class="p">,</span> <span class="n">signature</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">secret</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
    <span class="s">"""
    Verifies the X-Razorpay-Signature header.
    
    Args:
        request_body: Raw bytes of the POST body
        signature: The hex string from X-Razorpay-Signature header
        secret: Your webhook secret configured in Dashboard
    """</span>
    <span class="n">generated_signature</span> <span class="o">=</span> <span class="n">hmac</span><span class="p">.</span><span class="n">new</span><span class="p">(</span>
        <span class="n">key</span><span class="o">=</span><span class="nb">bytes</span><span class="p">(</span><span class="n">secret</span><span class="p">,</span> <span class="s">'utf-8'</span><span class="p">),</span>
        <span class="n">msg</span><span class="o">=</span><span class="n">request_body</span><span class="p">,</span>
        <span class="n">digestmod</span><span class="o">=</span><span class="n">hashlib</span><span class="p">.</span><span class="n">sha256</span>
    <span class="p">).</span><span class="n">hexdigest</span><span class="p">()</span>
    
    <span class="c1"># hmac.compare_digest prevents timing attacks
</span>    <span class="k">return</span> <span class="n">hmac</span><span class="p">.</span><span class="n">compare_digest</span><span class="p">(</span><span class="n">generated_signature</span><span class="p">,</span> <span class="n">signature</span><span class="p">)</span>
</code></pre></div></div>

<h3 id="idempotency-handling-duplicates">Idempotency: Handling Duplicates</h3>

<p>Idempotency ensures that processing the same event multiple times yields the same result as processing it once.</p>

<p><strong>The Anti-Pattern:</strong> checking your database for <code class="language-plaintext highlighter-rouge">if transaction_exists: return</code>.
<strong>The Better Pattern:</strong> Using a distributed lock (Redis) combined with an Event Log.</p>

<p>You should use the <code class="language-plaintext highlighter-rouge">x-razorpay-event-id</code> header or the <code class="language-plaintext highlighter-rouge">payload.id</code> to uniquely identify the event.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/code_flow-beyond-the-happy-path-architecting-faulttolerant-recurring-payments-with-razorpay-3-2025-12-17-65241.png" alt="Code Flow Diagram" /></p>

<p><strong>Implementation with Redis (Python/Pseudo-code):</strong></p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">redis</span>
<span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">transaction</span>

<span class="n">r</span> <span class="o">=</span> <span class="n">redis</span><span class="p">.</span><span class="n">Redis</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="s">'localhost'</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="mi">6379</span><span class="p">,</span> <span class="n">db</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>

<span class="k">def</span> <span class="nf">process_webhook</span><span class="p">(</span><span class="n">event_id</span><span class="p">,</span> <span class="n">payload</span><span class="p">):</span>
    <span class="c1"># 1. Atomic Check via Redis
</span>    <span class="c1"># SETNX sets the key only if it doesn't exist. 
</span>    <span class="c1"># We set an expiry (TTL) of 24 hours to prevent infinite growth.
</span>    <span class="n">is_new_event</span> <span class="o">=</span> <span class="n">r</span><span class="p">.</span><span class="nb">set</span><span class="p">(</span><span class="sa">f</span><span class="s">"webhook_event:</span><span class="si">{</span><span class="n">event_id</span><span class="si">}</span><span class="s">"</span><span class="p">,</span> <span class="s">"processing"</span><span class="p">,</span> <span class="n">ex</span><span class="o">=</span><span class="mi">86400</span><span class="p">,</span> <span class="n">nx</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
    
    <span class="k">if</span> <span class="ow">not</span> <span class="n">is_new_event</span><span class="p">:</span>
        <span class="c1"># We have seen this event. Return 200 to stop Razorpay retries.
</span>        <span class="k">return</span> <span class="mi">200</span><span class="p">,</span> <span class="s">"Event already processed"</span>

    <span class="k">try</span><span class="p">:</span>
        <span class="c1"># 2. Database Transaction
</span>        <span class="k">with</span> <span class="n">transaction</span><span class="p">.</span><span class="n">atomic</span><span class="p">():</span>
            <span class="c1"># ... process business logic (provision access, record payment) ...
</span>            <span class="k">pass</span>
            
        <span class="c1"># 3. Mark as success in Redis (Optional, for debugging)
</span>        <span class="n">r</span><span class="p">.</span><span class="nb">set</span><span class="p">(</span><span class="sa">f</span><span class="s">"webhook_event:</span><span class="si">{</span><span class="n">event_id</span><span class="si">}</span><span class="s">"</span><span class="p">,</span> <span class="s">"success"</span><span class="p">,</span> <span class="n">xx</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
        <span class="k">return</span> <span class="mi">200</span><span class="p">,</span> <span class="s">"OK"</span>

    <span class="k">except</span> <span class="nb">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
        <span class="c1"># 4. Critical: If processing fails, DELETE the key so we can retry
</span>        <span class="n">r</span><span class="p">.</span><span class="n">delete</span><span class="p">(</span><span class="sa">f</span><span class="s">"webhook_event:</span><span class="si">{</span><span class="n">event_id</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
        <span class="k">raise</span> <span class="n">e</span> <span class="c1"># Let the server 500 so Razorpay retries
</span></code></pre></div></div>

<p>This logic ensures that even if two requests with the same ID hit your servers in parallel (race condition), only one will acquire the Redis lock and process the logic.</p>

<hr />

<h2 id="3-handling-upgrades--downgrades-proration">3. Handling Upgrades &amp; Downgrades (Proration)</h2>

<p>Moving a user from a “Basic” plan (₹500/mo) to a “Pro” plan (₹1000/mo) in the middle of a billing cycle is mathematically complex. You cannot simply switch the <code class="language-plaintext highlighter-rouge">plan_id</code>. You must account for the money already paid for the unused days of the Basic plan.</p>

<h3 id="schedule_change_at-the-lever-of-control"><code class="language-plaintext highlighter-rouge">schedule_change_at</code>: The Lever of Control</h3>

<p>Razorpay provides the <code class="language-plaintext highlighter-rouge">schedule_change_at</code> parameter in the subscription update API. You have two choices:</p>

<ol>
  <li><strong><code class="language-plaintext highlighter-rouge">cycle_end</code>:</strong> The upgrade happens when the current month finishes. Easy logic, but delayed revenue and user gratification.</li>
  <li><strong><code class="language-plaintext highlighter-rouge">now</code>:</strong> The upgrade happens immediately. This triggers <strong>Proration</strong>.</li>
</ol>

<p><img src="https://blogs.buildwithmanish.com/assets/images/comparison_chart-beyond-the-happy-path-architecting-faulttolerant-recurring-payments-with-razorpay-4-2025-12-17-40674.png" alt="Comparison Chart" /></p>

<h3 id="the-proration-flow">The Proration Flow</h3>

<p>When you update a subscription with <code class="language-plaintext highlighter-rouge">schedule_change_at="now"</code> and <code class="language-plaintext highlighter-rouge">proration_behavior="create_prorations"</code>, Razorpay does the following:</p>

<ol>
  <li>Calculates the unused amount of the current plan (Credits).</li>
  <li>Calculates the cost of the new plan for the remaining days of the cycle (Debits).</li>
  <li>Creates an immediate invoice for the difference: <code class="language-plaintext highlighter-rouge">(New Plan Cost) - (Old Plan Unused Credit)</code>.</li>
</ol>

<p><strong>Architectural Warning:</strong> Your local database must stay in sync with this logic. Do not try to calculate proration yourself. Rely on the webhook <code class="language-plaintext highlighter-rouge">subscription.charged</code> that results from the upgrade to update your local “Plan Tier” field.</p>

<p><strong>Code Example: Performing an Immediate Upgrade</strong></p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">upgrade_subscription</span><span class="p">(</span><span class="n">sub_id</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">new_plan_id</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
    <span class="k">try</span><span class="p">:</span>
        <span class="c1"># Razorpay handles the math (credits/debits)
</span>        <span class="n">resp</span> <span class="o">=</span> <span class="n">client</span><span class="p">.</span><span class="n">subscription</span><span class="p">.</span><span class="n">edit</span><span class="p">(</span><span class="n">sub_id</span><span class="p">,</span> <span class="p">{</span>
            <span class="s">"plan_id"</span><span class="p">:</span> <span class="n">new_plan_id</span><span class="p">,</span>
            <span class="s">"schedule_change_at"</span><span class="p">:</span> <span class="s">"now"</span><span class="p">,</span> 
            <span class="s">"proration_behavior"</span><span class="p">:</span> <span class="s">"create_prorations"</span><span class="p">,</span>
            <span class="s">"quantity"</span><span class="p">:</span> <span class="mi">1</span>
        <span class="p">})</span>
        
        <span class="c1"># NOTE: Do NOT update user access to 'Pro' here immediately.
</span>        <span class="c1"># Wait for the successful payment webhook (subscription.charged)
</span>        <span class="c1"># generated by this change. 
</span>        <span class="c1"># If the proration charge fails, the user stays on the old plan.
</span>        
        <span class="k">return</span> <span class="n">resp</span>
    <span class="k">except</span> <span class="nb">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
        <span class="c1"># Handle scenarios like "Upgrade not possible on halted sub"
</span>        <span class="n">handle_error</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
</code></pre></div></div>

<p><strong>Key Takeaway:</strong> Never update user permissions synchronously during an upgrade request. The proration charge might fail (card declined). Always drive permission updates via webhooks.</p>

<hr />

<h2 id="4-resiliency-patterns-the-dunning-process">4. Resiliency Patterns: The “Dunning” Process</h2>

<p>Dunning is the process of communicating with customers to ensure the collection of accounts receivable. In SaaS, it means “How do we handle failed renewal charges without losing the customer?”</p>

<p>A naive implementation listens for <code class="language-plaintext highlighter-rouge">subscription.charged</code> (failed) and immediately sets <code class="language-plaintext highlighter-rouge">user.is_active = False</code>. This leads to high involuntary churn. A user’s card might fail due to a bank server timeout, and you’ve just locked them out of their account.</p>

<h3 id="the-grace-period-architecture">The Grace Period Architecture</h3>

<p>You need to decouple <strong>Payment Status</strong> from <strong>Access Status</strong>.</p>

<ol>
  <li><strong>Payment Status:</strong> Tracks the Razorpay state (<code class="language-plaintext highlighter-rouge">active</code>, <code class="language-plaintext highlighter-rouge">pending</code>, <code class="language-plaintext highlighter-rouge">halted</code>).</li>
  <li><strong>Access Status:</strong> Tracks whether the user can use the app (<code class="language-plaintext highlighter-rouge">granted</code>, <code class="language-plaintext highlighter-rouge">revoked</code>).</li>
</ol>

<p>When a payment fails (<code class="language-plaintext highlighter-rouge">subscription.pending</code>), the user enters a <strong>Grace Period</strong>.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/technical_diagram-beyond-the-happy-path-architecting-faulttolerant-recurring-payments-with-razorpay-5-2025-12-17-33473.png" alt="Technical Diagram - Dunning" /></p>

<p><strong>The Logic Flow:</strong></p>

<ol>
  <li><strong>T+0 (Due Date):</strong> Razorpay attempts charge. Fails.
    <ul>
      <li>Webhook: <code class="language-plaintext highlighter-rouge">subscription.pending</code>.</li>
      <li>System Action: Send email “Payment Failed”. <strong>Keep access active.</strong></li>
    </ul>
  </li>
  <li><strong>T+1 to T+3:</strong> Razorpay auto-retries.
    <ul>
      <li>If Success (<code class="language-plaintext highlighter-rouge">subscription.charged</code>): Send “Thank You” email. Extend validity.</li>
      <li>If Fail: Do nothing. Keep access active.</li>
    </ul>
  </li>
  <li><strong>T+5 (Max Retries Exhausted):</strong> Razorpay gives up.
    <ul>
      <li>Webhook: <code class="language-plaintext highlighter-rouge">subscription.halted</code>.</li>
      <li>System Action: <strong>Enter Grace Period (Internal).</strong> Give them 48 more hours to fix it manually.</li>
    </ul>
  </li>
  <li><strong>T+7 (Hard Cutoff):</strong>
    <ul>
      <li>Cron Job checks <code class="language-plaintext highlighter-rouge">subscription.halted</code> + <code class="language-plaintext highlighter-rouge">grace_period_expired</code>.</li>
      <li>System Action: Revoke Access.</li>
    </ul>
  </li>
</ol>

<h3 id="configuration">Configuration</h3>

<p>In your Database, your Subscription model should look something like this:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Subscription</span><span class="p">(</span><span class="n">models</span><span class="p">.</span><span class="n">Model</span><span class="p">):</span>
    <span class="n">razorpay_id</span> <span class="o">=</span> <span class="n">models</span><span class="p">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">50</span><span class="p">)</span>
    <span class="n">status</span> <span class="o">=</span> <span class="n">models</span><span class="p">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">choices</span><span class="o">=</span><span class="p">[</span><span class="s">'active'</span><span class="p">,</span> <span class="s">'pending'</span><span class="p">,</span> <span class="s">'halted'</span><span class="p">,</span> <span class="s">'cancelled'</span><span class="p">])</span>
    <span class="n">current_period_end</span> <span class="o">=</span> <span class="n">models</span><span class="p">.</span><span class="n">DateTimeField</span><span class="p">()</span>
    
    <span class="c1"># The field that actually controls login/feature access
</span>    <span class="n">access_valid_until</span> <span class="o">=</span> <span class="n">models</span><span class="p">.</span><span class="n">DateTimeField</span><span class="p">()</span> 
    
    <span class="k">def</span> <span class="nf">handle_payment_failure</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="s">"""
        Called on subscription.pending.
        We don't revoke access yet, but we might flag the UI.
        """</span>
        <span class="bp">self</span><span class="p">.</span><span class="n">ui_warning_flag</span> <span class="o">=</span> <span class="bp">True</span>
        <span class="bp">self</span><span class="p">.</span><span class="n">save</span><span class="p">()</span>

    <span class="k">def</span> <span class="nf">handle_halted</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="s">"""
        Called on subscription.halted.
        Set a hard stop date, perhaps 2 days from now.
        """</span>
        <span class="bp">self</span><span class="p">.</span><span class="n">access_valid_until</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">.</span><span class="n">now</span><span class="p">()</span> <span class="o">+</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
        <span class="bp">self</span><span class="p">.</span><span class="n">save</span><span class="p">()</span>
</code></pre></div></div>

<p>This decoupled architecture ensures that temporary banking glitches do not result in a degraded user experience.</p>

<hr />

<h2 id="5-system-architecture--scaling">5. System Architecture &amp; Scaling</h2>

<p>To tie this all together, your backend architecture needs to be robust. You cannot rely on a single monolithic web server to handle user traffic and webhook processing simultaneously.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/architecture_diagram-beyond-the-happy-path-architecting-faulttolerant-recurring-payments-with-razorpay-6-2025-12-17-68307.png" alt="Architecture Diagram" /></p>

<h3 id="the-recommended-stack">The Recommended Stack</h3>

<ol>
  <li><strong>Webhook Ingestion Layer:</strong> A lightweight API endpoint that does nothing but signature verification and pushing the payload to a Message Queue (SQS, RabbitMQ, Kafka). This ensures you respond <code class="language-plaintext highlighter-rouge">200 OK</code> to Razorpay instantly (under 2 seconds is the requirement).</li>
  <li><strong>Worker Fleet:</strong> Asynchronous workers consume events from the queue. This is where the heavy lifting happens (Database writes, Email triggers, Redis locking).</li>
  <li><strong>Reconciliation Cron:</strong> Even with the best webhook architecture, events get dropped. Run a nightly cron job that fetches all <code class="language-plaintext highlighter-rouge">active</code> local subscriptions and compares their state with the Razorpay API. If a mismatch is found (Local: <code class="language-plaintext highlighter-rouge">active</code>, Remote: <code class="language-plaintext highlighter-rouge">halted</code>), auto-correct the local state.</li>
</ol>

<h2 id="conclusion">Conclusion</h2>

<p>Building a recurring payment system is 20% integrating APIs and 80% handling failure states. The difference between a junior and senior implementation lies in how the system behaves when things go wrong.</p>

<p>By implementing a robust State Machine that understands the difference between <code class="language-plaintext highlighter-rouge">pending</code> and <code class="language-plaintext highlighter-rouge">halted</code>, securing your webhooks with HMAC and Redis-based idempotency, and architecting a compassionate Dunning process, you turn payment failures from a churn event into a recovery opportunity.</p>

<p>The goal isn’t just to process payments; it’s to build a system that maintains trust with your users, even when their bank declines the card.</p>]]></content><author><name>Manish Tiwari</name></author><category term="technical" /><category term="deep-dive" /><summary type="html"><![CDATA[]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" /><media:content medium="image" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Building a Virtual Data Analyst: Architecting Agentic AI Workflows over Power BI</title><link href="https://blogs.buildwithmanish.com/deep-dive/gen-ai/power-bi/agentic-workflows/python/2025/12/17/building-a-virtual-data-analyst-architecting-agentic-ai-workflows-over-power-bi.html" rel="alternate" type="text/html" title="Building a Virtual Data Analyst: Architecting Agentic AI Workflows over Power BI" /><published>2025-12-17T00:00:00+00:00</published><updated>2025-12-17T00:00:00+00:00</updated><id>https://blogs.buildwithmanish.com/deep-dive/gen-ai/power-bi/agentic-workflows/python/2025/12/17/building-a-virtual-data-analyst-architecting-agentic-ai-workflows-over-power-bi</id><content type="html" xml:base="https://blogs.buildwithmanish.com/deep-dive/gen-ai/power-bi/agentic-workflows/python/2025/12/17/building-a-virtual-data-analyst-architecting-agentic-ai-workflows-over-power-bi.html"><![CDATA[<p><img src="https://blogs.buildwithmanish.com/assets/images/hero-building-a-virtual-data-analyst-architecting-agentic-ai-workflows-over-power-bi-0-2025-12-17-46659.png" alt="Building a Virtual Data Analyst" /></p>

<p>In the rush to adopt Generative AI, organizations have flooded their backlogs with “Chat with your Data” initiatives. Most of these projects follow a predictable, often disappointing, pattern: a text-to-SQL layer slapped on top of a raw data warehouse. While this works for simple questions (“How many widgets did we sell?”), it fails catastrophic when applied to enterprise analytical questions (“Why is Q4 margin degrading despite higher volume?”).</p>

<p>The failure stems from a fundamental architectural oversight: <strong>Enterprise business logic does not live in the raw database tables; it lives in the Semantic Model.</strong></p>

<p>In the Microsoft ecosystem, that logic is encapsulated in Power BI datasets-specifically in DAX measures, calculated columns, and complex relationships. To build a true “Virtual Data Analyst,” we cannot simply ask an LLM to write SQL. We must architect an <strong>Agentic Workflow</strong> capable of interacting with the Power BI Service via XMLA endpoints and REST APIs, mimicking the cognitive processes of a human analyst: introspection, planning, execution, and synthesis.</p>

<p>This deep dive explores how to build that architecture using Python, LangChain, and the Power BI REST API.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/architecture_diagram-building-a-virtual-data-analyst-architecting-agentic-ai-workflows-over-power-bi-1-2025-12-17-87202.png" alt="Architecture Diagram" /></p>

<h2 id="the-core-problem-why-text-to-sql-fails-power-bi">The Core Problem: Why Text-to-SQL Fails Power BI</h2>
<p>Before writing code, we must understand the semantic gap. A raw SQL database contains <code class="language-plaintext highlighter-rouge">Sales_Amount</code> and <code class="language-plaintext highlighter-rouge">Cost_Amount</code>. However, the concept of <code class="language-plaintext highlighter-rouge">Gross Margin %</code> isn’t a column; it’s a DAX measure defined as:</p>

<pre><code class="language-dax">Gross Margin % = DIVIDE([Total Sales] - [Total Cost], [Total Sales], 0)
</code></pre>

<p>Furthermore, this measure might rely on Time Intelligence functions (<code class="language-plaintext highlighter-rouge">SAMEPERIODLASTYEAR</code>) or complex filter contexts (<code class="language-plaintext highlighter-rouge">CALCULATE</code>).</p>

<p>If you point a standard Text-to-SQL agent at the underlying SQL database, it will hallucinate the math, ignore the verified logic defined in Power BI, and return numbers that don’t match your executive dashboard. To fix this, our agent must speak <strong>DAX</strong> (Data Analysis Expressions) and query the <strong>Semantic Model</strong> directly.</p>

<h2 id="architectural-pattern-the-plan-and-execute-agent">Architectural Pattern: The Plan-and-Execute Agent</h2>
<p>We will move beyond simple RAG (Retrieval Augmented Generation) to a “Plan-and-Execute” agentic pattern. A human analyst doesn’t just run a query immediately. They:</p>
<ol>
  <li><strong>Introspect:</strong> Look at the data model to see what fields and measures exist.</li>
  <li><strong>Plan:</strong> Break the business question into logical steps.</li>
  <li><strong>Execute:</strong> Write and run the query.</li>
  <li><strong>Verify/Refine:</strong> Check if the results make sense.</li>
  <li><strong>Synthesize:</strong> Write the email/report.</li>
</ol>

<p>Our AI agent will replicate this loop.</p>

<h3 id="the-component-stack">The Component Stack</h3>
<ol>
  <li><strong>Orchestrator:</strong> LangChain or LangGraph (Python).</li>
  <li><strong>LLM:</strong> GPT-4o or Claude 3.5 Sonnet (Models with high coding capability are essential for DAX).</li>
  <li><strong>Interface Layer:</strong> Power BI REST API (<code class="language-plaintext highlighter-rouge">executeQueries</code> endpoint) or XMLA endpoint via <code class="language-plaintext highlighter-rouge">pythonnet</code>/<code class="language-plaintext highlighter-rouge">pyadomd</code>.</li>
  <li><strong>Tools:</strong> Custom Python functions for Schema Introspection and DAX Execution.</li>
</ol>

<p><img src="https://blogs.buildwithmanish.com/assets/images/code_flow-building-a-virtual-data-analyst-architecting-agentic-ai-workflows-over-power-bi-2-2025-12-17-20014.png" alt="Code Flow Diagram" /></p>

<h2 id="phase-1-introspection-the-metadata-layer">Phase 1: Introspection (The Metadata Layer)</h2>
<p>An LLM cannot query a dataset it cannot see. We need a tool that allows the agent to read the “map” of the data. We use Dynamic Management Views (DMVs) over the XMLA endpoint or the Power BI REST API to fetch metadata.</p>

<p>We need to extract:</p>
<ul>
  <li>Table Names</li>
  <li>Column Names</li>
  <li><strong>Measure Names and their DAX Expressions</strong> (Crucial for context)</li>
  <li>Relationships</li>
</ul>

<h3 id="implementation-the-schema-fetcher">Implementation: The Schema Fetcher</h3>
<p>Here is how we implement a robust schema fetcher using the Power BI REST API. This tool will be available to our Agent.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">import</span> <span class="nn">json</span>

<span class="k">class</span> <span class="nc">PowerBIMetadataService</span><span class="p">:</span>
    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tenant_id</span><span class="p">,</span> <span class="n">client_id</span><span class="p">,</span> <span class="n">client_secret</span><span class="p">,</span> <span class="n">workspace_id</span><span class="p">,</span> <span class="n">dataset_id</span><span class="p">):</span>
        <span class="bp">self</span><span class="p">.</span><span class="n">base_url</span> <span class="o">=</span> <span class="s">"https://api.powerbi.com/v1.0/myorg"</span>
        <span class="bp">self</span><span class="p">.</span><span class="n">workspace_id</span> <span class="o">=</span> <span class="n">workspace_id</span>
        <span class="bp">self</span><span class="p">.</span><span class="n">dataset_id</span> <span class="o">=</span> <span class="n">dataset_id</span>
        <span class="bp">self</span><span class="p">.</span><span class="n">token</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">_get_aad_token</span><span class="p">(</span><span class="n">tenant_id</span><span class="p">,</span> <span class="n">client_id</span><span class="p">,</span> <span class="n">client_secret</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">_get_aad_token</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tenant_id</span><span class="p">,</span> <span class="n">client_id</span><span class="p">,</span> <span class="n">client_secret</span><span class="p">):</span>
        <span class="n">url</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"https://login.microsoftonline.com/</span><span class="si">{</span><span class="n">tenant_id</span><span class="si">}</span><span class="s">/oauth2/v2.0/token"</span>
        <span class="n">payload</span> <span class="o">=</span> <span class="p">{</span>
            <span class="s">'grant_type'</span><span class="p">:</span> <span class="s">'client_credentials'</span><span class="p">,</span>
            <span class="s">'client_id'</span><span class="p">:</span> <span class="n">client_id</span><span class="p">,</span>
            <span class="s">'client_secret'</span><span class="p">:</span> <span class="n">client_secret</span><span class="p">,</span>
            <span class="s">'scope'</span><span class="p">:</span> <span class="s">'https://analysis.windows.net/powerbi/api/.default'</span>
        <span class="p">}</span>
        <span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">post</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">payload</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">response</span><span class="p">.</span><span class="n">json</span><span class="p">().</span><span class="n">get</span><span class="p">(</span><span class="s">'access_token'</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">get_semantic_schema</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="s">"""
        Executes a DAX query to retrieve DMVs (Dynamic Management Views)
        simulating schema introspection.
        """</span>
        <span class="c1"># Query to get Tables and Columns
</span>        <span class="n">dax_query</span> <span class="o">=</span> <span class="s">"""
        EVALUATE
        SELECTCOLUMNS(
            INFO.COLUMNS,
            "Table", [TableID],
            "Column", [ExplicitName],
            "Description", [Description]
        )
        """</span>
        
        <span class="c1"># Query to get Measures (The most important part)
</span>        <span class="n">measures_query</span> <span class="o">=</span> <span class="s">"""
        EVALUATE
        SELECTCOLUMNS(
            INFO.MEASURES,
            "Table", [TableID],
            "Measure", [Name],
            "Expression", [Expression]
        )
        """</span>
        
        <span class="n">columns</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">execute_dax</span><span class="p">(</span><span class="n">dax_query</span><span class="p">)</span>
        <span class="n">measures</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">execute_dax</span><span class="p">(</span><span class="n">measures_query</span><span class="p">)</span>
        
        <span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">_format_schema_for_llm</span><span class="p">(</span><span class="n">columns</span><span class="p">,</span> <span class="n">measures</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">execute_dax</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dax_query</span><span class="p">):</span>
        <span class="n">url</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">base_url</span><span class="si">}</span><span class="s">/groups/</span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">workspace_id</span><span class="si">}</span><span class="s">/datasets/</span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">dataset_id</span><span class="si">}</span><span class="s">/executeQueries"</span>
        <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span>
            <span class="s">"Authorization"</span><span class="p">:</span> <span class="sa">f</span><span class="s">"Bearer </span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">token</span><span class="si">}</span><span class="s">"</span><span class="p">,</span>
            <span class="s">"Content-Type"</span><span class="p">:</span> <span class="s">"application/json"</span>
        <span class="p">}</span>
        <span class="n">payload</span> <span class="o">=</span> <span class="p">{</span>
            <span class="s">"queries"</span><span class="p">:</span> <span class="p">[{</span><span class="s">"query"</span><span class="p">:</span> <span class="n">dax_query</span><span class="p">}],</span>
            <span class="s">"serializerSettings"</span><span class="p">:</span> <span class="p">{</span><span class="s">"incudeNulls"</span><span class="p">:</span> <span class="bp">True</span><span class="p">}</span>
        <span class="p">}</span>
        
        <span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">post</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">,</span> <span class="n">json</span><span class="o">=</span><span class="n">payload</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">response</span><span class="p">.</span><span class="n">json</span><span class="p">()</span>

    <span class="k">def</span> <span class="nf">_format_schema_for_llm</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">columns_raw</span><span class="p">,</span> <span class="n">measures_raw</span><span class="p">):</span>
        <span class="c1"># Helper to convert raw JSON to a compact text string
</span>        <span class="c1"># enabling the LLM to ingest the schema without blowing up the context window.
</span>        <span class="n">schema_summary</span> <span class="o">=</span> <span class="s">"DATASET SCHEMA:</span><span class="se">\n</span><span class="s">"</span>
        <span class="c1"># ... transformation logic ...
</span>        <span class="k">return</span> <span class="n">schema_summary</span>
</code></pre></div></div>

<p><img src="https://blogs.buildwithmanish.com/assets/images/concept_illustration-building-a-virtual-data-analyst-architecting-agentic-ai-workflows-over-power-bi-3-2025-12-17-24503.png" alt="Concept Illustration: Introspection" /></p>

<h2 id="phase-2-dynamic-dax-generation-strategies">Phase 2: Dynamic DAX Generation Strategies</h2>
<p>The hardest part of this workflow is preventing DAX syntax errors. DAX is unforgiving; a missing bracket or an invalid relationship reference breaks the query.</p>

<p>To mitigate this, we employ <strong>Constraint-Based Prompt Engineering</strong> and <strong>Few-Shot Learning</strong>.</p>

<h3 id="the-prompt-strategy">The Prompt Strategy</h3>
<p>We don’t just ask the model to “Write DAX.” We provide a strict template.</p>

<p><strong>System Prompt Template:</strong></p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>You are an expert Power BI Architect and DAX developer. 
Your goal is to answer business questions by generating valid DAX queries.

RULES:
1. ALWAYS begin by using the `get_schema` tool to understand the dataset.
2. Never invent columns or measures. Only use what exists in the schema.
3. Use SUMMARIZECOLUMNS for querying data. Do not use SUMMARIZE.
4. When filtering, remember that text fields are case-sensitive in Power BI depending on collation.
5. If the user asks for a comparison (e.g., Year over Year), look for existing Time Intelligence measures first. 
   If none exist, construct them using CALCULATE and SAMEPERIODLASTYEAR.

OUTPUT FORMAT:
Return ONLY the raw DAX query inside a code block.
</code></pre></div></div>

<h3 id="handling-hallucinations-via-self-correction">Handling “Hallucinations” via Self-Correction</h3>
<p>In an agentic loop, if the generated DAX fails execution, the error message from the Power BI API (e.g., “Column ‘Rev’ does not exist”) is captured and fed back to the LLM.</p>

<p><strong>The Correction Loop:</strong></p>
<ol>
  <li>Agent generates DAX.</li>
  <li>Tool executes DAX.</li>
  <li><strong>Error:</strong> <code class="language-plaintext highlighter-rouge">Query (1, 4) The column 'Sales'[Rev] was not found.</code></li>
  <li>Agent receives error.</li>
  <li>Agent Thought: “Ah, I used ‘Rev’ but the schema says ‘Revenue’. I will rewrite the query.”</li>
  <li>Agent generates corrected DAX.</li>
</ol>

<p><img src="https://blogs.buildwithmanish.com/assets/images/technical_diagram-building-a-virtual-data-analyst-architecting-agentic-ai-workflows-over-power-bi-4-2025-12-17-10764.png" alt="Technical Diagram: Execution Loop" /></p>

<h2 id="phase-3-the-execution-layer-python--langchain">Phase 3: The Execution Layer (Python &amp; LangChain)</h2>
<p>Now we combine the tools into a LangChain Agent. We utilize the <code class="language-plaintext highlighter-rouge">OpenAIFunctionsAgent</code> or a <code class="language-plaintext highlighter-rouge">LangGraph</code> state machine for more control.</p>

<h3 id="the-agent-definition">The Agent Definition</h3>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">langchain.agents</span> <span class="kn">import</span> <span class="n">AgentExecutor</span><span class="p">,</span> <span class="n">create_openai_tools_agent</span>
<span class="kn">from</span> <span class="nn">langchain_openai</span> <span class="kn">import</span> <span class="n">ChatOpenAI</span>
<span class="kn">from</span> <span class="nn">langchain_core.prompts</span> <span class="kn">import</span> <span class="n">ChatPromptTemplate</span><span class="p">,</span> <span class="n">MessagesPlaceholder</span>
<span class="kn">from</span> <span class="nn">langchain.tools</span> <span class="kn">import</span> <span class="n">tool</span>

<span class="c1"># Initialize Service
</span><span class="n">pbi_service</span> <span class="o">=</span> <span class="n">PowerBIMetadataService</span><span class="p">(...)</span>

<span class="o">@</span><span class="n">tool</span>
<span class="k">def</span> <span class="nf">get_dataset_schema</span><span class="p">(</span><span class="n">dummy_arg</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s">"ignore"</span><span class="p">):</span>
    <span class="s">"""
    Retrieves the table names, columns, and measures from the Power BI dataset.
    Always call this FIRST to understand what data is available.
    """</span>
    <span class="k">return</span> <span class="n">pbi_service</span><span class="p">.</span><span class="n">get_semantic_schema</span><span class="p">()</span>

<span class="o">@</span><span class="n">tool</span>
<span class="k">def</span> <span class="nf">execute_dax_query</span><span class="p">(</span><span class="n">dax_query</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
    <span class="s">"""
    Executes a DAX query against the Power BI dataset and returns the results as JSON.
    Use SUMMARIZECOLUMNS for best performance.
    """</span>
    <span class="k">try</span><span class="p">:</span>
        <span class="n">result</span> <span class="o">=</span> <span class="n">pbi_service</span><span class="p">.</span><span class="n">execute_dax</span><span class="p">(</span><span class="n">dax_query</span><span class="p">)</span>
        <span class="c1"># Parse the 'results' key from Power BI API response
</span>        <span class="n">rows</span> <span class="o">=</span> <span class="n">result</span><span class="p">[</span><span class="s">'results'</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">'tables'</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">'rows'</span><span class="p">]</span>
        <span class="k">return</span> <span class="n">json</span><span class="p">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">rows</span><span class="p">)</span>
    <span class="k">except</span> <span class="nb">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
        <span class="k">return</span> <span class="sa">f</span><span class="s">"DAX Execution Error: </span><span class="si">{</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)</span><span class="si">}</span><span class="s">"</span>

<span class="c1"># Setup Agent
</span><span class="n">llm</span> <span class="o">=</span> <span class="n">ChatOpenAI</span><span class="p">(</span><span class="n">model</span><span class="o">=</span><span class="s">"gpt-4-turbo"</span><span class="p">,</span> <span class="n">temperature</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="n">tools</span> <span class="o">=</span> <span class="p">[</span><span class="n">get_dataset_schema</span><span class="p">,</span> <span class="n">execute_dax_query</span><span class="p">]</span>

<span class="n">prompt</span> <span class="o">=</span> <span class="n">ChatPromptTemplate</span><span class="p">.</span><span class="n">from_messages</span><span class="p">([</span>
    <span class="p">(</span><span class="s">"system"</span><span class="p">,</span> <span class="s">"You are a specialized Data Analyst Agent for Power BI..."</span><span class="p">),</span>
    <span class="p">(</span><span class="s">"user"</span><span class="p">,</span> <span class="s">"{input}"</span><span class="p">),</span>
    <span class="n">MessagesPlaceholder</span><span class="p">(</span><span class="n">variable_name</span><span class="o">=</span><span class="s">"agent_scratchpad"</span><span class="p">),</span>
<span class="p">])</span>

<span class="n">agent</span> <span class="o">=</span> <span class="n">create_openai_tools_agent</span><span class="p">(</span><span class="n">llm</span><span class="p">,</span> <span class="n">tools</span><span class="p">,</span> <span class="n">prompt</span><span class="p">)</span>
<span class="n">agent_executor</span> <span class="o">=</span> <span class="n">AgentExecutor</span><span class="p">(</span><span class="n">agent</span><span class="o">=</span><span class="n">agent</span><span class="p">,</span> <span class="n">tools</span><span class="o">=</span><span class="n">tools</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">handle_parsing_errors</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>

<span class="c1"># Execution
</span><span class="n">response</span> <span class="o">=</span> <span class="n">agent_executor</span><span class="p">.</span><span class="n">invoke</span><span class="p">({</span>
    <span class="s">"input"</span><span class="p">:</span> <span class="s">"Analyze the sales trend for Q4 2024 specifically for the 'Enterprise' segment. Compare it to the previous year."</span>
<span class="p">})</span>
</code></pre></div></div>

<h2 id="phase-4-synthesis-and-narrative-generation">Phase 4: Synthesis and Narrative Generation</h2>
<p>The output of <code class="language-plaintext highlighter-rouge">execute_dax_query</code> is a raw JSON array. While accurate, it is not a “response.” The final step in the agentic workflow is <strong>Synthesis</strong>.</p>

<p>Once the agent has the data in its context, it switches roles from “Data Engineer” to “Strategy Consultant.”</p>

<p><strong>Data Context:</strong></p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="w">
  </span><span class="p">{</span><span class="nl">"Month"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Oct 2024"</span><span class="p">,</span><span class="w"> </span><span class="nl">"Sales"</span><span class="p">:</span><span class="w"> </span><span class="mi">150000</span><span class="p">,</span><span class="w"> </span><span class="nl">"Sales_LY"</span><span class="p">:</span><span class="w"> </span><span class="mi">140000</span><span class="p">},</span><span class="w">
  </span><span class="p">{</span><span class="nl">"Month"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Nov 2024"</span><span class="p">,</span><span class="w"> </span><span class="nl">"Sales"</span><span class="p">:</span><span class="w"> </span><span class="mi">160000</span><span class="p">,</span><span class="w"> </span><span class="nl">"Sales_LY"</span><span class="p">:</span><span class="w"> </span><span class="mi">155000</span><span class="p">},</span><span class="w">
  </span><span class="p">{</span><span class="nl">"Month"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Dec 2024"</span><span class="p">,</span><span class="w"> </span><span class="nl">"Sales"</span><span class="p">:</span><span class="w"> </span><span class="mi">200000</span><span class="p">,</span><span class="w"> </span><span class="nl">"Sales_LY"</span><span class="p">:</span><span class="w"> </span><span class="mi">180000</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span></code></pre></div></div>

<p><strong>Synthesis Prompt:</strong></p>
<blockquote>
  <p>“Review the data retrieved. Identify key trends, calculate the Year-over-Year growth percentage for the quarter, and summarize the findings in a professional paragraph suitable for a CEO.”</p>
</blockquote>

<p>This separation of concerns-Data Retrieval (DAX) vs. Insight Generation (Natural Language)-is vital for accuracy.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/comparison_chart-building-a-virtual-data-analyst-architecting-agentic-ai-workflows-over-power-bi-5-2025-12-17-36313.png" alt="Comparison Chart: Text-to-SQL vs Agentic PBI" /></p>

<h2 id="advanced-scenarios-and-edge-cases">Advanced Scenarios and Edge Cases</h2>

<h3 id="1-handling-large-result-sets">1. Handling Large Result Sets</h3>
<p>The Power BI <code class="language-plaintext highlighter-rouge">executeQueries</code> REST API has a row limit (typically 100k rows) and the LLM has a context window.</p>
<ul>
  <li><strong>Solution:</strong> The agent must be instructed to <strong>aggregate</strong> data in DAX. Never request <code class="language-plaintext highlighter-rouge">SELECT *</code>. The prompt should enforce usage of <code class="language-plaintext highlighter-rouge">SUMMARIZECOLUMNS</code> grouped by high-level dimensions (Date, Region, Category) rather than individual transaction IDs.</li>
</ul>

<h3 id="2-row-level-security-rls">2. Row Level Security (RLS)</h3>
<p>One of the massive benefits of using the Power BI API over direct SQL access is that <strong>RLS is preserved</strong>. When the Service Principal (or the user represented by an explicit <code class="language-plaintext highlighter-rouge">EffectiveIdentity</code> in the API call) queries the dataset, Power BI automatically filters the data.</p>
<ul>
  <li><strong>Implementation:</strong> In the <code class="language-plaintext highlighter-rouge">executeQueries</code> payload, inject the <code class="language-plaintext highlighter-rouge">impersonatedUserName</code> property if you are building a user-facing chatbot to ensure they only see their specific data.</li>
</ul>

<h3 id="3-measures-with-dependencies">3. Measures with Dependencies</h3>
<p>Complex measures often depend on other hidden measures.</p>
<ul>
  <li><strong>Solution:</strong> In the Introspection phase, we query <code class="language-plaintext highlighter-rouge">INFO.MEASURES</code> including the <code class="language-plaintext highlighter-rouge">Expression</code> column. We can feed these expressions into the LLM context (using RAG) so it understands that <code class="language-plaintext highlighter-rouge">[Net Profit]</code> is actually <code class="language-plaintext highlighter-rouge">[Gross Profit] - [Opex]</code>. This allows the LLM to explain <em>why</em> a number changed, not just <em>what</em> the number is.</li>
</ul>

<h2 id="performance-considerations">Performance Considerations</h2>
<p>XMLA vs REST API:</p>
<ul>
  <li><strong>REST API (<code class="language-plaintext highlighter-rouge">executeQueries</code>):</strong> Easier to set up, works over standard HTTP, standard JSON output. Good for light-to-medium workloads.</li>
  <li><strong>XMLA Endpoint:</strong> Requires Premium/Fabric capacity. Allows connection via <code class="language-plaintext highlighter-rouge">ADOMD.NET</code> (wrapped in Python). Much faster for large data retrieval and supports more complex DAX operations.</li>
</ul>

<p>For a production-grade Virtual Analyst, I recommend using the <strong>XMLA endpoint</strong> via the <code class="language-plaintext highlighter-rouge">pythonnet</code> library to load the Microsoft Analysis Services client libraries. It provides a more robust, stateful connection compared to the stateless REST API.</p>

<p><img src="https://blogs.buildwithmanish.com/assets/images/concept_illustration-building-a-virtual-data-analyst-architecting-agentic-ai-workflows-over-power-bi-6-2025-12-17-50317.png" alt="Concept Illustration: Final Workflow" /></p>

<h2 id="conclusion">Conclusion</h2>
<p>Building a Virtual Data Analyst is not about fine-tuning a model to write SQL. It is about respecting the Semantic Layer. By architecting an agent that can Introspect, Plan, and Execute against Power BI’s native logic, we bridge the gap between “Generative AI toy” and “Enterprise Decision Engine.”</p>

<p>This approach ensures consistency. The number the chatbot gives the CEO is the exact same number the dashboard shows, because they are both generated by the same DAX engine. That-not just the ability to chat-is the holy grail of AI in Business Intelligence.</p>

<h3 id="next-steps-for-implementation">Next Steps for Implementation</h3>
<ol>
  <li><strong>Enable XMLA Read/Write</strong> in your Power BI Premium Capacity settings.</li>
  <li><strong>Create a Service Principal</strong> in Azure Entra ID and grant it access to your Workspace.</li>
  <li><strong>Start Small:</strong> Pick a dataset with a clean schema and well-named measures.</li>
  <li><strong>Iterate on Prompts:</strong> The system prompt is your new code base. Version control it and test it against edge cases.</li>
</ol>

<p>The future of BI isn’t just dashboards; it’s intelligent agents that can read those dashboards for you.</p>]]></content><author><name>Manish Tiwari</name></author><category term="deep-dive" /><category term="gen-ai" /><category term="power-bi" /><category term="agentic-workflows" /><category term="python" /><summary type="html"><![CDATA[]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" /><media:content medium="image" url="https://blogs.buildwithmanish.com/assets/images/Blogs.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>