If you don’t follow me don’t forget to do so and read these 2 posts I made about my Windows to Linux migration:

• Defenestrate Windows
• I Defenestrated Windows and Adopted a Penguin

Intro

For years, the “year of the Linux desktop” has been a running joke in the tech community. But in the world of data science and AI development, that year might have already arrived — and it’s orange.

As a data professional, I’ve spent more hours than I care to admit wrestling with CUDA versions, matching toolkit dependencies to specific kernels, and fixing environments that spontaneously broke after a routine update. However, after looking at the roadmap for the upcoming Ubuntu 26.04 and the new tooling Canonical is introducing, I believe it is time to seriously consider Ubuntu as the primary OS for AI development.

Here is why the “orange Linux” is evolving into an AI-native platform.

1. One Command for Hardware Acceleration

The most significant friction in AI development has always been the “driver hell.” Whether you use Nvidia or AMD, setting up the hardware stack is often a manual, error-prone process.

Starting with Ubuntu 26.04, this changes completely. We will be able to run apt install cuda or apt install rockm and get the exact, verified version that works with our specific GPU and OS version—out of the box. By partnering directly with silicon giants like Nvidia, AMD, and Intel, Ubuntu ensures that the kernel supports not just CPUs and GPUs, but the newer MPUs and TPUs from day one.

2. Optimized Local AI via “Inference Snaps”

Testing a new model often feels like it requires a “PhD in Hugging Face” just to manage the dependencies and inference engines.

Ubuntu is solving this with Inference Snaps. These are high-quality, silicon-optimized AI models (like DeepSeek, Gemma, or Llama) packaged in a security-confined format. When you install one, it automatically detects your hardware — AMD, Nvidia, or Intel — and fetches the version tuned by the silicon vendor itself. They even come with an OpenAI-compatible API endpoint, allowing you to point your existing local tools directly to a production-grade local model with a single command.

3. Limiting the “Blast Radius” of AI Agents

As our workflows shift toward using autonomous agents, security becomes a practical concern. We’ve all seen stories of agents accidentally deleting files or exhausting system resources.

Ubuntu provides built-in sandboxing through LXD and Multipass. I’ve started using these to create “disposable” environments for my agents. If an agent makes a catastrophic error, it only affects an isolated container or VM, not my entire host system. This isolation doesn’t just provide security; it ensures my local context stays clean and efficient.

4. Enterprise-Grade Stability for the Long Haul

The reality is that Ubuntu already powers the majority of AI workloads in the cloud. By developing on the same OS that handles production on AWS or Google Cloud, we eliminate the “works on my machine” problem.

Furthermore, Canonical’s commitment to 15 years of security maintenance — including their “LTS anything” initiative for Docker containers and Python dependencies — means that the infrastructure we build today won’t become a security liability in the future.

Final Thoughts

Ubuntu isn’t just “another Linux distribution” anymore; it is becoming a specialized workbench for AI. By removing the friction of hardware drivers and providing secure, optimized ways to run local models, it allows us to focus on what actually matters: building and deploying models.

If you are tired of troubleshooting your operating system and want to start building, I highly recommend giving Ubuntu another look.

Igor Comune.

Post inspired by this video: https://www.youtube.com/watch?v=0CYm-KCw7yY

The 3 AM Wake-Up Call

Picture this: You’re a data scientist at a fintech company. Your fraud detection model is crushing it in notebooks — 98% precision, everyone’s happy. You hand it off to engineering, they deploy it, and… suddenly you’re catching 40% less fraud than in testing.

What the hell happened?

We’ve all been there… The model looked perfect… The metrics were solid… But somewhere between your Jupyter notebook and production, something broke. And here’s the kicker:

it’s probably not your model. It’s your FEATURES.

This is where Feature Stores enter the picture. But instead of starting with “here’s what a Feature Store is,” let’s do something different. We’re going to break things systematically, watch them fail in real-time, then fix them properly.

Our weapon of choice? A hands-on Jupyter notebook that simulates a real fraud detection pipeline. We’re talking about the same problems you face in production every single day.

The Five Horsemen of ML Production Hell

Before we dive in, let’s name our enemies. Feature Stores solve five critical problems that plague every ML system at scale:

1. Training vs Production Consistency — Your training code uses Pandas, production uses SQL. Same features, different bugs.
2. Feature Reuse Across Teams — The risk team computed user velocity last month. Now the fraud team is recomputing it. Same logic, different bugs.
3. Low-Latency Feature Serving — Your model needs to decide in 50ms, but your features take 200ms to compute. Whoops.
4. Point-in-Time Correctness — You accidentally trained on future data. Your model is a time traveler, and it sucks at production.
5. Feature Governance & Discovery — Where are the features? Who owns them? What version? Nobody knows.

Sound familiar? Yeah, we thought so.

Let’s systematically break and then fix each one using our fraud detection scenario.

Setup: Getting Your Hands Dirty

DOWNLOAD THE NOTEBOOK PYTHON HERE or

https://github.com/IgorComune/mlops_feature_store_crash_course

You’ll have a better learning experience if you read the blog with and follow the steps on Google Colab.

Execute cells 1–2 to get started.

These cells install Feast and the necessary dependencies. Nothing fancy here — just standard ML stack stuff. The key thing is we’re setting up a complete environment that mirrors real production constraints.

Cell 3 initializes the Feast repository. This is important: Feast needs a config file that defines where features live, how they’re stored, and how they’re served. We’re using SQLite for simplicity, but in production, you’d swap this for something like Redis or DynamoDB.

Notice the dual storage model being configured: registry (metadata about features) and online_store (the actual feature values). Keep this architecture in mind — it’s central to how Feature Stores solve latency problems.

Problem #1: Training-Serving Skew (The Silent Killer)

THE PROBLEM: Your data scientist writes feature logic in Pandas. Your ML engineer rewrites it in SQL for production. Same intent, different implementations. Guess what? They diverge. Your model trained on one thing, serves on another.

Seeing the Problem

Execute cells 4–9 to generate synthetic transaction data and see the naive approach.

Cell 4 creates a realistic fraud dataset — transactions with amounts, merchants, countries, timestamps. The key detail: we’re generating fraud patterns based on user behavior over time. High-velocity users (lots of transactions in a short window) are more likely to be fraudulent. This behavioral pattern becomes the foundation for our feature engineering — we’ll track how many transactions a user makes within a 24-hour window, and use that velocity as a fraud signal.

This is the core logic we’ll need to replicate in production. Remember this — we’ll come back to it.

Cells 5–6 create the “Data Science approach” — computing features in Pandas during training. Here’s where we see the first implementation: using DataFrame operations to group transactions by user and compute aggregations like transaction count and average amount. It’s clean, readable, and exactly how every data scientist naturally thinks about feature engineering. The Pandas groupby and transform operations make this trivial — a few lines of code and you’ve got your features.

Simple, clean, Pythonic. This is how every data scientist writes their features.

Now here’s where it gets ugly.

Cells 7–8 show the “Engineering approach” — the same features, but reimplemented for production. Instead of Pandas operations on a complete DataFrame, we’re now computing features for individual transactions as they arrive in real-time. This means hitting the database, running SQL queries to aggregate historical transactions, applying filters, and handling edge cases. We’re also simulating the reality of production: every query has latency (around 30ms), and you need to handle scenarios like new users with no transaction history.

Notice what just happened:

1. Different language (SQL vs Pandas)
2. Different aggregation windows (subtly different time boundaries)
3. Added latency (30ms per query)
4. Different handling of edge cases (what if user has no history?)

These implementations should produce the same results. But they won’t. They never do.

Execute cell 9 to see the divergence. The output shows you’ve got 10,000 training examples with features computed successfully, and production feature computation taking about 45ms per transaction. But here’s the warning that should make you nervous: the implementations may differ in edge cases.

The metrics look similar, but dig deeper and you’ll find discrepancies in:

• How nulls are handled (Pandas fills with NaN, SQL might return NULL or 0)
• Timezone conversions (UTC vs local time, daylight savings)
• Aggregation window boundaries (inclusive vs exclusive, timestamp precision)
• Rounding differences (floating point arithmetic across different libraries)

This is training-serving skew. Your model trained on one distribution, serves on another. And you won’t know until production starts bleeding money.

Problem #2: Feature Duplication (The N² Scaling Nightmare)

THE PROBLEM: You have 5 teams. Each team needs user velocity. Each team computes it independently. Now you have 5 implementations of the same feature. 5 bugs. 5 maintenance burdens. And god help you when you need to change the definition.

Seeing the Problem

Execute cells 10–11 to see feature duplication in action.

Cell 10 simulates what happens in real organizations. We create three different functions, each representing how a different team implemented the exact same concept — user transaction velocity. The fraud team counts all transactions for a user. The risk team applies a 24-hour sliding window. The credit team uses a different grouping mechanism entirely. Same business logic, three completely different implementations.

This isn’t a strawman. Walk into any company with multiple ML teams and you’ll find this exact scenario. The fraud detection team built their velocity feature six months ago. The credit risk team needed the same thing last quarter but didn’t know it existed, so they built their own. The anti-money laundering team is about to do the same thing next sprint.

Cell 11 quantifies the damage. You’ll see that we now have three duplicated features across the organization. The code maintenance cost is tripled — when you need to update the business logic (say, changing the velocity window from 24 hours to 12 hours), you need to update three separate codebases. The bug surface area is tripled — find a bug in one implementation, and you’ve probably got the same bug in two other places. And here’s the kicker: consistency is never guaranteed. These implementations will drift over time as different teams patch bugs, add edge case handling, or optimize performance.

When the product manager says “change how we calculate velocity,” you now have to update 3 codebases, coordinate 3 deployments, and pray they all stay in sync. Spoiler: they won’t.

Problem #3: Latency (The Production Killer)

THE PROBLEM: Your model needs to approve or reject transactions in real-time. You have a 50ms SLA. But your features take 200ms to compute because you’re hitting the database on every prediction.

Seeing the Problem

Execute cells 12–14 to measure naive serving latency.

Cell 12 simulates production serving without any optimization. When a transaction comes in, we need to make a prediction immediately. The naive approach does this: compute the features on-demand (which means querying the database for historical transactions, running aggregations, computing statistics), then feed those features into the model for inference. The model inference itself is fast — maybe 1–2ms. But the feature computation? That’s where we pay the price.

Execute cell 13 to run 1000 predictions and see the damage. The output shows average latency around 44.5ms, with P95 latency at 48.2ms and P99 latency breaking through 52.1ms. You’re seeing that about 12% of predictions are exceeding your 50ms SLA.

You’re barely making SLA on average, but your tail latencies are killing you. And this is with a simple feature set — just user velocity and average transaction amount. Add more features (merchant category frequency, time-of-day patterns, device fingerprinting, location-based features), more joins, more complexity, and you’re toast. Each additional feature means another database query, another aggregation, another 10–30ms added to your latency budget.

Cell 14 visualizes the problem with a histogram of latencies. You’ll see a distribution that’s mostly clustered around 40–50ms, but with a long tail stretching out to 70–80ms for the unlucky requests. That long tail? Those are the transactions that timeout, get retried, cause user frustration, create duplicate charges, and ultimately cost you money in lost conversions and customer support tickets.

The core issue: You’re computing features on-demand at prediction time. Every prediction triggers database queries, aggregations, joins. This doesn’t scale. As your feature set grows, as your transaction volume increases, as your database gets loaded, those latencies compound. What works fine at 100 QPS falls apart at 10,000 QPS.

Problem #4: Point-in-Time Correctness (The Data Leakage Timebomb)

THE PROBLEM: You’re training your model on historical data. You accidentally use a feature value from the future. Your model learns to predict the past using information it wouldn’t have in production. It looks amazing in training, worthless in production.

This is the most insidious problem because it’s silent. Your model metrics look great. Your code reviews pass. Then production happens.

Seeing the Problem

Execute cells 15–18 to see data leakage in action.

Cell 15 shows the wrong way to build training data — and it’s the way most people naturally do it first. When you have a DataFrame of transactions and you want to compute user velocity, the obvious approach is to group by user_id and count transactions. Simple, right? But here’s the trap: that groupby operation counts all transactions for each user in the dataset, including transactions that happened in the future relative to each row.

The bug is subtle but devastating. When you’re processing a transaction that happened on January 1st, you’re computing the user’s velocity based on their total transaction count across the entire dataset — which includes transactions from January 2nd, 3rd, all the way through the end of your data. You’re using future information to predict the past.

Execute cell 16 to see the impact quantified. The model trained with this data leakage achieves an AUC of 0.94 — fantastic! But when you train the same model correctly (without the leakage), you get an AUC of 0.82. That’s a 12.8% performance drop. Your model looked like a rockstar in training because it was cheating. In production, where it can’t see the future, it’s mediocre.

This happens all the time. You build your training set, compute features, train a model, get great metrics, ship it to production, and watch it underperform. You blame the model, the data quality, concept drift — everything except the real culprit: temporal data leakage in your feature engineering.

Cells 17–18 show the correct approach — point-in-time joins. For each transaction, we only use data that existed before that transaction occurred. This means for a transaction on January 5th, we compute user velocity based only on transactions that happened before January 5th. No future peeking allowed.

This is correct, but notice what we just did: we turned a simple aggregation into a sequential operation. Instead of one groupby operation across the entire dataset (which takes seconds), we’re now iterating through each transaction individually, filtering historical data for each one, and computing features (which takes hours). Training time just went from minutes to hours or even days for large datasets.

This is the curse of point-in-time correctness — doing it right is expensive. You need to maintain temporal consistency, which means you can’t use the fast vectorized operations that make Pandas so efficient. You’re forced into slower, more complex logic that respects time boundaries.

Problem #5: Feature Governance (The “Who the Hell Owns This?” Problem)

THE PROBLEM: You have 50 features scattered across 15 notebooks, 8 services, and 3 data warehouses. Someone asks “what’s the definition of user_velocity?” Nobody knows. Someone asks “is this feature tested?” Nobody knows. Someone asks “can I reuse this?” Nobody knows.

Seeing the Problem

Execute cells 19–20 to see the governance chaos.

Cell 19 simulates the typical ML codebase after a year of rapid iteration. Features are scattered everywhere. Version 1 of the notebook defined user_velocity and user_avg_amount. Version 2 (created three months later when someone needed to add a new feature) redefined user_velocity slightly differently and called it user_velocity_v2, while also adding user_max_amount. Meanwhile, production is running with yet another set of features that may or may not match either version.

Which version is canonical? Which one is actually running in production? If you want to use user_velocity for a new model, which implementation should you trust? Good luck figuring it out. You’ll need to grep through repositories, ask around on Slack, hope someone who worked on the original project is still on the team, and probably end up re-implementing it yourself because you can’t trust any of the existing versions.

Cell 20 quantifies this organizational chaos. Across your organization, you have 47 total features. Of those, exactly 3 have documentation explaining what they compute and why. Only 1 has tests verifying it works correctly. 5 have a known owner who you could ask questions. And the average time to find a feature definition? 4.3 hours. That’s more than half a workday just to figure out if a feature already exists and how it’s computed.

This isn’t hypothetical. This is every ML org at scale. Features are tribal knowledge. The person who implemented user_velocity_v2 left the company six months ago. The original notebook that defined merchant_category_freq has been deleted. The only person who knows how device_fingerprint works is on vacation. And you’re on a deadline to ship a new model that needs all three of these features.

So what do you do? You reimplement them. Again. Adding to the chaos. Contributing to problem #2 (feature duplication). And the cycle continues.

Enter the Feature Store: Solving All Five Problems Systematically

Now that we’ve thoroughly broken everything, let’s fix it. This is where Feast comes in.

The key insight: All five problems stem from the same root cause — features are not first-class citizens. They’re scattered, duplicated, unversioned, undocumented artifacts of individual pipelines.

Feast makes features first-class by providing:

1. A centralized registry (solves governance)
2. A single source of truth for feature definitions (solves duplication + skew)
3. Pre-computed online storage (solves latency)
4. Built-in point-in-time joins (solves data leakage)
5. Dual offline/online stores (same features, training and serving)

Let’s see this in action.

Solution Part 1: Defining Features as Code

Execute cells 21–23 to see how Feast defines features.

Cell 21 creates an Entity — the primary key for our features. In our fraud detection system, the entity is the user_id. Think of entities as the “join keys” of your feature world. Every feature belongs to an entity — user features belong to users, merchant features belong to merchants, transaction features belong to transactions. This is how Feast knows how to link features together when you request them.

The entity definition includes not just the name, but also a description that serves as documentation. This is governance in action — when someone wants to know what user_id means, there’s a canonical answer in the registry.

Cell 22 defines a Feature View — where the actual feature logic lives. This is the magic that solves problems #1, #2, and #5 simultaneously. We’re defining a set of features (user_velocity_24h, user_avg_amount, user_total_amount) that all belong to the user entity. We specify their data types, their source (a parquet file in this case, but it could be a data warehouse table), and their TTL (time-to-live, which tells Feast how long these features stay fresh).

Here’s what just happened: feature logic defined once, used everywhere. No more Pandas vs SQL debates. No more reimplementing the same feature across teams. No more wondering which version is correct. There’s one definition, stored in a centralized registry, version-controlled, and guaranteed to be used consistently in both training and serving.

The Feature View also includes a schema — explicit data types for every feature. This prevents the subtle bugs that happen when one implementation returns an integer and another returns a float, or when null handling differs between implementations.

Cell 23 registers these definitions with Feast. When you run this cell, Feast takes your entity and feature view definitions and stores them in the registry (that SQLite database we configured earlier). Now anyone in your organization can query the registry to discover what features exist, who owns them, what they compute, and how to use them.

Execute this and check the output. You’ll see confirmation that the entity and feature view were registered successfully. You now have a centralized catalog of features. Anyone can query the registry to discover what exists, who owns it, and how it’s defined. Problem #5 (governance) is solved. Problem #2 (duplication) is solved — there’s one canonical definition. And problem #1 (training-serving skew) is about to be solved because we’ll use this same definition for both training and serving.

Solution Part 2: Training with Point-in-Time Correctness (Automatic)

Execute cells 24–25 to see point-in-time joins in action.

Cell 24 prepares training data using Feast. This is where the magic happens. We create what Feast calls an “entity DataFrame” — essentially a list of entities (user IDs) and timestamps representing when we want to fetch features. In our fraud detection case, each row represents a transaction, and we want to know what the user’s features looked like at the exact moment that transaction occurred.

Then we call Feast’s get_historical_features method. This is where problem #4 (point-in-time correctness) gets solved automatically. Feast takes our entity DataFrame and, for each row, retrieves the feature values that were valid at that specific timestamp. For a transaction on January 5th at 2:30 PM, Feast will fetch the user_velocity_24h value as it existed at exactly 2:30 PM on January 5th—computed using only transactions that occurred before that moment.

This is the critical part: Feast handles all the complex temporal logic for you. You don’t need to write loops. You don’t need to manually filter historical data. You don’t need to worry about off-by-one errors in timestamp comparisons. Data leakage is impossible by design because Feast enforces temporal consistency at the infrastructure level.

Execute cell 25 to compare the results with the naive approach. The output shows a crucial difference: when we compute user velocity with Feast using point-in-time joins, the maximum velocity for any user in our training set is 8 transactions. This makes sense — it’s the actual maximum number of transactions any user had completed at any single point in time during the training period.

But when we computed it naively (the wrong way from cells 15–16), we saw maximum velocities of 156. That’s because the naive approach counted all transactions for each user across the entire dataset, including future transactions. A user who ended up making 156 transactions total would have that count applied to every single one of their historical transactions, even their very first one.

Feast enforces correctness by construction. You can’t accidentally leak data even if you try. And you get this guarantee without sacrificing performance — Feast optimizes the point-in-time joins internally so you’re not stuck with the slow sequential processing we implemented manually in cell 17.

Solution Part 3: Online Serving (Low Latency by Design)

Execute cells 26–27 to see online serving.

Cell 26 is where we solve problem #3 (latency). The key operation here is called materialization — Feast takes all the feature values, pre-computes them, and stores them in the online store (that SQLite database we configured, though in production you’d use Redis, DynamoDB, or another low-latency key-value store).

Think about what just happened: instead of computing features on-demand during prediction time (query the database, aggregate historical transactions, calculate statistics — all that expensive stuff that took 45ms), we’re computing everything upfront and storing the results. When a transaction comes in and we need to make a prediction, we just look up the pre-computed feature values by user_id. That’s a simple key-value fetch, which takes microseconds instead of milliseconds.

This is the architectural shift that Feature Stores enable: from compute-on-read (slow, unpredictable latency) to compute-on-write (fast, consistent latency). You pay the computational cost during the materialization process (which runs offline, on a schedule, without latency constraints), and you get instant lookups at serving time.

Cell 27 shows online retrieval in action. When we need to make a prediction for a transaction, we call get_online_features with the user_id. Feast immediately returns the pre-computed feature values from the online store. No database queries. No aggregations. No waiting. Just a fast lookup.

Execute cell 27 and watch the latency numbers. The output shows average latency dropping from 44.5ms (without Feast) to 5.5ms (with Feast). That’s an 8x speedup. More importantly, look at the P95 and P99 latencies — they’re now consistently under 10ms, which means you’re making your 50ms SLA with room to spare.

We went from barely scraping by (with 12% of predictions exceeding SLA) to comfortably meeting our latency requirements. And this is with a simple feature set — as you add more features, the gap widens. The naive approach gets slower linearly with each feature (more queries, more joins), while Feast stays constant (one lookup fetches all features for an entity).

This is the difference between a good user experience and a terrible one. This is the difference between a fraud detection system that catches fraud in real-time versus one that has to batch-process transactions hours later when it’s too late to prevent the fraud.

The Full Picture: Before and After

Execute cell 28 to see the final comparison table.

The output shows a comprehensive side-by-side comparison of every dimension we’ve discussed:

Point-in-Time Correctness: Without a Feature Store, you’re manually implementing temporal logic, hoping you got all the edge cases right, and constantly worrying about data leakage. With Feast, it’s automatic — the infrastructure enforces correctness.

Code Duplication: Without a Feature Store, your data scientists are writing features in Pandas, your engineers are rewriting them in SQL, and every team has their own slightly-different implementation. With Feast, you have a single definition that everyone uses.

Latency: The numbers speak for themselves — P95 latency drops from 48ms to 6.5ms. That’s not incremental improvement, that’s a qualitative shift in what’s possible.

Governance: Without a Feature Store, features are scattered across notebooks, repositories, and tribal knowledge. With Feast, you have a centralized registry where every feature is documented, versioned, and discoverable.

Feature Versioning: Without a Feature Store, you’re tracking feature versions in Git by hoping notebook filenames make sense. With Feast, versioning is native — you can see exactly which version of which features were used to train which model.

Team Reusability: Without a Feature Store, every team reimplements every feature they need. With Feast, a feature defined once is immediately available to every team, every model, every use case.

Look at that latency drop: 48ms → 6.5ms. That’s 87% faster. And you get correctness guarantees, governance, and reusability for free. This is why Feature Stores exist — they solve multiple critical problems simultaneously, problems that are painful to solve independently and nearly impossible to solve consistently across an organization.

When Should You Actually Use a Feature Store?

Feature Stores aren’t free. They add complexity, operational overhead, and cognitive load. So when does it make sense?

Use a Feature Store when:

✅ You have 3+ models in production — Shared feature infrastructure pays for itself

✅ You have latency SLAs < 50ms — Pre-materialization is essential

✅ You have multiple teams — Centralized catalog prevents duplication

✅ You’ve been burned by training-serving skew — Single source of truth saves your …

✅ You care about data leakage — Point-in-time joins are hard to do right manually

Don’t use a Feature Store if:

❌ You’re running 1–2 models — Overhead isn’t worth it

❌ You’re doing batch scoring — Latency doesn’t matter

❌ You’re in pure research mode — Too early to standardize

❌ Your team is < 3 people — Coordination overhead exceeds benefits

The Hard Truth

Feature Stores are infrastructure. They solve real, painful problems. But they’re not magic.

You’ll still need to:

• Design good features (garbage in, garbage out)
• Monitor feature drift (distributions change in production)
• Maintain pipelines (data quality doesn’t fix itself)
• Handle versioning (features evolve)

What Feature Stores give you is systematic solutions to systematic problems. They don’t make ML easy. They make ML operationally sound.

If you’ve shipped models to production, you’ve felt these problems. Now you know how to fix them.

Appendix: Cell-by-Cell Execution Guide

For quick reference:

• Cells 1–3: Setup and installation
• Cells 4–9: Problem #1 (Training-Serving Skew)
• Cells 10–11: Problem #2 (Feature Duplication)
• Cells 12–14: Problem #3 (Latency)
• Cells 15–18: Problem #4 (Point-in-Time Correctness)
• Cells 19–20: Problem #5 (Governance)
• Cells 21–23: Solution #1 (Feature Definitions)
• Cells 24–25: Solution #2 (Point-in-Time Joins)
• Cells 26–27: Solution #3 (Online Serving)
• Cell 28: Final Comparison

Run them in order. Don’t skip the problem cells — you need to feel the pain to appreciate the solution.

Igor Comune.

• Igor Comune - Medium
• IgorComune - Overview

https://www.linkedin.com/in/igor-comune/

The Tech Challenge for Phase 4 of my Master’s in Machine Learning Engineering at FIAP came with a clear assignment: build an LSTM neural network to predict stock closing prices and deploy it as a REST API in production.

The requirements were explicit:

• Collect and preprocess historical stock data
• Develop an LSTM model with proper evaluation metrics (MAE, RMSE, MAPE)
• Save and export the trained model
• Deploy as a RESTful API (Flask or FastAPI)
• Implement monitoring for production performance

The team: Igor Comune (me), Éder Ray, Mário Gotardelo, João Marcelo, and Felippe Maurício.

This project IS NOT about predicting the markets. This is about deployment of Machine Learning Models

The Model Architecture

Here’s the breakdown:

Input Layer (4 features: Open, High, Low, Volume)
    ↓
LSTM Layer 1 (128 hidden units)
    ↓
Dropout (0.2)
    ↓
LSTM Layer 2 (128 hidden units)
    ↓
Dropout (0.2)
    ↓
Fully Connected Layer (25 units, ReLU)
    ↓
Output Layer (Close price)

Key decisions:

• Lookback window: 30 days. Why 30? Because we needed enough data to capture monthly patterns without overfitting on noise. We tested 30, 60, and 90 days.
• 2 layers with 128 units each. More layers = more capacity, but also more risk of overfitting. We found 2 layers was the sweet spot. Going to 3 layers didn’t improve performance.
• Dropout at 0.2. Regularization was essential. Without dropout, the model memorized training data and failed on validation.
• Adam optimizer (learning rate = 0.001). SGD was too unstable; Adam converged smoothly without much hyperparameter tuning.

The Data Pipeline (Where Things Got Messy)

Data Collection

We’re using Yahoo Finance Python Library (yfinance) to import the data from the markets, src/utils/data_collection.py and in the same file we clear it from outliers and export it to data/processed/df.csv

Training

With the processed data ready, we train the model using src/models/lstm_model.py and src/utils/model_training.py, which implements the full training loop, validation strategy, checkpointing, and experiment tracking.

The model is trained to predict the next closing price using OHVL features (Open, High, Low, Volume), deliberately excluding Close from the inputs, as it is the prediction target.

Price-level prediction was chosen to simplify deployment, evaluation, and monitoring in this phase of the project. While predicting returns or direction would be more appropriate for trading-oriented systems, predicting absolute price levels allows for clearer error interpretation, easier debugging, and a more straightforward API contract during early production experimentation.

This choice reflects an engineering trade-off: prioritizing system design, reproducibility, and observability over financial realism in the initial iteration.

Temporal Validation Strategy

Time series models are especially sensitive to data leakage, so a standard random train/test split is not appropriate.

In this project, we adopted a strict chronological split, preserving the natural order of the data. The dataset was divided into training (70%), validation (15%), and test (15%) segments along the time axis.

Each split was then transformed into supervised learning samples using a sliding window approach (30-day lookback), where the model predicts the next closing price given the previous 30 observations. Importantly, windows are constructed only within each split, ensuring that no future information leaks into the training or validation phases.

This setup corresponds to an expanding-window, single-holdout evaluation, which reflects a realistic production scenario: the model is trained on past data and evaluated on strictly unseen future periods. While this approach does not capture all regime changes, it provides a conservative and leakage-safe estimate of model performance for this phase of the project.

Evaluation

And using src/utils/model_evaluation.py we evaluate the model itself.

This setup corresponds to an expanding-window, single-holdout evaluation, which reflects a realistic production scenario: the model is trained on past data and evaluated on strictly unseen future periods. While this approach does not capture all regime changes, it provides a conservative and leakage-safe estimate of model performance for this phase of the project.

Results (And Why R² = 0.98 Isn’t Too Good to Be True)

Metrics on the test set:

• MAE: $0.18 — Average error of 18 cents
• RMSE: $0.11 — Penalizes large errors more than MAE
• MAPE: 5.10% — Percentage error, excellent for financial time series
• R²: 98 — Model explains 98 of variance

This plot shows that the predicted values follow the general trend of the real values, indicating the model learned the overall dynamics of the series.

The R² = 0.98 Problem

When you see R² = 0.98, your first instinct should be skepticism. Is this overfitting? Data leakage?

In early phases, metrics like R² can be misleadingly high due to autocorrelation. Recognizing this was part of our learning curve.

We asked the same questions. Here’s what we validated:

1. Train vs. validation loss: They tracked closely without diverging. No overfitting.
2. Out-of-sample testing: We tested on unseen periods (January 2024 forward). Performance held.
3. Manual inspection: We checked predictions during volatile events (market crashes, earnings surprises). The model responded sensibly.

This repeated error distribution confirms that most errors remain small and centered near zero, reinforcing that the model is stable on average.

The real reason for high R²: Stock prices are highly autocorrelated. Today’s price is a strong predictor of tomorrow’s price. The model isn’t “too good” — it’s capturing that inherent autocorrelation.

Important caveat: This doesn’t mean the model is useful for trading. Predicting the direction (up/down) is harder than predicting the value.

This plot shows that residuals are not randomly distributed around zero over time, indicating that the model’s errors are structured rather than purely random.

Out-of-pattern behavior & possible causes:
The increasing variance and large residual spikes toward the end of the series suggest heteroscedasticity and regime change, meaning the model was trained on past dynamics that no longer hold, lacks features to capture volatility shifts, or is effectively extrapolating beyond its learned domain.

All of the results were saved both on the folder /results and inside /mlruns (folder created by the experiments inside MLFlow.

The API

It was required basically 3 endpoints from us: prediction, health and monitoring.

All of them were implemented using FAST API inside src/api, using main.py and the following command to launch the server and all of the endpoints:

uvicorn src.api.main:app --reload --port 8000

Why FastAPI?

We considered Flask. FastAPI won for these reasons:

1. Automatic validation: Pydantic schemas catch errors before they hit the model.
2. Interactive docs: Swagger UI out-of-the-box. No manual documentation needed.
3. Async by default: Better performance for I/O-bound operations.
4. Type hints: Fewer runtime bugs.

Flask is fine. FastAPI is better for production ML.

The endpoints:

/predict: receives the input data and outputs the prediction
/metrics: evaluate the consumption of RAM, CPU...
/health: health of the server

Why Prometheus? Industry standard. Integrates with Grafana. Pull-based architecture simplifies networking.

And for testing the endpoint of prediction you’ll need just to run (in parallel, of course the server) and the file infer.py in the root folder of the project.

So, we have:

1. The data collection and transformations
2. The model
3. The training
4. The evaluation
5. The deployment
6. The model prediction output

Using Python, Docker, MLFlow, Prometheus, FastAPI, PyTorch, SkLearn and Pandas.

Containerization

Docker was non-negotiable for reproducibility:

FROM python:3.12-slim

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

RUN apt-get update \
    && apt-get install -y --no-install-recommends curl \
    && rm -rf /var/lib/apt/lists/*

RUN useradd -m appuser
USER appuser

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir --upgrade pip \
    && pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8000

HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
  CMD curl --fail http://localhost:8000/health || exit 1

CMD ["uvicorn", "src.api.main:app", "--host", "0.0.0.0", "--port", "8000"]

“This container focuses on simplicity over optimization. In later stages, I plan to separate training and inference images and adopt multi-stage builds.”

Production notes:

• Multi-stage builds would reduce image size (we didn’t do this for the Tech Challenge, but you should).
• Don’t run as root in production.
• Scan for vulnerabilities with Trivy or Snyk.
• Use a reverse proxy (Nginx) instead of exposing port 8000 directly.

What We Didn’t Do (And Should Have)

Let’s be honest about limitations:

1. No fundamental data: The model is purely technical. It doesn’t know about earnings, news, or sentiment.
2. Fixed lookback window: 30 days works well on average, but during high volatility, a shorter window might be better. We didn’t implement adaptive windowing.
3. Single-stock prediction: We predict each stock independently. Portfolio-level models that consider correlations between stocks would be more robust.
4. No online learning: The model is static. In production, you’d want to retrain periodically as new data arrives (concept drift is real in financial markets).

Engineering Trade-offs & Learning Progression

• Used a single Docker image for simplicity; later phases will split training and inference containers.
• Metrics are strong due to autocorrelation; next iterations will benchmark against naive baselines.
• Model is retrained offline; future work includes scheduled retraining and drift detection.
• Monitoring focuses on system metrics; model performance monitoring will be expanded.

Conclusion

This project wasn’t just about achieving good metrics. It was about understanding the entire ML pipeline: data collection, preprocessing, model architecture, deployment, monitoring, and documentation.

The real challenge: Making technical decisions under constraints (time, compute, data quality) and justifying those decisions.

Full code available at: github.com/IgorComune/masters_tc_LSTM_stock_prediction

Video: https://www.youtube.com/watch?v=SfdFywHQDzw

Acknowledgments

This project wouldn’t have been possible without Éder Ray, Mário Gotardelo, and Felippe Maurício. Working as a team brought diverse perspectives on model design, software engineering, and ML best practices.

Thanks to FIAP for structuring the Tech Challenge in a way that simulates real-world ML projects.

Disclaimer: This project is for educational purposes only. Stock price prediction models do not constitute financial advice. Always consult qualified professionals before making investment decisions. Past performance does not guarantee future results.

Igor Comune.

IgorComune - Overview

https://www.linkedin.com/in/igor-comune/

Éder Ray

ederray - Overview

https://www.linkedin.com/in/ederray/

Mário Gotardelo

MariolGotta - Overview

https://www.linkedin.com/in/m%C3%A1rio-gottardello-2456a818a/

Felippe Maurício

felippemauricio - Overview

https://www.linkedin.com/in/felippemauricio/

João Marcelo

joaomendonca-py - Overview

https://www.linkedin.com/in/joaomarcelomendonca/

Now imagine trying the same conversation with someone you just met. You would need to explain every detail, provide full addresses, recount the entire story from the beginning. The difference between these two conversations is precisely what context engineering seeks to solve in the world of artificial intelligence.

The Problem of Invisible Gaps

When humans talk, we operate at a surprisingly high level of informational entropy. We leave words out, use implicit references, and rely on emotional cues. We do this because we know our conversation partner can “fill in the gaps” — actively reducing information entropy through shared knowledge and situational awareness.

Machines, on the other hand, are like that recent acquaintance. They cannot guess what you meant. They do not understand your implicit references. They do not pick up on your emotional cues. As a result, we need to “pre-process” our context for them — transforming high-entropy information into low-entropy representations they can understand.

This transformation represents the central effort of context engineering: the work you need to invest to make machines understand what you mean when you say “that place.”

The Decades-Long Journey: Learning to Talk to Machines

Contrary to popular belief, context engineering was not born with ChatGPT. It has existed for over 20 years, continuously evolving as machines become smarter.

Era 1.0: The Age of Forms and Clicks (1990–2020)

Think of the first phase as that extremely formal conversation with a strict bureaucrat. You need to fill out every field, follow every protocol, and use exactly the right words. Nothing can be left implicit.

In this era, dominated by ubiquitous computing and graphical interfaces, humans did all the heavy lifting. Designers acted as “intention translators,” converting complex human goals into structured, machine-readable formats. Clicks on menus, simple sensors, GPS coordinates — everything had to be explicitly defined and translated.

The Context Toolkit, developed by Anind Dey in the early 2000s, was a landmark of this time. It organized context through modular abstractions: Widgets captured sensor signals, Interpreters derived higher-level meanings, Aggregators integrated multiple sources. It was like teaching a child to recognize patterns — but every pattern needed to be carefully programmed.

Era 2.0: The Natural Language Revolution (2020-Present)

The release of GPT-3 in 2020 changed everything. Suddenly, machines started to understand natural language. It was as if that recent acquaintance had finally learned your cultural references and could keep up with more fluid conversations.

Now you can say “look up related documentation for me” and the system understands. You can send images, audio, and conversation history. The machine doesn’t just receive structured data — it interprets ambiguous intentions, fills in information gaps, and actively collaborates.

But here is the challenge: the more context we accumulate, the harder it becomes to manage. It’s like that friend who remembers every single detail of every conversation you have ever had — helpful, but potentially overwhelming.

The Three Pillars: Collect, Manage, Use

1. Collection: Capturing the Unsaid

In Era 1.0, we collected context through limited sensors: GPS location, time of day, mouse clicks. In Era 2.0, collection has expanded dramatically. Smartphones capture text, image, audio, location, and touch. Smartwatches monitor heart rate and movement. Brain-computer interfaces are starting to capture neural signals and emotional states.

The key is not the volume of data, but minimal sufficiency: collecting only what is necessary to support the task. And maintaining semantic continuity: preserving meaning, not just raw data.

2. Management: Organizing Memories

Imagine your personal memory. You don’t remember every second of every day with equal clarity. Some things stay in short-term memory and fade. Others consolidate into long-term memory. The most important ones you revisit and reinforce.

Modern systems adopt similar architectures:

• Short-term memory: Recent context with high temporal relevance, stored for quick access
• Long-term memory: Important information processed and abstracted for permanent storage
• “Self-baking”: The process of “baking” raw context into structured knowledge — like transforming episodic experiences into semantic wisdom

Claude Code, for example, maintains structured notes. Key information is periodically written out of the context window into external memory and retrieved when needed. This allows the agent to track complex tasks over thousands of steps, even after restarts.

3. Usage: Choosing What Matters

Not all available context is useful for the current task. In fact, empirical evidence shows that the performance of AI agents often decreases when context windows exceed about 50% of their capacity. Too much context can be as detrimental as insufficient context.

Modern systems employ various selection strategies:

• Semantic relevance: Searching for vector similarity to find related content
• Logical dependency: Tracking which information relies on previous steps
• Recency and frequency: Prioritizing recent or frequently accessed information
• User preference: Learning from feedback what each user values

It’s attention before attention — choosing what deserves to be observed.

Sharing Context: The Conversation Between Machines

When multiple agents need to work together, a new challenge arises: how to share context effectively? Imagine three friends planning a trip. Each has different information, and everyone needs to be on the same page.

Modern systems use three main approaches:

• Context embedded in prompts: An agent summarizes its thought process, and the next one reads and continues
• Structured messages: Agents exchange fixed-format messages (task type, input data, results)
• Shared memory: Agents write to and read from a common space, communicating indirectly

Each method has its trade-offs between clarity, efficiency, and flexibility.

The Horizon: Moving Towards Human-Level Intelligence

Looking ahead, we can foresee two more eras:

Era 3.0: Human-Level Intelligence

In this phase, machines will understand context as humans do. They will pick up on social cues, emotional states, and rich environmental dynamics. They will be able to sustain truly natural conversations, acting as knowledgeable and effective colleagues. It will be like talking to that close friend — where “that place” is instantly understood.

Era 4.0: Superhuman Intelligence

In the final, speculative phase, machines will not just understand our context — they will transcend it. With a “god’s eye view,” they will comprehend our intentions more deeply than we do ourselves. Instead of passively adapting to human-defined context, they will actively construct new contexts for us, revealing hidden needs and guiding our thinking.

We already see glimpses of this: in Go, professional players learn new, superhuman strategies from the AI. Machines become sources of insight and inspiration.

Challenges on the Path

The journey faces significant hurdles:

• Limited collection: We still rely heavily on explicit user input
• Scaling storage: How to organize ever-growing context scalably?
• Imperfect understanding: Current systems struggle with complex logic and relational information
• Performance bottlenecks: Transformer architectures suffer from quadratic complexity in long contexts
• Context selection: How to filter noise and keep only what is relevant?

A New Form of Digital Existence

In the age of context-centric AI, this idea gains new computational meaning: individuals are increasingly defined not by their physical presence or conscious activity, but by the digital contexts they generate — their conversations, decisions, and interaction traces.

These contexts can persist, evolve, and even continue to interact with the world through AI systems long after a person’s departure. The human mind may not be uploaded, but the human context can be — transforming context into an enduring form of knowledge, memory, and identity.

Conclusion: Bridging the Gaps Together

Context engineering is fundamentally about reducing the distance between how humans think and how machines process. It is about transforming that formal, bureaucratic conversation into something closer to a chat between friends — where much can be left between the lines because shared understanding exists.

As machines become smarter, they take on more responsibility for interpreting, reasoning, and even constructing context. The human effort gradually decreases. We are moving toward a future where machines will not only understand us but will illuminate and expand our understanding of ourselves.

And perhaps, in that future, when you just say “that place,” the machine will not only know which restaurant you are talking about — it will also suggest the perfect dish based on your current mood, remember the last time you were there, and book a table at the ideal time.

Because in the end, context is not about data. It is about shared meaning. And we are just starting to teach machines to share ours.

Post inspired by this paper.

Igor Comune.

If you’re a data scientist you must read this.

22/oct/2025 marks my last day using Windows.

Intro

In my last post I described my why I choose Linux Ubuntu.

Defenestrate Windows

Why I switched to Linux as a data scientist — and how I split work and play without losing my mind.

Today is my very last day using Windows. This post exists to convince you to adopt a penguin too — or at least to make you stop reflexively reaching for a Windows key every time something breaks.

Quick summary for the lazy

If you work with data, there are three practical needs you’ll hit quickly: GPU support for CUDA, stable system for development, and the freedom to tweak things without paying a ransom in reboot time. My choice: Ubuntu 24.04 LTS as my primary work distro, Cachy OS on a separate SSD for gaming. Dual-boot, separate SSDs — because mixing a workstation and a gaming rig on the same OS is asking for drama.

Which distro and why

• Ubuntu — best all-around for deep learning and robotics simulations (Isaac Sim, Gazebo, Unity/Unreal integrations tend to behave better here). If you’re doing deep reinforcement learning, this is the safe, boring, reliable choice — the distro equivalent of buying a Swiss watch.
• Pop!_OS — very friendly for data scientists, excellent out-of-the-box hardware support and smoother GPU setup for many users. Great if you don’t need the extra robotics stack stability.
• Mint / Fedora — also fine, but each has quirks (repos, packaging, or driver handling) that may require extra fiddling.
• Gaming distros (Cachy OS, Nobara, Bazzite) — tuned for low-latency gaming performance and bleeding-edge kernels. Great for frames-per-second; not ideal for ML workloads unless you enjoy troubleshooting drivers at 2 AM.

Yes, all the above can game fine — but gaming-focused distros often prioritize kernel and scheduler tweaks that aren’t always ideal for reproducible ML experiments.

My setup

Main SSD — Ubuntu 24.04 LTS
I run Ubuntu 24.04 LTS on my main SSD because — bluntly — it’s the best compromise between up-to-date kernel/drivers and distribution-level stability for heavy ML/robotics work. CUDA and NVIDIA tooling have first-class documentation and installers for mainstream Linux distros (Ubuntu is the most commonly supported target), so you waste less time beingfriend driver hell and more time training models. If you plan to run Isaac Sim, many NVIDIA robotics/simulation stacks expect Linux (and Ubuntu variants are the path of least resistance).

Why that matters practically: when you need GPU-accelerated training, you want a distro where the CUDA toolkit and NVIDIA drivers install cleanly, where Docker + NVIDIA Container Toolkit work predictably, and where community help (and compatibility notes) exist in abundance. Ubuntu ticks those boxes.

Second SSD — CachyOS (gaming only)
CachyOS is an Arch-based, performance-tuned distro with kernels and scheduler tweaks focused on responsiveness and gaming packages pre-bundled. It offers gaming meta-packages, optimized kernels (BORE / Zen variants) and compilation optimizations — good for squeezing extra FPS and low-latency behaviour out of your hardware. It’s perfect as the “nightclub” drive where you chase frame rates, but it’s not the tidy production environment you want for reproducible ML experiments.

Why separate SSDs?
Because mixing “work” and “play” on the same OS is an invitation to accidental kernel changes, driver experiments and repo upgrades that break reproducibility. Physical separation = deliberate separation of concerns. Analogy: one SSD is your museum (careful, catalogued), the other is the rave. Keep them physically apart.

Tools that made the switch bearable

LinuxToys — one-click installers and post-install scripts
LinuxToys is a curated collection/GUI that automates many common installs and tooling across distros (it helps install gaming launchers, dev tools and device helpers). Instead of copy-pasting 17 commands and hunting missing dependencies, LinuxToys bundles common flows into sane installers. That’s why I used it to provision Steam/Proton tools, utilities, and a few convenience optimizations. (Yes, it’s like having someone hand you pizza at 2AM.)

WinBoat — Windows apps inside a containerized VM
WinBoat runs a Windows VM inside Docker and exposes Windows apps as windows on your Linux desktop using RDP/RemoteApp tricks. For local Power BI Desktop usage (which has no native Linux client), WinBoat is a pragmatic, user-friendly way to run the real Windows app inside an automated VM/container — less fiddly than hand-rolling a VM and more integrated than full remote desktop. It’s not magic; it automates the engineering so you can keep working.

Caveat (the boring but important part): WinBoat depends on the underlying Windows image, QEMU/Docker plumbing and FreeRDP integration. Sometimes you’ll hit edge cases (driver passthrough, RDP quirks) that require tweaking or updated releases. The community is active and improving it, but keep the expectation: it’s an engineered workaround that solves a problem, not an official port.

Power BI in Linux — the pragmatic truth

Short answer: there is no official Power BI Desktop for Linux. Microsoft provides Power BI Desktop for Windows; the “official” Linux route is to use the Power BI Service (web) for many tasks, or run the Desktop inside a Windows VM/container for local, offline or heavy authoring workflows.

1. Power BI Service (web) — use the browser-based service for most sharing and many modelling tasks. Best for collaboration and zero local-app headaches. Requires cloud/tenant licensing.
2. Run Power BI Desktop in a Windows VM — local VM (VirtualBox/QEMU) or containerized VM solutions like WinBoat are realistic for offline/advanced desktop features. This gives you native Power BI Desktop behaviour at the cost of a Windows guest.
3. Hosted/cloud Windows VM — spin up a Windows VM in the cloud and RDP in. Useful if your workstation is weak but latency is acceptable.
4. Third-party “hosted desktop” services — providers that run Power BI Desktop in the cloud and stream the UI (less control, but fast to start). (Examples exist; vet security/cost.)

So yes: your WinBoat solution is pragmatic and commonly used in the Linux community precisely because Microsoft hasn’t shipped a native Linux client. You’re not being lazy — you’re being realistic.

Final wisdom

If you’re serious about data science, machine learning, and reproducible engineering, Linux is the practical upgrade — not a hobbyist detour. You get:

• Better developer tooling and native shells, package managers, and scripting that actually scale with automation.
• Cleaner GPU and container workflows for CUDA, Docker + NVIDIA Container Toolkit, and a proper environment for sim/robotics stacks (Isaac Sim, Gazebo, ROS) that expect Linux.
• Less vendor lock-in for your workflows: you control updates, packages, kernels, and can reproduce environments with Docker/containers and infrastructure as code.
• Performance options for gaming (if you want them) without sacrificing a stable, reproducible workstation — just put your games on a different SSD and sleep well.

Is there effort involved? Of course — but the effort pays back in reliability, control and fewer “works on my machine” tragedies. People who think Windows “just works” for every heavy data workflow are usually replaying an old advert in their head. Linux will make you a better engineer: you’ll read logs, fix the cause, and automate the fix — and you’ll stop blaming tools for your process gaps.

Igor Comune.

For the past six years, I’ve been using Windows primarily for Power BI and gaming (mainly on Steam, Squad). Everything else would arguably run better on Linux.

There are two key reasons behind my shift — worth noting. WSL and Google Colab have been amazing, but as I started experimenting with computer vision and reinforcement learning, the limitations became glaring… and quickly.

Microsoft’s AI obsession hasn’t helped either. From December 2022 until now, it feels like AI is everywhere — Cortana, CoPilot… AI, AI, AI. Meanwhile, their “wonderful” operating system seems almost like an afterthought.

Add to that the constant friction between GitHub projects designed for Linux versus Windows, Windows performance issues, the need to debloat, libraries that work exclusively on Linux, and version mismatches for tools like TensorFlow, PyTorch, CUDA, Gymnasium, and PySpark… it was chaos. Honestly, the reasons to switch are endless — you probably know them better than I do.

It was time to defenestrate Windows.

Two things finally pushed me over the edge:

1. Linux Gaming subreddit — an amazing, knowledgeable community.
2. Winboat — arguably the most important reason.

I’ve been following the Linux Gaming subreddit for at least two years, and their collective experience finally convinced me. The last major obstacle? Power BI. It simply couldn’t run on Linux — any Linux distribution — despite every tool the community had created, from Wine to Bottles.

And then, I was randomly scrolling down youtube videos and this video (in Brazilian Portuguese) from the youtuber called “psygreg” showed up.

And then, throught this video I met the wonderful https://www.winboat.app/. This app is basically a Windows running on a Docker, inverse of the WSL.. the new LSW where you run anything

And in that very moment, I realized that I could FINALLY and DEFINITELY make the full switch to Linux.

At first, I thought it would be as simple as picking my favorite distro and that’s it… but if you’re thinking the same, slow down. Let me tell you — it’s not nearly that simple.

Investigation

Linux is the kernel; distributions are essentially Linux optimized for different purposes.

For the sake of time, I’ll explain how I chose my distro — something that might be useful if you work with data.

First, to choose a distro, you need to know your requirements. If you’re a data scientist working with computer vision and reinforcement learning, like I’ve been doing recently, pay attention.

Main apps/programs: Python, CUDA, PyTorch/TensorFlow, Gymnasium, OpenCV, Unity, Unreal Engine, Gazebo, PX4, Steam, Epic Games, ROS, Isaac Sim, and Nvidia Omniverse.

I started with the basics: Ubuntu, Pop!_OS, and Mint. I also explored other distros, but many were either not as performant as they could be or extremely complicated, like Arch. That’s when I came across EndeavourOS and CachyOS.

Then began the comparison: Ubuntu, Pop!_OS, Mint, EndeavourOS, and CachyOS. Mint was almost immediately removed from the list, followed by EndeavourOS. That left Ubuntu, Pop!_OS, and CachyOS.

Let’s pause and analyze the choices a bit. Ubuntu and Pop!_OS are Debian-based, offering stability and user-friendliness. CachyOS, on the other hand, is a performance-oriented Arch-based distro: friendly, faster, and lighter.

Due to stability concerns I found online — particularly for reinforcement learning — I decided to remove Pop!_OS.

As tempting as CachyOS was, offering higher FPS and pre-tuned performance, Isaac Sim presented serious issues on it, along with some minor problems related to reinforcement learning. Unfortunately, that meant CachyOS was out.

The final contenders? Ubuntu 22, Ubuntu 24, and, reluctantly, CachyOS… which could have been the performance winner, but compatibility won this round.

The Final Choice

The final choice was guided by technical compatibility and long-term stability.
Or Ubuntu 24.04 LTS presented the best balance between:

1. Performance in IA loads and simulation (Isaac Sim, Docker, CUDA).
2. Stability for professional and academic use.
3. Integration with robotics tools and graphics engines (Gazebo, Unity, Unreal).
4. Flexibility for gaming via Steam, no need for dual boot.
5. Works better with GeForce series 40 (my new laptop).

The result of the analysis was a rational decision: o Ubuntu 24 offers the most complete and coherent ecosystem for development, simulation and research in AI, combining industrial compatibility, NVIDIA support and production performance in a unique operational environment.

(I created 2 bootable USB’s: ubuntu 24 and CachyOS, because I’m still considering running most of the applications on docker)

Conclusion

While I haven’t fully transitioned from Windows to Linux just yet, the decision is made and the groundwork is laid. Once I complete the switch, I’ll share my firsthand experience, detailing the challenges, surprises, and benefits of moving entirely to Ubuntu 24, so anyone considering the leap can see what it’s really like.

Igor Comune.

If you’ve seen that error, you know it’s not a mysterious Power BI bug — it’s a rate-limit. Google is politely (and correctly) preventing you from hammering their API.

Below I’ll be blunt and practical: what actually causes this, what I usually check first, and a small, reusable Power Query (M) pattern I built that solved it for me: a generic retry wrapper with exponential backoff you can drop into the Advanced Editor and reuse for any Google Sheets URL.

🎁 Get friend links for all of our 1000> Power BI learning articles here 🎁

The problem (fast)

• 429 = you made too many requests in a short time, or your project hit a quota.
• Common real-world triggers: multiple reports/dashboards hitting the sheets api, aggressive scheduled refresh intervals, parallel queries asking different ranges at the same time, or many services using the same API key/credential.
• The immediate effect: Power BI fails to fetch the content and the dataset refresh stops.

If the 429 is temporary (bursts of traffic, intermittent), the right approach is a retry with exponential backoff — not blind retries that double-down on the problem. Think of it like tapping someone on the shoulder vs. repeatedly shouting in their ear.

The solution I used (drop-in Power Query M)

Below is a compact, generic snippet: you just set Url and MaxRetries, paste into Advanced Editor, and the query will call GoogleSheets.Contents with retries and exponential backoff. It’s intentionally minimal — no transforms, no extra stages — because you asked for a general wrapper.

Power BI Desktop > Transform Data > Advanced Editor

let
    // === CONFIGURE HERE ===
    Url = "https://docs.google.com/spreadsheets/d/YOUR-SHEET-ID-HERE/", // replace with your sheet URL
    MaxRetries = 5, // maximum number of attempts (0 = no retries)
   // Generic retry function with exponential backoff
    Retry = (operation as function, optional maxRetries as nullable number) =>
    let
        maxR = if maxRetries = null then 5 else maxRetries,
        fn = (attempt as number) =>
            let
                tryRes = try operation(),
                result =
                    if tryRes[HasError] then
                        if attempt < maxR then
                            // wait 2^attempt seconds, then try again
                            (Function.InvokeAfter(()=> fn(attempt + 1), #duration(0,0,0, Number.Power(2, attempt))))()
                        else
                            // exhausted attempts: raise original error
                            error tryRes[Error]
                    else
                        tryRes[Value]
            in
                result
    in
        fn(0),
    // Call GoogleSheets.Contents wrapped with retry
    Source = Retry(()=> GoogleSheets.Contents(Url), MaxRetries)
in
    Source

I also tested a different function that creates a “delay” between the requests.

let
    // === CONFIGURE HERE ===
    Url = "https://docs.google.com/spreadsheets/d/YOUR-SHEET-ID-HERE/", // replace with your sheet URL
    MaxRetries = 5,
    InitialDelaySeconds = 10,      // sleep before the first attempt (seconds)
    FixedDelaySeconds = 5,         // fixed sleep between attempts (seconds)
    UseExponentialBackoff = true,  // if true, adds 2^attempt seconds to the wait

    // Sleep helper: blocks for given seconds using Function.InvokeAfter
    Sleep = (secs as number) =>
        let
            s = Number.RoundDown(secs),
            _ = if s > 0 then Function.InvokeAfter(()=> null, #duration(0,0,0, s)) else null
        in
            _,

    // Retry function with initial sleep, fixed sleep, and optional exponential backoff
    Retry = (operation as function, optional maxRetries as nullable number, optional initialDelay as nullable number, optional fixedDelay as nullable number, optional useExp as nullable logical) =>
    let
        maxR = if maxRetries = null then 5 else maxRetries,
        initD = if initialDelay = null then 0 else initialDelay,
        fixD = if fixedDelay = null then 0 else fixedDelay,
        useE = if useExp = null then true else useExp,

        fn = (attempt as number) =>
            let
                // initial delay only before first attempt
                _init = if attempt = 0 and initD > 0 then Sleep(initD) else null,

                // try the operation
                tryRes = try operation(),

                // on error and still have retries left -> compute wait and retry
                result =
                    if tryRes[HasError] then
                        if attempt < maxR then
                            let
                                backoff = if useE then Number.Power(2, attempt) else 0,
                                delay = fixD + backoff,
                                _sleep = if delay > 0 then Sleep(delay) else null
                            in
                                // recurse to next attempt after sleeping
                                fn(attempt + 1)
                        else
                            // no retries left -> surface original error
                            error tryRes[Error]
                    else
                        // success -> return value
                        tryRes[Value]
            in
                result
    in
        fn(0),

    // Call GoogleSheets.Contents wrapped with the enhanced Retry
    Source = Retry(()=> GoogleSheets.Contents(Url), MaxRetries, InitialDelaySeconds, FixedDelaySeconds, UseExponentialBackoff)
in
    Source

IMPORTANT

Both of the soluctions solved the SERVICE side of the problem but it just MITIGATED the DESKTOP problem.

Igor Comune.

👏🏻 Clap 🔎 Follow 📩 Subscribe ✍🏻Comment

🎁 Get friend links for all of our 1000> Power BI learning articles here 🎁

Don’t forget to subscribe to

👉 Power BI Publication

👉 Power BI Newsletter

and join our Power BI community:

Microsoft Power BI Masterclass | Linktree

Power BI Masterclass Article Classification

Level: Beginner

Category: Integration

Tags: Tips & Tricks, Integration

Power BI Service: Google Sheets Error (429): Too Many Requests was originally published in Power BI on Medium, where people are continuing the conversation by highlighting and responding to this story.

I started this project long time ago, but just recently I have enough time to end this little project.

Thank-you OWI for updating Squad to Unreal Engine 5, I no longer can play it because I’m an owner of a GeForce 1650 and now I have enough time to code.

Background

Few months ago I got into the world of Reinforcement Learning and somehow I ended up trying to use Godot and I had a wonderful idea…

Create an AI Agent that supports game development in Godot.

Godot is Open Source, so, I decided that the entire project should be open source.

But now, I’m leaving Godot for Unity and I’m publishing this idea because I don’t want to die, the project is fully open source.

Setup

• Python 3.13
• Dolphin3.0-Llama3.1–8B-Q3_K.gguf
• text-generation-webui
• LangChain
• PyTorch
• Chroma DB
• Github Repo (https://github.com/IgorComune/godot_rag_project)

The Project

I won’t use Godot for Reinforcement Learning anymore, I wanna start using Unity instead, so, if you want to take over this project or fork it, feel free to do so, everything in this project is open source.

This project was made to challenge all my data skills. Let me explain you…

1. I scraped all links of the Godot Documentation
2. After that I scraped all content from all links of its docs.
3. Vectorize it into a ChromaDB
4. Created a script to test the ChromaDB query
5. Downloaded the WebUI, download a LLM model and finally
6. Integrated everything together

The project is still a mess, but it’s working perfectly. You can run it locally using only Open source solutions from web.

In order to create the virtual environment I used Anaconda which makes it easier to deal with multiple python versions, envs and lib versions.

1. Gitclone this repo: https://github.com/IgorComune/godot_rag_project.git
2. Create your virtual environmental using Python 3.13.
3. pip install -r requirements.txt
4. Run godot_scraper_links.py (it will create a list of all pages to be scrapped
5. Run godot_scraper_content.py (it will download all of the content from all of the links, if you don’t want to do it, just unzip the file in godot_docs/pages.zip
6. Run txt_to_vector.py in order to vectorize all of the content downloaded
7. pip install -r requirements/portable/requirements.txt — upgrade
8. in your terminal type: python server.py — extensions rag

the step 8 will run this file:

from langchain.vectorstores import Chroma
from langchain.embeddings import HuggingFaceEmbeddings

# Caminho relativo do seu Chroma DB
PERSIST_DIRECTORY = "godot_chroma_db"

# Inicializa embeddings
embedding_function = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")

# Inicializa Chroma DB
vectordb = Chroma(persist_directory=PERSIST_DIRECTORY, embedding_function=embedding_function)

# ----------------------------
# Função que modifica o input
# ----------------------------
def input_modifier(prompt, state=None, is_chat=False):
    try:
        docs = vectordb.similarity_search(prompt, k=3)
        context_text = "\n".join([doc.page_content for doc in docs])
        print("[RAG DEBUG] Contexto adicionado ao prompt:")
        print(context_text)
        return f"{context_text}\nPergunta: {prompt}\nResposta:"
    except Exception as e:
        print(f"[RAG Extension] Erro ao consultar o Chroma DB: {e}")
        return prompt  # fallback: prompt original

# ----------------------------
# Configura parâmetros da extensão
# ----------------------------
params = {
    "display_name": "RAG Extension",
    "is_tab": False,  # fica no final da aba Text generation
}

# ----------------------------
# Hooks do WebUI
# ----------------------------
def setup():
    """
    Retorna os hooks que o WebUI vai chamar.
    """
    return {
        "input_modifier": input_modifier,
    }

This file will initiate the WebUI server along with the extension (tool) right above. It will sends your input inside the chat area to the query that queries the ChromaDB and return the content to the LLM in order to answer you.

your input text > query chroma db > chroma db answer > send to llm > llm answers you

Because I’m moving to Unity to learn Reinforcement Learning I decided to create this little post to draw someone’s attetion that can save this project from demise. The project itself has lots of problems, like: mixture between portuguese and english, useless files, lack of standards… but you can save it ❤

I’ll probably start a similar project to do the same with Unity but now, integrating with MCP and CrewAI for research.

Igor Comune.

Inspired by my Master’s course, I decided to try to solve the Gymnasium’s Lunar Lander V3 problem by my own… and I did it!

The Project:

• WSL Ubuntu 22.04.5 LTS
• Python 3.10.12
• Github repo
• Gymnasium Lunar Lander

Objective

• Implement both Discrete and Continous agents to solve the lunar lander problem.

Intro

This project implements both discrete (DQN) and continuous (TD3) reinforcement learning agents for the LunarLander-v3 environment (Gymnasium).
It includes modular training and testing scripts, pretrained models, and automated video recording of agent performance.
Experiments are organized into separate folders for discrete and continuous agents, with outputs saved under `videos/` for easy review.

What Is Reinforcement Learning

Reinforcement Learning (RL) is a branch of machine learning where an agent interacts with an environment by taking actions, receiving feedback in the form of rewards, and learning a policy to maximize long-term return. The environment is typically modeled as a Markov Decision Process (MDP), where the next state and reward depend only on the current state and action, not on the full history. This agent–environment interaction loop forms the foundation of RL, enabling agents to learn behaviors through trial and error.

My Main Difficulties in This Project

Looking back, my main struggles followed a pretty clear progression:

1. Environment setup
   At first, I had trouble just configuring the environment. I assumed I could pass parameters like gravity or wind_power directly into LunarLander-v3, but the actual API behaved differently. I had to slow down and read the official documentation instead of guessing.
2. Understanding the TD3 architecture
   Implementing TD3 was way harder than I expected. I didn’t fully understand what each part of the agent was doing — the DDQN, the replay buffer, the soft updates, and the delayed policy updates. This was more of a conceptual block than a coding one.
3. Mixing discrete and continuous control
   When I built my first training script, I mixed logic for both discrete and continuous agents. Later I realized that was a mistake — I only needed continuous control. The codebase became unnecessarily complicated just because I didn’t make that decision early.
4. Parameter consistency between training and testing
   My testing script broke because I was passing parameters (add_noise) that didn’t even exist in my TD3Agent class. That made me realize how easy it is to get training and testing scripts out of sync.
5. Managing action noise
   I also got confused about noise. I tried to control it at initialization, when in fact it should be handled inside the act method. This created a mess when I wanted noise for training but a clean policy for testing.
6. Practical details like video recording
   Even small things tripped me up, like recording videos with Gymnasium wrappers. I kept running into warnings and overwritten files just because I stopped the program mid-episode or reused the same output path.

The Discrete Problem

Solving the LunarLander-v3 discrete problem means training an agent to control a lunar module using four discrete actions (do nothing, fire left engine, fire main engine, fire right engine). The goal is to land safely between the landing pads, maximizing the total reward.

I started with the basics — the discrete version of the problem. Since I had no prior experience with Reinforcement Learning, even the fundamentals felt overwhelming. Concepts like how the learn function or the replay_buffer actually fit into a Markov Decision Process, or what “exploration vs. exploitation” really meant, were far from obvious. In short: the basics were already hard. So, I decided to call in my GenAI “mob” to help me push through this challenge.

After a lot of trial and error, I finally managed to train my first agent successfully. That felt like a milestone. But as soon as I tried to test it, I got stuck again. It wasn’t until I gave Claude this paper and compared its explanations with my own scripts that some concepts finally started to click.

One example was almost embarrassingly simple: I had forgotten to set the variable greedy=True during testing. Because of that, my agent kept choosing random actions instead of using the policy it had learned. To put it in plain words: I had imported the brain (the trained agent), but when it was time to perform, I basically told it, “ignore everything you know and just act randomly.”

With that paper in hand and a constant back-and-forth with my GenAI helpers, I finally pieced things together. I can’t say I mastered Reinforcement Learning, but I did build a solid enough understanding of the core concepts to succeed in this challenge.

• All the codes are here Github repo.

If you want to see the videos of the training and testing check the links below:

Reinforcement Learning — Gymnasium — Lunar Lander v3 — Discrete Agent Train

Reinforcement Learning — Gymnasium — Lunar Lander v3 — Discrete Agent Test

Do not forget to subscribe to my channel because there will be huge projects in reinforcement learning coming in soon to this channel that will be announced here on Medium.

The Continous Problem

Solving the LunarLander-v3 continuous problem means training an agent to control the lunar module with continuous thrust values for the main and side engines. The objective is to achieve a smooth and safe landing between the pads while maximizing the total reward.

My main challenge was moving from discrete to continuous control in the LunarLander environment. Switching from DQN to TD3 wasn’t just a matter of replacing the agent — it required rethinking the entire training loop.

First, I had to properly configure the environment in continuous mode, which took some trial and error. Then came the bigger hurdle: adapting the code. DQN logic didn’t translate to TD3, so I had to rewrite how actions, replay memory, and updates were handled.

Another issue was separating training and testing pipelines. During training, the agent needs exploration and network updates; during testing, it should run deterministically without noise. Mixing these up gave me inconsistent results until I made a clean separation.

Finally, I struggled with TD3 hyperparameters (batch size, tau, gamma, policy delay) and with setting up proper logging and checkpoints. Without consistent metrics, it was hard to know if the agent was truly improving.

In short: the hardest part wasn’t TD3 itself, but integrating it correctly — getting the environment setup right, restructuring the code, and keeping training and testing strictly apart.

If you want to see the videos of the training and testing check the links below:

Reinforcement Learning — Gymnasium — Lunar Lander v3 — Continuous Agent Train

Reinforcement Learning — Gymnasium — Lunar Lander v3 — Continuous Agent Test

In short, my difficulties were a mix of conceptual gaps (understanding TD3), architectural mistakes (mixing discrete/continuous, parameter mismatch), and practical issues (environment setup, noise handling, video recording).

Looking back, this project was much more than just solving LunarLander — it was about learning how to think in terms of Reinforcement Learning, dealing with frustration, debugging conceptual mistakes, and realizing that even the “small” details (like noise or video recording) can derail an experiment if ignored. The discrete and continuous problems pushed me to rework my mental models, and I leave this challenge with a stronger foundation to face bigger ones ahead. A special thanks to FIAP, where I am currently pursuing my Master’s in Machine Learning Engineer, for being both the spark and the structure behind this journey.

Igor Comune.

https://www.linkedin.com/in/igor-comune/
https://igorcomune.medium.com/
https://github.com/IgorComune
https://www.youtube.com/@DroneRLArena

Every year, in the fall, there’s a special full moon called the Hunter’s Moon.
Long ago, this moon gave hunters extra light to find food before winter.

But maybe, just maybe, this moon also gives us a clue about something deeper — how different brains work, and how AI might change everything.

Every autumn, people in the Northern Hemisphere see a special full moon called the Hunter’s Moon. In ancient times, this moon helped hunters find food after sunset, before the cold winter arrived. It was a symbol of instinct, timing and survival.

But what if this old symbol also represents a modern way of thinking?
What if it connects to how our brains work — especially for those who struggle to fit into today’s “normal” world?

Let’s talk about Hunters vs. Farmers, ADHD, and how AI might finally bring balance to this ancient brain divide.

🧠 Two Brain Types: Hunter and Farmer

This idea comes from the book “ADHD — A Hunter in a Farmer’s World”, written by Edward Hallowell and John Ratey — both doctors who also have ADHD… as I am.

They say that ADHD is not a disorder. It’s actually a different kind of brain, one that used to be perfect for survival, but doesn’t always work well in today’s society.

They call it the Hunter’s Brain.

🏹 The Hunter’s Brain

Imagine our ancestors, thousands of years ago, living in nature.
Some people had to hunt for food: animals move fast, they’re unpredictable, and you must react quickly.
These people had brains that were:

• Fast-thinking: always scanning the environment.
• Easily distracted: but could hyperfocus in the right situation.
• Creative: connecting ideas quickly.
• Impulsive: but full of energy and passion.
• Restless: always moving, always curious.

These are the traits of what we now call ADHD.
But back then, this brain was perfect for surviving as a hunter.

🌾 The Farmer’s Brain

Later in history, humans started farming.
This was the beginning of stable, organized societies.
To farm, you need to:

• Plan long-term.
• Follow routines.
• Wake up early and do the same tasks every day.
• Wait patiently for results (like harvests).

So, the Farmer Brain evolved:

• Focused: good with repetition.
• Organized: detail-oriented.
• Patient: able to follow steps and stick to routines.
• Disciplined: preferring stability over change.

This brain type is great for modern life: school, offices, meetings, deadlines, calendars.

🧨 The Problem: A World Built for Farmers

Today’s society is mostly designed for Farmers.

In school, you have to sit still, focus, wait your turn, and follow instructions.
In work, you need to complete tasks on time, be organized, and stay consistent.

If you’re a Hunter, that’s exhausting.

You might feel like:

• You’re always “too much” or “not enough.”
• You’re lazy, messy, or bad at adulting.
• You have great ideas but can’t finish them.
• People don’t understand how your brain works.

So what happens?
Hunters get diagnosed with ADHD, take medication, and try to fit into the Farmer mold.
But it’s like forcing a cheetah to plow a field — it just doesn’t work.

🤖 The Game-Changer: How AI Helps Hunters Thrive

Now, something new is here: Artificial Intelligence.

And this technology might finally help Hunters shine, without needing to become Farmers.

Here’s how:

1. 🚫 No More Boring Work

Hunters hate routine.
They get bored easily with repetitive tasks like emails, reports, spreadsheets, or filling forms.

• Write emails for you
• Summarize long documents
• Analyze data
• Schedule meetings
• Transcribe and organize voice notes
• Translate text
• Generate images and presentations
• Clean and organize datasets
• Write and debug code
• Turn ideas into full blog posts or scripts (What I’m doing right now)

This means Hunters can focus on what they love — thinking big, solving problems, making new things — and let AI handle the boring stuff.

2. 🧭 Help With Focus and Memory

Hunters often struggle with executive function. That means:

• Forgetting tasks.
• Losing track of time.
• Getting lost in distractions.
• Struggling to plan and finish things.

But now we have:

• AI reminders that adjust to your habits.
• Smart assistants that help you plan step by step.
• Tools like Notion AI, Google Gemini, or ChatGPT that keep you organized.

It’s like having a Farmer friend in your pocket, keeping things in order while you chase ideas.

3. ⚡ Turning Fast Thoughts into Action

Hunters think fast.
They often have brilliant ideas but can’t turn them into action quickly enough. They get stuck.

AI tools help by:

• Turning rough notes into clean documents.
• Turning sketches into designs.
• Turning ideas into code.
• Speeding up content creation — blogs, social media, videos, etc.

Now, the Hunter’s fast brain becomes a superpower, not a weakness.

4. 🔀 Making Sense of the Chaos

Hunters live in mental chaos.
There are so many thoughts, ideas, emotions — it’s hard to organize them.

AI can help bring structure to this creative storm:

• Visualize plans.
• Organize brainstorms.
• Help write clear messages.
• Find patterns and insights in your thinking.

With AI, Hunters no longer need to slow down or become someone they’re not.
Instead, they can use AI to translate their brain into results.

🌕 The Hunter’s Moon Is Rising Again

The Hunter’s Moon once gave ancient people the light they needed to survive.
Now, AI is giving the same light to a new kind of hunter — the modern mind that thinks differently.

This isn’t a war between Hunters and Farmers.
It’s a new age where we finally recognize that not all brains work the same way — and that’s okay.

AI makes it possible to work with your brain, not against it.

• If you’re a Farmer, AI can help you be more creative and faster.
• If you’re a Hunter, AI can help you stay focused and finish what you start.

Together, we build a world where different minds can finally thrive.

💬 So, What About You?

• Do you feel like a Hunter — full of energy, ideas, and chaos?
• Or more like a Farmer — calm, focused, and organized?
• How is AI changing the way you work, think, or create?

Leave a comment — let’s talk.
Maybe we’re part of the same tribe, walking under the light of the Hunter’s Moon. 🌕

Partially written by AI.

Igor Comune.