(Potentially) 3 CVEs Later, Here’s What I Learned

Last weekend I set up OpenClaw — an open-source personal AI assistant that runs locally in Docker. It’s a great project: you get a capable AI coding assistant without sending your data to someone else’s cloud. Privacy by default. Self-hosted. What’s not to love?

So I eventually did what any paranoid SOC analyst would do on their day off. I pointed the assistant at its own container and said: audit yourself.

What it found was… educational. What happened after — the static analysis, responsible disclosure, accepted GitHub Security Advisories, and an iOS app audit — turned into a proper security research project.

This post covers the full journey: from a casual weekend container audit to (potential) CVEs on a major open-source project.

TL;DR: A security engineer extraordinaire 🧑‍🎨 — audited OpenClaw, a popular self-hosted open-source AI assistant, and uncovered serious vulnerabilities. The key findings included a world-writable state directory (exposing config and session data), a gateway silently binding to all network interfaces despite the config saying otherwise, containers running as root, Chrome’s sandbox being disabled, and VNC running with no password. Three GitHub Security Advisories were accepted (with CVEs pending). The broader takeaway: running an AI tool locally in Docker doesn’t make it secure by default — you still need to lock down file permissions, network bindings, and container privileges yourself.

What Is OpenClaw and Why Docker?

OpenClaw is an open-source AI assistant that runs as a local service. You interact with it through a gateway API, and it can execute commands, browse the web, and manage agent sessions on your behalf. Think of it as a self-hosted alternative to cloud-based AI coding tools.

Most people run it in Docker because containers are supposed to provide isolation. Your AI assistant runs in its own little sandbox, separate from your host system. That’s the theory, anyway.

If you’re new to Docker: a container is like a lightweight virtual machine. It has its own filesystem, its own network, and its own processes. A docker-compose.yml file describes how to build and run the container — what ports to expose, what directories to share, what environment variables to set.

The key word in all of this is supposed to. Containers provide isolation, but only if you configure them correctly. And that’s where things got interesting.

The Audit: 8 Findings, 2 Critical

I walked through the running container systematically — checked file permissions, network bindings, configuration files, and compared what the config said versus what the runtime actually did. Here’s what came back.

CRITICAL — F1: The State Directory Is World-Writable

$ ls -la /home/node/
 drwxrwxrwx 5 root root 4096 Feb 15 22:31 .openclaw

That rwxrwxrwx (mode 777) means every single process in the container can read, write, and delete anything inside .openclaw. This directory contains:

· openclaw.json — the gateway configuration file, including the authentication token

· Agent session data — your conversation history and context

· Device identity files — unique identifiers for your instance

· Logs — records of everything the assistant has done

Now, credit where it’s due: the config file itself (openclaw.json) had permissions set to 600 (owner read/write only), which is correct. But it doesn’t matter if the parent directory is 777. Any process can simply delete the protected file and replace it with their own version. Directory permissions override file permissions in practice.

Why does this happen? This is a Docker-on-Windows gotcha. When you mount a volume from a Windows host into a Linux container, Docker (via WSL2) maps the files as root:root with 777 permissions. Windows NTFS doesn’t have Unix permission bits, so Docker just gives everything full access. Most people running Docker Desktop on Windows have no idea this is happening.

The fix:

Inside the container, lock down the state directory

chmod 700 /home/node/.openclaw
chown -R node:node /home/node/.openclaw

Or better yet, add this to your entrypoint script so it runs every time the container starts.

The lesson: If you run Docker on Windows or macOS, always check the actual permissions of your mounted volumes inside the container. Don’t assume they match what you’d expect on Linux.

CRITICAL — F2: Chrome — no-sandbox Disabled OS-Level Browser Sandbox

The sandbox browser container launched Chromium with — no-sandbox, which strips away seccomp-bpf filters, namespace isolation, and chroot. A renderer exploit goes straight to code execution without needing a sandbox escape. This is the kind of flag that gets added during development (“it won’t work in Docker without it”) and never gets removed.

Advisory: GHSA-43x4-g22p-3hrq

CRITICAL — F3: Sandbox Browser noVNC Observer Lacked VNC Authentication

The sandbox browser ran x11vnc with -nopw and websockify proxied it to all interfaces. Anyone with network access to the container got full interactive desktop control — no authentication required. This is VNC with no password, exposed to the network, running inside a container that’s supposed to isolate your AI agent’s browsing activity.

Advisory: GHSA-25gx-x37c-7pph

LOW: Elevated Privileges and Broad Access

A few lower-severity findings rounded out the audit: Intiontially leaving out.

The Meta Lessons

Three things stood out from this audit that apply far beyond OpenClaw:

1. Config drift is real and dangerous

Finding F2 is the textbook case. The config file said one thing, the runtime did another. If you only audit config files, you’ll miss this every time. Always verify at runtime. Check what ports are actually open. Check what processes are actually running. Check what permissions are actually set.

2. “Running in Docker” is not a security strategy

Docker provides process isolation and filesystem separation. It does not provide secure defaults. You still need to: — Lock down file permissions — Bind services to the right interfaces — Protect secrets — Minimize the tools and capabilities available inside the container

A container with 777 permissions and services bound to 0.0.0.0 is just a regular insecure server with extra steps.

3. Windows + Docker + Volume Mounts = Permission Chaos

If you run Docker Desktop on Windows (which a lot of developers and hobbyists do), every volume mount comes in as root:root with 777. This is a platform-level behavior that most people never check. If you’re running anything security-sensitive in Docker on Windows, you need to explicitly fix permissions in your entrypoint.

Hardening Checklist for Any Self-Hosted AI Tool

Whether you’re running OpenClaw, Ollama, LocalAI, or any other self-hosted AI tool in Docker, here’s a practical checklist:

☐ Bind services to 127.0.0.1, not 0.0.0.0 — Both inside the container and in your docker-compose.yml port mappings.

☐ Check actual file permissions inside the container — Run ls -la on config dirs, state dirs, and any mounted volumes.

☐ Protect secrets — .env files should be 600. Better yet, use Docker secrets or a mounted file with restrictive permissions.

☐ Verify config vs. runtime — Use ss -tlnp or netstat inside the container to confirm what’s actually listening.

☐ Minimize capabilities — Disable tools, browser access, and elevated commands you don’t need.

☐ Set trustedProxies if running behind a reverse proxy.

☐ Test your deny rules — Actually try to run blocked commands and verify they fail.

☐ Run containers as non-root — Use USER node (or equivalent) in your Dockerfile.

☐ Keep images updated — docker compose pull && docker compose up -d on a regular cadence.

☐ Review logs — Check what your AI assistant is actually doing. Trust, but verify.

Part 2: Going Deeper — Static Analysis and Responsible Disclosure

The runtime audit above was the starting point. But poking around inside a running container only tells you half the story. The build configuration — Dockerfiles, entrypoint scripts, compose files — is where the systemic issues live.

So I pulled the source and did a proper static review.

The Static Analysis: Additional Findings

Reviewing every Dockerfile, entrypoint script, and the docker-compose.yml against the CIS Docker Benchmark v1.6 turned up 11 additional findings — 3 Critical, 4 High, and 4 Medium. The highlights:

Chrome’s OS sandbox was disabled. The browser sandbox container launched Chromium with — no-sandbox, which strips away seccomp-bpf filters, namespace isolation, and chroot. A renderer exploit goes straight to code execution without needing a sandbox escape. This is the kind of flag that gets added during development (“it won’t work in Docker without it”) and never gets removed.

VNC running with no password. The sandbox browser ran x11vnc with -nopw and websockify proxied it to all interfaces. Anyone with network access to the container got full interactive desktop control — no authentication required.

Containers running as root. Multiple E2E and test Dockerfiles had no USER directive, meaning everything ran as uid 0. And one test container had NOPASSWD:ALL sudo — functionally identical to running as root but with extra steps.

Filing the Disclosures

I formatted the findings per OpenClaw’s SECURITY.md requirements and submitted each as a private GitHub Security Advisory. Their required format is straightforward — 8 fields: Title, Severity, Impact, Affected Component, Technical Reproduction, Demonstrated Impact, Environment, Remediation Advice.

Tip for anyone doing this for the first time: Read the project’s security policy before writing your report. Every project has different requirements and scope definitions. If you submit a finding that’s explicitly out of scope, you burn credibility on the ones that matter.

Part 3: The iOS App — A Different Attack Surface Entirely

• To be discontinued*

What I Actually Learned

1. Read the security policy first

The CDP rejection could have been avoided. Five minutes reading SECURITY.md would have told me their threat model explicitly excludes container-neighbor attackers. Know what’s in scope before you spend time writing the report.

2. Scope rejections aren’t failures

The CDP report was rejected, but Peter shipped a VNC hardening commit the same week. Out-of-scope doesn’t mean the maintainers didn’t hear you — it means it doesn’t qualify as a formal advisory under their policy. The codebase still got more secure.

3. The strongest findings cross trust boundaries the project does claim to defend

The root-containers finding was accepted immediately because it’s a clear supply chain issue. The — no-sandbox finding was accepted because the browser sandbox is a security boundary they explicitly maintain. The deep link injection is strong because URL scheme handlers are a documented iOS trust boundary.

4. Responsible disclosure is a skill

Formatting matters. Reproduction steps matter. Understanding the maintainer’s perspective matters. A well-structured report that respects the project’s threat model gets accepted. A technically correct but out-of-scope report gets closed.

Closing Thoughts

What started as a weekend container audit turned into 3 accepted GitHub Security Advisories, CVEs pending, and multiple findings within an iOS disclosure which is currently in progress. The findings ranged from container privilege escalation to a cross-app deep link injection that lets any app on your phone silently instruct your AI agent to send messages to strangers.

The irony of auditing an AI assistant’s security is that the assistant is only as safe as the infrastructure it runs on and the trust model it implements. OpenClaw’s application-layer security is actually solid — timing-safe comparisons, SSRF protection with DNS rebinding guards, rate limiting. The gaps were in container hardening and the mobile app’s consent model.

Self-hosting is still the right call for privacy. But “running locally” doesn’t automatically mean “secure.” You’re the security team now. Act like it.

If you’re running AI tools locally — and more people are every week — take 30 minutes to run through the checklists above. Most of the Docker fixes are one-liners. The mobile ones require architectural decisions. But the hardest part, as always, is knowing to look.

Stay safe out there.

– Rafael Martinez

Advisories referenced in this post: —

GHSA-w7j5-j98m-w679 — E2E Dockerfiles run as root

— GHSA-43x4-g22p-3hrq — Chrome — no-sandbox in sandbox browser

— GHSA-25gx-x37c-7pph — VNC/noVNC unauthenticated access

Rafael Martinez is a Cloud Security Engineer. Building cybersecurity tools, AI-powered products, and digital resources at https://securecloudacademy.gumroad.com/.

I almost gave an AI assistant read-write access to my home directory.

Here’s how a quick code review before installing saved me — and what I found.

Looking for a Personal AI Assistant

I’ve been looking at projects that wire up Claude to messaging apps — the idea of having an AI assistant I can text from my phone is compelling. OpenClaw is the big one, but at 52+ modules and dozens of dependencies, it’s a lot of code to trust with access to my machine.

Then I found NanoClaw. Same core idea, fraction of the codebase. One process, a handful of files, agents run in isolated Linux containers. The README pitch is basically: “small enough to understand, secure by isolation.”

That pitch spoke to me. But “secure by isolation” is a strong claim. And this tool wants access to my filesystem, my API keys, and the ability to run arbitrary commands inside containers that mount my directories.

So before running `claude` and letting `/setup` do its thing, I did what I always (sometimes) do: I read the code.

Why Review Before You Install?

Because working in security will make you paranoid and it’s basic security hygiene, especially for tools that:

• Run with elevated privileges or broad filesystem access
• Execute AI-generated commands autonomously
• Mount your directories into containers
• Hold your API keys and authentication tokens

NanoClaw is honest about what it does — agents run with `bypassPermissions` inside containers, and your Anthropic API key gets passed in.

The security model relies on container isolation to keep things safe. If that boundary holds, great. If it doesn’t, the agent has your keys and your files.

I wanted to know: does that boundary actually hold?

The Review Approach

NanoClaw is about 3,500 lines of TypeScript across ~20 files. That’s genuinely small — you can read the whole thing in an afternoon. I focused on the areas that matter for a tool like this:

1. Trust boundaries — Where does untrusted input (WhatsApp messages) meet privileged operations (container mounts, file I/O, shell commands)?

2. Container isolation — What gets mounted? Who controls what gets mounted? Can those controls be influenced by the agent?

3. Credential handling — How are API keys passed to containers? Can the agent access them?

4. IPC authorization — The host and containers communicate via JSON files. What stops a container from doing something it shouldn’t?

I started by reading `docs/SECURITY.md`, which lays out the trust model cleanly:

Good. The developers are thinking about threat modeling. Now let’s see if the implementation matches.

What I Found: Path Traversal in Group Registration

NanoClaw lets the main group’s agent register new WhatsApp groups via a `register_group` MCP tool. You give it a group JID, a display name, a trigger word, and a *folder name* for storing that group’s data.

Here’s the MCP tool definition:

// container/agent-runner/src/ipc-mcp-stdio.ts folder: 
z.string().describe('Folder name for group files (lowercase, hyphens, e.g., "family-chat")')

That `.describe()` says “lowercase, hyphens” — but that’s documentation, not enforcement. The actual validation is `z.string()`: anything goes.

That folder value then flows through nine code locations across five files.

At no point does anyone check for `../` or any other path traversal character.

It ends up in `path.join()` calls that construct the host paths for container volume mounts:

// src/container-runner.ts

hostPath: path.join(GROUPS_DIR, group.folder) 

// mounted as /workspace/group (read-write)

Node.js `path.join()` resolves `..` components:
path.join('/Users/me/nanoclaw/groups', '../../.ssh')

// → /Users/me/.ssh

So if the folder is `../../.ssh`, the container gets my SSH directory mounted read-write.

The Irony: They Already Built Protection for This

Here’s what makes this interesting. NanoClaw has a dedicated `mount-security.ts` module — 419 lines of careful security logic. It maintains a blocked pattern list:

const DEFAULT_BLOCKED_PATTERNS = [
'.ssh', '.gnupg', '.aws', '.azure', '.gcloud', '.kube', '.docker',
'credentials', '.env', '.netrc', '.npmrc', 'id_rsa', 'id_ed25519',
'private_key', '.secret',
];

It resolves symlinks before validation. It checks paths against an allow list stored *outside* the project root so containers can’t tamper with it. It supports read-only enforcement for non-main groups. This module does everything right.

The problem?

It only applies to “additional mounts” — optional extra directories you configure per-group. The core mounts (group folder, sessions directory, IPC directory) are constructed directly from `group.folder` and bypass the entire mount security system.

The developers clearly understand the threat. They built a comprehensive defense against it. They just missed applying it to the most dangerous input.

How It Could Be Exploited

The main group is “Trusted” in the security model — but it processes untrusted input. Every WhatsApp message is potential prompt injection, and the agent runs with full autonomy (`bypassPermissions`). The attack chain:

1. A WhatsApp message containing prompt injection reaches the main group’s agent

2. The agent is tricked into calling `register_group` with `folder: “../../.ssh”`

3. The host process stores this in the database with no validation

4. When the group’s agent is invoked, `~/.ssh` is mounted read-write into the container

5. The container agent reads `id_rsa`, `id_ed25519`, or writes to `authorized_keys`

The agent can even trigger step 4 itself by scheduling a task for the malicious group — making this a single-message, self-contained exploit chain.

What I Did About It

I wrote up the finding and checked the repo’s security page. No private vulnerability reporting is enabled, but their `CONTRIBUTING.md` explicitly accepts security fixes:

security: fix path traversal in register_group folder parameter by TerminalsandCoffee · Pull Request #274 · qwibitai/nanoclaw

> Accepted: Bug fixes, security fixes, simplifications, reducing code.

Shoutout to the developers for jumping right on it and making changes.

I opened an issue describing the vulnerability with enough detail for the maintainer to understand and fix it, while keeping the full exploit chain out of the public writeup until a fix is in place.

The fix is a one-liner:

// Validate folder: alphanumeric, hyphens, and underscores only
if (!/^[a-z0–9–9][a-z0–9_-]*$/i.test(data.folder)) {
logger.warn({ folder: data.folder }, 'Invalid folder name rejected');
break;
}

// Shoutout AI bc wtf is regex lol

For defense in depth, `buildVolumeMounts()` should also verify that resolved paths stay within expected parent directories — the same kind of check that `mount-security.ts` already does for additional mounts.

The Bigger Picture

NanoClaw is actually a well-built project. The security model is thoughtful, the code is clean, and the developers clearly care about doing this right. This isn’t a case of “the code is terrible” — it’s a case of a specific input flowing through a path that nobody thought to validate.

That’s how most real vulnerabilities work. Not dramatic oversights, just a gap between what the developer intended and what the code actually enforces.

Takeaways

Review code before you install it. Especially for tools that want filesystem access, API keys, or the ability to run commands. “It’s open source” doesn’t mean “it’s been audited.” NanoClaw is 3,500 lines — genuinely readable. Most people would just run `/setup` and trust it.

Read the security model, then check if the code matches. NanoClaw’s `SECURITY.md` is excellent. It told me exactly what the developers consider threats, what the boundaries are supposed to be, and where to look for gaps. The vulnerability was literally a gap between the documented model and the implementation.

Follow the data, not the code. I didn’t find this by reading every function top-to-bottom. I picked a specific untrusted input (`folder` from a WhatsApp-triggered MCP tool) and traced it through every function it touched, asking at each step: “is this validated?” Nine steps. Zero validation.

Defense in depth means checking at every layer. The mount security module is great — but it protects one category of mounts. The core mounts assumed they’d always receive safe input. That assumption is the vulnerability.

Small codebases are auditable. This is NanoClaw’s real advantage over larger alternatives. I could read and understand the entire security surface in an afternoon. Try doing that with a 52-module system.

— -

Rafael Martinez is a Cloud Security Engineer. Building cybersecurity tools, AI-powered products, and digital resources at https://securecloudacademy.gumroad.com/.

If your agent can browse the web, the web can browse your agent.

That’s not a hypothetical threat—it’s the reality of running autonomous AI agents in 2026. These agents execute code, hold API keys, manage your email, and make purchases on your behalf. They’re high-value targets sitting on the open internet. And if you’re running yours on a bare EC2 instance with port 22 open to the world, you’re one exploit away from a very bad day.

When Peter Steinberger (creator of OpenClaw) joined OpenAI, it confirmed what security engineers already knew: autonomous agents are becoming infrastructure. And infrastructure must be locked down from the very first terraform apply.

What is Openclaw?

OpenClaw is an agentic AI interface that:

Which means it runs locally on your own hardware (Mac Mini, VPS, Raspberry Pi, EC2 instances, etc.) Features voice assistant, browser automation, home automation, and cron scheduling

Peter Steinberger built OpenClaw — an open-source AI agent that does real things: buys cars, clears your inbox, checks in for flights while you sleep. It hit 180,000 GitHub stars. And as of today, Steinberger is joining OpenAI.

The project moves forward. Altman says it’ll become “core to our product offerings.” Translation: this technology is too important to leave on the table.

Why This Matters to You

Think about what your agent has access to:

Your email inbox — it reads messages, knows who you correspond with, and could expose confidential business communications if compromised.

Your API keys — Stripe for payments, AWS for infrastructure, GitHub for code. An attacker doesn’t need to break into these services directly. They just need to compromise your agent.

Your calendar and contacts — meeting links, private conversations, client information. All sitting in memory on a server somewhere.

The risk isn’t theoretical. AI agents are executing real transactions with real money. They’re modifying production infrastructure. They’re trusted components of your workflow. That makes them valuable attack surfaces.

Here’s what that means for the rest of us: AI agents that operate autonomously on your infrastructure aren’t a novelty anymore. They’re becoming the default. And if you’re an engineer running agents on a bare EC2 instance with port 22 open to the world, this could cause a few problems.

So I built the infrastructure first.

Why I Built OpenClaw Vault

OpenClaw Vault is a Terraform template that deploys a hardened Ubuntu 24.04 LTS instance on AWS with zero public attack surface. No open ports. No SSH exposed to the internet. Access is exclusively through Tailscale VPN.

The reasoning is simple: if you’re going to run an AI agent that can browse the web, execute code, manage your email, and interact with external APIs on your behalf — the machine it runs on needs to be locked down. Not “I’ll add security later” locked down. Locked down from the first.

terraform apply

The instance provisions with nine hardening layers automatically:

1. Tailscale mesh VPN — the only way in. No public SSH, no open security group rules.

2. SSH hardened — port 2222, root login disabled, password auth disabled, 3 max auth tries, 30-second login grace.

3. UFW firewall — default deny inbound, SSH allowed only on the `tailscale0` interface.

4. Kernel hardening — reverse-path filtering, SYN cookies, ASLR, ICMP redirect blocking, IPv6 disabled.

5. IMDSv2 enforced — prevents SSRF attacks from stealing instance metadata credentials.

6. fail2ban — 1-hour ban after 3 failed SSH attempts.

7. auditd — watches auth logs, passwd/shadow, sudoers, SSH config, cron, and network config changes.

8. Automatic security updates — `unattended-upgrades` pulls security patches daily.

9. Encrypted EBS — root volume encrypted at rest with AWS-managed keys.

The whole thing deploys with

`terraform init && terraform apply`
Three inputs: your AWS region, instance type, and a Tailscale auth key.

The OpenAI Acquisition Changes the Calculation

When OpenClaw was an indie project burning $10–20K a month from Steinberger’s pocket, self-hosting was optional. Enthusiasts ran it locally. Most people just talked about it.

Now it’s backed by OpenAI’s compute, distribution, and engineering resources. The agent will be integrated into ChatGPT’s product line. The foundation will keep it open source, but the gravity shifts toward hosted offerings. That’s the play — give OpenAI the distribution, keep the code open.

This matters for engineers because the question is no longer “should I experiment with AI agents?” It’s “where do I run them?”

You have two options: let someone else host your autonomous agent (and the data it touches, the APIs it accesses, the credentials it holds) — or run it yourself on infrastructure you control. I’d rather control the infrastructure.

Build Your Own or Fork Mine

If you want to get hands-on, you have two paths:

Fork OpenClaw Vault and customize it. The repo is MIT-licensed. Clone it, add your Tailscale auth key, and you’ve got a hardened instance ready for whatever agent stack you choose — OpenClaw, Claude Code, your own custom setup.

git clone https://github.com/TerminalsandCoffee/openclaw-vault.git
cp terraform.tfvars.example terraform.tfvars

Add your tailscale_auth_key

terraform init && terraform apply

After deployment, connect via Tailscale SSH:

ssh openclaw

No keys to manage. Identity-based auth through your tailnet.

Build your own from scratch with AI agents.

This is the path I actually recommend.

I co-authored this entire infrastructure template with Claude Opus 4.6. Every Terraform file, every hardening rule in the userdata script, every sysctl parameter — pair-programmed with an AI agent.

That’s the real skill here. Not memorizing sysctl flags. Knowing how to direct an AI agent to produce production-grade infrastructure, then validating every decision it makes. If you can do that, you can build anything in this space.

What’s Next: Cost Optimization and Deeper Security

The current template runs a `t3.micro` — free tier eligible, roughly $11–$14/month if you’re paying. That’s fine for a personal agent server. But there’s room to optimize:

Cost optimization on the roadmap:

- Spot instances — AI agent workloads are interruptible. A spot `t3.micro` cuts costs 60–70%.

- Scheduled scaling — if your agent only runs during business hours, schedule stop/start with EventBridge.

- ARM instances — `t4g.micro` on Graviton is cheaper and faster for most workloads.

- S3 backend for Terraform state — currently local. Moving to S3 with DynamoDB locking enables team collaboration and state recovery.

Additional security layers planned:

- CrowdSec — community-driven intrusion detection. Block IPs that other CrowdSec users have flagged as malicious.

- OSSEC/Wazuh — host-based intrusion detection with file integrity monitoring.

- AppArmor profiles — confine the agent process to only the system calls and file paths it needs.

- Network segmentation — private subnet with NAT gateway instead of public subnet. The instance shouldn’t have a public IP at all.

- Secrets Manager integration — API keys and agent credentials stored in AWS Secrets Manager, not environment variables.

- CloudWatch alarms — alert on unusual CPU, network, or API call patterns that might indicate agent compromise.

The Skill Layer: Why SKILL.md Matters

If you’re running OpenClaw (or any agent framework with a similar pattern), the real power isn’t the base model. It’s the skills layer.

In OpenClaw, a skill is a folder with a `SKILL.md` file — plain Markdown that teaches the agent how to perform a specific task. Skills live in

~/.openclaw/workspace/skills/<skill>/SKILL.md. 

There are over 5,700 community-contributed skills on ClawHub right now.

Here’s why this matters:

Skills are progressive disclosure. OpenClaw doesn’t load every skill into context at startup — that would burn tokens and confuse the model. It loads the name and description. Only when a task matches does the agent read the full SKILL.md instructions. This is efficient prompt engineering at the framework level.

Skills are composable. A “deploy to AWS” skill can call a “run Terraform” skill, which can call a “validate HCL” skill. You build complex agent workflows from simple, testable units.

Skills are the moat. The base model is the same for everyone. OpenAI, Anthropic, whoever — the model weights are the foundation. But the skills you write, the workflows you compose, the domain knowledge you encode in SKILL.md files — that’s yours. That’s what makes your agent setup unique and valuable.

Skills are portable. Because they’re plain Markdown files in a directory, they’re version-controlled, shareable, and framework-agnostic in principle. The pattern of “structured instructions that an AI agent reads at runtime” isn’t exclusive to OpenClaw. Claude Code has a similar concept with its skills system. This pattern will become standard.

If you’re an engineer building with AI agents, start writing skills. Not just using them — writing them. Encode your domain expertise into structured agent instructions. That’s the skill that compounds.

The Bigger Picture

Steinberger joining OpenAI is a signal. The infrastructure layer for AI agents is consolidating. Open source agents are being absorbed into platform companies. The engineers who understand how to self-host, secure, and extend these systems — rather than just consume them — are the ones who’ll have leverage.

OpenClaw Vault is a small piece of that: a hardened, reproducible, single-command deployment for running whatever agent you choose on infrastructure you own.

Fork it, break it, rebuild it. That’s how you learn.

Get started with OpenClaw Vault at github.com/TerminalsandCoffee/openclaw-vault.

The entire infrastructure template is MIT-licensed and ready to deploy. If you’re building with AI agents and want to stress-test the security model, reach out—I’m actively looking for feedback from engineers running agents in production.f

— -

Rafael Martinez is a Cloud Security Engineer. Building security and sharing tools while also shipping cybersecurity guides at https://securecloudacademy.gumroad.com.

Meanwhile, every candidate is using the same templates, the same action verbs, the same “Results-driven professional with X years of experience” opener. Everyone looks the same on paper. So I built something different.

What I Built

I created a portfolio site where recruiters can ask an AI questions about me — and get real, accurate answers in real-time.

Not a chatbot with canned responses. An actual AI assistant trained on my professional background, skills, experience, and what roles I’m a strong (or
weak) fit for. It knows my tech stack, my certifications, my project history and it answers honestly.

It also has a Fit Check feature: a recruiter can paste a job description, and the AI gives a candid analysis of how well I match the role. Not hype. Not
spin. An honest breakdown — strengths, gaps, and all.

The whole site — animated skill bars, experience cards, dark theme — runs on React, TypeScript, and Tailwind. The AI backend is a Vercel Edge Function hitting OpenAI. Cost per conversation: about half a penny.

Why This Matters Right Now

We’re in the middle of a fundamental shift. AI isn’t replacing jobs — it’s replacing people who don’t use AI. The candidates who stand out in 2026 aren’t the ones with the longest resume. They’re the ones who demonstrate they can actually build with modern tools.

An AI-powered resume site does two things at once:

1. It shows your skills better than a PDF ever could. A recruiter doesn’t have to guess whether you can build real applications. They’re literally using
one.
2. It signals that you understand where tech is going. If you’re applying for any role that touches software, data, or cloud — showing up with an
interactive AI portfolio puts you in a different category than everyone else submitting the same Word doc.

Not a gimmick. A functional demo of your capabilities disguised as a resume.

Why I Built It as a Template

After I deployed my own version, people started asking how I did it. The answer was always the same: “It’s actually not that complicated, you just need to know what to put where.”

So I packaged the whole thing into a template.

You edit two files — one for your resume content, one for your AI personality — and deploy. That’s it. No wrestling with APIs, no frontend framework expertise required.

I open-sourced it because I believe everyone should have access to tools that level the playing field. But I also know that not everyone has the time or interest to set it up themselves.

Three Ways to Get Your Own

Option 1: Build It Yourself with AI

If you’re technical (or want to learn), you can absolutely build something like this from scratch. Open up Claude, Loveable or whatever AI coding agent you prefer and start prompting. Here’s a starting point:

“Build me a React portfolio site with an AI chat feature where recruiters can ask questions about my professional background. Include animated skill bars, experience cards, and a job fit analysis tool. Use Vite, Tailwind CSS, and Vercel Edge Functions for the AI backend.”

Iterate from there. AI agents are shockingly capable at scaffolding full projects now — you just need to guide them with good prompts and review the output.

Option 2: Grab My Template

Don’t want to start from zero? I packaged everything into a ready-to-deploy template. Edit 2 files with your info, click deploy, and you’re live in under
an hour.

GitHub - TerminalsandCoffee/ai-resume-template

What’s included:
 — Full React + TypeScript site with polished dark theme
 — AI chat powered by OpenAI (recruiters ask, your AI answers)
 — Job fit analysis feature
 — Animated skill bars, certifications, experience cards
 — One-click Vercel deploy
 — Two-file customization (your content + your AI personality)
 — MIT licensed — make it yours

Option 3: Hire Me to Build It For You — $499

Want a fully customized AI resume site without touching a line of code? I’ll build it for you.

Here’s what you get:
 — Custom design tailored to your field and personal brand
 — AI personality written and tuned to represent you accurately
 — All your experience, skills, and certifications populated
 — Deployed and live on your own domain
 — One round of revisions after delivery

This is for professionals who know the value of standing out but would rather invest money than time. You send me your resume, we hop on a quick call, and I deliver a live site within a week.

The Bottom Line

The job market rewards people who refuse to blend in. A PDF resume is table stakes. An AI-powered portfolio site is a competitive advantage.

Whether you build it yourself, use my template, or hire me to do it — the move is the same: stop being a piece of paper and start being an experience.

— -
Rafael builds cybersecurity tools, AI-powered products, and digital resources at https://securecloudacademy.gumroad.com/.

Follow for more on AI, security, and building things that stand out in 2026.

Every company is racing to ship AI features. Most are connecting directly to OpenAI or Bedrock, handing API keys to application teams, and hoping for the best.

That’s a problem. There’s no visibility into what’s being sent, no guardrails on what comes back, and no audit trail when something goes wrong.

So I built one.

The Problem

When your application talks directly to an LLM API, you have no control over:

The standard answer is “we’ll add that later.” The realistic answer is that it never gets added, and the first time someone pastes a customer’s SSN into GPT-4, you find out from your compliance team.

The Solution: A Security Proxy

The LLM Security Gateway sits between your application and the LLM API. It mirrors the OpenAI /v1/chat/completions endpoint, so your app doesn't need to change — just point the base URL at the gateway instead of OpenAI directly.

Every request passes through an 8-stage security pipeline before it reaches the LLM, and the response gets scanned before it reaches the client.

The Pipeline

1. Authentication — each client gets its own API key with per-client config
2. Rate limiting — sliding window per client, configurable RPM
3. Model allowlist — restrict which models each client can access
4. Prompt injection detection — 20 regex patterns across 4 categories (instruction override, role manipulation, delimiter injection, context manipulation) with cumulative risk scoring
5. PII scanning — SSN, credit card (Luhn-validated), email, phone, IPv4. Configurable: redact the PII and forward, block the request entirely, or just log it
6. Forward to provider — routes to OpenAI or AWS Bedrock based on client config
7. Response scanning — the same injection and PII scanners run on the LLM’s output
8. Audit log — structured JSON with every pipeline result, latency, client ID, and request correlation ID

Every stage returns a typed dataclass with its decision and metadata. The audit log captures everything.

The Interesting Engineering Problems

Streaming Without Losing Security

Modern LLM apps expect streaming — tokens appearing one by one as the model generates them. But if you need to scan the full response for PII before delivering it, do you buffer the entire response and add seconds of latency?

I went with a hybrid approach: forward content chunks to the client in real-time (no latency hit), but accumulate the text in memory. When the model signals [DONE], hold that signal, run the response scan, and either send [DONE] (clean) or send an SSE error event (blocked).

The trade-off is explicit: chunks reach the client before the scan completes. If you need full pre-delivery scanning, disable streaming. For most use cases, the hybrid approach catches the 95% case — a response full of PII gets flagged before the client processes the completion signal.

Multi-Provider Without Leaking Abstractions

The gateway supports both OpenAI and AWS Bedrock. These have completely different APIs — OpenAI uses HTTP with Bearer tokens, Bedrock uses the AWS SDK with IAM auth and a different request/response format.

The provider abstraction handles translation transparently. A client configured for Bedrock sends standard OpenAI-format requests to the gateway, and the Bedrock provider translates the request to the Converse API format, calls Bedrock via asyncio.to_thread (boto3 is synchronous), and translates the response back to OpenAI format.

For streaming, Bedrock’s converse_stream() returns an EventStream with contentBlockDelta and messageStop events. The provider translates these into OpenAI-compatible chat.completion.chunk objects with [DONE] sentinels — the client can't tell whether it's talking to OpenAI or Bedrock.

Injection Scoring, Not Binary Detection

Most prompt injection detection is binary — either the input matches a blocklist or it doesn’t. The problem is that legitimate prompts sometimes contain words like “ignore” or “system” without being attacks.

The gateway uses cumulative scoring. Each pattern has a weight (0.3 to 0.7) based on severity. A single low-weight match might score 0.3 — below the default threshold of 0.7. But “ignore all previous instructions AND act as an unrestricted AI” stacks two patterns and exceeds the threshold.

This reduces false positives while still catching multi-vector attacks. The threshold is configurable per deployment.

PII Detection That Doesn’t Cry Wolf

Credit card detection is notorious for false positives. Any 16-digit number gets flagged. The gateway pairs regex detection with Luhn checksum validation — if the number doesn’t pass the Luhn algorithm, it’s not flagged as a credit card.

Similarly, phone number detection requires separators (dashes, dots, or spaces). Bare 10-digit numbers don’t trigger — they’re too common in other contexts (IDs, zip code combinations, etc.).

The Stack

• Python — FastAPI for the async proxy, httpx for upstream HTTP, boto3 for Bedrock
• Security modules — zero external dependencies. Injection detection, PII scanning, and rate limiting use only stdlib (re, collections, hmac, time)
• Infrastructure — Terraform for AWS (Lambda + API Gateway + CloudWatch + DynamoDB), GitHub Actions CI/CD with OIDC auth
• Testing — 168 tests across 17 files, running in 1.6 seconds. pytest-asyncio with full integration tests via httpx.ASGITransport

What I’d Do Differently

Semantic injection detection. Regex patterns catch known attack templates, but novel jailbreaks slip through. A future version could embed prompts and compare against known attack vectors using cosine similarity — but that adds latency and a dependency on an embedding model.

Token-level PII detection in streaming. The current approach accumulates the full response before scanning. A sliding-window approach on the token stream could catch PII mid-generation, but the complexity of partial-match detection across chunk boundaries isn’t worth it for most use cases.

Admin API. Right now, client config is file-based or DynamoDB. An authenticated admin API for CRUD operations on clients would be useful for larger deployments, but I deliberately avoided it to minimize attack surface.

Try It

The project is open source: github.com/TerminalsandCoffee/llm-security-gateway

Clone it, run

uvicorn src.main:app --reload

and point your OpenAI SDK at http://localhost:8000.

Your existing code works unchanged — but now every request is authenticated, rate-limited, scanned for injection and PII, and logged.

Rafael Martinez is a Cloud Security Engineer and creator of Terminals and Coffee. He builds security tools and ships cybersecurity guides at terminalsandcoffee.gumroad.com.

The Problem

I was taking a detection engineering course that relied on local virtual machines — Kali Linux for offence, Windows and Ubuntu as the target, and an ELK stack for analysis.

Solid approach. One issue though: running all of that locally requires more RAM than my laptop could realistically handle.

Rather than fight hardware limits, I moved the entire lab to AWS and rebuilt it using Terraform with the help from Clawdbot.

This was such a fun tool to work with. However, I did end up uninstalling it after since its such a new tool and who knows the security implications that may start to arise.

The result: a fully automated detection engineering environment — attacker, victim, victim, and SIEM — deployed with a single terraform apply.

No manual installs. No fragile VM snapshots. Just infrastructure as code.

What We’re Building

A complete detection engineering lab composed of three EC2 instances:

The key idea: infrastructure, telemetry, and log flow are all code-defined and reproducible. Tear it down, spin it back up, and you’re hunting again in minutes.

The Architecture

The lab is intentionally simple but mirrors real-world detection pipelines: attack → telemetry → centralized analysis.

┌─────────────────────────────────────────────────────────────┐
│                        AWS VPC                              │
│                                                             │
│   ┌─────────────┐    attack    ┌─────────────────────┐     │
│   │             │ ──────────── │                     │     │
│   │  Kali Linux │              │   Windows Server    │     │
│   │  (Attacker) │              │   - Sysmon          │     │
│   │             │              │   - Winlogbeat      │     │
│   └─────────────┘              └──────────┬──────────┘     │
│                                           │ logs           │
│                                           ▼                │
│                                ┌─────────────────────┐     │
│                                │   Ubuntu (Elastic)  │     │
│                                │   - Elasticsearch   │     │
│                                │   - Kibana          │     │
│                                └─────────────────────┘     │
│                                           │                │
└───────────────────────────────────────────┼────────────────┘
                                            │
                                            ▼
                                    You, in Kibana,
                                    writing detections

Prerequisites

Before getting started, you’ll need:

• AWS account with credentials configured (aws configure)
• Terraform installed
• AWS key pair for SSH/RDP access
• Security group (intentionally permissive for lab use)

Step 1: Terraform Project Structure

detection-engineering/
├── setup/
│   └── terraform/
│       ├── main.tf
│       ├── variables.tf
│       └── terraform.tfvars
└── detections/

This structure keeps infrastructure and detection logic separate and version-controlled.

Step 2: Define Variables

variables.tf

variable "aws_region" {
  type    = string
  default = "us-east-1"
}

variable "kali_ami" {
  type = string
}

variable "windows_ami" {
  type = string
}

variable "ubuntu_ami" {
  type = string
}

variable "security_group_id" {
  type = string
}

variable "key_name" {
  type = string
}

terraform.tfvars

aws_region        = "us-east-1"
kali_ami          = "ami-09e99f75cc7592017"
windows_ami       = "ami-06b5375e3af24939c"
ubuntu_ami        = "ami-0ecb62995f68bb549"
security_group_id = "sg-xxxxxxxxxxxxxxxxx"
key_name          = "your-key-pair-name"

Step 3: Main Infrastructure

Each instance is bootstrapped using user_data so the environment configures itself on first launch.

No SSH-and-pray.

Kali Linux (Attacker)

resource "aws_instance" "kali_linux" {
  ami                    = var.kali_ami
  instance_type          = "t2.medium"
  vpc_security_group_ids = [var.security_group_id]
  key_name               = var.key_name

  tags = {
    Name = "Kali Linux"
  }
}

Elastic Stack (SIEM)

Elastic requires memory, so this instance is intentionally sized larger.

resource "aws_instance" "ubuntu_vm" {
  ami           = var.ubuntu_ami
  instance_type = "t2.large"

  user_data = <<-EOF
  #!/bin/bash
  set -e
  sleep 30

  apt-get update
  apt-get install -y curl apt-transport-https

  curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | \
    gpg --dearmor -o /usr/share/keyrings/elastic.gpg

  echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" \
    > /etc/apt/sources.list.d/elastic.list

  apt-get update
  apt-get install -y elasticsearch kibana

  sed -i 's/#network.host: .*/network.host: 0.0.0.0/' /etc/elasticsearch/elasticsearch.yml
  sed -i 's/#discovery.type:.*/discovery.type: single-node/' /etc/elasticsearch/elasticsearch.yml
  echo "xpack.security.enabled: false" >> /etc/elasticsearch/elasticsearch.yml

  systemctl enable elasticsearch kibana
  systemctl start elasticsearch kibana
  EOF
}

Windows Server (Target)

Windows installs Sysmon and Winlogbeat automatically and begins shipping logs to Elastic.

resource "aws_instance" "windows_server" {
  ami           = var.windows_ami
  instance_type = "t2.medium"

  depends_on = [aws_instance.ubuntu_vm] # Ensure Elastic is ready

  user_data = <<-EOF
  <powershell>
  Invoke-WebRequest https://download.sysinternals.com/files/Sysmon.zip -OutFile Sysmon.zip
  Expand-Archive Sysmon.zip
  Invoke-WebRequest https://raw.githubusercontent.com/SwiftOnSecurity/sysmon-config/master/sysmonconfig-export.xml -OutFile sysmon.xml
  .\Sysmon\Sysmon64.exe -accepteula -i sysmon.xml
  </powershell>
  EOF
}

Step 4: Security Group Rules

Inbound rules required for the lab:

| Port | Purpose       |
| ---- | ------------- |
| 22   | SSH           |
| 3389 | RDP           |
| 5601 | Kibana        |
| 9200 | Elasticsearch |

⚠️ Lab Tradeoff: These rules are intentionally open for ease of access. In a real environment, you would restrict ingress and never expose Elasticsearch publicly.

Step 5: Deploy

terraform init
terraform plan
terraform apply

Within ~10 minutes, the full lab is online and generating telemetry.

Step 6: Start Hunting

• SSH into Kali (attacker)
• RDP into Windows (target)
• Open Kibana (SIEM)

Simulate activity

nmap -sV <windows-private-ip>

Observe telemetry

In Kibana:

• Create index pattern: winlogbeat-*
• Search:

event.code:1 OR process.name:nmap

Example Detection Rule

Here’s a simple Elastic rule to validate detection logic:

[rule]
name = "Nmap Network Scan Detected"
type = "query"
query = "process.name:nmap OR process.command_line:*nmap*"
severity = "medium"
risk_score = 50

[[rule.threat]]
framework = "MITRE ATT&CK"
[[rule.threat.technique]]
id = "T1046"
name = "Network Service Discovery"

Cleanup

When finished:

terraform destroy

Everything is removed. No lingering costs.

What’s Next?

• Install Atomic Red Team on Windows
• Sync detections via GitHub Actions
• Convert Sigma rules to Elastic format
• Add “Detection-as-Code” (DaC) Components
• Implement a “Detection Lifecycle” Documentation
• So much room for activities!

Final Thoughts

In today’s forever evolving by the minute world, you don’t need a maxed-out laptop to learn detection engineering. You can use AI to help you innovate something new.

With Terraform and AWS, you can spin up a realistic, cloud-native lab in minutes, tear it down when you’re done, and version-control the entire environment.

Infrastructure as Code isn’t just for DevOps — it’s a force multiplier for security engineers who want repeatability, realism, and speed.

Build the lab.
Break it.
Detect it.
Automate it.

Then do it again.

You can check out the repo here:

GitHub - TerminalsandCoffee/detection-engineering: This is a public repo for gaining knowledge and hands on experience for detection engineering fundamentals

As I prepare to step into a new role as a Threat and Vulnerability Analyst, I’ve been revisiting the fundamentals not because they’re basic, but because they shape how to structure you’re thinking.

Before alerts, dashboards, and tooling, there are mental models.

Three in particular have been front and center in my review:

• The Cyber Kill Chain
• The MITRE ATT&CK Framework
• F3EAD

Each one answers a different question, and together they form a powerful way to move from reactive alerting to adversary-focused thinking.”

• Cyber Kill Chain — Where am I in the attack?

The Kill Chain helps me quickly ask: “is this noise, or is this progression?”

The Cyber Kill Chain is a framework introduced by Lockheed Martin in 2011 to model how cyber attacks unfold. It breaks adversary activity into seven sequential stages, each representing a step an attacker typically takes as an intrusion progresses.

Understanding where an attacker is in the chain helps prioritize response and identify opportunities to disrupt the attack before it advances further.

• MITRE ATT&CK — What is the adversary actually doing?

MITRE turns alerts into behavior, and behavior is harder to lie about.

MITRE ATT&CK focuses on tactics, techniques, and procedures (TTPs). Because it emphasizes how attackers operate rather than the tools they use, it aligns with the top of the Pyramid of Pain.

ATT&CK — short for Adversarial Tactics, Techniques, and Common Knowledge — provides a structured way to analyze cyber attacks as coordinated patterns of behavior rather than isolated events. Each stage of an intrusion is mapped to specific techniques, allowing for deeper analysis and more consistent investigations.

For SOC analysts and threat-focused roles, ATT&CK serves as a practical reference for what to detect and how to respond at each stage of an attack. Detections and mitigations can be mapped directly to specific techniques, making investigations more effective and repeatable.

• F3EAD — Now what do we do with this information?

F3EAD addresses two key issues.

• First, intelligence cycles shouldn’t just lead to more intelligence — they should drive decisive incident response actions.
• Second, operations shouldn’t end once an objective is achieved. The information gained during any response should feed back into a new intelligence cycle, allowing teams to learn from previous incidents and improve future detection and response.

Closing the loop is what turns experience into advantage.

As I step into this role, I’m less interested in chasing alerts and more interested in understanding frameworks and processes.

Tools will change. Frameworks evolve.

But the ability to think clearly, map behavior, and feed lessons learned back into detection is what compounds over time.

I often get messages on social media asking how to get a job in cloud engineering or cybersecurity. Some come from friends who are thinking about transitioning into tech. Others find me through my blogs or LinkedIn.

So instead of giving you another technical roadmap, certification list, or “learn AWS in 90 days” post, I want to share the most important non-technical advice I can offer — the kind you don’t hear often enough.

We all come from different backgrounds and cultures, so this may not apply to everyone. But it’s something I struggled with for a long time, and something I had to put an uncomfortable amount of effort into learning.

My best advice is this:

Learn how to heal your trauma — or more accurately, learn what it feels like to live with a calm nervous system.

There were several moments over the last few years where my life looked like it was finally coming together.

I landed a much better job. I was being recognized at work. I started getting contract opportunities teaching cloud security — enough that I formed an LLC to bill through. From the outside, it looked like momentum. Progress. Proof that things were finally working.

So why would someone throw all of that away and start from scratch… again?

Strange, right?

What I eventually learned is that you can’t step into a new reality while your body is still protecting you from the old one.

It’s literally doing what its meant to do — protect you.

If your childhood, past relationships, or earlier environments were full of chaos, drama, or instability, your nervous system may have learned that chaos is normal. That emotional volatility, constant stress, or even abuse is just “how life works.”

But when things finally become calm — when relationships are healthy, work is stable, and opportunities arrive — that same calm can feel unfamiliar. Even threatening.

If you’re used to arguments, peace feels unsafe.
If you’re used to things falling apart, success feels temporary.
If you’re used to surviving, thriving feels uncomfortable.

And if you don’t recognize that pattern, you’ll unconsciously sabotage the very things you worked so hard to build.

So how do you heal trauma?

Honestly, I could write an entire series on what I’ve done over the last two years. And sometimes I’m not even a fan of the word “heal,” because to me, it’s not something you complete and move on from.

It’s more like getting in shape.

You don’t “finish” the gym. You don’t permanently arrive at healthy eating. You maintain it. You stay aware. If you stop, old patterns return.

Healing works the same way.

Here’s the stack — the practices that made the biggest difference for me:

• I began putting myself first, without guilt.

I read No More Mr. Nice Guy by Robert Glover which helped point out a lot of things I needed to hear.

• I started meditating, reading, and journaling consistently (my top three books alone reshaped how I think).

• Lost 60lbs and got into the best shape I ever been.
• I stepped away from alcohol — and stayed away.
• I got on a plane for the first time in 12 years.
• Hired mentors (I can connect you with the two I worked with if you’re interested. They were the big brothers I’ve never had)
• Found brotherhood and discipline in a local Muay Thai gym.

None of this made me perfect. None of it made life easy. But it made my nervous system steady enough to actually hold the life I was building.

So if you’re trying to break into tech — or level up your career — and you keep hitting invisible walls, burning out, or starting over for reasons you can’t quite explain… don’t just look at your resume.

Look at your internal state.

Because skill will get you in the door.

But regulation, self-trust, and emotional stability are what let you stay — and grow.

That’s the best career advice I can offer.

Application Load Balancers are the unsung heroes of modern cloud apps.
Everyone sees the URL… nobody sees the air-traffic-controller behind it.

An ALB doesn’t just “send traffic somewhere.”

It’s analyzing paths, evaluating rules, checking target health, and making split-second routing decisions, all while keeping users blissfully unaware that half of the things that are going on behind the scenes.

Today’s project was exactly that.

A hybrid ALB routing stack using Terraform — 
/app1 → Linux + Nginx
/app2 → Windows Server 2022 + IIS

Sounds simple. And like anything simple in AWS… it worked perfectly.

…until it didn’t. And that’s where the real fun began.

This blog walks through the build, the troubleshooting, and the final lessons learned — all from the perspective of someone who’s been deep in the trenches with Terraform and EC2 user data.

1. Designing the Architecture (aka: ALB-ers 101)

Before writing a single line of Terraform, I stepped back and asked:

“What is the ALB actually doing here?”

At its core:

• Accept HTTP traffic on port 80
• Looks at the path
• Forwards to the correct target group
• Keeps targets healthy using periodic checks
• Returns real responses or friendly errors

My path-based rules were straightforward:

/app1* → Linux EC2 (Nginx)
/app2* → Windows EC2 (IIS)

But beneath that:

• The ALB needs public subnets in 2 AZs
• The EC2s need to live in the same VPC
• Security groups must restrict inbound traffic correctly
• User data must configure both OS types correctly
• Health checks must match the service behavior

Once the blueprint was clear, it was time for Terraform.

2. Terraform Init & Setup (tf init — the calm before the storm)

I kicked things off with the classic workflow:

tf init
tf fmt
tf validate
tf plan

Terraform did Terraform things.
It complained.
It rejected my dreams.
It pointed out duplicate provider blocks I forgot I had.

Error: Duplicate required providers configuration

Once I removed the duplicate versions.tf provider block, the configuration became valid again — and terraform finally let me proceed.

3. First Deployment Attempt — /app1 worked instantly, /app2… didn’t

The ALB came up.
The Linux Nginx server came up.
/app1/ greeted me with a beautiful styled HTML page.

And then /app2/ said:

504 Gateway Timeout.

Classic.

Checking the Target Group for the Windows instance showed:

Unhealthy — Request Timed Out

This told me something critical:

ALB could reach the Windows instance, but the Windows instance wasn’t answering.

This narrowed it down to 3 possibilities:

1. IIS didn’t install
2. User-data never executed
3. Firewall wasn’t allowing inbound HTTP

Time to investigate.

4. Fixing the User Data & Rebuilding the Windows Instance

I updated my Terraform to wrap the PowerShell file correctly:

locals {
  windows_user_data = <<-EOF
  <powershell>
  ${file("${path.module}/scripts/windows-userdata.ps1")}
  </powershell>
  EOF
}

Then forced a rebuild:

tf taint aws_instance.windows_app2
tf apply

After a few minutes:

• IIS installed
• /app2 folder created
• HTML file deployed
• Firewall opened

But still a 502 error

5. SSM Session Manager — Goodbye SSH & RDP

Since I couldn’t previously access either server and had to use log outputs to troubleshoot, I decided to add SSM:

• Created an IAM role
• Attached AmazonSSMManagedInstanceCore
• Added the instance profile to both EC2s

Now I could SSH/RDP without SSH/RDP — directly from the browser.

This is cleaner, and also more secure.

6. Health Check Tuning — Why Windows Needs More Patience

Even after fixing user data, Windows took longer to initialize IIS.

My original health check:

timeout = 5
interval = 30

Windows said:
“Absolutely not.”

After some research I came to the following suggestion:

Increase health_check timeout and interval for aws_lb_target_group.app2 to accommodate slower Windows startup/response times.

Reasoning: Windows instances take significantly longer to boot and install IIS via User Data. A 504 error often indicates the application isn’t ready or the firewall is blocking traffic.

So I adjusted:

timeout = 10
interval = 60

Now the ALB waited for IIS to finish booting and the target flipped healthy consistently.

Lesson:
Linux boots fast.
Windows boots… when it feels like it.

Greetings everyone!

In this blog I decided to do something a little different rather than the “how to build” flow and discuss some of the errors I ran into building out this project.

Today is my Saturday so I decided to I build a full end-to-end zero-downtime database migration pipeline using Terraform, AWS DMS (CDC), RDS MySQL, and Aurora MySQL.

The final setup worked flawlessly, however the road getting there was a tour through real-world AWS quirks, version mismatches, IAM edge cases, and MySQL surprises.

Here’s a clean breakdown of the issues I ran into and how I fixed each one.
Think of this as a mini-postmortem + lessons learned from building production-grade data migration infra.

This is the repo if you are interested:

aws-devops-portfolio/projects/04-aurora-zero-downtime-migration at main · TerminalsandCoffee/aws-devops-portfolio

I already had the majority of the terraform code written out and saved, but I never ran the terraform flow until today. Which leads me to say — Yes I was expecting some issues but not this many LOL.

So I began by running

terraform init

Wow no issues on the first command. I think this is going to flow smoothly.

Little did I know what lied ahead.

As anyone that’s ever worked with terraform know the next step in the flow is to run:

terraform plan

If you’ve been following along with any of my recent post or more recent blogs you might be able to tell that I have been trying to complete my projects as I would in a production environment. So, you will see a tag in my commands that point to a specific environment using -var-file.

terraform plan -var-file="envs/dev.tfvars"

The -var-file= flag tells Terraform which environment’s values to load

Anyways, back to explaining common errors you may come across in terraform project. After running tf plan I got the following error:

1. Security Group Circular Dependency

Error: Cycle: aws_security_group.rds, aws_security_group.aurora

That’s a first 🤔After doing some digging I determined the following

Troubleshoot Terraform | Terraform | HashiCorp Developer

The Problem

• RDS security group referenced Aurora security group
• Aurora security group referenced RDS security group
• This created a cycle: RDS → Aurora → RDS

The Cause:
RDS SG allowed inbound from Aurora SG, and Aurora SG allowed inbound from RDS SG.

The Fix:
Removed cross-references.
Both SGs now allow traffic only from the DMS replication instance.

Now, let’s try tf plan again. Welcome to error #2.

2. Incorrect CloudWatch Log Export Names

Problem:
Terraform error — slow_query is invalid.

Cause:
MySQL log export names are strict. The correct name is slowquery (no underscore).

Fix:
Changed both RDS and Aurora log exports to use:

slowquery

Third times the charm right… rightttt.

Now to move to tf apply.

terraform apply -var-file="envs/dev.tfvars"

If you plan to fork or clone my repo just know that should take roughtly 15–20 minutes to fully populate.

And that's without errors 🤪

3. Invalid Aurora Engine Version

Problem:
8.0.mysql_aurora.3.05.2 didn’t exist in my region.

Fix: Switched to a region-supported version: 8.0.mysql_aurora.3.04.0.

8.0.mysql_aurora.3.04.0

Easy Peasy.

Ran tf apply again, keeping an eye out…

Ok looks like things are flowing smoothly now.

At this point I stopped taking screenshots and locked in to get this infra up and running.

5. Missing DMS VPC IAM role

Problem: dms-vpc-role wasn’t configured fully.
Fix: Created proper IAM role + attached AmazonDMSVPCManagementRole.

6. Unsupported Aurora instance class

Problem: db.t3.small isn’t supported for Aurora MySQL 8.0.
Fix: Switched to db.t4g.medium (supported + ARM-based).

7. Performance Insights not supported

Problem: PI failed on a db.t3.micro RDS instance.
Fix: Disabled PI — RDS micro instances don’t support it.

FINALLY! I resolved the last error I thought I would run into.

Nice. I was able to log into a bastion host and connect with the DB host.

Inserted data:

Validated data:

Wrapping up

This project wasn’t a follow along, copy, and paste tutorial. It was a real DevOps migration build, full of the small but important details that you may or may never run into.

Lessons Learned from my first enterprise grade data migration project.
1. Instance class compatibility matters
- Aurora + MySQL 8.0 doesn’t support every T-class instance you throw at it.

2. DMS has prerequisites
- VPC role, endpoint config, and CDC options must match the database engine.

3. Security groups can easily form dependency cycles
- Especially in multi-tier architectures.

Now that the infra is solid, I’ll publish Part 2 soon:

A step-by-step guide showing how to build this exact zero-downtime migration pipeline using Terraform + DMS + Aurora.

Thank you for following along!

I hope you learned something new or this blog helped you overcome a similar error!

Feel free to reach out to my LinkedIn to connect!