Aaqa Ishtyaq

What AI Infrastructure Actually Needs from MicroVMs

Unknown — Sun, 12 Apr 2026 00:00:00 +0000
I have been thinking about AI infrastructure from the compute side more than the model side.
There is a lot of noise around AI infrastructure right now.
Most of it is about models.
The part I care about is everything around the model: where work runs, how fast sandboxes start, how isolated they are, how they reach private tools, how you measure usage, and how you keep the whole thing understandable.
That is where microVMs get interesting.
In this post, I want to keep the discussion grounded in the kind of systems work that actually shows up when you build these platforms.
AI workloads are not just training jobs</h2>
When people say "AI infra," they often picture giant GPU clusters.
That is real, but it is only one slice.
A lot of useful AI work looks more like this:
short-lived agent sandboxes</li>
code execution environments</li>
retrieval or tool-calling workers</li>
notebook-like interactive sessions</li>
batch jobs that need strong isolation</li> </ul>
Those workloads want a different set of platform traits.
A simple example is a code interpreter sandbox. It needs to boot fast, run untrusted code in isolation, reach a few internal tools, then disappear cleanly when the task is done.
What these workloads need</h2>
In plain terms, they need:
- strong isolation</li>
- fast startup</li>
- reproducible environments</li>
- private networking</li>
- simple ingress when needed</li>
- usage metering</li>
- enough observability to explain cost and latency</li> </ul>
 That is a pretty good match for microVM-based compute.
 Why containers are not always enough</h2>
 Containers are great. I use them a lot.
 But there are cases where a stronger boundary is worth the extra machinery:
 - running untrusted user code</li>
 - isolating agent tasks from each other</li>
 - giving each workload a cleaner kernel boundary</li>
 - making network identity and lifecycle more explicit</li> </ul>
 MicroVMs sit in a useful middle space. They are lighter than full traditional VMs and stronger than "just another container on the host."
 That middle space is attractive for AI products that need speed and isolation at the same time.
 Fast startup matters more than people admit</h2>
 A lot of AI workloads are bursty.
 An agent wakes up, does work, calls tools, maybe spins up a second task, then disappears. A user opens an interactive environment and expects it to feel ready now, not after a long cold boot. A background job fans out for a few minutes and then goes quiet.
 That means startup time is not a side metric.
 If you want microVMs to fit these products, you need:
 
 image preparation off the hot path</li>
 clone or snapshot paths</li>
 warm slots for common environments</li>
 scheduling that knows which nodes can activate fast</li> </ul>
 That is not AI magic. That is platform discipline.
 Why this is different from classic web workloads</h2>
 These workloads are often:
 
 burstier</li>
 more isolated</li>
 more expensive when they go wrong</li> </ul>
 That changes what you care about. Startup, teardown, and usage accounting all matter more.
 Private networking is the sleeper requirement</h2>
 AI workloads often need to reach private things:
 
 internal APIs</li>
 databases</li>
 vector stores</li>
 queues</li>
 company tools</li> </ul>
 That means the compute environment cannot just be isolated. It has to be connected in a controlled way.
 This is where a microVM platform needs a real private network story, not just egress to the public internet.
 Metering matters because AI costs move fast</h2>
 One weak spot in a lot of AI platforms is cost visibility.
 A request fans out into a few workers, maybe a browser sandbox, maybe a code executor, maybe a retrieval task, and suddenly nobody is sure what the real billable footprint was.
 A microVM platform can help here if it tracks usage cleanly:
 
 how long the machine lived</li>
 what shape it had</li>
 what resources it consumed</li>
 what path it took through the platform</li> </ul>
 That does not solve all billing problems, but it gives you a real foundation.
 Teardown matters too. If these workloads start fast but linger forever after the useful work is done, the bill still gets ugly.
 The shape I find compelling</h2>
 This is the platform picture I keep coming back to:
 user request | v control plane | +--> warm microVM sandbox | +--> private tools / data over internal network | +--> usage, logs, traces, cleanup</code></pre> The machine is isolated. The startup is fast. The network is private. The lifecycle is visible. That is a strong base for AI workloads. The real point</h2> I do not think "AI infrastructure" needs a totally separate class of systems. I think it needs better compute primitives. MicroVMs become interesting when they stop being a novelty and start acting like those primitives: reliable</li> quick to activate</li> easy to meter</li> easy to reason about</li> safe enough for hostile or messy code</li> </ul> If you can give people that, a lot of AI product shapes get easier to build. That is why I find this work interesting. The model gets the headlines. The compute platform decides whether the product feels real. That part deserves more attention than it gets. AI products do not just need models. They need fast, isolated, measurable compute.


Scheduling Dense Compute Without Lying to Yourself
Unknown — Sat, 11 Apr 2026 00:00:00 +0000
I have been thinking a lot about scheduling lately, especially once warm starts and reservations enter the picture.</p>
Schedulers are easy to make impressive in a slide deck.</p>
They are much harder to make honest.</p>
You can pack nodes aggressively, overcommit a little, and brag about utilization. Then a few bad placement calls land on the wrong hosts and the whole thing feels brittle.</p>
That is why I care less about flashy scheduling algorithms and more about honest inputs.</p>
In this article, I want to focus on that part.</p>
The first rule</h2>
Do not schedule from fantasy.</p>
That means your scheduler needs real inputs:</p>

current capacity</li>
reserved capacity</li>
ready slots</li>
machine shape</li>
node health</li>
whether the node should take work</li>
</ul>
If any of those are stale or vague, "smart" placement becomes random placement with extra steps.</p>
Why free CPU is not enough</h2>
A node can have free CPU and still be a bad target.</p>
Maybe it is unhealthy.
Maybe it is still recovering.
Maybe it has capacity on paper but not the right prepared images.
Maybe it is already holding reservations for requests that have not committed yet.</p>
This is why a better scheduler tracks more than free resources.</p>
It needs to know the difference between:</p>

cold capacity</li>
reserved capacity</li>
hot capacity</li>
</ul>
Those are not the same thing.</p>
Here is a very simple example:</p>
node A has more free CPU, but only cold create capacity</span></span>
node B has less free CPU, but already has a matching ready slot</span></span>
for latency, node B is the better answer</span></span></code></pre>
That is why raw free capacity is not enough.</p>
The reservation model fixes a lot</h2>
One thing I like about reservation-based placement is that it forces the scheduler to act with discipline.</p>
Instead of firing real create requests at multiple nodes, it can:</p>

rank candidates</li>
reserve one</li>
commit to one</li>
activate a ready slot when available</li>
</ol>
That cuts wasted work and keeps placement cleaner.</p>
It also gives the system a way to talk honestly about partial ownership of resources before the machine fully exists.</p>
Bad scheduler inputs</h2>
The scheduler gets weird quickly if any of these are stale:</p>

node health</li>
reservations</li>
ready-slot counts</li>
image availability</li>
</ul>
At that point, even a clever ranking function starts making dumb choices.</p>
Dense packing needs guardrails</h2>
I am not against dense packing or measured overcommit.</p>
I am against pretending they are free.</p>
If you want to pack nodes tightly, you need:</p>

clear machine classes</li>
known resource envelopes</li>
health gating</li>
fast rollback when activation fails</li>
good observability around pressure</li>
</ul>
Otherwise density turns into support load.</p>
Scheduling is a product decision too</h2>
This part gets missed a lot.</p>
The scheduler shapes the product.</p>
If placement is sloppy:</p>

starts get slower</li>
noisy nodes get noisier</li>
users hit surprise failures</li>
ready slots get wasted</li>
</ul>
If placement is sharp:</p>

fast paths stay fast</li>
node failures are easier to contain</li>
prepared capacity gets used well</li>
the product feels responsive</li>
</ul>
That is why I treat scheduler inputs like user-facing behavior, not background math.</p>
What I want from a scheduler</h2>
I want the system to answer a few simple questions well:</p>

which nodes are healthy enough to take work?</li>
which nodes can do a cold create?</li>
which nodes can do a warm activation right now?</li>
which node is already close to the needed image and shape?</li>
</ul>
That is enough to get good results without pretending the platform has perfect foresight.</p>
The lie to avoid</h2>
The lie is telling yourself the scheduler is smart because it has a ranking function.</p>
A ranking function over stale or incomplete state is not smart. It is decorative.</p>
Good scheduling starts with good state.</p>
That is why so many "scheduler" problems are really state-distribution problems, health-model problems, or lifecycle-model problems in disguise.</p>
Fix those first.</p>
Then the placement code gets a lot better, even if the algorithm itself stays pretty simple. A scheduler is only as good as the state it believes.</p>


Observability for a MicroVM Platform
Unknown — Fri, 10 Apr 2026 00:00:00 +0000
I love logs, but logs alone have not been enough for the kind of machine lifecycle work I have been doing.</p>
You do not really know a platform until something slow or broken happens.</p>
That is when the nice diagrams stop helping.</p>
A request takes five seconds instead of two hundred milliseconds. A machine gets stuck during startup. A route exists, but traffic still fails. A node looks healthy until you ask one slightly better question.</p>
That is when observability earns its keep.</p>
In this post, I want to talk about the kind of visibility I actually want from a microVM platform.</p>
Logs alone are not enough</h2>
Logs matter. I want them. I want structured logs, and I want machine IDs in them.</p>
But logs alone are a rough way to understand a distributed machine lifecycle.</p>
You still end up asking:</p>

which service started the action?</li>
which node handled it?</li>
which transition was slow?</li>
how long did rootfs prep take?</li>
did networking finish before the guest timed out?</li>
</ul>
That is why traces and metrics matter too.</p>
Different signals answer different questions.</p>
What I want to see</h2>
For a microVM platform, I want visibility at a few levels:</p>

request level: what API call came in?</li>
machine level: what happened to this machine over time?</li>
node level: is the host healthy and under pressure?</li>
service level: which component is slow or failing?</li>
</ul>
If those layers do not connect, debugging turns into archaeology.</p>
The fields I never want to miss</h2>
There are a few fields I always want available when I look at a request or a machine:</p>

machine id</li>
node id</li>
request id</li>
transition name</li>
</ul>
Without those, even good logs and traces start to blur together.</p>
The machine is the unit that matters</h2>
The cleanest organizing idea is to treat the machine as the thread through the system.</p>
Every log, metric, and span that can reasonably attach to a machine should do it.</p>
That way you can follow one path:</p>
API request</span></span>
   -> placement</span></span>
   -> worker FSM</span></span>
   -> rootfs prep</span></span>
   -> network setup</span></span>
   -> guest boot</span></span>
   -> proxy exposure</span></span></code></pre>
Without that thread, you get disconnected facts. With it, you get a story.</p>
Why spans help so much</h2>
A machine FSM is a perfect place for spans.</p>
Each transition already has a name and timing boundary. That means the system can show you not just that startup was slow, but where it was slow.</p>
For example:</p>
reserve_node: 8ms</span></span>
prepare_rootfs: 420ms</span></span>
setup_network: 95ms</span></span>
wait_for_guest_agent: 1.8s</span></span></code></pre>
Now you have something real to act on.</p>
Without that, "create was slow" is just a complaint.</p>
That also gives you a real debugging loop:</p>
create request looks slow</span></span>
trace shows rootfs prep is normal</span></span>
guest readiness wait is huge</span></span>
look inside guest startup path next</span></span></code></pre>
That is a much better place to be than guessing between five different services.</p>
Local collection matters</h2>
I prefer a node-local collector for a simple reason: it keeps the system robust.</p>
Each host can emit:</p>

OTLP logs</li>
metrics</li>
traces</li>
machine boot logs</li>
</ul>
Then a local collector ships them onward.</p>
That makes instrumentation simpler and keeps the service code cleaner.</p>
Good observability changes behavior</h2>
Once the signals are strong, the engineering work changes.</p>
You stop guessing.</p>
You stop saying things like:</p>

"it might be networking"</li>
"maybe the guest was slow"</li>
"could be the proxy"</li>
</ul>
You can actually see:</p>

retry storms</li>
stuck transitions</li>
noisy health probes</li>
node-specific hot spots</li>
slow image paths</li>
</ul>
That makes the platform better because the fixes get sharper.</p>
The trap to avoid</h2>
The trap is collecting everything and understanding nothing.</p>
A microVM platform already has enough moving parts. If the telemetry model is sloppy, the data turns into clutter.</p>
I would rather have:</p>

good machine identifiers</li>
a few honest spans</li>
useful node metrics</li>
clean request logs</li>
</ul>
than a giant bucket of unlabeled noise.</p>
The boring outcome I want</h2>
When a user says, "machine create was slow," I want the answer within minutes.</p>
Not because someone remembered a host by instinct.</p>
Because the system can show:</p>

where the request went</li>
how placement behaved</li>
which machine transition stalled</li>
what the host looked like at the time</li>
</ul>
That is what observability is for. Not dashboards for their own sake. Explanations. That is the bar. Good observability does not just collect data. It shortens the path to the right fix.</p>


Host Reconciliation with a Host Agent
Unknown — Thu, 09 Apr 2026 00:00:00 +0000
I actually like SSH. It is useful, fast, and hard to avoid when you are building infrastructure.</p>
Still, there is a point in every infrastructure project where SSH becomes a bad habit.</p>
At first it feels fast.</p>
Jump into a box. Edit a file. Restart a service. Fix the thing.</p>
Then the fleet grows, and you realize your operating model is just "hope the last person remembers what changed."</p>
That is not a platform. That is group memory with root access.</p>
This is the thinking behind having a host agent in the first place.</p>
Why hosts need their own owner</h2>
A host running a worker daemon, WireGuard, Corrosion, DNS, and proxy services has real state outside the guest workload.</p>
That state needs ownership.</p>
You need one place that can answer:</p>

what should be running here?</li>
what revision is applied?</li>
are configs rendered correctly?</li>
is the node healthy enough for placement?</li>
what should happen if the mesh drifts or a service crashes?</li>
</ul>
That is the job I want a host agent to do.</p>
What a host agent should own</h2>
For me, the host agent should own the boring but important things:</p>

config rendering</li>
service state</li>
revision tracking</li>
health reporting</li>
</ul>
If those responsibilities are split across too many scripts and side channels, the host starts drifting faster than the team can keep up.</p>
The rule I like</h2>
The host agent should be the only live owner of host mutation.</p>
That sounds strict. It should be.</p>
If half your changes come from bootstrap scripts, a quarter come from direct SSH, and the rest come from local service side effects, you do not have clean recovery. You have overlapping authority.</p>
That leads to drift.</p>
Manifest-driven hosts are calmer hosts</h2>
The idea is simple:</p>

describe the node in a manifest</li>
let the host agent reconcile toward that state</li>
expose health and admin APIs from the same agent</li>
</ul>
That gives you a clear loop:</p>
desired node manifest</span></span>
        |</span></span>
        v</span></span>
   host agent</span></span>
        |</span></span>
        v</span></span>
 services / configs / health state</span></span></code></pre>
Now the node stops being a snowflake and starts being something you can reason about.</p>
Why this matters for placement</h2>
Host health is not just an ops concern. It affects scheduling.</p>
If a node has enough free CPU but its mesh is broken, its config is stale, or its core services are unhealthy, it should not take new work.</p>
That means the host agent should publish health that the control plane can trust:</p>

ready or not</li>
schedulable or not</li>
drained or not</li>
last error</li>
applied revision</li>
</ul>
That turns host management into a real input to compute placement.</p>
Self-healing without nonsense</h2>
I like self-healing, but only when it is bounded.</p>
"Self-healing" should not mean a daemon thrashing forever while making the node harder to debug.</p>
It should mean a small set of safe actions:</p>

rerun reconcile</li>
restart one broken service</li>
re-check health</li>
cordon if still bad</li>
escalate if retries are exhausted</li>
</ul>
That is enough.</p>
The host agent should be able to help, not hide the problem.</p>
Here is the kind of drift case I worry about:</p>
service binary updated</span></span>
mesh config not updated</span></span>
node looks half healthy</span></span>
node should report degraded and stop taking work</span></span></code></pre>
That is exactly the sort of thing I want a host agent to catch early.</p>
Why direct SSH should become the exception</h2>
SSH is still useful. You need it for deep debugging.</p>
But it should not be the main operating path.</p>
The more routine work you can move into a host agent, the better:</p>

service restarts</li>
drain and uncordon</li>
health checks</li>
config rendering</li>
support bundle collection</li>
</ul>
That is how you make the fleet operable by design instead of by folklore.</p>
This matters even more for small teams</h2>
Big teams can sometimes absorb messy host practices with process and staffing.</p>
Small teams cannot.</p>
If a small team wants to run real infrastructure, it needs systems that reduce memory load. You do not want every fix to require remembering which host has which hand-edited file and which bootstrap flag was last used there.</p>
You want one path.</p>
That is why I think host reconciliation is a first-class platform feature, not just an operations convenience.</p>
The simple bar</h2>
A healthy host management story should let you answer these questions quickly:</p>

what should be on this node?</li>
what is actually on this node?</li>
is it healthy?</li>
can it take more work?</li>
if not, what is the next safe action?</li>
</ul>
If the answer still starts with "SSH in and check around," I do not think the system is done. If hosts mutate from too many places, recovery turns into guesswork.</p>


Why We Built Our Own Machine Proxy
Unknown — Wed, 08 Apr 2026 00:00:00 +0000
When I started thinking about exposing services from machines, one plain question kept showing up:</p>
How does traffic reach a machine?</p>
That question sounds smaller than it is.</p>
Public ingress is not just HTTP. It is TLS, SNI, raw TCP, SSH, WebSSH, machine ownership, and route cleanup when the guest disappears in the middle of the day.</p>
That is a lot of stuff to hand-wave.</p>
In this post, I want to explain why I think a machine platform often wants a proxy layer that is tightly tied to the machine lifecycle.</p>
The normal answer</h2>
The normal answer is to reach for a big proxy.</p>
That is often reasonable. Envoy, HAProxy, Nginx, Caddy, and friends exist for a reason. You should not rebuild them out of ego.</p>
But sometimes the fit is bad.</p>
In a microVM platform, the proxy has to care about things that general-purpose ingress does not naturally care about:</p>

per-machine route ownership</li>
fast route add and delete</li>
namespace-aware connections</li>
SSH and WebSSH alongside HTTP</li>
tight integration with machine lifecycle</li>
</ul>
When that friction grows, a custom proxy stops sounding like vanity and starts sounding like less glue code.</p>
What I wanted from the proxy</h2>
I wanted the host-level proxy to do a few things well:</p>

terminate TLS</li>
expose stable public endpoints</li>
route traffic to the right machine</li>
support HTTP, raw TCP, and SSH flows</li>
clean up routes when machines die</li>
</ul>
That is not a generic public ingress problem. That is a machine platform problem.</p>
Why lifecycle integration matters</h2>
A machine proxy should not feel like a detached box on the side.</p>
It needs to move with the lifecycle:</p>

machine starts</li>
route gets registered</li>
DNS gets updated</li>
traffic flows</li>
machine stops</li>
route disappears</li>
</ul>
If the route layer lags behind machine truth, you get bad failure modes:</p>

traffic to dead machines</li>
ghost routes</li>
wrong backend after a restart</li>
delayed exposure after machine readiness</li>
</ul>
The proxy has to speak the same language as the machine system.</p>
The shape of the design</h2>
This is the rough picture:</p>
internet</span></span>
   |</span></span>
   v</span></span>
+---------+</span></span>
| proxy   |</span></span>
| TLS     |</span></span>
| SSH     |</span></span>
| routes  |</span></span>
+----+----+</span></span>
     |</span></span>
     v</span></span>
+----------+</span></span>
| machine  |</span></span>
| netns or |</span></span>
| backend  |</span></span>
+----------+</span></span></code></pre>
The important part is not the box. It is the ownership.</p>
The proxy should know which machine owns which route and update that state as the machine lifecycle changes.</p>
What the proxy really stores</h2>
At a minimum, the proxy layer usually needs to know:</p>

hostname</li>
target machine</li>
protocol</li>
port</li>
route ownership</li>
</ul>
That is enough to make route updates explicit instead of magical.</p>
Why not push this into every guest</h2>
You could make every machine solve its own public ingress.</p>
That would be a mistake.</p>
Now every guest needs:</p>

TLS setup</li>
cert management</li>
public binding logic</li>
exposure policy</li>
maybe even SSH exposure logic</li>
</ul>
That is duplicated work and a bigger security surface.</p>
A host-level proxy gives the platform one clean place to manage exposure.</p>
That is simpler for operators and better for users.</p>
There is also a very real cleanup race here:</p>
machine dies</span></span>
route still exists</span></span>
next request arrives</span></span>
request goes nowhere</span></span></code></pre>
That is why I keep treating ingress as part of the machine lifecycle instead of a detached networking concern.</p>
One underrated benefit: better product shape</h2>
Once ingress is owned at the platform layer, you can do nicer things:</p>

stable public hostnames per machine</li>
platform-managed certs</li>
clean SSH entry points</li>
policy around what gets exposed</li>
shared observability around requests and failures</li>
</ul>
That turns raw compute into a product with doors and handles.</p>
Building your own does not mean building everything</h2>
There is a good way to do this and a bad way.</p>
The good way is to build the narrow thing your platform actually needs:</p>

route storage</li>
lifecycle hooks</li>
protocol support you really use</li>
cert modes you can operate</li>
</ul>
The bad way is to convince yourself you are now in the business of recreating every feature a major proxy has learned over a decade.</p>
I have no interest in the second path.</p>
The goal is not "beat Envoy." The goal is "remove enough mismatch that the system gets simpler."</p>
The test I use</h2>
If a machine starts, can the route appear quickly?</p>
If a machine dies, can the route disappear cleanly?</p>
If an operator asks who owns a route, is the answer obvious?</p>
If the answer to those questions is yes, the proxy is doing its job.</p>
That is the level I care about. Not theoretical flexibility. Real ownership. That is what I care about here. Ingress is a lifecycle problem as much as a networking problem.</p>


Making Internal DNS Boring for Private Machines
Unknown — Tue, 07 Apr 2026 00:00:00 +0000
Once private networking starts to work, DNS becomes the next obvious problem.</p>
If private networking is the road system, DNS is the street sign.</p>
And like street signs, you only notice it when it is wrong.</p>
Nothing makes a platform feel flaky faster than this:</p>

the machine is up</li>
the network path works</li>
the name still does not resolve</li>
</ul>
That gap is small on paper and painful in real use.</p>
I have run into this enough times that I now care a lot about keeping internal DNS boring.</p>
What I want from internal DNS</h2>
I want internal DNS to be so boring people stop asking how it works.</p>
That means:</p>

every machine can resolve private names</li>
the answer does not depend on which node happens to host the resolver</li>
moving a machine should not change the model</li>
stale node-local state should not quietly break resolution</li>
</ul>
If DNS depends too much on local accidents, the whole private network feels less trustworthy.</p>
Stable identity beats node-local cleverness</h2>
One tempting design is to make the guest ask a node-specific resolver.</p>
That sounds easy because the host is right there.</p>
The problem is that it leaks host details into the guest model. Now the guest is coupled to where it landed. If the node changes, the path changes. If node-local state drifts, resolution changes. If the resolver is healthy on one node and stale on another, you get weird half-failures.</p>
A better model is simple:</p>

the guest uses one stable DNS service identity</li>
the host makes that identity work locally</li>
</ul>
That keeps the guest contract clean.</p>
Why this matters more than it sounds</h2>
Service discovery is one of those things people treat as an accessory until they build real workloads.</p>
The moment you have:</p>

app talking to database</li>
app talking to cache</li>
one machine calling another by name</li>
a short-lived environment that should behave like a real network</li>
</ul>
you need names you can trust.</p>
Something as simple as db.internal</code> should just work. The user should not have to know which node hosts the database or which host is running the resolver.</p>
IP addresses are fine for debugging. They are bad user experience.</p>
The shape I like</h2>
Here is the guest-facing model:</p>
guest:</span></span>
  query .internal name</span></span>
      |</span></span>
      v</span></span>
stable DNS VIP</span></span>
      |</span></span>
      v</span></span>
worker-local dns service</span></span>
      |</span></span>
      v</span></span>
answer from replicated machine/network state</span></span></code></pre>
The guest always talks to the same destination.</p>
The host does the local termination and lookup work.</p>
That split is what makes the system feel stable.</p>
Keep lookup data close to the truth</h2>
The second part of the DNS problem is data freshness.</p>
If name records are built from stale side tables or delayed repair logic, DNS becomes the first place you notice drift.</p>
That is usually what bad DNS feels like in practice:</p>

the machine is up</li>
packets are flowing</li>
the name still fails</li>
</ul>
That kind of bug wastes a lot of time because it makes people question the whole network even when only one part is broken.</p>
That is why the data feeding DNS should come from the same control-plane truth that placement and machine ownership use. Different products can wire that differently, but the principle stays the same:</p>
Do not make DNS invent its own reality.</p>
It should reflect the authoritative view of:</p>

what machine exists</li>
what network it belongs to</li>
what private address it owns</li>
</ul>
Freshness matters too. If DNS updates lag too much, users stop trusting names and start pasting IPs into config files.</p>
A small thing that changes the whole product</h2>
When internal DNS is good, everything above it gets easier:</p>

config files get simpler</li>
local dev feels closer to production</li>
private services stop needing copy-pasted IPs</li>
cross-machine workflows become natural</li>
</ul>
When internal DNS is bad, users build shadow systems:</p>

hardcoded addresses</li>
host file hacks</li>
startup scripts that query the control plane directly</li>
</ul>
That is usually a sign the platform contract is weak.</p>
Boring is the win</h2>
The best internal DNS story is not impressive.</p>
It is just this:</p>
"Every machine gets a private name, every other machine can resolve it, and nobody has to think about which worker host is involved."</p>
That is the result I want. Not clever. Just reliable. If users keep reaching for IP addresses, the internal DNS contract is probably weak.</p>


Private Networking for MicroVM Fleets with WireGuard and IPv6
Unknown — Mon, 06 Apr 2026 00:00:00 +0000
After getting machines to boot reliably, networking was the next thing that forced me to slow down and think properly.</p>
Private networking sounds simple when you say it quickly.</p>
"Just let the machines talk to each other."</p>
Then you try to do it across hosts, across regions, across reboots, without handing every guest a pile of custom firewall rules and mystery routing.</p>
That is where the clean model matters.</p>
In this post, I want to explain the network shape I find the most reasonable for a microVM fleet.</p>
What I wanted from the network</h2>
For a microVM platform, the private network should feel boring from inside the guest.</p>
The guest should not care:</p>

which host it landed on</li>
which physical node hosts its peer</li>
whether a node got replaced</li>
how the control plane exchanged peer state</li>
</ul>
The guest should know one thing: it has a private address and packets to that private world work.</p>
That is the standard.</p>
Why IPv6 helps</h2>
IPv4 can do this, but it starts fights you do not need.</p>
With IPv6, you can give every machine a clean private identity without squeezing everything through tiny address space and heavy NAT habits. It makes the addressing model easier to reason about.</p>
That matters because the address can carry meaning:</p>

node identity</li>
machine identity</li>
network membership</li>
policy boundaries</li>
</ul>
Not magical meaning. Useful meaning.</p>
Why WireGuard fits</h2>
Once you have more than one host, you need a private fabric between them.</p>
WireGuard is a good fit because it is simple, fast, and kernel-native. It also gives you a clean identity model with keys instead of a giant pile of TLS setup for every internal hop.</p>
At a high level:</p>
guest VM --tap--> worker host --WireGuard--> remote worker host --tap--> guest VM</span></span></code></pre>
The worker hosts form the mesh. The guests ride on top of it.</p>
That lets the platform keep a simple promise:</p>
"Your machine has a private address, and that address works across the fleet."</p>
Here is the packet path I keep in my head:</p>
guest</span></span>
  -> tap device</span></span>
  -> local worker host</span></span>
  -> WireGuard mesh</span></span>
  -> remote worker host</span></span>
  -> remote tap device</span></span>
  -> remote guest</span></span></code></pre>Do not mix guest routing and DNS identity</h2>
One subtle mistake is to blur two separate ideas:</p>

the next hop a guest should route through</li>
the stable identity of the DNS service it should query</li>
</ul>
Those are not the same thing.</p>
The next hop is local. It is about how a packet leaves the guest and reaches the host.</p>
The DNS service identity should be stable. It should not change just because the guest moved or the node got replaced.</p>
Once those concepts stay separate, the network gets less weird.</p>
The control-plane trap</h2>
There is a bigger design lesson here too.</p>
Do not make your whole control plane depend on the same network it is still trying to heal.</p>
That circular dependency creates awful failure modes.</p>
If peer updates only arrive over the mesh, and the mesh is broken because peer updates are stale, you have built a trap for yourself.</p>
The data plane and the control plane can talk to each other, but they should not hold each other hostage.</p>
Failure modes that matter</h2>
The failure modes here are not always dramatic. They are often annoying:</p>

a stale peer endpoint means packets go to the wrong place</li>
a bad route means the guest has an address but no path</li>
the guest can reach the network, but DNS still fails</li>
</ul>
Those are the cases that make a platform feel flaky even when part of the design is working.</p>
What "good" looks like</h2>
A good private network for microVMs has a few traits:</p>

every guest gets a stable private identity</li>
east-west traffic works across hosts</li>
guest routing is host-local and predictable</li>
policy can be enforced without hand-crafted snowflake rules</li>
DNS does not depend on the guest knowing host details</li>
host replacement does not force the guest model to change</li>
</ul>
That is the shape I want:</p>
            private fleet network</span></span>
</span>
   VM A            worker A           worker B            VM B</span></span>
  fdaa::10  <-->   mesh0    <====>    mesh0    <-->     fdaa::20</span></span>
                     WG mesh          WG mesh</span></span></code></pre>
The guest sees a private world. The host does the ugly work.</p>
Why this matters for compute products</h2>
Private networking is not a side quest.</p>
It is what makes the platform useful for anything beyond isolated toy workloads.</p>
You need it for:</p>

service-to-service communication</li>
private databases</li>
multi-machine test environments</li>
agent sandboxes that call internal tools</li>
distributed jobs that should not be public on the internet</li>
</ul>
Without a clean private network, everything gets forced through public exposure or weird host tunnels. That is not a platform. That is a workaround farm.</p>
The real win</h2>
The best network design is the one users forget.</p>
They should not need to know which node owns the WireGuard peer, which route got reconciled, or which host-local device delivered the packet.</p>
They should know their machine has a private address and the packet gets where it should go. That is boring. Boring is the goal. The guest should see one private network, even if the hosts underneath are messy.</p>


Fast Starts: Snapshots, Clones, and Warm Slots for MicroVMs
Unknown — Sun, 05 Apr 2026 00:00:00 +0000
One of the things I care about most in a compute platform is startup time.</p>
People say they want fast machine starts.</p>
What they usually mean is they do not want to wait while your platform does cold work right in front of them.</p>
That distinction matters a lot.</p>
If you create a machine from scratch on demand, you are paying for a bunch of slow steps in public:</p>

prepare a root filesystem</li>
create overlays</li>
set up networking</li>
boot or restore Firecracker</li>
wait for the guest agent</li>
</ul>
Even when each step is pretty fast, the total is not.</p>
Here is a rough way to think about that budget:</p>
rootfs prep: hundreds of ms to seconds</span></span>
network setup: tens to hundreds of ms</span></span>
boot or restore: hundreds of ms to seconds</span></span>
guest readiness: hundreds of ms to seconds</span></span></code></pre>
None of those numbers are scary on their own.</p>
Stack them together on the request path and they stop feeling cheap.</p>
Cold create is not the enemy</h2>
Cold create is fine. You need it. It is the fallback path and the general path.</p>
The mistake is pretending cold create can also be your instant path.</p>
If you care about interactive workloads, dev environments, agent sandboxes, or bursty compute, you need a different shape:</p>

prepare ahead of time</li>
reserve capacity before doing expensive work</li>
activate something already close to ready</li>
</ul>
That is where snapshots, clones, and warm slots come in.</p>
In this post, I want to walk through how I think about those three paths.</p>
Three levels of speed</h2>
I like to think about machine startup in three buckets.</p>
1. Fresh boot</h3>
This is the full path. Pull image. Build filesystem. Boot guest. Wait for readiness.</p>
It is the most flexible and the slowest.</p>
2. Clone from a prepared source</h3>
This path reuses a known-good base, often through snapshot or overlay tricks. It cuts a lot of setup time, but there is still work to do.</p>
It is much better, but not instant.</p>
3. Warm activation</h3>
This path claims something already prepared and nearly ready.</p>
That is the path that can feel snappy.</p>
Here is the rough picture:</p>
fresh create:</span></span>
  image pull -> rootfs prep -> network -> boot -> ready</span></span>
</span>
clone:</span></span>
  prepared base -> overlay/snapshot -> network -> restore -> ready</span></span>
</span>
warm activation:</span></span>
  claim ready slot -> attach identity -> expose -> run</span></span></code></pre>
The third path is where good latency lives.</p>
Why reservation matters</h2>
One easy mistake is to fan out real work to multiple nodes and let the fastest winner keep running.</p>
That looks clever until you notice the waste.</p>
Two or three nodes may do expensive rootfs and startup work for one user request. The losers then roll back after burning CPU, IO, and time.</p>
That is not a fast path. It is a messy betting strategy.</p>
A better model is:</p>

rank nodes</li>
reserve one node</li>
commit cold work to that node only</li>
use warm activation when an exact match already exists</li>
</ol>
This keeps the fast path honest.</p>
The importance of exact match</h2>
Warm pools sound great until you make them fuzzy.</p>
If a warm slot is only "kind of close" to what the user asked for, activation turns into mutation, and mutation turns back into cold work.</p>
So the identity of a warm slot needs to be strict:</p>

image digest</li>
machine shape</li>
restore capability</li>
any other setup that changes startup cost or correctness</li>
</ul>
That lets the scheduler ask a sharp question:</p>
"Do I already have a prepared thing that exactly matches this request?"</p>
If yes, activate it.</p>
If no, reserve a node for cold work and move on.</p>
What breaks warm starts</h2>
Warm starts stop being warm pretty quickly when one of these shows up:</p>

the image digest does not match</li>
the machine shape does not match</li>
the prepared state is stale</li>
the guest still has expensive init work left to do</li>
</ul>
That is why I do not like fuzzy warm capacity. Either it is a real ready slot for this request, or it is not.</p>
Speed without lying</h2>
The biggest risk in performance work is cheating.</p>
You call something fast because one internal benchmark looked nice, but the real path still hides slow work in the same request.</p>
I do not want "fast" to mean:</p>

it was cached on one node once</li>
the loser nodes cleaned up eventually</li>
we skipped a correctness check</li>
the guest was not actually ready yet</li>
</ul>
I want it to mean the user asked for a machine and got a usable machine quickly, without hidden waste and without crossed fingers.</p>
The user-facing payoff</h2>
When this is done well, the platform feels different.</p>
It stops feeling like "please wait while we build your environment" and starts feeling closer to "your environment is here."</p>
That opens the door for better product shapes:</p>

on-demand dev machines</li>
ephemeral test environments</li>
short-lived AI sandboxes</li>
burst compute for jobs that should start now</li>
</ul>
Those experiences depend less on the hypervisor and more on how smart the control plane is about prepared capacity.</p>
The real challenge</h2>
The hard part is not building one fast restore path.</p>
The hard part is managing the whole life around it:</p>

how many warm slots to keep</li>
which image digests deserve them</li>
when to recycle them</li>
how to avoid stale prepared state</li>
how to represent ready slots to the scheduler</li>
</ul>
That is why fast activation is a control-plane problem as much as a Firecracker problem.</p>
The simple summary</h2>
Cold create is necessary.</p>
Clone paths are useful.</p>
Warm activation is what changes the feel of the product.</p>
And if you want warm activation to work well, you need more than a snapshot trick. You need clear reservation semantics, exact identity, and a scheduler that knows the difference between "this node has free CPU" and "this node has a prepared machine I can claim right now." That difference is where a lot of the speed comes from. Fast start is mostly about moving slow work off the user path.</p>


Stateless Regional Control Planes and Replicated State
Unknown — Sat, 04 Apr 2026 00:00:00 +0000
I like control planes that can die without drama.</p>
If a regional API process restarts, I do not want a rescue mission. I want it to come back, rebuild what it needs, and keep serving traffic.</p>
That pushes you toward a simple rule:</p>
The control plane should not own more durable state than it has to.</p>
I ran into this while thinking through the regional control plane. The more state I wanted to push into it, the less I liked the system.</p>
The old temptation</h2>
The easy design is familiar.</p>
Workers report into a central database. The regional service reads and writes that database. Over time it becomes a big cache, a queue, and a graveyard of stale rows all at once.</p>
That works until it does not.</p>
The cracks show up in boring places:</p>

a node reboots and stale machine records survive</li>
capacity data lags behind reality</li>
different regional instances read slightly different views</li>
cleanup relies on rows that no longer mean what they used to</li>
</ul>
The database becomes the place where old assumptions go to live forever.</p>
A better split</h2>
The worker already has the facts about its own machines. So let it publish enough of those facts for the control plane to read.</p>
That changes the flow:</p>
before:</span></span>
  worker -> central DB -> control plane</span></span>
</span>
after:</span></span>
  worker -> replicated cluster state</span></span>
  control plane -> read local replica</span></span></code></pre>
The second model has better failure behavior.</p>
The control plane is now a reader, scheduler, and router. It is not the sole keeper of facts it did not create.</p>
What the control plane should not know</h2>
There is also plenty the control plane does not need to know in detail:</p>

every host-side cleanup step</li>
every boot log line</li>
every process-level detail on the worker</li>
</ul>
It needs enough shared state to decide and route well. It does not need to mirror the entire private life of every node.</p>
Why this matters in practice</h2>
Stateless does not mean dumb.</p>
The regional control plane still does real work:</p>

auth</li>
placement</li>
network lifecycle</li>
machine lookup</li>
routing decisions</li>
health-based placement</li>
</ul>
The difference is that it does not need its own database to do that work well.</p>
It can boot, connect to its local replica, warm an in-memory cache if needed, and start answering questions.</p>
If it dies, no truth is lost.</p>
That is a much healthier role.</p>
What replicated state buys you</h2>
The control plane needs cluster facts, but not all facts.</p>
It mostly needs:</p>

node health</li>
node capacity</li>
machine placement</li>
network allocation state</li>
enough metadata to route and schedule safely</li>
</ul>
Those are all good candidates for shared replicated state.</p>
Corrosion became the interesting piece here. It shifts the cluster away from "everything goes through one central writer" and toward "the writer is close to the machine activity it is reporting."</p>
That matches reality better.</p>
Reading from a local replica helps too. It cuts the cost of always reaching back to one central place before the control plane can answer a simple question.</p>
The hard part is not replication</h2>
The hard part is deciding what deserves replication.</p>
If you replicate too little, the control plane gets blind.</p>
If you replicate too much, you turn shared state into a kitchen sink and make rollouts painful.</p>
The trick is to replicate the facts needed for decisions, not every private detail of a node's internal life.</p>
I think about it like this:</p>
local only:</span></span>
  step-by-step recovery detail</span></span>
  process-local cleanup bookkeeping</span></span>
  short-lived machine state</span></span>
</span>
replicated:</span></span>
  node capacity</span></span>
  machine ownership</span></span>
  placement-visible health</span></span>
  network membership</span></span>
  routeable identity</span></span></code></pre>
That boundary keeps the cluster readable.</p>
Stateless makes life easier</h2>
There is a human reason to want this too.</p>
A control plane with its own fragile state store becomes a thing you are scared to touch. You babysit migrations. You wonder if a failover will leak meaning. You start planning around the database instead of around the product.</p>
A stateless regional service is easier to operate, easier to scale, and easier to trust.</p>
Start a new one. Stop an old one. Move it. Replace it.</p>
That should be routine.</p>
One more benefit: the architecture gets honest</h2>
When the regional layer stops pretending to own everything, the system reads more clearly:</p>

workers report machine state</li>
the cluster replicates scheduling truth</li>
the regional service consumes that truth to make decisions</li>
</ul>
That is a crisp model.</p>
Here is the simplest version of it:</p>
          write state                  read state</span></span>
worker -----------------> replicated state -----------------> regional control plane</span></span>
   ^                                                              |</span></span>
   |                                                              |</span></span>
   +--------------------- commands back --------------------------+</span></span></code></pre>
The loop is tight, and each side has a job. That is what I want from a control plane. Not magic. Just clean ownership. The control plane needs enough shared state to decide. It does not need every detail to operate.</p>


How a Worker Node Runs a MicroVM
Unknown — Fri, 03 Apr 2026 00:00:00 +0000
After removing the earlier draft for this slot, I still wanted a post here that explains one of the most important parts of the platform: what actually happens on the worker node when a machine gets created.</p>
This part matters because a lot of the rest of the system only makes sense once you know what the worker is responsible for.</p>
The control plane can pick a node. The proxy can expose traffic. DNS can hand out names. None of that means much until one worker host actually does the job of turning a request into a running microVM.</p>
The worker is where the request becomes real</h2>
At a high level, the worker daemon is the part of the system that takes a machine request and does the host-side work needed to bring it to life.</p>
That usually means:</p>

validating the machine request</li>
preparing the filesystem</li>
setting up the network</li>
starting Firecracker</li>
waiting for the guest to come up</li>
exposing the machine to the rest of the platform</li>
cleaning up if something fails halfway through</li>
</ul>
That is a lot of responsibility for one service, but it is the right place for it. The worker host is where the machine actually lives, so it needs the code that knows how to deal with the messy details.</p>
The flow I keep in my head</h2>
This is the simplest version of the worker-side flow:</p>
create request arrives</span></span>
        |</span></span>
        v</span></span>
  validate + record intent</span></span>
        |</span></span>
        v</span></span>
  prepare root filesystem</span></span>
        |</span></span>
        v</span></span>
  setup network namespace, tap, routes</span></span>
        |</span></span>
        v</span></span>
  write VM config + start Firecracker</span></span>
        |</span></span>
        v</span></span>
  wait for guest agent to become ready</span></span>
        |</span></span>
        v</span></span>
  register machine with proxy / DNS / control plane</span></span></code></pre>
Real life is messier than that, but if this mental model is clear, the rest of the platform gets easier to follow.</p>
What the worker leaves behind on disk</h2>
Even a short-lived machine usually leaves behind a few concrete things on the host while it is running:</p>

a kernel image</li>
a root filesystem or overlay</li>
a VM config</li>
logs</li>
sockets or control files</li>
</ul>
That is another reason cleanup matters so much. These are not abstract objects. They are real files and real resources on a real box.</p>
Step 1: take the request seriously</h2>
The first part sounds boring, but it is important.</p>
When the worker gets a create request, it cannot just charge ahead and hope for the best. It needs to make sure:</p>

the shape of the machine makes sense</li>
the node has enough room</li>
the image or base is available</li>
the request can be tracked if the host dies halfway through</li>
</ul>
This is one reason I like durable lifecycle logic so much. If the process crashes after step three out of seven, the worker still needs enough information to continue or clean up properly when it comes back.</p>
Step 2: prepare the filesystem</h2>
Before the guest boots, the worker has to prepare the disk the machine will use.</p>
Depending on the path, that can mean:</p>

pulling an image</li>
unpacking a root filesystem</li>
creating an overlay</li>
cloning from a prepared base</li>
restoring from a snapshot-backed setup</li>
</ul>
This step has a huge effect on startup time.</p>
It is also one of the places where wasted work hurts the most. If two nodes both prepare filesystems for one request and only one wins, that is real IO and real time thrown away.</p>
That is why I care so much about reservations and prepared capacity in the later posts.</p>
The slow parts</h2>
If I had to point to the places that most often dominate startup time, it would usually be these:</p>

filesystem prep</li>
network bring-up</li>
waiting for the guest to be ready</li>
</ul>
The worker does all three, which is why it ends up sitting right on the critical path.</p>
Step 3: build the network around the guest</h2>
Once the filesystem is ready, the worker still has to build the network shape the machine expects.</p>
That usually means creating or wiring up:</p>

a network namespace</li>
a tap device</li>
routes</li>
addressing</li>
any host-side policy needed for isolation</li>
</ul>
This is the part that makes a machine feel less like "a Firecracker process" and more like "a guest that actually lives on a host and can talk to things."</p>
It is also the step that causes a lot of annoying bugs if you are sloppy. A machine can boot fine and still be useless because the host-side network was only half-configured.</p>
Step 4: start Firecracker</h2>
Once the assets are in place, the worker can finally hand things over to Firecracker.</p>
That means building the VM config, pointing Firecracker at the right kernel and drives, starting the process, and then watching closely.</p>
This is the step people usually think is the whole problem.</p>
It is not.</p>
It is a very important step, but by the time you get here, the worker has already done a lot of heavy lifting.</p>
Step 5: wait for the guest to be usable</h2>
A VM process existing is not the same thing as a usable machine.</p>
This is one of those details that matters a lot in practice.</p>
If you return success too early, the rest of the system starts acting as if the machine is ready when it really is not. Then you get weird timing bugs:</p>

the proxy points to a machine that is not listening yet</li>
the control plane thinks startup succeeded</li>
the user connects before the guest finished booting</li>
</ul>
So the worker needs a clear readiness signal from inside the guest. Until that signal shows up, the machine is still starting.</p>
Step 6: plug the machine into the platform</h2>
Once the guest is actually ready, the worker can expose it to the rest of the system.</p>
That can include:</p>

registering routes with the proxy</li>
publishing private network identity</li>
publishing DNS data</li>
reporting machine state back to the control plane</li>
</ul>
This is the point where the machine stops being "something being built on one host" and becomes "something the platform can route to and reason about."</p>
Step 7: clean up hard when things go wrong</h2>
This part is easy to underestimate.</p>
Machines do not only fail at clean points. They fail halfway through.</p>
You can end up with:</p>

a prepared filesystem but no VM</li>
a VM process but broken networking</li>
a route registered for a machine that never became ready</li>
a half-created overlay</li>
</ul>
That is why the worker daemon cannot just know how to start machines. It has to know how to recover and how to clean up.</p>
For me, that is one of the main differences between a demo and a platform.</p>
The worker is the heart of the node</h2>
Once I started looking at the system this way, a lot of other design choices made more sense.</p>
The control plane should stay focused on placement and coordination.</p>
The worker should own the machine lifecycle on the node.</p>
The proxy should care about exposure.</p>
DNS should care about names.</p>
The host agent should care about host-level drift and health.</p>
That split keeps the jobs clear.</p>
Why I wanted this post in the series</h2>
I wanted one post early in the series that answers a simple question:</p>
When a machine gets created, what does one worker host actually do?</p>
For me, the answer is:</p>
It takes a request, prepares the machine, builds the network around it, starts the guest, waits until it is actually usable, plugs it into the rest of the platform, and cleans up if anything goes wrong.</p>
That is a lot of work.</p>
It is also where the platform becomes real. The worker does much more than start Firecracker. It builds the machine around it.</p>


Why a MicroVM Platform Needs a Crash-Resistant Machine FSM
Unknown — Thu, 02 Apr 2026 00:00:00 +0000
While working on the worker side of the platform, one thing became obvious pretty quickly: machine lifecycle code looks easy only when you test happy paths.</p>
Create a machine. Start it. Stop it. Delete it.</p>
That idea falls apart the first time a host reboots halfway through startup.</p>
Now you have harder questions:</p>

Was the VM process created?</li>
Did networking finish?</li>
Did the root filesystem get prepared?</li>
Did the API return before the host died?</li>
Should restart continue, roll back, or wait?</li>
</ul>
You cannot answer those questions with request handlers and hope.</p>
That is why I keep coming back to a durable machine FSM.</p>
In this article, I want to explain why I think this matters so much.</p>
The problem with one-shot handlers</h2>
A normal HTTP handler wants to do work and return. That is fine for small things. It is not fine for machine lifecycle.</p>
Machine lifecycle is long, messy, and full of partial failure. It has steps that touch:</p>

local storage</li>
network namespaces</li>
cgroups</li>
Firecracker process state</li>
proxy registration</li>
DNS registration</li>
snapshots</li>
cleanup</li>
</ul>
If each endpoint owns its own little workflow, recovery gets scattered everywhere. One path retries here. Another path cleans up there. A third path adds a special boot-time scan to heal the leftovers.</p>
You end up with a system where every bug fix adds one more side lane.</p>
What the FSM gives you</h2>
The useful part of an FSM is not the word "state machine." The useful part is discipline.</p>
You write down:</p>

the current durable state</li>
the desired next state</li>
the allowed transitions</li>
the work for each transition that is safe to run twice</li>
what to do when the process comes back after a crash</li>
</ul>
That forces the system to stop improvising.</p>
I like to think of it this way:</p>
request handler:</span></span>
  validate input</span></span>
  record desired change</span></span>
  wake machine run</span></span>
</span>
machine run:</span></span>
  inspect durable state</span></span>
  inspect what is really on the host</span></span>
  do the next safe step</span></span>
  persist progress</span></span>
  repeat until steady</span></span></code></pre>
That split is boring, which is exactly why it works.</p>
The key idea: one main machine run</h2>
The clean model is one main run per machine.</p>
Not one FSM for create, another for stop, another for cleanup, and a startup reconciler that tries to guess what the others meant.</p>
One machine run.</p>
That run owns the job of moving the machine toward its desired state.</p>
It can decide:</p>

should this machine be started?</li>
does it need cleanup?</li>
is the VM still alive?</li>
did the last transition finish?</li>
do we need to recover from a crash?</li>
</ul>
This is the shape:</p>
desired state + durable record + observed reality</span></span>
                  |</span></span>
                  v</span></span>
            +-----------+</span></span>
            | machine   |</span></span>
            |   run     |</span></span>
            +-----------+</span></span>
                  |</span></span>
      +-----------+------------+</span></span>
      |           |            |</span></span>
      v           v            v</span></span>
   create      start        destroy</span></span>
   network     wait         cleanup</span></span>
   rootfs      recover      release</span></span></code></pre>
That machine run becomes the adult in the room.</p>
A real failure case</h2>
Here is the kind of failure that changed how I think about this:</p>
network setup succeeds</span></span>
Firecracker starts</span></span>
guest never becomes ready</span></span>
host reboots</span></span>
machine run resumes</span></span>
system decides whether to retry, wait, or destroy</span></span></code></pre>
If your lifecycle model cannot explain that sequence cleanly, it is going to leak mess all over the node.</p>
Safe retries are not optional</h2>
The most important phrase in lifecycle code is not "fast." It is "safe to run twice."</p>
If the host crashes after setting up a tap device but before marking the step complete, you need to be able to run that step again without making things worse.</p>
Same for:</p>

creating directories</li>
allocating leases</li>
registering routes</li>
restoring snapshots</li>
deleting leftovers</li>
</ul>
The system should be able to ask, "What is true right now?" and then move one safe step forward.</p>
If your lifecycle logic depends on perfect memory of the previous process, it will fail in production.</p>
Cleanup has to be first-class</h2>
A lot of systems treat cleanup like an apology at the end.</p>
That is backwards.</p>
Cleanup is part of the lifecycle. It needs the same level of care as create.</p>
When a machine dies badly, the leftovers are not random junk. They are real resources:</p>

disk overlays</li>
snapshots</li>
proxy routes</li>
IP allocations</li>
namespaces</li>
DNS records</li>
leases</li>
</ul>
If cleanup is scattered across handlers, deferred callbacks, and boot-time repair code, it will drift.</p>
When cleanup is owned by the same machine run that owns creation, the system gets simpler. There is one place that knows what the machine owns and one place that can release it safely.</p>
Why this matters for user-facing latency</h2>
This is not just a correctness story. It affects speed too.</p>
A good FSM lets you separate cold work from fast work.</p>
It lets you:</p>

prepare assets ahead of time</li>
resume interrupted work instead of starting over</li>
expose exact lifecycle state to the control plane</li>
measure where time really goes</li>
</ul>
Without that, every slow start looks the same from the outside. You have no idea whether the machine was waiting on rootfs prep, network setup, guest boot, or cleanup from a previous failure.</p>
What I watch for now</h2>
When I read lifecycle code, I ask a few blunt questions:</p>

Is there one owner of recovery and cleanup?</li>
Can it resume after a crash?</li>
Are transitions explicit?</li>
Is cleanup part of the same model?</li>
Can every step run twice safely?</li>
</ul>
If the answer is no, the code may still work on a laptop. It will not stay honest under load.</p>
For me, that is the real value of the machine FSM. It gives the platform a way to explain half-finished work instead of just leaving a mess behind. If a machine lifecycle cannot resume after a crash, it is not finished.</p>


Building a MicroVM Control Plane in Public
Unknown — Wed, 01 Apr 2026 00:00:00 +0000
Over the past couple of years, I have been spending a lot of time with Firecracker, microVMs, and a distributed compute platform I have been building.</p>
It started with a simple goal: run Firecracker on a machine I control and understand every moving part.</p>
That sounds small, but it really is not.</p>
The first few steps are easy enough to explain. Boot a VM. Get KVM working. Start Firecracker. Launch a guest. Once that works, the next question shows up right away: what if this was not one machine, but a fleet?</p>
That is where the real work starts.</p>
Running one microVM is a neat trick. Running thousands means you need a control plane. You need placement, state, networking, ingress, observability, cleanup, and recovery when the host does something rude like reboot in the middle of a machine start.</p>
In this post, I want to set up the rest of the series and talk about that jump.</p>
The shape of the system</h2>
Over time, the repo turned into a small compute platform with a few clear pieces:</p>

a regional control-plane service decides where work should go</li>
a worker daemon runs machines on worker hosts</li>
an image service prepares root filesystems from OCI images</li>
an ingress proxy handles public traffic, TLS, and SSH access</li>
an internal DNS service gives private names to machines</li>
a host agent owns host-level changes and node health</li>
a guest agent runs inside the guest and brings the machine to life</li>
</ul>
At a high level it looks like this:</p>
                public API</span></span>
                    |</span></span>
                    v</span></span>
             +--------------+</span></span>
             | regional API |</span></span>
             | placement    |</span></span>
             | auth         |</span></span>
             +------+-------+</span></span>
                    |</span></span>
          commands / reads</span></span>
                    |</span></span>
     +--------------+------------------+</span></span>
     |                                 |</span></span>
     v                                 v</span></span>
+------------+                   +------------+</span></span>
| worker     |                   | worker     |</span></span>
| worker A   |                   | worker B   |</span></span>
+------+-----+                   +------+-----+</span></span>
       |                                |</span></span>
       | local services                 | local services</span></span>
       v                                v</span></span>
  image service / ingress proxy / DNS / host agent / Firecracker</span></span></code></pre>
That diagram is clean. The real thing was not.</p>
A request in motion</h2>
One of the easiest ways to understand the platform is to follow one machine request from start to finish:</p>
client asks for a machine</span></span>
        |</span></span>
        v</span></span>
control plane picks a worker</span></span>
        |</span></span>
        v</span></span>
worker prepares and starts the machine</span></span>
        |</span></span>
        v</span></span>
proxy and DNS get updated</span></span>
        |</span></span>
        v</span></span>
user reaches a running machine</span></span></code></pre>
That flow looks simple on paper.</p>
The hard part is everything hiding inside each step.</p>
The first lesson: every shortcut becomes a subsystem</h2>
Early on, a lot of things can live in shell scripts and one-off flows. That is normal. You need speed while you are learning.</p>
Then the edges show up.</p>
You need network setup that works every time, not just on a good day. You need logs that survive long enough to explain a bad boot. You need to know if a machine is still real after the host process dies. You need a way to expose HTTP and SSH without making every guest solve TLS and routing on its own.</p>
Each shortcut turns into a real component:</p>

Root filesystem prep becomes a real image service.</li>
Traffic routing becomes a real ingress proxy.</li>
Guest boot logic becomes a guest agent.</li>
Host reconciliation becomes a host agent.</li>
Cluster state becomes a control-plane problem, not a local script problem.</li>
</ul>
That is the pattern I keep coming back to: small experiments turn into real features faster than you think.</p>
The second lesson: the hard parts are not the obvious parts</h2>
People hear "microVM platform" and think the hard part is booting Firecracker.</p>
It is not.</p>
Booting Firecracker is table stakes. The real pain lives in everything around it:</p>

deciding where a machine should run</li>
cleaning up half-failed starts</li>
preserving state across crashes</li>
connecting private networks across hosts</li>
making machine names resolve correctly</li>
exposing services without leaking the whole host</li>
understanding why one request took 150ms and the next took 5s</li>
</ul>
The control plane is not glamorous, but it is the difference between a demo and a platform.</p>
The third lesson: state is the whole game</h2>
Once you run more than one node, you stop asking "did the API succeed?" and start asking better questions:</p>

What happens if the writer crashes after the side effect but before the state update?</li>
What if two nodes both think they own the same action?</li>
What data can be rebuilt, and what data must survive?</li>
</ul>
That is why so much of this series ends up talking about FSMs, state replication, networking, and cleanup paths. Those are not side details. That is the platform.</p>
Why I am writing this series</h2>
There is a lot of writing about containers, and there is a lot of writing about big cloud systems. There is less writing about the middle ground: building your own compute platform when you still want to understand every moving piece.</p>
That middle ground is where most of the interesting tradeoffs live.</p>
You do not have infinite headcount. You do not want five databases because a diagram looked impressive. You do not want a service mesh because you got bored. You want a system that can survive failure, stay understandable, and move fast enough to keep building.</p>
That is the lens for the posts that follow.</p>
What is coming next</h2>
This series will cover:</p>

durable machine state and crash recovery</li>
how a worker node actually brings up a machine</li>
how a stateless regional control plane can still make good decisions</li>
faster starts with snapshots, clones, and warm slots</li>
private networking for microVM fleets</li>
boring internal DNS</li>
ingress and SSH routing</li>
host reconciliation and self-healing</li>
observability that can explain real failures</li>
scheduling compute without cheating</li>
what this all means for AI workloads</li>
</ul>
That is the kind of system I am aiming for here: small enough to understand, but solid enough to run real work. A single microVM is mostly a runtime problem. A fleet of microVMs is a coordination problem.</p>


Move untracked git files to another directory
Unknown — Tue, 19 Mar 2024 00:00:00 +0000
I had a lot of files in a git repository, which I didn't want to add in git</code>, So to remove all of them
to a temporary directory I used a collection of Unix commands.</p>
git</span> status</span> -</span>-porcelain</span> |</span> awk</span> '</span>{print $2}</span>'</span> |</span> xargs</span> mv</span> -</span>t</span> ../scratch</span></span></code></pre>


Animal Farm by George Orwell
Unknown — Thu, 07 Mar 2024 00:00:00 +0000

  
  Image generated using Adobe Firefly AI</figcaption>
</figure>
I picked up Animal Farm</em> right after 1984</a>. I expected something smaller and easier. It is smaller. It is not easier.</p>
The plot is almost too neat. Animals kick out the farmer, promise a better world, and end up under a new set of tyrants. Orwell keeps it plain, which is exactly why it works. Nothing feels dressed up. You watch the hope drain out of the farm step by step.</p>
That slow slide is the whole book. Nobody wakes up one morning and says the revolution is dead. The rules change a little. The language gets slippery. The animals get tired. They start doubting what they saw with their own eyes. By the time the truth is fully broken, they do not have much left to fight with.</p>
Boxer got to me more than anyone else. He is loyal, decent, hardworking, and completely unequipped for the kind of lies being used against him. Napoleon is awful in a colder way. He does not need charm. He just needs control, patience, and the right people to repeat his version of events.</p>
What stayed with me most is how familiar the book feels. Not because life is literally a farm run by pigs, obviously. Because Orwell understands how people get managed. Keep them busy. Keep them confused. Give them a slogan. Change the record. Do it again.</p>
It is a sharp book. Mean, too. I finished it with that bad taste you get when a story stops feeling like a story and starts feeling like a pattern you have seen before.</p>

If you have thoughts on the book, ping me on Twitter @aaqaishtyaq</a>.</em></p>


Pulga, Kalga - Himachal Pradesh 2024
Unknown — Tue, 05 Mar 2024 00:00:00 +0000
In the heart of Himachal's Parvati Valley lie its hidden gems: Pulga and Kalga. Picture-perfect villages embraced by towering mountains, where every corner tells a story of nature's magnificence. Here, snow-capped peaks stand tall, kissing the sky, while icy waterfalls dance down the slopes, painting the landscape with their pure, untamed beauty. Despite the biting cold, every moment feels like a warm embrace from Mother Nature herself. It's not just a trip; it's a journey of awe, wonder, and connection with the raw essence of the mountains.</p>


  
  </figcaption>
</figure>
 \</p>


  
  </figcaption>
</figure>
 \</p>


  
  </figcaption>
</figure>
 \</p>


  
  </figcaption>
</figure>
 \</p>


  
  </figcaption>
</figure>
 \</p>


1984 by George Orwell
Unknown — Sat, 03 Feb 2024 00:00:00 +0000

  
  Image generated using Adobe Firefly AI</figcaption>
</figure>
I knew the broad shape of 1984</em> before I read it. Big Brother. Surveillance. Thought Police. All that. What I was not ready for was how suffocating the book feels page by page.</p>
Orwell does not rely on giant twists or clever worldbuilding tricks. He just keeps pressing on the same sore spot. The telescreen is always there. The slogans are always there. The Party is always there, reaching right into memory and language and private thought. It wears you down while you read.</p>
That is the part that hit me hardest. The regime does not only want obedience. It wants to wreck your grip on what is real. It wants you to doubt your own memory before anyone else even has to argue with you. That feels more disturbing than the surveillance itself.</p>
Winston works because he is small and shaky and ordinary. He is not built like a movie rebel. He is a lonely man trying to protect a corner of his mind. Sometimes that corner is a memory. Sometimes it is desire. Sometimes it is just the urge to say that two plus two still matters.</p>
There is no fake heroism in this book. Orwell is too ruthless for that. He keeps asking what power can do when it does not stop at laws or prisons and goes straight for the self. Not the body first. The self.</p>
I finished 1984</em> feeling tired in the best way. It is brutal, but it earns that brutality. A lot of books warn you about tyranny. This one makes tyranny feel intimate.</p>

If you have thoughts on the book, ping me on Twitter @aaqaishtyaq</a>.</em></p>


The Idiot by Fyodor Dostoevsky
Unknown — Mon, 01 Jan 2024 00:00:00 +0000

  
  Image generated using Adobe Firefly AI</figcaption>
</figure>
The Idiot</em> is not a smooth read. It is crowded, restless, dramatic, and at times almost too much. That is also why I liked it.</p>
Prince Myshkin is the reason the whole thing works. He is kind in a way that feels dangerous to everyone around him. Not flashy. Not saintly. Just open, honest, and weirdly free of the usual games people play. That makes him hard to place. Some people see innocence. Some see weakness. Some want to crush him for it.</p>
What makes the book interesting is that goodness does not solve anything here. If anything, it makes the mess easier to see. Myshkin walks into rooms full of vanity, wounded pride, jealousy, self-pity, and desire, and all of it starts showing its face faster.</p>
Nastasya Filippovna is impossible to ignore. So is Rogozhin. Every time either of them shows up, the temperature changes. You can feel trouble coming even in the quiet scenes. That tension carries the book through its longer stretches.</p>
Dostoevsky is great at letting people embarrass themselves on the page. They talk too much. They misread each other. They act out. They want love, status, forgiveness, revenge, sometimes all at once. Nobody stays simple for long.</p>
I did not come away from The Idiot</em> with a clean moral. I came away thinking about how badly people want tenderness and how often they wreck it the second it gets close. That felt true. Painfully true.</p>

If you have thoughts on the book, ping me on Twitter @aaqaishtyaq</a>.</em></p>


Firecracker Dev Machine Setup (GCP)
Unknown — Thu, 26 Jan 2023 00:00:00 +0000
VM on Google Compute Engine (GCE) supports nested virtualisation, unlike AWS, and allows to run KVM.
We can create a VM on GCE to run and test Firecracker microVM.</p>
Here are the steps</h3>


Setup GCP Project and Zone</p>
FC_PROJECT</span>=</span>y</span>o</span>u</span>r</span>_</span>n</span>a</span>m</span>e</span>-</span>f</span>i</span>r</span>e</span>c</span>r</span>a</span>c</span>k</span>e</span>r</span></span>
FC_REGION</span>=</span>u</span>s</span>-</span>e</span>a</span>s</span>t</span>1</span></span>
FC_ZONE</span>=</span>u</span>s</span>-</span>e</span>a</span>s</span>t</span>1</span>-</span>b</span></span></code></pre>gcloud</span> config</span> set</span> project</span> $</span>{</span>FC_PROJECT</span>}</span></span>
gcloud</span> config</span> set</span> compute/region</span> $</span>{</span>FC_REGION</span>}</span></span>
gcloud</span> config</span> set</span> compute/zone</span> $</span>{</span>FC_ZONE</span>}</span></span></code></pre></li>

Create a VM image (Machine Image) able to run KVM</p>
FC_VDISK</span>=</span>d</span>i</span>s</span>k</span>-</span>u</span>b</span>2</span>2</span></span>
FC_IMAGE</span>=</span>u</span>b</span>2</span>2</span>-</span>n</span>e</span>s</span>t</span>e</span>d</span>-</span>k</span>v</span>m</span></span>
</span>
gcloud</span> compute</span> disks</span> create</span> $</span>{</span>FC_VDISK</span>}</span> \</span></span>
  -</span>-image-project</span> ubuntu-os-cloud</span> \</span></span>
  -</span>-image-family</span> ubuntu-2204-lts</span></span>
</span>
gcloud</span> compute</span> images</span> create</span> $</span>{</span>FC_IMAGE</span>}</span> \</span></span>
  -</span>-source-disk</span> $</span>{</span>FC_VDISK</span>}</span> \</span></span>
  -</span>-source-disk-zone</span> $</span>{</span>FC_ZONE</span>}</span> \</span></span>
  -</span>-licenses</span> "</span>https://www.googleapis.com/compute/v1/projects/vm-options/global/licenses/enable-vmx</span>"</span></span></code></pre></li>

Create the VM</p>
FC_VM</span>=</span>f</span>i</span>r</span>e</span>c</span>r</span>a</span>c</span>k</span>e</span>r</span>-</span>v</span>m</span></span>
</span>
gcloud</span> compute</span> instances</span> create</span> $</span>{</span>FC_VM</span>}</span> \</span></span>
  -</span>-zone</span> $</span>{</span>FC_ZONE</span>}</span> \</span></span>
  -</span>-image</span> $</span>{</span>FC_IMAGE</span>}</span></span></code></pre></li>

Connect to the VM via SSH.</p>
gcloud</span> compute</span> ssh</span> $</span>{</span>FC_VM</span>}</span></span></code></pre></li>

Verify that VMX is enabled, enable KVM</p>
$</span> grep</span> -</span>cw</span> vmx</span> /proc/cpuinfo</span></span>
1</span></span>
$</span> sudo</span> setfacl</span> -</span>m</span> u:</span>$</span>{</span>USER</span>}</span>:rw</span> /dev/kvm</span></span>
$</span> [</span> -</span>r</span> /dev/kvm</span> ]</span> &&</span> [</span> -w</span> /dev/kvm </span>]</span> &&</span> echo</span> "</span>OK</span>"</span> |</span>|</span> echo</span> "</span>FAIL</span>"</span></span>
OK</span></span></code></pre></li>
</ol>


Linux Container Networking from Scratch
Unknown — Thu, 26 Jan 2023 00:00:00 +0000
In this article, we will be looking into setting up networking on a linux box from scratch.</p>
We will be creating a fresh new VM using Lima on macOS.
You can create a new VM using VirtualBox, Vagrant, or even create a VM on OCI for free.</p>
Isolation based on Network Namespace</h2>
The common thing that constitutes a network stack includes the:</p>

Network Devices</li>
Routing Rules</li>
Netfilter Hooks</li>
</ul>
We can create a non-comprehensive inspect-net-stack</code> script.</p>
#!</span>/usr/bin/env bash</span></span>
</span>
echo</span> "</span>> Network devices</span>"</span></span>
ip</span> link</span></span>
</span>
echo</span> -</span>e</span> "</span>\n> Route table</span>"</span></span>
ip</span> route</span></span>
</span>
echo</span> -</span>e</span> "</span>\n> Iptables rules</span>"</span></span>
iptables</span> -</span>-list-rules</span></span></code></pre>
After the execution of the inspect script on my machine produces the following input:</p>
%</span> sudo</span> ./inspect-net-stack</span></span>
></span> Network devices</span></span>
1:</span> lo:</span> <</span>LOOPBACK,UP,LOWER_U</span>P</span>></span> mtu</span> 65536</span> qdisc</span> noqueue</span> state</span> UNKNOWN</span> mode</span> DEFAULT</span> group</span> default</span> qlen</span> 1000</span></span>
    link/loopback</span> 00:00:00:00:00:00</span> brd</span> 00:00:00:00:00:00</span></span>
2:</span> eth0:</span> <</span>BROADCAST,MULTICAST,UP,LOWER_U</span>P</span>></span> mtu</span> 1500</span> qdisc</span> fq_codel</span> state</span> UP</span> mode</span> DEFAULT</span> group</span> default</span> qlen</span> 1000</span></span>
    link/ether</span> 52:55:55:a3:ad:92</span> brd</span> ff:ff:ff:ff:ff:ff</span></span>
    altname</span> enp0s1</span></span>
</span>
></span> Route table</span></span>
default</span> via</span> 192.168.5.3</span> dev</span> eth0</span> proto</span> dhcp</span> src</span> 192.168.5.15</span> metric</span> 100</span></span>
192.168.5.0/24</span> dev</span> eth0</span> proto</span> kernel</span> scope</span> link</span> src</span> 192.168.5.15</span> metric</span> 100</span></span>
</span>
></span> Iptables rules</span></span>
-P</span> INPUT</span> ACCEPT</span></span>
-P</span> FORWARD</span> ACCEPT</span></span>
-P</span> OUTPUT</span> ACCEPT</span></span></code></pre>
We want to make sure that each of the containers we are going to create soon will get a separate network stack.</p>
Linux namespaces used for container isolation are called network namespaces</a>. We won't be creating the fully isolated container, rather restrict the scope to only the network stack. One of the ways to create a network stack is by using the ip</code> tool, part of iproute2</code></a></p>
%</span> sudo</span> ip</span> netns</span> add</span> netns0</span></span>
%</span> ip</span> netns</span></span>
netns0</span></span></code></pre>
We have a new network namespace but to start using the network namespace, We can use a command called nsenter</code>. It enters one or more of the specified namespaces and then executes the given program:</p>
#</span> This is create a new bash process under netns0 namespace</span></span>
%</span> sudo</span> nsenter</span> -</span>-net=/var/run/netns/netns0</span> bash</span></span>
</span>
</span>
%</span> sudo</span> ./inspect-net-stack</span></span>
></span> Network devices</span></span>
1:</span> lo:</span> <</span>LOOPBAC</span>K</span>></span> mtu</span> 65536</span> qdisc</span> noop</span> state</span> DOWN</span> mode</span> DEFAULT</span> group</span> default</span> qlen</span> 1000</span></span>
    link/loopback</span> 00:00:00:00:00:00</span> brd</span> 00:00:00:00:00:00</span></span>
</span>
></span> Route table</span></span>
</span>
></span> Iptables rules</span></span>
-P</span> INPUT</span> ACCEPT</span></span>
-P</span> FORWARD</span> ACCEPT</span></span>
-P</span> OUTPUT</span> ACCEPT</span></span></code></pre>
As you can see from the above output, the bash</code> process running inside the netns0</code> namespace sees a different network stack. There are no routing rules and no custom iptables chain. Only one loopback device (lo</code>).</p>
                        Network</span> Namespace</span></span>
┌─────────────────────────────────────────────────────────────────────┐</span></span>
│</span>                                                                     │</span></span>
│</span>                                          Root</span> Namespace</span>             │</span></span>
│</span>                                                                     │</span></span>
│</span> lo0</span>                    ┌────────────────────────────────────────┐</span>   │</span></span>
│</span> ┌─┐</span>                    │</span>                                        │</span>   │</span></span>
│</span> │</span> │</span>                    │</span>    lo0</span>             Container</span> Namespace</span> │</span>   │</span></span>
│</span> │</span> │</span>                    │</span>    ┌─┐</span>              netns0</span>             │</span>   │</span></span>
│</span> │</span> │</span>    ┌────────────┐</span>  │</span>    │</span> │</span>                                 │</span>   │</span></span>
│</span> │</span> │</span>    │</span>            │</span>  │</span>    │</span> │</span>  ┌───────────┐</span>                  │</span>   │</span></span>
│</span> │</span> │</span>    │</span> routes</span>     │</span>  │</span>    │</span> │</span>  │</span>           │</span>                  │</span>   │</span></span>
│</span> │</span> │</span>    │</span>            │</span>  │</span>    │</span> │</span>  │routes</span>     │</span>                  │</span>   │</span></span>
│</span> │</span> │</span>    └────────────┘</span>  │</span>    │</span> │</span>  └───────────┘</span>                  │</span>   │</span></span>
│</span> └─┘</span>                    │</span>    └─┘</span>                                 │</span>   │</span></span>
│</span>                        │</span>                                        │</span>   │</span></span>
│</span> ┌─┐</span>    ┌────────────┐</span>  │</span>         ┌───────────┐</span>                  │</span>   │</span></span>
│</span> │</span> │</span>    │</span>            │</span>  │</span>         │</span>           │</span>                  │</span>   │</span></span>
│</span> │</span> │</span>    │</span> iptables</span>   │</span>  │</span>         │iptables</span>   │</span>                  │</span>   │</span></span>
│</span> │</span> │</span>    │</span>            │</span>  │</span>         └───────────┘</span>                  │</span>   │</span></span>
│</span> │</span> │</span>    └────────────┘</span>  │</span>                                        │</span>   │</span></span>
│</span> │</span> │</span>                    │</span>                                        │</span>   │</span></span>
│</span> │</span> │</span>                    │</span>                                        │</span>   │</span></span>
│</span> └─┘</span>                    │</span>                                        │</span>   │</span></span>
│</span> eth0</span>                   └────────────────────────────────────────┘</span>   │</span></span>
│</span>                                                                     │</span></span>
│</span>                                                                     │</span></span>
└─────────────────────────────────────────────────────────────────────┘</span></span></code></pre>
Beware</em>: The nsenter</code> command from above started a nested bash session in the netns0 network namespace. Don't forget to exit from it.</p>

In the upcoming posts, I will discuss connecting the host with this namespace. Interconnecting various network switches, Just like patching the physical switch with a LAN cable.</p>


Git exclude files from working copy
Unknown — Fri, 29 Jul 2022 00:00:00 +0000
How to ignore new files</h2>
Globally</h3>
Add the path(s) to your file(s) which you would like to ignore to your .gitignore</code> file (and commit them). These file entries will also apply to others checking out the repository.</p>
Locally</h3>
Add the path(s) to your file(s) which you would like to ignore to your .git/info/exclude</code> file. These file entries will only apply to your local working copy.</p>
How to ignore changed files (temporarily)</h2>
In order to ignore changed files to being listed as modified, you can use the following git command:</p>
git</span> update-index</span> -</span>-assume-unchanged</span> <</span>file-</span>1></span> <</span>file-</span>2></span> <</span>file-</span>3></span></span></code></pre>
To revert that ignorance use the following command:</p>
git</span> update-index</span> -</span>-no-assume-unchanged</span> <</span>file-</span>1></span> <</span>file-</span>2></span> <</span>file-</span>3></span></span></code></pre>
Note: If you find errors, typos or would like to add new tips, feel free to
reach out to me on twitter. I'm @aaqaishtyaq</a>. Thank
you for reading ! And if you find this useful, share it with your friends and
coworkers !</em></p>


Open markdown links in a new tab
Unknown — Fri, 05 Jun 2020 00:00:00 +0000
I created a static website linkbin</a>, Where I could save links in a markdown file that can be viewed later.
It is serving its purpose and because it uses hugo</a>, I don't have to think about VPS cost.</p>
Recently, I ran into an issue where my website serves links but when you click on it, It will open the link in the same tab. Ugggh! I don't want to close my website just to open a link. Ideally, it should open the link in a new tab.</p>
How can we do it? It would be easy Eh?</p>
The generic way</h2>
Since markdown is eventually rendered as HTML, The most naive way I found was to write HTML in markdown</p>
<</span>a</span> href</span>=</span>"</span>http://example.com/</span>"</span> target</span>=</span>"</span>_blank</span>"</span>></span>example</span></</span>a</span>></span></span></code></pre>
No way, I am going to rewrite all my links into HTML. It adds cost later on as well. I will have to add HTML tags to write the link instead of this syntax:</p>
[</span>link</span>]</span>(</span>example.com</span>)</span></span></code></pre>The specialized way</h2>
This could vary depending upon the markdown parser you are using.
If you're using Jekyll, like this website, then you can use the following syntax to open links in a new tab.</p>
[</span>link</span>]</span>(</span>url</span>)</span>{:target="_blank"}</span></span></code></pre>
But, I am not using Jekyll for linkbin</a>. It uses a fairly newer version of Hugo</a>. Hugo ,v0.62.0 or later</code>, uses Goldmark</a> parser and supports Markdown render hooks</a>. Markdown Render Hooks offer you several ways to extend the default markdown behavior, e.g. resizing uploaded images, opening links in new tabs, or creating mermaid diagrams from code. You can do this by creating templates in the layouts/_default/_markup</code> directory with base names render-link</code> or render-image</code> or render-codeblock</code>. Your directory layout may look like this:</p>
layouts</span></span>
└──</span> _default</span></span>
    └──</span> _markup</span></span>
        ├──</span> render-image.html</span></span>
        ├──</span> render-image.rss.xml</span></span>
        ├──</span> render-link.html</span></span>
        ├──</span> render-codeblock.html</span></span>
        └──</span> render-codeblock-bash.html</span></span></code></pre>Render hook to open link in a new tab</h3>
Add the following HTML template file (or render hook) at layouts/_default/_markup/render-link.html</code>:</p>
</span>
<</span>a</span> href</span>=</span>"</span>{{ .Destination | safeURL }}</span>"</span>{{</span> with</span> .Title}}</span> title</span>=</span>"</span>{{ . }}</span>"</span>{{</span> end</span> }}{{</span> if</span> strings.HasPrefix</span> .Destination</span> "http"</span> }}</span> target</span>=</span>"</span>_blank</span>"</span> rel</span>=</span>"</span>noopener</span>"</span>{{</span> end</span> }}</span>></span>{{ .Text | safeHTML }}</span></</span>a</span>></span></span></code></pre>
You'll find that the links now open in a new tab!
For internal blog links (which you would want to open in the same tab), you can use the relative link of the post, e.g. for an other-post.md</code> file within the posts</code> directory, you could use</p>
[</span>Other post</span>]</span>(</span>/posts/other-post/</span>)</span></span></code></pre>

If I’ve missed something or made a horrible mistake if you have any questions regarding this article then feel free to ping me on Twitter. I’m
@aaqaishtyaq</a>.</p>


The Recipe for a Great RESTful API
Unknown — Sun, 30 Dec 2018 00:00:00 +0000
I personally love writing APIs. It's my favourite thing about backend
development. As developers, we interact with APIs all the time: to process
payments with Stripe, handle geolocation with Google Maps, retrieve tweets,
etc.</p>
So what exactly makes a RESTful API great? What can you do to ensure that it's
easy for other developers to use? Let's dive right in.</p>
Responses</h2>
In your responses, you should separate the metadata from the body and put them in their own objects.</p>
//</span> Example: GET /api/user/2</span></span>
</span>
{</span></span>
</span>
    "</span>meta</span>"</span>:</span> {</span></span>
        "</span>type</span>"</span>:</span> "</span>success</span>"</span>,</span></span>
        "</span>code</span>"</span>:</span> 200</span>,</span></span>
        "</span>request_id</span>"</span>:</span> "</span>someid</span>"</span></span>
        //</span> etc ...</span></span>
    }</span>,</span></span>
</span>
    "</span>body</span>"</span>:</span> {</span></span>
</span>
        "</span>id</span>"</span>:</span> 2</span>,</span></span>
        "</span>name</span>"</span>:</span> "</span>Aaqa</span>"</span>,</span></span>
        "</span>location</span>"</span>:</span> "</span>Delhi</span>"</span>,</span></span>
        "</span>job</span>"</span>:</span> "</span>Software Developer</span>"</span></span>
    }</span></span>
</span>
}</span></span></code></pre>
When creating a new resource, always return a 201 created</code> instead of a 200 OK</code>. It's a common mistake developers make.
Another best practice is to return a url to the newly created resource in the location header of your response.
You can also choose to send a serialised version of the resource in the response body. (I personally like to do that).</p>
</span>
//</span> Example: POST /api/users</span></span>
</span>
//</span> Response Headers</span></span>
</span>
//</span> Content-Type: application/json</span></span>
//</span> ...</span></span>
//</span> Location: https://yoursite.com/api/users/2</span></span>
</span>
{</span></span>
    "</span>meta</span>"</span>:</span> {</span></span>
        "</span>type</span>"</span>:</span> "</span>success</span>"</span>,</span></span>
        "</span>code</span>"</span>:</span> 201</span>,</span></span>
        "</span>request_id</span>"</span>:</span> "</span>some_id</span>"</span></span>
    }</span>,</span></span>
</span>
    "</span>body</span>"</span>:</span> {</span></span>
</span>
        "</span>message</span>"</span>:</span> "</span>User successfully added</span>"</span>,</span></span>
        "</span>user</span>"</span>:</span> {</span></span>
</span>
            "</span>id</span>"</span>:</span> 3</span>,</span></span>
            "</span>name</span>"</span>:</span> "</span>Harry_Potter</span>"</span>,</span></span>
            "</span>location</span>"</span>:</span> "</span>London</span>"</span>,</span></span>
            "</span>job</span>"</span>:</span> "</span>Auror</span>"</span>,</span></span>
            "</span>href</span>"</span>:</span> "</span>https://yoursite.com/api/users/3</span>"</span></span>
        }</span></span>
    }</span></span>
}</span></span></code></pre>Errors</h3>
Properly dealing with errors is crucial to provide the best experience possible
to your users.  Just like with normal responses, always include the status code in the meta part.</p>
You should also provide a nice descriptive message detailing the cause of the
error as well as a potential solution to fix it.</p>
Doing this will help the consumers of your API to gracefully handle these errors on the client side. They will thank you for it.</p>
//</span> Example GET /api/secretendpoint</span></span>
</span>
{</span></span>
</span>
    "</span>meta</span>"</span>:</span> {</span></span>
        "</span>type</span>"</span>:</span> "</span>error</span>"</span>,</span></span>
        "</span>status</span>"</span>:</span> 401</span>,</span></span>
        "</span>request_id</span>"</span>:</span> "</span>weoifwe</span>"</span></span>
    }</span>,</span></span>
    "</span>body</span>"</span>:</span> {</span></span>
       "</span>message</span>"</span>:</span> "</span>Unauthorized. Please login or create an account</span>"</span></span>
    }</span></span>
}</span></span></code></pre>Status codes</h2>
This part is subject to many passionate debates, apparently it's a very touchy
topic.</p>
Below I will list the most common status codes, what they mean and how I use
them.</p>

200 - OK - Request went well, errthang's fine.</li>
201 - Resource created - The resource was successfully saved to the database.</li>
204 - No Content - There used to be a resource at this endpoint but not
anymore (Used for DELETE</code> requests)</li>
400 - Bad request - The request was badly formatted. (ex: Invalid JSON)</li>
401 - Unauthorized - Authentication failed due to invalid credentials.</li>
403 - Forbidden - Authentication passed but the user does not have permission to access the resource</li>
404 - Not Found - Well, you know what it is</a></li>
405 - Method Not Allowed - The HTTP verb used to interact with the resource is not allowed. (eg: User tried to POST to a GET only endpoint)</li>
409 - Conflict - The data passed in the payload conflicts with the resource in
the server. (I return this status code in cases where a user creates/updates a
resource with a unique field constraint attached to it such as a username or
email)</li>
422 - Unprocessable Entity - Again, another controversial one. I return a 422
when the user sends an incomplete payload thereby violating the NOT NULL constraint attached to those fields.</li>
429 - Too many requests - You made too many requests.</li>
50x - Server Related Errors - Something went wrong server side.</li>
</ul>
Caching</h3>
Caching is great and is a must when deploying an API. It helps a huge
amount with performace by reducing server load as you are not required to
perform the same action multiple times. You could choose to do it manually or use a reverse proxy like nginx or even something like varnish to handle that for you.</p>
Caching will return a 304 Not Modified</code> if the resource hasn't changed. The great thing about this is that it does not affect your rate limiting, so there's
literally no excuse not to implement it.</p>
Rate Limiting</h3>
For security and performance reasons you want limit the amount of requests made
against your API.
You must return the maximum and remaining number of request allowed for a given user in a particular time window (which you are free to determine).</p>
CORS</h3>
You want to enable CORS to allow requests from browsers using AJAX. This can
be done by setting the following headers:</p>
Access-Control-Allow-Origin:</span> *</span> #</span> this means allow all domains to interact with the API</span></span>
Access-Control-Allow-Headers:</span> #</span> All the headers you want to accept should be listed here</span></span></code></pre>A few tips</h3>

Return urls to resources and total count when GETting collections</li>
Map your endpoints to collections and resources</li>
Always use plural nouns (Ex: /api/users not /api/user)</li>
Actions on resources must be described with HTTP verbs</li>
for partial updates of resources use PATCH instead of PUT</li>
for versioning, use dates instead of numbers in the URL (foursquare does exactly that)</li>
</ul>

If you have any comment, suggestions or remarks of any sort, you can ping
me on twitter @aaqaishtyaq</a> !</p>


How to Set Up the Scrapy Splash Plugin
Unknown — Sat, 01 Sep 2018 00:00:00 +0000
Scrapy is great for scraping static web pages with Python, but when it comes to
dynamic pages it can only do so much. That's where Selenium</code> usually comes
in, but as good as Selenium is, Scrapy still beats it in terms of speed.</p>
The web these days is full of dynamic JS-based pages and AJAX. For exactly that
scenario, the folks over at scrapy-plugins</a> created scrapy-splash</code>.
Scrapy-Splash is a plugin that connects Scrapy with Splash (a lightweight,
scriptable browser as a service with an HTTP API). In a nutshell, Splash takes
the response received from the server and renders it. Then it returns a
render.html</code> response which is static and can be scraped easily.</p>
0 - Setting up the machine</h2>
A. Before we begin you need to install Docker</code> first, You can follow the official instruction</a> as per your Operating System.</p>
B. After installing Docker, navigate to your project folder, activate
virtualenv</code> and install the scrapy-splash plugin</p>
pip3</span> install</span> scrapy-splash</span></span></code></pre>
C. Pull the Splash Docker Image and run it</p>
docker</span> pull</span> scrapinghub/splash</span></span>
docker</span> run</span> -</span>p</span> 8050:8050</span> scrapinghub/splash</span></span></code></pre>1 - Configuration</h2>
A. Add the Splash server address to settings.py</code> of your Scrapy project like this:</p>
SPLASH_URL</span> =</span> '</span>http://localhost:8050</span>'</span></span></code></pre>
If you are running docker on your local machine then you can simply use http://localhost:<port></code> , but if you are running it on a remote machine you need to specify it's I.P. Address like this http://192.168.59.103:<port></code></p>
B. Enable the Splash middleware by adding it to DOWNLOADER_MIDDLEWARES</code> in your settings.py</code> file and changing HttpCompressionMiddleware priority:</p>
DOWNLOADER_MIDDLEWARES</span> =</span> {</span></span>
    '</span>scrapy_splash.SplashCookiesMiddleware</span>'</span>:</span> 723</span>,</span></span>
    '</span>scrapy_splash.SplashMiddleware</span>'</span>:</span> 725</span>,</span></span>
    '</span>scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware</span>'</span>:</span> 810</span>,</span></span>
}</span></span></code></pre>
C. Enable SplashDeduplicateArgsMiddleware by adding it to SPIDER_MIDDLEWARES in your settings.py:</p>
SPIDER_MIDDLEWARES</span> =</span> {</span></span>
    '</span>scrapy_splash.SplashDeduplicateArgsMiddleware</span>'</span>:</span> 100</span>,</span></span>
}</span></span></code></pre>
D. Set a custom DUPEFILTER_CLASS:</p>
DUPEFILTER_CLASS</span> =</span> '</span>scrapy_splash.SplashAwareDupeFilter</span>'</span></span></code></pre>2 - Scraping with Splash</h2>
Before you use scrapy-splash</code> you need to import it in your spider.
You can do that by adding this line:</p>
from</span> scrapy_splash</span> import</span> SplashRequest</span></span></code></pre>
from now on insted of using scrapy.Request</code> you can simply use SplashRequest</code> to get response from Splash</code> insted of directly from ther server.</p>
Bonus - Using Scrapy-Splash in Shell</h2>
It's all well and good but actual spider buiding does not happens in vim</code> or sublime</code>, it takes place in shell</code>.</p>
So how to use Splash in the shell?</strong></p>
Good Question.</p>
Insted of invoking shell with:</p>
scrapy</span> shell</span></span>
>>></span> fetch(http://domain.com/page-with-javascript.html</span>)</span></span></code></pre>
or with this:</p>
scrapy</span> shell</span> http://domain.com/page-with-javascript.html</span></span></code></pre>
You invoke shell with this</strong>:</p>
scrapy</span> shell</span> '</span>http://localhost:8050/render.html?url=http://domain.com/page-with-javascript.html&timeout=10&wait=0.5</span>'</span></span></code></pre>
Let me explain</p>

localhost:port</code> is where your splash service is running</li>
url</code> is url you want to crawl</li>
render.html</code> is one of the possible http api endpoints, returns redered html page in this case</li>
timeout</code> time in seconds for timeout</li>
wait</code> time in seconds to wait for javascript to execute before reading/saving the html.</li>
</ul>
If I’ve missed something, made a horrible mistake of if you have any questions regarding this article then feel free to ping me on Twitter. I’m
@aaqaishtyaq</a>.</p>


Build and Deploy a React App with Sass and Nginx
Unknown — Sat, 02 Jun 2018 00:00:00 +0000
A couple of days ago (at the time of writing), I started my newest side
project. It's a portfolio showcasing my (very very very) amateur
photography. It's written in React.js with Sass and I have to say it was
extremely enjoyable to work on. Unsurprisingly though, I ran into some issues
while deploying to production which, after a lot of head banging against every
possible flat surface I could find, I managed to sort out. So this post will be
about how to make React.js work with Sass in production and how to serve the
project using Nginx as a front-end web server.</p>
We'll be using the official starter kit / CLI tool provided by the Facebook
team called create-react-app</code>.</p>
You can install it by running the following command:</p>
npm</span> install</span> -</span>g</span> create-react-app</span></span></code></pre>Adding Sass to a React project</h2>
Including Sass in a React app can be done in two ways: You can either eject the
project and manually modify the webpack config files or you can follow the
procedure in the official docs</a>. I've personally chosen to go with the first approach.
You can just follow the steps in this tutorial</a> to get up and running.</p>
Note: If you went with the second approach, you can just skip the following and
directly jump to the next section</em></p>
Once you've done that, open the config/webpack.config.prod.js file, locate the
rules</code> section under module.exports</code> and add the following snippet to it.</p>
{</span></span>
    test</span>:</span> /</span>\.</span>sass</span>$</span>/</span>,</span></span>
    use</span>:</span> ExtractTextPlugin</span>.</span>extract</span>(</span>{</span></span>
        fallback</span>:</span> require</span>.</span>resolve</span>(</span>"</span>style-loader</span>"</span>)</span>,</span></span>
        use</span>:</span> [</span>require</span>.</span>resolve</span>(</span>"</span>css-loader</span>"</span>)</span>,</span> require</span>.</span>resolve</span>(</span>"</span>sass-loader</span>"</span>)</span>]</span></span>
    }</span>)</span>,</span></span>
    include</span>:</span> paths</span>.</span>appSrc</span>,</span></span>
}</span></span></code></pre>
This config will be executed when you call npm run build</code>. If you don't do
this, you'll end up with an empty css file.</p>
Important note</h2>
Before building the project for the first time, we must first unregister the
service worker. Doing this will ensure that the static assets aren't cached by
the client's browsers. It's important because if you skip this step, any
subsequent change / build you'll deploy won't necessarily be reflected right away
client side. (I'm not knowledgable enough on service workers to provide details
on this behaviour, but it's something to note. If you really need service
workers in your project, you might want to explore other solutions to avoid
asset caching).</p>
Your index.js file should now look something like this:</p>
import</span> {</span> unregister</span> }</span> from</span> '</span>./registerServiceWorker</span>'</span></span>
import</span> App</span> from</span> '</span>./App</span>'</span>;</span></span>
ReactDOM</span>.</span>render</span>(</span><</span>App</span> /></span>,</span> document</span>.</span>getElementById</span>(</span>'</span>root</span>'</span>)</span>)</span>;</span></span>
unregister</span>(</span>)</span></span></code></pre>Building the project</h2>
This is the part where we actually build our project. By building I mean
compiling all React files and their related dependencies, transpiling them into
vanilla Javascript, compiling sass files into css, minifying them etc ...
To do this, we can simply run npm run build</code> and voila ! You should have a
brand new build</code> folder in your project.</p>
Creating our deploy script</h2>
Your app is now compiled and ready to be served. Now is a good time to start
thinking about deployment strategies. In this basic scenario
(and quite frankly, most scenraios), you really just want to upload the build
folder to a remote server and have it accessible as a static site.</p>
You can use ftp to transfer the files and it would be perfectly acceptable, but
it's not the most flexible solution. The alternative is to use a CLI utility called
rsync</code> (which is available on mac and linux, not sure about windows). With
rsync, you can synchronise files and folders within the same computer or across
machines you have ssh access to.</p>
Here's the command we would run to synchronise the build folder to a server
on the internet:</p>
#</span> Asssuming we're inside the project folder</span></span>
rsync</span> -</span>avP</span> build/</span> username@remote_ip:/destination/path</span></span></code></pre>
Let's break down this command:</p>
-a</code> means archive, which is a shortcut for multiple switches. It
recursively syncs all files and subfolders within build</code> to the
destnation path, keeping the modification dates, permissions and other metadata
unchanged.</p>
-v</code> means verbose. It just outputs the steps to the screen so you can see
what happens in real time.</p>
-P</code> stands for progress. This is particularly useful in this case because you
rely on a network connection to sync the files. Using this option will
display a progress bar for each file in the queue.</p>
But you don't want to keep doing all of that every time we want to push now do
you ?</p>
Thankfully, you can use create a bash script to automate this process a litte
bit. Here's how mine looks like:</p>
#!</span>/bin/sh</span></span>
</span>
echo</span> "</span>[log] - Merging branch to master</span>"</span></span>
git</span> checkout</span> master</span> &&</span> git</span> merge</span> develop</span> &&</span> git</span> push</span> origin</span> master</span></span>
echo</span> "</span>[log] - Merge completed</span>"</span></span>
</span>
echo</span> "</span>[log] - Compiling project to build folder ...</span>"</span></span>
npm</span> run</span> build</span></span>
echo</span> "</span>[log] - Build process done</span>"</span></span>
</span>
echo</span> "</span>[log] - Deploying files to server</span>"</span></span>
rsync</span> -</span>avP</span> build/</span> user@host:/destination/path</span></span>
echo</span> "</span>[log] - Deployment completed</span>"</span></span>
</span>
echo</span> "</span>[log] - Switching to develop</span>"</span></span>
git</span> checkout</span> develop</span></span>
echo</span> "</span>[log] - Done!</span>"</span></span></code></pre>
Again, let's walk through that script section by section:</p>

I checkout to master, merge develop and push. This ensures that my master
branch is always up to date with the latest version of my working codebase.</li>
I execute npm run build</code> which, as previously explained, will create the
build directory with our compiled, ready to be deployed files.</li>
I use rsync to copy over the contents of the build folder to the destination
path in the remote machine I administer. (notice the trailing slash after
build/, this tells rsync to copy the contents of the folder and not the
folder itself).</li>
I switch the current working branch back to develop so that I can start
developing without accidentally altering the state of master</code>.</li>
</ol>
Obviously this is very basic and in a more complex project, you'd have to run
unit tests and do other things your project requires.</p>
Finally, you need to give executable permissions to the file by running:</p>
sudo</span> chmod</span> +x</span> deploy.sh</span></span></code></pre>
Now all you have to do when you want to deploy your project to production is run</p>
./deploy.sh</span></span></code></pre>Serving our site with Nginx</h2>
Note: this assumes your server is running ubuntu or any other debian based
distro</em></p>
Ok, so if you've followed the steps correctly, you should have your project
files uploaded to your remote server. Now we need to use Nginx to make the site
accessible to the internet.</p>
First, create a new config file inside /etc/nginx/sites-available</code>.</p>
cd</span> /etc/nginx/sites-available</span></span>
touch</span> mywebsiteconfig</span> #</span> File extension is not required in this case</span></span></code></pre>
Next step is to edit the file using either vim or nano (sudo privileges may be
required).</p>
server</span> {</span></span>
</span>
    listen</span> 80</span>;</span></span>
    server_</span>name</span> mywebsite.com www.mywebsite.com</span>;</span></span>
</span>
    location</span> / </span>{</span></span>
        root</span> /path/to/your/project</span>;</span></span>
        index</span> index.html index.htm</span>;</span></span>
</span>
        default_type</span> "</span>text/html</span>"</span>;</span></span>
    }</span></span>
</span>
    access_log</span> /var/log/nginx/mywebsite_access.log</span>;</span></span>
    error_</span>log</span> /var/log/nginx/mywebsite_errors.log</span>;</span></span>
}</span></span></code></pre>
If you're not familiar with Nginx, let me explain what you just copied.</p>

We created a server block to hold our configuration and keep it separate from
other configs we may add in the future (like https for example).</li>
We declare the listen directive which tells nginx to listen on port 80.</li>
We set the server_name to our domain name. This tells nginx to apply the
config settings to any incoming request from any of listed urls.</li>
Finally, we specify the paths to both the access and error logs. It's
optional but highly recommended, so that you know exactly where to look when
errors happen. This will save you a tonne of time when troubleshooting issues
in the future.</li>
</ol>
Close the file and exit nano or vim.</p>
Nginx keeps its configuration files in two separate directories:
/etc/nginx/sites-available</code> and /etc/nginx/sites-enabled</code>. It will serve
any website whose configuration file is in the latter folder.</p>
All you have to do now is create a symlink (think of it like a shortcut to an app
in a desktop GUI) to your config, and store it in sites-enabled</code>.
That way, if you ever decide to shut down the site, you'll simply need to
delete the symlink and you're good.</p>
Before creating the symlink, it's good measure to check if the configuration
file has any errors in it. To check for errors just run the following command:</p>
sudo</span> nginx</span> -</span>t</span></span></code></pre>
It will scan all of your config files and check for errors (and return them to
you if there are any).</p>
Now you can create the symlink by executing:</p>
sudo</span> ln</span> -</span>s</span> /etc/nginx/sites-available/mywebsite</span> /etc/nginx/sites-enabled</span></span></code></pre>
Note: If you decide to use service workers with your project and you're still
running into caching issues, you should know that Nginx can also be used as a
static assets server. I haven't looked into this scenario yet. I might test that
approach in the future and detail the process in another post.</em></p>
Lastly, reload nginx and your website should now be accessible via its URL
(provided you correctly setup the DNS settings with your domain name registrar)</p>
sudo</span> service</span> nginx</span> reload</span></span></code></pre>
If you read this article all the way through, thanks for sticking with me ! You
can send me questions, remarks, or comments on twitter, I'm @aaqaishtyaq</a> on
twitter.</p>


The Ultimate PostgreSQL Cheat Sheet
Unknown — Wed, 09 May 2018 00:00:00 +0000
So I'd been working with Django, Flask, and Express.js for a while, and my
database of choice for pretty much every project had, of course, always been
Postgres.
What makes these frameworks great (regardless of language) is the ability to use
an ORM (Object Relational Mapper) that sits between your code and the DB.
It does all the heavy lifting for you and takes care of executing SQL queries on
your behalf.</p>
This is great, but I couldn't stop thinking, "what if I have to manually debug
something directly in Postgres?" That was when reality slapped me in the face:
I barely knew SQL. So I thought it'd be fun to create a cheat sheet that both I
and you could keep around for those times when you absolutely need to set a
column to UNIQUE</code> and you don't know how.</p>
Enjoy ! (and yeah I went all in with the pokemon references)</p>
Create a User (or Role)</h3>
/*</span> create a user without privileges</span>*/</span></span>
CREATE</span> ROLE</span> aaqa;</span></span>
</span>
/*</span> create a user with privileges</span>*/</span></span>
CREATE</span> ROLE</span> aaqa </span>LOGIN</span> CREATEDB CREATEROLE REPLICATION;</span></span>
</span>
/*</span> Add privileges to existing user</span>*/</span></span>
ALTER</span> ROLE</span> aaqa </span>WITH</span> LOGIN</span> CREATEROLE CREATEDB REPLICATION;</span></span></code></pre>Make a user superuser (bump their privileges)</h3>
ALTER</span> ROLE</span> aaqa </span>WITH</span> superuser;</span></span></code></pre>Rename an existing user</h3>
ALTER</span> ROLE</span> psyduck RENAME </span>TO</span> brock;</span></span></code></pre>Create a DB</h3>
CREATE</span> DATABASE</span> pokemons</span>;</span></span></code></pre>Rename a DB</h3>
ALTER</span> DATABASE</span> pokemons RENAME </span>TO</span> charizard;</span></span></code></pre>Create a table in a DB</h3>
CREATE</span> TABLE</span> trainers</span>(</span></span>
</span>
    id </span>INT</span> PRIMARY KEY</span> NOT NULL</span>,</span></span>
    pokemon_type </span>CHAR</span>(</span>50</span>)</span>,</span></span>
    name</span> CHAR</span>(</span>50</span>)</span> NOT NULL</span> UNIQUE</span>,</span></span>
    gender </span>CHAR</span>(</span>50</span>)</span> NOT NULL</span> UNIQUE</span></span>
);</span></span></code></pre>Delete a DB</h3>
DROP</span> DATABASE</span> pokemons;</span></span></code></pre>Delete a user</h3>
/*</span> assuming there's a pikachu role on the system </span>*/</span></span>
DROP</span> ROLE</span> pikachu;</span></span></code></pre>Change DB ownership</h3>
ALTER</span> DATABASE</span> pokemons </span>OWNER</span> TO</span> aaqa;</span></span></code></pre>Rename a table</h3>
ALTER</span> TABLE</span> trainers RENAME </span>TO</span> gym_trainers;</span></span></code></pre>Change column type</h3>
ALTER</span> TABLE</span> gym_trainers </span>ALTER</span> COLUMN pokemon_type </span>TYPE</span> TEXT</span>; </span>/*</span> it makes zero</span></span>
sense to want to change this column type to TEXT but YOLO </span>*/</span></span></code></pre>Rename a column</h3>
ALTER</span> TABLE</span> gym_trainers </span>ALTER</span> COLUMN </span>name</span> RENAME </span>TO</span> trainer_name;</span></span></code></pre>Add a column to a table</h3>
ALTER</span> TABLE</span> gym_trainers </span>ADD</span> COLUMN bio </span>TEXT</span>;</span></span>
</span>
/*</span></span>
ALTER TABLE <table_name></span></span>
ADD COLUMN <column_name> <data_type> <constraints if any></span></span>
*/</span></span></code></pre>Add a column with a UNIQUE constraint</h3>
ALTER</span> TABLE</span> gym_trainers </span>ADD</span> COLUMN age </span>INT</span> UNIQUE</span>; </span>/*</span> cause why not </span>*/</span></span></code></pre>Add a column with a NOT NULL constraint</h3>
ALTER</span> TABLE</span> gym_trainers </span>ADD</span> COLUMN main_pokemon </span>CHAR</span>(</span>60</span>)</span> NOT NULL</span>;</span></span></code></pre>Remove NOT NULL CONSTRAINT from a column</h3>
ALTER</span> TABLE</span> gym_trainers </span>ALTER</span> COLUMN main_pokemon </span>DROP</span> NOT NULL</span>;</span></span></code></pre>Add a column with a NOT NULL constraint and a DEFAULT value</h3>
ALTER</span> TABLE</span> gym_trainers </span>ADD</span> COLUMN city </span>CHAR</span>(</span>80</span>)</span> NOT NULL</span> DEFAULT</span> '</span>Indigo</span></span>
Plateau</span>'</span>;</span></span></code></pre>Add a column with a CHECK constraint</h3>
</span>
CREATE</span> TABLE</span> trainers</span>(</span></span>
</span>
    id </span>INT</span> PRIMARY KEY</span>,</span></span>
    trainer_name </span>CHAR</span>(</span>50</span>)</span> NOT NULL</span>,</span></span>
</span>
    /*</span></span>
        method 1</span></span>
        Add the check yolo style</span></span>
    */</span></span>
    age </span>INT</span> NOT NULL</span> CHECK</span> (age </span>></span> 18</span>)</span></span>
</span>
    /*</span></span>
        method 2</span></span>
        Add a named constraint for better error handling</span></span>
    */</span></span>
    age </span>INT</span> CONSTRAINT</span> legal_age </span>CHECK</span> (age </span>></span> 18</span>) </span>NOT NULL</span></span>
</span>
    /*</span></span>
        method 3</span></span>
        Add the constraint at the end</span></span>
        for more clarity</span></span>
    */</span></span>
    age </span>INT</span> NOT NULL</span>,</span></span>
    CONSTRAINT</span> legal_age </span>CHECK</span> (age </span>></span> 18</span>) </span>NOT NULL</span></span>
);</span></span></code></pre>Add a CONSTRAINT to an existing column</h3>
ALTER</span> TABLE</span> trainers </span>ADD</span> CONSTRAINT</span> unique_name </span>UNIQUE</span> (trainer_name);</span></span></code></pre>Remove a named CONSTRAINT from a table</h3>
ALTER</span> TABLE</span> trainers </span>DROP</span> CONSTRAINT</span> unique_name;</span></span></code></pre>Insert a row into a table</h3>
INSERT INTO</span> trainers </span>VALUES</span> (</span>1</span>, </span>23</span>, </span>'</span>brock</span>'</span>);</span></span>
</span>
/*</span> or </span>*/</span></span>
</span>
INSERT INTO</span> trainers (age, trainer_name)</span></span>
VALUES</span> (</span>1</span>, </span>23</span>, </span>'</span>brock</span>'</span>);</span></span>
</span>
/*</span></span>
    Note that in the second case we don't have pass</span></span>
    the id. Postgres will automatically generate and autoincrement</span></span>
    it for us. To omit the id column we must use named inserts otherwise an</span></span>
    error is raised.</span></span>
*/</span></span></code></pre>Insert multiple rows into a table</h3>
</span>
INSERT INTO</span> trainers (age, trainer_name) </span>VALUES</span></span>
(</span>19</span>, </span>'</span>misty</span>'</span>),</span></span>
(</span>22</span>, </span>'</span>chen</span>'</span>),</span></span>
/*</span> ... </span>*/</span></span>
/*</span> ... </span>*/</span></span></code></pre>Clear a table (without deleting it)</h3>
TRUNCATE</span> trainers;</span></span></code></pre>Set the primary key type to a serial (An auto incrementing integer)</h3>
</span>
/*</span> Considering this table structure </span>*/</span></span>
</span>
CREATE</span> TABLE</span> pokemon_list</span> (</span></span>
</span>
    id </span>INT</span> PRIMARY KEY</span>,</span></span>
    /*</span> ... </span>*/</span></span>
    /*</span> ... </span>*/</span></span>
    /*</span> ... </span>*/</span></span>
);</span></span>
</span>
</span>
/*</span></span>
    1.</span></span>
    Create a sequence for the auto generating prinary key</span></span>
    It follows the tablename_columnname_seq</span></span>
*/</span></span>
CREATE</span> SEQUENCE</span> pokemon_list_id_seq</span>;</span></span>
</span>
/*</span> 2. Set the id column to not null </span>*/</span></span>
ALTER</span> TABLE</span> pokemon_list </span>ALTER</span> COLUMN id </span>SET</span> NOT NULL</span>;</span></span>
</span>
/*</span> 3. Set the default value to the next value in the sequence</span>*/</span></span>
ALTER</span> TABLE</span> pokemon_list</span></span>
ALTER</span> COLUMN pokemon_list</span></span>
SET</span> DEFAULT</span> nextval(</span>'</span>pokemon_list_id_seq</span>'</span>);</span></span>
</span>
/*</span> 4. Link the sequence to the correct table and column </span>*/</span></span>
ALTER</span> SEQUENCE</span> pokemon_list_id_seq OWNED </span>BY</span> pokemon_list</span>.</span>id</span>;</span></span></code></pre>Import data from a file</h3>
/*</span></span>
    Must use absolute path and the user must have appropriate permissions</span></span>
    Defaults to importing using TAB as the default parameter.</span></span>
    We'll use a csv file as an example</span></span>
*/</span></span>
</span>
COPY</span> pokemon_list </span>FROM</span> '</span>/path/to/yourfile.csv</span>'</span> DELIMITER </span>'</span>,</span>'</span>;</span></span>
</span>
/*</span></span>
    This only works if a pk is specified for each row</span></span>
    The (my) prefered way to do it is the following</span></span>
*/</span></span>
</span>
COPY</span> pokemon_list (</span>name</span>, </span>level</span>, </span>type</span>) </span>FROM</span> '</span>/path/to/yourfile.csv</span>'</span> DELIMITER </span>'</span>,</span>'</span>;</span></span>
</span>
/*</span> It's much more flexible because you control what data you actually import</span>*/</span></span></code></pre>Export a table to a file</h3>
To be able to export a table to a file, we need to ensure that postgres has
write permissions to the file.</p>
sudo</span> chmod</span> 777</span> /path/to/directory</span> #</span> This is just an example, edit this as needed</span></span></code></pre>
We can now safely copy the table to the file.</p>
COPY</span> pokemon_list </span>TO</span> '</span>/path/to/file.csv</span>'</span> DELIMITER </span>'</span>,</span>'</span>;</span></span></code></pre>Select columns by using aliases</h3>
SELECT</span> name</span> AS</span> pokemon_name, </span>type</span> AS</span> pokemon_type</span></span>
FROM</span> pokemon_list;</span></span></code></pre>Select elements based on a criteria</h3>
SELECT</span> *</span></span>
FROM</span> pokemon_list</span></span>
WHERE</span> id </span>></span> 3</span>;</span></span></code></pre>Select elements based on string comparison</h3>
SELECT</span> *</span></span>
FROM</span> pokemon_list</span></span>
WHERE</span> type</span> LIKE</span> '</span>%water%</span>'</span>;</span></span></code></pre>Select all results and order them by id in reverse</h3>
SELECT</span> *</span></span>
FROM</span> pokemon_list</span></span>
ORDER BY</span> id </span>DESC</span>;</span></span></code></pre>Select all results and order them by a column name</h3>
SELECT</span> *</span></span>
FROM</span> pokemon_list</span></span>
ORDER BY</span> level</span>;</span></span>
</span>
/*</span> If the column you're ordering by is not of type INT then the ordering will be</span></span>
done alphabetically </span>*/</span></span></code></pre>Select DISTINCT column from table</h3>
SELECT DISTINCT</span> type</span> AS</span> pokemon_type</span></span>
FROM</span> pokemon_list;</span></span></code></pre>Limit the results from a SELECT query</h3>
SELECT</span> *</span></span>
FROM</span> pokemon_list</span></span>
LIMIT</span> 3</span>;</span></span></code></pre>Select the last 3 items</h3>
SELECT</span> *</span></span>
FROM</span> pokemon_list</span></span>
ORDER BY</span> id </span>DESC</span></span>
LIMIT</span> 3</span>;</span></span></code></pre>Create two tables with a foreign key relationship</h3>
CREATE</span> TABLE</span> pokemon_types</span>(</span></span>
</span>
    id </span>SERIAL</span> PRIMARY KEY</span>,</span></span>
    type_name </span>CHAR</span>(</span>120</span>)</span> NOT NULL</span></span>
)</span></span>
</span>
CREATE</span> TABLE</span> pokemon_list</span>(</span></span>
</span>
    id </span>serial</span> PRIMARY KEY</span>,</span></span>
    pokemon_name </span>CHAR</span>(</span>120</span>)</span> NOT NULL</span>,</span></span>
    pokemon_level </span>INT</span> NOT NULL</span>,</span></span>
    pokemon_type </span>INT</span> REFERENCES</span> pokemon_types(id) </span>NOT NULL</span>,</span></span>
    CONSTRAINT</span> pokemon_level_not_zero </span>CHECK</span> (pokemon_level </span>></span> 0</span>)</span></span>
</span>
);</span></span></code></pre>Perform Joins based on a criteria</h3>
SELECT</span> name</span>, </span>level</span>, </span>pokemon_types</span>.</span>name</span> AS</span> type</span></span>
FROM</span> pokemon_list</span></span>
JOIN</span> pokemon_types</span></span>
ON</span> pokemon_type_id  </span>=</span> pokemon_types</span>.</span>id</span></span>
WHERE</span> pokemon_type_id </span>=</span> 1</span>;</span></span>
</span>
/*</span> Will return the name, level and type name for all water pokemons </span>*/</span></span></code></pre>Perform joins</h3>
SELECT</span> name</span>, </span>level</span>, </span>pokemon_types</span>.</span>name</span> AS</span> type</span></span>
FROM</span> pokemon_list</span></span>
JOIN</span> pokemon_types</span></span>
ON</span> pokemon_type_id  </span>=</span> pokemon_types</span>.</span>id</span>;</span></span></code></pre>Create a VIEW based on a JOIN</h3>
CREATE</span> VIEW</span> pokemonswithtypes</span> AS</span></span>
SELECT</span> name</span>, </span>level</span> pokemon_types</span>.</span>name</span> AS</span> type</span></span>
FROM</span> pokemon_list</span></span>
JOIN</span> pokemon_types</span></span>
ON</span> pokemon_type_id </span>=</span> pokemon_types</span>.</span>id</span>;</span></span>
</span>
/*</span> To see the data </span>*/</span></span>
</span>
SELECT</span> *</span> FROM</span> pokemonswithtypes;</span></span></code></pre>Update the VIEW (Change the query)</h3>
SELECT</span> name</span>, </span>pokemon_types</span>.</span>name</span> AS</span> type</span></span>
FROM</span> pokemon_list</span></span>
JOIN</span> pokemon_types</span></span>
ON</span> pokemon_type </span>=</span> pokemon_types</span>.</span>id</span></span>
WHERE</span> pokemon_type </span>=</span> 1</span>;</span></span>
</span>
/*</span> only show the name and type for water pokemons </span>*/</span></span></code></pre>Delete the VIEW</h3>
DROP</span> VIEW</span> pokemonswithtypes;</span></span></code></pre>Use aggregate functions (MIN, MAX, SUM, COUNT, AVG)</h3>
/*</span> MAX </span>*/</span></span>
SELECT</span> MAX</span>(</span>pokemon_level)</span></span>
FROM</span> pokemon_list;</span></span>
</span>
/*</span> MIN </span>*/</span></span>
SELECT</span> MIN</span>(</span>pokemon_level)</span></span>
FROM</span> pokemon_list;</span></span>
</span>
/*</span> AVG </span>*/</span></span>
SELECT</span> AVG</span>(</span>pokemon_level)</span></span>
FROM</span> pokemon_list;</span></span>
</span>
/*</span> ROUND </span>*/</span></span>
SELECT</span> ROUND</span>(</span>AVG</span>(</span>pokemon_level))</span></span>
FROM</span> pokemon_list;</span></span>
</span>
/*</span> COUNT </span>*/</span></span>
SELECT</span> COUNT</span>(</span>*</span>)</span></span>
FROM</span> pokemon_list;</span></span>
</span>
/*</span> SUM </span>*/</span></span>
SELECT</span> SUM</span>(</span>pokemon_level)</span></span>
FROM</span> pokemon_list;</span></span></code></pre>Use boolean aggregate functions</h3>
/*</span> Add a column is_legendary of type boolean to table pokemon_list </span>*/</span></span>
ALTER</span> TABLE</span> pokemon_list </span>ADD</span> COLUMN is_legendary BOOL </span>NOT NULL</span> DEFAULT</span> TRUE;</span></span>
</span>
/*</span>  BOOL_AND</span></span>
    returns a result if **ALL** records have that column set to true</span></span>
*/</span></span>
</span>
SELECT</span> BOOL_AND(is_legendary) </span>FROM</span> pokemon_list;</span></span>
</span>
</span>
/*</span></span>
    BOOL_OR</span></span>
    returns a result if one or more records have that column set to true</span></span>
*/</span></span>
</span>
SELECT</span> BOOL_OR(is_legendary) </span>FROM</span> pokemon_list;</span></span></code></pre>Update a table and change all column values</h3>
UPDATE</span> pokemon_list</span></span>
SET</span> is_legendary </span>=</span> FALSE;</span></span></code></pre>Update a table and change value based on a criteria</h3>
UPDATE</span> pokemon_list</span></span>
SET</span> is_legendary </span>=</span> TRUE</span></span>
WHERE</span> id </span>=</span> 2</span>;</span></span></code></pre>Delete row with specific id</h3>
DELETE</span> FROM</span> pokemon_list </span>WHERE</span> id </span>=</span> 4</span></span></code></pre>Delete rows withing a range of ids</h3>
DELETE</span> FROM</span> pokemon_list</span></span>
WHERE</span> id </span>BETWEEN</span> 1</span> AND</span> 4</span>;</span></span></code></pre>Delete all rows</h3>
DELETE</span> FROM</span> pokemon_list;</span></span></code></pre>
Note: The difference between DELETE and DROP or TRUNCATE is that the former can
be undone (rolled back) the latter can't</em></p>
Alter a table to drop a CONSTRAINT if it exist</h3>
ALTER</span> TABLE</span> pokemon_types</span></span>
DROP</span> CONSTRAINT</span> IF</span> EXISTS</span> unique_type_name;</span></span></code></pre>Comments</h3>
COMMENT ON TABLE </span>pokemon_types </span>is</span> '</span>pokemon with types</span>'</span></span>
</span>
/*</span> To display the comment, in psql simply run \dt+. It will return a description</span></span>
column containing that comment. It's useful when working on a legacy database</span></span>
for example</span>*/</span></span>
</span>
/*</span> Please note that comments aren't exclusive to tables, they can be executed on</span></span>
schemas and multiple other objects.</span>*/</span></span></code></pre>
Note: If you find errors, typos or would like to add new tips, feel free to
reach out to me on twitter. I'm @aaqaishtyaq</a>. Thank
you for reading ! And if you find this useful, share it with your friends and
coworkers !</em></p>


The Ultimate Remote Development Setup
Unknown — Tue, 08 May 2018 00:00:00 +0000
I'm a programmer. Like most programmers, I use an Apple laptop. They're just the
best on the market and the best purchase you can make as a developer. No
fanboyism here, just stating facts. They're fast, well built, and durable.</p>
I rely on this laptop for all my personal / client work. I run Vagrant boxes,
Docker containers on it and all the usual stuff.</p>
As a result, I've developed this irrational fear that it would get stolen or
fall off my bedroom window (things like that happen, trust me). Interestingly
enough, it coincides with my growing interest for remote work.</p>
So I started to look for ways to create a development server that would allow me
to remain productive even if I lose / break my laptop, or when I'm on the road.</p>
Why</h2>
Ok, so you've read all this and you're thinking "How is it going to benefit me ?".</p>
The main selling points to creating a remote development environment are the
following:</p>

It's cheap</li>
You learn linux</li>
You become OS agnostic (I said no fanboyism)</li>
You can work from almost anywhere</li>
</ul>
What you need</h2>
Laptop</h3>
Because all of your work is now done on a remote machine you don't really have
to care about what computer you're using. It can be a super expensive (notice I
didn't say overpriced) Apple laptop or any of its really good windows competitors (the DELL xps 13/15 comes to mind) or even a super cheap, 35$ raspberry Pi. Some people even use chromebooks ! They stick ubuntu on them and use them as their primary machines (SSH is a bit tricky to setup on chrome OS) but hey! Official Linux support</a> is coming for you ChromeOS folks.</p>
Dev Tools</h2>
Git</h3>
This one's obvious. While git is a life saver and a great tool for collaboration in large teams (distributed or not), you can also use it as a backup system for your code when you're a solo developer.</p>
Vim</h3>
I've used sublime text but felt bad for continuously extending the trial because I
couldn't pay for it. Then I switched back to VSCode and got frustrated by how slow it
runs after using Sublime. Then I tried vim, and I never
looked back. It's the lightest, fastest and overall best text editor out there by a huge
margin (in my opinion).</p>
Vim is highly customizable and lets you save your settings inside a
.vimrc</code> file, which makes it version control friendly. It also makes your vim
environment 100% portable as it comes standard in most linux server distros.</p>
Tmux</h3>
The best companion to vim. Tmux is a terminal multiplexer. With Tmux you can
essentially access multiple terminal sessions inside a single window. This gives
you the ability to work on multiple projects at a time. You can even save
sessions, attach and reattach to them. It's painless.
Just like vim, it's extremely customizable. All of your settings can be stored
inside of a .tmux.conf</code> file.</p>
A VPS</h3>
To start developing on a remote server, you'll need ... well, a remote
server. If you don't know what a VPS is, it stands for Virtual Private Server</strong>.
It's essentially a virtual machine that you pay for monthly, which runs a server
distribution of linux (Ubuntu, CentOS ...), or Unix (FreeBSD). You can connect to it via SSH (it has a public ip address) and start playing around. You have complete control over the server, you can configure it however way you want.</p>
There's a large range of VPS providers on the market, the most notable
ones are Linode</a>, Digital Ocean</a>, and Amazon EC2</a>.</p>
Dotfiles</h3>
As I previously mentioned above, you can save your settings for vim and tmux in
what are called dotfiles</strong> (files that start with a period). However dotfiles
are not exclusive to vim and tmux, you can for example save your shell configuration in a .zshrc</strong> or .bashrc</strong> config file, or your git settings in a .gitconfig</strong> file.</p>
Doing this is extremely powerful because you can store these configuration files
on github and always pull the latest version when you launch a new development server.</p>
You can check out my own dotfiles</a> on github to help you get started.</p>
(Optional) Ansible</h3>
Manually setting up servers is fun for the first couple of times, then it just
becomes repetitive. And what do you do with repetitive tasks ? You automate
them.</p>
Ansible is a provisioning tool written in python that will help you do just that.
You give it the ip address (or addresses) of the server you want to configure and it will execute all the tasks you tell it to.</p>
With Ansible you can:</p>

run shell commands</li>
install packages</li>
create directories and files</li>
add users and groups</li>
clone git repos</li>
use templates and pass variables to them</li>
and much more</li>
</ul>
If you don't like Ansible, there are other server provisioning tools like
puppet</a>, chef</a> and salt</a> that will work just as good.</p>

Caveats</h2>

it takes time to learn linux and networking (SSH etc)</li>
vim has a learning curve to it</li>
you become reliant on an internet connection</li>
</ul>


Deploy a Django App with Git
Unknown — Mon, 07 May 2018 00:00:00 +0000
I'm going to make a bold statement: Django replaced Ruby on Rails in the hearts
of many developers. With this increase in popularity, we've seen tons of
articles, videos, and websites dedicated to setting up Django and creating apps
with the framework.</p>
Unfortunately, when it comes to deployment, many of these resources only mention
heroku or pythonanywhere. While these are excellent solutions for quickly
shipping your MVP or prototype, they lack a bit of flexibility if you
want to create your custom deployment pipeline.</p>
tl;dr: If you manage your own server infrastructure, we're going to set up a
Git deployment workflow with Django.</strong></p>
What you'll need</h2>

Working knowledge of ssh</li>
Working knowledge of git</li>
Working knowledge of the bash shell</li>
Basic linux command line skills (sorry hipsters)</li>
(Very) Basic knowledge of vi/vim</li>
Patience</li>
</ul>
The typical workflow usually looks like this:</p>

You have your development environment, either on your local machine or a remote server</li>
A git server (on GitHub, BitBucket, GitLab ...) that you and your team push your work to</li>
A production server (aka your live app).</li>
</ul>
Usually when you commit and push work you do something like:</p>
git</span> push</span> origin</span> <</span>branch_nam</span>e</span>></span></span></code></pre>
origin</code> being the name of the remote server your code is being pushed to.
What took me a while to realise is that you can have many remotes for your repo
that point to different servers.</p>
The idea here is to add a new remote to our repo. It will point to our
production server such that when we run git push live master</code>, our code will be
copied over to there.</p>
On the production server</h2>
To achieve this, we have some setup work to do on our live server. So go ahead
and connect to it via ssh.</p>
ssh</span> user@host_or_ip_address</span></span>
</span>
#</span> If your server's ssh service listens on a port other than 22, you'll need</span></span>
#</span> to add the -p switch</span></span>
</span>
ssh</span> -</span>p</span> PORT</span> user@host_or_ip_address</span></span></code></pre>
Once we're in, we need to create a new directory for our application. This is
where our deployed code will be copied to.</p>
mkdir</span> -</span>p</span> /home/user/sites/myawesomedjangoproject</span></span>
</span>
#</span> Some people prefer to use /var/www/<project_name>, it's really up to you. Just</span></span>
#</span> make sure remember the path to your project</span></span></code></pre>
Now head over to /var</code> and create another directory called repos</code></p>
cd</span> /var</span></span>
sudo</span> mkdir</span> repos</span></span>
#</span> Depending on your setup, you might need sudo priveleges</span></span></code></pre>
Inside that directory, we need to create a folder named after our project
(or domain name) and append it with .git</code> (not necessary but it's good practice)</p>
sudo</span> mkdir</span> myawsomedjangoproject.com.git</span></span></code></pre>
Inside this folder we'll create what is called a bare</code> repository. To do this
just run:</p>
git</span> init</span> -</span>-bare</span></span></code></pre>
If you run ls</code> inside that folder you'll see a bunch of files and directories
(the same ones found inside the .git</code> folder in normal repos). One of these
directories is called hooks</code>.</p>
Inside that folder, we'll need create a file called post-receive.</p>
#</span> Assuming you are inside /var/repos/yourproject.git</span></span>
sudo</span> touch</span> hooks/post-receive</span></span></code></pre>
Now open it up with vi/vim</p>
sudo</span> vim</span> hooks/post-receive</span></span></code></pre>
Hit i</code> to switch to insert mode, and add the following to the file:</p>
#!</span>/bin/bash</span></span>
DEPLOYDIR</span>=</span>/</span>h</span>o</span>m</span>e</span>/</span>u</span>s</span>e</span>r</span>n</span>a</span>m</span>e</span>/</span>s</span>i</span>t</span>e</span>s</span>/</span>m</span>y</span>a</span>w</span>e</span>s</span>o</span>m</span>e</span>d</span>j</span>a</span>n</span>g</span>o</span>p</span>r</span>o</span>j</span>e</span>c</span>t</span> #</span> or whatever path you chose</span></span>
GIT_WORK_TREE</span>=</span>"</span>$</span>DEPLOYDIR</span>"</span> git</span> checkout</span> -</span>f</span></span></code></pre>
Please note that the first shebang line is important, it instructs git to use
bash instead of the default shell. Otherwise it won't activate our (soon to be
created) virtual environment</strong></p>
exit vim by hitting :wq</code> (which in vim lingo means write and quit)</p>
What we've done here is set two variables. DEPLOYDIR</code> is an alias for our
project path on the server, and GIT_WORK_TREE</code> which is a special variable that
tells git to copy the code it receives inside of our DEPLOYDIR</code>. This ensures
that we're always running the latest version of our code.</p>
As you've probably noticed, this post-receive file looks very much like a shell
script. That's because it is (as explained above). It's executed every time you
push code to the repo.</p>
The last thing we need to is make the script executable, so as soon as you're
back in the shell run:</p>
sudo</span> chmod</span> +x</span> hooks/post-receive</span></span></code></pre>
You can now exit the server and go back to your local machine.</p>
On our local dev environment</h2>
Now that we've created our remote repository, we need to add it to our
project (I like to call mine live</code>).</p>
It takes one simple command:</p>
git</span> remote</span> add</span> live</span> root@ip_address:/var/repos/myawesomedjangoproject.git</span></span>
</span>
#</span> And if your server's ssh service listens on a different port :</span></span>
</span>
git</span> remote</span> add</span> live</span> ssh://root@ip_address:PORT/var/repos/myawesomedjangoproject.git</span></span>
</span></code></pre>
To make sure it was added, you can print the list of available remotes by running:</p>
git</span> remote</span> -</span>v</span> #</span> v for verbose</span></span>
</span></code></pre>
and that's it ! You can now make changes locally, commit and deploy
them live (or staging if it's a staging server) and see your changes instantly.</p>
You can obviously still push to github/lab or bitbucket with
git push origin <branch></code>
like you normally would.</p>
Bonus</h2>
As I mentioned in the first part, the post-receive hook is a shell script. Which
means you can use it to perform all kinds of tasks against your code, like
running front-end builds, installing dependencies, etc ...</p>
Here's an example for a basic Django App:</p>
#!</span>/bin/bash</span></span>
</span>
DEPLOYDIR</span>=</span>/</span>h</span>o</span>m</span>e</span>/</span>u</span>s</span>e</span>r</span>n</span>a</span>m</span>e</span>/</span>s</span>i</span>t</span>e</span>/</span>m</span>y</span>a</span>w</span>e</span>s</span>o</span>m</span>e</span>d</span>j</span>a</span>n</span>g</span>o</span>p</span>r</span>o</span>j</span>e</span>c</span>t</span></span>
</span>
echo</span> "</span>[log] - Starting code update </span>"</span></span>
GIT_WORK_TREE</span>=</span>"</span>$</span>DEPLOYDIR</span>"</span> git</span> checkout</span> -</span>f</span></span>
echo</span> "</span>[log] - Finished code update </span>"</span></span>
</span>
if</span> [[</span> -d</span> "</span>$</span>DEPLOYDIR</span>/ENV_projectname</span>"</span>]]</span>;</span> then</span></span>
    echo</span> "</span>[log] - Cleaning virtualenv</span>"</span></span>
    cd</span> "</span>$</span>DEPLOYDIR</span>"</span>;</span> rm</span> -</span>rf</span> ENV_projectname</span> cd</span> -</span></span>
    echo</span> "</span>[log] - Finished creating virtualenv</span>"</span></span>
fi</span></span>
</span>
echo</span> "</span>[log] - Creating virtualenv</span>"</span></span>
cd</span> "</span>$</span>DEPLOYDIR</span>"</span>;</span> virtualenv</span> -</span>p</span> python3</span> ENV_projectname</span>;</span> cd</span> -</span></span>
echo</span> "</span>[log] - Finished creating virtualenv</span>"</span></span>
</span>
echo</span> "</span>[log] - Activating virtualEnv</span>"</span></span>
cd</span> "</span>$</span>DEPLOYDIR</span>"</span>;</span> source</span> ENV_projectname/bin/activate</span>;</span> cd</span> -</span></span>
echo</span> "</span>[log] - Finished activating virtualenv</span>"</span></span>
</span>
echo</span> "</span>[log] - Pulling down pip dependencies</span>"</span></span>
cd</span> "</span>$</span>DEPLOYDIR</span>"</span>;</span> pip</span> install</span> -</span>r</span> requirements.txt</span>;</span> cd</span> -</span></span>
echo</span> "</span>[log] - Finished pulling down pip dependencies</span>"</span></span>
</span>
echo</span> "</span>[log] - Staring DB migration</span>"</span></span>
cd</span> "</span>$</span>DEPLOYDIR</span>"</span>;</span> python</span> manage.py</span> makemigrations</span>;</span> python</span> manage.py</span> migrate</span>;</span> cd</span> -</span></span>
echo</span> "</span>[log] - Finished DB migration </span>"</span></span>
</span>
echo</span> "</span>[log] - Pulling Node Dependencies</span>"</span></span>
cd</span> "</span>$</span>DEPLOYDIR</span>"</span>;</span> sudo</span> npm</span> install</span>;</span> cd</span> -</span></span>
echo</span> "</span>[log] - Finished Pulling Node Dependencies</span>"</span></span>
</span>
echo</span> "</span>[log] - Building the Front end</span>"</span></span>
cd</span> "</span>$</span>DEPLOYDIR</span>"</span>;</span> sudo</span> gulp</span> build</span>;</span> cd</span> -</span></span>
echo</span> "</span>[log] - Finished building the Front end</span>"</span></span>
</span>
echo</span> "</span>[log] - Collecting static assets</span>"</span></span>
cd</span> "</span>$</span>DEPLOYDIR</span>"</span>;</span> python</span> manage.py</span> collectstatic</span> -</span>-clear</span> -</span>-no-input</span>;</span> cd</span> -</span></span>
echo</span> "</span>[log] - Finished collecting static assets</span>"</span></span>
</span>
echo</span> "</span>[log] - Restarting App</span>"</span></span>
sudo</span> service</span> myawesomedjangoapp</span> restart</span>;</span></span>
echo</span> "</span>[log] - Finished collecting static assets</span>"</span></span>
</span></code></pre>
I run my Django Apps as systemd services, if you don't you can just call python
manage.py runserver. If you want to know how to setup Django the way I do just
follow this very comprehensive tutorial over on Digital Ocean</a></em></p>
Conclusion</h2>
I am fully aware that there are more sophisticated methods of deployment through
Docker, Travis (For continious integration) etc. But if you have a small app that
you want to ship and you already have an infrastructure, I've found this method
to be more than suitable.</p>
Please report any missing info, mistake, error, typo. I'm on  twitter </a> if you
wanna chat.</p>


Useful File Operations in Python
Unknown — Wed, 28 Mar 2018 00:00:00 +0000
In this short post (spoiler: it's actually quite lengthy), I'm going to walk
through a list of useful methods in the os</code> module (which is part of the
Python standard library) for handling files and directories.</p>
1. Create a directory</h2>
This one is pretty straightforward. If you're comfortable with the linux shell,
you know that mkdir</code> is the command to use to create directories.
Unsurprisingly, Python uses the same naming convention.</p>
Example:</p>
import</span> os</span></span>
</span>
os</span>.</span>mkdir</span>(</span>"</span>my_awesome_directory</span>"</span>)</span></span></code></pre>
The method takes a string as its argument and will create the directory under the
file's parent folder. (For instance, if the path to the file calling os.mkdir()
is /home/username/Documents/app.py, the "my_awesome_directory" will be created
under /home/username/Documents)</p>
2. Get a file's parent directory.</h2>
This is useful if you want the path to a file or folder that you know lives
under the working file's parent directory. If that sounded confusing, here's an
example.</p>
In the Linux shell (bash or otherwise), you can issue the command pwd</code>
(which I believe stands for "print working directory") to quickly print your
current location within a given session.</p>
In Python you would achieve this like so:</p>
import</span> os</span></span>
</span>
CURRENT_DIR</span> =</span> os</span>.</span>getcwd</span>(</span>)</span></span></code></pre>
Notice I used all caps for the variable name. This is because it's usually a
constant. This variable isn't meant to be changed.
You can now use this variable to locate any file or folder within that
directory. Keep reading and I'll show you how.</p>
3. Concatenate paths</h2>
This is something you'll find yourself doing a lot. Especially on large projects
that require configuration files and other such things. If you have experience
working with Django for example, they have a settings.py</code> file littered with
calls to the os module. There are many benefits to this approach. Perhaps the
most obvious being is that if you ever decide to move your project to another
location, you don't want to keep modifying the path every time. Remember,
programming is all about being lazy.</p>
So this is how you would do it:</p>
import</span> os</span></span>
</span>
#</span> Assuming this file is located at /home/username/myproject/app.py</span></span>
#</span> and that you want to operate on a a file called config.cfg within the same</span></span>
#</span> directory :</span></span>
</span>
CURRENT_DIR</span> =</span> os</span>.</span>getcwd</span>(</span>)</span> #</span> evaluates to /home/username/myproject</span></span>
MY_TEXT_FILE</span> =</span> os</span>.</span>path</span>.</span>join</span>(</span>CURRENT_DIR</span>,</span> "</span>config.cfg</span>"</span>)</span></span>
</span>
print</span>(</span>MY_TEXT_FILE</span>)</span> #</span> /home/username/myproject/config.cfg</span></span></code></pre>
An important note: os.path.join() merely concatenates the two paths together.
It doesn't check whether the path is valid. So be careful when using this
method. Also notice how the method call is to os.path.join() and not os.join()</em></p>
4. Check that a path exists</h2>
The other day I was working on a small web scraper for a side project of mine.
After the it was done fetching data, my script would save the results into a
pickle file (don't worry if you don't know what it is) that would be read by my
program, saving me the trouble of sitting there waiting to fetch the same info
over and over again each time I run the script.</p>
The solution was to tell my script to check whether a specific file (let's call
it results.pkl) exists at a given path. If it does, the program continues and
if not, the program executes the crawler function.</p>
This is clever because now I only have to fetch the data and if the file gets
deleted I know I can rely on the program to go and crawl the sites as expected.</p>
And now for the example:</p>
import</span> os</span></span>
</span>
CURRENT_DIR</span> =</span> os</span>.</span>getcwd</span>(</span>)</span></span>
RESULTS_FILE</span> =</span> os</span>.</span>path</span>.</span>join</span>(</span>CURRENT_DIR</span>,</span> "</span>results.pkl</span>"</span>)</span></span>
</span>
def</span> crawl_data</span>(</span>)</span>:</span></span>
    #</span> scrapes a bunch of websites and saves the result in a file called</span></span>
    #</span> results.pkl under the current directory</span></span>
    pass</span></span>
</span>
if</span> not</span> os</span>.</span>path</span>.</span>exists</span>(</span>RESULTS_FILE</span>)</span>:</span></span>
    crawl_data</span>(</span>)</span></span>
</span>
else</span>:</span></span>
    #</span># the file exists so we can open it and work with its content</span></span></code></pre>
The same thing can be done to check that the path exists AND that it's a
directory:</p>
CURRENT_DIR</span> =</span> os</span>.</span>getcwd</span>(</span>)</span></span>
MY_DIRECTORY</span> =</span> os</span>.</span>path</span>.</span>join</span>(</span>CURRENT_DIR</span>,</span> "</span>my_directory</span>"</span>)</span></span>
</span>
if</span> os</span>.</span>path</span>.</span>exists</span>(</span>MY_DIRECTORY</span>)</span> and</span> os</span>.</span>path</span>.</span>isdir</span>(</span>MY_DIRECTORY</span>)</span>:</span></span>
    #</span># do something with the files inside the folder</span></span>
else</span>:</span></span>
    os</span>.</span>mkdir</span>(</span>MY_DIRECTORY</span>)</span></span></code></pre>5. List files within a given directory</h2>
Very useful when you want to read several files that are under the same
directory.
It can be done in two ways: conventional and pythonic. I'll show you both.</p>
for</span> file</span> in</span> os</span>.</span>listdir</span>(</span>"</span>/path/to/dir</span>"</span>)</span>:</span></span>
    #</span> do something with the filename (open it, copy it,  move it, rename it...)</span></span>
</span></code></pre>
or</p>
filenames</span> =</span> [</span>file</span> for</span> file</span> in</span> os</span>.</span>listdir</span>(</span>"</span>/path/to/dir</span>"</span>)</span>]</span></span></code></pre>
Guess which way is more pythonic!</p>
Conclusion</h2>
These have been the most useful file/directory functions for me in Python. I
really love the fact that the method names sound natural and are (for the most
part) similar to linux commands. What are your favourite file operation methods? Any tip or trick you want to share with me? Something I've missed? Ping on
twitter! I'm @aaqaishtyaq</a>.</p>
If you found this article useful, please share it with your nerd friends/coworkers and spread the word!</p>


Mind-Blowing Python Tips and Tricks
Unknown — Thu, 15 Feb 2018 00:00:00 +0000
A short list of Python tricks that make day-to-day code a little nicer.
Nothing too magical, just handy stuff.</p>
0 - Loop over a range of numbers</h2>
Use range</code> instead of xrange</code>.
In Python 3, the former creates an iterator-like object that produces values
one at a time, which makes it much more efficient.</p>
</span>
nums</span> =</span> [</span>0</span>,</span>2</span>,</span> 34</span>,</span> 55</span>,</span> 32</span>]</span></span>
for</span> i</span> in</span> range</span>(</span>nums</span>)</span>:</span></span>
    print</span> i</span></span>
</span></code></pre>1 - Looping backwards</h2>
Just use reversed</code>.</p>
</span>
names</span> =</span> [</span>"</span>Case</span>"</span>,</span> "</span>Molly</span>"</span>,</span> "</span>Armitage</span>"</span>,</span> "</span>Maelcum</span>"</span>]</span></span>
for</span> name</span> in</span> reversed</span>(</span>names</span>)</span>:</span></span>
    print</span> name</span></span>
</span></code></pre>2 - Looping over a list and its indices</h2>
To keep track of the index of each item in a collection, enumerate is your buddy.</p>
names</span> =</span> [</span>"</span>Case</span>"</span>,</span> "</span>Molly</span>"</span>,</span> "</span>Armitage</span>"</span>,</span> "</span>Maelcum</span>"</span>]</span></span>
for</span> index</span>,</span> name</span> in</span> enumerate</span>(</span>names</span>)</span>:</span></span>
    print</span> index</span>,</span> name</span></span>
</span></code></pre>3 - Looping over two lists simultaneously</h2>
Yeah, you could use zip</code>, but izip</code> is faster, so use that instead.</p>
</span>
from</span> itertools</span> import</span> izip</span></span>
</span>
names</span> =</span> [</span>"</span>Case</span>"</span>,</span> "</span>Molly</span>"</span>,</span> "</span>Armitage</span>"</span>,</span> "</span>Maelcum</span>"</span>]</span></span>
ages</span> =</span> [</span>23</span>,</span> 27</span>,</span> 41</span>,</span> 24</span>]</span></span>
for</span> name</span>,</span> age</span> in</span> izip</span>(</span>names</span>,</span> ages</span>)</span>:</span></span>
    print</span> name</span>,</span> age</span></span>
</span></code></pre>4 - Looping over a sorted list</h2>
You can sort out the list first and then loop through it, or you could use
sorted.</p>
</span>
names</span> =</span> [</span>"</span>Case</span>"</span>,</span> "</span>Molly</span>"</span>,</span> "</span>Armitage</span>"</span>,</span> "</span>Maelcum</span>"</span>]</span></span>
for</span> name</span> in</span> sorted</span>(</span>names</span>)</span>:</span></span>
    print</span> name</span></span>
</span></code></pre>
And BAM, you're ... sorted.</p>
5 - Call a function until a sentinel value is returned</h2>
To do that, use iter().</p>
Bad example:</p>
Loop over a file containing a list of names
until the loop returns an empty string,
in which case we break out of it.</p>
</span>
names</span> =</span> [</span>]</span></span>
while</span> True</span>:</span></span>
    name</span> =</span> file</span>.</span>read</span>(</span>32</span>)</span></span>
    if</span> name</span> =</span> "</span>"</span>:</span></span>
        break</span></span>
    names</span>.</span>append</span>(</span>name</span>)</span></span></code></pre>
Beautiful example:</p>
In this case, we call a function (f.read) until it returns the sentinel value
passed as a second argument to iter.
That way we avoid having to make the unnecessary if check.</p>
for</span> name</span> in</span> iter</span>(</span> partial</span>(</span>f</span>.</span>read</span>(</span>32</span>)</span>)</span>,</span> "</span>"</span>)</span>:</span></span>
    print</span> name</span></span></code></pre>6 - Looping over a dictionary</h2>
The normal way to do it:</p>
molly</span> =</span> {</span> "</span>name</span>"</span>:</span> "</span>Molly Millions</span>"</span>,</span> "</span>Age</span>"</span>:</span> 27</span>,</span> "</span>Occupation</span>"</span>:</span> "</span>Professional Killer</span>"</span>}</span></span>
</span>
for</span> key</span> in</span> molly</span>:</span></span>
    print</span> key</span></span></code></pre>
If you wish to mutate the data, prefer dict.keys()</code>.</p>
</span>
molly</span> =</span> {</span> "</span>name</span>"</span>:</span> "</span>Molly Millions</span>"</span>,</span> "</span>Age</span>"</span>:</span> 27</span>,</span> "</span>Occupation</span>"</span>:</span> "</span>Professional Killer</span>"</span>}</span></span>
</span>
for</span> key</span> in</span> molly</span>.</span>keys</span>(</span>)</span>:</span></span>
    #</span> do the mutation</span></span>
</span></code></pre>7 - Looping over a dict keys AND values</h2>
Don't do this:</p>
</span>
molly</span> =</span> {</span> "</span>name</span>"</span>:</span> "</span>Molly Millions</span>"</span>,</span> "</span>Age</span>"</span>:</span> 27</span>,</span> "</span>Occupation</span>"</span>:</span> "</span>Professional Killer</span>"</span>}</span></span>
</span>
for</span> key</span> in</span> molly</span>:</span></span>
    print</span> molly</span>[</span>key</span>]</span></span>
</span></code></pre>
It's slow because we have to rehash the dictionary and do a lookup everytime.</p>
Instead choose iteritems()</code>:</p>
</span>
molly</span> =</span> {</span> "</span>name</span>"</span>:</span> "</span>Molly Millions</span>"</span>,</span> "</span>Age</span>"</span>:</span> 27</span>,</span> "</span>Occupation</span>"</span>:</span> "</span>Professional Killer</span>"</span>}</span></span>
</span>
for</span> key</span>,</span> value</span> in</span> molly</span>.</span>iteritems</span>(</span>)</span>:</span></span>
    print</span> key</span>,</span> value</span></span>
</span></code></pre>8 - Create a dict out of two lists</h2>
Just instantiate a new dict with two zipped lists. Real magic.</p>
</span>
from</span> itertools</span> import</span> izip</span></span>
</span>
names</span> =</span> [</span>"</span>Case</span>"</span>,</span> "</span>Molly</span>"</span>,</span> "</span>Armitage</span>"</span>,</span> "</span>Maelcum</span>"</span>]</span></span>
ages</span> =</span> [</span>23</span>,</span> 27</span>,</span> 41</span>,</span> 24</span>]</span></span>
</span>
characters</span> =</span> dict</span>(</span>izip</span>(</span>names</span>,</span> ages</span>)</span>)</span></span>
</span></code></pre>9 - Use named tuples for returning multiple values</h2>
Like in the case of an API response in Flask.</p>
</span>
from</span> collections</span> import</span> namedtuple</span></span>
</span>
Response</span> =</span> namedtuple</span>(</span>'</span>APIResponse</span>'</span>,</span> [</span>'</span>status_code</span>'</span>,</span> '</span>body</span>'</span>,</span> '</span>headers</span>'</span>]</span>)</span></span>
</span>
@</span>app</span>.</span>route</span>(</span>'</span>/users/1</span>'</span>)</span>:</span></span>
</span>
    try</span>:</span></span>
        user</span> =</span> db</span>.</span>getuserbyid</span>(</span>1</span>)</span></span>
    except</span>:</span></span>
        return</span> Response</span>(</span>404</span>,</span> user</span>.</span>notfound</span>(</span>)</span>,</span> {</span>'</span>content-type</span>'</span>:</span> '</span>application/json</span>'</span>}</span></span>
    else</span>:</span></span>
        return</span> Response</span>(</span>200</span>,</span> user</span>.</span>json</span>(</span>)</span>,</span> {</span>'</span>content-type</span>'</span>:</span> '</span>application/json</span>'</span>}</span></span>
</span></code></pre>Other</h2>

Always clarify function calls by using keyword arguments</li>
</ul>
If you learned something from this article, share it with your co-workers and
fellow hackers. If you notice any typo, error etc let me know on
twitter</a>.</p>


Mind-Blowing Git Tips for Beginners
Unknown — Mon, 05 Feb 2018 00:00:00 +0000
As developers, we all (hopefully) use git. It's not extremely hard or
time-consuming to get started with it, and you'll surely thank your future self
for taking the time to learn it. Ok, that was the cringy intro.</p>
Now let's get down to business: what I really want to share in this post is a
list of tricks I've learned during the past 2 1/2 years of using git. Some of
it might seem trivial to seasoned developers, but if you're just getting
started, stick with me because a couple of these might just blow your mind.</p>
The difference between git fetch and pull</h2>
git fetch</code> only updates the tracking remote branches. If you actually want to
update the local repo, you need to merge the local branch with the remote
tracking branch using git merge</code>.</p>
remote branches are prefixed with the name of the remote and a slash:
origin/branchname</strong></p>
git pull</code>, on the other hand, will execute both commands for you so when you issue
a git pull</code> in a branch it will fetch that branch from the remote repo and</strong>
merge it with yours.</p>
To list all remote tracking branches, git branch -r</code> is your friend.</p>
Merge conflicts</h2>
At some point you'll inevitably have to deal with merge conflicts. Essentially what this means is that git noticed a file was modified on both branches and it does not know which version is correct. It leaves it up to you to decide which one you want to push.</p>
If you open the file you will see a HEAD</code> part prefixed with <<<<</code> signs and a
second part containing the code on the branch you're trying to merge. Both
sections are separated with an equal sign.</p>
The top HEAD section contains the version of the file as it is on your current
branch. The other shows you what the code looks like on the branch you're trying to merge from.</p>
To resolve the conflict just delete the part you don't want (Including all the equal signs etc), save the file and commit it again.</p>
Note that as a safety measure it's always good practice to do a git pull</code> to
see if you don't have any remaining conflicts. If not you can just push your code and you're all set!</p>
As for the tips:</p>
git merge --abort</code> will clean up the current working directory and go back to
the last version before the merge (it pretty much nullifies the merge).</p>
git merge --squash</code> is an interesting one as it will bring the changes you made in the feature branch and create a new commit on your current branch without
mixing the two histories.</p>
Dealing with files</h2>
Now this one's tricky !</p>
To remove a file from both</strong> the working directory and version control, you
can use git rm <filename></code> but if you already deleted a file in the cli like I (and almost everyone) usually do then just run git rm <path to the file></code> and it will stage it as deleted so you can commit.</p>
But what if you ramrafed (rm -rf</code>) a bunch of files in your working directory?
You don't really want to manually stage every file for deletion, do you? (DO YOU ?) So what do you do in that case? Well, you just run git add -u</code> (u for updating the working tree) and all of your deleted files will be staged so you can commit and push.</p>
"What if a man accidentally tracked a file and man doesn't want it deleted
from the index ?" I hear you ask with an exquisite South London accent.</p>
Well, my G, just run:</p>
git</span> rm</span> -</span>-cached</span> <</span>name</span> of</span> the</span> fil</span>e</span>></span></span></code></pre>
and you're done. Efficience ting.</p>
Something else that happens quite often is moving/renaming files (which is
essentially the same thing on Linux systems, as the path/address of the file
changed).</p>
Here's an example: imagine you have a file called index.html that you
want to rename to home.html. you can use git mv</code> which takes a source and a
destination:</p>
git</span> mv</span> index.html</span> home.html</span></span></code></pre>
if you want to move and</strong> rename just run</p>
git</span> mv</span> index.html</span> path/to/home.html</span></span></code></pre>
However if you moved / renamed the file manually on the command line, running
git status</code> will tell you that index.html was deleted and home.html is untracked.</p>
To fix this you will have to run two commands</p>
git</span> rm</span> index.html</span> #</span> aka the file you moved/renamed</span></span></code></pre>git</span> add</span> home.html</span> #</span> aka the file with the new name x path</span></span></code></pre>
Running git status again will mark the file as renamed.</p>
Note that this also works if you don't rename the file. For example, if you just
want to move index.html to src/index.html. The same command will apply
(git mv index.html src/index.html</code>).</p>
The other way of achieving this is:</p>
git</span> add</span> -</span>A</span></span></code></pre>
which will pick up the changes and automatically stage them for commit.</p>
So there you go, I hope you learned something useful in this article. If you know
other mind-blowing tips and tricks for git, @ me on twitter</a>.</p>


Don't Be Scared, Switch to Vim
Unknown — Sat, 03 Feb 2018 00:00:00 +0000
I'm currently sitting at the most boring meetup I've probably ever attended in
Delhi. It's about chatbots. I don't care about chatbots, I care about free
stickers and pizza. So I'm using this opportunity to open up about a subject
that's dear to my heart: vim.</p>
I used to believe vim was exclusive to this superior race of developers who
gulp coffee like it's water and seem to only read HN and nothing else. (Hi, if
you're coming from HN). Architecture and software design come naturally to them,
they never run into bugs and they can recognize the most obscure of algorithms
at a glance (Shout out to Shashank, one of my mentors).</p>
Shanky is a good, productive developer. Shanky uses vim. I want to be like Shanky. I want
to use vim.</p>
There are a million reasons why you should jump ship and join the cult. In the
next paragraphs, I will detail some of these reasons.</p>
It's not (that) hard</h2>
There's a learning curve to vim, but it's worth the
trouble. And if you're on Linux or MacOS, there's a built-in tool called
vimtutor</code> (just fire it up from a terminal, I am not sure about Windows though) and
a wide variety of online tools to learn vim. Namely openvim</a>,
vim adventures</a>, and vim genius</a>.</p>
Personally, The way I learned was by using it on small, fun side projects of
mine during the weekends, mostly to become familiar with the new mental model.
And just like everything in life, shit takes time, and practice makes perfect.
So keep at it and you'll eventually come to your "aha" moment.
As you get more and more comfortable using vim, it will become harder and harder
to go back to a regular editor / IDE.</p>
It's Fast and Customisable</h2>
Because it runs on the terminal, you'll never have to wait 20 seconds to get
on with your work. (Atom anyone ?)</p>
And if you like pretty things, there's a large selection of colorschemes</a>
for you to choose from. On top of that, there's a plugin for just about anything
you might need. And if there isn't, you can program your own.</p>
Ubiquity</h2>
Not really, but I wanted to place a complicated word to sound smart.
Seriously though, it's everywhere. On Mac OS, Windows and of course Linux/Unix. If
you work on remote servers you can quickly edit files on the fly without having
to use nano. (Don't use nano)</p>
Say for example a coworker/friend is running into a bug, you come to help and
they're using an IDE you're not familiar with, well you can just access the files
from their terminal and start debugging right away.</p>
Or if you're like me, and you spill water on your Macbook keyboard and it becomes
toast, you can spin up a VPS on Digital Ocean or AWS, and pick up where you
left off (almost) right away.</p>
Bonus: Some of my favorite plugins</h2>
My color scheme of choice (at the time of writing) is afterglow</a>.</p>
And here's a list of my favorite plugins:</p>

 Nerdtree </a> (A tree explorer much like the sidebar in traditional IDEs)</li>
 Airline </a> (A sleek, customizable status bar)</li>
 Surround </a> (Helpful tool that helps with "surrounding" words with brackets etc)</li>
 CtrlP </a> (A fuzzy finder for vim)</li>
 UtilSnips </a> (Snippet utility for many languages)</li>
 Vim  Markdown</a> (Markdown syntax highlighting)</li>
 Goyo </a> (Allows for distraction-free editing)</li>
</ul>
I'll end this article with a quote from a Chamillionaire:</p>

They see you vimmin', they hatin'. Patroling they tryna catch me coding dirty</p>
</blockquote>


Algorithms in Python: Understanding Bubble Sort
Unknown — Mon, 08 Jan 2018 00:00:00 +0000
Some theory</h2>
Bubble sort is one of the most commonly known sorting algorithms. The idea is
simple: scan a list of items (say integers) sequentially, from left to right,
and compare consecutive pairs of elements starting at index 0.</p>
Example:</p>
</span>
my_numbers</span> =</span> [</span>92</span>,</span>11</span>,</span>45</span>,</span>2234</span>,</span>0</span>,</span>7</span>,</span>65</span>]</span></span>
#</span> 92 is index 0 and the consecutive pairs are</span></span>
#</span> (92,11), (11,45), (45,2234) and so on ...</span></span></code></pre>
First we compare elements (list[0],list[1]) then (list[1],list[2]) then
(list[2],list[3]) and so on until we reach the end of the list.</p>
When comparing, we check if element i is greater than element i + 1. If it is,
we swap the two elements and move on to the next pair. If not, it means the
pair is already sorted, so we still move on to the next pair.</p>
Example:</p>
my_numbers</span> =</span> [</span>92</span>,</span>11</span>,</span>45</span>,</span>2234</span>,</span>0</span>,</span>7</span>,</span>65</span>]</span></span>
</span>
#</span> Let's compare my_numbers[0] and my_numbers[1]</span></span>
if</span> my_numbers</span>[</span>0</span>]</span> ></span> my_numbers</span>[</span>1</span>]</span>:</span></span>
    swap</span>(</span>my_numbers</span>[</span>0</span>]</span>,</span> my_numbers</span>[</span>1</span>]</span>)</span></span>
</span>
print</span>(</span>my_numbers</span>)</span> #</span> [11, 92, 45, 2234, 0, 7, 65]</span></span></code></pre>
This process has to be repeated for however many items are on the list. So if
the list holds 9 items, it means we need to loop through it 9 times at most.
But what if our original list is partially sorted? We might not need 9 passes
through the list.</p>
One way for us to know that the list is fully sorted is if we have made no
swaps during our pass. For that, we need a variable to keep track of how many
swaps were made during a pass.</p>
Example:</p>
my_numbers</span> =</span> [</span>92</span>,</span>11</span>,</span>45</span>,</span>2234</span>,</span>0</span>,</span>7</span>,</span>65</span>]</span></span>
</span>
#</span> Elements (0,1) are compared and swapped. List is now 11,92,45,2234,0,7,65</span></span>
#</span> Elements (1,2) are compared and swapped. List now 11,45,92,2234,0,7,65</span></span>
#</span> Elements (2,3) are compared and not swapped. List remains the same.</span></span>
#</span> Elements (3,4) are compared and swapped. List is now 11,45,92,0,2234,0,7,65</span></span>
#</span> Elements (4,5) are compared and swapped. List is now 11,45,92,0,7,2234,65</span></span>
#</span> Elements (5,6) are compared and swapped. List is now 11,45,92,0,7,65,2234</span></span>
</span>
#</span> This represents one unique pass through the list.</span></span></code></pre>
Notice how after each pass the highest value number is pushed at len(list) - 1.</p>
Some code</h2>
Let's look at how to implement Bubble Sort in Python:</p>
def</span> bubble_sort</span>(</span>some_list</span>)</span>:</span></span>
</span>
    is_sorted</span> =</span> False</span></span>
</span>
    while</span> not</span> is_sorted</span>:</span></span>
</span>
        is_sorted</span> =</span> True</span></span>
</span>
        for</span> i</span> in</span> range</span>(</span>0</span>,</span> len</span>(</span>some_list</span>)</span> -</span> 1</span>)</span>:</span></span>
</span>
            if</span> some_list</span>[</span>i</span>]</span> ></span> some_list</span>[</span>i</span> +</span> 1</span>]</span>:</span></span>
</span>
                some_list</span>[</span>i</span>]</span>,</span> some_list</span>[</span>i</span>+</span>1</span>]</span> =</span> some_list</span>[</span>i</span>+</span>1</span>]</span>,</span> some_list</span>[</span>i</span>]</span></span>
                is_sorted</span> =</span> False</span></span></code></pre>
This works fine and it will sort any list you throw at it. However we can
slightly optimise it: We know that, after each pass the highest value element is
guaranteed to be sorted and placed at len(some_list) - 1. Because of this, for
each subsequent pass, we can stop comparing the last sorted item. instead of
comparing pairs that we know are already sorted.
This is what it looks like:</p>
def</span> bubble_sort</span>(</span>some_list</span>)</span>:</span></span>
</span>
    is_sorted</span> =</span> False</span></span>
    last_sorted_item</span> =</span> len</span>(</span>some_list</span>)</span> -</span> 1</span></span>
</span>
    while</span> not</span> is_sorted</span>:</span></span>
</span>
        is_sorted</span> =</span> True</span></span>
</span>
        for</span> i</span> in</span> range</span>(</span>0</span>,</span> last_sorted_item</span>)</span>:</span></span>
</span>
            if</span> some_list</span>[</span>i</span>]</span> ></span> some_list</span>[</span>i</span> +</span> 1</span>]</span>:</span></span>
</span>
                some_list</span>[</span>i</span>]</span>,</span> some_list</span>[</span>i</span>+</span>1</span>]</span> =</span> some_list</span>[</span>i</span>+</span>1</span>]</span>,</span> some_list</span>[</span>i</span>]</span></span>
                is_sorted</span> =</span> False</span></span>
</span>
        last_sorted_item</span> -=</span> 1</span></span></code></pre>
After each pass through the loop, we know the right side of the list is sorted
so we decrement the value of last_sorted_item. What this means is that the 1st
pass will loop from 0 to len(some_list) -1, the second time, it will be from 0
to len(some_list) - 2 and so on ...</p>
Time complexity</h2>
The rate of growth of this algorithm is quadratic. Expressed as O(n^2) in
"big-oh" notation.</p>
def</span> bubble_sort</span>(</span>some_list</span>)</span>:</span></span>
</span>
    is_sorted</span> =</span> False</span>                       #</span> time here is constant</span></span>
    last_sorted_item</span> =</span> len</span>(</span>some_list</span>)</span> -</span> 1</span></span>
</span>
    while</span> not</span> is_sorted</span>:</span> #</span> We go through this first loop n times</span></span>
</span>
        is_sorted</span> =</span> True</span></span>
</span>
        for</span> i</span> in</span> range</span>(</span>0</span>,</span> last_sorted_item</span>)</span>:</span> #</span> we go through this loop n-1 times</span></span>
</span>
            if</span> some_list</span>[</span>i</span>]</span> ></span> some_list</span>[</span>i</span> +</span> 1</span>]</span>:</span></span>
</span>
                #</span> execution here is constant</span></span>
                some_list</span>[</span>i</span>]</span>,</span> some_list</span>[</span>i</span>+</span>1</span>]</span> =</span> some_list</span>[</span>i</span>+</span>1</span>]</span>,</span> some_list</span>[</span>i</span>]</span></span>
                is_sorted</span> =</span> False</span></span>
</span>
        last_sorted_item</span> -=</span> 1</span> #</span> constant time</span></span></code></pre>
It's O(n^2) because, for each pass through the loop n times, we loop n times
through the consecutive pairs. It's not a very efficient algorithm when used on large samples of data. It should only be used if you have a
specific case on a small data set.</p>
Next in the series is QuickSort, another interesting and more efficient sorting
algorithm. As always, if you have questions, comments or if you spotted a typo
or a mistake, please feel free to let me know on Twitter, I'm
@aaqaishtyaq</a> and always happy to help!</p>


Algorithms in Python: Understanding Quick Sort
Unknown — Mon, 08 Jan 2018 00:00:00 +0000
Theory</h2>
Quicksort is a "divide and conquer" type of algorithm. One good thing about
it is that the worst case can almost always be avoided by using what's called a
randomized version of quicksort (more on that later).</p>
The idea of Quicksort is to take an unsorted list and select one element from
it called a "pivot". Then the list is rearranged such that all elements greater
(in value) than the pivot are placed to its right, and all elements lesser (in
value) are placed to its left.</p>
This process is called partitioning. At this stage in the execution of the
algorithm, the order of the elements doesn't matter so long as the
lesser/bigger values are placed on the correct side of the pivot.</p>
Partitioning will produce two sublists with the pivot as a separator (
this is because the pivot will be at its natural place after the first pass aka
sorted). The problem then becomes sorting these two sublists.</p>
Note: Partitioning does not require creating copies of the
list, we work on it directly as long as we keep track of the start and end
indices of each sublist.</em></p>
To sort the two sublists, we can apply the same logic as above (choosing a
pivot, and sorting the two resulting sublists) because QuickSort is a recursive
algorithm.</p>
When a sublist only contains a single element, it's already sorted so we can
stop the recursion at this point, it's our exit condition.</p>
Note on choosing a pivot</strong></p>
Some people use the last item of the list, and some people use the median of the
first, last, and medium elements but the most common way is to choose a random
pivot to ensure n log n</code> execution.</p>
Some Code</h2>
def</span> swap_values</span>(</span>lst</span>,</span> val1</span>,</span> val2</span>)</span>:</span></span>
    lst</span>[</span>val1</span>]</span>,</span> lst</span>[</span>val2</span>]</span> =</span> lst</span>[</span>val2</span>]</span>,</span> lst</span>[</span>val1</span>]</span></span>
</span>
def</span> quicksort</span>(</span>array</span>,</span> start</span>,</span> end</span>)</span>:</span></span>
</span>
    if</span> start</span> <</span> end</span>:</span></span>
</span>
        partition_index</span> =</span> partition</span>(</span>array</span>,</span> start</span>,</span> end</span>)</span> #</span></span>
        quicksort</span>(</span>array</span>,</span> start</span>,</span> partition_index</span> -</span> 1</span>)</span></span>
        quicksort</span>(</span>array</span>,</span> partition_index</span> +</span> 1</span>,</span> end</span>)</span></span>
</span>
def</span> partition</span>(</span>array</span>,</span> start</span>,</span> end</span>)</span>:</span></span>
</span>
    pivot</span> =</span> end</span></span>
    partition_index</span> =</span> start</span></span>
</span>
    for</span> i</span> in</span> range</span>(</span>start</span>,</span> end</span>)</span>:</span></span>
</span>
        if</span> array</span>[</span>i</span>]</span> <</span> array</span>[</span>pivot</span>]</span>:</span></span>
            print</span>(</span>"</span>{}</span> is less than </span>{}</span>"</span>.</span>format</span>(</span>array</span>[</span>i</span>]</span>,</span> array</span>[</span>pivot</span>]</span>)</span>)</span></span>
            swap_values</span>(</span>array</span>,</span> partition_index</span>,</span> i</span>)</span></span>
            partition_index</span> +=</span> 1</span></span>
</span>
    array</span>[</span>pivot</span>]</span>,</span> array</span>[</span>partition_index</span>]</span> =</span> array</span>[</span>partition_index</span>]</span>,</span> array</span>[</span>pivot</span>]</span></span>
    return</span> partition_index</span></span></code></pre>
A randomized version of Quicksort would look similar to what's above except that
we must randomize the selection of our pivot.</p>
import</span> random</span></span>
#</span> ...</span></span>
def</span> partition</span>(</span>array</span>,</span> start</span>,</span> end</span>)</span>:</span></span>
    if</span> start</span> <</span> end</span>:</span></span>
        pivot</span> =</span> random</span>.</span>randint</span>(</span>start</span>,</span> end</span>)</span></span>
        array</span>[</span>end</span>]</span>,</span> array</span>[</span>pivot</span>]</span> =</span> array</span>[</span>pivot</span>]</span>,</span> array</span>[</span>end</span>]</span></span>
        partition_index</span>(</span>array</span>,</span> start</span>,</span> end</span>)</span></span>
        #</span> ...</span></span></code></pre>
Here, we set the pivot to a random integer in the range between start</code>
and end</code>. Then, we swap the value at that index with the value at array[end].
If you run the code successively, you'll notice that the pivot is
different every time. It's a nice optimization that can save some time.</p>
Time Complexity</h2>
It's one of the most efficient sorting algorithm. In fact, most sorting
functions that come packaged in many language's standard libraries use an
implementation of QuickSort.</p>
The order of growth for QuickSort in the worst case is quadratic O(n^2). The
average case, however, which is the most common scenario, has a complexity of
O(n log n).</p>
QuickSort works best when used on large sets of data because of its recursive nature.</p>
Aaqa Ishtyaq

What AI Infrastructure Actually Needs from MicroVMs

Scheduling Dense Compute Without Lying to Yourself

Observability for a MicroVM Platform

Host Reconciliation with a Host Agent

Why We Built Our Own Machine Proxy

Making Internal DNS Boring for Private Machines

Private Networking for MicroVM Fleets with WireGuard and IPv6

Fast Starts: Snapshots, Clones, and Warm Slots for MicroVMs

Three levels of speed</h2> I like to think about machine startup in three buckets.</p>

1. Fresh boot</h3> This is the full path. Pull image. Build filesystem. Boot guest. Wait for readiness.</p> It is the most flexible and the slowest.</p>

2. Clone from a prepared source</h3> This path reuses a known-good base, often through snapshot or overlay tricks. It cuts a lot of setup time, but there is still work to do.</p> It is much better, but not instant.</p>

Stateless Regional Control Planes and Replicated State

How a Worker Node Runs a MicroVM

Why a MicroVM Platform Needs a Crash-Resistant Machine FSM

Building a MicroVM Control Plane in Public

Move untracked git files to another directory

Animal Farm by George Orwell

Pulga, Kalga - Himachal Pradesh 2024

1984 by George Orwell

The Idiot by Fyodor Dostoevsky

Firecracker Dev Machine Setup (GCP)

Linux Container Networking from Scratch

Isolation based on Network Namespace</h2> The common thing that constitutes a network stack includes the:</p>

Git exclude files from working copy

How to ignore new files</h2>

Globally</h3> Add the path(s) to your file(s) which you would like to ignore to your .gitignore</code> file (and commit them). These file entries will also apply to others checking out the repository.</p>

Open markdown links in a new tab

The specialized way</h2> This could vary depending upon the markdown parser you are using. If you're using Jekyll, like this website, then you can use the following syntax to open links in a new tab.</p>

Render hook to open link in a new tab</h3> Add the following HTML template file (or render hook) at layouts/_default/_markup/render-link.html</code>:</p>

The Recipe for a Great RESTful API

Responses</h2> In your responses, you should separate the metadata from the body and put them in their own objects.</p>

Status codes</h2> This part is subject to many passionate debates, apparently it's a very touchy topic.</p> Below I will list the most common status codes, what they mean and how I use them.</p>

CORS</h3> You want to enable CORS to allow requests from browsers using AJAX. This can be done by setting the following headers:</p>

How to Set Up the Scrapy Splash Plugin

1 - Configuration</h2> A. Add the Splash server address to settings.py</code> of your Scrapy project like this:</p>

Three levels of speed</h2>
I like to think about machine startup in three buckets.</p>

1. Fresh boot</h3>
This is the full path. Pull image. Build filesystem. Boot guest. Wait for readiness.</p>
It is the most flexible and the slowest.</p>

2. Clone from a prepared source</h3>
This path reuses a known-good base, often through snapshot or overlay tricks. It cuts a lot of setup time, but there is still work to do.</p>
It is much better, but not instant.</p>

Globally</h3>
Add the path(s) to your file(s) which you would like to ignore to your `.gitignore</code> file (and commit them). These file entries will also apply to others checking out the repository.</p>`
