Building a Natural Language Desktop Agent (Inspiration)

I got tired of my desktop being a static grid of windows. Even with a powerful tiling manager like Hyprland, I was doing all the heavy lifting manually. I wanted to build something that felt less like a tool and more like an extension of my intent—a background agent that actually "sees" the system state and bridges the gap between natural language and the low-level compositor.

The Build

The architecture is a hybrid. I used Go for the heavy lifting because it handles Unix sockets and concurrency effortlessly. The Go daemon stays glued to the Hyprland socket, serving up a live JSON map of every window and workspace.

On the logic side, I used Python with LangGraph to build the "brain." Instead of a linear script, I built a cyclic graph that follows a sync-think-act loop. It grabs the OS state, decides on a tool action, and immediately re-syncs. I'm running local models via Ollama to keep everything private and snappy, specifically using tool-calling models that know how to talk to my Go API.

What I Picked Up

The biggest lesson was that an agent is useless if its "eyes" are lagging. I learned you can't just tell an AI to "close the terminal"; you have to give it the exact address of that terminal in real-time.

• I dove deep into async Python to ensure the AI's "thinking" didn't lock up my system calls.
• I walked away with a much better grasp of how Linux IPC actually functions under the hood.

The Headaches

It wasn't smooth sailing. Keeping the Go daemon and the Python graph in sync was tricky at first.

• Race Conditions: I dealt with plenty of instances where the agent tried to focus windows I had just closed.
• Prompt Engineering: I spent a lot of time "jailbreaking" the LLM—I didn't want a polite chatbot that apologized; I needed a robotic engine that just executed commands efficiently.

Getting that zero-latency precision took a lot of trial and error.

Built With

• domain
• golang
• langgraph
• python
• sockets
• unix