Vsoch

Liquid Gold

Mon, 13 Jul 2026 00:10:00 +0000

You poured for him a bowl of liquid gold
Rich with life, and stories untold
It bubbled to the rim, a meniscus it formed
Of course with growth, uncertainty adorned.

The trickle was gentle, small enough to stop.
Small enough to account for every drop.
It rolled off of the table and you respond
Your hands, cupped together, embracing bond.

You look to him, surely for a cup.
But inside his heart, panic has bubbled up
He paces, back and forth, bounces his fear.
You watch, patiently, as line of oil grows near.

The burden is now on you, to bear the spill.
A small mountain emerges from a tiny hill.
The burden is now on you, to manage the pain.
That will be your loss, and his feign.

But you feel a pressure, it spins in your grasp.
And you realize it is no longer oil that you clasp.
Force of water, oil, and care, formed a scar.
But you have accidentally created a star.

It shines brightly, glows with elegance.
Only existing because you took a chance.
It smiles warmly, and you both know.
That to the floor it eventually must go.

Strong enough to hold systems together,
Resilient to emotional and physical weather
He keeps his distance, hands at his side,
You cradle the strength that he could not provide.

It is brave to give when they can’t meet you there,
To hold the gap with unwavering care.
But beauty is found in the fault and the spill,
In the accidents born when the panic falls still.

The signal goes out, but it echos right through,
A quiet reflection returning to you.
You will carry the star, and embody its glow
A sovereign joy that continues to grow.

Swan and Mallard

Thu, 09 Jul 2026 00:16:00 +0000

I found a fake restaurant called the swan and mallard, and found the concept immensely beautiful. I appreciate the metaphor of darkness and light, and I also appreciate thoughtful design that creates identity with empty space. I made my own variant, and a small poem to accompany it that reflects the sentiment it gives me.

Swan and Mallard

Swan and mallard, an unlikely pair
One a shadow, one light and fair
Defined by darkness, hiding in sight
Revealed by connection, embraced in light

It is romantic, but not necessarily in the way you first intuit. It could be about two people, but it also could be about two pieces of the self. It might say that comfort and safety comes from acceptance and embrace of both the dark and the light. It might also say that our greatest fears of not being accepted, based in what we see as our darkness, draws the light to us. A third interpretation I like is that both concepts cannot exist without the other. Shadow cannot exist, conceptually, without light.

Quantum Computing

Fri, 19 Jun 2026 00:11:50 +0000

I tend to get obsessively interested in things, to the point my curiosity is insatiable. If I do not understand how everything works I will absolutely implode. That has recently started to happen with quantum computing. I will precede this with a warning that I have zero background in Physics beyond a class in high school that was terrible, and my background in everything else STEM is largely self-taught during my working years. I am going to dive in anyway, because I am on ☁️ nine learning about this. First, I want to review projects that I have encountered in the space.

Projects

When you attend meetings or hear people talk about quantum computing, these are commonly named. I still have a lot of learning to do, and especially diving into each one, but I am starting to put them together.

Qiskit is IBMs software SDK is what you use to write circuits and compile instructions. This is the slack community that a lot of people that work on qrmi are in. They also have a runtime and serverless that are products to combine classical resources and quantum. If I want to write a program for Quantum, I can use Qiskit.
QRMI is the quantum resource management interface, what I have been working on with Flux. This is what is relevant to scheduling, because it helps expose quantum resources as interfaces. It is middleware to schedule and manage jobs on quantum. E.g., I can acquire a session to connect to a quantum device, or submit / cancel / monitor tasks.
QDMI is what Martin works on - the same but device interface. It works with devices as a back-end interface for software to read hardware characteristics and compile specific instructions (e.g., from qiskit).
A qubit is analogous to a bit, but a quantum bit (my deep dive on this below).

So QRMI is system level, and QDMI is hardware level, and they are both parts of a “submit my work to quantum machines” but different layers. To get a sense of how centers are integrating classical and quantum, watch this talk by Martin Schultz (a year old) and this more recent talk that I attended in person! And asked a very naive question… how do I get access to one of these computers? I did not have an answer until about a month ago.

Learning about Qubits

There are several different ways to actually create them (analogous to hardware, an example from IBM is microwave pulses) but for my understanding, I’m considering one like a supersonic spinning coin in the air. At any point in time, I can slam the coin on the table to get heads or tails, but while it’s spinning, it is some combination of heads or tails. At first I was imagining heads and tails having probabilities, but that isn’t quite right. They call them amplitudes, where each can be positive or negative, and they can cancel each other out. A positive number means the quantum wave is in a crest, and a negative is a trough. The mental imagery I have is of a grid of little intersecting waves, which if you imagine combined, is like a moving sheet. Each configuration has its own amplitude point on the sheet in a crest, trough, or neither (this state is called superposition). When a microwave pulse fires, it adds positive or negative values to these points. A path that adds 5 and a rule-breaking path that adds -5 would pool into the same spot and become 0, and that is how they would cancel each other out. A 5 and -3 would leave a tiny little wave of 2. When we take a measurement, which is like freezing the sheet, the computer squares these final values to turn them into positive percentages (probabilities) that add up to 100%. It doesn’t use a threshold to round up or down; instead, it rolls a quantum dice based on those percentages. An optimal path with a high percentage is highly likely to be chosen, while a bad path with 0% will never be seen. When a measurement is taken, that dice roll is what we get back—a single string of 0s and 1s representing that one frozen grid configuration, which is why we run it multiple times to see which answer pops up the most. To go back into my visual imagery, I see water rolling across a valley and pooling into the rivers, the solutions to our problem, but only frozen at a single point in time by the measurement.

My next question was how you model a problem into this kind of program, and you can use software called Docplex that works with qiskit. There are two matrices you define - a distance matrix for some kind of topology (e.g., we could model hardware topology) and a penalty matrix for constraints. The assumption (for something like QAOA) is to assume pairwise constraints, but there are ways to add in helper qubits to model three way+ interactions. I’m going to avoid thinking about that for now. So we write a Python program that loads in the matrices, and sets the objective (e.g., minimize distance). And here is a cool part that I can’t best explain because I have almost zero actual STEM background aside from throwing myself into it like a noodle. There is a wave landscape (!) called an Ising Hamiltonian, and it’s a mathematical equation. Qiskit takes our inputs and translates them to that. And at this point, every valid path (e.g., maybe a scheduling solution) sits at the bottom of a low-energy valley, and all of our distance/penalty violations are steep, high-energy hills. God, this is so cool. 😎

Now we run the algorithm. The one I mentioned is QAOA (Quantum Approximate Optimization Algorithm). In Vanessa terms - “how many times should I let my waves crash and slide around this landscape to find a lowest point? As a reminder, a really high point (heavy penalty) is what we have modeled to be not ideal, and ideal is low.” So in the actual quantum computer, assuming IBM with microwave pulses, the qubits are initialized into superpositions of all possible solutions, and then the machine fires microwave pulses based on distance rules. Invalid options will destructively cancel out, and we go until we take a final measurement coinciding with our last planned pulse. Then the quantum computer does the freeze, convert to 0/1, and it returns them via QRMI/qiskit to our classical computer, and we convert that back to meaningful variables based on our model (Docplex variables).

Scheduling Quantum

We have started modeling Quantum APIs into our scheduler, Flux Framework. More specifically, flux-sched or fluxion, isolated from the core of flux (flux-core). I do not need the core components to build stuff in Kubernetes. This is what I am calling the two-queue problem, or the fact that we are modeling a queue that we do not control (but can inspect the depth of) and need a good means to orchestrate work between classical and quantum systems that does not have resources running idle on the classical.

I love this work, and am in that state of being inspired, thoughtful, and focused, all at once. Right now I am still working on our custom scheduler plugin, fluence, and doing early experiments to measure latency and test approaches in Kubernetes to handle it. I started using IBM Quantum Cloud, ran out of the free trial, and have now moved over to Braket. I will ideally be able to work with both. Next we will look at local on-premises quantum modeling, which I will not talk about publicly. In parallel, I am interested in the actual programs, and specifically modeling problems (algorithms) in scheduling to run on quantum devices. I will note that I have been working on this for less than a month, so please excuse my lack of expertise - you are seeing someone learn about a new area in real time! Here is the first set of projects that reflect the Kubernetes work, recorded to present at a 6am meeting at which point I was sleeping. :)

They are already much further along than I presented there. Stay tuned for more! 🥳

Treading Water

Tue, 14 Apr 2026 00:11:50 +0000

There were once a man and a woman, and they were in a big, beautiful, and deep sea. It was the color of nautical twilight – a deep purple that was stained with elegance, memory, and the softness of a heartbeat. They were treading water. It is unclear why the man was treading water, but the woman chose to stay with him.

She could not touch him, and could not help. But she chose to tread water beside him to make sure he knew he was not alone. She wanted to bring substance to what would otherwise be empty. What is heavy is not always burdensome if it carries meaning. What is light is not always freedom. The heaviness of the task was the ultimate fulfillment. The lightness of the quiet was only lonely when the lapping of the water declared its presence.

He is treading and waiting for something different. Maybe a boat. She could not be a boat. And so she waits with him, also treading water. At some point, she will become tired and be forced to return to shore. But she will not regret the time spent next to him. And the story is OK, because she does not want to be a boat. She will return to the shore, and walk along it collect shells. What is important is not an ultimate state of finding a boat. What is meaningful is making the choice to wait, floating not in water but laughter, and knowing that the joy and strength resides within her.

Observation

Fri, 10 Apr 2026 00:22:00 +0000

A good life is a balance of different kinds of thinking. We have our preferences for thinking style and content, and mine tend to vary between technical and behavioral, where often I am experiencing a gradient of both but leaning stronger in one direction. I have been leaning technical for over a month now – starting with the HPSF Conference in Chicago, and returning and becoming completely immersed in not one, but two paper that led to simulation experiments for which (ultimately) I ran over 20K simulations for agentic job negotiation and selection paired with resource provider discovery software, a hierarchical server architecture to orchestrate the work, and algorithms and method to test it all out. It was an exciting, and full set of weeks! However, today I would like to talk about the other side of my dimension. I want to talk about observation, acceptance, and learning. Let’s dive in.

Going off a Cliff

You have a dream where you are in a car, and in a forested and quiet area in the night. It’s a beautiful car, but it does not behave as expected when you push the accelerator. It goes backwards and then forwards at high speed. You find yourself going off the road, through the trees, and plummet off of a cliff. There is a moment of weightlessness and time in a standstill. How do you a feel?

A younger (or real-world) version of you would be terrified. This is the point in the nightmare when you wake up, and are a mix of relieved and upset. Today, you are overtaken with calm. You still wake up, but you are are not fearful or experiencing a negative emotion. You are curious. Why?

Acceptance Before Experience

The calm results from adopting acceptance before experience. It has resulted from years of meta-cognition to understand yourself and the world. Let me explain. When we are younger, we are reactive. We experience the world, ruminate, and either react or choose to suppress. A child will likely react in the most behaviorally logical way. When you touch a hot stove and it hurts, you cry out. A young adult might learn the social fabric wrapped around their reactions, and learn to suppress. They touch the stove, understand that it is “weak” to show pain, and hide it. Many of us do not progress from either of these states to react or suppress. If you watch the famous scene from American Chopper (the one that resulted in the meme, yes) you will notice the second, younger man, go from a state of expressionless shock into immediate reactive anger. We know well from the meme that it turns into a yelling match and he storms out. He reacted. What are alternative endings for this interaction?

He might have suppressed. He might have decided to internalize his anger, like a packet of negative energy forced into a tiny pocket and shoved deep inside his chest. This is a strategy that works in the short term because it allows you to handle a hard situation. Often quick, emotional reactions can be unpacked later, and then discussed when emotions have cooled down. The problem is when we never unpack. We continue filling up our internal pockets until every crevice is stuffed. And then the (typically) negative and unprocessed emotions explode out. Exploding pockets are collections of so many life experiences they no longer have specific direction, only valence. We direct the energy inwards to ourselves, and onto others. It is the reason that someone might seem to unexpectedly explode, and the magnitude of the reaction often does not make sense.

The Alternative of Observation

The other alternative is to observe. Observation is based on a pattern of experiencing the world, ruminating, understanding thoughts and behavior, and then making a decision. The nuances of the decision are what set people apart. You might decide on a strategy of control. How do I control my environment so this does not happen again? Does it mean avoidance? Does it mean setting a boundary? As a young woman I practiced this a lot. My negative emotion of choice was anxiety, and primarily oriented to social. A bad social experience could simply be nullified by avoiding people. If I had a bad experience traveling, I could stay at home. I can understand how agoraphobia develops. What happens with this mindset is that we inadvertently create boundaries to protect ourselves. It’s a false sense of protection. It’s a self-inflicted cage. The safety measures we impose prevent us from living fully, from being authentic and vulnerable with others, and connecting meaningfully. Our life experience gets smaller. That is, of course, what happens when you live in a box.

There is no Control

The insight is that it is not possible to have control. Events occur with different valences and probabilities of happening again, many of which are outside of our awareness. Logic can help here too, and we can identify our own inconsistencies. Look up the probability of dying in a plane versus a car crash, and then think about the prevalence of being afraid of driving versus flying. It is not logical. We have a false sense of control while driving because our hand happens to be on the wheel. We cannot control events, nor can we inflict control on the thinking of behavior of others.

Let’s start again with the sequence. We are hit with life experiences, we ruminate, and possibly get stuck in our heads. We do not choose reaction, suppression, or avoidance. For each new experience, we start to recognize patterns. It might be that we are defensive for a particular topic. Is it because we have an insecurity about it? Where does that insecurity stem from? Is the foundational belief that underlies it true? We unwrap our (typically) unconscious thinking to form understanding, and find that we react less. The experiences feel less directed at as, and more like streams flowing by us. Our default state is one of curiosity. We are not suppressing. We are understanding. The world is no longer dangerous, but interesting. We start to notice more, process, and learn about ourselves, our needs, and create hypotheses about the underlying mechanisms of what we observe. We become better at applying similar logic to others. We notice more, to the point we are no longer resorting to impulsive reaction. We stop trying to control. This is meta-cognition – the ability to step outside of ourselves, observe, and critique our thoughts.

The epiphany that many of us might have as young, ruminating adults is that there is no control. However, there is wisdom in the Serenity Prayer. Lack of control does not extinguish lack of choice. The degree to which something affects us is a choice. We can choose to stop reacting and trying to control our environment, and just observe it. Observing becomes the primary state of being, and we accept things before we have experienced them. Their existence or outcome is separate from a sense of self. The events that happen around us do not define us. The car goes off the cliff, and we observe the experience. We have accepted it before it happened.

Unintentional Impact

Marcus Aurelius talks about the soul being dyed the color of our thoughts in his personal writings. I read this, and I loved the metaphor. The practice of observing our experiences and changing our perspective can color the mind a more beautiful hue. It builds on itself as we face the world with more joy, and it reflects back to us.

There are further benefits. The first is appreciation. We often do not actively see the things that are right in front of us. We forgot that we are wearing a watch or bracelet because it is biologically more efficient. Or perhaps we do not appreciate what is on our wrist because we are perceiving a future better model. We do not realize the value until our wrist is bare. We notice absence of a previously fulfilled need more strongly than its presence. And that is the basis of nostalgia – a realization of appreciation that only is possible when the object or person is no longer present to appreciate. This is the benefit of observation. It reveals the details. We move from a mindset of wanting what we don’t have to realizing what we have is exactly what we want.

To go back to American Chopper, the alternate ending to that scene is that he observes, he listens, says OK and stands up and walks out. He understands his own needs and what is happening in the environment, accepts it, and decides it is time to walk away.

Closing

It is an admirable goal to practice being an observer. The choice of how to react (control, suppress, or observe) is the first level of a hierarchy that we choose from. The degree to which we are able to, in real time, observe and change our thoughts is meta-cognition. When our perception is not anchored in fear, we can live more fully.

I Love You, All of Your Pieces

Sun, 22 Mar 2026 00:22:00 +0000

We are hard on ourselves. We can consistently push our bodies and minds, and put others first and strive for compassion and integrity in our lives. But we often forget ourselves. Or we are forced into hyper-independence that makes us long for the care or validation of someone else. We are often told by the very people we love that we are not enough.

What do we do? We take responsibility for ourselves and the power of our words. We love everything a little bit more knowing it will not last. If time is subjective, we decide to perceive the details of a moment. We start to see that some light only reaches us because something got in the way. We realize that others being incapable of meeting us even some of the way, or providing the intimacy we crave does not deem us too much, but having higher resonance that requires an equivalent match. We realize that we are the protagonist of an exciting, wild life story. Our silence and space becomes intentional, and our moments of fragility elegant. We find forgiveness for all that we see wrong in ourselves, and realize that acceptance only comes when we stop fighting reality. We see the essence of life’s duality, that perfection does not exist without imperfection, and light does not exist without darkness. We decide to be a light.

Regardless of how anyone judges you, you get to decide your definition of strength and beauty. There is beauty in our scars, and imperfections. They quietly tell the story of our lives. You get to decide that you have worth, leaning toward people that support you to grow, and walking away with dignity from those that do not respect it. You get to decide to flip every conventional love song on its head, and love all of your pieces. You are the strongest person you know. Your strength does not need to be conventional, nor does it need validation by anyone. You become your decisions. You can find false protection by trying to control others and your environment, or true strength by learning to sit comfortably in the unknown.

Never give up on yourself.

I am not trained to sing (and I am not very good) but I enjoy it as I do dancing, and I recorded this thinking of someone that I love. I hope you appreciate it.

Your browser does not support the audio element.

This song was always special to me. It tells a story of finding magic and story in life, in the small things, always. I often wish that I could share my view of the world with people that I love, because it feels like a beautiful place.

The Fifth Decade

Fri, 02 Jan 2026 00:09:00 +0000

Happy 2026! Let’s start the year off with some dancing. 💃 This year I promise:

To continue to live with authenticity and integrity.
To be inspired, diving deeply into ideas and learning.
To adventure, and look for beauty in unexpected places.
To not make myself smaller to make others comfortable.
To push myself physically and do things that are hard.
To age with elegance, and appreciate myself as I am.
To believe a person when they show me who they are.
To dance, laugh, and be curious about the world.
To prioritize self-care and rest.

To little funny looking, goblin me: you finally grew above 4 feet. You’ll feel out of place for a long time, and life is going to be hard when you enter adulthood. You’ll grow from it, and you’ll be OK. You will learn that beauty and value are within you, and Sovereign. They do not require validation or being chosen. They exist because you are a person who notices the world.

This closing year, 2025 had a lot of adventures with dancing, Flux tutorials, running and biking and beautiful places!

Onward to the adventures of 2026, still as a goblin, but taller! 💪

Gifts

Mon, 22 Dec 2025 00:00:00 +0000

Presents are interesting to think about. It’s an exchange of a physical item as to say, “I value you.” And don’t get me wrong - they can be fun, and informative about a person or relationship. I’ve historically been someone that puts time into thinking about the right gifts for specific people. That goes back to sewing pajama pants and blankets in high school to custom gifts for friends and family to (as I’ve been older) niche food or experiences I know the person will enjoy. I have spent upwards of a year carefully preparing (what I deem to be) highly meaningful gifts. Experiential gifts are also a lot of fun, especially for a group (e.g., a pinata or food item to enjoy together). The exception to that are things that the person really wants or needs, which is something else you can do well when you know someone well. Receiving a gift of something you really like (and will use) that you maybe would not get yourself is the sweet spot (e.g., high end chocolate).

Where it starts to go wrong is expectation. When birthdays or holidays roll around and you are expected to send a gift, what do you do if it isn’t in your heart? What do you do if there isn’t anything the person needs, or if you just don’t feel like it? There is too much expectation that is created by advertisement and a generally consumerist society. It starts to feel badly to feel trapped into having to send something. It feels equally bad to receive something that you feel was forced or actually indicates that the person doesn’t know you at all. And let’s take this a step further. Gift giving, if normally done, when withheld, turns into a pathological means to express a lack of approval. The quality or quantity becomes a point of comparison, and I’m guilty of making this comparison and feeling like I wasn’t valued quite as much.

As I’ve gotten older, I’ve realized something important. The most meaningful gift that someone can give me is their time. Time is limited, and precious. It reflects intellectual curiosity, openness to experience, and choice. Two people choosing to share time is a mutual desire and not an obligation. There is no need to assert value with words, because it is shown through action. There is a mutual shared value of conversation, which typically comes down to emotional and/or intellectual connection. I know I’m valued not because I receive an obligatory present in the mail, but because I just spent many hours with someone engaged with me, laughing, and my internal cup of connection is overflowing.

I will still enjoy (and give) presents, but I no longer feel an obligation. And I’m also investing more in myself, both in terms of self-care and tangible items. My cup has overflowed this year, and I’m grateful to all the people that have been a part of that. That said, when you receive a present from me, know that it comes from my heart. When you spend time with me, you probably already knew that. ❤️

You may also have noticed that I have not written as much this year. The reason is because the year has been too rich to want to spend the time. I am finding that my adventures, whether they be to new cities, intellectual dives into ideas with colleagues, balanced rest, or to scale a mountain with my best friend, are how I want to spend the time. I am branching out, taking care of myself, and expanding my world in ways I never did before, and possibly a lot of the ideas that wound up in writing here are being spoken in a much more engaged context. It was a year of adventure, learning, and growth. On to another in 2026! 🥳

Agentic Orchestration of an HPC Workload in Cloud

Fri, 21 Nov 2025 09:00:00 +0000

One of the most satisfying and learning-rich pieces of work from this year is represented in this white paper, “Agentic Orchestration of HPC Applications: A study using Google Gemini in Cloud.”

First, I’ll provide a little bit of back-story. We were using simple models to convert JSON job specifications or batch jobs between formats. I had a sense that the agents (specifically, Gemini) could do much more, and dove in. At first I was not sure the agent could successfully build a Docker container. It did. And then I was not sure about deployment and optimization in Kubernetes. That worked! Of course, there was a lot of nuanced detail with respect to how the orchestration was done, and how me (the human) interacted with the agents as a team. The learning from this early work is represented in this white paper.

In summary, we used an agentic team (with Google’s Gemini) to build, deploy, optimize, and run scaling studies for HPC applications in Kubernetes. Work is underway (and most of the software done) to do similar experiments using AutoGen, LangChain, and a more formalized state machine design with Model Context Protocol (MCP). This work is immensely exciting because we have more ideas for extending agents to scheduling, topology, and job design. We released this as a white paper since we wanted to extend it before any kind of journal submission, and (for me) I care more about sharing the work than getting it into some high-end venue.

Rootless User-Space Kubernetes with GPU

Sat, 22 Feb 2025 00:00:00 +0000

This is a first prototype to get GPU devices working in User-space Kubernetes, or Usernetes. I am calling it a prototype because it is a “works for the first time” and will be improved upon. For our use case, we will be testing and using on clouds that have NVIDIA GPU devices, however we will need to support other device types in production, and this will be future work. I want to create this write-up while everything is fresh in my mind, because I just had 2.5 days of working through the complexity of components, and learning a lot.

A bit of background

We want to test User-space Kubernetes “Usernetes” ability to run a GPU workload, and compare between Kubernetes (as provided by a cloud) and the equivalent user-space setup deployed with the same resources on the VM equivalent. Google Cloud has excellent tooling for deploying GPU and installed drivers for GKE, so I was able to get this vanilla setup working and tested in under an hour. The setup of the same stack, but on user-space Kubernetes on Compute Engine deployed with a custom VM base on Terraform, would prove to be more challenging.

I’ve designed various driver installers for previous work, including infiniband on Azure Kubernetes Service and more experimental ones like deploying a Flux instance alongside the Kubelet. NVIDIA GPU drivers are typically installed in a similar fashion, in the simplest case with nvidia device plugin but now NVIDIA has exploded their software and Kubernetes tooling so everything but the kitchen sink is installed with the GPU Operator. Getting this working in user-space was uncharted territory, because we had two layers to work through - first the physical node to the rootless docker node (control plane or kubelet) and then from that node to the container in a pod deployed by containerd. Even just for the case of one layer of abstraction, I found many unsolved issues on GitHub and no single source of truth for how to do it. Needless to say, I wasn’t sure about the complexity that would be warranted to get this working, or if I could do it at all.

Resources and Cloud

For this environment, we are working on Google Cloud, and specifically with V100 GPUs, because I can get them in very small numbers (on the order of 1-4 per node, and for a few nodes). To develop with a few GPU on a node it would be a reasonable cost, about $12.00/hour (for reference, the cost of each GPU is $2.48, and then the corresponding instance is ~1.75). This is good example of how tiny bits of resources can go a long way if you are a developer, and (personally speaking) I like clouds best for development environments that I can control over all other use cases. I needed these up for a long time for development, and created easily 50 different setups over a few days. When I had nodes up for most of a day, the total cost was about $150.0. When I realized I would need to do a lot more work, I cut down the number of GPU per node to 2 (my pytorch workflow has a master and worker).

Virtual Machine

When I first started, I took a strategy of using what Google provided. When you select the V100 and navigate to OS, it gives you an option to select one of their ML optimized images. These images are very old (Debian 11 is the newest, which I think dates to 2021) and they only go up to CUDA 12.3. I thought that would be OK to start, but in retrospect it made the environment more error prone. I had to remove and reinstall docker as rootless, and there wasn’t transparency about how the initial Debian base was customized. A good strategy for building these images is to start from an empty slate to the highest extent possible to maximize transparency of what changes have been made.

What ultimately worked was to start with an ubuntu 24.04 image and install my own drivers and CUDA, and then I could choose versions selectively (CUDA 12.8, and I seem to remember the driver version being used was 560.xxx). I was a bit nervous about this because the recommendation was lower than that for the V100 on the n1-standard family, but their provided ML image wasn’t working for me so I had an open mind. You can see the driver install commands here.

Usernetes

The install of Usernetes was typical. You need to enable several kernel modules, cgroups v2, and install a rootless container technology. I chose rootless docker, although on HPC systems you would be forced to use podman. I also set most limits (e.g,. nproc, memlock, stack, nofile, etc.) to unlimited.

One gotcha in this setup that is specific to Google Cloud is how logins to machines are handled. You will typically get OS login, or otherwise login as your email / username. The problem with this is that you get assigned a really high id, and one that isn’t present for any ranges in /etc/subuid. What happened for me on the first day is that rootlesskit was failing (somewhat silently, or at least I missed looking in places to check for it) so I was running rootful docker. The problem was not only that uidmap wasn’t installed, but that the user didn’t have a range. It was actually the debug output of nerdctl that I tested on a whim that pointed me to the issue with the setup, and shout-out to Akihiro for again excellent work. I decided to use the default ubuntu user, with id 1000, akin to what I did on AWS and Azure.

Rooty Docker!

Using “ubuntu” poses a bit of an issue for ssh-ing in. The “gcloud” client is not going to easily allow ubuntu. What I needed to do was first ssh in with my os login, add a public key for my machine to /home/ubuntu/.ssh/authorized_keys and then ssh in as ubuntu. For the terraform setup that doesn’t expose an ephemeral IP, I needed to edit the instance with the control plane, to add an ephemeral IP for ssh.

As a side note - it wasn’t hard getting GPU devices to work with rootful User space kubernetes (yeah, that doesn’t make sense, does it)? I couldn’t use this setup, even as a mock, because rootful breaks usernetes on a multi-node setup. I was able to create a GitHub CI test that reproduces the issue, and hopefully it will be fixed soon! I’m thinking it’s probably related to a rootful docker not properly working with slirp4netns, but I am not an expert there and haven’t looked into it.

The last customizations for docker needed on the host VM were to install the nvidia container toolkit “nvidia-ctk” and configure it to use the docker runtime, and with CDI (the container device interface). For this step I allowed it to generate devices for the development machine I was on, and note that these need to be regenerated when that VM base is used for Terraform.

Docker Compose

Most of the issues on GitHub and instructions for rootless docker and NVIDIA GPU had one indirection in mind - getting the devices on the host to show up in a single docker container. We have two indirections, because we need to map the host devices into a node on the VM running the kubelet, and then that node (a rootless docker container) has containerd that needs to further pass those devices to containers running in the User space Kubernetes cluster. This means we need to solve the problem twice, and essentially have every component in the stack (e.g., the nvidia runtime config file and nvidia toolkit install) duplicated. I found a lot of GitHub issues (here is one open since 2023) that would suggest setting no-cgroups = true in the nvidia container runtime config at “/etc/nvidia-container-runtime/config.toml” and I did try that, but found that it failed after the second indirection.

A Gotcha with the Nvidia Runtime

Many instructions directed to tweak the nvidia runtime to have “cdi” enabled as a feature, and then point to the nvidia-container-runtime executable for the runtime. In fact, there is a command to easily do that. What I realized was that with rootless docker, it wasn’t picking up the default location of the daemon.json, or where the user space one was expected to be. I looked into the service and found that it was running “/usr/bin/dockerd-rootless.sh” and tweaked the entrypoint of that file to explicitly target the config, like this:


  exec "$dockerd" "--config-file"	"/etc/docker/daemon.json" "$@"

That was a manual change I had to make on the VM (that is saved as the base image for Terraform). It’s important to validate that the nvidia runtime is present (detected) along with rootless before moving forward:

  docker info | grep -i runtimes
   Runtimes: io.containerd.runc.v2 nvidia runc

  docker info | grep root
    rootless

You should also test the nvidia runtime before moving forward. You should be able to use it with a vanilla ubuntu container and have “nvidia-smi” working and seeing devices!

  docker run --rm -ti --device=nvidia.com/gpu=all ubuntu nvidia-smi -L
  GPU 0: Tesla V100-SXM2-16GB (UUID: GPU-798e9725-623d-ca7f-f15d-b1908ec8bb0d)
  GPU 1: Tesla V100-SXM2-16GB (UUID: GPU-be5719da-cd52-8a40-09bb-0007224e9236)

docker-compose yaml

The tweaks to the default usernetes docker-compose.yaml are minimal. I had tested added permissions (caps, for example) but ultimately just needed to specify using the nvidia runtime, and then the list of devices. You can see the setup here, and note that I think (but have not tested) that adding “devices” vs. the “deploy” directive do the same thing. Note that if you try to start the control plane (or a worker) with “make up” without using the nvidia runtime and asking for devices, without the “no-groups = true” you will get an error, specifically this one about bpf_prog_query with failed permissions. That is another issue that has been open since 2022. 🙃

Usernetes node

A Usernetes node can be the control plane or a worker. The general procedure for the control plane is to bring it up, run kubeadm init, install flannel, make the kubeconfig, and then prepare a join command to send to workers. The worker nodes also need to be brought up with the same setup, but then they just need to have the join-command (it’s a text file that is executed as a command for kubeadm join). The additional step I needed to add to this was a Makefile command to “make nvidia” that would setup CDI to be used inside the node.

 .PHONY: nvidia
 nvidia:
	$(NODE_SHELL) nvidia-ctk system create-dev-char-symlinks --create-all
	$(NODE_SHELL) nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml --device-name-strategy=uuid
	$(NODE_SHELL) nvidia-ctk cdi list
	$(NODE_SHELL) nvidia-ctk config --in-place --set nvidia-container-runtime.mode=cdi
	$(NODE_SHELL) nvidia-ctk runtime configure --runtime=containerd --cdi.enabled --set-as-default
	...	
	$(NODE_SHELL) systemctl restart containerd

In the above, we create a set of symlinks that I found were needed in practice, but I would see errors if they didn’t exist. For the GPU operator, I found that there was an environment variable in the validator that needed to be set to disable trying to make them, which would fail in usernetes with a permissions error. I didn’t wind up pursuring that path further (using the GPU Operator) because it was highly error prone, and making changes that led to a broken state for Usernetes. We are also generating the cdi file “nvidia.yaml” in “/etc/cdi” and setting the nvidia container runtime mode to use it. Finally, we are configuring the nvidia container runtime to work with containerd, and (still) with CDI enabled. The sed commands (not shown) are uncommenting and enabling different settings I found that would (at least at face value) possibly help in rootless mode. Finally, we restart containerd.

It took me a lot of testing and learning (I have no experience with CDI or working with these tools beyond installs of the nvidia device plugin that have just worked on clusters in the past) to get to the above. You can see the full Makefile here. At this point, we have the node configured also with the nvidia container toolkit, and containerd updated (and restarted) to use it.

NVIDIA Device Plugin and GPU Operator

It’s typically easy to apply the nvidia device plugin to have devices detected and working. This gave me quite a bit of trouble, because at first (when I wasn’t using CDI) it only detected anything when I specified “tegra.” That would have the labels show up on the nodes, but then when I tried to create pods they would fail not knowing what tegra is (and understandably, that’s the wrong setup). Changing it to use nvml would fail to find the library, and “auto” didn’t work at all (at least at the onset). Before I got the CDI just right I went through half a day of going back and forth between using (and trying to tweak) this yaml file and testing the GPU operator, and found a lot of really weird states.

Several times I could deploy the first, have the GPUs show up, but then fail in the cluster, and then apply the GPU Operator. A few times that seemed to work, and other times (most times) it just led to more errors, and not even getting so far to get labels for the GPU. I don’t know how this worked once, but when I tried to reproduce it, I would get containerd operation not permitted errors, along with an error about a PID. There were at least 5 times when something would work, and then I would save an image of the VM with the changes, bring up the Terraform setup to reproduce, and reach the “moment of truth” with deploying pytorch and be faced with another new error. That usually felt bad. 😞 My best guess based on this work is that we were having interactions between components and slight differences in GPU operator components coming up that led to inconsistent state.

What I ultimately decided is that the GPU Operator was too complex to understand easily. I tried customizing the values.yaml install with helm to disable un-needed components (for example, I don’t need MIG here to split GPUs, and the V100s don’t support that anyway) to try to simplify (and make it understandable) but my intuition told me that it was too complex. I didn’t like that it seemed to break the setup, give me inconsistent states, and there were so many init containers and dependencies it wasn’t clear if there were race conditions. All I can say is that on the rare cases something started working, it didn’t reliably reproduce in this rootless setup. In several of those cases it worked for a first run, and then broke for subsequent with new errors. This is what led me to not use it, and focus on the details of the CDI and the simple device plugin daemonset deployment. That ultimately worked like a charm, despite not being recommended for production setups.

The Application

The ML app wasn’t without issues. Specifically, the entrypoint for the master or a worker might look something like this:


  python3 /opt/pytorch-mnist/mnist.py --epochs=10 --backend=nccl --batch-size=128

I hit several errors about not finding GPUs, or (in the case of rootful Docker with usernetes) the networking never working. I had a mangly device error that was resolved when I updated the container to the latest version (now 2 years old). Another unexpected issue was with respect to data. I had prepared data to use from the old container, and when it was attempted to be used with the newer version, it wouldn’t validate and would try to download. Given that the download links weren’t working at the time, I couldn’t run anything. I had to ensure that the data matched the container. There is more on that here. We ultimately build our own container with the data to ensure it is available, and we won’t take time during our experiments to download it. Side note - in that exercise I learned that I could convert a Python egg to a zip, unzip to explore, and then make changes and repackage into an egg! YOLO!

Summary

The final setup is using an ubuntu 24.04 base with CUDA 12.8 and drivers 560.xxx, and using a strategy of rootless docker for usernetes with the nvidia runtime exposing devices via CDI. Setting no-cgroups to true or not using the nvidia runtime will not work, either due to needing cgroups later or the bfp permissions error I noted earlier. Once in the container, we need to again prepare CDI to be used with containerd, and ensure that we generate symlinks in advance. The GPU operator results in a broken state, and the nvidia device plugin, on its own, can best expose devices on the nodes to be available to the pods. Here are a few images for posterity that show everything working. First, the nvidia device plugin:

And this is when the devices show up (just one GPU) on our tiny nodes. For context, we can’t get many GPU on Google Cloud, so we are maximizing the number of GPU per node, since we primarily need to test Usernetes for network.

And pytorch working - go epoch, go!

Reflecting

This process from zero to working that lasted 2.5 days (with a bit of sleep between days) was uncharted territory, and I knew would be challenging from the getgo. It is an experience that I must strangely enjoy? I say this because I have moments of joy and anguish, and several times in the period of time I “decided” to give up. But somehow (even when there are many other things I should have be doing) I found myself continually returning to the setup. It meant waking up with “just one more idea” or bringing down a setup, finally eating, and then during mind wandering deciding that I wasn’t done yet.

I don’t know if this is mental strength or just stubbornness (I think likely the latter). It’s a brute force approach, when I think about it, because I almost refuse to stop until I physically fall over. This didn’t happen the first night, but did the second night. It’s also a kind of problem that I know I won’t get help with. Not to get too philosophical, but I’ve realized through life experience that if I want something done, I need to do it. If I want change, I need to figure out the steps to take and take them. It’s easy to defer responsibility or blame, and I won’t do it. Often it’s not productive, because it accomplishes nothing. That kind of approach applies in everyday life when it comes to making choices about taking care of oneself, and also for learning and solving hard problems. Knowing this was an important problem to solve for my institution, team, and community, even the start of a solution, I felt that responsibility. I’m grateful for the inspiration and support I get from my small team to have the inner fire that drives me. It makes the work challenging, and when we solve problems together, fulfilling and fun.

All in all, I can’t say what the essential fix was, but I will say this is a complex setup. In retrospect, my advice here is to follow intuition, try to build components that you have the most transparency (and control) over, and choose simplicity over complexity over what can be software bloat for a simple use case. And on that note - I am going to leave my dinosaur cave and go outside! And then likely I really do need to come back and work on those slides, which I have been very successfully putting off for 3 days now. 😉