Now, given this situation, would you still say using LLMs to generate massive chunks of code given a feature specification is a good idea? In my opinion, definitely not. Any change you make needs to be thought through a minimum of 10 times (or any number of times you want; you definitely don’t want your change to blow up something :)). But where LLMs come in handy is when you are brainstorming and understanding the landscape, discovering code paths, potential reasons behind certain implementations, writing extensive and exhaustive unit tests, and so on.

But source code changes going to production should be as lean as possible, especially when the blast radius is close to infinity. To build a proof-of-concept, yes, I completely exploit LLMs to the maximum, but when the same is to be productionized, I always err to the side of caution. Code should be as minimal as possible, each line thought through enough with an extensive test suite — doing so makes it easier to isolate and catch bugs if and when they occur, plus, it also makes it easier to reason about.

While velocity has increased, and it has become easier to onboard onto a new project, slow & steady wins the race applies here.

I’ve spent the past few weeks running performance benchmarks across different machine architectures, and if there is one thing I have realized, it is that the very infrastructure and code that powers benchmarks must be reliable — reliable and unbiased always, no matter what. Benchmarks are a reflection of your system performance, you definitely wouldn’t want to see a 9/10 on day one but 7/10 on day 2 without any changes whatsoever.

Along with reliability, the setup should ideally mirror what real-world production loads and conditions will look like, since, only then it is possible to estimate the best throughput & latency that can be achieved. Yes, I see arguments against this, but how useful is it to only estimate the best case performance when it is almost guaranteed that such levels will never be reached.

Some uber-level observations — latency and queries per second (QPS) are inversely proportional, the higher the latency, the lower the QPS, an average of n-number of runs gives you a good idea of the overall system, the repeatability factor is crucial here, across n-runs you would want to achieve the least possible variance, i.e., if run 1 reports a QPS of 100, run 2 shouldn’t report a QPS of 900, instead the best case is if it hovers around the 100-range.

Outside of systems and performance, evaluations govern a large part of our lives — exams and grades at every step, performance evaluations at work and so on. For anything you do, there is a metric of measure that tells you how good you are at it. Do these metrics capture what they are intended to, can’t say for sure. I would argue that most of the benchmarks in real life are biased — do exams mirror what happens in the messy real-world, probably not, do they capture the specific skills a student has — also probably not. Another parallel thread of thought is how the evaluation mechanisms vastly differ from what is valuable in the real world, what one does on the job is often poles apart from what they are tested on. After having worked in a variety of environments — big tech, startups, building my own, I can say nothing more than the ability to understand a problem statement, look for a solution and talk to the relevant stakeholders matter.

However, I have to agree that designing benchmarks and evaluation criteria is no trivial task, in a world where every student is different and unique in their own way, how do you compare them against a single metric?

Leaving us with no option, those who perform well against those benchmarks are considered to be the best, but what happens to the rest, seems like they’re stuck in a vicious cycle.

Any sort of knowledge work — remembering a configuration, a sequence of steps to achieve something, theoretical proofs and guarantees, none of this, which was probably considered as coveted knowledge at some point of time, is irrelevant.

The barrier to entry to new field is significantly lowered, there’s absolutely no reason to say you “do not know something”, the window to learn & do is the smallest it can ever be today, but of course, there is a need to learn “how” to employ, or rather exploit AI tools.

What has been most effective for me — while onboarding onto a new team at work, or debugging why code can’t be loaded into a PLC controller or understanding the basics of PLC and ROS — has been the ability to identify the right tools and ask the right questions. I would attribute asking the right questions to the creativity of the user, the better you ask, the better you get — or rather garbage in, garbage out!

In the context of onboarding onto a new team, or learning ROS — how can you get the LLM to teach you new concepts and relate it to what you already know, in the context of loading code into a PLC — how best can you explain the errors and provide context through different modalities.

Over the past few months, I’ve experienced this time and again, getting into something new & building has become extremely easy; making deep domain expertise, ultimately leading to creativity more valuable than ever before.

What’s something you recently learnt, that you probably couldn’t ever before?

Sitting in the silence of the night,

Under the bright moonlight,

I am in deep thought,

About the for why's and for what's,

Life is so uncertain,

Yet, we treat it like it's a burden,

Forgetting to live life like how it is meant to be,

constantly indulging in jealousy and greed

The title says it, this is an account of selling not software, but cakes & pastries at a community event. I’ve probably spoken about what it takes to build & sell software and what I’ve seen over the course of the past few years, but this experience felt very different.

Food is fleeting, you buy, you eat, you forget. It’s not like software, which you buy once and most likely end up using forever unless something is terribly wrong, or you replace it with an internal version you build, thanks to AI. The faster the the impact of what you spend money on ends, perhaps, the lesser you think about it — this is an observation I made today, reflect on the last few purchases you made…

To set some more context, I sold cakes in a physical stall, not one where someone places an order somewhere and someone delivers it to them. There were a lot of other stalls too — that sold clothes, salads, art work, other kinds of food and some even sold cakes and pastries.

I’m going to put together today’s experiences & draw parallels to the world of software:

Competitors are selling the same, how will we survive

There were other stalls selling cakes and pastries, the immediate thought yesterday evening was, but how are we going to sell / get customers if others are selling exactly what we have. The reality however, was that each piece of cake was sold, so much so that I didn’t get a chance to eat what I sold! This means there is demand — people will buy no matter what, because people love cakes and pastries. Not to say there was no competition, but this only made the sales pitch (the way people were invited to the stall) better. Are there other companies selling what you are building, probably yes, is that a good sign, probably also yes!

Budget for the unknown unknowns :)

No one uses cash anymore in India, UPI is the way to go. A single QR code was printed, and people were expected to make payments to that phone number. But apparently, there is a limit on the number of UPI transactions that can be done received a day — I was aware of a limit on the amount of money you can send, but definitely didn’t anticipate this. Did this cause chaos, yes — instead of scanning QR codes, I was reciting an alternate phone number to people, but there was a way out. While building software, you always take into account unknown unknowns — in my experience, there are always unanticipated issues that arise. Securing permission to read a database requires permissions that requires 5 approvals which means 10 days, a feature that you believed a library should support doesn’t even exist, there’s absolutely no internal documentation on something you are trying to do, which means you figure out how to do it, and also write the documentation — all of these always happen, but with AI, I can say navigating a few of these have atleast become easier. You add a sufficient buffer before committing a project timeline. Anyways, who doesn’t appreciate it if you deliver a project ahead of time?!

Pricing is dynamic

You’ve probably seen, for enterprise contact sales on most products. Probably means their pricing is specific to the organization they are selling to. Those who are wealthier can afford to pay more, and that’s what keep the cycle of money going. Don’t get me wrong, but as a salesperson, it is your responsibility to read the person you are selling to. Will this person pay at the price I am quoting, some factors that come into play — is this person in a hurry (they need your software no matter what, which means you have leverage now and have the upper hand, could be due to compliance reasons, or something your customer’s customer wants), have they already bought a lot of things (your customer is wealthy, they are okay spending on your software), what time of the day is it, energy levels drop as the day passes (this is probably variable in the context of software, but are they purchasing in the busy quarters of the year, Q4 is usually lighter than the rest of the year)

Reach out to your customers, just enough so they aren’t annoyed

A lot of people passed by my stall. I tried to call out / ask people if they were interested in buying cakes. Some of them stopped and bought nothing, some of them stopped and bought multiple cake pieces, some of them couldn’t care less, while some came on their own. Discovery is a challenge, or as they say, distribution is king in today’s AI world, where code is commodity. The real wins were when people who otherwise wouldn’t have made a purchase did, since I called them to take a look!

FOMO is the ultimate card; create a scarcity mindset

This is the last cake we have, you wouldn’t want to miss trying it out. Yes, people do give into this. People never want to miss out on anything. But in today’s AI world, this is way too hard, and if anything code is only abundant today.

To conclude, FOMO draws people, there is no greater leverage.