In the last few weeks of the year, I’ve been re-examining how I use AI tools both personally and professionally. The benefit of being able to go a step further in doing things, faster, without understanding the hows and whys, or worse, the false sense of confidence that I could achieve with AI, has really started to concern me.
One part of that is that it gives me the confidence to use technology in ways I wouldn’t normally reach for, both personally and professionally. An example of that is the amount of self-hosting I’ve been doing. I’m someone who prefers the right tool for the job, even when it doesn’t align with the architecture or the broader landscape of the long-term plan. In my role at work, I have more direct control over the direction of our strategy for how we do business, both in theory and in practice. However, AI has given me too much assurance that the decision to buy was right.
Over the last year, in many cases, I’ve examined the cost, supportability, and integration opportunities for leveraging internally built (read: AI-developed) or OSS applications at work to replace many of our back-office systems. Marketing, support, and financials are all leveraging a set of tools stitched together via APIs and automation, which, on the surface, sounds like the dream for a business. Automation driving performance. However, when it came time to review metrics at the end of Q3 this year, I found that many hadn’t met their goals as I had hoped. While time savings from automation meant data exchanges were working smoothly, the system’s mental load was carried by me too heavily. Or worse, LLMs had a better understanding of the landscape and the systems, yet the outcomes missed the mark. I had fallen into a “what’s the best tool” vs “doing the thing” problem without realizing it.
A silent challenge for every application, server, or piece of code is the need for updates, maintenance, change management, and risk management. And at the start of Q4, I realized a large burden was on my shoulders of the platforms and automation rather than the outputs, which drove the metrics I really cared about.
If my objective was to increase net-new logos or improve time to case resolution, the parts in the middle from my days in data governance took hold more in making sure the data was moving well and of higher quality, not so much in achieving the objectives.
Taking a step back from that, I thought hard about how I got here and what I needed to do to move the needle quickly for the remainder of the year. What did I need to change, and how did I need to personally change my outlook to be more focused on the best outcomes for each objective rather than how the objective was met? In an example, we had implemented a self-hosted solution for mass email marketing this year (which, if you need mass email, I can not begin to recommend enough Listmonk, it’s great software), which allowed us to place the server in our perimeter.
I went back and examined my ChatGPT chats over the last while and found where I started a chat with an LLM about the pros and cons of different options, costs, and how to measure reach. What I hadn’t considered, and which in hindsight I knew, was that our partners offered mass email systems, which they are always encouraging us to use more. But did the LLM know that? It’s not publicly available information, and you only really understand it if you work closely with upstream software partners. I had fallen into a trap of “here’s a Docker Compose file and some notes, this will be easy!” which I took as truth.
A point to clarify in this is that I don’t think self-hosting is wrong. In fact, I think for many reasons you should self-host software. Avoiding vendor lock-in, data residency, and downstream compliance/security requirements all make great cases for this. The problem is more so - if you do it yourself, and if you’re in a smaller business, whose shoulders does that fall to? In my case, the problems fell to me. For every application, automation, or system I stood up to make life easier, I added technical debt across the board. It meant that our Cloud Engineers, who we’re not always confident about the inner workings, had to stand in front of the technology and speak to the rest of the business when I was offline or unavailable, and moved a lot of mental burden onto them. While I could spend a lot of time adding documentation for these things, it occurred to me in reflecting on all this, “What value does this bring to the business and them having this automated?” and more so, “Does this move the needle for my objectives?”
While LLMs or GenAI have helped me build confidence in my answers, they have ultimately led me to think independently and build my own confidence that we could accelerate work at scale. But the challenge became the mental load on my shoulders, and the difficulty of passing it on to others, where it didn’t add value to the work they needed to do to move the needle on our objectives.
To put it simply, does a restaurant need to build and run its own point-of-sale system? I think it just depends. If it moves the needle to making sales more effective or reducing costs, then maybe they should. Otherwise, a restaurant’s primary business is selling food (and arguably an experience, but I digress). Anything else is a distraction from selling to more and more customers.
The good news is that over the last three months, I’ve managed to remove all but one application I’ve deployed either by leveraging other applications we had at the business, which we’re better suited to do the job, or by creating documentation and a few Python scripts for when I need the automation to do something once a month. The security risk of running our own infrastructure for those services dropped dramatically, the cost of servers and other infrastructure dropped dramatically (the AWS account we use for backoffice/presales work reduced costs by 50%), and my mental load dropped dramatically as well. And as an upside, moving to leverage our partner marketing, as I mentioned above, added value in our partnership, which was an unintended surprise.
Now, as I chat with LLMs and bounce ideas off them, I am taking more time to reflect more on the output, looking for the unknown and the unexpected. The things the LLM might miss while it’s trying to be accurate at speed.
