Chris has seen his share of enthusiastic implementations that don’t always work out. Funnily enough, it’s not always the clients and partners that get carried away with AI; sometimes it’s Atlassian. Chris noted that the company is just like many others, trying to figure out the impact of AI as they go on. The one thing to point out is the usage of agents:

I think about agents inside Atlassian, and as a team member adopting them, it’s really interesting to see how quickly they get spun up, but then how quickly they only have only one or two uses.

Then you even get to the point where there’s almost like a clean up process. Many agents, duplicated names and activities, and you have to clean those up. You almost need agents to clean up agents.

Chris told us that, in Atlassian, they’re trying to move fast and react to what customers need and want. This often results in making services and face challenges in duplication.

Don’t use AI selfishly

Despite AI agents and tools’ ability to understand almost everything about software, building things with AI is something most of us do by ourselves. We create tools and use AI to write code we’re working on, but rarely we go and seek how we could work with others. You could argue that using AI has become a bit of a selfish thing; we’re all trying to see what we can do on our own. Chris told us that teams that legitimately change the way they work with AI are usually the ones who use it to collaborate with others.

Maybe they’re opening up more of an interface that’s like self-serve to another part of the organization. You know, you think like QA management and test and development, that interfance, how can we make that better?

It’s not about what we’re coding in our IDE, it’s more about how that work is then moving to the next part of the organization. So I really look for where are connections across that whole development lifecycle, not just the coding bit.

AI is magnifying some challenges

An interesting trend in software development today is adding AI for the sake of AI, thus adding more steps in an otherwise very streamlined process. Chris said that we, as humans, need to be very intentional in finding steps where AI can help. We should not just try and apply AI to every single part of every development lifecycle. Doing that just causes something Chris calls “drag”:

Where it causes drag is when poor quality comes down the pipe. We’ve known for years that it costs a lot more to fix poor quality later in the lifecycle than earlier, and I just think we’re magnifying that challenge with AI because it’s so convincing.

Humans can be a little bit lazy sometimes, said Chris, and added that we might let a lot of things pass through unintentionally. Those things get further down the pipeline and AI then causes a real drag because it’s harder to find and fix, and it takes more effort from everyone.

It’s challenging to lead in the AI era

Engineering leaders today, such as higher-level managers and executives, do not have it easy today. Chris echoed this statement, saying that he “feels for engineering leaders at the moment”, and adding that there’s a whole other layer of dealing with both the technical and human challenges of this AI era.

In terms of where leaders go wrong, Chris used a high reliance on data as an example. In his opinion, with more data of all sorts available today, leaders tend to rotate more towards it than towards qualitative metrics:

Managers can defend and they can articulate well with data. There is a risk that you over-rotate purely to data and not those qualitative metrics. So, engineering managers need to also keep qualitative metrics, developer surveys and developer pulses.

That’s how you stay close to what the feeling is on the ground, from both a performance perspective but also a human perspective.

Ask your AI to challenge you

Using AI tools without being skeptical and critical of them might be the single worst trend not just in software development today, but in almost every industry (and in personal lives of many). Chris noted that it’s really dangerous how convincingly wrong AI tools and agents might be, and without knowing how to properly work with them, you might be a victim of their hallucinations without ever noticing it. Chris explained:

A practice I see many teams do is you use AI to challenge you. You have to have a viewpoint and a hypothesis about what you’re doing and why you’re doing it and how you’re approaching it. Then you use AI to challenge that.

Guaranteed, it will come up with things you didn’t think of. It will come up with elements of code or something in the code that you wouldn’t have seen, and it would not have been detected had it not been for AI.

Ask how you could use AI to help other teams

Wrapping up our chat with Chris, we asked him how would he advise those working with AI every day to make the most of their experience, and collaboration with AI tools:

A practical change might be to take some time out of your week, draw a circle and put all the teams that you work with around that circle, and then start to pull your top three or five teams that are critical to that flow of work, and ask yourself, how could we be using AI better in our relationship with those teams?

If you switch your thought process related to AI from thinking how can the technology make your job easier to how it can make working with others easier, it might just be the thing you need to get the edge. Just don’t go using it without questioning the things it says.

The post Chris Davidson, Atlassian: “Ask AI to challenge you” appeared first on ShiftMag.

I soon learned that college did not prepare me for the job, got some kind of idea of the skills and knowledge required to ship production software, and figured out that the more I knew the less stressful the job would be.

So over the course of many years, much against my will, I became an expert. Then AI came into the picture, and nothing fundamentally changed, because the job of a software engineer is to help achieve business goals by leveraging their understanding of the technology, and now there was yet another technology to learn.

Get ahead of security issues

I have decided to focus on making the job less miserable for me. My original approach, once I learned how to use tools such as Claude, was to do the same amount of work as before and then just chill.

Nobody ever complained about my performance, so I figured if I just did the same amount of stuff in less time, I would have time for the things I enjoyed, like drinking coffee with other slackers on the balcony or writing pieces such as this one.

This felt good for a while, but still there were annoying things that needed to be done. One of them is upgrading packages to mitigate security issue. I don’t know about you, but I fall asleep each time I need to read a CVE Record.

So I get AI to write me a short horror story written from the perspective of a sarcastic and frustrated application security engineer whose warnings companies have ignored at their own peril.

The story always has some snippets that help me understand how this exploit might happen on services I am responsible for, and it always ends up with my company going bankrupt, this really makes the lessons stick and motivates me to fix the issues.

Sure I could completely automate the bumping, but where is fun in that, and what do I learn?

The production can be cruel

The other thing I don’t like is the production; the production is cruel. You surround your code with all kinds of tests, you create the production like environment with simulated traffic, you use feature flags, you deploy small increments often, and still production finds the ways to surprise you, and it has a tendency not to do that during business hours.

So you do what you must, add a bunch of observability, create runbooks and alerts. Now, with all that telemetry in place and a coding agent I can finally engage with production properly: not reactively when something goes wrong, but every day I can chat with the production, ask it about its concerns, about what is going on lately, if something feels wrong.

I do this by invoking the skill that gathers all the data and then reports it with a certain tone of voice. And I’ll tell you one thing, gaze long enough into production telemetry and the production gazes back.

Chat instead of jot

One last example, and I envy the juniors for having the ability to do this from the start, is the change in the way how I learn stuff. When I was starting, I used to jot down the things and concepts I did not understand, I would look them up in articles and books, I would go through half-baked internet tutorials, blog posts and man pages, and still come out with questionable understanding.

Now I start a chat session with an AI model, give him a personality, most often it is an old and grumpy but well-mannered distinguished engineer who explains topics with a lot of examples and then asks challenging questions while throwing casual insults, he asks me a challenging question and then roasts my answer before proceeding with the next lesson.

Hope you find some of these ideas useful, because I feel much of the talk has been on how to use AI to speed things up, and too little of the talk is about how to use AI to make software more reliable or the process of building and maintaining it more enjoyable.

The post I Talk to Production Every Morning. It Talks Back. appeared first on ShiftMag.

What’s the earliest reduced version we can put in front of real customers to validate against something we actually want to learn?

At worst, it’s just a date and an announcement with nothing clearly scoped behind it, so that gap needs to be closed before anyone starts building toward it.

That’s where things usually start to go wrong.

Business, product, and engineering often mean different things by “early release”

Once one customer flow is split across several teams, an early release becomes real coordination work.

Someone has to lead it end to end, and that usually falls to no one. The reduced scope drifts and nobody is coordinating the whole release. With each new team that holds a certain piece of the project, coordinating them gets harder. The fix is good management, and it requires someone to actually lead the release.

Part of the trouble is that the same words mean different things to different people who are, of course, the part of the same meeting. When the business asks for an early release, it usually means a reduced scope of something, made to be more of a barebones product than a full-sized one. This is something that’s enough to validate assumptions and deliver value while the rest is still being built. 

To most engineers, “done” means complete, so “early release” gets heard as “the whole thing, but fast.” Put an MVP, the smallest version that tests your least-sure assumption in the mix alongside full GA, the finished product for everyone, and one phrase is now pointing at three different targets.

Nobody draws the line between them, so scope drifts toward GA by default: a team builds one target while the plan assumed another, and that drift is what sinks these releases. 

How one customer journey becomes nobody’s problem 

Multi-team releases break along a predictable line, and it was mapped a long time ago. In 1968, Melvin Conway published a paper called How Do Committees Invent? with an observation that later became known as Conway’s Law. The law states that organizations that design systems are constrained to produce designs that copy their own communication structure.  

Applied to a release, it predicts the break before anyone writes code: split one customer journey across five teams that mostly talk within their own walls, and the software inherits the same boundaries the organization has. The release breaks at the boundary between two teams, where the calls that need the whole picture don’t belong to anyone.

Put that prediction next to a real feature. I kept it deliberately small here, just to picture the shape; the real ones are bigger and messier.  

A customer sends across dozens of destinations, each with its own registration rules. Provisioning sets the customer up to send, sender registration gets each sender approved per destination. The hard cases raise questions neither team can answer alone: when a sender is fine in one country and refused in the next, does provisioning collect a fallback, does registration substitute one, or does the campaign ship partial?  

No owners means no decisions

Since there’s no owner, there is no decision.

Instead, each team codes to its own assumption and passes its own tests, and the assumptions only collide in some cases that cross between teams. The first customer with a mixed set of destinations goes live approved in some and silently refused in others. This is a fix that looks obvious at this size, but at real scale it’s dozens of these calls across five teams, and the someone who should make them is still no one. 

It starts when the date is set before the scope or target customer is clear. Then teams start building without knowing where to stop. One team works on an edge case for full GA, while another builds for customers who aren’t onboarded yet.

“Early release” becomes “the whole thing” slowly, one decision at a time. It stays invisible because each team is doing good work on its own. The drift only exists at the level nobody is watching, which is the whole end-to-end journey, or the big picture. A release that goes out with no defined scope, no chosen customer, and no one leading it pushes the risk onto whoever is downstream.

Cutting scope on purpose is different. You choose what to leave out and handle the rest manually, so the customer still gets value without needing the unfinished parts.

The case against shared ownership

Shared ownership across those teams is where the whole falls through. Everybody owns a piece, which really means no one owns the whole.

When multiple teams each hold a piece of a customer journey, someone has to lead the end-to-end release. This should ideally be one or two people who know the flow in real detail: which service does what, where the boundaries between teams are, what a real customer actually experiences from first call to final result.  

A steering committee can set priorities and handle escalations, but it can’t lead the release day to day. A manager who only sees team status reports can’t do that either. The details that decide the release depend on how the pieces fit together, and neither of them is close enough to manage that.

In practice, that leadership is coordination work, not a box on the chart. The owner learns the whole flow and keeps it in their head. They draw the reduced-scope line and publish it, so every team is working from the same picture of what “first version” means.  

They watch for symptoms of a team drifting, which can include hardening a GA edge case or generalizing too early in the process. They decide where “reduced scope” ends and “next phase” begins and call the release. Most orgs already have someone near this role. Often, that’s a product manager or a TPM. 

Try making one person responsible for the release end to end

Define the reduced scope and target customer before you set the date. Write it down. Give it one owner who understands the whole flow. And when the first customer hits a gap, fix it in the shared path so the next customer doesn’t run into it too.

The reason it doesn’t happen is that leading five teams end to end is hard, unglamorous management work, and in most organizations it isn’t any single manager’s job. Each manager runs one team and looks after that team’s part of the release. The coordination across teams is no one’s job.

Some organizations avoid this because leading the whole is already built into someone’s job, or the structure is flat enough that the whole stays visible. The teams aren’t weak and nobody is being lazy; the job that needs doing simply isn’t anyone’s job. The fix is to make one person responsible for the release end to end, and to give them the room to actually lead it. 

Sometimes, though, one person genuinely can’t lead the whole thing, however skilled they are. In a matrix organization, an agency where the client signs off every step, or a regulated field where the approval path is set from outside, the flow crosses boundaries a single coordinator can’t reach across. 

If your structure genuinely prevents one person from leading end to end, no release process will fix it, because the real constraint is how the organization is wired. The release is only where that constraint first becomes visible. 

The post Why Early Releases Often Fail in Multi-Team Organizations appeared first on ShiftMag.

This article looks at those five individual-contributor levels. It uses Netflix job descriptions, the company’s culture and compensation policies, and reported pay data from Levels.fyi to show the kind of work associated with each level. Those hiring examples should not be treated as Netflix’s internal promotion criteria.

Engineering levels and compensation

Levels.fyi labels L3 through L7 as Engineer, Engineer II, Senior Engineer, Staff Engineer, and Principal Engineer. Netflix’s recruiting titles vary, sometimes using only a number. The examples below describe particular advertised roles; they are not a uniform specification for every engineer at that level.

Based on the last information checked on October 9, 2026, the individual US level pages on Levels.fyi displayed average annual total compensation of approximately $214K at L3, $321K at L4, $509K at L5, $704K at L6, and $1.14M at L7.

Separately, the company overview reported a median of $430K across US software engineer submissions. What should be noted is these are reported compensation figures, not Netflix’s official pay bands or a representative census of its employees.

Netflix’s engineering listings generally describe annual compensation without a separate bonus. Its Work Life Philosophy says eligible employees choose each year how much compensation to receive as salary or stock options. Under its Personal Top of Market approach, the company assesses an individual’s market value rather than defining the market through fixed bands and grades. The advertised ranges still depend on the role and can vary by location.

For example, the L4 DevOps listing specifies $250K to $413K, and the L6 Developer Productivity listing specifies $600K to $1.066M. The published L7 Growth Engineering description gives a range of $240K to $1.2M. Such overlapping ranges are a reason to check the specific role and offer: they cannot be used as a standard salary schedule for L4, L6, or L7.

L3 and L4 Experience requirements and project ownership

Netflix has a dedicated US new-graduate program that includes technical roles, onboarding, and training. Its careers page says new-graduate openings typically appear in September or October, depending on hiring needs. Levels.fyi identifies L3 as entry level, but that does not mean every Software Engineer 3 vacancy is suitable for someone straight out of university.

The Software Engineer 3 role at Ink, for example, asks for three to five years of software engineering experience, preferably in animation production. It involves working with senior colleagues on tools and platforms for AI-assisted production. That is a specific hiring requirement for a specialist role, not a universal experience threshold for L3.

Speaking about L4, the DevOps and CI/CD posting asks for at least four years of professional experience or equivalent practical experience. It describes work on defined parts of larger platform initiatives, with guidance from senior engineers and a manager. An Android L4 posting expects ownership from project scoping through delivery, while still stressing the importance of team support.

Together, those examples suggest increasing independence at L4, with the amount of guidance depending on the role. Project ownership and support from more experienced colleagues appear together in these descriptions.

L5 Senior engineering and system ownership

Globally popular streaming platform has advertised Senior Software Engineer positions at L5. A useful example is its Analysis role in the Experimentation Platform team, which combines responsibility for production systems with input into architecture and the team’s roadmap.

The archived Analysis listing describes workflows processing billions of events each day. Its responsibilities include developing experiment-analysis tooling, responding to incidents, improving reliability, and agreeing data and API contracts with other teams. The engineer is also expected to understand how product teams use experiments. These details show that the job extends beyond implementing a predefined feature.

In this role, senior responsibility includes accounting for the operational consequences of a design and the needs of the teams using it. Netflix’s emphasis on independent judgment provides context for that work, but the Culture Memo applies across the company rather than defining separate expectations for L5.

L6 Staff engineering across teams

At L6, Netflix explicitly advertises Staff Software Engineer roles. The AIMS Platform Systems examples make the wider scope concrete: the engineer works on infrastructure used by other engineering and research teams.

The Developer Productivity role covers development environments, build and test systems, CI/CD, and tools used to move research into production experiments. It also calls for measuring build times and iteration speed, supporting adoption, and setting shared technical standards.

The Staff ML role focuses on observability, infrastructure cost, and reliability, alongside contributing to architecture and migration planning with other teams.

These examples suggest that Staff responsibility includes getting shared systems adopted and making them useful across AIMS. The developer productivity posting makes that test concrete: the tools need to improve how other teams build and run experiments. The work combines hands-on engineering with coordination across teams.

L7 Technical direction across an organization

Levels.fyi calls Netflix L7 Principal Engineer, while the Growth Engineering job description uses Software Engineer L7. Its responsibilities are defined around the Growth Engineering organization, so the scope is more informative than the title alone.

The published role reports to a Senior Director and asks for at least 15 years of professional software development experience as an individual contributor. It includes setting architectural direction across Growth Engineering, connecting business goals to technology investments, and working with senior leaders and other L7 engineers. The experience requirement belongs to this vacancy, not to a published rule for every L7 role.

The remit is broad technical leadership within Growth Engineering. It combines architectural decisions with continued expertise in production systems and mentoring other engineers, including helping managers support their teams’ development.

Comparing level names across companies

Level numbers are company-specific. Levels.fyi places Google’s Senior Software Engineer title at L5 and Staff at L6, while Amazon uses L6 for Senior SDE and L7 for Principal SDE. Meta uses E-level labels in the same database. These naming conventions can help a candidate navigate vacancies, but they do not establish equivalent responsibilities or hiring standards.

The distinction matters most above Senior. For example, Google’s L7 is Senior Staff and its Principal title appears at L8, whereas Levels.fyi labels Netflix L7 Principal. A shared title or similar number does not show where an engineer would be placed after changing companies. The role’s technical remit and the employer’s assessment remain necessary context.

How culture affects engineering work

Netflix’s Culture Memo highlights judgment, candid feedback, and giving employees enough context to make decisions. It also expects managers to stay involved and coach their teams. Autonomy should therefore not be mistaken for an absence of support, nor treated as a benefit reserved for senior levels.

The keeper test is part of that performance culture. Managers are asked whether they would try to retain someone or choose to hire them again. If the answer is no, Netflix says it favors ending the employment relationship promptly. The memo also says to assess a person’s overall record, discuss performance regularly, and support employees through short-term setbacks.

For a candidate, the practical next step is to ask how the hiring team defines scope at the advertised level, what support is available, and how progression is assessed. Public postings can help frame that conversation. They cannot supply a guaranteed promotion timetable or the terms of an individual offer.

The post Netflix Engineering Levels Explained: L3 to L7 Career Ladder, Scope, and Pay appeared first on ShiftMag.

In my team in Infobip, we handle message sending and serve as an endpoint for OTT providers such as WhatsApp, Viber, and Apple. Because of that, message sending is our core responsibility, and we need to be aware of any bottlenecks.

Initially, our service was measured at around 1,000 requests per second, but we didn’t know why.

From time to time, this raised concerns in the team: we should understand better where the bottleneck is and what causes it. Once we know that, we can make informed decisions to address client needs – for example, answering whether we can handle the traffic they expect to send. That was partly what led to this investigation.

How to run the tests

Application

Initially, with no experience in performance testing, I viewed the service as a set of methods – essentially a stack of calls. I assumed I could analyze it with a profiler. I also knew I shouldn’t run it on my computer, because other applications would affect the service’s performance.

That turned out to be a bad assumption. I’d suggest starting by measuring the application as a whole to get a broader view of its performance.

Integration

Since our application forwards messages further down the infrastructure and ultimately to the provider, it had to be adapted so that it would not send real traffic, while preserving as much of the original logic as possible.

Initial tests of this version showed that the service could handle a request in 0 ms, which seemed unrealistic. I realized that I needed to introduce a delay based on production, where the rounded average request processing time is around 50 ms.

Initial measurements

Having established that the service handles one request in 50 ms, we can estimate that it can process about 20 requests per second. Since the service is Jetty-based, it has roughly 200 threads available by default, which gives a theoretical peak of around 4,000 requests per second. With that in place, the service can now be tested.

Testing environment

• DO NOT run such tests on a local machine.
• Dedicated machine on our testing env was created for it.
• K6 was chosen as test suite.
• Rundeck to automate and parameterize remotely run tests.

Endpoints to be tested against:

• /mapi-mock/1/dummy – performs Thread.sleep(50) – so it is blocking code run on Jetty thread pool
• /mapi-mock/1/dummyReactor – also performs Thread.sleep(50)but publishing result on boundedElastic thread pool – it is still blocking code, but in that case one thread of reactor’s pool will be used
• /mapi-mock/1/messages – non-blocking code which sends request through WebClient to real service
• /mapi-mock/1/messagesReactor – non-blocking code which sends request through WebClient but publishing it on boundedElastic

JVM tuning

Make sure the correct memory parameter is used. In my case, HEAP_PERCENT was set to 60, meaning the JVM heap used 60% of the machine’s RAM. That allowed me to focus on machine RAM only, and the heap size would be adjusted automatically. This is not ideal, but it was good enough for our tests.

Metrics gathering

Initially, I ran the tests for 40 seconds, but I forgot that the metrics were collected every 30 seconds. That explains why I didn’t see any issues in the service metrics during the first run. When I extended the test duration to 180 seconds (a value I borrowed from similar tests) everything became visible.

You can also observe the machine directly with tools like htop or top, or any other tool that shows current CPU and memory usage. In the end, you can use whichever tool helps you complete the task.

One obvious but important rule: do not change two variables at the same time unless you know what you are doing. If you do, you won’t know which parameter had the effect. Change one thing, test it, then change the next thing, and repeat until you understand what is happening.know what is going on.

Gathering measurements

Initially, I decided to send up to 5k requests toward the service – it is below theoretical machine efficiency, but higher than what was experienced in the past. Lower CPU and RAM measurements are somewhat weird: 4 GB of RAM is not recommended as a base machine setup, because there is less than 2 GB of RAM left for the operating system, where not only our service lives, but also supporting applications. But both measurements for 2 CPU showed that it is way too low for handling any traffic efficiently and it showed quite high CPU and RAM usage, meaning that it could be improved. I finished the tests with 5k requests/s, when performance was quite near predicted performance.

The theoretical peak was reached, so it was time to check whether it could be improved. I tried switching from Jetty pool to Reactor pool, but it turned out (my bad for not remembering Project Reactor documentation by heart) that to reach the initial performance I had to configure the machine with 20 CPU, because Reactor constructs the pool to have 10 * CPU threads. So when I matched theoretical performance again, but now with the Reactor pool, I thought about what could be done next to improve. The last thing to check was how non-blocking code would perform, so I started using the /mapi-mock/1/messages* endpoints for testing. As one would predict, it was hit. The application peaked at 7,000 requests/s, but with quite large demands for hardware. So the last thing to check was to try to find reasonable settings to have big enough performance with not much resources assigned.

Now we are focusing on the non-blocking endpoint, so it more reflects the real-case scenario, and on limiting resources. The first thing that is apparent is that when it comes to non-blocking code, there is no difference between Jetty thread pool and boundedElastic thread pool (the latter results in some queuing because the rule of 10 * CPU threads is always true). We can see performance downgrade with CPUs removal, but not apparently linear. I had to say “stop” somewhere, and I decided to stop with 8 CPU and 8 GB of RAM, for which we received ~3.5k requests/s. Good enough with not much resources assigned. As we can scale horizontally, adding more machines will use more resources, but give more resiliency back.

 The thing with VT…

As you can see, I focused on measuring reactive vs non-reactive code.

It was our go-to architecture at that time, while VT was being adopted slowly (we were afraid of thread pinning, so we waited until it was addressed in JDK 24 to start adoption). After I published it internally, I was asked specifically about VT – how the app would perform when configured to use this not-so-new-but-fixed feature.

It turned out that VT gave the app a boost. With optimal configuration (8 CPU and 8 GB of RAM) the app was able to handle 10k requests/s. Why is that? Any code, regardless of whether it is blocking or not, will be run on a virtual thread, which ideally should not block a system thread, leading to the case where system threads are used only for real logic and not for waiting. VT can improve applications where there is a lot of blocking code. In the rest of the cases, it is “just” syntactic freedom, no more looking at which operator do I need?

The outcome

When I performed the tests, I observed that just switching to boundedElastic (i.e. “use Reactor and you will get higher TPS”) didn’t increase throughput – it actually reduced it. Not remembering the Project Reactor documentation by heart, I was surprised, but after some research I found that boundedElastic provides 10 * CPU threads, which gave us a total of 60 threads to service all requests previously handled by 200 Jetty threads.

When the CPU count was adjusted to match 200 Jetty threads (20 CPUs) we reached the expected throughput. But it was wasteful in terms of resources. In the Reactor world, people often say that blocking is bad, and my service was actually doing exactly that, on purpose. I thought it was time to switch to a different endpoint.

So I was able to nearly reach the estimated performance of the service, which was good. That allowed me to move on and investigate other ways of improving it, such as using non-blocking code or VT.

What I’ve learned

It is always about finding the bottleneck and finding out if we can improve it, for example by adding resources or redesigning it. If it is not to be improved, then this is the peak capacity of the service.

Sometimes questions from clients can actually help. It is just like with a new person joining the team: they will ask obvious questions about all-known solutions, but if you want to respond accurately, you have to dig deeper, and sometimes you find out that this simple question leads to an investigation and even to questioning whether it is correct at all.

Also, be aware of architecture. In that scenario, we tested a service which accepts requests, so it is one of the first applications, or APIs, which the client experiences. Being first means that there is the rest of the system, which can in turn have throttling applied. What does it mean? Even if we accept 10k requests/s, sometimes we cannot send them immediately, because an independent system cannot handle, for example, more than 2k requests/s. So we can be fast for clients, in the sense that we accept their traffic as fast as they want, but we should always remember that this is not the whole picture, and the client should be informed about it.

In my test there was no DB, just API-to-API calls. Having any limited, and even worse blocking, resource like a DB in the way of processing a request will definitely impact its performance.

The post How 5,000 RPS Exposed the Real Bottleneck in Our Jetty Messaging Service appeared first on ShiftMag.

“Voice AI is an infrastructure problem because voice itself is an infrastructure problem,” says Kosta Pribić, a Senior Principal Engineer at Infobip.

He works with voice services – real-time communications across the globe, across multiple data centers, and for everyone reachable on the planet. Yep, it’s a lot.

That scale makes voice AI a much more complex challenge than developers might initially expect.

Developers can take control of the call in real time

Kosta sees voice AI as a two-layer system. Smaller models built into phones can handle real-time issues like packet loss and background noise. They should do their work quietly in the background, so the device just feels like it works.

Larger models, meanwhile, usually run in the cloud and need access to the audio being exchanged between devices. That’s where Infobip comes in: it provides the communications infrastructure developers need to process that audio in real time.

We give you control over the call. We enable you to interject, do something smart with the media.

He says a lot of the infrastructure behind this, from global data centers to enterprise deployments, was originally built for web pages. But web pages aren’t real-time communication.

So when teams use that infrastructure for voice AI, they end up carrying over decisions that were made decades ago. Those systems weren’t built for a live AI-assisted call, which means developers are often stuck working around constraints they probably wouldn’t choose if they were starting from scratch today.

Biggest challenges are real-time performance and cost

Developers entering voice AI often expect hallucinations to be the biggest challenge. But Kosta says they usually run into a different set of problems first, like custom signaling protocols, real-time delivery, and tricky networking edge cases. Networks aren’t built for this kind of traffic by default, so for developers without a voice background, it can feel like a completely new set of problems to learn.

“The main problem is understanding all of this. It’s completely new,” he says. That’s why Infobip’s first internally built LLM-enabled application was a voice AI troubleshooter. Kosta sees troubleshooting as a practical place to start, because teams first need to understand the communication problems they’re actually dealing with.

From there, model choice comes down to the business case and the margins. Kosta says a large model can make sense when latency isn’t a major concern and the economics work.

If your business case allows for using huge LLMs and you don’t care that much about latency, great, go for it.

But in most cases, he says, latency matters. Teams may need smaller LLMs to process calls, or fast, lightweight models to handle audio. Cost matters too, especially when a model is running on every call, because a large model can become expensive fast.

Voice AI has to be private, reliable, and real-time

People expect voice-enabled communications to be private and confidential, Kosta says. They also expect calls to be as reliable as the telecom services they already use, which he describes as operating at “five nines.” Once that trust is lost, it’s much harder to win back.

For developers who are new to voice, understanding those expectations is part of understanding the system they’re building.

That becomes especially clear in assistive use cases. Kosta points to an Infobip customer using a generative model to create voices for people with damaged vocal cords:

They can speak in a whisper while the person on the other end hears the generated voice. That’s, I think, the most heartwarming implementation I’ve ever done.

In the end, before choosing a model, developers need to understand how a call works, where it can break down, and how much delay and cost the application can handle. Even the best model won’t help if its response can’t make it back to the caller in time.

The post The Real Problem in Voice AI Is Latency appeared first on ShiftMag.

It is Thursday afternoon, and your team is preparing a release promised for Monday. The deadline has already moved once. Sales and leadership know about it, and everybody wants this thing to ship.

During the final review, one of the engineers notices something strange in the database migration. Nothing is obviously broken, no dashboard is red, and no test is failing. But something just doesn’t feel right.

We reviewed this three times already. Are we really reopening this now?

There is a short silence, then: “No, no. It’s probably fine.”

So they ship.

On Monday, production goes down, and during the incident review someone asks, “How did we not see this coming?”

The interesting part is that somebody did but the problem was that saying more about the risk felt more dangerous than staying quiet.

That is what psychological safety is really about: not whether people are nice to each other, but a more useful question – what does honesty cost on your team?

“If I stay silent, probably nothing happens to me today. “

People constantly make small risk calculations at work. If I speak up, I might look difficult or incompetent, annoy my manager, or damage my reputation. If I stay silent, probably nothing happens to me today.

When the social risk of speaking is higher than the immediate risk of staying quiet, teams learn silence.

Amy Edmondson defines psychological safety as a shared belief that the team is safe for interpersonal risk-taking. My engineering translation is simpler:

Can I say the thing that might make me look bad?

In engineering teams, that risk shows up in ordinary moments: admitting a mistake, challenging someone’s thinking, bringing bad news, or saying “I don’t know.”

Those moments all ask the same thing: what happens if I say this out loud?

That is why I think of them as four simple tests: if you want to understand a team, just watch what happens when someone says one of them.

Psychological safety is not about being nice

A common misunderstanding is that psychological safety means avoiding conflict or lowering standards but it’s not. Sometimes the safest thing to say is, “This is not good enough.”

That can be uncomfortable, but the goal is having hard conversations early enough to change something.

For example, in a design review, a senior engineer sees a scaling concern but doesn’t want to turn a thirty-minute meeting into another debate. So they say:

Looks good overall, just one small thought: maybe we should think a bit more about how this scales, but definitely not a blocker.

The manager hears: “No blocker.”

The meeting ends, everyone was polite, nobody was aggressive, and nobody left uncomfortable.

Three weeks later, the system starts struggling and the is a question:

Why didn’t anyone raise this during the design review?

The engineer thinks: “I did.”

The manager thinks: “No, you didn’t, you said it wasn’t a blocker.”

Neither person is the villain here. The engineer was trying to be polite, but the manager heard something different. The gap between what was meant and what was said was the problem.

The engineer meant: “This could seriously hurt us in production. Maybe we should keep an eye on it.”

The manager heard: “No real concern.”

Words like just, small, maybe, perhaps, and not a blocker can soften a message until the point gets lost. The answer is clarity without extra threat. The goal is candor with high standards.

Let’s talk about mistakes without blame

Another common misunderstanding is that psychological safety means a comfort zone where everyone is nice and nobody gets challenged, but that’s just low standards.

We can challenge each other and still expect excellent work.

Psychological safety also makes it easier to talk about mistakes without blame. And every hard conversation teaches people how costly honesty feels here – whether it’s saying, “I made a mistake,” “I think you’re wrong,” “I have bad news,” or “I don’t know.”

“I made a mistake.”

Imagine an engineer saying, “I pushed the wrong config. This incident is on me.”

A very natural response, especially when production is down, is: “How did you let this happen?”

Maybe the person saying it is not trying to blame anyone. They are stressed, customers are affected, and everyone wants to know what happened. But the effect can still be the same: next time, make sure you have a defense before admitting the mistake.

A more useful first response could be: “Thanks for raising it quickly. Let’s stabilize first, then understand what happened, what was in your control, and what we need to change.”

Psychological safety means safe to report but it doesn’t mean free from accountability.

“I think you are wrong.”

This sentence gets more expensive as the other person gets more senior. Disagreeing with a peer is one thing; disagreeing with your manager, principal engineer, architect or VP is another.

When raising a concern, separate the decision from the person who made it. Instead of, “This architecture is wrong,” try, “I’m worried about X because of Y. Can we test that assumption?” For example: “I’m worried about what happens when traffic doubles because the queue becomes a single bottleneck. Can we test that assumption before we commit?”

The concern is still there, but the conversation stays on the work, not the person.

The same applies on the receiving side. Instead of, “We already decided this,” try, “What are you seeing that I’m not?” or “What evidence would make us reconsider?”

Challenge assumptions, decisions and behaviour before you challenge identity.

“I have bad news.”

Psychological safety doesn’t eliminate bad news, just brings it earlier, while it is still a risk rather than an incident.

The earlier we talk about it, the more options we usually have. But engineers often want to bring a solution with the problem, so they wait, investigate a little longer, and try one more thing.

Sometimes the most professional thing you can say is: “We may miss the deadline. I don’t have the solution yet, but you should know now.”

And the response matters too: “I would rather know today than next Friday. What changed? What options do we have?”

If you punish early bad news, you do not get better news – you get later news.

“I don’t know.”

The fourth sentence is quieter. “I don’t know” gets harder to say as your title gets longer.

In engineering, pretending to understand can look professional for about five minutes and then it becomes risk. Because of that you need to ask the “stupid” question before uncertainty becomes hidden risk.

For senior people, model fallibility. When someone senior says they are unsure, they show that not knowing is okay and that asking questions does not reduce your status. That can be as simple as saying, “I’m not sure either,” or “What assumption are we making here?” or “Can someone explain this from first principles?”

When someone senior says that, it becomes easier for everyone else to stop pretending and start asking.

The first 10 seconds matter

Most leaders don’t destroy psychological safety through dramatic intimidation, and most teams do not become silent because of one movie-villain manager. Usually, it is smaller than that: an eye roll or interrupting someone with “No, because…”

The hard part is that your intention is private, but your reaction is observable. You may mean to move the discussion forward quickly and the other person may hear: do not bring this up again.

So the first few seconds matter. When someone challenges your decision, brings bad news, or points out a risk, your instinct may be to explain, defend, or correct. That is human but you do not have to act on it right away.

Instead of asking, “How do I defend my position?”, ask, “What might they be seeing that I’m not?”

Sometimes the simplest thing is: “Tell me more.” You are not agreeing with the person, just trying to understand what they are actually seeing. You can also ask, “What makes you concerned?” or “What would happen if we are wrong?”

The goal is to show curiosity. If they can see you are trying to understand before judging, the next uncomfortable conversation gets a little easier.

What if you are not the leader?

There is a danger in talking about communication skills in this context because it can sound like the answer to a bad culture is simply: employees need to learn how to speak better.

If you are not the leader, you are not responsible for fixing the whole culture yourself. Communication skills help, but they can’t make up for an environment where speaking honestly repeatedly comes at a cost. What you can often do is make your signal easier to hear.

Instead of, “This architecture is bad,” say, “I’m concerned about X because Y. The consequence could be Z.”

You can also calibrate your certainty: “I may be wrong, but…” “I’m about 60% sure that…” “I’ve seen this fail before, although the context was slightly different.”

Uncertainty and assertiveness are not opposites.

Good communication can improve a difficult conversation but it cannot fix a pattern where honesty is repeatedly met with defensiveness or dismissal. That is why it matters to read the pattern, not just one reaction. One defensive response can be a bad moment but repeatedly making candor expensive is a signal about the environment.

Back to Thursday

Now go back to the database migration. It is the same team, the same deadline, and the same engineer with the same concern.

Engineer: “I’m not convinced this migration is safe.”

Manager: “Okay. What are you seeing? How confident are you?”

Engineer: “Maybe 60%.”

That conversation doesn’t magically produce the right technical decision but that’s not the point. Psychological safety doesn’t guarantee the right decision but it gives the team access to more of the information that already exists, and that changes how they handle uncertainty.

You still have hard conversations, and you still make wrong decisions. The difference is that you catch them early enough to respond.

Polite. Open. Assertive.

I like a simple communication profile for these conversations: POA – Polite, Open, Assertive.

Polite means respecting the person. Open means staying curious. Assertive means saying the real thing clearly.

Together, they give you respect, curiosity, and clarity. Polite keeps candor from becoming cruelty. Open keeps it from becoming ego. Assertive keeps it from becoming vague.

It doesn’t create psychological safety by itself, but it gives both sides of a difficult conversation something practical to work with.

Your engineers already see problems, the question is whether you get to hear about them.

Psychological safety makes it easier to raise uncomfortable truths while there is still something you can do about them.

Make honesty less risky than silence.

The post Make Honesty Less Risky Than Silence for Your Engineering Team appeared first on ShiftMag.

When businesses create messaging campaigns on Infobip, they must go through an approval process. They share brand details and campaign information, such as the purpose, target age group, and campaign type, which is checked for consistency and compliance with platform safety rules like WhatsApp or Viber terms of use.

Once a campaign goes live, it’s our job as developers to ensure the messages being sent continue to reflect what was approved.

There’s a whole spectrum of ways a campaign can drift, though. Some changes are obvious safety problems, like phishing or illegal content. Others are more subtle, like using the wrong brand voice, adding marketing language to transactional messages, or changing personalized content outside the approved template. The hardest cases are the ones that look compliant at first but still blur the line.

…and that’s precisely what makes this a tough engineering problem.

Keyword matching isn’t enough to catch message drift

Legitimate traffic and drifting traffic can look very similar, even when they mean different things. Rule-based systems and keyword matching can’t always tell the difference. Detecting drift requires understanding meaning, context, and who is sending the message.

If that’s a little hard to picture in the abstract, here’s an example. It’s rarely this overt in practice, but it makes the shape of the problem obvious.

Imagine you’re a long-time client of an eye clinic. Every year, like clockwork, you receive the same familiar birthday message from the same phone number:

Hello Sam, this is Clermont Family Eyecare, we want to wish you a happy birthday!

Generic, but consistent – a small customer-care touch from a business you trust.

This year, the message arrives slightly altered, from that same number:

Hello Sammy, this is For Your Eyes Only Eye Care, we want to wish you a very happy birthday!

The tone is friendly and the structure looks the same. But the brand name has changed, and the link is new. You tap it, and suddenly you’re on a page with explicit content, not eyecare. In one click, trust is broken – even though the message came from the same number and passed every superficial check.

On average, this system, when completed, is meant to process just short of 300 million SMS and RCS messages daily (crazy, right?). So, the engineering question becomes: how do you detect drift semantically, in production, at that scale, without reasoning about every message individually? Let me walk you through how we got to an answer.

The system needs to be designed from first principles

A few observations about the traffic led to the architecture.

The first is the most obvious: messaging campaigns are repetitive by nature. Messages are templated, sometimes identical within a campaign. At our scale, this immediately points toward grouping messages and reasoning about drift at the group level rather than per message.

There’s an important operational detail here you should know: when businesses create messaging campaigns on our platform, they go through an approval process. They provide brand information and campaign metadata like intent, recipient age or campaign type categorization, which is reviewed for internal consistency and against platform safety guidelines (think WhatsApp or Viber Terms of Use). Once a campaign goes live, it’s our job to ensure the messages being sent continue to reflect what was approved.

You can see that the problem isn’t really about tracking a semantic distribution of campaign messages and watching it shift over time, even though the word drift might suggest that. Historical traffic isn’t what defines drift here. The problem is closer to continuous content validation: preserving the intent stated in the original campaign metadata and looking for new patterns that warrant separate interpretation and potential action.

Even though this framing focuses on novel data, historical traffic still matters. If incoming messages reliably match traffic that has already been evaluated, we can decide in real time whether they’re legitimate or drifting. Messages that match approved intent should be handled immediately, and only unexplained content needs to be isolated for further analysis.

A message entering our system takes one of two paths:

• A real-time classification path, which assigns it to a previously observed group with a known verdict, or
• A deferred discovery path, where it’s held aside to help form a new group.

And there you have it: the gist of the design. The rest of the post is about how we turn that into a production system.

The architecture

In practice, this is a multi-stage microservice pipeline. A message flows through it as follows:

1. Ingestion and reassembly. The orchestrator service ingests incoming traffic and reassembles multi-part messages into their original textual form.
2. Vectorization. To compare messages by intent rather than wording, we map them into a shared semantic space. A dedicated vectorization service takes a batch of messages and returns their embedding vectors. Messages expressing similar intent end up close together in this space, even when the wording differs. That way, grouping by proximity in the embedding space gives us grouping by semantics.
3. Search. The orchestrator queries an Elasticsearch index of known cluster representatives using the message’s embedding. If a match is found, the message is assigned to that cluster, and the cluster signature is promoted to a local cache, so subsequent traffic from the same sender resolves faster. This is the classification path in action.
4. Noise accumulation. If no match is found, the message takes the discovery path: it’s written to a noise index. The name comes from clustering terminology: technically, it’s an “unclassified” index, holding candidate data for future clusters.
5. Clustering. A scheduled job continuously pulls accumulated noise and runs clustering to discover new semantic groups. Clustering is performed on sender level. In practice, we’ve observed that for a sender, message volume is a great primary signal for parameter configuration, and after an obscene amount of per-sender experiments, we did manage to settle on a base of volume-keyed parameter sets. Although not as ubiquitously successful as maybe a best-of-n configs approach would have been, at our scale the compute savings make it the right call.
6. LLM-as-a-judge. Finally, cluster representatives are relayed to an LLM, which produces a verdict for the cluster, not for individual messages. The model also outputs per-message reasoning, which is what makes the verdict actionable downstream.

Walking the eyecare message through the pipeline

Let’s replay the message path from the start with this architecture in mind.

The message arrives at ingestion alongside other traffic from the same eyecare clinic sender. The orchestrator reassembles it and the vectorization service returns its embedding. The orchestrator queries Elasticsearch for known clusters associated with this sender, and let’s imagine that there are several: the standard birthday message, an annual check-up reminder, a glasses-ready-for-pickup notification.

The incoming embedding sits near the birthday cluster, but not near enough to match. The brand name swap and the shift from “Sam” to “Sammy” are subtle on paper, but they move the vector just far enough away.

No match. The message goes to the noise index.

Over the next clustering interval, more messages like it accumulate (you can be pretty sure they would, this reads like a classic sender takeover). The clustering job runs, a new cluster forms around them, and a handful of representatives are stored and passed to the LLM judge.

The judge compares the cluster against the sender’s approved campaign metadata and flags it for brand impersonation. (The changed recipient personalization remains unflagged: on its own, it’s a minor surface variation that doesn’t shift the underlying intent.) Reasoning is attached. A human reviewer picks it up, and the sender is paused before the next batch of spam goes out.

The message arrived through the same number, used the same structure, and would have passed any rule-based check. The only reason the system caught it was because the meaning had shifted.

Clustering is the leverage point

At 300 million messages a day, per-message LLM inference is a non-starter on cost and latency alone. It would also be wasteful, because most incoming traffic is heavily templated or duplicated. Grouping by intent first means we make one judgment instead of a million.

Embeddings group messages together, the cluster amortizes the cost of judging them, and the LLM is reserved for the moment judgment is actually necessary: when something new appears.

Two (in my opinion) interesting design decisions follow from this:

• First, we deliberately don’t let clusters evolve. A cluster represents a stable behavior snapshot, a fixed meaning at a point in time. Letting clusters expand would let them quietly absorb new variations in intent and language, which is exactly what we’re trying to detect. Keeping them fixed forces any notable change in semantics to surface as a new cluster, and makes downstream judgment more coherent because the examples within a cluster genuinely belong together.
• Second, this tightening of group intent allows us to represent each cluster by a few select examples rather than its full membership. This keeps the classification index lean for fast matching, and it gives the next stage, cluster judgment, a focused, reviewable input.

Tradeoffs we accepted

As no architecture is without its flaws, the most interesting one is overclustering.

The system inherits the limitations of its embedding model: embeddings capture intent well, but not every kind of shift. We expected greedy cluster expansion, but in practice the embedding space often treats benign surface variation (like a swapped greeting or different recipient name) as meaningfully different. Legitimate traffic from one campaign gets fragmented across many clusters, which inflates downstream cost and dilutes the signal we care about.

The lesson is that embeddings alone won’t always get us to clean intent groups.

The other two costs are more predictable:

• Discovery latency. The classification path is real-time, but the discovery path isn’t: messages that don’t match an existing cluster sit in the noise index until the next clustering job runs. For genuinely novel drift, this means a delay between first appearance and first judgment. We tune the noise accumulation period to keep this window short for senders above a certain message volume, assuming that low-volume senders will continue to grow with time and graduate to a size above the threshold.
• Cluster proliferation from non-evolving clusters. Forcing semantic shifts to surface as new clusters means a sender with legitimate variation across message templates ends up represented by many clusters. We manage this with cluster aging policies explicitly, but the lean cluster representations also help in that regard, so the memory cost remains bounded.

Model provides a recommendation, not a decision

A verdict on a cluster may end the pipeline, but the case is only closed when a human says it is.

When the LLM judge flags a cluster, the verdict and reasoning go to a human review queue. Reviewers see the cluster representatives, the approved campaign metadata, and the model’s reasoning side by side. They can confirm the verdict, override it, or adjust its severity, and their decision propagates downstream.

From there, action depends on severity: mild drift leads to client notification and guidance, more serious cases can block specific message types and trigger a root cause analysis, and the most severe, spam-grade drift can block senders entirely.

The important thing to note is that the model provides a recommendation, not a decision.

The scariest drift is the one that still looks trusted

A few things became clearer as we built this out than they were going in:

• The most interesting drift is quiet: a message that looks familiar, comes from the same number, but means something slightly different and slowly erodes trust before anyone notices.
• Judging clusters instead of individual messages changes the question. Is this message okay? is narrow and easy to answer wrong. What is this sender doing now? is the question that matters, and a group of similar intent is the right unit to ask it about.

What’s immediately next is concrete, on two fronts.

1. The first is resolving the overclustering issue described earlier: collapsing benign surface variation back into single clusters so the LLM judge isn’t doing redundant work.  Lightweight named entity recognition and a smidge of the old regex to capture emails or phone numbers should do the trick, we hope.
2. The second is scaling: the system currently runs in a subset of regions and handles on the order of tens of millions of messages a week, and the immediate work is extending that toward the 300-million-a-day ceiling it’s designed for.

The eyecare message at the start of this post is just a small example of a bigger problem. What we’re really trying to defend is the meaning of a trusted channel.

And for now, please don’t click weird links in your texts. Sam learned the hard way, and we’re not quite there yet on link-checking.

The post How We Catch Messaging Campaign Drift Before It Spreads appeared first on ShiftMag.

The developer already tested it, so why do you need to test it too?

If you work in tech, you’ve probably heard this sentence a thousand times.

A developer writes the code, runs the checks, all the tests pass, and the feature works.

So why do companies still hire Quality Assurance (QA) engineers? Are we just here to double-check developers’ work, or is there a deeper purpose?

The short answer is: no.

The longer answer is that there’s a fundamental difference between writing code and creating a great customer experience. The value of QA lies in perspective, risk awareness, and understanding how people actually use a product.

The builder

To build a truly great product, you need two very different mindsets: the builder and the chaos maker.

A developer’s brain is wired for creation, logic, and structure. While QA starts the day with a finished product, a developer starts with a blank screen. Their job is to create something out of nothing and assemble the many moving parts of a complex system into something that works exactly as intended.

Developers are responsible for creating the magic. To build even a single feature, they have to master a labyrinth of tools, frameworks, databases, and configurations. They carry the responsibility of making sure the architecture is secure, the system can handle heavy load, and the new code integrates cleanly with millions of lines of existing logic. Their work is highly complex and demands deep understanding, precision, and sustained focus. They are the architects of the digital world.

Because their primary mission is building and stabilizing the system, their testing mindset naturally reflects that. When developers test their code, they are checking whether their blueprint holds up. They focus on the happy path, along with a few common scenarios, in a controlled environment, using standard data to confirm that the core functionality works as intended. Their attention is on making sure the foundation is solid.

Think of a developer as an architect and engineer combined. They design a beautiful, highly complex door, check that the handle turns smoothly, inspect the hinges, and confirm that it opens perfectly.

Now what? Aren’t developers already responsible for the whole thing?

The chaos maker

While developers are incredibly skilled at what they do, they cannot be expected to carry the entire burden of the user experience alone. That is where QA steps in – to relieve some of that pressure and help refine the product for the real world.

If a developer’s brain is wired for creation, a QA’s brain is wired for exploration and chaos. The goal is not to prove that the software works, but to discover what happens when the scenario is not perfect. We test unusual cases, unpredictable behavior, and the edge cases that real users may actually trigger. We test for the messy real world.

Because of this, QA goes far beyond engineering. The work calls for empathy, human psychology, and an eye for the emotional, reliability, and invisible factors that make customers want to use a product.

As a QA, I’m constantly stepping into the user’s shoes and asking myself a very different set of questions:

• Is this accessible and intuitive to use?
• What emotion does this product create for me – frustration or delight?
• Would I still enjoy using it tomorrow, or would it already be starting to annoy me?

What happens in the real world

To make this point come to life, let me share a personal experience.

I’ve seen that even a large team of skilled developers means every single scenario is not automatically covered. Tight deadlines, heavy workloads, and the sheer mental focus required to build complex architecture make it nearly impossible for any single perspective to catch everything. Once a feature is done with implementation, it’s QA’s turn to step in, look beyond the happy path, and hunt down every possible edge case.

The “blank space” API trap

For instance, when new feature was implemented, it was time to test it. My team and I ran automated tests for our Voice APIs.

During testing, I decided to try something slightly unusual: I sent a call using a sender name that contained a space. Immediately, the calls started failing. This turned out to be a critical issue. If a real client wanted to do something as simple and common as putting a space in their sender name (e.g., “Global Bank” instead of “GlobalBank”), their entire voice campaign would fail. We only discovered this flaw because we had an automated QA project specifically designed to stress-test the exact parameters our clients use daily, which even shows the importance of regression tests and their benefits before the release.

The edge case that shouldn’t break the system

Another instance happened while I was testing the delivery reports for our system. The documentation stated that the record limit field was limited to 1 to 1000. My QA brain asked, “What happens if a user accidentally types 0?” When I entered 0, the system crashed and returned a 500 Internal Server Error.

Technically, it was just a wrong number. But from a customer experience standpoint, this is a big deal. A 500 error tells the client, “Your servers are broken,” which causes panic, especially during real-time actions. It should have gracefully returned a 400 Bad Request, which politely tells the user, “Oops, your input was invalid, please try again.”

Logically, it makes no sense for a user to request 0 records. But it is our duty as QA to make sure that even when a user does something illogical, the system handles it elegantly instead of catching on fire.

The trapped user

The chaos isn’t always in the backend, either. Recently, while performing manual UI testing, I found a scenario where a user could fill out an entire complex form, only to find that the “Save” button remained permanently disabled because of a tiny edge-case conflict in the UI logic. Imagine the frustration of spending ten minutes typing out intricate details, only to be trapped at the finish line and unable to save the work.

All of these instances prove why additional testing is necessary. It is incredibly easy for crucial edge cases to slip through when a team’s primary focus is building complex core functionality. No one should set a limit to 0. A sender name shouldn’t break an API. But humans aren’t strictly logical, and neither is the real world. That is why QA exists.

QA and devs are two halves of the same brain

Ultimately, this story comes back to the same point: success. Despite our different approaches, both teams are chasing the same finish line. We come together, we collaborate, and we bring an idea to life in the best possible way.

Both developers and QA want the same thing: to deliver a product that is simple, easy to use, unique, and highly functional. We want to build something so reliable and intuitive that the end user does not even have to think about how it works under the hood – it just does.

Because my job involves finding edge cases, logging bugs, and occasionally breaking a developer’s hard work, it is easy for outsiders to assume that developers and QA are at war. But that is far from the truth. We are not enemies; we are an essential partnership. We are a continuous feedback loop where logic meets empathy.

Just like a human being cannot function with only half a brain, a truly great product cannot survive in the wild with only one mindset. It needs the builder’s solid foundation, and it needs the chaos maker’s reality check.

The developer brain ensures the software is built right. The QA brain ensures we’re building the right software for the customer. Together we create the product that customers love.

Before you go…

Next time someone asks, “The developer already tested it, why do you need to test it too?” you’ll know exactly what to say:

Code lives on servers, but customer experience lives in the real world, and QA is the bridge between the two.

The post QA Is What Happens When Software Meets Reality appeared first on ShiftMag.

We help companies put AI to work in real environments.

We spoke with him about what it takes to move AI agents into production, how developers can make them reliable and secure, and which skills will matter most as AI changes software development.

Production starts with evals

As Luis hinted in the intro, Forward Deploy Engineers (FDE) help companies turn AI demos into real products. So we asked him what developers most often underestimate when they make that jump.

According to him, the real challenge starts when a demo reaches production. A demo can work well in controlled conditions, but production forces teams to understand exactly what works, what does not, and why failures happen.

That is where eval-driven (Evaluation-driven) development becomes important. He says OpenAI uses it extensively in its engagements to track and measure different workflows and tasks:

We have a scientific approach to measure the performance of the models doing a task. When, basically, you clear those benchmarks, those evals, you are good to go to production in a really good and safe way.

The deployment gap is the real challenge

We wanted to understand what makes an AI agent reliable enough for production, rather than something that only works well in a controlled environment.

Luis said it all comes down to “cracking the evals.” As he explained, today’s models already have impressive intelligence, but putting them to work in a company’s own environment requires giving them access to the company’s data, procedures, and tools:

You know, those three things, we kind of call the deployment gap, right? And this is what we’re trying to close as FD, basically.

Agents went from six-minute to 16-hour tasks

But what happens when an agent works on a task for hours rather than minutes? How should developers think about errors, memory, context, and human approval as those workflows become longer?

Luis answered with an example from his own experience. While checking the latest data from METR’s long time horizon benchmark, he was surprised by how quickly agents had improved. In just over three years, they went from handling tasks that would take a human around six minutes to tasks that would take around 16 hours.

That represents a massive jump, he said:

On one hand, you have the models getting better at long-term coherency and persistence, but on the other hand, you also have the harnesses improving a lot. Back in the days, a developer needed to care about context management, memory creation, and all of that. Right now, as I say, with harnesses such as Codex, all that is gone.

For Luis, that progress comes from combining more capable models with more powerful harnesses. Together, they allow engineers and developers to run increasingly complex workflows over much longer periods of time.

Agents should follow the same access rules as employees

When it comes to security, we wanted to understand the safest way to connect AI agents to company data without creating new permission and access risks.

Luis explained it through a familiar workplace example. When a company onboards someone who works in finance, it gives that person access to financial data, but not to HR data they do not need.

He argues that companies can apply the same principle to AI agents, using the same access controls and permission rules they already use for employees:

In the same way that when you onboard an employee into your company, and let’s say he or she is working in finance, you only give access to the finance data. You don’t give access to the HR data, right? So, in the same way, those same principles, those same ACLs, can also apply to AI agents, right?

For him, the principle comes down to giving agents access only to what they require to perform their tasks:

So, in my view, it’s about having the roles, the permissions, and the setup for the agents to only see what they need to see for completing their workflow, basically.

Evals are fundamental to AI development

Finally, we asked Luis what developers should test and measure before releasing an AI agent to real users.

Once again, he came back to evals. He said he keeps returning to the concept because it remains critical and fundamental to AI development:

I think we go back to evals once again, right? And I keep coming back to that concept because it’s absolutely critical and fundamental for AI development, right? If you’re able to put together that representative eval set containing easy cases, medium cases, and also edge cases, and you’re able to highly lean on your skills, on your guardrails, on the prompts you send to the model, that will give you the guarantees that when you deploy to production, nothing will break.

He added that evals also give developers a way to test for regressions when they upgrade to a new model and make sure the change does not break existing workflows:

Furthermore, when you upgrade your model to a new one, you have a way to test regressions and make sure that that upgrade won’t break anything.

System design taste will never go away

Before joining OpenAI, Luis also worked at Google, so over the years he has seen plenty of developers, roles, and career dilemmas up close. That is why, to close the interview, we asked him which skills developers should focus on today to stay relevant in the age of AI agents.

He believes that, in this new era, engineers will no longer create code as their primary output, because code has become very cheap to produce.

Instead, he sees much more value in everything developers build around the model:

I think now, in this new era we’re entering, the primary thing that engineers are creating is not code anymore. Code is very cheap to produce. It’s all about building the system around it, so the system of intent, the guardrails, the constraints for the agent to work reliably, right? So I think creating those harnesses around the model will be absolutely critical.

Luis ended our interview with a simple but sharp statement:

Plus, I always say, like, system design taste will never go away.

The post OpenAI’s Luis Velasco: Code is very cheap to produce, but the system around it isn’t appeared first on ShiftMag.