We in the Test Engineering team at Stuart are very excited to share how we use Cypress as a testing framework on all our front-end applications.

There are several reasons why we want to do this, but these are our top two:

• Today, UI tests are written in Java with Selenium and maintained by QA, which inhibits collaboration with other FE engineers as it’s not their main stack.
• Stubbing network requests/responses are handled with an external mockserver.

How does Cypress solve these issues?

• We can implement the tests with Typescript, the stack that our FE engineers use daily
• Stubbing network requests/responses are handled internally.

Even though team collaboration is the key advantage of this change, there are other architectural advantages to Cypress — keep reading as we dig deeper into this stubbing topic:

In Selenium, test cases use the WebDriver component to communicate with the Browser Driver, which will then interact with the actual browser to execute the commands. Communications between all the components throughout this route are two-way, so that information can seamlessly flow back to the WebDriver from the actual browser. Likewise, developers will need different browser drivers for different types of browsers. Simply stated, Selenium runs outside the browser and executes the commands via the network.

In contrast, Cypress executes test cases directly inside the browser. A server process that powers Cypress makes it possible for Cypress to execute code in the same run loop as the application. Both Cypress and the server process constantly communicate with each other to perform tasks, enabling Cypress to respond to application events in real-time. This communication also allows Cypress to interact with OS components for tasks outside the browser, such as taking screenshots.

Cool, right?

To stub, we use cy.intercept() command.

As our FE application consumes a GraphQL API, we need to be able to stub it seamlessly. Here we see our first limitation.

Since GraphQL exposes a single resource with different queries/mutations, stubbing becomes more challenging. We will have to implement a stub for that exact resource.

cy.intercept({ method: "POST", url: "/gql" }, (req) => {
  req.reply({fixture: "aResponseBody.json",});
});

The first attempt to stub a different query was to repeat that exact stub but change the response fixture.

cy.intercept({ method: "POST", url: "/gql" }, (req) => {
  req.reply({fixture: "anotherResponseBody.json",});
});

However, since the matching criteria {method, url} are the same for requests, the first one will not be considered anymore (as per Cypress implementation, the last stub stays).

To solve this and to differentiate requests, we can use operationName, which is unique. The previous code will evolve as follows:

cy.intercept({ method: "POST", url: "/gql" }, (req) => {
  if (req.body.operationName.includes("anOperationName")) {
    req.reply({ fixture: "aResponseBody.json" });
  }
  if (req.body.operationName.includes("anotherOperationName")) {
    req.reply({ fixture: "anotherResponseBody" });
  }
})

If we want to deal with only a few requests, this approach is more than valid, but our Dashboard application, for example, produces many requests. This means that simply for logging, the code looks like:

cy.intercept({ method: "POST", url: "/gql" }, (req) => {
  if (req.body.operationName.includes("GetClient")) {
    req.reply({ fixture: "/gql/GetClient/corporateClient.json" });
  }
  if (req.body.operationName.includes("GetZonePackageTypes")) {
    req.reply({ fixture: "/gql/getZonePackageTypes.json" });
  }
  if (req.body.operationName.includes("GetPackageSizes")) {
    req.reply({ fixture: "/gql/getPackageSizes.json" });
  }
  if (req.body.operationName.includes("GetScheduledSlots")) {
    req.reply({ fixture: "/gql/getScheduledSlots.json" });
  }
  if (req.body.operationName.includes("GetTransportTypes")) {
    req.reply({ fixture: "/gql/getTransportTypes.json" });
  }
  if (req.body.operationName.includes("GetClosedZones")) {
    req.reply({ fixture: "/gql/getClosedZones.json" });
  }
  if (req.body.operationName.includes("GetJobs")) {
    req.reply({ fixture: "/gql/getJobs.json" });
  }
  if (req.body.operationName.includes("GetTermsAndConditions")) {
    req.reply({ fixture: "/gql/getTermsAndConditions.json" });
  }
  if (req.body.operationName.includes("GetSavedPlaces")) {
    req.reply({ fixture: "/gql/getSavedPlaces.json" });
  }
  if (req.body.operationName.includes("GetZones")) {
    req.reply({ fixture: "/gql/getZones.json" });
  }
  if (req.body.operationName.includes("getClientTier")) {
    req.reply({ fixture: "/gql/getClientTier.json" });
  }
})

And the problem can (and will) snowball as soon as we want to either add a new stub or modify an existing one (Remember: we will need to execute the stub again with the new value).

Commands to the rescue!

We’ve have created a command that will store in a key-value pair the operation name with their corresponding responses that have already been registered:

const registeredResponses = new Map<string, string>();

Cypress.Commands.add(
  "interceptGql",
  (operationName: string, response: string) => {
    registeredResponses.set(operationName, response);
    cy.intercept({ method: "POST", url: "/gql" }, (req) => {
      if (registeredResponses.has(operationName)) {
        req.reply({
          fixture: `/gql/${registeredResponses.get(
            req.body.operationName
          )}.json`,
        });
      }
    });
  }
);

This way, every time we want to add or overwrite an already stubbed response, we don’t need to mock it all at once; it will internally do so.

Imagine we want to change a stub during the test execution for the package sizes we have. The new implementation would look like this:

describe("/new screen tests", () => {
  it("should stub different package sizes seamlessly", () => {
  cy.interceptGql("GetPackageSizes", "getPackageSizes");
  // Do something
  // Do something else
  // I write some more lines just in case Elon Musk acquires Stuart
  // I don't want to be fired
  // And more lines of code means better code
  cy.interceptGql("GetPackageSizes", "getAnotherZonePackageSizes");
  });
});

This is how we overcame this limitation. Do you have any other solution that you would like to share? Please share your thoughts by adding your comments!

Looking to join the Stuart team? Take a look at our open roles here.

GraphQL Stubbing Using Cypress was originally published in Stuart Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Chapter 1: Early Days

It was early 2020, the beginning of the Covid pandemic. As in many European countries, France was locked down except for essential services. That gave me plenty of time to catch up on my favourite series — but a few weeks in, I realised the curve wasn’t getting any flatter!

Despite the odds, Stuart was growing exceptionally. In fact, our growth was exponential, which introduced a couple of issues, one of them being the question of scaling our capability to act. I was part of a team of just three BI analysts about to face a rather big challenge: How do we quickly empower analysts to generate insights from our data and fast?

In this upcoming series, I’ll take you through our transformational journey (pun intended) of how we scaled our teams, enabling them to do their (analytical) jobs autonomously without relying on my team and me. I’ll cover this topic in the following chapters:

Chapter 1: Early Days

Chapter 2: Drawing Board

Chapter 3: New Beginnings

The very start of our journey

Like many other start-ups, we initially opted for a fleshy open-source data stack. Our team of three had the basics covered; we had a set of pipelines running in Airflow with several custom-made operators that allowed us to ingest data from APIs, do SQL-based transformations, and report on the results. In theory, the stack worked and worked for a while, but as we entered the turbulent growth phase, we — the BI team — became the bottleneck.

Enter the modern data stack phase

We were confident using Redshift as our data warehouse, but we needed the rest of the stack to scale. Quickly, we decomposed our tooling behind the Extract (E), Load (L) and Transform (T) components. For the E & L, we built an in-house Pipeline Framework in combination with Fivetran, which I’ll cover in a separate post. Crucially, however, we started looking for alternatives to implement the T in ELT.

Quickly, dbt came out on top as a potential candidate for transforming data. We saw the huge potential of embedding jinja in SQL and immediately fell in love with its dependency management options. We started trialling the tool as an internal PoC project, and shortly after, we accepted it into our tool stack.

Stuart kept growing exponentially, and it was the right time to put dbt to the test. We had an ambitious goal of migrating all our legacy pipelines to dbt. This meant not only doing the physical migration of our pipelines but also onboarding new analysts outside our team to dbt. This would enable us to gain momentum as new models would all be built under the same framework.

We were already doing development on a few pipelines and had even deployed some processes to production. After going through this process for a few cycles, we realised that there were friction points that would stump the adoption of dbt as a company-wide tool.

Shoot for the moon. Even if you miss, you’ll land among the stars.

As lazy developers, we love to build custom scripts to reduce the cognitive load of remembering commonly used patterns, a.k.a. shortcuts. In our case, we’d built a small collection of recipes, which worked well for our usage (local testing, grouping common models, adding runtime parameters, etc.).

#!/bin/zsh

#usage: zsh backfill_loop.zsh -s 2021-02-21 -e 2021-03-01 -m dbt_staging.my_great_model > my_great_model.log &
while getopts s:e:m: flag
do
    case "${flag}" in
        s) start_date=${OPTARG};;
        e) end_date=${OPTARG};;
        m) dbt_model=${OPTARG};;
    esac
done

d=$start_date
while [[ $d < $end_date ]]; do
    echo $d
    dbt run --target prod --models $dbt_model --vars '{"start_date": '$d'}'
    d=$(date -j -v +1d -f "%Y-%m-%d" "$d" +%Y-%m-%d)
done

Example of one of our scripts for backfilling models one day at a time (yes, some models aren’t able to fast forward to the latest date, but that will be the discussion of a later post…)

On that same note, we set out to build a custom script that would package these and other common patterns and give a unified user interface to other users in the company. We spent a good few weeks working out the kinks and were set to showcase this to a selected subset of analysts. The big day came. We had two days to conduct a workshop introducing dbt to our beta users. We had our pot of coffee and started onboarding users — only to face a setback: the challenge of local machines.

1. Most of our analysts work on Apple machines, and no two analysts have the same setup.
2. Out of all the installations we performed, three-quarters went without a blip.
3. About 20% needed hands-on support to manage different settings (virtual environments, docker versions, shell differences, etc.).
4. About 5% had hard-to-resolve issues (including one user with a different hardware architecture than the rest).

In the next chapter, I’ll cover the lesson learnt and how it inspired us to go back to the drawing board.

Stay Tuned!

Stuart’s Data Journey: How Gustavo and the BI engineering team polished the T in ELT was originally published in Stuart Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Initial idea

It’s been a long time since we started working on the creation of an app for our clients that will give them a huge improvement on some of the management systems we use in Stuart.

Our frontend team was in the front line of all these changes and they decided to start a new project creating a specific repository for this development. They focused their efforts on creating an application based on React that was capable of managing a Single Sign On (SSO) system of login giving the users access to multiple tools inside. In our case, the SSO system was a Keycloak instance. Each of these tools has its own different permissions based on the client. So, depending on the permissions a user has on the SSO system, they will be able to see some tools or others. And this concept is key to understanding the problem we faced.

Since different users had access to different tools inside the app, the traditional approach of having one token to authenticate against the API was a liability as this would give the users access to graphQL queries and mutations that they should not be able to access. To go around this, each tool was set up to be an OAuth client requiring users to generate a new token as they navigated between tools. The Frontend server handled this by matching the current path the user was on to a tool and then redirecting to Keycloak with the appropriate client ID. Keycloak redirected the user back to the application with an authentication code which was then used in combination with the client secret to request access and refresh token.

The idea to start using Cypress was discussed for a long time between the QA and FE teams. This new application required mocking graphQL API calls and converting with E2E tests a lot of dynamic changes on the interface that seemed perfect to introduce it.

The problems we found

The first important step after the installation was related to our SSO login system. Any test on the application should bypass this. We were interested in testing the login in isolation, but we don’t want to perform the login on each test as it will be very time-consuming. After some research, we found a possible valid solution by using a combination of approaches.

We found a third-party library created specifically for mocking Keycloak called cypress-keycloak-commands. It’s a simple and easy-to-use tool that perfectly does what it was made for. We also needed to define a cookie with our client id and secret credentials and we set it directly using Cypress due to our technical requirements.

It was nice to run the first test after implementing this approach. It worked! The login system was bypassed and we entered the main menu of our project. But… Here we faced THE PROBLEM.

After logging in with the mock generated by the library and reaching the internal list of available tools, we found that we were unable to mock the second permissions asking the SSO using that library because of what we told about how our app works. Our Cypress server couldn’t access our FE Server which was the one doing the request asking for permissions on the SSO, so there was no possible mock for this. Neither directly in Cypress nor using the external library.

The real solution. Frontend team to the rescue!

It became very clear that we needed to somehow mock the SSO authentication flow on localhost but we also did not want to change any production code to cater for tests. Pointing the application to a local server instead of Keycloak and getting a mock token would not be a problem as all Keycloak URLs were set as environment variables and all data would be mocked during tests.

In the end, the solution was inspired by how our development environment is set up. During development mode there are two express servers running, one is the same server that runs on production and the other one handles hot reloading and serving assets.

After some brainstorming, it was decided to create a similar script to start up two express servers for cypress. One server handles server-side rendering and serves the app the same as production and the other one has two routes added to it. One route handled redirecting back to the main server with a mock authentication code and the other route responded with a mock access and refresh token.

app.get("/openid-connect/auth", (req, res) => {
  const { redirect_uri: redirectUri, state } = req.query;

  res.redirect(
    `${redirectUri}?session_state=${sessionState}&code=${code}&state=${state}`
  );
});

app.post("/openid-connect/token", (req, res) => {
  res.status(200).json(tokenPayload);
});

Final test

With those changes in mind, we just needed to adapt our CI scripts on Jenkins so the frontend server app was configured correctly to run Cypress tests as explained. And it all started working like a charm!

After creating some tests, we were able to login into the app and see all the tools listed but we were also capable of accessing any tool and, using the intercept method and some fixtures for mocking, we just got the information that we wanted to test!

Special thanks to Tonio Buttigieg for all the time dedicated to the mocking system and writing this article.

How to mock an SSO Keycloak React app for Cypress testing was originally published in Stuart Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article I share the experience of my first three months at Stuart as a Data Product Manager for our internal data platform; what it means to sit at this powerful intersection between data and product; and why it’s critical for a modern data platform team like ours to “think product” in truly delivering business value. I’ll draw on the challenges faced when looking at a data platform through a product lens, redefining it from simply a “tech stack” to what it really is: a product that delivers value to hundreds of business users day in day out. My hope is that this will serve as a guide to data platform teams at any stage of becoming more product-led while contributing to the emerging space of data products more generally.

The uncomfortable questions

Over recent years most companies have adopted a data platform as a way to manage how data is ingested, stored, transformed and accessed at scale. The end goal is typically to support the business in making fast and reliable data-driven decisions, and here at Stuart we’re no different.

In the face of exploding volume and complexity of data which needs no further explanation, the focus of data platform teams have understandably been concentrated on stability and reliability at scale. For example, ensuring the smooth running of business-critical pipelines, minimising downtime or optimising query efficiency.

While companies large and small have made considerable gains in building a scalable and sustainable architecture, we’re left with the uncomfortable questions: is what we’re doing truly providing value? Do we really know who our users are and understand their needs? If so, can they generate insights in a fast and reliable way? As long as users don’t complain and pipelines don’t fail, does that mean all is well? For all our investment in data, are we seeing the return?

These questions that touch on the concept of value are particularly difficult for an internal platform that doesn’t sell to or make money from our users. Moreover, the added complexity of quantifying the value that data brings forces us to consider alternative definitions of value:

• Top-down: How does the data platform align with and contribute to company goals?
• Bottom-up: Would users choose to use the data platform, and is it easy to use?

While the importance of the former cannot be underestimated and should play a critical part in goal setting and prioritisation, this is often a question of modifying process. Meanwhile, the latter is a more complex topic, and especially so for a data platform intended for business (i.e. human) decision-making. To better understand users requires more than a change in process — we need a change of mindset.

When the data platform is viewed purely from a technical lens, we put ourselves at risk of forgetting our users and their needs, building solutions before we’ve understood the problem and further increasing the data/value gap.

These questions are particularly important to ask now, when the growing discussion around data mesh suggests we’re on the verge of a radical shift in how business interacts with and governs data. This only further increases the need to proactively and continuously understand our users and their evolving needs.

The data platform: More than a tech stack

Much in the same way that experimentation platforms, algorithms, and even the data itself are being increasingly considered products, we turn our attention to the data platform. For simplicity, consider the data platform to refer to all the tech and tools that collectively serve the business to make data-driven decisions.

If the previous section wasn’t enough to convince us, there is one concept that helps sum up the need for product thinking for a data platform more than anything:

Accessing and working with data is a user experience.

In the same way we use applications for almost everything in our personal lives from measuring our heart rates to finding a new house, the experience of accessing data to make decisions should be no different. But the concept of UX is rarely heard among data teams. This is hardly surprising if the data platform is viewed entirely as a technical problem and resourced accordingly; but perhaps added to this is the idea that as user-facing data tools are frequently bought and often come with a UI, that these UIs automatically “take care” of our user needs, i.e our job is done.

But a good UI cannot replace the need for a good UX. To think that in most modern organisations, hundreds or even thousands of users are interacting with data on a daily basis, the idea of not continuously investing in discovering users’ needs, pain points and desires, of not using this user context to shape our roadmap for success, in other words, not thinking of our data platform as a product should concern anyone investing in it in the first place.

Here at Stuart the Data team is fortunate to sit alongside a strong Product team whose philosophy is centred on continuous discovery: identifying opportunities in the form of user needs, pain points and desires and using this to guide our vision, strategy and goals. This means that as well as building products that make sense both from a technical standpoint and aligned with company goals, we’re also strongly led by our users. This approach means we’re more likely to deliver a data platform that users love, and less likely to jump into solutions that ultimately only we in the platform teams love.

Data Platform as a Product: First Principles

Until now we’ve mostly discussed the why. It’s now time to move from theory to reality and put this into practice: the how. The complex make-up of the data platform doesn’t make it easy to define as a product by any means. Here we present our “first principles” approach to this, which focused on three broad areas:

• First, at the end of almost all of our data efforts is a user experience. But users are not one and the same; they are diverse and evolving, fast. For this reason we should start by considering our different user personas (“who are our users?”).
• Second, we also need to consider the motives and reasons behind needing a data platform in the first place. We should at least give consideration to user intents (“what do our users want to do with data?”), broadly similar to use cases but at a much higher level.
• Finally, for the multifaceted data platform we need to make life easy for both our users and ourselves by breaking it down into easily identifiable product components (“how do we present our data platform to our users?”). These components are accessed by users through interfaces, typically representing third party or internal tools.

Together with other data leads and interviews with key business stakeholders we investigated each of these areas one by one. To ensure it didn’t become a one-off exercise we made them as visible as possible internally, e.g as part of onboarding a new Data or BI Engineer. Doing this also incentivises us to keep them relevant and up to date.

User Personas

If we want to shift to a more user-centric approach, understanding who our different users are feels like a logical starting point.

Approach

• If our goal is to provide constant reminders to our data platform teams that there’s a user at the end of our efforts, we need to reflect reality as much as possible. We used names, photos and real quotes from users.
• We avoided creating personas directly along job titles or business departments, as it often obfuscates the nuances between teams and is less flexible to change. e.g. a data analyst in one team could have a very different profile to a data analyst in another.
• We tried to at least get a rough number of the size of each persona through looking at usage logs etc. This helps give more meaning and context to a persona.

Outcome

User Intents

Why would anyone make use of a data platform in the first place? What are the key actions our combined user personas should expect to be able to perform autonomously? These were the questions we asked during this step.

Approach

• We placed the word autonomously front and centre of this discussion. As we scale we must strive to reduce the dependency on the data platform teams so users can act fast, albeit in an aligned and secure manner.
• With this in mind, we took an aspirational approach in considering both what the user can and cannot do with data today by themselves.
• We avoided listing all the specific use cases at this early stage due to the sheer volume of these, which could easily run into the several hundreds. Instead, we focused on summarising high-level intents as described by a single action (verb). As we mature and evolve we can add (and remove) nodes to this “tree” of user intents, possibly then going down to specific use cases.
• As soon as broad categories of user needs emerged we began to group these. For example, the “need right now” vs. “need for future”. The first represents ad-hoc needs for data answers immediately; while the latter represents intents to build or automate in order to avoid repetition in future.

Outcome

Product Components

With a better overview of the users and motives for using a data platform, we turn our focus inwards and ask what our data platform offers. Similar to how market-facing products sit on a shelf or under the “Our Products” dropdown on a website, how do we represent our data platform as a portfolio of products?

Approach

• We only considered business user-facing aspects of the platform as product components. The goal here is for users to get an easy answer to “what’s in it for me?”
• We divided the platform into components that make sense to our combined user personas. This is an important point, because the purpose of this exercise is not to end with another technical architecture diagram. Components should be named in a way that clearly communicates their purpose.
• Just as for user intents, we shouldn’t be bound only by the current or “live” platform components. Even if there are some that are planned or still under consideration there is no reason not to show them among the components.

Outcome

One of the advantages of dividing our platform in this way is that we can build a set of KPIs for each product component, which would then form the basis of future OKRs:

A Note about Interfaces

At a later stage, we mapped the more familiar user interfaces (usually third party tools) to each component, so that users can clearly navigate to the specific tool(s) for accessing each component. We felt it important to distinguish between components and interfaces for these reasons:

• Interfaces change. If we were to replace our primary visualisation interface from Tool A to Tool B, the interface has changed but the component remains unchanged.
• Interfaces are increasingly multi-purpose. Many third party tools are expanding beyond their core offering and some even considered platforms in themselves. This means their purpose isn’t always obvious to users; components help to clarify this. This also means there isn’t a one-to-one mapping of interface to component. For this reason, we introduce the idea of primary and secondary interfaces. The primary interface is where the user should typically be directed. Failing that, the secondary interface may be partially able to serve as that product component.

Joining the dots

By taking a step back and linking the use cases, user personas, components this enables us to:

• Look at our data platform as a manageable set of product components we can measure more easily and build OKRs around.
• Identify gaps in our offering, where important user intents are not linked to a live product component. This is particularly useful as a starting point for building a longer-term strategy.
• Decide how we best organise our teams around common KPIs.
• How we structure our roadmap and communications to the business on what we’re delivering, to who and why.

Where we’re heading next

We’ve introduced the concept of a data platform as a product, moving beyond simply viewing it as a tech stack. We did this by identifying who our users are, their intents and what our product offering is that can meet those needs.

As we’ve witnessed at Stuart most modern data platforms are complex and multifaceted, so the simple act of putting these on paper and visualising our platform as a collection of products that deliver value to our users is already a meaningful first step.

Our work is by no means done here — we are only at the start of our journey to becoming more product-led. These practices and principles will form a solid foundation for what comes next: discovering opportunities on an ongoing basis. Opportunities that will play a key role in building our strategy and OKRs for the coming year, becoming a data platform that continuously brings value to our users and to our business.

Main reading sources:

• Continuous Discovery Habits, Teresa Torres
• Inspired: How to Create Tech Products Customers Love, Marty Cagan
• How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh, Zhamak Dehghani
• How to Build your Data Platform like a Product, Barr Moses

How we’re building our data platform as a product was originally published in Stuart Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Discover the strategies of Stuart’s Head of Agile for developing high-performing teams.

The topic of High Performance Agile teams is one that comes up frequently in conversations in Agile circles. There is a lot of confusion around it, I think in part because it’s difficult to come to an agreement of how we define a high performing team and what the benefits are. At Stuart we recently had a discussion around High Performing Agile teams in our Agile Community of Practice (CoP). We wanted to gain a common understanding of what we mean by it, what are the benefits of high performing teams, what are some patterns and anti-patterns, and most importantly, what are some tips and tricks for developing high performing teams. So, we put our heads together and came up with some thoughts that I am now sharing as part of this blog post.

This is the second article in a series about creating and developing a high performance Agile team. If you missed Part 1, check it out here.

How can you help your team develop through the Tuckman Model?

7 tips and tricks to help teams progress from one stage to the next.

Set a clear purpose and mission for the team and revisit it throughout the process. It’s important to have the answers to the following questions and make sure they are clearly understood by the team.

• Why does the team exist?
• Who are the personas that represent the customers of the team?
• Who are the stakeholders and business sponsors and what are their expectations?
• What values matter to the stakeholders and to the team?
• What problem(s) will you solve and why do you need to solve it?
• What opportunities are you trying to capture?
• What value will the team bring to the company?

Set ground rules and make sure they are followed. For this we found it useful to establish working agreements with the team.

• What behavior is acceptable? What is not?
• How will the team interact?
• How often will the team meet?
• What tools will the team use?
• How will the team organise and remain open and transparent?

Share leadership across the team and create an environment where everyone is empowered to lead.

• Everyone on the team should be empowered to lead the team, whether it be through a process, a discussion to solve a problem, or an Agile Ceremony such as backlog refinement or a cycle review.
• Every team should have a facilitator for their Agile Ceremonies but (and this is important) it doesn’t need to be the same person all the time. In fact, it SHOULDN’T be the same person every time.
• Encourage people to be Servant Leaders and coach and guide them to become one.
• Choose the most qualified person to lead a particular discussion or exercise.
• People should be accountable for their solutions, which means they need to be empowered to identify and own them.

Don’t try to avoid conflicts. They are normal and can be healthy.

• The benefit of working in a team is that you have access to diverse experiences, skills and opinions that aren’t possible alone.
• When members disagree about something, listen to each side. But don’t take one. Search for common ground. Disagreements are often a result of people having different perspectives and the best solution is one that combines multiple perspectives.
• When conflicts are resolved correctly and in a positive way, it can improve team dynamics and bring the team closer together.

Coach team members on how to become active listeners. Active listening is a technique that keeps you engaged with your team in a positive way. It allows the speaker to be heard and keeps the listener open to what is being said.

• Each person in your group holds some value, otherwise they wouldn’t be there, right? Remind your team to listen and to value each person’s insight.
• Create a safe environment that is open and non-judgmental to encourage everyone to voice their opinion.
• Encourage team members to actively seek the opinions of fellow colleagues on the team from different disciplines. Getting their perspective could prove invaluable.

Be open to change and to feedback–in other words, strive to continuously improve.

• Encourage the team to talk about what they are doing well. What do they need to improve?
• Provide a space where the team can retrospect on how they are doing and identify things they can improve on. Commit to those things as a team.
• Do not blame individuals for the shortcomings of a team. Work with the team on making things better.
• Accept that the team will make mistakes, celebrate them, learn from them and make improvements based on them.
• Identify solutions for the improvement opportunities identified by the team. Do not turn your retrospective into a meeting where everyone complains and nothing positive comes out of it.
• Tell teams what they are doing right as well as what they need to improve.

There is no “I” in “team”; everyone contributes to the team’s success.

• Everyone plays a part and contributes to the team’s success.
• When one person on the team is struggling, the entire team struggles. We should work to support each other.
• It is important to instill this sense of responsibility in the team.
• Make sure the team has the right tools to collaborate and that they are being used correctly.
• If you are blocked by someone else’s work, instead of waiting for them to finish so you can start your work, try collaborating with them to complete the work faster so you are working on a common goal.

Why are team dynamics so important? Why should you care about them? The answer is that it’s because team dynamics directly impact team performance. Think about it for a minute. Intrinsically, we know it’s true. It’s true in almost any industry. In sports, for example, there is a lot of information out there about how team dynamics contribute to their success…. or failure. An Athlete Assessment’s article entitled Sports Team Chemistry and Dynamics in Sport talks about this. Team dynamics also play a significant role in business performance of course. Katie Popp talks about this in her article, Understanding Your Team’s Dynamics on the Mission Capital’s website. She also provides a graphical representation of the impact that Tuckman Model phases have on performance (see below).

What is particularly interesting to me is that there is a lot of overlap between Logistics and Team Dynamics. Both, for example, speak about setting a clear purpose and vision, using the right tools, team alignment and working agreements, embracing change and conflict. I believe that, as Agile practitioners, we bring the most value to a team by helping to build an environment for success. It’s not by telling them what to do or how to do it. It’s about creating an atmosphere where the team comes together to identify and define their own solutions. Focusing on the areas where team logistics and dynamics intersect is an extremely powerful strategy. It allows us, as Agile practitioners, to work on improving both the operational and interpersonal aspects of the team at the same time.

Do’s and don’ts of developing a high-performing team

As part of our discussion in the Agile Community of Practice at Stuart, we discussed some practical tips on what to do and what not to do to develop high-performing teams. We identified six categories that we thought were especially relevant to developing teams. Below is the output of our conversation.

Our Stuart values

At Stuart we have a set of values we strive to embody and consciously develop within the organisation. These are values that bind us, bring us closer together and provide a north star to members of the organisation. Developing high-performing teams directly supports several of our internal values, including “Build by empowerment”, “Start with humility”, “Share through cooperation”, and “Solve the problem, right”.

My hope is that this blog article has given you some insight and food for thought, and that you have some practical knowledge and tools to help you develop high-performing teams. Whether your team is an Agile team or not, it doesn’t matter. Every team and every organisation benefits from high-performing, self-organised and empowered teams!

Like what you see? We’re hiring! 🚀 Check out our open positions here!

Creating and developing a high performance Agile team — Part 2 was originally published in Stuart Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Discover the benefits of a high performance Agile Team and learn to recognise teams that achieve their goals and grow together.

The topic of High Performance Agile teams is one that comes up frequently in conversations in Agile circles. There is a lot of confusion around it, I think in part because it’s difficult to come to an agreement of how we define a high performing team and what the benefits are. At Stuart we recently had a discussion around High Performing Agile teams in our Agile Community of Practice (CoP). We wanted to gain a common understanding of what we mean by it, what are the benefits of high performing teams, what are some patterns and anti-patterns, and most importantly, what are some tips and tricks for developing high performing teams. So, we put our heads together and came up with some thoughts that I am now sharing as part of this blog post.

Before we begin, it is important to note that developing a high performing agile team is a never ending journey and that it is not necessarily linear. A team may become more or less mature for a number of reasons such as a change in scope or vision, changing, adding or removing team members, and organisational restructuring. Below is a summary of our conversation blended together with my own thoughts and experiences. I also included some research I conducted online that supports the need for high performing teams and highlights the struggles to get there. Let’s start by asking ourselves: why do we want high performing agile teams anyway?

Why do we want them?

High performing teams bring many benefits to the organisation, including:

• Better quality solutions
• Sustainable development
• More engaged and empowered team members
• Removing leadership bottlenecks
• Happier teams which are more productive and result in people staying with a company longer
• Better communication
• Trust
• Efficiency

How do we define them?

We started our conversation in the CoP by defining what we mean by high performing Agile teams. Are we talking about lines of code? Number of releases? Business or customer value? Velocity in the case of Scrum or cycle time in the case of Kanban? The answer is more nuanced than it would appear at first glance. This is because different teams define success in different ways and we need to be careful not to track vanity metrics such as lines of code or velocity. These types of metrics do not speak to the value that the team is delivering. Business value and customer value are great indicators of success but only if you have a way to define and measure them. Many organisations do not. How did we define a high performing Agile team then, you ask? Well, we stuck to the most basic definition possible. First we aligned on the definition of an Agile Team, and then identified what makes it a high performing one.

An Agile Team is a team composed of cross functional, fully autonomous individuals with the skills necessary to effectively accomplish the work in their backlog. Everyone in the team is empowered to make their own decisions and is held accountable for them. (This is a fairly standard definition of an Agile Team and can be found by doing a quick Google search.)

A High Performing Agile Team: Is an Agile team made up of highly motivated individuals that consistently delivers exceptional results (however you define them) over a period of time, regardless of the challenges and uncertainty the team may encounter.

High Performing Agile teams don’t just spring out of nowhere. It takes dedication and leadership at all levels. A lot of effort goes into developing such a team. The results of the team are often due to the sustained efforts of committed people who embody Servant Leadership values and principles. As I stated earlier, it is an infinite game. High performing teams are rarely satisfied with their performance even if they are performing well. They are constantly looking for ways to do better.

That being said, the definition of a high performing team can be somewhat vague and general so we continued our conversation by identifying first what is NOT a high performing team and where teams struggle. Secondly, we thought about what we can do as Agile Practitioners to move the team past those struggles to achieve high performance.

Why do Teams Struggle to Gain High Performance?

There are many reasons why a team might struggle to achieve high performance. I find it useful to break it down into two main categories: Logistics and Team Dynamics. As Agile Coaches our role should be to ensure that every team has an environment that contributes to and is conducive to their success. We should help every team to discover their own solutions without dictating to them what they should be. That is not to say we cannot suggest possible options to them based on our experience working with other teams. Our expertise is why we are Agile Coaches, after all. However it’s important that we approach each team individually. Every team has a set of unique characteristics, team dynamics and struggles. Understanding and recognising them is the key to their success.

Team Logistics

Let’s talk about the first category I mentioned, Logistics. Logistics is about processes, tools and vision. It’s about how the team accomplishes their goals. For example, does the team have a common vision? Do they have a roadmap? How was that roadmap created? Do they understand their purpose? Do they know how to use the tools at their disposal and are they using them effectively? Are they using the right tools for the task at hand? Does their process work for them or are they working for the process? There is nothing worse than a team following a process that works against their success. This is the fastest way to decrease a team’s happiness and productivity and it will result in sub-optimal solutions. People won’t want to be a part of the team and could potentially leave it.

The below Matrix is taken — and slightly altered — from a Blog post created by Uday Varma in an article titled 8 Keys to Transforming into a High-Performance Agile Team in the Agile Connection website, which is an incredibly useful resource for Agile Coaches and Practitioners. I find it extremely valuable when discussing possible reasons why Agile teams fail to reach high performance and I reference it often. I love being able to use materials that are already out there and created by other Agile Practitioners to help us in our quest for continuous improvements and learning. If anyone finds anything I write useful, it would bring me immense pleasure to know it is being referenced and used to help others in their Agile Evolution. In any case, I want to give a shoutout to Uday!

We used Uday’s matrix as a starting point. We then added the last column on the right as part of our discussion about potential strategies for reaching high performance, and we captured it in a confluence page for everyone in the organisation to make use of. I should point out that there are other reasons a team may fail to reach logistical high performance. If you are an Agile Coach for such a team, work with them to identify the reason, its root cause, and strategies to address it. Below is the matrix that we, at Stuart, have put together by combining Uday’s original Matrix with the strategies we identified to help us resolve them.

These aren’t the only areas where a team might struggle logistically. They are the areas we spoke about in our Community of Practice. Your team may share some of these struggles and may have some that are unique to your circumstance. Even in our community there were teams that associated more strongly with some of these areas than others. That’s ok. My point here is not to iterate every single area where a team might struggle but to make the point that Logistical discipline is one of the aspects that is key to your team’s success.

Team Dynamics

Let’s talk about the other category I identified, Team Dynamics. This is where it gets tricky because every team dynamic is different. To create outstanding Team Dynamics, Servant Leadership, influence management and social engineering all come together for the greater good of the team. An Agile Coach, or anyone in an Agile team, needs to be aware of the dynamics within the team and needs to have an understanding of how to help the team dynamics evolve. There are many lines of thought around this, from things like learning frameworks like Shu Ha Ri to different leadership styles. Regardless of how you approach it, if you understand team dynamics, then it almost doesn’t matter what kind of leadership style you employ (except for command and control, we don’t like command and control). To understand team dynamics I use the Tuckman model. I find it useful and easy to follow and it is universally relevant. Every team, in my experience, goes through these stages. It doesn’t matter if they are an Engineering team, HR, Finance, Operations, or an Agile team. EVERY team goes through this process. It is also important to point out that teams can progress or regress through the model depending on the circumstance. So…. What is it?

The Tuckman model describes the stages a team goes through in its journey to high performance. There are five stages. Each describes the team’s overall characteristics at that point in time to give insight into where and why a team might struggle. In our conversation with the Community of Practice, we discussed the model and identified some strategies for how to develop a team into the next stage. Again, these are just examples, you may come up with more strategies for your specific team. Additionally, teams may progress through some phases faster than others. The key is to have patience and understand that a team is made up of individuals, each with their own needs and desires.

The Five Phases

The image below is taken from Toggl Track: 5 Stages of Team Development and I feel it succinctly and elegantly describes the five stages.

Understanding how to identify the phases and the characteristics that make up each phase is only the first step in the process of team development. Developing strategies to move a team through higher levels of maturity is key, as is understanding what may cause a team to move back to a previous phase.

In Part II of this blog post, we will explore Tips and Tricks to developing a high performing Agile Team so keep your eyes open for it! It is my sincere hope that you found this post helpful, feel free to leave comments or suggestions and to share your thoughts.

Creating and Developing High Performance Agile Team — Part 1: The benefits of High Performance… was originally published in Stuart Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

As Agile Practitioners, we are normally the only Agile Practitioner on the team. We do not have someone questioning our decisions, besides the development team (always and nonstop 😉 — which we appreciate!). But we do not have another person from the same profession challenging what we are doing. Who makes sure that we deliver high-quality work? How do we make sure to learn from each other? How do we get new ideas? How do we stay on top of our game? How do we make sure to be aligned in our ways of agile development? And how do we make sure that everyone can handle the workload and pushback we get from the teams?

Recently, these questions were flying around in the Stuart Agile team and we were looking for ways to solve them. Of course, we implemented initiatives, which mitigated some of these risks. For example, we created “brain meeting”, a space to deep-dive into agile discussions and align on practices. Or team meetings to talk about organisational topics, workload, and happiness. But we asked ourselves: Is that enough?

What would we suggest to our development teams if they had the described issues or questions?

If this conversation was with our development teams, what would we suggest they do? And the answer was super clear: Code reviews.

Why is this standard tool, which we expect all of our development teams to use, not part of our way of working? We took the decision to look into it and figure out how we could adapt the best practice of code reviews to our work as agile practitioners.

What we proposed

First, we needed to clarify what our “code” was, or in other words, what exactly we wanted to be reviewed. We brainstormed some ideas around this and quickly found out this was not a trivial topic. We decided to call it “deliverables”, which included things like workshops, trainings, retrospectives, kick-off meetings, how we do stand-ups, etc. It was clear that not all these deliverables could be reviewed in the exact same way or with the same cadence if we wanted to make the process effective and avoid overwhelming the team.

For the deliverables which consisted of something we created (a training, documentation, an article…) it was apparent that we could just review it asynchronously and collect feedback. The challenge was for the non-tangible things, such as reviewing certain meetings. In those occasions, we suggested organising a short session, around 30 minutes, with the reviewer to have a conversation. The proposed agenda for those conversations was:

• Giving context about the team and the goal you had in mind with the session
• Explaining how the session went (sticking to the facts) and what the outcome was
• Commenting on challenges and potential issues you encountered during the session: for instance, things that didn’t work as expected, problems with engaging people, pushback, etc.
• Finally, the reviewer gives feedback on all the points above and suggests alternatives when appropriate.

Additionally, as this was a new thing, we proposed to have a last part of the session in which we collected feedback about the session itself. We wanted to know if the reviews were useful for the team and to spot potential areas of improvement.

The next question we needed to answer was when the reviews should happen. Should we do one every time we work on something intended to help our teams? After every stand-up? When does it make sense to complete an agile review and when doesn’t it?

We concluded that this also depended on what we were reviewing. So we proposed dividing deliverables into two groups: Deliverables that need reviews every time and deliverables that only need it periodically.

Deliverables that required review every time included things such as the retrospective outcomes, training, big changes happening in the teams, and kick-off sessions for new teams.

On the other hand, deliverables that don’t require review every time included things like one-to-one sessions, refinement sessions, preparations for retrospectives, and stand-ups. In this group, we agreed the reviews could be requested on-demand when we felt it was necessary, but also established a baseline cadence for reviews. For instance, every quarter we will review how we are approaching stand-ups.

Finally, we proposed the creation of a specific Slack channel to request these reviews. Any member of the Agile team could volunteer to review any deliverable from the others. Multiple reviewers were also allowed where it made sense.

Ah! We also needed a name, since Code Reviews didn’t make sense for us. So we decided to simply call our new method Agile Reviews.

Implementation of the idea

After developing the idea we did two test reviews to try it out and to get an idea of how it feels to be reviewed and to be the reviewer. Also, we wanted to get some initial ideas on where we could improve. With the insights from these tests, we made some adjustments to the Agenda and to the expected time needed for specific points.

We also talked to some developers to deepen our knowledge about code reviews and figure out some tips and tricks from them. The main insight from these talks was that we should add the following ground rules:

• Making sure to give feedback with empathy
• Not being too picky

As the next step, we presented the idea to the team, discussed details and got their buy-in to try it out. One learning we had from that session was that we had to define the deliverables more in detail as well as making the goals and not-goals very explicit.

Goals:

• Learn from each other
• Ensure the quality of our work
• Align on “how to do things” (identify areas in which we want to be aligned, similar to coding standards in the engineering world)
• Help maybe identify blockers or hidden problems
• Identify hidden personal struggles (is the agile practitioner happy in the team, stressed, unhappy, any personal issues in the team)

Not goals:

• Controlling
• Comparing
• Having everyone work the same way

How did it go?

We have been using agile reviews for three months in the team and so far we have done over 60 reviews in a team of eight people (including the review of this blog article!).

The feedback has been generally positive, helping us improve the quality of the work we deliver to the teams. It has also helped us to share information. A good example of this was a small presentation about how to improve the way we give and receive feedback. This presentation was originally created for a specific team to help them address some issues. However, after doing the agile reviews, some of the reviewers had an interest in using the same materials for their respective teams.

Another thing we can observe from this period is that there is a lot of interest in knowing what others in the Agile team are doing. When preparing this proposal, one of our concerns was that it could become difficult to find volunteers to be reviewers and that it could create too much overhead in the team. Far from it! What we have seen is that on many occasions there are multiple volunteers for the reviews. This is great because we are enabling people to share and learn new things, and also allows the reviewee to get more opinions.

Feedback

We can corroborate that we achieved some of our goals with the feedback that we have collected. Some examples are:

• “Overall it has helped me see other structures and different points of view.”
• “I like the feedback very much (when I present something that I did), it’s an improvement goal for me and it works.”
• “The quality of our work has gone up, as has team collaboration and engagement.”
• “I feel the sharing of ideas has been really fruitful, it allows us to synthesise our thoughts and to learn from one another”

It’s also fair to mention that some people expressed they got a bit anxious about the reviews, the most recurring points being:

• Fear of missing out on what was going on if they couldn’t review certain things
• Having a bit of a “competition” feeling when they were not submitting things for review while others did
• Introducing the reviews may have slowed down our speed as sometimes the process takes too long

These are undesired side effects and we are going to discuss them with the team. But we believe we can learn a lot from these insights as well. For example, developers may have similar concerns about code reviews and this experience can help us to understand their feelings.

Conclusions

Overall, we are happy with the results and we have learned a lot from them. This practice will definitely continue and, at the same time, we will keep collecting feedback and applying changes to remove pain points and help participants to feel less anxious.

After this experience, we are convinced that, in the same way, those code reviews are a best practice for development teams. Agile reviews are a great way to boost the quality, knowledge, and collaboration in a team of agile practitioners. We would definitely recommend applying something similar to all the agile teams out there!

Written by Àngel Sebastián Esparza and Magdalena Lorenz

Agile reviews: How we adapted an engineering best practice was originally published in Stuart Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

9 months ago I joined Stuart as a Product Designer, and started my journey into the world of logistics. I have never worked in such a dynamic and fast-changing environment before, and one of my biggest challenges was to quickly incorporate and research the Supply and Demand side of logistics from scratch.

Now, looking back to the beginnings of my journey at Stuart I can say it’s been an amazing time when I rapidly learned a lot; I believe I selected the appropriate set of exercises and I am excited to share it with the community. Maybe it will help another designer to start their journey smoothly into the complex domain!

Principal advice: Set your research goals.

When I joined Stuart we did not have a dedicated User Research Department, nor any urgent upcoming project. I adapted to the current situation (the cross-functional team I joined was in search of engineers, and the roadmap for the team was not defined yet) and set the goal of my research by myself, which was to holistically research the topic of Supply and Demand to get myself prepared for future projects.

Do: Talk to your manager, PM/PO, or whoever made a decision of you joining the project to understand what was the reason you are in the project/company, and/or collect their vision/expectations. Try to find out the current state of the project, what is the biggest need/problem the company needs to address. Is there any research done already about that topic? Depending on the answer, you should think of your research goals and methods.

Below I am sharing some advice based on my experience researching complex processes from scratch — meaning there was no research done before I joined, and no relationship established with my users and stakeholders.

Advice #1: Know who your user is.

Do: It’s extremely important to connect to the right people and ask them the right questions. How to identify the right people? Check the organisational chart, internal documentation (Confluence, Notion, etc.), simply ask whoever is onboarding you which people you should meet. Along with learning about company culture, building the right network are the most important steps of a newbie.

UX deliverable

Users- or stakeholders map. It’s needed to know the scale of your research (so you can estimate timeframes), and map out existing relationships (consider both official and unofficial), so once you start talking to your users you can link workflows/insights and grab a bigger picture.

Advice #2 & 3: Know what you’d like to learn from users, and what you’d like to achieve from the first touchpoint.

It sounds very cliche, but indeed it’s crucially important to identify what you would like to take from your touchpoint(s) with the user. Talking to experts in their domain could be challenging, because they have so much specific knowledge to share they simply might not know how to pass it on.

Do: Ask the right questions and frame the interview. Consider studying internal documentation of the department your user is working in if possible, and/or sector-related materials so you can have a better understanding of users’ pain points. Ask them what kind of projects they participate in, who they closely work with, what kind of tools they use, etc. Also, if you plan to collaborate closely with your users in the future, remember that the first touchpoint is not only about you learning about your user, but also an opportunity to get a buy-in from them! Explain what Product/UX design is about and how you expect the relationship with that person to look in the near future — maybe you are already planning some shadowing sessions, usability testing, or a second round of interviews?

UX deliverable — 1

Interview guide. First think of what you’d like to learn (goals), then translate it into questions (interview guide). Don’t forget about timeframing, introduction, ice-breaker (more in the next advice), closing and Q&A part.

UX deliverable — 2

Interview documentation. Once interviews/touchpoints are over, it’s important to document what users have shared with you. This can be a simple transcript where you group insights, empathy map, user journey, jobs to be done, etc.: whatever suits your research goals best, and would help you in the next phase of your research/project.

Advice #4: Treat your users as people, not robots.

Even though your users might do repetitive tasks and have very little “human” part in their daily routine, we are human after all!

Do: I always like kicking off the first interview by asking my users how they found their way to their workplace, what they like and dislike about their job, and what their daily motivation is. It always helps to break the ice and establish a human connection. Understanding what motivates your user is great knowledge you can benefit from in your future research and design processes!

UX deliverable

Ice-breaker part in your interview guide.

Advice #5: Sharing is caring.

Sharing your learnings, especially when you start investigating a topic no one has ever touched before, could be the most helpful thing you can do for your team and company. Even if you don’t know who can benefit from your research outside of your team, it’s still important to keep it accessible.

Do: Document your research in such a way that it can be shared to the biggest range of people. Share it actively (on Slack, email, calls) and passively (make it searchable).

Advice #6: Find your way to dive deeper into the problematic topic.

Once you feel comfortable with the domain knowledge, do not hesitate to run a second round of investigation.

Do: Find a method that suits your project goal best. Talk to the PM/PO if needed to align on what you’d like to clarify and what level of detail is needed. For the project we run I scheduled a set of shadowing sessions with a few users, where we asked them to perform their tasks and explain the decision-making process in detail.

UX deliverable

Depends on the problem/project. It could be an interview, shadowing/screen-recording session, survey, etc. As I was dealing with complex processes, I always liked to validate process diagrams I created after every shadowing session with users to be sure I interpreted all that I saw correctly.

After running all those exercises I closed the initial part of my research and really felt confident to embrace upcoming challenges (I am on it right now!). During my research process I heavily relied on Nielsen Norman group materials, here happily sharing some of them:

To watch:

• Complex Apps 101
• Design Patterns For Complex Apps and Workflows

To read:

• Complex application design framework
• Stakeholder Analysis for UX Projects
• Task Analysis: Support Users in Achieving Their Goals

What about you? Have you ever had an experience dealing with complex processes research? What would your advice be?

How I researched complex processes at Stuart from scratch. was originally published in Stuart Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

It was blood, sweat, and tears for Stuart’s Solutions Engineers team until we found the right combination of configurations in order to ensure our tests using Jest always pass.

Before jumping to our recommendations, we’ll set the scene on who we are and what we’re trying to do.

👋 Context: The Solutions Team

In January 2020, we created a new team within Stuart Tech: the Solutions Engineers team.

The team’s purpose is to bridge the gap that exists between the business and the products that we have at Stuart. If the business team wants to trial a new business use-case, they ask for support from the new solutions engineers team. We then build solutions to unblock their needs in a short period of time.

To put it in another way: We are a startup inside a startup!

The team mainly works on projects with a short life span, adding the maximum value in the least amount of time possible. To achieve this, we had to make technical decisions on our development stack. That’s why our Solutions stack looks like this:

• Node JS and Express JS (using Typescript) for the backend;
• React JS for the frontend;
• NPM as the package manager;
• Jest for tests;
• ESlint as linter;
• Jenkins as CI.

All these tools are set up using a monorepo, in which we share all common code between Solutions in a very easy way.

🤔 The problem

Jest is a resource eater. Doesn’t matter if you have 8GB, 16GB, or 32GB of RAM. Jest will use it all. How about the CPU? Exactly the same. Two cores? Yummy! Eight cores? Yuuuummy! 😋

This is a pretty interesting situation. We are very used to running software that is extremely badly optimised. So badly that most of it usually uses a single CPU and just a few MB or GB of RAM, because the single-threaded application acts as a bottleneck. But that’s not the case of Jest. Jest adapts to your maximum resources and drains them all.

This is why all over the internet, you can very easily find recommendations for forcing Jest to run in a single thread, or a maximum of 50% of your CPUs.

And luckily, you can do it quite simply by setting parameters in Jest such as maxWorkers. This will be enough for development purposes. But… How about when running it in a CI tool such as Jenkins? If you have 4 workers available per Jenkins node, then 4 Jenkins jobs can be executed in parallel.

In the scenario of running 4 jobs in parallel with Jest’s maxWorkers to 1, you’ll end up having 4 CPUs consuming 100% (not exactly, but for the purpose of this example, close enough!).

On the other hand, you cannot control Jest RAM usage. It’ll use all your RAM if needed. So again, if you have 16GB of RAM, your individual Jest executions, running in the same Jenkins node, will try to consume 16GB each!

Because of the high CPU and RAM usage, the Jenkins daemons will not be able to communicate with the controller node—and the whole Jenkins execution will fail.

😱 We need to prevent Jest from killing our nodes!

👩‍🏫 Our approach

Our Jenkins machines are 15GB machines, with 4 workers per Jenkins node. As a result, we can have up to 4 Jest commands running at the same time in the same machine.

Limiting the CPU

Thanks to Jest, this is extremely easy. Just use:

--maxWorkers=20%

Why 20%, you might wonder? Simple maths: 4 (Jenkins workers) * 20% of the CPU = 80%. We’ll leave 20% for the system (so that the Jenkins Daemon can still respond, OS stuff, etc.).

Limiting the RAM

Here’s the biggest challenge. Jest doesn’t allow you to set a maximum of RAM. What’s even worse, Jest won’t trigger any Garbage collecting process unless it is running out of RAM or you force it to do so. That means that the RAM usage will increase and increase and increase—up until the point where GC is triggered. If that’s not enough to free some memory (:cough: leaks :cough:), swapping starts to happen. By then, tests are running really, really slow.

In order to limit the RAM, we went for the Docker approach. For every test suite we want to run, we spin up a Docker image and run it while limiting its resources based on the host resources.

Considering that our Jenkins nodes have 4 workers, we need to limit the resources based on the maximum available RAM, which in our case is 15GB.

We need to understand that RAM behaves slightly differently than the CPU. It takes time for RAM to be fully used, so in this case we can be a little more aggressive in allocating the resources. In our case, we decided to set the Docker limit to 4.3GB. 4.3GB * 4 = 17,2GB — slightly above the RAM limit.

How can you limit the RAM in Docker, you might wonder?

docker run -m 4300m tests_image

Both things together

If you really want to make sure Jest doesn’t drain all your CPUs, you can also benefit from Docker’s CPU limitations. Our host has 8 logical CPUs. So to be consistent with the 20% CPU limitation we set on the Jest command, we will set a CPU limitation on every Docker container to consume up to 6.5~ logical CPUs.

Again, as we will be running a maximum of 4 dockers in parallel in a Jenkins node, that’s around 1.5 CPUs.

docker run --cpus=’1.5’ tests_image

And that, in combination with the RAM setting, becomes:

docker run -m 4300m --cpus=’1.5’

🏇 Does this speed up tests?

It depends…

Our biggest pain point was our Jenkins agents dying because of high CPU and RAM usage. With this approach, we resolved that issue.

It does improve speed for the test suites that are small. For those that eventually consume close to 4GB of RAM, the tests run slowly. This is not a big deal, because thanks to our Jenkins setup built by our DevOps team, the Jenkins nodes are auto-scaled. By using Jenkins pipelines, we can simply parallelise as many tests as we want into different Jenkins jobs.

🚰 Working around the memory leaks

In order to improve tests’ performance and ensure they do not slow down because they are consuming too much RAM, our recommendation is not to run all tests in a single Jest command.

Instead, divide your tests into multiple executions. In our case, we have a monorepo with more than 10 applications. We have divided these 10 suites into 5 different Jest executions. By restarting Jest, you’ll make sure you start from 0, and not with previous leaks affecting your current tests.

🐌 Still too slow?

It could be that soon after starting running the tests, they’ll already be running slowly. It’s then highly likely that you’ve already hit the max RAM allowed. To confirm this, run “docker stats”. It will show the current memory usage and its limit.

If the usage hits 100% very quickly—unless you find a big memory leak and manage to resolve it—the only solution is to increase docker RAM: run the test suite without setting any docker limitations, and while monitoring with docker stats, write down the highest peak.

If the peak is way beyond the RAM you have set to the Docker container, you should increase the RAM limits in the docker container, but also make sure that the maths still make sense. If we had to increase our RAM further for our tests, we’d have two options:

1. Increase nodes RAM from 15GB to 32GB.
2. Reduce the Jenkins workers from 4 to 3.

🏁 Conclusions

We spent a couple of weeks with a trial and error approach until we found the perfect numbers for our infrastructure. You should not expect to copy-paste any configuration. You need to understand your infrastructure and adapt to it.

We also found it pretty interesting to learn that a tool which makes use of all the hardware available is, in reality, a problem. We are so used to running programs in over-dimensioned hosts that when it’s not the case, we struggle to figure out how to fix that.

Would you like to see more? We are hiring! 🚀 Check out our open positions.

🔍 A practical guide to removing frustration with Jest and Jenkins was originally published in Stuart Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

How we dramatically increased our database write speed

Optimization Target

Stuart is a last-mile delivery company. When riders are active in the platform their phones send GPS locations so the application knows where everyone is.

Servers are hammered with GPS locations, so this is a part of the system were optimizations may have a significant performance impact. On reception, each GPS location gets inserted into MySQL, and then a number of things follow. (This is a simplification, but faithful enough for the purposes of this post.)

The business is growing fast, and MySQL started to struggle with the load during peak hours. This continued up to the point where manual intervention from the on-duty team was needed to reduce the pressure on the platform.

While some long-term solutions were being worked on, we needed a guerrilla tactic to win time and regain tranquility until those solutions landed.

We wanted to optimize MySQL INSERTs for the same entity performed by separate threads.

These threads could be the ones responsible for HTTP endpoints in a web server. Or they could be BEAM processes in a Phoenix application. In our case, threads execute as Sidekiq workers, but conceptually they all represent the same scenario.

Bulk Data Loading

Before we explain the technique, let’s take one step back.

Let’s imagine you need to import 1,000,000 records into a database in a batch process. Which are the options?

Sequential INSERTs 🐌

The simplest approach is to write a loop that inserts them one after the other.

In Ruby:

It works, and it may be enough for a small number of records, but it is really inefficient for a large set.

Parallel INSERTs 🚀️️️

The next step is to spawn N threads, with their own database connection each, and split the work among them.

In Ruby:

As a rule of thumb, this may speed things up to nearly Nx.

It’s a helpful technique for batch processes. In the problem at hand, however, applications already leverage this as a side-effect of being multi-threaded.

Bulk INSERTs 🚀️🚀️

But we can still do better. One single INSERT statement is able to insert multiple rows at once. In this code snippet we insert three rows:

INSERT INTO table_name
  (col1, col2, col3)
VALUES
  (val11, val12, val13),
  (val21, val22, val23),
  (val31, val32, val33);

This is way faster than doing three individual INSERTs.

MySQL documentation has a page dedicated to this optimization where you can see the server works less.

Another important difference is the amount of network round-trips: Every statement you execute has a response of some sort from the database. In a procedural API, you issue the INSERT, wait for the server to commit the transaction and reply, process the reply, next record, etc.

Therefore, if you issue 1,000 sequential INSERTs, the client needs to wait for MySQL and process its response 1,000 times. That waiting is interleaved with all the operations. On the other hand, if you execute one single INSERT with 1,000 rows, the client waits for MySQL once.

All these observations compound, and therefore the bigger the batch size, the bigger the speed up. To show you the magnitude of the acceleration, these are some factors that I have seen:

Groups of    5 →   5 times faster
Groups of  100 →  57 times faster
Groups of  500 → 189 times faster
Groups of 1000 → 284 times faster

Factors depend on latency, complexity of the statement, etc. But in any case, for a back-of-the-napkin calculation, we can see an opportunity lies in here for huge performance improvements.

CSV Imports 🚀️🚀️🚀️

Can we still squeeze more rows per time unit? Yes!

Importing data in CSV format is even more performant. However, we’ll leave it at bulk INSERTs for this post.

Connecting the Dots

The idea of the technique is to apply this to a dynamic situation where you are receiving a data stream in a multi-threaded endpoint.

There are several ways to do this, and the trade-offs depend on the details of each particular situation. We implemented a process-level buffer and a timer thread. GPS locations received are pushed to the buffer. Every second, that buffer is flushed. Done.

Simple, but the impact is dramatic.

Ruby Implementation

The problem statement in this post, and the technique we use to address it are quite generic. However, let’s see some concrete source code in Ruby.

The Sidekiq worker that receives locations only pushes to a buffer. Can’t be more cheap:

Schematically, this is the batch writer:

I like the clarity of the the timer thread, which allows you to unit test flush properly in a deterministic manner. The timer thread is not needed in the test suite beyond a trivial coverage that guarantees it flushes as expected.

You could also flush on buffer size. But a time threshold is normally necessary because otherwise flushing could be too infrequent in valleys. In an Elixir application where I have written something similar for DynamoDB I have both logics in place: buffer size, and at most t time between flushes.

You have to choose a value for MAX_BATCH_SIZE that makes sense for the application. Let’s suppose your workload today is less than 100 records per second per process. If you set MAX_BATCH_SIZE to 100, when the platform is quiet maybe you send batches of 15, and when the platform is busy maybe you send batches of 75.

Here’s the point: As we have seen above, batches of 75 are way more efficient than batches of 15. Thus, the busier the system, the faster it runs.

Batch insertion can be done in different ways. My input is a trusted hash table and I build a raw SQL string that is executed directly with exec_insert. If you want to work with ARs, Active Record 6.0 shipped with insert_all and friends. The gem activerecord-import is also worth mentioning.

Finally, we need to ensure the buffer is flushed when the server shuts down:

Caveats

Due to buffering, there’s a small delay between reception and availability of the event in the platform.

On the other hand, MySQL does not give you the IDs of the new records. If the primary key is auto-incremented, however, MySQL guarantees they are consecutive. Thus, if you need them, you can retrieve the last ID inserted and do the math.

Summary

We’ve seen a way to boost parallel INSERTs with a per-process buffer flushed regularly using one single INSERT with multiple rows.

It’s been a smooth sailing since this was deployed. 🎉

Like what you see? We’re hiring! 🚀

Boosting Parallel INSERTs was originally published in Stuart Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.