Then, it evolved to be a platform to build ML models for generic software engineering purposes: we now no longer only have models that operate on bug reports, but also on test data, patches/commits (e.g. to choose which tests to run for a given patch and to evaluate the regression riskiness associated to a patch), and so on.

Its infrastructure also evolved over time and slowly became more complex. This post attempts to clarify its overall infrastructure, composed of multiple pipelines and multi-stage deployments.

The nice aspect of the continuous integration, deployment and production services of bugbug is that almost all of them are running completely on Taskcluster, with a common language to define tasks, resources, and so on.

In bugbug’s case, I consider a release as a code artifact (source code at a given tag in our repo) plus the ML models that were trained with that code artifact and the data that was used to train them. This is because the results of a given model are influenced by all these aspects, not just the code as in other kinds of software. Thus, in the remainder of this post, I will refer to “code artifact” or “code release” when talking about a new version of the source code, and to “release” when talking about a set of artifacts that were built with a specific snapshot (version) of the source code and with a specific snapshot of the data.

The overall infrastructure can be seen in this flowchart, where the nodes represent artifacts and the subgraphs represent the set of operations performed on them. The following sections of the blog post will then describe the components of the flowchart in more detail.

Continuous Integration and First Stage (Training Pipeline) Deployment

Every pull request and push to the repository triggers a pipeline of Taskcluster tasks to:

• run tests for the library and its linked HTTP service;
• run static analysis and linting;
• build Python packages;
• build the frontend;
• build Docker images.

Code releases are represented by tags. A push of a tag triggers additional tasks that perform:

• integration tests;
• push of Docker images to DockerHub;
• release of a new version of the Python package on PyPI;
• update of the training pipeline definition.

After a code release, the training pipeline which performs ML training is updated, but the HTTP service, the frontend and all the production pipelines that depend on the trained ML models (the actual release) are still on the previous version of the code (since they can’t be updated until the new models are trained).

Continuous Training and Second Stage (ML Model Services) Deployment

The training pipeline runs on Taskcluster as a hook that is either triggered manually or on a cron.

The training pipeline consists of many tasks that:

• retrieve data from multiple sources (version control system, bug tracking systems, Firefox CI, etc.);
• generation of intermediate artifacts that are used by later stages of the pipeline or by other pipelines or other services;
• training of ML models using the above (there are also training tasks that depend on other models to be trained and run first to generate intermediate artifacts);
• check training metrics to ensure there are no short term or long term regressions;
• run integration tests with the trained models;
• build Docker images with the trained models;
• push Docker images with the trained models;
• update the production pipelines definition.

After a run of the training pipeline, the HTTP service and all the production pipelines are updated to the latest version of the code (if they weren’t already) and to the last version of the trained models.

Production pipelines

There are multiple production pipelines (here’s an example), that serve different objectives, all running on Taskcluster and triggered either on cron or by pulse messages from other services.

Frontend

The bugbug UI lives at https://changes.moz.tools/, and it is simply a static frontend built in one of the production pipelines defined in Taskcluster.

The production pipeline performs a build and uploads the artifact to S3 via Taskcluster, which is then exposed at the URL mentioned earlier.

HTTP Service

The HTTP service is the only piece of the infrastructure that is not running on Taskcluster, but currently on Heroku.

The Docker images for the service are built as part of the training pipeline in Taskcluster, the trained ML models are included in the Docker images themselves. This way, it is possible to rollback to an earlier version of the code and models, should a new one present a regression.

There is one web worker that answers to requests from users, and multiple background workers that perform ML model evaluations. These must be done in the background because of performance reasons (the web worker must answer quickly). The ML evaluations themselves are quick, and so could be directly done in the web worker, but the input data preparation can be slow as it requires interaction with external services such as Bugzilla or a remote Mercurial server.

I belong to the second “school”: I have always thought, intuitively, that patches without tests are more likely to cause postrelease regressions, and so having test coverage decreases risk.

A few days ago, I set out to confirm this intuition, and I found this interesting study: Code Coverage and Postrelease Defects: A Large-Scale Study on Open Source Projects.

The authors showed (on projects that are very different from Firefox, but still…) that there was no correlation between project coverage and the amount of bugs that are introduced in the project and, more importantly, there was no correlation between file coverage and the amount of bugs that are introduced in the file.

File coverage vs Patch coverage

So, it seemed to show coverage is not a good predictor at all. I thought though: we are not as interested in existing bugs as we are interested in not introducing new ones. The study showed a file with high coverage has a similar possibility of having bugs than a file with low coverage, but… What happens if we look at things at the patch level?

Maybe those files are covered just incidentally (because their functions are called while running tests for other related files), but the tests were not specifically designed for them.

Maybe the tests were written hastily when those files got introduced, and so they are already full of bugs.

What about if we look instead at whether there is a correlation between patch coverage and the number of regressions that are introduced by those patches?

Of course, correlation is not causation, but we can at least make the case that, when new code is not automatically tested, we should be more careful and review it in more detail, or increase manual testing, or delay it to a new release to get more beta testing.

Data collection

Over the past few years, we have been collecting coverage data about Firefox patches.

It is not fully complete, as it doesn’t include Mac, it doesn’t include all possible Firefox configurations (e.g. Fission) and some languages like Python, but it does include the languages we use the most (JavaScript, C/C++, Rust), and it includes Linux and Windows (considering that a lot of Firefox code is cross-platform, I don’t see the missing Mac as a big limitation).

This means that we excluded the portions of the patches that touch Mac-specific code, Python code, etc., from the analysis.

I collected the data using bugbug and set out to explore it. I got 10938 clean patches, and 2212 buggy ones.

Box plots of bug-introducing vs clean patch code coverage

First, I plotted the data using a box plot:

It doesn’t tell me much. The median is higher in the case of “clean” patches though (~92% vs ~87%).

Scatter plot of lines added/covered by bug-introducing vs clean patches

Then, I plotted a scatter plot with “added lines” as the x-axis, “covered lines” as the y-axis, and color being “clean” or “buggy”. What I expected to find: more “clean” patches than “buggy” ones close to the line f(added) = covered (that is, covered == added).

There are a lot of outliers, so let’s zoom in a bit:

It seems to partly match what I expected, but… hard to tell really.

Distributions of bug-introducing vs clean patch coverage

Next step, plotting two empirical distribution functions for the two distributions. These functions show the proportion of patches, in the two distributions, at or above a given level of coverage (e.g. for the distribution of “clean patches”, when x is 0.8, y is 0.4, which means that 80% of clean patches have more than 40% coverage):

Now, we can clearly see that while 45% of clean patches have 100% coverage, only around 25% of buggy patches have 100% coverage. The proportion of patches that is between 100% and 80% coverage is similar, but non-buggy patches more often present higher coverage than buggy ones. There are no meaningful differences for 40% of the patches (from 0.6 to 1.0 in the chart above).

What I learned

To summarize, we can see that there is a correlation between coverage and bug proneness. In particular, fully covered patches more rarely cause post-release bugs. Of course, correlation does not mean causation. We can’t tell for sure that writing tests for new code changes will reduce the riskiness of those changes, but we can definitely say that we can increase our attention level when changes are not covered by tests.

Source-based code coverage was recently introduced in Rust. It is more precise than the gcov-based coverage, with fewer workarounds needed. Its only drawback is that it makes the profiled program slower than with gcov-based coverage.

In this post, I will show you a simple example on how to set up source-based coverage on a Rust project, and how to generate a report using grcov (in a readable format or in a JSON format which can be parsed to generate custom reports or upload results to Coveralls/Codecov).

Install requirements

First of all, let’s install grcov:

cargo install grcov

Second, let’s install the llvm-tools Rust component (which grcov will use to parse coverage artifacts):

rustup component add llvm-tools-preview

At the time of writing, the component is called llvm-tools-preview. It might be renamed to llvm-tools soon.

Build

Let’s say we have a simple project, where our main.rs is:

use std::fmt::Debug;

#[derive(Debug)]
pub struct Ciao {
    pub saluto: String,
}

fn main() {
    let ciao = Ciao{ saluto: String::from("salve") };

    assert!(ciao.saluto == "salve");
}

In order to make Rust generate an instrumented binary, we need to use the -Zinstrument-coverage flag (Nightly only for now!):

export RUSTFLAGS="-Zinstrument-coverage"

Now, build with cargo build.

The compiled instrumented binary will appear under target/debug/:

.
├── Cargo.lock
├── Cargo.toml
├── src
│   └── main.rs
└── target
    └── debug
        └── rust-code-coverage-sample

The instrumented binary contains information about the structure of the source file (functions, branches, basic blocks, and so on).

Run

Now, the instrumented executable can be executed (cargo run, cargo test, or whatever). A new file with the extension ‘profraw’ will be generated. It contains the coverage counters associated with your binary file (how many times a line was executed, how many times a branch was taken, and so on). You can define your own name for the output file (might be necessary in some complex test scenarios like we have on grcov) using the LLVM_PROFILE_FILE environment variable.

LLVM_PROFILE_FILE="your_name-%p-%m.profraw"

%p (process ID) and %m (binary signature) are useful to make sure each process and each binary has its own file.

Your tree will now look like this:

.
├── Cargo.lock
├── Cargo.toml
├── default.profraw
├── src
│   └── main.rs
└── target
    └── debug
        └── rust-code-coverage-sample

At this point, we just need a way to parse the profraw file and the associated information from the binary.

Parse with grcov

grcov can be downloaded from GitHub (from the Releases page).

Simply execute grcov in the root of your repository, with the --binary-path option pointing to the directory containing your binaries (e.g. ./target/debug). The -t option allows you to specify the output format:

• “html” for a HTML report;
• “lcov” for the LCOV format, which you can then translate to a HTML report using genhtml;
• “coveralls” for a JSON format compatible with Coveralls/Codecov;
• “coveralls+” for an extension of the former, with addition of function information. There are other formats too.

Example:

grcov . --binary-path PATH_TO_YOUR_BINARIES_DIRECTORY -s . -t html --branch --ignore-not-existing -o ./coverage/

This is the output:

This would be the output with gcov-based coverage:

You can also run grcov outside of your repository, you just need to pass the path to the directory where the profraw files are and the directory where the source is (normally they are the same, but if you have a complex CI setup like we have at Mozilla, they might be totally separate):

grcov PATHS_TO_PROFRAW_DIRECTORIES --binary-path PATH_TO_YOUR_BINARIES_DIRECTORY -s PATH_TO_YOUR_SOURCE_CODE -t html --branch --ignore-not-existing -o ./coverage/

grcov has other options too, simply run it with no parameters to list them.

In the grcov’s docs, there are also examples on how to integrate code coverage with some CI services.

Coverage information is uploaded either automatically for revisions after they are landed in mozilla-central (for example for release managers when looking at uplift requests), or on-demand for in-progress revisions.

For revisions under review, in order to upload coverage you just need to trigger a try push containing code coverage builds and tests, e.g. by using:

$ mach try fuzzy --full

and selecting the relevant ccov builds and test suites. In the future, we will also likely automatically trigger coverage try builds for revisions we deem to be risky, alongside the on-demand option.

Here is an example of a try build which produced the coverage information for my revision:

Once the try build and linked tests finish, the coverage artifacts get parsed and uploaded to the Phabricator revisions corresponding to the commits in the try push. The analysis works on all commits in the try push that are linked to Phabricator revisions. Stacks of revisions are supported as well.

The coverage information is shown on Phabricator both at a high-level view, in the Revision Contents section, and at a detailed view in the Diff section.

The Revision Contents section contains a list of the files modified by the revision, showing both the coverage percentage of the whole file and the coverage percentage of touched lines. Here’s the screenshot of the section from my revision:

The Diff section instead shows the coverage line per line, coloring the bar on the right-hand side. Orange corresponds to uncovered lines, light blue corresponds to non-executable lines (e.g. a comment, a definition, and so on), dark blue corresponds to covered lines. When hovering the bar, the corresponding line is highlighted in the same color. The following screenshots show excerpts of my revision, with covered, uncovered and non-executable lines:

Why do we need to tell if a bug is actually a bug? There are several reasons, the main two being:

• Quality metrics: to analyze the quality of a project, to measure the churn of a given release, it can be useful to know, for example, how many bugbugs are filed in a given release cycle. If we don’t know which bugs are bugbugs and which are feature requests, we can’t precisely measure how many problems are found (= bugbugs filed) in a given component for a given release, we can only know the overall number, confusing bugbugs and feature work;
• Bug prediction: given the development history of the project, one can try to predict, with some measure of accuracy, which changes are risky and more likely to lead to regressions in the future. In order to do that, of course, you need to know which changes introduced problems in the past. If you can’t identify problems (i.e. bugbugs), then you can’t identify changes that introduced them!

On BMO, we have some optional keywords to identify regressions vs features, but they are not used consistently (and, being optional, they can’t be. We can work on improving the practices, but we can’t reach perfection when there is human involvement). So, we need another way to identify them. A possibility is to use handwritten rules (‘mozregression’ in comment → regression; ‘support’ in title → feature), which can be precise up to a certain accuracy level, but any improvement over that requires hard manual labor. Another option is to use machine learning techniques, leaving the hard work of extracting information from bug features to the machines!

The bugbug project is trying to do just that, at first with a very simple ML architecture.

We have a set of 1913 bugs, manually labelled between the two possible classes (bugbug vs nobug). We augment this manually labelled set with Bugzilla bugs containing the keywords ‘regression’ or ‘feature’, which are basically labelled already. The augmented data set contains 10818 bugs. Unfortunately we can’t use all of them indistinctly, as the dataset is unbalanced towards bugbugs, which would skew the results of the classifier, so we simply perform random under-sampling to reduce the number of bugbug examples. In the end, we have 1928 bugs.

We split the dataset into a training set of 1735 bugs and a test set of 193 bugs (90% - 10%).

We extract features both from bug fields (such as keywords, number of attachments, presence of a crash signature, and so on), bug title and comments.

To extract features from text (title and comments), we use a simple BoW model with 1-grams, using TF-IDF to lower the importance of very common words in the corpus and stop word removal mainly to speed up the training phase (stop word removal should not be needed for accuracy in our case since we are using a gradient boosting model, but it can speed up the training phase and it eases experimenting with other models which would really need it).

We are then training a gradient boosting model (these models usually work quite well for shallow features) on top of the extracted features.

This very simple approach, in a handful of lines of code, achieves ~93% accuracy. There’s a lot of room for improvement in the algorithm (it was, after all, written in a few hours…), so I’m confident we can get even better results.

This is just the first step: in the near future we are going to implement improvements in Bugzilla directly and in linked tooling so that we can stop guessing and have very accurate data.

Since the inception of bugbug, we have also added additional experimental models for other related problems (e.g. detecting if a bug is a good candidate for tracking, or predicting the component of a bug), turning bugbug into a platform for quickly building and experimenting with new machine learning applications on Bugzilla data (and maybe soon VCS data too). We have many other ideas to implement, if you are interested take a look at the open issues on our repo!

Being able to do it directly in the browser can make the workflow much smoother.

To support this use case, I’ve built an extension for Phabricator that integrates Searchfox code search functionality directly in Phabricator differentials. This way reviewers can benefit from hovers, go-to-definition and find-references without having to resort to the IDE or without having to manually navigate to the code on searchfox.org or dxr.mozilla.org. Moreover, compared to searchfox.org or dxr.mozilla.org, the extension highlights both the pre-patch view and the post-patch view, so reviewers can see how pre-existing variables/functions are being used after the patch.

To summarize, the features of the extension currently are:

1. Highlight keywords when you hover them, highlighting them both in the pre-patch and in the post-patch view;
2. When you press on a keyword, it offers options to search for the definition, callers, and so on (the results are opened on Searchfox in a new tab).

Here’s a screenshot from the extension in action:

I’m planning to add support for sticky highlighting and blame information (when hovering on the line number on the left side). Indeed, being able to look at the past history of a line is another sought after feature by reviewers.

You can find the extension on AMO, at https://addons.mozilla.org/addon/searchfox-phabricator/.

The source code, admittedly not great as it was written as an experiment, lives at https://github.com/marco-c/mozsearch-phabricator-addon.

Should you find any issues, please file them on https://github.com/marco-c/mozsearch-phabricator-addon/issues.

This is perfectly fine, but web workers are not so easy to use or debug. To take Prism.js again as an example, the option I mentioned earlier is false by default. Why?

In most cases, you will want to highlight reasonably sized chunks of code, and this will not be needed. Furthermore, using Web Workers is actually slower than synchronously highlighting, due to the overhead of creating and terminating the Worker. It just appears faster in these cases because it doesn’t block the main thread. In addition, since Web Workers operate on files instead of objects, plugins that hook on core parts of Prism (e.g. modify language definitions) will not work unless included in the same file (using the builder in the Download page will protect you from this pitfall). Lastly, Web Workers cannot interact with the DOM and most other APIs (e.g. the console), so they are notoriously hard to debug.”

Another alternative to achieve the same result, without using Web Workers and making things more difficult, is to use requestIdleCallback. This function allows a callback to be scheduled when the browser is idle, enabling us to perform background work / low priority work on the main thread without impacting animations / input response. N.B.: This will still be slower than synchronous, but might be cheaper than a Web Worker since you don’t have to pay the price of the Worker initialization.

Here’s an example, using promises and asynchronous functions we can also avoid callback hell and keep using normal loops.

function idle() {
  return new Promise(resolve => requestIdleCallback(resolve));
}

async function work() {
  let deadline = await idle();

  for (let job of jobs) {
    if (deadline.timeRemaining() <= 1) {
      deadline = await idle();
    }

    // Do something with `job`...
  }
}

I’m doing something similar in my Searchfox in Phabricator extension, to operate on one source line at a time and avoid slowing down the normal Phabricator operation. Here’s where I’m doing it.