In this post well estimating if the remote build cache is worth it using the internals of Gradle. This post is part of a series that dives in the details of whether it is better to rerun tasks locally or download them from the remote build cache on a slow internet connection. Read Part 1 first for the basics of estimating build cache performance.

Diving into Gradle

This post involves a bit of Gradle hackery using internal APIs. If you want to skip straight to the code, here it is. To perform an estimation on your project, scroll to the bottom of this post.

Estimating benefits of the remote build cache

To create a real world estimate for a build we will:

1. Run a typical build for a project and measure the build duration
2. Create a map of build cache keys for all the tasks in the project
3. Map the build cache keys to tasks which ran
4. Find all recently-added artifacts in the build cache
5. Sum their size
6. Sum of build cache artifacts / build duration = minimum remote cache connection speed

Sounds simple enough? Let’s get to it.

Automating the estimation

Run a typical build for a project

Let’s estimate how long ./gradlew :application:assembleDebug takes.

The first time we run it, it takes 60 seconds. But if we run it again, it only takes 3 seconds because all the tasks are up to date or cached. We need to temporarily disable task caching by forcing a series of Gradle tasks to re-run.

The following code snippet will re-run all SourceTasks in the module:

tasks.withType(SourceTask).configureEach {
  outputs.upToDateWhen { false }
}

With this in our buildscript, we can re-run ./gradlew :application:assembleDebug and get a consistent 55 second build duration excluding configuration time.

Create map of build cache keys for all tasks in the project

It sounds straightforward, but unfortunately, there is no public API to get a build cache key for a task.

We can see the build cache key easily in the build scans, so Gradle must be producing it somehow. I did some poking around in the Gradle open source project to find BuildOperationListener . This is a powerful internal API, but also quite dangerous if misused.

The API gives us all sorts of build events. We are going to listen for SnapshotTaskInputsBuildOperationType since it contains two pieces of information that we need:

1. hashBytes — the build cache key
2. the parent task id — needed to determine if the task was executed

We put all these events in a map.

Map the build cache keys to tasks which ran

To find whether or not a task ran, we need to listen for another type of event using the same BuildOperationListener : the ExecuteTaskBuildOperationType event. It has the two other pieces we need:

1. the parent task id to connect to the build cache key
2. skipMessage — null if the task was executed

With both ExecuteTaskBuildOperationType and SnapshotTaskInputsBuildOperationType we can use the parent task id to link the events together and calculate all the build cache keys for all the tasks which were just executed.

val cacheKeys = mutableListOf<String>()
executeTaskIdsMap.entries.forEach { (operationId, task) ->
    if (task.wasExecuted()) {
        val cacheKey = snapshotIdsMap[operationId].hashByte
        cacheKeys.add(cacheKey)
    }
}

Above is the code snippet that returns a list of build cache keys for all executed tasks in the project.

Find all recently-added artifacts in the build cache

The Gradle build cache is located by default in your Gradle user home in a folder called caches/build-cache-1 .

The build cache artifacts are stored in a compressed format in this directory with the build cache key as the file name.

The size of a compressed artifact for a task with a build cache key of ef41ae2ece5d is simply the following code in kotlin:

File("caches/build-cache-1", "ef41ae2ece5d").length()

Sum their size

We can sum the size of the compressed artifacts as follows:

val cacheSizeBytes = cacheKeys.sumBy { cacheKey ->
    File(cacheDir, cacheKey).length().toInt()
}

Calculate minimum remote build cache connection speed

val minimumConnectionSpeed = cacheSizeBytes / buildExecutionTime

In part 3 of this series we’ll do some in depth calculations based on real world data and see whether certain tasks are better to store remotely than others.

Try it out yourself

Instead of doing all this work yourself, I’ve automated it for you in the Gradle Doctor plugin. To find out if a remote build cache is worth using on your internet connection run any gradle task with -PbenchmarkRemoteCache.

./gradlew :app:assembleDebug -PbenchmarkRemoteCache

You’ll get a nice report like the one below:

=================Gradle Doctor Prescriptions========================
= Remote Build Cache Benchmark Report =
Executed tasks created compressed artifacts of size 159,93 MB
Total Task execution time was 208,85 s

In order for a remote build cache to save you time, you would need an internet connection to your node of at least 0,77 MB/s.
Check a build scan to see your connection speed to the build cache node.
Build cache node throughput may be different than your internet connection speed.

A 1 MB/s connection would save you 48,92 s.                                      A 2 MB/s connection would save you 128,88 s.               
A 10 MB/s connection would save you 192,86 s.
                                                                                                Note: This is an estimate. Real world performance may vary. This estimate does not take in to account time spent decompressing cached artifacts or roundtrip communication time to the cache node.   ====================================================================

Let me know how it works for you and your workload. Is using the remote build cache worth it?

This question is becoming ever more important as we all switch from our blazing fast office internet connections to our home internet connections.

This is part 1 of a 3 part series. Part 2 is now published.

Downloading a Single Task

How do we know if a remote build cache is worth it? Let’s take a simple Gradle task and calculate whether having a remote build cache is worth it.

If a task takes 10 seconds to execute and generates 1 MB of output, you would need an internet connection faster than 0.1 MB/s in order to save time. An internet connection of 1MB/s would mean that you could download the output of the task in 1 second, saving you 9 seconds of execution time.

Calculation for a real Gradle task

To do this for a Gradle task, you can print its outputs by adding the following to your build.gradle file:

tasks.named("compileDebugKotlin").configure {
  outputs.files.forEach {
    println(it)
  }
}

For the KotlinCompile task, the outputs are placed in that module’s build/tmp/kotlin-classes directory.

$ du -sh libs/analytics/build/tmp/kotlin-classes/
 136K

This KotlinCompile task produces 136KB of output and takes about 5 seconds to run on my computer. This would require a build cache connection of 0.136MB/5s = .0272 MB/s in order to be able to download the task faster than it would take to run locally.

The download size is actually smaller than 136KB because Gradle compresses artifacts before placing them in the build cache. This speeds up download time but adds decompression time. We’ll dive in to this later.

Let’s check a build scan

A build scan will tell us exactly how much time we saved by reusing the outputs of the task from the build cache. There is one caveat, it uses the amount of time that it took to run the task on a different machine when the remote build cache was seeded. In this case, the build cache was seeded from a significantly slower machine. This task only takes 5 seconds to run on my machine, but took 15 seconds when seeding the cache. To get a more accurate estimate, we need to estimate the minimum connection speed based on the local machines power.

Calculating parallel computation

Machines these days can perform multiple Gradle tasks at once. To perform the same calculation for two tasks run in parallel, we just take the length of the overall build, and divide it by sum of the two task’s outputs.

If two 10 second tasks run in parallel, the length of the overall build would still be 10 seconds. If both tasks produce 1MB of output, the sum of the two task’s outputs would be 2MB.

2MB/10s = 0.2MB/s minimum connection speed

The more tasks our local machine can run in parallel, the faster our connection to the build cache node needs to be!

Compression

Gradle compresses artifacts before storing them in the build cache. In the case of the above KotlinCompile task, the compressed artifacts are only 20KB instead of 136KB. This means our minimum connection speed to make the build cache worth it is much lower: 0.02MB / 5 seconds = 0.004 MB/s. This assumes the artifact decompression time is insignificant.
To calculate the size of the compressed artifact, we need to know the build cache key. We find this in the build scan by clicking on the task.

The build cache stores artifacts by default in the ~/.gradle/caches/build-cache-1 directory where the name of the file is the build cache key, for example:

du -h ~/.gradle/caches/build-cache-1/b59e646b297605c55f49a251bdd7814
 20K

Calculating a typical build

We want to know if the build cache is worth it in a real scenario like assembling our app. To make this calculation, we would divide the amount of time a build would take by the total amount of compressed artifacts that the build produces. For the build time, we should only include the execution phase of the Gradle build. We exclude configuration time since tasks are neither executed nor downloaded during that time.

We run into a bit of a problem when trying to find the total size of all the compressed build cache artifacts. A Gradle build may run hundreds if not thousands of tasks. There is no public API to get the build cache key for a Gradle task. The only way to find the cache key is to manually look at a build scan. If we had to do that for 1000 tasks, we would be here all day.

In order to provide a real estimate, we will dive in to the internals of Gradle in part 2 of this post.

If you want to run an automatic estimation in your project, apply the Gradle Doctor plugin. Then simply run any gradle task with -PbenchmarkRemoteCache.

./gradlew :app:assembleDebug -PbenchmarkRemoteCache

You’ll get a nice report like the one below:

=================Gradle Doctor Prescriptions========================
= Remote Build Cache Benchmark Report =
Executed tasks created compressed artifacts of size 159,93 MB
Total Task execution time was 208,85 s

In order for a remote build cache to save you time, you would need an internet connection to your node of at least 0,77 MB/s.
Check a build scan to see your connection speed to the build cache node.
Build cache node throughput may be different than your internet connection speed.

A 1 MB/s connection would save you 48,92 s.                                      A 2 MB/s connection would save you 128,88 s.                
A 10 MB/s connection would save you 192,86 s.
                                                                                                Note: This is an estimate. Real world performance may vary. This estimate does not take in to account time spent decompressing cached artifacts or roundtrip communication time to the cache node.   ====================================================================

Let me know how it works for you and your workload. Is using the remote build cache worth it?

Thank you to Júlio Zynger for proofreading this post.

How fast does my internet need to be to use the Gradle Remote Build Cache? (Part 1) was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

This blog post dives in to a few reasons why builds might behave differently in Android Studio vs the terminal.

Injected Properties

Android Studio build is built with Gradle just like any other build. But it adds some additional information to each build by injecting some special project properties.

You can see all the properties that are injected by Android Studio by adding the following to your build.gradle.

for (def prop : project.properties) {
  if (prop.key.startsWith('android.injected')) {
    println(prop)
  }
}

This will yield the following properties when invoked from the IDE.

android.injected.invoked.from.ide=true
android.injected.build.abi=x86
android.injected.build.density=420dpi
android.injected.build.api=28

android.injected.invoked.from.ide=true

This property is set to true when running a build from the IDE. The property does not exist when running from the terminal. You can use this to tag your build scans.

buildScan {
  if (project.hasProperty("android.injected.invoked.from.ide")) {
   tag 'AndroidStudio'
  } else {
   tag 'Cmd'
  }
}

If you don’t use build scans, you can send this information to whatever tool you use to keep track of build speeds in order to determine if Android Studio builds are faster than terminal builds for your project.

The Android Gradle Plugin (AGP) looks at the other properties to try to avoid unnecessary work and optimize the build.

android.injected.build.abi=x86

If you have native libraries (C/C++), theExternalNativeBuildTask in AGP will read the injected ABI property to skip building unnecessary binaries. In this caseandroid.injected.build.abiis set to =x86 so we skip building binaries for ARM.

android.injected.build.api=28

Using the injected API property, in this case 28, the Android Gradle Plugin can make dexing optimizations for Instant Run. This property is used in InstantRunPatchingPolicy. AGP can use a faster patching policy if the target device is using ART instead of Dalvik.

Even if Instant Run isn’t running, AGP can upgrade from legacy multi-dex to native multidex in TaskManager if the project’s min sdk < 21 but the targeted device’s min SDK > 21.

// Upgrade from legacy multi-dex to native multi-dex if possible when using with a device
if (dexingType == DexingType.LEGACY_MULTIDEX) {
    if (variantScope.getVariantConfiguration().isMultiDexEnabled()
            && variantScope
                            .getVariantConfiguration()
                            .getMinSdkVersionWithTargetDeviceApi()
                            .getFeatureLevel()
                    >= 21) {
        dexingType = DexingType.NATIVE_MULTIDEX;
    }
}

android.injected.build.density=420dpi

Using the density property, AGP find the right split APK by screen density in SplitOutputMatcher. A smaller APK will strip out unnecessary resources in order to reduce APK transfer and installation time.

Instant Run

Android Studio also injects a special property when invoking Instant Run.

android.optional.compilation=INSTANT_DEV

This property is defined in the enumOptionalCompilationStep. AGP will use the Instant Run mode optimized for the target device’s SDK version by looking at the android.injected.build.api injected property.

Instant Run disables Gradle’s build cache. Gradle’s build cache stores the outputs of tasks and builds to reuse them later and avoid rebuilding things which are already built. Instant run completely ignores the Gradle build cache. Because of this, building from the IDE and switching branches means you could end up spending a lot of time rebuilding.

I would dive deeper in to Instant Run, but I think it isn’t worth spending too much time on since it was removed in Android Gradle Plugin 3.5.0. You won’t have to worry about it slowing down your build anymore.

If you are using Android Gradle Plugin 3.4.0, you can ensure that your fellow developers are not plagued by slow or inconsitent Instant Run builds by adding the following code to your build.gradle:

def instantRun = project.getProperties().get("android.optional.compilation")?.contains("INSTANT_DEV")

if (instantRun) {

  throw new IllegalStateException("Disable Instant Run: Android Studio -> Preferences -> Build, Execution, Deployment -> Instant Run")

}

Built-in JDK

Android Studio comes bundled with a JDK and uses it by default. Your terminal is likely pointed to a different JDK installation. A different JDK makes the Gradle Daemon incompatible between Android Studio and the terminal which breaks incremental compilation. If you notice really slow builds when switching back and forth it might be caused by this incompatibility.
To test for the incompatibility you can switch from Android Studio to the terminal. If you see the following message you may have an incompatibility.

Starting a Gradle Daemon, 1 incompatible and 0 stopped Daemons could not be reused, use — status for details

Svyatoslav Chatchenko’s post covers how to make your Daemon compatible.

The instructions have changed a bit for Android Studio 3.5.0. You need to open Project Structure -> SDK location.

You can ignore the message that says “To use the same Gradle daemon…select JAVA_HOME.” In Android Studio 3.5.0, this will not work.

When you open the dropdown, selecting the JAVA_HOME option will give you the same JDK location as the embedded JDK. This is a known issue in Android Studio.
Because of this bug, you’ll need to copy paste your JAVA_HOME in the setting. You can obtain your JAVA_HOME with the following command:

echo $(/usr/libexec/java_home)

Once you have entered your JAVA_HOME you shouldn’t get an error message indicating that the Gradle Daemons are incompatible when switching between Android Studio and the terminal. If you still get incompatibilities, you can find more information on how to debug it here.

Which is faster?

I didn’t really answer the question of which is faster. The thing is, depending on the project configuration, the size of the project, and many other factors one might be faster than the other. It’s important to minimize the differences to ensure consistent builds. It’s also important to measure build speeds. These are just a few differences between building from Android Studio and the terminal. There may be more and things have changed in Android Studio 3.5.

At SoundCloud, builds from the terminal and Android Studio for the assembleDebug task in our main app were both just about 58 seconds on average in the last 28 days.

Edit: A couple people have pointed out that this is a clickbait-y title and I didn’t provide sufficient metrics in this post. Thank you for challenging me. It made me realize that even the assembleDebug metrics I provided don’t capture the full picture. Ideally we should also measure app install time because Instant Run optimizes the installation process by stripping out unnecessary pieces from the APK. However we cannot compare installDebug in this case because Android Studio doesn’t use the installDebug task. It instead installs the APK by directly calling ADB. This makes it a bit more difficult to compare the two builds. If anyone has thoughts or ideas on how to better measure this, please let me know. I’ll keep digging.

Thanks to Tom Sammons for inspiring and proofreading this post.
Thanks to John Rodriguez for this gist to easily examine AGP sources and also proofreading this blog post.

Need some help improving Gradle build speeds? I’m available for help!
Did I help improve your build speed? Buy me a chocolate. nom nom

First we can define a boolean in our Gradle build script.

def useAgpRC = (project.properties['useAgpRC'] ?: 'false').toBoolean()

This sets the property disabled by default. Use true instead to enable it by default.

You can enable the property by running a build with -PuseAgpRC=true

You can also add useAgpRC=true to your gradle.properties.

Individual developers on your team can opt in by adding useAgpRC=true to ~/.gradle/gradle.properties.

This special Gradle Properties file affects all Gradle projects on your machine but has a lower precendence than the gradle.properties in the project.

When enabled, we switch to use the Android Gradle Plugin 3.5 Release Candidate.

if (useAgpRC) {
  classpath 'com.android.tools.build:gradle:3.5.0-rc02'
} else {
  classpath 'com.android.tools.build:gradle:3.4.2'
}

This is what our buildscript dependency block looks like putting all that together.

buildscript {

 dependencies {

  def useAgpRC = (project.properties['useAgpRC'] ?: 'false').toBoolean()

  // To enable AGP 3.5 add `useAgpRC=true` to your
  // `~/.gradle/gradle.properties`.

  if (useAgpRC) {
    println('Using AGP 3.5 RC')
    classpath 'com.android.tools.build:gradle:3.5.0-rc02'
  } else {
    println('Using AGP 3.4 Stable')
    classpath 'com.android.tools.build:gradle:3.4.2'
  }
}

If you are using the Gradle Remote Build Cache and seeding it on CI machines. Make sure to seed the cache with -PuseAgpRC=true and -PuseAgpRC=false.

If using Build Scans, you can also tag your build with

buildScan { 
  value "AGP RC", useAgpRC.toString() 
}

Was that cool? Fund my chocolate addiction.

I’m also available for Gradle help.

Dagger 2.23 introduces the HasAndroidInjector interface. It reduces some boilerplate in your Application class. Here is how you can destroy some of that boilerplate.

You might have an Application class that looks like this:

class MyApplication : Application(), HasActivityInjector, HasServiceInjector, HasBroadcastReceiverInjector, HasSupportFragmentInjector

And therefore you will also have some of these:

@Inject lateinit var androidActivityInjector : DispatchingAndroidInjector<Activity>

@Inject lateinit var androidFragmentInjector : DispatchingAndroidInjector<Fragment>

@Inject lateinit var androidBroadcastReceiverInjector : DispatchingAndroidInjector<BroadcastReceiver>

@Inject lateinit var androidServiceInjector : DispatchingAndroidInjector<Service>

And you also have to implement those Has*Injector interfaces like so:

override fun activityInjector(): AndroidInjector<Activity> = androidActivityInjector

override fun serviceInjector(): AndroidInjector<Service> = androidServiceInjector

override fun broadcastReceiverInjector(): AndroidInjector<BroadcastReceiver> = androidBroadcastReceiverInjector

override fun supportFragmentInjector(): AndroidInjector<Fragment> = androidFragmentInjector

With Dagger 2.23, you can replace those 4 interfaces with just one:

class DaggerExampleApplication : Application(), HasAndroidInjector

And instead injecting 4 different injectors, you can just inject one:

@Inject lateinit var androidInjector : DispatchingAndroidInjector<Any>

Now you just have one method to implement:

override fun androidInjector(): AndroidInjector<Any> = androidInjector

Check out this sample commit performing this boilerplate reduction.

If you already extend from DaggerApplication, you will not need to make any changes.

Click here for the rest of the Dagger 2.23 release notes.

Thanks to the Dagger team for incorporating the feedback that dagger has a lot of boilerplate and setup!

P.S. Use Dagger 2.24 is the latest version.

Did I help save you some time? Fund my chocolate addiction.
I’m also available for Gradle help.