Testableapple

Retrying Android Tests and Clearing the Database Between the Runs

2025-03-12T00:00:00+00:00

Intro

While great minds are arguing with each other about whether it is better to disable flaky tests or to retry them, I’d like to show you the way how you can achieve the latter.

This is for the brave, so do it at your own risk! 🤠

Implementing retry strategy

Let’s create two classes:

Retry - this one will be able to retry any specific test
RetryRule - this one will be responsible for the whole test suite

import org.junit.rules.TestRule
import org.junit.runner.Description
import org.junit.runners.model.Statement

/** Annotation to retry a specific failed test. **/
@Retention(AnnotationRetention.RUNTIME)
@Target(AnnotationTarget.FUNCTION)
public annotation class Retry(val count: Int)

/** Rule to retry all failed tests. **/
public class RetryRule(private val count: Int) : TestRule {

    override fun apply(base: Statement, description: Description): Statement = statement(base, description)

    private fun statement(base: Statement, description: Description): Statement {
        return object : Statement() {
            @Throws(Throwable::class)
            override fun evaluate() {
                val retryAnnotation: Retry? = description.getAnnotation(Retry::class.java)
                val retryCount = retryAnnotation?.count ?: count
                var caughtThrowable: Throwable? = null

                for (i in 0 until retryCount) {
                    try {
                        System.err.println("${description.displayName}: run #${(i + 1)} started.")
                        base.evaluate()
                        return
                    } catch (t: Throwable) {
                        System.err.println("${description.displayName}: run #${(i + 1)} failed.")
                        caughtThrowable = t
                    }
                }

                throw caughtThrowable ?: IllegalStateException()
            }
        }
    }
}

Then let’s create two simple test cases that will hopelessly try to assert the inequality of two different numbers:

import org.junit.Assert.assertEquals
import org.junit.Rule
import org.junit.Test

abstract class BaseTestClass {

    @get:Rule
    val retryRule = RetryRule(count = 3)
}

class SampleTest : BaseTestClass() {

    @Test // <~~~~~~ This test have 3 attempts to pass
    fun testNewFancyRetryRule() {
        assertEquals(33, 42)
    }

    @Retry(count = 4)
    @Test // <~~~~~~ This test have 4 attempts to pass
    fun testNewFancyRetryAnnotation() {
        assertEquals(42, 33)
    }
}

Clearing database between retries

More than likely, you would like the environment to be as clean as possible for each retry.

To achieve this, let’s implement the DatabaseOperations class, which will be responsible for clearing the database between each retry of the test:

import android.database.sqlite.SQLiteDatabase
import androidx.test.platform.app.InstrumentationRegistry
import java.io.File

public open class DatabaseOperations {

    public open fun clearDatabases() {
        val databaseOperations = DatabaseOperations()
        val dbFiles = databaseOperations.getAllDatabaseFiles().filterNot { shouldIgnoreFile(it.path) }
        dbFiles.forEach { clearDatabase(it, databaseOperations) }
    }

    private fun shouldIgnoreFile(path: String): Boolean {
        val ignoredSuffixes = arrayOf("-journal", "-shm", "-uid", "-wal")
        return ignoredSuffixes.any { path.endsWith(it) }
    }

    private fun clearDatabase(dbFile: File, dbOperations: DatabaseOperations) {
        dbOperations.openDatabase(dbFile).use { database ->
            val tablesToClear = dbOperations.getTableNames(database).filterNot { it == "room_master_table" }
            tablesToClear.forEach { dbOperations.deleteTableContent(database, it) }
        }
    }

    private fun getAllDatabaseFiles(): List<File> {
        return InstrumentationRegistry.getInstrumentation().targetContext.let { context ->
            context.databaseList().map { context.getDatabasePath(it) }
        }
    }

    private fun openDatabase(databaseFile: File): SQLiteDatabase {
        return SQLiteDatabase.openDatabase(databaseFile.absolutePath, null, 0)
    }

    private fun getTableNames(sqLiteDatabase: SQLiteDatabase): List<String> {
        sqLiteDatabase.rawQuery("SELECT name FROM sqlite_master WHERE type IN (?, ?)", arrayOf("table", "view"))
            .use { cursor ->
                val tableNames = ArrayList<String>()
                while (cursor.moveToNext()) {
                    tableNames.add(cursor.getString(0))
                }
                return tableNames
            }
    }

    private fun deleteTableContent(sqLiteDatabase: SQLiteDatabase, tableName: String) {
        sqLiteDatabase.delete(tableName, null, null)
    }
}

Thus, the statement function of the RetryRule class will undergo some changes:

private fun statement(base: Statement, description: Description): Statement {
    return object : Statement() {
        @Throws(Throwable::class)
        override fun evaluate() {
            val retryAnnotation: Retry? = description.getAnnotation(Retry::class.java)
            val retryCount = retryAnnotation?.count ?: count
            val databaseOperations = DatabaseOperations() // <~~~~~~ new line
            var caughtThrowable: Throwable? = null

            for (i in 0 until retryCount) {
                try {
                    System.err.println("${description.displayName}: run #${(i + 1)} started.")
                    base.evaluate()
                    return
                } catch (t: Throwable) {
                    System.err.println("${description.displayName}: run #${(i + 1)} failed.")
                    databaseOperations.clearDatabases() // <~~~~~~ new line
                    caughtThrowable = t
                }
            }

            throw caughtThrowable ?: IllegalStateException()
        }
    }
}

TLDR

To retry, or not to retry, that is the question. Even though it remains unanswered, you can find the full sample code here 🙂

Implementing Test Coverage in a Multi-module Android Project with Sonar and JaCoCo/Kover

2025-01-12T00:00:00+00:00

Intro

For a long time, setting up test coverage in Android projects has been a quite complex and frustrating process. Developers often found themselves wading through a rabbit hole of conflicting information, scrolling the docs, surfing the net, scouring the forums and blog posts in search of elusive solutions. What worked seamlessly for one dev might fail spectacularly for another, leading to countless hours and days of trial and error. This was the reality during the era of JaCoCo.

However, the landscape has shifted dramatically with the arrival of Kover, a Kotlin-first coverage plugin that simplifies the process and makes reliable test coverage setups more accessible than ever before. Kover addresses many of the pain points that JaCoCo users faced, providing a streamlined and intuitive approach to achieving robust test coverage metrics.

In this article, I’ll guide you through a practical and proven setup for implementing test coverage in a multi-module Android project using both JaCoCo and Kover. While I’ll walk you through the configuration of both tools (and you’ll likely notice a stark difference in how much simpler Kover is), I strongly recommend adopting Kover for its modern features and developer-friendly approach. As a bonus, I’ll show how to upload the results of both to SonarCloud.

So, let’s dive straight into it!

Set up Sonar

In short, Sonar is a continuous inspection tool. From the test coverage perspective, Sonar will be used as a host platform that displays the metrics and updates them as soon as they change.

File: sonar.gradle

apply plugin: "org.sonarqube"

ext.sonar = [
    ignoreModules : [
        // add here any module you want to ignore
        "documentation",
        "demo"
    ],
    excludeFilter : [
        // add here any regex you want to ignore
        "**/test/**",
        "**/androidTest/**",
        "**/R.class",
        "**/R2.class",
        "**/R$*.class",
        "**/BuildConfig.*",
        "**/Manifest*.*",
        "**/*Test*.*"
    ]
]

ext.sonar.ignoreModules.each {
    ext.sonar.excludeFilter << "**/${it}/**"
}

sonarqube {
    properties {
        property("sonar.host.url", "https://sonarcloud.io")
        property("sonar.token", "${System.getenv("SONAR_TOKEN")}")
        property("sonar.organization", "YOUR ORGANIZATION NAME")
        property("sonar.projectKey", "YOUR PROJECT KEY")
        property("sonar.projectName", "YOUR PROJECT NAME")
        property("sonar.java.coveragePlugin", "jacoco") // this value stays the same for both jacoco and kover
        property("sonar.sourceEncoding", "UTF-8")
        property("sonar.java.binaries", "${rootDir}/**/build/tmp/kotlin-classes/debug")
        property("sonar.coverage.exclusions", rootProject.ext.sonar.excludeFilter)
    }
}

File: build.gradle.kts

apply(from = "${rootDir}/sonar.gradle")

Set up Test Coverage

From here, we branch into two realities: dark and bright. Feel free to pick your path and skip the other.

JaCoCo a.k.a Old-school

File: jacoco.gradle

if (!rootProject.ext.sonar.ignoreModules.contains(name)) {
    apply plugin: "jacoco"
    apply plugin: "org.sonarqube"

    def sources = "src/main/kotlin"
    def testTask = "testDebugUnitTest"
    def jacocoResults = "${buildDir}/reports/jacoco/reportDebug.xml"

    if (hasProperty("android")) {
        android {
            buildTypes {
                debug {
                    testCoverageEnabled = true
                    enableUnitTestCoverage = true
                    enableAndroidTestCoverage true
                }
            }
        }
    }

    afterEvaluate {
        tasks.withType(Test).configureEach {
            jacoco.includeNoLocationClasses = true
            jacoco.excludes = [
                "jdk.internal.*",
                "androidx.core.*",
                "com.android.*",
                "android.*"
            ]
        }

        tasks.register("testCoverage", JacocoReport) {
            dependsOn testTask

            reports {
                xml.required.set(true)
                xml.outputLocation.set(file(jacocoResults))
            }

            executionData.setFrom(fileTree(dir: buildDir, includes: [
                "outputs/unit_test_code_coverage/debugUnitTest/${testTask}.exec"
            ]))

            sourceDirectories.setFrom(files([
                sources
            ]))

            classDirectories.setFrom(files([
                fileTree(
                    dir: "${buildDir}/tmp/kotlin-classes/debug",
                    excludes: rootProject.ext.sonar.excludeFilter
                )
            ]))
        }
    }

    sonarqube {
        properties {
            property "sonar.junit.reportPaths", "${buildDir}/test-results/${testTask}"
            property "sonar.coverage.jacoco.xmlReportPaths", jacocoResults
            property "sonar.sources", sources
        }
    }
}

File: build.gradle.kts

plugins {
    id("org.sonarqube")
}

apply(from = "${rootDir}/sonar.gradle")

subprojects {
    apply(from = "${rootDir}/jacoco.gradle")
}

Project structure

Terminal

./gradlew :first-module:testDebugUnitTest \
          :first-module:testCoverage \
          :second-module:testDebugUnitTest \
          :second-module:testCoverage \
          sonar

This command will run tests for all modules in the Debug build variant, generate an XML report using JaCoCo, and upload both test and code coverage results to SonarCloud.

Kover a.k.a New-school

🚧 Keep in mind that, while Kover worked like a charm for me, it is still in beta at the time of writing this article.

File: kover.gradle

if (!rootProject.ext.sonar.ignoreModules.contains(name)) {
    apply plugin: "org.jetbrains.kotlinx.kover"
    apply plugin: "org.sonarqube"

    if (hasProperty("android")) {
        android {
            buildTypes {
                debug {
                    testCoverageEnabled = true
                    enableUnitTestCoverage = true
                    enableAndroidTestCoverage true
                }
            }
        }
    }

    sonarqube {
        properties {
            property "sonar.junit.reportPaths", "${buildDir}/test-results/testDebugUnitTest"
            property "sonar.coverage.jacoco.xmlReportPaths", "${buildDir}/reports/kover/reportDebug.xml" // this key stays the same for both jacoco and kover
            property "sonar.sources", "src/main/kotlin"
        }
    }
}

File: build.gradle.kts

plugins {
    id("org.sonarqube")
    id("org.jetbrains.kotlinx.kover")
}

apply(from = "${rootDir}/sonar.gradle")

subprojects {
    apply(from = "${rootDir}/kover.gradle")
}

Project structure

Terminal

./gradlew :first-module:koverXmlReportDebug \
          :second-module:koverXmlReportDebug \
          sonar

This command will run tests for all modules in the Debug build variant, generate an XML report using Kover, and upload both test and code coverage results to SonarCloud.

Resources

Optimizing Android Test Runs: Parsing Test Names and Implementing Orchestration

2024-12-02T00:00:00+00:00

Intro

Today I’d like to talk about test orchestration on Android. And about parsing test names in particular, as one of the main pitfalls along the way.

Imagine you have hundreds or thousands of tests that take more than an hour in a raw. You better start thinking how to speed them up or run them in parallel.

There are multiple options to parallel the tests:

Parallelization at the CI level: when you split the tests and run them on different jobs (e.g. via GitHub Actions matrix)
Parallelization at the test runner level: when you split the tests and run them on different devices as part of a single CI job (some test runners are smart and offer this out of the box, some are dumb and should be executed in different threads)
Parallelization at the test framework level: really depends on the framework, but usually this means splitting the tests into suites
<…>

Thus, most options require a list of test names or test classes in order to run them in parallel. But how can we parse the tests, get their names, and not miss a single one? On Android, Dex Test Parser comes to the rescue.

All the code examples below are presented using fastlane lanes for the simplicity. Feel free to extract the main idea and translate it to your favourite language/DSL if required.

Precondition

I assume you already have an apk with your tests and an apk with your test target. If not, you can do something like this:

./gradlew assembleDebugAndroidTest assembleDebug

or using fastlane:

lane :build do
  gradle(tasks: ['assembleDebugAndroidTest', 'assembleDebug'])
end

Install test parser

First of all, let’s install dex-test-parser. According to the docs, this can be done using Maven, Gradle or manually. I choose the latter option:

wget -O test-parser.jar 'https://linkedin.jfrog.io/artifactory/open-source/com/linkedin/dextestparser/parser/2.2.1/parser-2.2.1-all.jar'

or using fastlane:

lane :install_test_parser do
  version = '2.2.1'
  output = 'test-parser.jar'
  url = "https://linkedin.jfrog.io/artifactory/open-source/com/linkedin/dextestparser/parser/#{version}/parser-#{version}-all.jar"
  sh("wget -O #{output} '#{url}' 2>/dev/null") unless File.exist?(output)
end

Parse the tests

Now let’s parse the tests using dex-test-parser:

java -jar test-parser.jar "${path_to_test_apk_path}" ./

This produces an AllTests.txt file that contains all absolute test paths.

Batch the tests

To batch the tests we need to know the total amount of batches and what is the current batch number:

lane :batch_tests do |options|
  current_batch = []
  if options[:batch_number] && options[:batch_count]
    sh("java -jar test-parser.jar #{options[:test_apk_path]} ./")
    test_names = File.read('AllTests.txt').split
    current_batch = test_names.each_slice((test_names.size.to_f / options[:batch_count].to_i).ceil).to_a[options[:batch_number].to_i]
    UI.success("Contents of the current batch: #{current_batch}")
  end
  current_batch
end

For instance, we have 3 batches in total and we want to get the tests for the first batch:

bundle exec fastlane batch_tests batch_count:3 batch_number:0 test_apk_path:"path/to/test.apk"

Keep in mind that the batches will always be identical and equally divided. If new tests are added, they will be distributed proportionally between the batches at the upcoming run.

Service the tests

To be able to run the tests with pre-built apks we have to install test services:

lane :install_test_services do
  device_api_level = sh('adb shell getprop ro.build.version.sdk').strip.to_i
  force_queryable = device_api_level >= 30 ? '--force-queryable' : ''
  version = '1.5.0'

  output = 'test-services.apk'
  url = "https://dl.google.com/dl/android/maven2/androidx/test/services/test-services/#{version}/test-services-#{version}.apk"
  sh("wget -O #{output} '#{url}' 2>/dev/null") unless File.exist?(output)
  sh("adb install #{force_queryable} -r #{output}")

  output = 'orchestrator.apk'
  url = "https://dl.google.com/dl/android/maven2/androidx/test/services/orchestrator/#{version}/orchestrator-#{version}.apk"
  sh("wget -O #{output} '#{url}' 2>/dev/null") unless File.exist?(output)
  sh("adb install #{force_queryable} -r #{output}")
end

Run the tests

Let’s sum it up and add the cherry on top:

lane :test do
  install_test_services
  install_test_parser

  apk_path = "path/to/app.apk"
  test_apk_path = "path/to/test.apk"

  sh("adb install -r #{apk_path}")
  sh("adb install -r #{test_apk_path}")

  current_test_batch = batch_tests(
    batch_number: options[:batch_number],
    batch_count: options[:batch_count],
    test_apk_path: test_apk_path
  )

  app_package_name = 'com.example'
  test_package_name = "#{app_package_name}.test"
  test_runner_package_name = 'androidx.test.runner.AndroidJUnitRunner'
  test_orchestrator_package_name = 'androidx.test.orchestrator/.AndroidTestOrchestrator'
  androidx_test_services_path = sh('adb shell pm path androidx.test.services').strip

  result = sh(
    "adb shell 'CLASSPATH=#{androidx_test_services_path}' " \
    'app_process / androidx.test.services.shellexecutor.ShellMain am instrument -w '\
    '-e clearPackageData true ' \
    "-e targetInstrumentation #{test_package_name}/#{test_runner_package_name} " \
    "-e class #{current_test_batch.join(',')} #{test_orchestrator_package_name}"
  )
  UI.user_error!('Tests have failed!') if result.include?('Failures')
end

Showcase

On GitHub Actions it would look like this:

name: CI

on:
  push:

jobs:
  build:
    name: Build
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4.2.2
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: 3.1
          bundler-cache: true
      - uses: actions/setup-java@v4
        with:
          distribution: adopt
          java-version: 17
      - name: Enable KVM group permissions
        run: |
          echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
          sudo udevadm control --reload-rules
          sudo udevadm trigger --name-match=kvm
      - run: bundle exec fastlane build
      - uses: actions/upload-artifact@v4.4.3
        with:
          name: apks
          path: |
            ./**/path/to/app.apk
            ./**/path/to/test.apk

  test:
    name: Test
    runs-on: ubuntu-24.04
    needs: build
    strategy:
      matrix:
        include:
          - batch_number: 0
          - batch_number: 1
          - batch_number: 2
    steps:
      - uses: actions/checkout@v4.2.2
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: 3.1
          bundler-cache: true
      - uses: actions/setup-java@v4
        with:
          distribution: adopt
          java-version: 17
      - uses: actions/download-artifact@v4.1.8
        with:
          name: apks
      - name: Enable KVM group permissions
        run: |
          echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
          sudo udevadm control --reload-rules
          sudo udevadm trigger --name-match=kvm
      - uses: reactivecircus/android-emulator-runner@v2
        with:
          api-level: 33
          disable-animations: true
          profile: pixel
          arch : x86_64
          emulator-options: -no-snapshot-save -no-window -no-audio -no-boot-anim -gpu swiftshader_indirect -camera-back none -camera-front none
          script: bundle exec fastlane test batch_count:$ batch_number:$
      - uses: actions/upload-artifact@v4.4.3
        if: failure()
        with:
          name: report
          path: ./**/build/reports/androidTests/*

If you want to randomize things a bit, consider:

moving the tests’ parsing step to the Build job
shuffling test names in AllTests.txt file
uploading it as an artifact alongside the apks
downloading it as part of the matrix step

After that, it should be pretty much the same as the initial plan.

Resources

Profiling app size on iOS

2024-08-26T00:00:00+01:00

Intro

Today I’d like to show you how to do a proper size analysis of an App or an SDK on iOS.

I’ll try to be as short as possible, so fasten the seat belts 🤠

Building the app

Let’s create a new empty app and integrate some third-party SDK via SPM. For example StreamChatSwiftUI.

Open Xcode
Create New Project... -> iOS App -> Product name: `Size`; Interface: `SwiftUI` -> Next -> Create
File -> Add Package Dependencies -> `stream-chat-swiftui` -> Add Package -> Add to Target: `Size` -> Add Package
Make sure the `DEBUG_INFORMATION_FORMAT` build setting is set to `DWARF with dSYM File` for more detailed analysis
Select a development team in the Signing & Capabilities editor
Product -> Archive -> `Size.xcarchive`

Profiling the size

This is where we get to a crossroads, because there are a bunch of ways to get the size of a binary.

We will look at some of them, from the simplest to those that offer the most comprehensive analysis.

Disk Usage CLI

Measure your app

 app_path="Size.xcarchive/Products/Applications/Size.app"
 size_in_mb=$(echo "scale=2; $(du -sk "${app_path}" | cut -f1) / 1024" | bc)
 echo "$size_in_mb MB"

Where:

 * `du` is an abbreviation for "Disk Usage"
 * `bc` is an abbreviation for "Basic Calculator" or "Bash Calculator"
 * `1024` is the number of KB in MB
 * `scale=2` is an argument to round the result to required number of decimal places
 * `-sk` is an argument to get the size of a file/folder in KB
 * `cut -f1` is an argument to get rid of the file path in the output

Result:

 13.23 MB

Measure your binary

 app_binary_path="Size.xcarchive/Products/Applications/Size.app/Size"
 size_in_mb=$(echo "scale=2; $(du -sk "${app_binary_path}" | cut -f1) / 1024" | bc)
 echo "$size_in_mb MB"

Result:

 12.08 MB

Bloaty

Install bloaty, a size profiler for binaries, crafted by Google
```
 brew install bloaty
```

Profile you binary

 app_binary_path="Size.xcarchive/Products/Applications/Size.app/Size"
 dsym_binary_path="Size.xcarchive/dSYMs/Size.app.dSYM/Contents/Resources/DWARF/Size"
 bloaty ${app_binary_path} --debug-file=${dsym_binary_path} -d compileunits -n 0

Where:

  * `-d compileunits` is an argument to get app size split per file
  * `-n 0` is an argument to specify the maximum fetch depth

Result:

     FILE SIZE       VM SIZE
  --------------  --------------
1%  1004Ki   7.9%  1004Ki    [__TEXT,__text]
8%   837Ki   6.7%   848Ki    [__LINKEDIT]
4%   293Ki   2.3%   293Ki    /Library/Developer/Xcode/DerivedData/Size-chfnnfxxtxfvxabrjimspcolpjyk/SourcePackages/checkouts/stream-chat-swift/Sources/StreamChat/WebSocketClient/Events/MessageEvents.swift
4%   292Ki   2.3%   292Ki    [__DATA_CONST,__const]
4%   291Ki   2.3%   291Ki    /Library/Developer/Xcode/DerivedData/Size-chfnnfxxtxfvxabrjimspcolpjyk/SourcePackages/checkouts/stream-chat-swiftui/Sources/StreamChatSwiftUI/ChatChannelList/SearchResultsView.swift
0%       0   2.1%   261Ki    [__DATA,__bss]
9%   234Ki   1.9%   234Ki    [__TEXT,__const]
9%   231Ki   1.8%   231Ki    [__DATA,__data]
6%   198Ki   1.6%   198Ki    [__TEXT,__swift5_typeref]
5%   184Ki   1.5%   184Ki    /Library/Developer/Xcode/DerivedData/Size-chfnnfxxtxfvxabrjimspcolpjyk/SourcePackages/checkouts/stream-chat-swiftui/Sources/StreamChatSwiftUI/ChatChannelList/ChatChannelNavigatableListItem.swift
4%   170Ki   1.4%   170Ki    /Library/Developer/Xcode/DerivedData/Size-chfnnfxxtxfvxabrjimspcolpjyk/SourcePackages/checkouts/stream-chat-swift/Sources/StreamChat/ChatClient.swift
    ...
    ...
    ...
0%  12.1Mi 100.0%  12.3Mi    TOTAL

Emerge Tools

And finally, let’s see the magic Emerge Tools is capable of 🪄

Zip your app's xcarchive: `zip -r Size.zip Size.xcarchive`
Sign up for a free account at emergetools.com
Verify your email
Open emergetools.com/dashboard
Uploads -> Upload a local build -> Drag your `Size.zip`
Click on the `View build analysis` link
Point to any module to find out its size or search for any filename, class name, etc...

Postscript

If you want to get an idea of how a third-party SDK affects the size of your app, just do the same without importing an SDK and compare the results.

Running iOS Performance Testing on CI

2024-07-31T00:00:00+01:00

Intro

The proof of concept we’ll discuss today goes a bit beyond the boundaries of what XCTest Performance offers out of the box. And that’s only because the standard tooling has limitations, which we’ll try to detail shortly.

Although if you have a self-hosted runner and a farm of physical devices that you gonna use for Performance testing then all good and you’re covered, just follow the official docs.

Otherwise, grab some 🍿 and enjoy reading.

How it’s supposed to work

Write the test
Run it the first time to capture the baseline
Run it again at some point to compare the current result with the baseline

Existing limitations

The idea behind XCTMetric is great and the API is really gorgeous, but as I mentioned there are quite some limitations. And the most frustrating thing of all is the lack of information about these boundaries.

The impression after reading the docs leaves you with a thought that it’s all rosy, just write these lightweight tests and run them wherever and whenever you wish. But, unfortunately, that’s not entirely true.

Target device model

You can’t use test results with baseline if baseline was recorded on another device model.
Running machine

You can’t use test results with baseline if baseline was recorded on another machine.
Metrics availability

The availability of metrics depends drastically on the architecture of the test device.
1. Available metrics on iOS Simulator:
  - Duration (a completely useless metric)
2. Available metrics on Physical Device:
  - Duration
  - Frame Count
  - Frame rate
  - Hitch time ratio
  - Hitches total duration
  - Number of hitches
To sum up

You have to run performance tests on the same machine and on the same physical device.

Sounds reasonable, isn’t it? But it kind of cuts off at the root the possibility of running these tests on CI. Unless, as I mentioned in the very beginning, you have a self-hosted runner and a physical device connected to it.

Push the boundaries

I assume you already have some Performance tests. And so what I offer is to:

Forget about the native Xcode baseline and create a JSON file with a custom baseline
- e.g.: {"duration":3,"frame_rate":75,"hitches":0,...}
Set up a Firebase TestLab account
- it allows to use Physical Devices at no-cost up to 30 min/day or 5 runs/day depending on the plan (which I think is grand for our task)
Choose one device and always run the performance tests on it
Parse the logs and get the metrics’ results
- the file name is xcodebuild_output.log
Compare the result with the custom baseline from the first step
- keep in mind that the Xcode’s default baseline threshold was 10%, adjust yours accordingly

This way:

Your CI will be waiting for the tests to be completed
The tests will always pass on Firebase TestLab due to the absence of a baseline
And then you will do your own math and fail the job if required

Here is an example of what it might look like (within fastlane):

lane :run_xctmetric do
  # Get your custom baseline
  expected_performance = JSON.parse(File.read('YOUR_BASELINE_CONFIG.json'))

  # Build the app and xctestrun
  scan(
    project: 'YOUR_PROJECT_PATH',
    scheme: 'YOUR_SCHEME_NAME',
    testplan: 'YOUR_PERFORMANCE_TESTPLAN_NAME',
    result_bundle: true,
    derived_data_path: 'derived_data/',
    sdk: 'iphoneos',
    skip_detect_devices: true,
    build_for_testing: true
  )

  Dir.chdir('../derived_data/Build/Products') do
    # Zip all the test-related schtuff
    sh('zip -r MyTests.zip .')

    # Upload the tests to Firebase TestLab and wait for the completion
    sh('gcloud firebase test ios run --test MyTests.zip --timeout 7m --results-dir test_output --device "model=iphone14pro,version=16.6,orientation=portrait"')

    # Download the logs from Google Cloud
    sh('gsutil cp -r gs://YOUR_FIREBASE_TESTLAB_BUCKET_ID/test_output/iphone14pro-16.6-en-portrait/xcodebuild_output.log xcodebuild_output.log')

    # Parse the logs and extract the performance metrics
    actual_performance = extract_xctmetric_result(log: File.read('xcodebuild_output.log'))

    # Compare the result with the custom baseline
    success = do_your_own_math(expected_performance, actual_performance)

    if success
      UI.success('🟢 Performance benchmark passed.')
    else
      UI.user_error!('🔴 Performance benchmark failed.')
    end
  end
end

private_lane :extract_xctmetric_result do |options|
  metrics = {}

  # Collect the metrics for all performance tests one by one
  ['test_YourFirstTestName', 'test_YourSecondTestName'].each do |test_name|
    hitches_total_duration = options[:log].match(/#{test_name}\]' measured \[Hitches Total Duration \(Scroll_DraggingAndDeceleration\), ms\] average: (\d+\.\d+)/)
    duration = options[:log].match(/#{test_name}\]' measured \[Duration \(Scroll_DraggingAndDeceleration\), s\] average: (\d+\.\d+)/)
    hitch_time_ratio = options[:log].match(/#{test_name}\]' measured \[Hitch Time Ratio \(Scroll_DraggingAndDeceleration\), ms per s\] average: (\d+\.\d+)/)
    frame_rate = options[:log].match(/#{test_name}\]' measured \[Frame Rate \(Scroll_DraggingAndDeceleration\), fps\] average: (\d+\.\d+)/)
    number_of_hitches = options[:log].match(/#{test_name}\]' measured \[Number of Hitches \(Scroll_DraggingAndDeceleration\), hitches\] average: (\d+\.\d+)/)

    metrics[test_name] = {
      'hitches_total_duration' => {
        'value' => hitches_total_duration[1].to_f.round(2),
        'ext' => 'ms'
      },
      'duration' => {
        'value' => duration[1].to_f.round(2),
        'ext' => 's'
      },
      'hitch_time_ratio' => {
        'value' => hitch_time_ratio[1].to_f.round(2),
        'ext' => 'ms per s'
      },
      'frame_rate' => {
        'value' => frame_rate[1].to_f.round(2),
        'ext' => 'fps'
      },
      'number_of_hitches' => {
        'value' => number_of_hitches[1].to_f.round(2),
        'ext' => ''
      }
    }
  end

  metrics
end

And here is an example of the custom baseline JSON file:

{
  "test_YourFirstTestName": {
    "hitches_total_duration": {
      "value": 10,
      "ext": "ms"
    },
    "duration": {
      "value": 2.6,
      "ext": "s"
    },
    "hitch_time_ratio": {
      "value": 4,
      "ext": "ms per s"
    },
    "frame_rate": {
      "value": 75,
      "ext": "fps"
    },
    "number_of_hitches": {
      "value": 1,
      "ext": ""
    }
  },
  "test_YourSecondTestName": {
    "hitches_total_duration": {
      "value": 10,
      "ext": "ms"
    },
    "duration": {
      "value": 2.6,
      "ext": "s"
    },
    "hitch_time_ratio": {
      "value": 4,
      "ext": "ms per s"
    },
    "frame_rate": {
      "value": 75,
      "ext": "fps"
    },
    "number_of_hitches": {
      "value": 1,
      "ext": ""
    }
  }
}

How to install old iOS Simulator Runtimes from Command-line

2024-06-30T00:00:00+01:00

Introduction

There came a day when we needed to run Unit or E2E tests on older iOS versions on CI. Sounds like a piece of cake. What could go wrong?

Let’s see what options we have.

xcode-install a.k.a xcversion

⛔️ Deprecated now. But back then it was a no-brainer.

$ gem install xcode-install
$ xcversion simulators --install='iOS 13.7'

###############################           82%
######################################## 100%
Successfully installed iOS 13.7 Simulator

xcodes

Picked up the baton, but… being broken for a few weeks every time Apple updates the API.

$ brew install xcodesorg/made/xcodes || mint install XcodesOrg/xcodes
$ sudo xcodes runtimes install "iOS 14.5"

Downloading Runtime iOS 13.7: 82%
Downloading Runtime iOS 13.7: 100%
Mounting DMG
Expanding pkg
Expanded pkg into /Users/runner/Library/Caches/XcodeInstall/com.apple.pkg.iPhoneSimulatorSDK14_5
Unmounting DMG
Setting package installation location
Rebuilding package
Please authenticate to install iOS 14.5 Simulator...
Successfully installed iOS 14.5 Simulator

ipsw

Easily handles the runtimes’ download, but when it comes to installation, it’s not all rosy. Let’s use it to download the runtime and combine it with some custom script to install the downloaded image.

$ brew install blacktop/tap/ipsw
$ echo "iOS 15.5 Simulator" | ipsw download xcode --sim

Downloading file=iOS_15.5_Simulator_Runtime.dmg
  4.35 GiB / 5.04 GiB [=============>-----| 33s ] 38.69 MiB/s
  5.04 GiB / 5.04 GiB [===================| ✅  ] 37.54 MiB/s

$ dmg_file=$(ls *.dmg)
$ sh install_runtime.sh "${dmg_file}"

Mounting iOS_15.5_Simulator_Runtime.dmg on /var/folders/rl/swm2ljqn187g5y81d6mp8h840000gn/T/tmp.NFC0jygSj0...
Checksumming Driver Descriptor Map (DDM : 0)
Checksumming  (Apple_Free : 1)
Checksumming Apple (Apple_partition_map : 2)
Checksumming disk image (Apple_HFS : 3)
Checksumming  (Apple_Free : 4)
Detected packaged runtime.
Found package /var/folders/rl/swm2ljqn187g5y81d6mp8h840000gn/T/tmp.NFC0jygSj0/iPhoneSimulatorSDK15_5.pkg (sdk iPhoneSimulatorSDK15_5).
Expanding package /var/folders/rl/swm2ljqn187g5y81d6mp8h840000gn/T/tmp.NFC0jygSj0/iPhoneSimulatorSDK15_5.pkg to /var/folders/rl/swm2ljqn187g5y81d6mp8h840000gn/T/tmp.Y3CN2N4Wgv/expanded...
Rewriting package install location to /Library/Developer/CoreSimulator/Profiles/Runtimes/iPhoneSimulatorSDK15_5.simruntime...
Re-assembling the package /var/folders/rl/swm2ljqn187g5y81d6mp8h840000gn/T/tmp.Y3CN2N4Wgv/iPhoneSimulatorSDK15_5.pkg...
Installing /var/folders/rl/swm2ljqn187g5y81d6mp8h840000gn/T/tmp.Y3CN2N4Wgv/iPhoneSimulatorSDK15_5.pkg...
installer: Package name is iPhoneSimulatorSDK15_5
installer: Upgrading at base path /
installer: The upgrade was successful.
Installed iOS 15.5.
Removing /var/folders/rl/swm2ljqn187g5y81d6mp8h840000gn/T/tmp.Y3CN2N4Wgv...
Unmounting /var/folders/rl/swm2ljqn187g5y81d6mp8h840000gn/T/tmp.NFC0jygSj0...
"disk12" ejected.
Removing /var/folders/rl/swm2ljqn187g5y81d6mp8h840000gn/T/tmp.NFC0jygSj0...

Conclusion

Turns out dealing with older runtimes is not that easy and obvious. All above come with pros and cons and decision is yours to take. Here I’ve just shared the options.

By the way, keep in mind that packaged iOS Simulator runtimes (everything below 16) are not supported on hosts after macOS 13.3.99. So juggling the macOS and Xcode versions is also on the table 🤠

Postscript

If you’re struggling to make it work on GitHub Actions, check out this tweet of mine, hope it will help you out.

Managing credentials and environment variables within Xcode and XCTest

2023-07-16T00:00:00+01:00

Credentials, what can be easier, right? Just get them from .env file or whatever and you’re good to go. Well, not so fast… not on iOS.

To be honest, it was a bit challenging to make it work from Xcode and from the Terminal at the same time. Passing credentials through the command-line only is not a problem at all, but as soon as you want to run these tests from Xcode it takes a ridiculous turn.

So, if you happen to be dealing with credentials in native tests on iOS, I’m afraid you’re limited to the following options:

Hard-code credentials in the test code
Pass credentials through a scheme or an xctestplan

Let’s pretend that the first option is not an option at all and jump straight to the second one.

Precondition

Create an environment variable in your xctestplan:

Open the .xctestplan file
Click on the Configurations tab
Arguments section → Env Vars row → ➕ button
Add a new variable with the name AWESOME_TOKEN and the value $(AWESOME_TOKEN)

Create a sample test that prints the value of our secret token:

import XCTest

class SampleTestCase: XCTestCase {

    func testToken() {
      let awesomeToken = ProcessInfo.processInfo.environment["AWESOME_TOKEN"]!
      print(awesomeToken)
      XCTAssertFalse(awesomeToken.isEmpty)
    }
}

How to manage it

in Xcode

Create env.xcconfig file in your project
- File → New → File… → Configuration Settings File
- Type the name of the file: env.xcconfig
- Choose the location of the file
- Choose the target with your tests
Attach env.xcconfig to the tests’ target
- Click on your project name in the Project Navigator
- Open Info tab
- Expand the Debug configuration in the Configurations section
- Add env.xcconfig file to the app target for UI tests or to the test target for Unit tests
Add the AWESOME_TOKEN variable to the env.xcconfig file
```
 AWESOME_TOKEN = 1234567890
```
Add env.xcconfig file to .gitignore!
Run the test 🤌

PS: you might want to share this file with your team so that they can run the tests as well.

on CI

Create a secret variable in your CI (e.g.: GitHub Actions)
Create an empty env.xcconfig file on the fly so that xcodebuild command doesn’t fail
```
 touch env.xcconfig
```

Run the test 🤌

 xcodebuild test \
   -project "Sample.xcodeproj" \
   -scheme "Sample" \
   -destination "platform=iOS Simulator,name=iPhone 14" \
   AWESOME_TOKEN=${{ secrets.AWESOME_TOKEN }} | xcpretty

Managing Xcode project files programmatically

2023-06-21T00:00:00+01:00

Let’s say you have a code generation tool that adds and deletes files from the project’s sources.

In addition to physically doing so, Xcode requires updating the project file to reflect these changes, i.e. adding and deleting file references.

Here is where the tricky part starts because this project.pbxproj file is an endless old-style ASCII property list that is not meant to be edited manually. So, how do we deal with it?

Xcodeproj to the rescue

There is one good old friend called Xcodeproj that can help us with that. It’s a Ruby gem made by CocoaPods team that allows us to programmatically manipulate Xcode project files.

gem install xcodeproj

In my humble opinion, the documentation is not its strong suit. It took me almost a day to build and test the script below. I’m even a little jealous of you. Anyways, fasten the belt and follow the comments. Let’s jump right into it.

Picturesque script

# import the gem
require 'xcodeproj'

# create a function that selects all swift files that start with a given prefix
def select_swift_files_from(files:, that_start_with:)
  files.select do |f|
    f.start_with?(*that_start_with)
  end.map do |f|
    f.split.drop(1).join
  end.select do |f|
    f.end_with?('.swift')
  end
end

# create an Xcode project object
project = Xcodeproj::Project.open(xcode_project)

# list all changed files
changed_files = `git status -s`.split("\n").map(&:strip)

# select all deleted swift files
deleted_swift_files = select_swift_files_from(files: changed_files, that_start_with: 'D')

# ...added ones
added_swift_files = select_swift_files_from(files: changed_files, that_start_with: ['A', '??'])

# ...renamed ones
renamed_swift_files = select_swift_files_from(files: changed_files, that_start_with: 'R')

# distribute renamed swift files between deleted and added ones
renamed_swift_files.each do |renamed_file|
  content = renamed_file.split.drop(1).join.split('->').map(&:strip)
  deleted_swift_files << content.first
  added_swift_files << content.last
end

deleted_swift_files.each do |file_path|
  # get the file reference
  file = project.files.find { |f| f.full_path.to_s == file_path }

  # delete the reference only if it exists
  file.remove_from_project if file
end

added_swift_files.each do |file_path|
  # determine if the file reference already exists
  file_reference_exists = project.files.find { |f| f.full_path.to_s == file_path }

  # move on to the next file if it does
  next if file_reference_exists

  # assign the main group to a variable
  group = project.main_group

  # represent the file path as an array of subfolders
  split_file_path = file_path.split('/')

  # get the file name and remove it from the array
  file_name = split_file_path.pop

  split_file_path.each do |subfolder|
    if group[subfolder]
      # move on to the next subfolder if the current one already has a reference
      group = group[subfolder]
    else
      # create a new Xcode group if the current subfolder doesn't have a reference
      group = group.new_group(subfolder)

      # for some reason Xcodeproj sets the full path for the newly created group,
      # whereas Xcode expects it to be just a folder name ¯\_(ツ)_/¯
      group.path = subfolder
    end
  end

  # create a new file reference
  file_reference = group.new_file(file_path)

  # once again, replace the full path in file reference to a file name
  file_reference.path = file_name

  # add the file reference to the first (main) target's `Compile Sources` build phase
  # even though the main target will work for most cases, you might want to replace it with a specific one
  project.targets.first.source_build_phase.add_file_reference(file_reference)
end

# save the project
project.save

Afterword

If you’re a Swift guy, you’d probably like to rewrite everything with XcodeProj made by Tuist, and I think that should be fun.

Let me know if you do, and share the source code, I’d love to see and try it out 🤠

How to access app logs from UI Tests on iOS

2023-06-04T00:00:00+01:00

There is no secret that the UI tests are pretty limited in terms of accessing app internals. That’s why sometimes you have to craft some hackish workarounds to bypass these limitations.

The app logs are one of those things that would be quite useful to have access to from UI tests to observe any events that happen in the app or whatnot.

Even though there is nothing out of the box that could help us, let’s see how we can achieve this after all.

Sample app

As an example, I created a simple app that has an image view and logs a message when it’s tapped:

import SwiftUI

struct ContentView: View {

    var body: some View {
        VStack {
            VStack {
                Image(systemName: "photo")
                    .font(.system(size: 150))
                    .accessibilityIdentifier("imageView")
                    .onTapGesture {
                        print("Image was tapped!")
                    }
            }
        }
        .frame(maxWidth: .infinity, maxHeight: .infinity)
        .background(Color.brown)
    }
}

@main
struct SampleApp: App {
    var body: some Scene {
        WindowGroup {
            ContentView()
        }
    }
}

That’s how our picturesque sample app looks like:

LogView injection

To access the logs from UI tests, we need to inject a view that’ll display them for us. But don’t worry:

it will be there only in debug builds
it will be completely transparent
so basically, nothing changes in UI, the tree hierarchy is where the magic happens

struct ContentView: View {
    @State private var log: String = ""

    var body: some View {
        VStack {
            VStack {
                Image(systemName: "photo")
                    .font(.system(size: 150))
                    .accessibilityIdentifier("imageView")
                    .onTapGesture {
                        let event = "Image was tapped!"
                        print(event)
                        log += event
                    }
                    #if DEBUG
                    .overlay {
                        Text(log)
                            .accessibilityIdentifier("logView")
                            .foregroundColor(.clear)
                    }
                    #endif
            }
        }
        .frame(maxWidth: .infinity, maxHeight: .infinity)
        .background(Color.brown)
    }
}

Sample test

Now, let’s access the logs from the sample UI test and assert that the event was logged:

import XCTest

final class SampleUITests: XCTestCase {

    func testExample() throws {
        let app = XCUIApplication()
        app.launch()

        app.images["imageView"].tap()
        XCTAssertTrue(app.staticTexts["logView"].label.contains("Image was tapped!"))
    }
}

Finally, that’s how the tree hierarchy snapshot looks like:

Element subtree:
 →Application, 0x153d0e440, pid: 61338, label: 'SampleApp'
    Window (Main), 0x153d10f20, ((0.0, 0.0), (393.0, 852.0))
      Other, 0x153d0aa00, ((0.0, 0.0), (393.0, 852.0))
        Image, 0x153d07220, ((110.0, 371.0), (173.0, 135.0)), identifier: 'imageView', label: 'Photo'
        StaticText, 0x153d07340, ((124.5, 417.3), (144.0, 42.3)), identifier: 'logView', label: 'Image was tapped!'

Afterword

Hope you got the idea. Making the LogView available across all app screens requires some boilerplate code or some smart architecture, so I’ll leave it up to you. Happy hacking! 🤠

Page Object Model with Maestro

2023-05-21T00:00:00+01:00

Page Object Model (POM) is a design pattern, used in test automation that creates Object Repository for UI elements.

Even though the word «Page» comes from the web world, the pattern applies to any UI, including mobile and desktop apps. Maybe think of it as a «Screen Object Model». Sounds a bit more applicable to mobile, doesn’t it?

Anyways, the main idea is to create an object for each screen of the app and encapsulate all the elements of that screen within the object. Thus these elements become reusable and easy to maintain.

Basic stuff, isn’t it? But take Maestro, for instance, which DSL has a YAML syntax, just like your configs and pipelines (which is, by the way, its killer feature). Not that obvious anymore, is it? So, let’s see how to implement POM with this beautiful framework.

Preconditions

Install Maestro:

 brew install facebook/fb/idb-companion
 curl -Ls "https://get.maestro.mobile.dev" | bash

Pick up an app for the sample test (e.g.: native Settings app)
The platform makes no difference in this particular exercise, so let’s stick with iOS

Collect screen elements

Run an iOS Simulator
Open the Settings app
Run Maestro Studio to collect some screen elements for the sample test
```
 maestro studio
```

Sample test w/o POM

flow.yaml

appId: com.apple.Preferences
---

- launchApp
- tapOn:
    id: "General"
- assertVisible:
    id: "About"

maestro

$ maestro test flow.yaml
 ║
 ║  > Flow
 ║
 ║    ✅  Launch app "com.apple.Preferences"
 ║    ✅  Tap on id: General
 ║    ✅  Assert that id: About is visible
 ║

Sample test with POM

page_objects.js

output.settingsPage = {
  generalLabel: 'General',
  safariLabel: 'Safari',
  // ...
};

output.generalPage = {
  aboutLabel: 'About',
  keyboardLabel: 'Keyboard',
  // ...
};

flow.yaml

appId: com.apple.Preferences
---

- runScript: page_objects.js
- launchApp
- tapOn:
    id: ${output.settingsPage.generalLabel}
- assertVisible:
    id: ${output.generalPage.aboutLabel}

maestro

$ maestro test flow.yaml
 ║
 ║  > Flow
 ║
 ║    ✅  Launch app "com.apple.Preferences"
 ║    ✅  Tap on id: General
 ║    ✅  Assert that id: About is visible
 ║

Disabling network on iOS Simulator from XCTest

2023-05-07T00:00:00+01:00

Offline support is one of the most complex features not only to implement but also to test.

In this post, I’ll show you how to disable the network on iOS Simulator from XCTest. Fasten your seat belts, it can open up a whole new world for your automated test cases.

Options

How can we test «Offline support»? There are a few options:

To modify the source code so you can control the «network» from the app (in DEBUG mode only)
- This is not always possible, and it may not be 100% appropriate to the real network conditions
To use a real iOS device instead and turn on/off Airplane mode
- Fair point, but I doubt you’re here for this option
To be a hacker
- It’s always fun, and sometimes it’s the only option 🤠

Becoming a hacker

The iOS Simulator does not have a built-in option to disable the network. Moreover, it relies on macOS network settings. And therefore, we need to disable the network on macOS in the first place.

Long story short, we need to access the Command-line from XCTest to disable the network on macOS and thus on iOS Simulator. All the magic is behind the native networksetup CLI:

for service in ["Ethernet", "Wi-Fi", "iPhone USB"] {
    exec("networksetup -setnetworkserviceenabled '\(service)' \(state) || true") // "state" can be "on" or "off"
}

Proof of concept

Create an HTTP server (I’m using express as an example)

Command-line:

  npm install express
  touch server.js
  open server.js

Source code:

  // server.js

  const process = require("child_process");
  const express = require("express");
  const app = express();
  const port = 4567;

  app.use(express.json());

  app.listen(port);

  app.post("/terminal", (req, res) => {
    const output = exec(req.body.command, req.query.async).toString("utf8").trim();
    res.send(output);
  });

  function exec(command, async) {
    if (async === "true") {
      return process.exec(command);
    } else {
      return process.execSync(command);
    }
  }

Create a class that will be responsible for everything around Command-line:

 class CommandLine {

     private static let host = "http://localhost"
     private static let port: UInt16 = 4567

     enum ConnectionState: String {
         case on
         case off
     }

     static func setConnection(state: ConnectionState) {
         let network = ["Ethernet", "Wi-Fi", "iPhone USB"]
         for service in network {
             exec("networksetup -setnetworkserviceenabled '\(service)' \(state) || true")
         }
     }

     @discardableResult
     private static func exec(_ command: String, async: Bool = false) -> String {
         let urlString = "\(host):\(port)/terminal?async=\(async)"
         guard let url = URL(string: urlString) else { return "" }

         var request = URLRequest(url: url)
         request.httpMethod = "POST"
         request.setValue("application/json", forHTTPHeaderField: "Content-Type")
         request.httpBody = try? JSONSerialization.data(withJSONObject: ["command": command], options: [])
         var output = ""

         if async {
             // Do not wait for the command to complete
             URLSession.shared.dataTask(with: request).resume()
         } else {
             // Wait for the command to complete
             let semaphore = DispatchSemaphore(value: 0)
             let task = URLSession.shared.dataTask(with: request) { data, response, error in
                 if let data = data, let string = String(data: data, encoding: .utf8) {
                     output = string
                     semaphore.signal()
                 }
             }
             task.resume()
             semaphore.wait()
         }
         return output
     }
 }

Create a sample test case that will:

disable the network on iOS Simulator
verify that the network is disabled
make sure that the network is enabled again after the test is finished

   import XCTest

   class SampleUITests: XCTestCase {

       let safari = XCUIApplication(bundleIdentifier: "com.apple.mobilesafari")

       override func setUp() {
           super.setUp()
           safari.launch()
           _ = safari.wait(for: .runningForeground, timeout: 10)
       }

       override func tearDown() {
           super.tearDown()
           CommandLine.setConnection(state: .on)
       }

       func testOfflineScenario() {
           CommandLine.setConnection(state: .off)

           open(url: "https://testableapple.com/")
           let msg = safari.staticTexts.matching(NSPredicate(format: "label CONTAINS 'your iPhone is not connected to the Internet'"))
           let networkDisabled = msg.firstMatch.waitForExistence(timeout: 10)

           XCTAssertTrue(networkDisabled, "Network should be disabled")
       }

       func open(url: String) {
           safari.textFields["TabBarItemTitle"].tap()
           safari.textFields["URL"].typeText(url + XCUIKeyboardKey.return.rawValue)
       }
   }

Now let’s run the server:
```
 node server.js
```
And the test case itself (either from Xcode or Terminal)

Pros

Testing of «Offline support» can be automated

Cons

Disabling the network on macOS obviously will affect everything on this machine that depends on the network connection
Running tests in parallel on the same machine might be no longer possible. Consider extracting these tests in their own xctestplan / scheme and run them separately

Sample project

Check out this GitHub repo for more details 🙂

Gaining access to Command-line from Maestro

2023-04-23T00:00:00+01:00

What’s this?

The command line is quite a powerful tool. It can be used to automate a bunch of things here and there. Obviously, it’s not only about Shell scripts, you can run any other tool from it as well. But using it directly is not always possible.

Recently, I showed how to gain access to the command line from XCTest. This time it’s Maestro’s turn, as it also works inside a sandbox (more precisely, a JavaScript sandbox). Let’s bypass this limitation and learn how to gain access to the command line from Maestro 🎹

So, basically, the same dish is on the menu today, just with a slightly different serving 👨‍🍳

And this is useful because…?

This uncovers a whole new world of simulator/emulator-related features. For example:
This opens up the possibility to create and use a standalone mock server that can completely replace the real backend for the testing purposes
This allows executing any macOS UI actions through Applescript from your Maestro tests
This gives the option to automate web-related actions via playwright or selenium instead of opening Safari/Chrome on iOS/Android
This offers a pretty flexible way to export anything you want from the test to the host machine (screenshots, mocks, test data, etc.)
And much much more 🤠

Cool, what’s the trick?

Even though Maestro does support a minimal subset of vanilla JavaScript APIs, it does not allow to access the filesystem. But we can still use the HTTP protocol to communicate with the host machine.

So, let’s create an HTTP server that will be listening for requests. Then we can send a request from Maestro to the server and run whatever we want. The server can even return the output back to Maestro if required.

Okay, show me how to do this

Proof of concept

GIVEN we have a server running on localhost:4567
AND we sent a request to /terminal endpoint from Maestro with the desired command
WHEN server receives the request
AND server executes the command
THEN server returns the result back to Maestro
AND Maestro verifies the result

Walkthrough

Install Maestro

 brew install facebook/fb/idb-companion
 curl -Ls "https://get.maestro.mobile.dev" | bash

Choose any web framework of your choice (they are pretty much all the same for our use case)

 {
   "js": "express", // https://github.com/expressjs/express
   "rb": "sinatra", // https://github.com/sinatra/sinatra
   "py": "flask", // https://github.com/pallets/flask
   "go": "gin", // https://github.com/gin-gonic/gin
   "..": "..." // ...
 }

Create an HTTP server (I picked up { "js": "express" } as an example)

Command-line:

  npm install express
  touch server.js
  open server.js

Source code:

  // server.js

  const process = require("child_process");
  const express = require("express");
  const app = express();
  const port = 4567;

  app.use(express.json());

  app.listen(port);

  app.post("/terminal", (req, res) => {
    const output = exec(req.body.command, req.query.async).toString("utf8").trim();
    res.send(output);
  });

  function exec(command, async) {
    if (async === "true") {
      return process.exec(command);
    } else {
      return process.execSync(command);
    }
  }

Run the server
```
 node server.js
```

Create a tiny layer between the server and the test

 // terminal.js

 var response = http.post(`http://localhost:4567/terminal?async=${async}`, {
   headers: { "Content-Type": "application/json" },
   body: JSON.stringify({ command: exec })
 })

 output.terminal = response.body;

Create the test file

 # test.yaml

 appId: ${app_id}
 ---

 - launchApp
 - runScript:
     file: terminal.js
     env:
       exec: "sw_vers -productName"
       async: false
 - assertTrue: ${output.terminal == "macOS"}

Launch the emulator/simulator

Run the test ✅

on iOS:

  udid="7E0929F4-CFB5-41E7-9943-85CA9370EE51" # This is a unique iOS Simulator id
  app_id="com.apple.Preferences" # This is a Settings app, just as an example
  maestro --udid ${udid} test --env app_id=${app_id} test.yaml

on Android:

  udid="emulator-5554" # This is a default id for the first running Android Emulator
  app_id="com.android.settings" # This is a Settings app, just as an example
  maestro --udid ${udid} test --env app_id=${app_id} test.yaml

Adjust the scripts to your needs and enjoy hacking!

Does this work on real devices?

Yup, but not on iOS, at least at the moment.

Alrighty, does this work on CI?

Sure thing, works like a charm!

Just make sure to run the server in the background on CI:

node server.js &

Thanks a mill!

No probs! Check out this GitHub repo for more details 🙂

Test-only accessibility values on iOS

2023-04-10T00:00:00+01:00

What is accessibility value?
It’s a localized string that contains the current value of an element. For example, the value of a slider might be 42% and the value of a text field is the text it contains.

When to use it?
Usually, it’s used only when an accessibility element can have a value that is not represented by its label.

Why to use it?
For example, a volume slider’s label might be «Volume» but its value is the current volume level. In this case, it’s not enough for users to know the identity of the slider, because they also need to know its current value.

What about «test-only» accessibility value?
Long story short, accessibility values are exposed in VoiceOver and other assistive technologies. So, if we want to rely on them in our tests, we have to ensure that no testing data is accessible to users. Here «test-only» thingy comes into play :)

Sample app

I created a super simple sample app with just one screen and one element.

Let’s pretend that this element represents a user’s authorization status. If the user is authorized, the element is green. Otherwise, it’s red.

This element is also a «switch» that changes the authorization status on tap event.

import SwiftUI

@main
struct SampleApp: App {

    var body: some Scene {
        WindowGroup {
            ContentView()
        }
    }
}

struct ContentView: View {

    @State private var isAuthorized: Bool = false

    var body: some View {
        Image(systemName: "circle.fill")
            .onTapGesture {
                isAuthorized = !isAuthorized
            }
            .foregroundColor(isAuthorized ? .green : .red)
            .accessibility(identifier: "statusIcon")
    }
}

Implementing test-only accessibility value

To archive this in SwiftUI we need to extend View class:

import SwiftUI

extension View {

    public func debugAccessibility(value: String) -> some View {
        #if DEBUG
        return self.accessibility(value: Text(value))
        #else
        return self
        #endif
    }
}

And implement this new function in ContentView:

struct ContentView: View {

    @State private var isAuthorized: Bool = false

    var body: some View {
        Image(systemName: "circle.fill")
            .onTapGesture {
                isAuthorized = !isAuthorized
            }
            .foregroundColor(isAuthorized ? .green : .red)
            .accessibility(identifier: "statusIcon")
            .debugAccessibility(value: isAuthorized ? "1" : "0")
    }
}

In case you’re using UIKit, then UIView class should be extended, something like this:

import UIKit

extension UIView {

    public func debugAccessibility(value: String) -> Self {
        #if DEBUG
        return self.accessibilityValue = value
        #endif
        return self
    }
}

Simply put, we’re using the DEBUG flag to make sure that this code is not included in the release build and is only used in the debug one. When this function is called in production, it will return the same view without any changes.

Checking test-only accessibility value

Alrighty, now, using XCTest, we can check out what it was all for:

import XCTest

final class SampleAppUITests: XCTestCase {

    func testAuthorizationStatus() {
        let app = XCUIApplication()
        app.launch()

        // Given the user is logged out
        let status = app.images["statusIcon"]
        XCTAssertEqual(status.value as? String, "0")

        // When the user taps the status icon
        status.tap()

        // Then the user is logged in
        XCTAssertEqual(status.value as? String, "1")

        // When the user taps the status icon again
        status.tap()

        // Then the user is logged out
        XCTAssertEqual(status.value as? String, "0")
    }
}

Cool, we can even create a new extension for XCUIElement to make our tests more readable:

import XCTest

extension XCUIElement {

    var isOn: Bool {
        (self.value as? String) == "1"
    }

    var isOff: Bool {
        (self.value as? String) == "0"
    }
}

It looks a lot better now, doesn’t it?:

func testAuthorizationStatus() {
    let app = XCUIApplication()
    app.launch()

    // Given the user is logged out
    let status = app.images["statusIcon"]
    XCTAssertTrue(status.isOff)

    // When the user taps the status icon
    status.tap()

    // Then the user is logged in
    XCTAssertTrue(status.isOn)

    // When the user taps the status icon again
    status.tap()

    // Then the user is logged out
    XCTAssertTrue(status.isOff)
}

Use cases

There are a bunch of use cases for «test-only» accessibility values. In fact, I showed you only one of them, even though it’s kind of the most common one.

Here’s a rough list of uses:

state - store the state of an element (e.g. on/off, enabled/disabled, etc.)
colour - store the colour name of an element (e.g. red, green, etc.)
shape - store the shape of an element (e.g. circle, square, etc.)
font - store the font of a text element (e.g. bold, italic, etc.)
whatever - yeah, this way, you can store almost any property you want as long as it can be represented as a string

In case you’re wondering, the same can be done with accessibility identifiers or any other accessibility properties. Happy hacking! 🤠

Painless BDD on iOS with XCTContext

2023-03-12T00:00:00+00:00

Do you like BDD?

To be honest, I don’t. I do like the idea behind it, but the implementation usually way too complicates things for no reason and makes it super painful to develop and support the automated testing frameworks.

People usually say it’s there to allow non-technical folks to read and write the automated tests on their own. Really? Okay. If something breaks under the hood, these folks will be the first ones to complain. And, yeah, they will be right. But it’s not the point. The point is that it’s not a good fit for automated tests.

There is one thing though that I really like about BDD. It makes the test scenario look like a story, which side effect is that everyone can understand it. GIVEN anything => WHEN something => THEN nothing, you know?

What if XCTest could offer something like this out of the box? It would be a win-win, wouldn’t it? Well, XCTest does offer this. It’s called XCTContext.

Let’s play with the following tiny test scenario:

GIVEN user opens chat
WHEN user sends message
THEN user observes new message

Then

I’ve seen multiple times on various projects that developers usually use comments to mimic BDD, something like:

import XCTest

class SampleTestCase: XCTestCase {
    func testExample() {
        func testExample() {
            let message = "test"

            // GIVEN user opens chat
            user
                .login()
                .tapOnChatButton()

                // WHEN user sends message
                .typeText(message)
                .tapOnSendButton()

                // THEN user observes new message
                .assertMessageContent(message)
                .assertMessageAuthor(user)
        }
    }
}

And this is not too bad if you want to avoid a complex BDD framework under the hood while still keeping the tests kinda readable.

However, the test report looks quite messy on success:

And a bit confusing on failure:

Now

I’ve already mentioned that XCTContext might help, so it will transform our test case into something like this:

func testExample() {
    let message = "test"

    GIVEN("user opens chat") {
        user
            .login()
            .tapOnChatButton()
    }
    WHEN("user sends message") {
        user
            .typeText(message)
            .tapOnSendButton()
    }
    THEN("user observes new message") {
        user
            .assertMessageContent(message)
            .assertMessageAuthor(user)
    }
}

It looks so much better — documentation as a code, isn’t it? It also helps to understand the test case structure either when you are developing, reviewing, or editing the automated test.

Besides, just look at this compact test report on success:

And on failure:

Multiply this by the number of steps a real test case usually has, and the difference in reports between the two approaches will be hilarious.

Under the hood

To make our beautiful «BDD framework» work we just need to extend the XCTest class and add a few new functions:

import XCTest

public extension XCTest {
    func GIVEN(_ name: String, actionStep: () -> Void) {
        step("GIVEN \(name)", step: actionStep)
    }

    func WHEN(_ name: String, actionStep: () -> Void) {
        step("WHEN \(name)", step: actionStep)
    }

    func THEN(_ name: String, actionStep: () -> Void) {
        step("THEN \(name)", step: actionStep)
    }

    func AND(_ name: String, actionStep: () -> Void) {
        step("AND \(name)", step: actionStep)
    }

    private func step(_ name: String, step: () -> Void) {
        XCTContext.runActivity(named: name) { _ in
            step()
        }
    }
}

And that’s it! We’ve literally just wrapped up the XCTContext.runActivity() into a small function step() and created the behaviour-driven keywords that we can use in our tests.

You may notice that I’ve added AND as a keyword as well. This is just a personal preference, but I like to use it in test scenarios that have more than 3 steps. To make this clear, just imagine our initial scenario with AND:

GIVEN user logs in
AND user opens chat
WHEN user types message
AND user sends message
THEN user observes new message
AND message belongs to user

Conclusion

I prefer to treat automated tests as documentation. And if I call it documentation, I have to keep it readable. Ideally without either outdated comments or BDD colliders.

XCTContext solves this issue perfectly. Just give it a try, and you’ll see how much easier it is to read, understand and maintain the automated tests.

Ruby within iOS development realm

2023-02-24T00:00:00+00:00

So, today, February 24th, Ruby turns 30 years old! It was first born in Japan back in 1993, thanks to Yukihiro «Matz» Matsumoto, and it’s been swiftly taking over the world of ours ever since!

It was designed with the idea that programming should be fun for programmers. It emphasizes the necessity for software to be understood by humans first and computers second. Even though right now there are a bunch of other languages that are faster/safer/more efficient, they lack the magic and charm of Ruby, you know?

Ruby has been and probably always will be my favourite language. It just brings me joy every time I use it. So yeah, take everything I say below with a pinch of salt =)

Anyways, I’d like to reflect on how Ruby has made a thing in the iOS development realm, became a role model and, at the same time, begun to lose the ground…

Are Ruby gems the actual gems?

Storytelling

Ruby is a language of careful balance. Its creator, Yukihiro «Matz» Matsumoto, blended parts of his favorite languages (Perl, Smalltalk, Eiffel, Ada, and Lisp) to form a new one that balanced functional programming with imperative programming.

He has often said:

I am trying to make Ruby natural, not simple, in a way that mirrors life.

Building on this, he adds:

Ruby is simple in appearance, but is very complex inside, just like our human body.

© ruby-lang.org

But what brought Ruby to iOS realm?

Well, it was the RubyCocoa framework which was started in 2001 by Hisakuni Fujimoto when he implemented a Ruby extension module to wrap NSObject and NSClassFromString function.

It provided a bridge between the Ruby and the Objective-C programming languages, allowing the user to manipulate Objective-C objects from Ruby, and vice versa.

In turn, it made it possible to write a Cocoa («SwiftUI of those days») application completely in Ruby as well as to write an application that mixes Ruby and Objective-C code.

An Apple project called MacRuby was under development to replace RubyCocoa in 2008. A proprietary spin-off called RubyMotion was subsequently released in 2012, available for iOS, macOS and Android. It was a commercial product that allowed developers to write native applications in Ruby.

I guess this literally untied the hands of indie contributors in the Apple community, and since then Ruby began to take a firm foothold in the iOS world, gradually introducing and developing new revolutionary tools and frameworks that were supposed to make the iOS development process easier and much more enjoyable.

I’m going to mention the most notable ones and briefly compare them with their native alternatives that have started joining the party in the last couple of years with the rise of Swift development.

Name-dropping

Automation in general

Ruby:

nomad-cli (2012-2021, archived)
fastlane (2014, still #1 automation tool for iOS development. Don’t forget to vote here, so it doesn’t end up in the Google graveyard)

Native alternatives:

xcodebuild (2003, it’s been included with the Xcode development environment since its first release. Unfortunately, super clumsy and complex)
Fastlane.swift (2017, still in beta. The wrapper on top of its Ruby ancestor)

puts "Ruby wins. Flawless victory! 🚀"
puts "fastlane has already done an incredible amount work to automate most of the iOS development workflows. It's really hard to compete with it. Fastlane.swift? Eh, will see..."

Package management

Ruby:

CocoaPods (2011)

Native alternatives:

Swift Package Manager (2017)

echo "Draw 🤝"
echo "Even though CocoaPods still is more common these days, it's clearly a legacy. Swift Package Manager has some limitations but already looks promising and starts to pull the blanket"

Management of executable packages

Ruby:

brew (2009)

Native alternatives:

Mint (2017)

echo "Draw 🤝"
echo "Even though brew is much more popular than Mint, it's already not that obvious in the iOS development realm"

Docs generation

Ruby:

jazzy (2014)

Native alternatives:

Swift-DocC (2021)

echo "Draw 🤝"
echo "This couple looks very similar to CocoaPods and SPM. One is mature and widespread, another one is more modern and simpler, but not as commonly used yet"

Logs decoration

Ruby:

xcpretty (2013)

Native alternatives:

xcbeautify (2018)

puts "Ruby wins 🎉"
puts "Almost every automation tool or script around iOS realm uses xcpretty. At least to date. Will see how it goes in the future"

Project management

Ruby:

Native alternatives:

XcodeGen (2017)
tuist (2018)

print("Swift wins 👍")
print("It wasn't an easy decision, but we have to face the music. Swift tools here are just too good to ignore")

Project modification

Ruby:

Xcodeproj (2011)

Native alternatives:

XcodeProj (2018)

echo "Draw 🤝"
echo "It was easy. One is copy-pasta of another. Both are pretty good, all depends on the use case"

Automation of Xcode installation

Ruby:

xcode-install (2015-2022, archived)

Native alternatives:

xcodes (2019)

print("Swift wins. Flawless victory! 🚀")
print("Community made a decision. xcode-install is not maintained anymore, xcodes is the way to go")

Test automation

Ruby:

calabash (2012-2020, archived)

Native alternatives:

XCTest (2013, it’s been included with the Xcode development environment since version 5.0)

print("Swift wins. Flawless victory! 🚂")
print("No comments, XCTest is the best 🤠")

Test coverage reports generation

Ruby:

xcov (2016)
slather (2014)

Native alternatives:

xccov (2018, it’s been included with the Xcode development environment since version 9.3)

puts "Ruby wins 🎉"
puts "These are a little different tools around the same thing. When xccov just parses and prints, slather and xcov generate and beautify"

Monkey testing

Ruby:

xcmonkey (2022)

Native alternatives:

SwiftMonkey (2016-2022, archived)

puts "Ruby wins 🥳"
puts "It was easy indeed, wasn't it? "

Summing-up

Domain	Ruby	Native alternatives
Automation in general	👍
Package management	👍	👍
Management of executable packages	👍	👍
Docs generation	👍	👍
Logs decoration	👍
Project modification		👍
Automation of Xcode installation		👍
Test automation		👍
Test coverage reports generation	👍
Monkey testing	👍
Whose birthday is this?	🎂	🥳

Thoughts

Apple let Ruby into its ecosystem of its own free will, and we can see firsthand how Ruby plugs holes within and around iOS workflows with its stunning frameworks and tooling for so many years in a row.

Time flies by, and Swift is gaining more and more traction and is becoming a more viable alternative to Ruby not only for direct iOS development, but also for development of executable and server-side tools. It is the future of iOS development and it’s only a matter of time before it becomes the future of iOS tooling as well.

With projects such as Tuist, which kind of took aim to copy-paste (and improve) the best of the best schtuff from Ruby to Swift, it becomes even more obvious. No matter how sad I am, it makes perfect sense. Swift is a more powerful language than Ruby and it’s much more convenient for iOS developers to use it for workflow automation as well as for software development.

After all, there is no smoke without fire:

Scripting language runtimes such as Python, Ruby, and Perl are included in macOS for compatibility with legacy software. Future versions of macOS won’t include scripting language runtimes by default, and might require you to install additional packages. If your software depends on scripting languages, it’s recommended that you bundle the runtime within the app.

© Apple

Anyways, I hope you enjoyed this post and found it helpful. If you, like me, don’t believe that Ruby will leave the iOS development realm anytime soon, and you’d like to improve your Ruby skills, I recommend you to play around with tools like:

They’re all great and will make your life so much easier while you cross the Rubicon and become a Rubyist 🤠

Beautifying XCTest results

2023-02-06T00:00:00+00:00

XCTest is an excellent tool for testing your apps on iOS. It’s fast, reliable, and easy to use. But the default test reports aren’t as pretty as they could be, especially for UI tests. In this post, I’ll show you how to make them look much better.

Quick intro

Usually, tests’ result representation has two «heads»:

trace in logs — basically test names with ✅ or ❌ prefixes
json/xml/xcresult reports — more detailed info that is packed into a file

Although the first option does show which tests are failing, and sometimes it even gives us an idea why, in some cases, it’s not enough because it lacks some important details such as screenshots, logs, videos, etc, which can lead to a lot of time spent on debugging.

The second one can give us a lot more if configured correctly but in a computer-readable format.

Let’s take as much as possible out of these two options and convert their output into a human-being-readable format.

Sample project

I created a sample project to show you how to beautify XCTest results from both logs’ and reports’ points of view. So if you’re in a hurry, just jump on this repo and copy-paste schtuff from there.

Beautifying the logs

There is one rock star tool that has been around for ages and is able to hide the useless parts of xcodebuild output for us so we can focus only on the important things. It’s called xcpretty.

gem install xcpretty

Just try to build or test any .xcodeproj using the naked xcodebuild command with and w/o xcpretty and you’ll see the difference.

Without xcpretty:

xcodebuild test -scheme XCTestReport -destination 'platform=iOS Simulator,name=iPhone 14,OS=16.2'

🙈

With xcpretty:

xcodebuild test -scheme XCTestReport -destination 'platform=iOS Simulator,name=iPhone 14,OS=16.2' | xcpretty

Beautiful, innit?

I won’t go into details about xcpretty here because its pros (and cons?) have been covered in many articles already. It’s a great tool, and I highly recommend using it, so you don’t strain your eyes.

Fun fact, if you’re using fastlane, it has xcpretty built-in. I bet you already know that, so let’s move on.

Beautifying the reports

Reporting is a bit more complicated thingy. There are a couple of popular tools for doing this job, for instance:

XCTestHTMLReport
Allure Report (XCTest integration tutorial)

Allure is cross-platform, XCTestHTMLReport is kind of «native», both are really great with their own advantages and disadvantages and both able to do what we’re up to. I’ll show you how to use both of them, in parallel, so you can choose the one you like more.

Installing XCTestHTMLReport is as easy as:

brew install xctesthtmlreport

Installing Allure Report is a bit more complicated, but still not rocket science:

brew install allure
DOWNLOAD_URL="https://github.com/eroshenkoam/xcresults/releases/download/1.13.1/xcresults"
curl -sL "${DOWNLOAD_URL}" -o ./xcresults
chmod +x ./xcresults

Let’s run the command we already know to execute the tests and generate .xcresult:
```
 xcodebuild test -scheme XCTestReport -destination 'platform=iOS Simulator,name=iPhone 14,OS=16.2' -resultBundlePath TestResult | xcpretty
```
- resultBundlePath argument takes the path where to save the tests’ result
- so passing -resultBundlePath TestResult to xcodebuild test command will create a TestResult.xcresult in the root folder
Now we can generate a report from the .xcresult file:

2.1. For XCTestHTMLReport:
```
 xchtmlreport TestResult.xcresult
```
- this command will create an index.html in the root folder
2.2. For Allure Report:
```
 ./xcresults export TestResult.xcresult allure-results
 allure generate allure-results
```
- these commands will create two folders in the root folder: allure-results and allure-report
Let’s open the report:

3.1. For XCTestHTMLReport:
```
 open index.html
```
Beautiful, innit?

3.2. For Allure Report:
```
 allure open allure-report
```
Gorgeous, innit?

All above on CI

Alrighty, let’s try to replicate all that on CI. I’ll use GitHub Actions for this purpose, but guess what, you can use almost any other CI service you want.

Let's start with the XCTestHTMLReport:

First of all, we need to create a workflow file. I’ll name it test.yml and put it into .github/workflows folder:

 name: Beautifying XCTest Results

 on:
   push:
     branches:
       - main

   workflow_dispatch:

 jobs:
   xctest_html_report:
     name: XCTestHTMLReport
     runs-on: macos-12
     steps:
       - uses: actions/checkout@v3.1.0

       - run: brew install xctesthtmlreport

       - run: xcodebuild test -scheme XCTestReport -destination 'platform=iOS Simulator,name=iPhone 14,OS=16.2' -resultBundlePath TestResult | xcpretty

       - run: xchtmlreport TestResult.xcresult

       - name: Deploy to GitHub Pages
         uses: peaceiris/actions-gh-pages@v3.9.2
         with:
           github_token: ${{ secrets.GITHUB_TOKEN }}
           publish_dir: ./

as you may have noticed, we’re installing XCTestHTMLReport as it’s not pre-installed unlike xcpretty, and then running the same commands we used locally
and then the third-party GitHub action peaceiris/actions-gh-pages is used to publish the report to GitHub Pages
see sample project for more details

PS: you might want to keep a history of test runs, so you can use the destination_dir: ${{ github.run_id }} option to publish the report to a separate folder for each run. Keep in mind that XCTestHTMLReport depends on *.xcresult which is super heavy and this will increase the size of the branch/repo, so you may need to clean up the old reports from time to time:

 - uses: peaceiris/actions-gh-pages@v3.9.2
   with:
     github_token: ${{ secrets.GITHUB_TOKEN }}
     publish_dir: ./
     destination_dir: ${{ github.run_id }}
     exclude_assets: '.github,XCTestReport.xcodeproj,XCTestReport,XCTestReportUITests'

Then let’s allow GitHub to publish the test reports for us. For this, we need to:
- create a branch called gh-pages:
- enable GitHub Pages for our project, Settings -> Pages:
- enable required workflow permissions, Settings -> Actions -> General -> Workflow permissions:
That’s basically it. Every time we run the tests on GitHub Actions, it will automatically update gh-pages branch and publish the test reports for us on GitHub Pages:

Example:
- https://testableapple.github.io/beautifying-xctest-results/4098130575/
- https://testableapple.github.io/beautifying-xctest-results/4098181045/

Let's implement Allure Report:

First of all, we need to add a new job to the existing workflow file .github/workflows/test.yml:

 allure:
   name: Allure Report
   runs-on: macos-12
   timeout-minutes: 20
   steps:
     - uses: actions/checkout@v3.1.0

     - run: |
         brew install allure
         DOWNLOAD_URL="https://github.com/eroshenkoam/xcresults/releases/download/1.13.1/xcresults"
         curl -sL "${DOWNLOAD_URL}" -o ./xcresults
         chmod +x ./xcresults

     - run: xcodebuild test -scheme XCTestReport -destination 'platform=iOS Simulator,name=iPhone 14,OS=16.2' -resultBundlePath TestResult | xcpretty

     - run: |
         ./xcresults export TestResult.xcresult allure-results
         allure generate allure-results

     - uses: actions/upload-artifact@v3
       with:
         name: Allure Report
         path: allure-report/

as you may have noticed, we’re installing allure and xcresults as they are not pre-installed unlike xcpretty, and then running the same commands we used locally
and then the native GitHub action actions/upload-artifact is used to upload the report to GitHub Actions artifacts
see sample project for more details

PS: we don’t publish the test report to GitHub Pages in this example. Consider using a third-party GitHub action simple-elf/allure-report-action if you’d like to achieve this.

That’s basically it. Every time we run the tests on GitHub Actions, it will automatically attach the test report to the action’s artifacts. Thus, we can download the report from there and open it locally as described above.

Example:
- https://github.com/testableapple/beautifying-xctest-results/actions/runs/4113524123

Quick outro

Setting up all the nice little things needed to make test results look gorgeous takes some time and effort, but it’s a one-time thingy and it’s worth it. This is a great way to make life easier for you and your team and make test results more readable and understandable.

Hope you enjoyed this post. See you in the next one! 🤠

Gaining access to Command-line from XCTest

2023-01-22T00:00:00+00:00

What’s this?

XCTest, for instance, works inside a sandbox, which means you literally can’t even stick your nose out of the box. Well, almost. Let’s bypass this limitation and learn how to gain access to the command line from tests.

And this is useful because…?

This uncovers a whole new world of simulator-related features that are available through simctl and idb but not through XCTest API yet. For example:
This opens up the possibility to create and use a standalone mock server that can completely replace the real backend for the testing purposes
This allows executing any macOS UI actions through Applescript from your XCTests
This gives the option to automate web-related actions via playwright or selenium instead of opening Safari on iOS
This offers a pretty flexible way to export anything you want from the test to the host machine (screenshots, mocks, test data, etc.)
And much much more 🤠

Cool, what’s the trick?

We need to create an HTTP server that will be listening for requests. Then we can send a request from XCTest to the server and run whatever we want. The server can even return the output back to XCTest if required.

Okay, show me how to do this!

Proof of concept

GIVEN we have a server running on localhost:4567
AND we sent a request to /terminal endpoint from XCTest with the desired command
WHEN server receives the request
AND server executes the command
THEN server returns the result back to XCTest
AND XCTest verifies the result

Walkthrough

Choose any web framework of your choice (they are pretty much all the same for our use case)

 {
   "js": "express", // https://github.com/expressjs/express
   "rb": "sinatra", // https://github.com/sinatra/sinatra
   "py": "flask", // https://github.com/pallets/flask
   "go": "gin", // https://github.com/gin-gonic/gin
   "..": "..." // ...
 }

Create an HTTP server (I picked up { "js": "express" } as an example)

Command-line:

  npm install express
  touch server.js
  open server.js

Source code:

  // server.js

  const process = require("child_process");
  const express = require("express");
  const app = express();
  const port = 4567;

  app.use(express.json());

  app.listen(port);

  app.post("/terminal", (req, res) => {
    const output = exec(req.body.command, req.query.async).toString("utf8").trim();
    res.send(output);
  });

  function exec(command, async) {
    if (async === "true") {
      return process.exec(command);
    } else {
      return process.execSync(command);
    }
  }

Run the server
```
 node server.js
```

Create the test file

 import XCTest

 class SampleUITests: XCTestCase {
     let host = "http://localhost"
     let port: UInt16 = 4567

     func testPlatformName() {
         let platformName = exec("sw_vers -productName")
         XCTAssertEqual(platformName, "macOS")
     }

     func exec(_ command: String, async: Bool = false) -> String {
         let urlString = "\(host):\(port)/terminal?async=\(async)"
         guard let url = URL(string: urlString) else { return "" }

         var request = URLRequest(url: url)
         request.httpMethod = "POST"
         request.setValue("application/json", forHTTPHeaderField: "Content-Type")
         request.httpBody = try? JSONSerialization.data(withJSONObject: ["command": command], options: [])
         var output = ""

         if async {
             // Do not wait for the command to complete
             URLSession.shared.dataTask(with: request).resume()
         } else {
             // Wait for the command to complete
             let semaphore = DispatchSemaphore(value: 0)
             let task = URLSession.shared.dataTask(with: request) { data, response, error in
                 if let data = data, let string = String(data: data, encoding: .utf8) {
                     output = string
                     semaphore.signal()
                 }
             }
             task.resume()
             semaphore.wait()
         }
         return output
     }
 }

Run the test ✅
Adjust the scripts to your needs and enjoy hacking!

Does this work on real devices?

Emm, yeah, but not out of the box. You need to make sure that the server is accessible from device.

There are two options to achieve this:

Connect to Wi-Fi and send requests to the server using the desktop IP address instead of localhost

Expose your local server to the internet (e.g.: via ngrok)

Alrighty, does this work on CI?

Sure thing, works like a charm!

Just make sure to run the server in the background on CI:

node server.js &

Thanks a million!

You’re more than welcome! Check out this GitHub repo for more details 🙂

Stress testing on iOS with xcmonkey

2023-01-08T00:00:00+00:00

Monkey testing

There are three things you can watch forever: fire burning, water falling, and monkey testing 📱🐒

In my humble opinion, automated monkey testing is the most mesmerizing testing technique. No, really, random or pseudo-random clicks and swipes, and yet it’s so breathtaking, isn’t it? Also, it’s such a fun way to stress-test apps that you’re developing, in a random yet repeatable manner.

How it all started

I think on iOS this story started with UIAutoMonkey. It was such an awesome tool to play with. Back then, there was neither Swift nor XCTest, so it leveraged Instruments that were coming with Apple’s developer tools and allowed to automate UI interactions using Javascript. It was deprecated around 2016 right after Apple removed the UI Testing instruments, which were replaced by XCTest.

After a while, SwiftMonkey popped up. You can guess from the name that it was written in Swift. It’s worth noting that it had such a beautiful design — you could literary observe the appearance of colourful monkey paws at the places where it was tapping. Unfortunately, in December 2022 it was archived.

So it goes, native development is awesome and it can bring a lot of benefits, but in return, it requires constant support and maintenance for obvious reasons. But look at Android, it has had monkey tool for ages, and it’s still there. Role model or monopoly?

Anyways, I’ve been looking for a replacement for UIAutoMonkey and SwiftMonkey and, oddly enough, ain’t much out there. Apple introduces breaking changes so often that these tools don’t last long. So, if the space is empty, why not fill it yourself? 🤠

Let’s craft something new

Say hi to xcmonkey — a brand new tool for doing randomised UI testing of iOS apps. It’s inspired by and has similar goals to the aforementioned monkey on Android.

Although it’s not that feature-rich yet, but it is a good start and it’s open-sourced so you can contribute and make it even better by opening issues or pull requests on GitHub. Let’s see if it finds its audience.

Under the hood, xcmonkey uses iOS Development Bridge as a driver, that’s why it’s pretty smart and can do a lot of things, such as taps, swipes and presses. All that comes «pseudo-random» because it has access to the screen hierarchy, and so can either do actions blindly (like tapping on random points) or precisely (like tapping on the existing elements).

Installation

First of all, you need to have idb preinstalled:

brew install facebook/fb/idb-companion
pip3.6 install fb-idb

Then you can install xcmonkey:

gem install xcmonkey

If you prefer to use bundler, then

add the following line to your Gemfile:
```
  gem 'xcmonkey'
```
and run:
```
  bundle install
```

Usage

As of today, xcmonkey has three commands:

test — runs the monkey testing

  $ xcmonkey test --udid "413EA256-CFFB-4312-94A6-12592BEE4CBA" --bundle-id "com.apple.Maps" --duration 100
  12:44:19.343: Device info: iPhone 14 Pro | 413EA256-CFFB-4312-94A6-12592BEE4CBA | Booted | simulator | iOS 16.2 | x86_64 | /tmp/idb/413EA256-CFFB-4312-94A6-12592BEE4CBA_companion.sock

  12:44:22.550: App info: com.apple.Maps | Maps | system | arm64, x86_64 | Running | Not Debuggable | pid=74636

  12:44:23.203: Tap: {
    "x": 53,
    "y": 749
  }

  12:44:23.511: Swipe (0.5s): {
    "x": 196,
    "y": 426
  } => {
    "x": 143,
    "y": 447
  }

  12:44:24.355: Press (1.2s): {
    "x": 143,
    "y": 323
  }

repeat — repeats the monkey testing from generated session

  $ xcmonkey repeat --session-path "./xcmonkey-session.json"
  12:48:13.333: Device info: iPhone 14 Pro | 413EA256-CFFB-4312-94A6-12592BEE4CBA | Booted | simulator | iOS 16.2 | x86_64 | /tmp/idb/413EA256-CFFB-4312-94A6-12592BEE4CBA_companion.sock

  12:48:16.542: App info: com.apple.Maps | Maps | system | arm64, x86_64 | Running | Not Debuggable | pid=73416

  12:48:20.195: Tap: {
    "x": 53,
    "y": 749
  }

  12:48:20.404: Swipe (0.5s): {
    "x": 196,
    "y": 426
  } => {
    "x": 143,
    "y": 447
  }

  12:48:21.155: Press (1.2s): {
    "x": 143,
    "y": 323
  }

describe — describes the given point on the screen

  $ xcmonkey describe -x 20 -y 625 --udid "413EA256-CFFB-4312-94A6-12592BEE4CBA"
  20:05:20.212: Device info: iPhone 14 Pro | 413EA256-CFFB-4312-94A6-12592BEE4CBA | Booted | simulator | iOS 16.2 | x86_64 | /tmp/idb/413EA256-CFFB-4312-94A6-12592BEE4CBA_companion.sock

  20:05:21.713: x:20 y:625 point info: {
    "AXFrame": "19, 624.3, 86, 130.6",
    "AXUniqueId": "ShortcutsRowCell",
    "frame": {
      "y": 624.3,
      "x": 19,
      "width": 86,
      "height": 130.6
    },
    "role_description": "button",
    "AXLabel": "Home",
    "content_required": false,
    "type": "Button",
    "title": null,
    "help": null,
    "custom_actions": [

    ],
    "AXValue": "Add",
    "enabled": true,
    "role": "AXButton",
    "subrole": null
  }

Fastlane Support

If you use fastlane, then you can easily integrate xcmonkey into your workflow. Just add the following lane to your Fastfile:

lane :test do
  bundle_id = 'com.apple.Maps'
  device = 'iPhone 14 (16.2)'
  sim = FastlaneCore::Simulator.all.detect { |d| device == "#{d.name} (#{d.os_version})") }

  xcmonkey(udid: sim.udid, bundle_id: bundle_id)
end

How it looks like

Conclusion

Although monkey testing is not a substitute for more comprehensive testing strategies, it still can be a useful complement to other testing approaches.

Try to stress-test your apps if you haven’t already, and don’t hesitate to give xcmonkey a try! 🐒

Allow XCTest to simulate location

2022-12-04T00:00:00+00:00

Many apps track users’ location. Some of them use it for navigation, some — for analytics. Is this for our good? Emm…

But in order to autotest related features, we need to mock some things. Today, I’ll show you how to do it in XCTest.

Precondition

Install any local web server (e.g.: Sinatra):

gem install sinatra

Configure the server

First of all, we need to create an endpoint that will receive a request with all the required parameters. In our case, it will be a POST request with the following params:

latitude — latitude of the initial location (default: 53.337)
longitude — longitude of the initial location (default: -6.27)
direction — direction to move (default: "up")
pace — movement speed (default: 0.0001)
duration — how long to move (default: 5 sec)

Then, depending on the direction parameter, we will move the location in the specified direction (within the given duration). For example, if the direction is "up", we will increase the latitude by the pace value. If the direction is "left", we will decrease the longitude by the pace value. And so on for the other directions.

To make it real, implement the following code into the server.rb file:

require 'json'
require 'sinatra'

post '/move' do
  body = JSON.parse(request.body.read)
  movement_direction = body['direction'] || 'up'
  movement_duration = body['duration'] || 5
  movement_pace = body['pace'] || 0.0001
  initial_latitude = body['latitude'] || 53.337
  initial_longtitude = body['longtitude'] || -6.27

  move(
    latitude: initial_latitude,
    longtitude: initial_longtitude,
    direction: movement_direction,
    pace: movement_pace,
    duration: movement_duration
  )
end

def move(latitude:, longtitude:, pace:, direction:, duration:)
  time_to_finish_movement = Time.now.to_i + duration
  while Time.now.to_i < time_to_finish_movement
    case direction
    when 'up'
      latitude += pace
    when 'down'
      latitude -= pace
    when 'right'
      longtitude += pace
    when 'left'
      longtitude -= pace
    end

    puts "Moving #{direction} to #{latitude},#{longtitude}"
    `xcrun simctl location #{params['udid']} set #{latitude},#{longtitude}`
  end
end

Link XCTest to the server

Alrighty, now we need to link our sandbox (XCTest) to the server. For this, we need to create a sendRequest function, that will get the required parameters and send a request to the server:

func sendRequest(endpoint: String, body: [String: Any], method: String = "POST") {
    let urlString = "\(host):\(port)\(endpoint)"
    guard let url = URL(string: urlString) else { return }

    var request = URLRequest(url: url)
    request.httpMethod = method
    request.httpBody = try? JSONSerialization.data(withJSONObject: body, options: [])
    URLSession.shared.dataTask(with: request).resume()
}

The whole test class without tests will look like this:

import XCTest

class LocationTests: XCTestCase {
    let app = XCUIApplication(bundleIdentifier: "com.apple.Maps")
    let host = "http://localhost"
    let port: UInt16 = 4567
    let movementDuration: UInt32 = 5

    override func setUpWithError() throws {
        app.launch()
    }

    func move(_ direction: Direction, for duration: UInt32) {
        sendRequest(
            endpoint: "/move",
            body: ["direction": direction.rawValue, "duration": duration]
        )
    }

    enum Direction: String {
        case right
        case left
        case up
        case down
    }

    func sendRequest(endpoint: String, body: [String: Any], method: String = "POST") {
        let udid = ProcessInfo.processInfo.environment["SIMULATOR_UDID"] ?? ""
        let urlString = "\(host):\(port)\(endpoint)?udid=\(udid)"
        guard let url = URL(string: urlString) else { return }

        var request = URLRequest(url: url)
        request.httpMethod = method
        request.httpBody = try? JSONSerialization.data(withJSONObject: body, options: [])
        URLSession.shared.dataTask(with: request).resume()
    }
}

You may notice that I used the default iOS Maps app as an example, feel free to replace it with the app of your choice.

Create the tests

So, it’s time to add some tests to make sure it all works together. Let’s create a simple one that will check if the user is able to move across the map in the right direction:

func testThatUserCanMoveRight() {
    move(.right, for: movementDuration)
    let myLocationFlag = app.otherElements["AnnotationContainer"].otherElements.firstMatch
    let initialLocation = CGPoint(x: myLocationFlag.frame.midX, y: myLocationFlag.frame.midY)
    sleep(movementDuration)
    let newLocation = CGPoint(x: myLocationFlag.frame.midX, y: myLocationFlag.frame.midY)
    XCTAssertGreaterThan(newLocation.x, initialLocation.x, "My location flag should move 'right'")
}

And one more test just to double-check that all good also with the longitude. To do that, let’s check the down direction:

func testThatUserCanMoveDown() {
    move(.down, for: movementDuration)
    let myLocationFlag = app.otherElements["AnnotationContainer"].otherElements.firstMatch
    let initialLocation = CGPoint(x: myLocationFlag.frame.midX, y: myLocationFlag.frame.midY)
    sleep(movementDuration)
    let newLocation = CGPoint(x: myLocationFlag.frame.midX, y: myLocationFlag.frame.midY)
    XCTAssertGreaterThan(newLocation.y, initialLocation.y, "My location flag should move 'down'")
}

Start the server

ruby server.rb

Run the tests

Open Xcode
Click on the Run test icon next to the testThatUserCanMoveRight function
If everything was done correctly, you’ll probably see something like this:
Same for testThatUserCanMoveDown:
Profit!

Feel free to adjust the source code to your needs and don’t hesitate to reach out if you have any questions. See ya!

Video recording of failed tests on iOS

2022-11-12T00:00:00+00:00

It’s no secret that tests can fail with or without a reason, especially those flaky UI ones on CI. Would be great to know exactly why they failed, wouldn’t it?

XCTest runs in the sandbox, which makes it hard to get anything out of it, but let’s hack it and record the failed tests!

Precondition

The easiest way to record a video is to run simctl command from the Terminal:

xcrun simctl io booted recordVideo "filename"

But how to run Shell commands from XCTest? Any web server will do the trick. Let’s use my favourite one:

gem install sinatra

Code outside the sandbox

Create a server.rb file
Fill it with the code below

require 'sinatra'
require 'fileutils'

post '/record_video/:udid/:test_name' do
  recordings_dir = 'recordings'
  video_base_name = "#{recordings_dir}/#{params['test_name']}"
  recordings = (0..Dir["#{recordings_dir}/*"].length + 1).to_a
  body = JSON.parse(request.body.read)
  FileUtils.mkdir_p(recordings_dir)

  video_file = ''
  if body['delete']
    recordings.reverse_each do |i|
      video_file = "#{video_base_name}_#{i}.mp4"
      break if File.exist?(video_file)
    end
  else
    recordings.each do |i|
      video_file = "#{video_base_name}_#{i}.mp4"
      break unless File.exist?(video_file)
    end
  end

  if body['stop']
    simctl_processes = `pgrep simctl`.strip.split("\n")
    simctl_processes.each { |pid| `kill -s SIGINT #{pid}` }
    File.delete(video_file) if body['delete'] && File.exist?(video_file)
  else
    puts `xcrun simctl io #{params['udid']} recordVideo --codec h264 --force #{video_file} &`
  end
end

Comments

Web server has only one endpoint: /record_video/:udid/:test_name
Simulator UDID and test name are passed as URL parameters
- :udid is used to run simctl command
- :test_name is used to name the video file
This endpoint gets POST requests with JSON:
- stop: Bool - should recording be stopped (will kill the recording process)
- delete: Bool - should recording be deleted (will delete the recording if test passed)
Web server saves the video in the recordings directory
- It’s smart enough to find the next available file name if test is run multiple times (e.g. retry strategy)

Code in the sandbox

Open an Xcode project
Create a test file
Fill it with the code below

import XCTest

final class SampleTest: XCTestCase {

  override func setUpWithError() throws {
    try super.setUpWithError()
    recordVideo()
    XCUIApplication().launch()
  }

  override func tearDownWithError() throws {
      recordVideo(stop: true)
      XCUIApplication().terminate()
      try super.tearDownWithError()
  }

  func testPass() {
    XCTAssertTrue(true)
  }

  func testFail() {
    XCTAssertTrue(false)
  }

  func recordVideo(stop: Bool = false) {
    let testName = String(name.split(separator: " ")[1].dropLast())
    let json: [String: Any] = ["delete": !isTestFailed(), "stop": stop]
    let udid = ProcessInfo.processInfo.environment["SIMULATOR_UDID"] ?? ""
    let urlString = "http://localhost:4567/record_video/\(udid)/\(testName)"
    guard let url = URL(string: urlString) else { return }

    var request = URLRequest(url: url)
    request.httpMethod = "POST"
    request.httpBody = try? JSONSerialization.data(withJSONObject: json, options: [])
    URLSession.shared.dataTask(with: request).resume()
  }

  func isTestFailed() -> Bool {
    if let testRun = testRun {
      let failureCount = testRun.failureCount + testRun.unexpectedExceptionCount
      return failureCount > 0
    }
    return false
  }
}

Comments

There are two tests: testPass and testFail, so we can make sure that the recording is saved only if the test failed. I suppose there is no need to keep recording if the test passed
setUpWithError and tearDownWithError are the best places to start and stop the recording
recordVideo method sends the request to the web server with all the required details
- host: localhost
- port: 4567 (default for Sinatra)
- endpoint: /record_video/:udid/:test_name
isTestFailed method is used to get a test result
Dictionary extension with jsonToString method is used to convert JSON body to String

Result

Run the web server:
```
 ruby server.rb
```
Run the tests
- from Xcode or from the Terminal, whatever you prefer
- on any iOS Simulator
Check out the recordings folder, you should only find the video from the failed test there

Feel free to adjust the scripts to your needs and don’t hesitate to reach out if you have any questions. See ya!

Getting iOS Simulator UDID from tests

2022-11-11T00:00:00+00:00

Sometimes it can be useful to have the UDID of the iOS simulator handy for automation purposes. Let’s learn how to extract it within XCTest.

Right off the boat 🤿

import XCTest

class SampleTest: XCTestCase {

  func testSimulatorUDID() {
    let env = ProcessInfo.processInfo.environment
    let udid = env["SIMULATOR_UDID"] ?? ""
    XCTAssertNotEqual(udid, "")
  }
}

Open an Xcode project
Create a test file
Fill it with the code above
Run it on any iOS Simulator
Enjoy 🤠

Testing push notifications within XCTest

2022-09-19T00:00:00+01:00

Xcode 11.4 introduced a handy feature that allowed us to test push notifications on Simulator (xcrun simctl push). Unfortunately, it’s still not possible to take advantage of it within the XCTest framework. So, let’s build our own bike!

Tooling

I will use the following tools:

XCTest - a native framework to write Unit and UI tests on iOS
Sinatra - an awesome DSL for quickly creating web applications in Ruby
Fastlane - the best tool for automating almost everything in and around iOS and Android platforms

Paths

Gemfile - a file that defines your Ruby dependencies
fastlane/Fastfile - a Fastlane configuration file
fastlane/sinatra.rb - a Sinatra app
fastlane/sinatra_log.txt - an output of the Sinatra server
fastlane/push_payload.json - Apple Push Notification service (APNs) payload

Precondition

We need to create a Gemfile:

  source 'https://rubygems.org'

  gem 'fastlane'
  gem 'sinatra'

And install the required gems using bundler:
```
  gem install bundler
  bundle install
```

Web server configuration

There is no way to execute macOS commands from XCTest as it works in the sandbox on the device/simulator. To bypass this limitation, we gonna build a tiny web server that will listen for HTTP requests and run an xcrun command for us as soon as it receives them. So let’s create a sinatra.rb file with the following content:

require 'sinatra'

post '/push/:udid/:bundle_id' do
  push_data_file = 'push_payload.json'
  File.open(push_data_file, 'w') { |f| f.write(request.body.read) }
  puts `xcrun simctl push #{params['udid']} #{params['bundle_id']} #{push_data_file}`
end

This script will run the web server and listen for the POST requests on the /push/:udid/:bundle_id endpoint. As soon as Sinatra receives a request, it will write the request body to the push_payload.json file and pass it to the xcrun command. The :udid and :bundle_id parameters are going to be passed to the xcrun command as well. The xcrun simctl push is exactly the guy who sends a push notification to the specified device.

Test framework configuration

I guess you already have an Xcode project so let’t open your *UITests.swift file and add the following functions and variables:

The function to send a request to the web server with the required details:

  func pushNotification(title: String, body: String, badge: Int = 1) {
    let notificationPayload = """
    {
      "aps": {
        "badge": badge,
        "alert": {
          "title": title,
          "body": body
        }
      }
    }
    """

    let targetBundleId = String(XCUIApplication().description.split(separator: "'")[1])
    let simulatorUdid = ProcessInfo.processInfo.environment["SIMULATOR_UDID"]!

    let urlString = "http://localhost:4567/push/\(simulatorUdid)/\(targetBundleId)"
    let url = URL(string: urlString)!

    var request = URLRequest(url: url)
    request.httpMethod = EndpointMethod.post.rawValue
    request.httpBody = notificationPayload.data(using: .utf8)
    URLSession.shared.dataTask(with: request).resume()
  }

The variable to access a push notification banner:

  var notificationBanner: XCUIElement {
    XCUIApplication(bundleIdentifier: "com.apple.springboard")
      .otherElements["Notification"]
      .descendants(matching: .any)
      .matching(NSPredicate(format: "label CONTAINS[c] ', now,'"))
      .firstMatch
  }

The function to tap on a push notification banner:

  func tapOnPushNotificationBanner() {
    notificationBanner.tap()
  }

The function to test a push notification banner:

  func assertPushNotification(title: String, body: String) {
    let bannerText = notificationBanner.label
    XCTAssertTrue(bannerText.contains(title), "'\(bannerText)' does not contain '\(title)'")
    XCTAssertTrue(bannerText.contains(body), "'\(bannerText)' does not contain '\(body)'")
  }

The function to launch the target application:

  func launchApp() {
    XCUIApplication().launch()
  }

The function to go to background:

  func goToBackground() {
    XCUIDevice.shared.press(XCUIDevice.Button.home)
    sleep(2)
  }

The test itself:

  func testPushNotification() {
    let notificationTitle = "Foo"
    let notificationBody = "Bar"

    launchApp()
    goToBackground()
    pushNotification(title: notificationTitle, body: notificationBody)
    assertPushNotification(title: notificationTitle, body: notificationBody)
  }

At this point, you already have a working test case that sends a push notification to the target application and verifies if the banner is displayed. See the complete code example here. To check it out:

Just start the Sinatra server in the Terminal:
```
  bundle exec ruby sinatra.rb
```
And run your test manually from Xcode

CLI configuration

Testing locally is fine, but we are here to automate things properly. So, let’s create a Fastfile with the following content:

The lane to start the Sinatra server:

  lane :start_sinatra do
    sh('nohup bundle exec ruby sinatra.rb > sinatra_log.txt 2>&1 &')
  end

The lane to run the provided test plan on the iPhone 8 simulator (make sure to update your project details in the scan action)

  lane :test_push_notification do
    start_sinatra
    scan(
      project: 'SampleApp.xcodeproj',
      scheme: 'SampleAppScheme',
      testplan: 'SampleAppTestPlan',
      devices: ['iPhone 8']
    )
  end

The lane and the method to always stop the Sinatra server after a Fastlane execution has completed:

  lane :stop_sinatra do
    sh('lsof -t -i:4567 | xargs kill -9')
  end

  after_all do |lane|
    stop_sinatra if lane == :test_push_notification
  end

See the complete code example here. To check it out just run the following command:

bundle exec fastlane test_push_notification

The easiest way to manage multiple Xcodes

2022-08-12T00:00:00+01:00

Dealing with multiple Xcode versions locally as well as on CI has never been easier than it is today. Installing Xcode command line tools, creating symlinks, switching between different versions, setting up the default one - from now on it’s just a matter of a couple of mouse clicks. Less words, more code snippets and examples.

GUI

For me, using Xcodes is by far the easiest way to manage multiple Xcode versions locally:

https://github.com/RobotsAndPencils/XcodesApp

Install the tool
```
 brew install --cask xcodes
```
Open the app
```
 open /Applications/Xcodes.app
```
Select the desired version of Xcode and click on the Install button
Once the app downloads an Xcode from developer.apple.com, you’ll be prompted to enter an access code
- don’t worry, Apple will send it to you (ofc only if you have an apple developer account)
When the installation is complete, you can click on the Open button 🎉
- if you already have multiple versions of Xcode on your machine, you can choose which one to make the default version by clicking the colorless checkmark next to the Open button
To delete an unwanted Xcode, simply right-click on the line with the Xcode name and click on the “Uninstall” button from the drop-down list

Magic, isn’t it? Xcodes acts like a native tool, juggling the Xcode versions as you wish.

CLI

xcodes

This is basically the CLI version of the GUI tool written in pure Swift:

https://github.com/RobotsAndPencils/xcodes

brew install robotsandpencils/made/xcodes

Example:

$ xcodes install 13.4.1
(1/6) Downloading Xcode 13.4.1: 100%
(2/6) Unarchiving Xcode (This can take a while)
(3/6) Moving Xcode to /Applications/Xcode-13.4.1.app
(4/6) Moving Xcode archive Xcode-13.4.1.xip to the Trash
(5/6) Checking security assessment and code signing
(6/6) Finishing installation
xcodes requires superuser privileges in order to finish installation.
macOS User Password:

Xcode 13.4.1 has been installed to /Applications/Xcode-13.4.1.app

xcversion

If you are a fastlane fan, picking this action is a no-brainer:

https://docs.fastlane.tools/actions/xcversion

gem install fastlane

Example:

before_all do
  xcversion(version: "~> 13.4.0") if is_ci # Selects the latest installed version from the 13.4.x set
end

xcode-install

The fastlane action above is also available as a separate and more powerful Ruby gem:

https://github.com/xcpretty/xcode-install

gem install xcode-install

Example:

$ xcversion install 13.4.1
###########################################################               82.1%
######################################################################## 100.0%
Please authenticate for Xcode installation...

Xcode 13.4.1
Build version 13F100

The standalone gem also allows us to install any iOS/iPadOS/WatchOS Simulators

Example:

$ xcversion simulators --install='iOS 15.5'
###########################################################               82.1%
######################################################################## 100.0%
Please authenticate to install iOS 15.5 Simulator...

Successfully installed iOS 15.5 Simulator

Running tests on multiple iOS versions on GitHub Actions

2022-08-01T00:00:00+01:00

As with everything, there are at least two ways to run tests in parallel on multiple iOS versions:

Parallelization at the CI level
Parallelization at the test framework level

In our case, the CI will be GitHub Actions and the test framework will be Fastlane - the wrapper on top of XCTest. Just to be clear, we are going to use GitHub Actions as well as Fastlane in both scenarios, the only difference is where exactly we will parallelize the tests.

Parallelization at the CI level

Let’s create a GitHub Actions workflow, that will:

be executed by cron
use a matrix with different iOS versions, devices and macOS versions if needed
install the required iOS Simulator
pass a device name and an iOS version to Fastlane

name: Cron

on:
  schedule:
    # Running the pipeline "At 03:00 every night"
    - cron: '0 3 * * *'

jobs:
  test:
    name: Run tests on multiple iOS versions
    strategy:
      matrix:
        ios: [12.4, 13.7, 14.5, latest]
        device: ["iPhone 8"]
        macos: ["macos-12"]
      fail-fast: false
    runs-on: ${{ matrix.macos }}
    steps:
    - uses: actions/checkout@v2

    # Grabbing the full name of a macOS image
    - run: echo "IMAGE=${ImageOS}-${ImageVersion}" >> $GITHUB_ENV

    # Caching ruby gems
    - uses: actions/cache@v2
      with:
        path: vendor/bundle
        key: ${{ env.IMAGE }}-gem-${{ hashFiles('**/Gemfile.lock') }}
        restore-keys: ${{ env.IMAGE }}-gem-

    # Installing ruby gems
    - run: bundle install

    # Installing iOS Simulator if required
    - name: Install iOS Simulator ${{ matrix.ios }}
      if: ${{ matrix.ios != 'latest' }}
      run: xcversion simulators --install='iOS ${{ matrix.ios }}' || true

    # Here we go
    - name: Run Tests
      run: bundle exec fastlane cron_checks device:"${{ matrix.device }}" ios:"${{ matrix.ios }}"

Then let’s create a Fastlane lane, that will:

take in a device name and an iOS version
run tests on the provided device

lane :cron_checks do |options|
  device_name = options[:ios] == 'latest' ? options[:device] : "#{options[:device]} (#{ios})"

  scan(
    project: 'SampleApp.xcodeproj',
    scheme: 'SampleApp',
    testplan: 'SampleAppTestPlan',
    configuration: 'Debug',
    devices: [device_name]
  )
end

This way, the tests are run on multiple iOS versions in parallel in different matrix jobs on GitHub Actions.

Parallelization at the test framework level

Let’s create a GitHub Actions workflow, that will:

be executed by cron
pass a device name to Fastlane

name: Cron

on:
  schedule:
    # Running the pipeline "At 03:00 every night"
    - cron: '0 3 * * *'

jobs:
  test:
    name: Run tests on multiple iOS versions
    runs-on: macos-12
    steps:
    - uses: actions/checkout@v2

    # Grabbing the full name of a macOS image
    - run: echo "IMAGE=${ImageOS}-${ImageVersion}" >> $GITHUB_ENV

    # Caching ruby gems
    - uses: actions/cache@v2
      with:
        path: vendor/bundle
        key: ${{ env.IMAGE }}-gem-${{ hashFiles('**/Gemfile.lock') }}
        restore-keys: ${{ env.IMAGE }}-gem-

    # Installing ruby gems
    - run: bundle install

    # Here we go
    - name: Run Tests
      run: bundle exec fastlane cron_checks device:"iPhone 8"

Then let’s create a Fastlane lane, that will:

take in a device name
install the required iOS Simulators
run tests on multiple iOS versions

lane :cron_checks do |options|
  ios = ['12.4', '13.7', '14.5', 'latest']

  # Installing iOS Simulators if required
  ios.each { |version| sh("xcversion simulators --install='iOS #{version}' || true") if version != ios.last }

  # Setting up device names
  devices = ios.reverse.drop(1).reverse.map { |v| "#{options[:device]} (#{v})" }.append(options[:device])

  scan(
    project: 'SampleApp.xcodeproj',
    scheme: 'SampleApp',
    testplan: 'SampleAppTestPlan',
    configuration: 'Debug',
    devices: devices
  )
end

Thus, the tests are run on multiple iOS versions in parallel in the same job on GitHub Actions.

Native mock server inside XCTest

2022-02-14T00:00:00+00:00

Introduction

An automated testing can be a tricky thing, especially when you’re dealing with unstable or constantly changing backend. What if we could get rid of the real backend and use a mock server in our tests instead? Perhaps that would solve a couple of our problems, eh?

Check out this repository to dive into the source code I’ll be playing with in this note.

Precondition

I’ll use Swifter just because I love it and it’s super easy to start hacking right away with it. But really there are several similar tools, so grab any other if Swifter does not suit you.

Create sample project

Open Xcode
Click on Create a new Xcode project
Choose iOS App as a template
Fill the product name and the team name
Choose SwiftUI as an interface
Click on the Next button
Open the ContentView.swift file
Fill it in with this code
Run an app on any iOS Simulator
Tap on the Game controller 🎮
^{The app pulls a random user from the backend}

Install Swifter

Open Xcode
Go to File => Add Packages...
Type https://github.com/httpswift/swifter.git in the search field
Click on the Add Package button
Open UITests target => Build Phases tab
Click on the plus button under the Link Binary With Libraries section
Find Swifter from the list under the Swift Package section
Click on the Add button
Open a <package name>UITests file
Import Swifter
```
import Swifter
```
Create this variable in the test class
```
private var server = HttpServer()
```

Create this method in the test class

private func startServer() {
  do {
    try server.start(4567)
    print("Server status: \(server.state)")
  } catch {
    print("Server start error: \(error)")
  }
}

Run this test to make sure everything goes fine (hopefully, you see a green light ✅)

func testUsername() {
    startServer()()
    XCUIApplication().launch()

    XCUIApplication().images["gamecontroller"].coordinate(withNormalizedOffset: CGVector(dx: 0.0, dy: 0.0)).tap()

    let username = XCUIApplication().staticTexts["username"]

    XCTAssertTrue(username.exists)
}

Mock production backend

Open the ContentView.swift file

Replace an existed function apiUrl() with:

 func apiUrl() -> URL {
     var urlString = "https://random-data-api.com"
     #if DEBUG
       if ProcessInfo.processInfo.arguments.contains("MOCK_SERVER") {
           urlString = "http://localhost:4567"
       }
     #endif
     let url = URL(string: urlString)!
     return url
 }

Run an app on iOS Simulator and tap on the Game controller 🎮

3.1 The result should remain the same, except for the username of course
Open a <package name>UITests file

Create new methods and variables to mock the required endpoint

 private var username = ""

 private func configureServer() {
   server["/api/v2/users"] = { [self] _ in
       var response = readData(fromFile: "users")
       response["first_name"] = username
       print("\nGAME STARTED 🏈\n\(response.toString(prettyPrinted: true))")
       return .ok(.json(response))
   }
 }

 private func mockUsername(_ username: String) {
     self.username = username
 }

 private func readData(fromFile name: String, ext: String = "json") -> [String: Any] {
     let bundle = Bundle(for: Self.self)
     let url = bundle.url(forResource: name, withExtension: ext)!
     let data = try! Data(contentsOf: url)
     return String(bytes: data, encoding: .utf8)!.json
 }

 extension String {
     var json: [String: Any] {
         try! JSONSerialization.jsonObject(with: Data(self.utf8),
                                           options: .mutableContainers) as! [String: Any]
     }
 }

Create a json template users.json in your project to mock the response from the backend

Update the test and run it again

 func testUsername() {
     startServer()()
     configureServer()

     XCUIApplication().launchArguments = ["MOCK_SERVER"]
     XCUIApplication().launch()

     let expectedUsername = "Frodo"
     mockUsername(expectedUsername)

     XCUIApplication().images["gamecontroller"].coordinate(withNormalizedOffset: CGVector(dx: 0.0, dy: 0.0)).tap()

     let actualUsername = XCUIApplication().staticTexts["username"].label

     XCTAssertEqual(expectedUsername, actualUsername)
 }

Look at the green light ✅
^{The app pulls the expected user from the mock server}

Explore the mock server

Get request details

server["/api/v2/users"] = { request in
  let headers = request.headers
  let queryParams = request.queryParams
  let address = request.address
  let body = request.body
  let params = request.params
  let path = request.path
  let method = request.method
  return HttpResponse.ok(.text("""
  { "hello": "world" }
  """)
  )
}

Mock dynamic endpoint

server["api/:version/users"] = { request in
  let apiVersion = request.params[":version"]
  return HttpResponse.ok(.text("""
  { "hello": "world" }
  """)
  )
}

Return internal server error

server["api/:version/users"] = { _ in
  .internalServerError
}

Redirect

server["api/:version/users"] = { _ in
  .movedPermanently("http://www.google.com")
}

Customize response details

server["api/:version/users"] = { _ in
    let body = """
    { "hello": "world" }
    """.data(using: .utf8)

    let statusCode = 33
    let message = "Test message"
    let headers = ["User-Agent": "fake", "Test-Header": "test"]
    return HttpResponse.raw(statusCode, message, headers, { (writer) in
        try writer.write(body!)
    })
}

WebSockets

server["/websocket-echo"] = websocket(text: { session, text in
  session.writeText(text)
}, connected: { [weak self] _ in
  print("WS connected")
}, disconnected: { [weak self] _ in
  print("WS disconnected")
})

Set up a random port for parallel tests

private func startServer(
  port: in_port_t = in_port_t.random(in: 8081..<10000),
  maximumOfAttempts: Int = 10
) {
  guard maximumOfAttempts > 0 else { return }

  do {
      try server.start(port, forceIPv4: true)
  } catch SocketError.bindFailed(let message) where message == "Address already in use" {
      startServer(maximumOfAttempts: maximumOfAttempts - 1)
  } catch {
      print("Server start error: \(error)")
  }
}

Mitmproxy: tips, tricks and scripts

2022-02-13T00:00:00+00:00

Useful links

Precondition

Let’s put this code at the top of our mitmproxy script file:

from mitmproxy import http
import json
import time
import os

server_folder = "mitmproxy_data"

def parse_json(content: bytes):
  try:
    return json.loads(content.decode("utf-8"))
  except:
    return None

Tips, tricks, scripts

Filter by host and endpoints

def request(flow: http.HTTPFlow) -> None:
  content = flow.request.content
  if any(url in flow.request.url for url in target_list()):
    print(content)

def response(flow: http.HTTPFlow) -> None:
  content = flow.response.content
  if any(url in flow.request.url for url in target_list()):
    print(content)

def target_list():
  return ["foo.bar", "sample-website.com/test/endpoint"]

Catch all json requests and responses

def request(flow: http.HTTPFlow) -> None:
  content = parse_json(flow.request.content)
  if content is not None:
    print(content)

def response(flow: http.HTTPFlow) -> None:
  content = parse_json(flow.response.content)
  if content is not None:
    print(content)

Embed tracking beacon in jsons

def request(flow: http.HTTPFlow) -> None:
  content = parse_json(flow.request.content)
  if content is not None:
    content = add_tracker(flow, content)
    print(content)

def response(flow: http.HTTPFlow) -> None:
  content = parse_json(flow.response.content)
  if content is not None:
    content = add_tracker(flow, content)
    print(content)

def add_tracker(flow: http.HTTPFlow, content: json):
  if isinstance(content, list):
    content += [{"mitmproxy_tracker": flow.request.url}]
  else:
    content["mitmproxy_tracker"] = flow.request.url
  return content

Write json data to a file

def request(flow: http.HTTPFlow) -> None:
  content = parse_json(flow.request.content)
  if content is not None:
    file = open(file_path("request"), 'w')
    file.write(beautify(content))
    file.close()

def response(flow: http.HTTPFlow) -> None:
  content = parse_json(flow.response.content)
  if content is not None:
    file = open(file_path("response"), 'w')
    file.write(beautify(content))
    file.close()

def file_path(type: str):
  os.makedirs(server_folder, exist_ok=True)
  return server_folder + "/" + type + "_%s.json" % int(time.time())

def beautify(content: json):
  return json.dumps(content, indent=4, sort_keys=True)

Return all jsons combined into one

def request(flow: http.HTTPFlow) -> None:
  if "mitmproxy/collect" in flow.request.url:
    jsons = []
    for file in os.listdir(server_folder):
      file_path = os.path.join(server_folder, file)
      content = ""
      with open(file_path) as file:
        for line in file:
          content += line
      jsons.append(content)

    json = "[" + ",".join(jsons) + "]"
    flow.response = http.Response.make(
      200,
      json,
      {"Content-Type": "application/json"}
    )

Clear all saved files

  if "mitmproxy/clear" in flow.request.url:
    for file in os.listdir(server_folder):
      os.remove(os.path.join(server_folder, file))
    flow.response = http.Response.make(200)

Kill request

def request(flow: http.HTTPFlow) -> None:
  block_hosts = ["appldnld.apple.com", "mesu.apple.com"]
  if any(host in flow.request.host for host in block_hosts):
    flow.kill

Update headers and queries in requests and responses

def request(flow: http.HTTPFlow) -> None:
  flow.request.query["mitm"] = "proxy"
  flow.request.headers["newheader"] = "foo"

def response(flow: http.HTTPFlow) -> None:
  flow.response.query["proxy"] = "mitm"
  flow.response.headers["newheader"] = "bar"

Change response status code

def response(flow: http.HTTPFlow) -> None:
  flow.response.status_code = 503

Emulate slow connection

def response(flow: http.HTTPFlow) -> None:
  time.sleep(5.0)

Redirect requests

def request(flow: http.HTTPFlow) -> None:
  flow.request.host = "127.0.0.1"
  flow.request.scheme = "http"
  flow.request.port = 4567

Anything else?

Stay in touch (:

How to programmatically bypass 2FA

2021-10-17T00:00:00+01:00

Introduction

There is a chance that one day you’ll be asked to enter a two-factor authentication (2FA) code. There is even a possibility that you’ll be asked to enter a code every time you log in. Such a good area for automation, isn’t it?

My notes are usually dancing around mobile automation, so let’s imagine we have an idea to bypass 2FA on Play Store as part of our automation script.

Action

Indeed the first time we have to log in manually, but instead of a QR code, we’ll use a secret key.

Navigate to myaccount.google.com/security using the account that will be used in the automation script
Click on 2-Step Verification section
Provide an email and a password and click on the Next button
Click on the Authenticator App section (or on the Change Phone link in that section if the app was already set up for this account)
Select the Android or iPhone radio button and click on the Next button
Click on the Can’t Scan It option (it will generate the secret key for manual use)
Open any Authenticator app on your iOS or Android device (e.g.: Duo Mobile)
Add a new account in the Authenticator app using the account email and the secret key from the previous step
Return to the browser window and click on the Next button
Generate a Time-based One-Time Password TOTP using the Authenticator app on your device
Jump back on the browser window, enter the TOTP verification code and click on the Verify button

That’s it, we logged in. Now it’s time to automate the process. We’re not going to open the Authenticator app every time we log in, are we?

Install OATH Toolkit on your machine:
```
 brew install oath-toolkit
```

Generate the TOTP verification code:

 oathtool --base32 --totp "${YOUR_SECRET_KEY}"

Cool beans, the output of the last command is the TOTP verification code. Check it out!

Conclusion

Although 2FA is a truly secure thing, but as you might have noticed, it’s also a kind of developer-friendly tool, and to bypass it from the script, automation test or whatever we can simply use oathtool at the moment of logging in. Hope it helps. See ya later (:

Fastlane, plugin them all

2021-09-14T00:00:00+01:00

Introduction

Have you ever dealt with a long-long Fastfile or a bunch of fastlane actions or even any build scripts that hang out side-by-side with a source code of your project and turned to be completely unreadable? Then I have a modern architectural idea to make such things less complicated and much more readable, reliable and extendable.

I suggest to craft your own private fastlane plugin that will wrap everything you have around building and such things and return you a gorgeous DIY DSL that can be used in your Fastfile. It might sound a little vague and complicated, but this is really awesome sauce, see, you will love it! Very-very roughly it will look like this:

Terminal

brew install fastlane
fastlane add_plugin awesome_sauce

Fastfile

lane :awesome_lane do
  edit_some_weird_properties # this action comes from my awesome sauce fastlane plugin
  my_hackish_way_to_build # same for this one
  my_tricky_way_to_test # same here
  ok_boy_ship_it # same
end

Auto-generating the plugin

First of all let’s install fastlane if you have not done it yet:
```
 brew install fastlane
```
Cool, now we can create our plugin, let’s name it awesome_sauce:
```
 fastlane new_plugin awesome_sauce
```
Enter a description and short summary of your choice
And that’s it, fastlane generated a template for us, check it out:
```
 ls fastlane-plugin-awesome_sauce
```

Crafting the plugin

Let’s add some super simple features to our plugin for proof of concept purposes. For instance minus and plus, just to show how these two actions will live side-by-side. Sure, these actions are useless, but who said that you can’t do some really cool stuff on your own? This is just a POC after all (:

Let’s start from the minus action:

Go to the actions folder:

cd fastlane-plugin-awesome_sauce/lib/fastlane/plugin/awesome_sauce/actions

Rename awesome_sauce_action.rb to minus_action.rb and open the file
Rename the class name AwesomeSauceAction to MinusAction in minus_action.rb file

Fill the description:

 def self.description
     "Subtracts one number from another"
 end

Add available options:

 def self.available_options
 [
     FastlaneCore::ConfigItem.new(
         key: :x,
         description: "The number from which to subtract",
         optional: false,
         is_string: false
     ),
     FastlaneCore::ConfigItem.new(
         key: :y,
         description: "The subtracted number",
         optional: false,
         is_string: false
     )
 ]
 end

Сreate a method for calculation in the helper/awesome_sauce_helper.rb file:
```
 def self.minus(x:, y:)
     x - y
 end
```

Call this method in minus_action.rb file:

 def self.run(params)
     Helper::AwesomeSauceHelper.minus(
         x: params[:x],
         y: params[:y]
     )
 end

Let’s do the same for the plus action:

Copy minus_action.rb and rename the copy to plus_action.rb
Rename the class name MinusAction to PlusAction in plus_action.rb file

Change the description:

 def self.description
     "Adds one number to another"
 end

Change the description of available options:

 def self.available_options
 [
     FastlaneCore::ConfigItem.new(
         key: :x,
         description: "The number to which to add",
         optional: false,
         is_string: false
     ),
     FastlaneCore::ConfigItem.new(
         key: :y,
         description: "The number to be added",
         optional: false,
         is_string: false
     )
 ]
 end

Сreate a method for a new calculation in the helper/awesome_sauce_helper.rb file:
```
 def self.plus(x:, y:)
     x + y
 end
```

Call this method in plus_action.rb file:

 def self.run(params)
     Helper::AwesomeSauceHelper.plus(
         x: params[:x],
         y: params[:y]
     )
 end

Here we go, our mega simple plugin is almost ready to use!

Testing the plugin

We all know there are some cowboy rules, so let’s add the tests to our plugin.

Go to the specs folder:
```
 cd spec/
```
Open awesome_sauce_action_spec.rb and clean it up

Describe a class we gonna test:

 describe Fastlane do
     describe Fastlane::FastFile do
         describe 'Very limited calculator' do
             it 'is dummy test' do
                 expect(42).to eq(42)
             end
         end
     end
 end

Add some tests for minus action:

 it 'minus positive number' do
     result = described_class.new.parse("lane :test do
         minus(x: 44, y: 2)
     end").runner.execute(:test)
     expect(result).to eq(42)
 end

   it 'minus negative number' do
     result = described_class.new.parse("lane :test do
         minus(x: 40, y: -2)
     end").runner.execute(:test)
     expect(result).to eq(42)
 end

 it 'minus float number' do
     result = described_class.new.parse("lane :test do
         minus(x: 44.5, y: 2.5)
     end").runner.execute(:test)
     expect(result).to eq(42)
 end

Add some tests for plus action:

 it 'plus positive number' do
     result = described_class.new.parse("lane :test do
         plus(x: 40, y: 2)
     end").runner.execute(:test)
     expect(result).to eq(42)
 end

 it 'plus negative number' do
     result = described_class.new.parse("lane :test do
         plus(x: 44, y: -2)
     end").runner.execute(:test)
     expect(result).to eq(42)
 end

 it 'plus float number' do
     result = described_class.new.parse("lane :test do
         plus(x: 40.5, y: 1.5)
     end").runner.execute(:test)
     expect(result).to eq(42)
 end

Go to the root of your plugin:
```
 cd ~/fastlane-plugin-awesome_sauce
```
Execute the tests (locally for now, but you can also run them on CI if you wish):
```
 rspec
```

P.S.: in «‎real world» you’d probably like to run tests as part of the rake command with rubocop linter and such things, but this is «‎hello world», so it’s grand (:

Releasing the plugin

On the one hand we could all the time use master branch as a source of truth for the plugin, but from my point of view it’s better to have a way to use the different versions of the plugin, this is simply a good practice.

So for that, we just need to tag the commit with the version number. I’ll also attach a gem file with the release just in case. Let’s do it using fastlane:

Open Fastfile:

cd ~/fastlane-plugin-awesome_sauce/fastlane/Fastfile

Create a new lane release (you need to have a Github API token for this step):

 lane :release do
 version_path = '../lib/fastlane/plugin/awesome_sauce/version.rb'
 release_version = File.read(version_path).scan(/\d+/).join('.')
 sh('rake build')
 set_github_release(
     repository_name: 'your_github_username/fastlane-plugin-awesome_sauce',
     api_token: ENV['GITHUB_TOKEN'],
     name: "Awesome Sauce v#{release_version}",
     tag_name: "v#{release_version}",
     description: "v#{release_version}",
     commitish: git_branch,
     upload_assets: ["pkg/fastlane-plugin-awesome_sauce-#{release_version}.gem"]
 )
 end

Initiate the release:
```
 fastlane release
```

Implementing the plugin

Open your main project:
```
 cd ~/my_very_important_project
```
Create a Pluginfile:
```
 touch fastlane/Pluginfile
```

Install the plugin:

 echo "gem 'fastlane-plugin-awesome_sauce', git: 'git@github.com:my_github_username/fastlane-plugin-awesome_sauce.git', tag: 'v0.1.0'" > fastlane/Pluginfile
 fastlane install_plugins

Use its actions to the fullest in your Fastfile:

 lane :awesome_lane do
     sum = plus(x: 1, y: 1) * minus(x: 4, y: 2)
     say(text: "The result is: #{sum}")
 end

Conclusion

Peep into a Sample Project

We’ve just crafted our very own fastlane plugin, even if it’s not so smart, more important is that now you know what secrets fastlane keeps. It’s time to find out how to make it work for you… Happy Mondays (:

How to re-run only failed jobs on GitHub Actions

2021-04-18T00:00:00+01:00

Introduction

Have you ever come across a GitHub Actions workflow that ~~stumbled into level ground~~ failed without good reason? If so I believe you’re well familiar with this ruthless button:

Yeah, the thing is that it restarts absolutely all the jobs (green, red, whatever), even if just the only one has failed. You can imagine how annoying this can be. While GitHub Actions doesn’t provide a way to re-run just the failed jobs out of the box, let’s build something on our own.

Implementation

Step-by-step guide

Set up a step to get the current timestamp

 - id: timestamp
   run: echo "::set-output name=timestamp::$(timestamp +%s)"

Name a cache container using the above timestamp

 - uses: actions/cache@v2
   with:
     path: |
       run_result
     key: ${{ github.run_id }}-${{ github.job }}-${{ steps.timestamp.outputs.timestamp }}
     restore-keys: |
       ${{ github.run_id }}-${{ github.job }}-

Restore the previous result

 - id: run_result
   run: cat run_result 2>/dev/null || echo 'default'

Set up a condition for further steps

 - uses: actions/checkout@v2
   if: steps.run_result.outputs.run_result != 'success'

Save the result on success

 - run: echo "::set-output name=run_result::success" > run_result

Sample

Pros

CI time will be noticeably reduced
Developers nerves will be safely rescued
This approach is 100% configurable and can be implemented only for the chosen jobs and steps

Cons

The only drawback I could come up with is that the logs of the successful jobs will be cleared on re-run

Conclusion

The idea was to make our GitHub Actions more flexible and compromisable to speed up the development cycle, avoid time-consuming obstacles and increase fault tolerance.

Although this pattern is controversial enough, but on the other hand, you are now fully equipped to give your failed jobs a second chance (:

How to use Bash anonymously

2021-02-20T00:00:00+00:00

Introduction

Today I’ll briefly explain how to provide the Tor anonymity guarantee to almost any scripts and CLI tools that you have in your pocket.

The most complicated question would be: «For what?», and, to be honest, I can’t come up with a strong answer. Maybe you are a hacker, or you just enjoy being hidden and safe, or you simply need to test your app from Madagascar, who knows?

Installation

brew install tor
brew install torsocks

Usage

Start proxy server:

tor

Figure out your current IP address and location:

ip2cc $(curl -s ifconfig.me)

Proxy it through Tor and check it out one more time:

ip2cc $(curl -s --socks5 127.0.0.1:9050 ifconfig.me)

Looks like it has changed, right? It was a quick win. Worth mentioning that by default Tor takes an IP address randomly, but we can specify the exact country using Tor country codes. Let’s open a connection, for instance, from Canada and close it from New Zealand.

Stop pre-created proxy server:

killall tor

Create torrc file:

touch torrc

Put entry and exit nodes in torrc file (ca and nz as we decided above):

echo "EntryNodes {ca}
ExitNodes {nz}" > torrc

Start proxy server providing the configured torrc file:

tor -f torrc

Check your location out:

ip2cc $(curl -s --socks5 127.0.0.1:9050 ifconfig.me)

Github

So, now you’d like to clone github repo anonymously. Why do you need that? Ok, never mind, let’s just do it:

curl -sOL --socks5 127.0.0.1:9050 https://github.com/{username}/{reponame}/archive/{branch}.zip

Wait, you even think about working on that repo anonymously? I didn’t go that far, but I have something for ya:

https://stackoverflow.com/a/37100346

The show must go on

It worth noticing that some CLI tools provide an argument to get a proxy server path (e.g.: youtube.dl) and this way you can strain all their traffic through Tor:

youtube-dl "youtube.com/watch?v={id}" --proxy socks5://127.0.0.1:9050

To be honest, there is a much easier way to cover all that stuff, it’s where torsocks comes. For example:

torify curl -s ifconfig.me

Torify simply wraps all the traffic that any provided tool will request and thoroughly proxies it through itself. Let’s take a look at it from a different example, imagine that for the sake of fun or for any other reasons you need to anonymously install some groundbreaking ruby gem.

Here we go:

torify gem install bundler

You’d say: «Now you’re talking», but there is always BUT. Sometimes you may face an error like:

ERROR: /usr/bin/{toolname} is located in a directory protected by Apple's System Integrity Protection.

If so, just copy-pasta that binary to your local execution folder (but it’s possibly no more a workaround for macOS Big Sur users):

cp /usr/bin/{toolname} /usr/local/bin

Conclusion

Fun fact: now you know how to wear a mask on your tools and scripts 😷

Finding leaks, exposing secrets

2021-01-03T00:00:00+00:00

Introduction

So, right off the bat, - leaks are one of the most dangerous types of vulnerabilities for companies, one of the yummiest things for hackers, and one of the most well-paid targets for bug hunters.

This note is full of referrals on multiple great third-party researches, docs, articles, and whatnot, so sorry in advance.

Preparation

Would be great to first of all pick up as many target values as possible. By target values I mean: domains, subdomains, keywords, key-values, etc. Everything here depends on your savvy and imagination. At least I would propose to get:

The domains in scope (and, of course, out of scope, as we aren’t gonna break the things, just looking for some leaks that can possibly have an influence on the domains in scope) from HackerOne and/or BugCrowd (e.g.: Twitter);
The data under the hood of iOS and Android apps (tbh, this is already a good place to start keeping an eye on the leaks: tokens, secrets, passwords, etc). I won’t go into details here, for more info you can simply jump on my previous notes about it (iOS, Android);
The name of an organization on GitHub and the nicknames of its members for further analysis (e.g.: Twitter).

Baby steps

I won’t dwell on this point for long, but still, it shouldn’t be written off as it’s a really powerful tool. Here I’m talking about GitHub search. Using it in the right way can allow us to find a lot of interesting shtuff.

Btw, there is a lot of content across the web about GitHub dorks, but I’ll refer to the one specific pretty good research on this.

Scan the project

The most popular tool for this purpose is gitleaks. But I prefer to go with earlybird due to it’s a bit more functional and customizable. So here we go:

Install the tool:

 git clone https://github.com/americanexpress/earlybird
 cd earlybird/ && ./build.sh && ./install.sh

Scan each repo of the target project (keep in mind that there is a bunch of arguments that can fit your needs):

 go-earlybird --git-project=twitter \
             --display-severity=high \
             --display-confidence=high \
             --show-full-line \
             --skip-comments

Scan any user’s repo (e.g.: any repo of the target organization members):

 go-earlybird --git=https://github.com/testableapple/fastlane-plugin-try_scan \
             --display-severity=low \
             --display-confidence=low \
             --show-full-line

Pay attention to any suspicious data in the output.

Going a little off-topic, you could also set up such tools for your own project for instance as CI checks on PRs, to make the world a bit more secure. It’s a good thing to think about.

Scan the whole hub

I would take a breath and develop something special by myself for this step, but I won’t ehh? There is a lovely tool for such things called git-hound. Let’s do something like this:

Install the tool:

 git clone https://github.com/tillson/git-hound
 cd git-hound/
 go build
 cp git-hound /usr/local/bin

Fill the target list with the domains, subdomains, and/or any other suspicious queries, for instance:

 touch target.txt
 printf "corp.company.com
 test.company.ie
 stage.company.ru" > target.txt

Search for leaks (keep in mind that there are plenty of arguments that can fit your needs):

 git-hound --subdomain-file target.txt \
           --dig-files \
           --dig-commits \
           --many-results \
           --no-gists

Dive deep into the interesting links from the output.

Scan the feck everything

This is almost rocket science, but let’s give it a try. There is one tool I’d like to share with you that’s just blowing my mind. And its name is shhgit. Just open the live version of it and you’ll get me.

Here are some steps for those daredevils who’d like to deploy it on their own side:

Clone the repo:

 git clone https://github.com/eth0izzle/shhgit.git

Edit the config.yaml file (i.e. adding your GitHub tokens and removing the signatures you are not interested in):
```
 cd shhgit && open config.yaml
```
Install and run the tool:
- if you require a web interface then you’ll have to go with Dockerfile:
```
 docker-compose build
 docker-compose up
 open http://localhost:8080/
```
- elsif CLI is pretty enough for you then just execute:
```
 go build
 cp shhgit /usr/local/bin
 shhgit
```
Enjoy.

Conclusion

I hope it was a fun journey and there were at least some useful tips and tricks. If so, don’t stop on the git related leaks, try to think about CI logs and things like that. There is terrific research about it and I couldn’t write it up better.

Happy hacking and tremendous swag (:

Speed up CI through impact analysis

2020-07-28T00:00:00+01:00

Definition

What does impact analysis actually mean in terms of CI?

in brief:

It’s about time-saving thing — we define what was impacted and skip unrelated actions (e.g.: building or testing).

in practice:

We did a change to production code. We do analysis of what was affected. We will make a decision about what should be built and which tests should be executed depending on that analysis.

Surface impact analysis based on filters and triggers

Here we are using keywords in TRIGGER and IGNORE environment variables and running our tests that have been pre-filtered via tags/classes. This can be pretty helpful in monorepo world.

Create a script (check_changes.rb) that will be responsible for which files were impacted:

 require 'json'

 required_keys = ENV['TRIGGER'].split(',').map { |key| key.strip }
 ignored_keys = ENV['IGNORE'].nil? ? [] : ENV['IGNORE'].split(',').map { |key| key.strip }
 response = JSON.parse(`curl -s -H "authorization: Bearer #{ENV['GITHUB_TOKEN']}" -X GET -G #{ENV['PULL_REQUEST']}/files`)
 impacted_files = response.map { |file| file['filename'] }
 impacted_files.select! do |path|
   required_keys.any? do |required_key|
     should_key_be_considered = ignored_keys.none? { |key| path.include?(key) }
     should_key_be_considered && path.include?(required_key)
   end
 end

 puts impacted_files.size

Create CI pipeline that will be responsible for providing any required info to the script above and will work depending on the output:

 name: Sample

 on: [pull_request]

 jobs:
   first_project:
     name: FirstProjectTests
     runs-on: [macos-latest]
     steps:
       - uses: actions/checkout@v2
       - name: Runner
         env:
           PULL_REQUEST: ${{ github.event.pull_request._links.self.href }}
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           TRIGGER: "FirstProjectRelatedFolder/, VeryImportantFolder/, Fastfile"
           IGNORE: "FirstProjectFolder/tests/, .md"
         run: |
           if [ $(ruby ./.github/scripts/check_changes.rb) = 0 ]; then
             echo "Required files were not touched"
           else
             fastlane first_project_tests
           fi

   second_project:
     name: SecondProjectTests
     runs-on: [macos-latest]
     steps:
       - uses: actions/checkout@v2
       - name: Runner
         env:
           PULL_REQUEST: ${{ github.event.pull_request._links.self.href }}
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           TRIGGER: "SecondProjectRelatedFolder/, VeryImportantFolder/, Fastfile"
           IGNORE: "SecondProjectRelatedFolder/tests/, .md"
         run: |
           if [ $(ruby ./.github/scripts/check_changes.rb) = 0 ]; then
             echo "Required files were not touched"
           else
             fastlane second_project_tests
           fi

Deep impact analysis based on code coverage tools

Here we are using code coverage tools to generate an impact map and running our tests that have been matched against the impacted files.

As nothing changed since Paul Hammant has breathtakingly described it, I’ll just leave a link here:

The Rise of Test Impact Analysis

And make a quick guide to action:

Configure code coverage tool (e.g.: iOS example)
Run each test one by one

2.1. Collect code-coverage report for each test

2.2. Extract involved source files from code-coverage report

Create a map (impact_map.json), where test names are the keys and arrays of source files are the values, for example:

 {
   "SampleSuite/SampleClass/testMethod_1": [
     "path/to/a.swift",
     "path/to/b.swift",
     "path/to/c.swift"
   ],
   "SampleSuite/SampleClass/testMethod_2": [
     "path/to/c.swift",
     "path/to/d.swift",
     "path/to/e.swift"
   ]
 }

Create a script (check_changes.rb) that will be responsible for which tests to run:

 require 'json'

 impact_map = JSON.parse(File.read('impact_map.json'))
 response = JSON.parse(`curl -s -H "authorization: Bearer #{ENV['GITHUB_TOKEN']}" -X GET -G #{ENV['PULL_REQUEST']}/files`)
 impacted_files = response.map { |file| file['filename'] }

 impact_map.select! do |test, related_source_files|
   related_source_files.any? { |file| impacted_files.include?(file) }
 end

 puts impact_map.keys.join(',')

Create CI pipeline that will be responsible for providing any required info to the script above and will run the tests depending on the output:

 name: Sample

 on: [pull_request]

 jobs:
   sample:
     name: Tests
     runs-on: [macos-latest]
     steps:
       - uses: actions/checkout@v2
       - name: Runner
         env:
           PULL_REQUEST: ${{ github.event.pull_request._links.self.href }}
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
           essential_tests=$(ruby ./.github/scripts/check_changes.rb)
           fastlane test only_testing:essential_tests

Conclusion

This was just a superficial analysis/rough idea of how we are able to influence on the speed of CI process. Keep it in mind and build your own pipeline as fast as you wish!

See ya (:

iOS apps under the microscope

2020-05-04T00:00:00+01:00

I once wrote a note about poking around Android apps. The other day I realised I still haven’t done it for iOS. Let’s fix the gap (:

Nice to have

Hopper Disassembler (free version is enough for now)
class-dump (binary selection depends on your app)
passionfruit (for those who like UI) – as it’s kind of archived I’ll update the note right after depreciation of the tool
objection (for those who adore CLI)
frida (em, no comments)
Jailbroken iDevice (whatever, e.g.: checkra1n)

Here we go

As an example I’ll take default Apple’s Calendar app:

com.apple.mobilecal

Binary neighbours

At first we should somehow get hold of decrypted app file (.ipa). To do so we can jump over to another branch where I explained the process in details.

As soon as we got the decrypded .ipa we are able to:

Imagine .ipa as archived folder:
```
 mv Calendar.ipa Calendar.zip
```
Unzip that folder
```
 unzip Calendar.zip
```
Get inside
```
 cd Payload/MobileCal.app/
```

Being there we can find binary itself (that names «MobileCal» without extension) and figure out what else lives near the binary, e.g.: .json and .plist files.

Strings and pseudocode

As soon as we got the binary file we can dive much deeper – inside the Assembler code via Hopper:

Open Hopper Disassembler app
Navigate to «File ↝ Open…»
Chose the app’s binary file
Confirm all default options by tapping on «OK» button
Wait for Hopper will analyze binary code

When Hopper has done its awesome work we are able to:

Consider strings «representation» of our sample app

Look for methods names

Enjoy pseudocode of those methods

Or block diagrams

Classes, methods and variables

Then we could get a list of the classes from the app in a readable format via class-dump:

As I mentioned before binary selection depends on your app:
- if your app is written in Objective-C, so grab ObjC binary
- elsif your app is written in Swift, just grab Swift binary
Put the binary to /usr/local/bin/ folder

Dump the app’s binary using class-dump

 class-dump-swift Payload/MobileCal.app/MobileCal > dump.txt

Enjoy classes content by opening dump.txt file

Keychain, local storage, url-schemes and much more

To get this data we could use for example: passionfruit or objection. For most day to day actions, these tools are pretty much the same with only one correction:

passionfruit is about UI
objection is about CLI

Just take your choice, but I’ll do a short intro how to get app’s data using both of them:

objection

objection -g "Calendar" explore

Command	Description
ios cookies get	Show app’s cookies
ios keychain dump	Show app’s keychain
ios nsurlcredentialstorage dump	Show nsurlcredentialstorage data
ios ui dump	Show current app’s screen hierarchy
ios bundles list_frameworks	Show app’s frameworks list
ios plist cat	Show content of chosen plist file
sqlite connect	Show content of chosen DB file

passionfruit

Run passionfruit server
```
 passionfruit
```
Open its main web page
```
 open http://localhost:31337
```
Chose your device and the app
Enjoy handy design with a lot of intuitive features

Widescale static analysis

There were many interesting tools, but let’s come back to old but cozy frida and grep all useful information about the app.

Just run the command and enjoy the output:

frida -U "Calendar" --codeshare interference-security/ios-app-static-analysis

Conclusion

Hope it helped on your way, see ya (:

Debugging third party iOS apps with lldb

2020-04-13T00:00:00+01:00

Requirements

macOS
Xcode
usbmuxd
Jailbroken iDevice connected via USB (e.g.: checkra1n)

Installation

Extracting debugserver (via hdiutil)

cd /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/DeviceSupport/<iOS version>
hdiutil attach DeveloperDiskImage.dmg
cp /Volumes/DeveloperDiskImage/usr/bin/debugserver ~/Desktop

Signing debugserver (via codesign)

touch entitlements.plist # fill the file using content bellow
codesign -s - --entitlements entitlements.plist -f debugserver

entitlements.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/ PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>com.apple.springboard.debugapplications</key> <true/>
  <key>run-unsigned-code</key> <true/>
  <key>get-task-allow</key> <true/>
  <key>task_for_pid-allow</key> <true/>
</dict>
</plist>

Connecting to iDevice (via iproxy)

iproxy 6666 6666 & iproxy 2222 44 & sleep 3 && ssh -p 2222 root@localhost

Copying debugserver to iDevice (via scp)

scp -P 2222 debugserver root@localhost:/usr/bin/

Running debugserver on iDevice

debugserver localhost:6666 -a "<app's CFBundleDisplayName>"

App’s CFBundleDisplayName can be obtained in many ways, e.g.:

Info.plist
ideviceinstaller
frida-ps

Running lldb on macOS

lldb
process connect connect://localhost:6666

Here we go

Now debug process is ready to go and possibly you would like to figure out more about each lldb command — to do so just jump to official lldb command map or their tutorial. Anyway I will share couple of lldb commands I think are most helpful and frequently used:

Common

Command	Description
help	Explore all available commands
po UIApplication.shared.delegate	Print object

Breakpoints

Command	Description
b	Set breakpoint
br dis 3	Disable third breakpoint
br en 3	Enable third breakpoint
br l	Show list of the existed breakpoints
br del 3	Remove third breakpoint
br del	Remove all breakpoints
br set -n viewDidLoad	Set breakpoint on the each viedDidLoad method
br s -S count	Set breakpoint on the each method with “count” argument
b Testme.swift:33	Set breakpoint on the line #33 in the Testme.swift file

Remote сontrol

Command	Description
c	Resume execution
n	Step over
s	Step in

Variables

Command	Description
e sum	Evaluate expression
e sum = 42	Set expression

Thread state

Command	Description
process status	Show process status
bt	Show the stack backtrace for the current thread
bt all	Show the stack backtraces for all threads
frame info	Show information about current frame
frame variable	Show all variables for current frame

Broad variety of waiting in Espresso

2020-02-22T00:00:00+00:00

What is «Waiting»?

Lifeguard from flaky tests.

What variety do we have?

Hmm, let me think… 🤔

We knew about Idling Resources

Pros:

It’s the most honest, fastest, and technically perfect way to wait;
It doesn’t eat up the memory.

Cons:

Sometimes it’s not a bit easier than writing production code;
Sometimes we’ll have to touch production code;
Most cases will probably require the own Idling Resources.

We heard about sleeps

Pros:

Kill me then.

Cons:

We’ll probably wait longer than we need;
We’ll perhaps wait less than we need.

We were guessing about the hacky way of injecting UiAutomator

Pros:

That’s something close to us since the time of Selenium.

Cons:

We’ll have to set an excess addiction;
We’ll be limited by minSdkVersion = 18;
We’ll have to play with weird UiObjects.

Go on, what about something else?

Okay… 😏

Let’s try one hacky way using Espresso

At first we have to extend ViewInteration class by adding new methods:

isDisplayed(): Boolean
doesNotExist(): Boolean
waitForAppear(millis: Long): ViewInteraction
waitForDisappear(millis: Long): ViewInteraction

fun ViewInteraction.isDisplayed(): Boolean {
    try {
        this.check(ViewAssertions.matches(ViewMatchers.isDisplayed()))
    } catch (ignored: NoMatchingViewException) {
        return false
    }
    return true
}

fun ViewInteraction.doesNotExist(): Boolean {
    try {
        this.check(ViewAssertions.doesNotExist())
    } catch (ignored: AssertionError) {
        return false
    }
    return true
}

fun ViewInteraction.waitForAppear(millis: Long = 5000): ViewInteraction {
    val timeout = System.currentTimeMillis() + millis
    while (System.currentTimeMillis() < timeout) {
        if (this.isDisplayed()) { break }
    }
    return this
}

fun ViewInteraction.waitForDisappear(millis: Long = 5000): ViewInteraction {
    val timeout = System.currentTimeMillis() + millis
    while (System.currentTimeMillis() < timeout) {
        if (this.doesNotExist()) { break }
    }
    return this
}

Then we can try to use it in the tests:

@Test
fun whenViewWillAppear() {
    val viewMatcher = ViewMatchers.withId(R.id.sample_view_id)
    Espresso.onView(viewMatcher)
        .waitForAppear()
        .check(ViewAssertions.matches(ViewMatchers.isDisplayed()))
}

@Test
fun whenViewWillDisappear() {
    val viewMatcher = ViewMatchers.withId(R.id.sample_view_id)
    Espresso.onView(viewMatcher)
        .waitForDisappear()
        .check(ViewAssertions.doesNotExist())
}

Pros:

Nothing is easier than use it for all the cases;
We’re able to control the waiting timeout for each case.

Cons:

Logcat will be littered;
After all, it a bit eats up the memory (comparing with Idling Resources).

Conclusion?

Yeah, waits are the delicate matter:

Follow the deadlines, needs, requirements and sanity;
Do whatever you want (using Espresso, of course 🧐);
Enjoy! 😉

Counting views in Espresso

2020-01-26T00:00:00+00:00

While doing pretty and flexible things with Espresso, have you ever faced with AmbiguousViewMatcherException? If so, go on reading, if no, you are lucky guy, just keep the article (:

When does this exception appear? Imagine, you have a RecyclerView with a lot of elements and some of them have the same resource-id as the neighbours have. Do you wanna count the specific element? Do you wanna tap on the first/last specific element? Do you wanna grep the elements by positions? You are more than welcome to stumble on the AmbiguousViewMatcherException!

Let’s try to play around it. I’ll show my vision, how to cover this exception and get a profit. Here we go!

The first way

is to create the object, that will handle the exception and grep a hierarchy of the views. It will give us an opportunity to get:

the first matching view;
the last matching view;
matching view at position;
the count of matching views.

object ViewCounter {
    fun first(matcher: Matcher<View>): Matcher<View?>? {
        return object : BaseMatcher<View>() {
            var isFirst = true
            override fun matches(item: Any): Boolean {
                if (isFirst && matcher.matches(item)) {
                    isFirst = false
                    return true
                }
                return false
            }

            override fun describeTo(description: Description) {
                description.appendText("should return first matching item")
            }
        }
    }

    fun last(matcher: Matcher<View>): Matcher<View?>? {
        return getElementFromMatchAtPosition(matcher, getCount(matcher) - 1)
    }

    fun getElementFromMatchAtPosition(matcher: Matcher<View>, position: Int): Matcher<View?>? {
        return object : BaseMatcher<View?>() {
            var counter = 0

            override fun matches(item: Any): Boolean = matcher.matches(item) && counter++ == position

            override fun describeTo(description: Description) {
                description.appendText("Element at hierarchy position $position")
            }
        }
    }

    fun getCount(viewMatcher: Matcher<View>, countLimit: Int = 5): Int {
        var actualViewsCount = 0
        do {
            try {
                Espresso.onView(ViewMatchers.isRoot())
                    .check(ViewAssertions.matches(withViewCount(viewMatcher, actualViewsCount)))
                return actualViewsCount
            } catch (ignored: Error) {
            }
            actualViewsCount++
        } while (actualViewsCount < countLimit)
        throw Exception("Counting $viewMatcher was failed. Count limit exceeded")
    }

    fun withViewCount(viewMatcher: Matcher<View>, expectedCount: Int): Matcher<View?>? {
        return object : TypeSafeMatcher<View?>() {
            var actualCount = -1
            override fun describeTo(description: Description) {
                if (actualCount >= 0) {
                    description.appendText("With expected number of items: $expectedCount")
                    description.appendText("\n With matcher: ")
                    viewMatcher.describeTo(description)
                    description.appendText("\n But got: $actualCount")
                }
            }

            override fun matchesSafely(root: View?): Boolean {
                actualCount = 0
                val iterable = TreeIterables.breadthFirstViewTraversal(root)
                actualCount = Iterables.filter(iterable, withMatcherPredicate(viewMatcher)).count()
                return actualCount == expectedCount
            }
        }
    }

    private fun withMatcherPredicate(matcher: Matcher<View>): Predicate<View?>? {
        return object : Predicate<View?> {
            override fun apply(@Nullable view: View?): Boolean {
                return matcher.matches(view)
            }
        }
    }
}

Comments

I see only one con: it limits us to the views, that we are considering at the moment, and hides from us everything that is outside the screen.

Showcase

@Test
fun sample() {
  val fab = withId(R.id.fab)

  onView(ViewCounter.first(fab)).perform(click())

  onView(ViewCounter.last(fab)).perform(click())

  onView(isRoot()).check(matches(ViewCounter.withViewCount(fab, 3))
}

The second way

is to create two classes:

The first class will match the RecyclerView. It will give us an opportunity to parse the RecyclerView and to get:

matching view at position

any view at position

  fun withRecyclerView(recyclerViewId: Int): RecyclerViewMatcher {
      return RecyclerViewMatcher(recyclerViewId)
  }

  class RecyclerViewMatcher(val recyclerViewId: Int) {
      fun atPosition(position: Int): Matcher<View> {
          return atPositionOnView(position, -1)
      }

      fun atPositionOnView(position: Int, targetViewId: Int): Matcher<View> {
          return object : TypeSafeMáatcher<View>() {
              var resources: Resources? = null
              var childView: View? = null
              override fun describeTo(description: Description?) {
                  val id = if (targetViewId == -1) recyclerViewId else targetViewId
                  var idDescription = id.toString()
                  if (resources != null) {
                      idDescription = try {
                          resources!!.getResourceName(id)
                      } catch (var4: NotFoundException) {
                          String.format("%s (resource name not found)", id)
                      }
                  }
                  description!!.appendText("with id: $idDescription")
              }

              override fun matchesSafely(view: View?): Boolean {
                  resources = view?.resources
                  if (childView == null) {
                      val recyclerView = view?.rootView?.findViewById(recyclerViewId) as RecyclerView
                      childView = if (recyclerView != null) {
                          recyclerView.findViewHolderForAdapterPosition(position)!!.itemView
                      } else {
                          return false
                      }
                  }
                  return if (targetViewId == -1) {
                      view === childView
                  } else {
                      val targetView: View = childView!!.findViewById(targetViewId)
                      view === targetView
                  }
              }
          }
      }
  }

The second class will handle the AmbiguousViewMatcherException and grep a hierarchy of the whole RecyclerView. Thanks to it we’ll be able to get:

the first matching view
the last matching view
the count of matching views

the possibility to consider all the views inside and outside the screen

  fun recyclerViewCounter(@IdRes recyclerViewId: Int, action: RecyclerViewCounter.() -> Unit) {
      action(RecyclerViewCounter(recyclerViewId))
  }

  class RecyclerViewCounter(@IdRes private val recyclerViewId: Int) {
      fun getView(atPosition: Int): Matcher<View> {
          return withRecyclerView(recyclerViewId).atPosition(atPosition)
      }

      fun getCount(expectedView: Matcher<View>): Int {
          val elementsCount = getCount()
          var expectedViewsCount = 0
          repeat(elementsCount) {
              val result = scanPosition(it, expectedView)
              if (result) {
                  expectedViewsCount++
              }
          }
          return expectedViewsCount
      }

      fun getLastMatchingPosition(expectedView: Matcher<View>): Int {
          val elementsCount = getCount()
          var lastMatchingPosition = -1
          repeat(elementsCount) {
              val result = scanPosition(it, expectedView)
              if (result) {
                  lastMatchingPosition = it
              }
          }
          return lastMatchingPosition
      }

      fun getFirstMatchingPosition(expectedView: Matcher<View>): Int? {
          val elementsCount = getCount()
          repeat(elementsCount) {
              val result = scanPosition(it, expectedView)
              if (result) {
                  return it
              }
          }
          return null
      }

      private fun getCount(): Int {
          val count = intArrayOf(0)
          val matcher: Matcher<View> = object : TypeSafeMatcher<View>() {
              override fun matchesSafely(item: View): Boolean {
                  count[0] = (item as RecyclerView).adapter!!.itemCount
                  return true
              }

              override fun describeTo(description: Description) {
                description.appendText("Count of the views inside the RecyclerView")
              }
          }
          onView(CoreMatchers.allOf(ViewMatchers.withId(recyclerViewId), ViewMatchers.isDisplayed()))
              .check(matches(matcher))
          return count.first()
      }

      private fun scanPosition(position: Int, expectedView: Matcher<View>): Boolean {
          try {
              onView(ViewMatchers.withId(recyclerViewId))
                  .perform(RecyclerViewActions.scrollToPosition<RecyclerView.ViewHolder>(position))
              onView(withRecyclerView(recyclerViewId).atPosition(position))
                  .check(matches(hasDescendant(expectedView)))
          } catch (ignored: AssertionError) {
              return false
          }
          return true
      }
  }

Comments

This way is a bit more complicated, but definitely more flexible and reliable. Also, it «almost» doesn’t contradict the Espresso paradigm (:

Showcase

@Test
fun sample() {
  recyclerViewCounter(R.id.recyclerViewId) {
    val fab = withId(R.id.fab)

    val firstFabPosition = getFirstMatchingPosition(fab)
    onView(getView(firstFabPosition)).perform(click())

    val lastFabPosition = getLastMatchingPosition(fab)
    onView(getView(lastFabPosition)).perform(click())

    val actualFabCount = getCount(fab)
    assertEquals(3, actualFabCount)
  }
}

Conclusion

The AmbiguousViewMatcherException and counting views in Espresso are not the problems anymore.

Wish you happy Mondays (:

Configure secure web proxy on macOS via command line

2019-06-25T00:00:00+01:00

Set secure web proxy on Wi-Fi

networksetup -setsecurewebproxy Wi-Fi "127.0.0.1" "1234"

Set secure web proxy on Ethernet

networksetup -setsecurewebproxy Ethernet "127.0.0.1" "1234"

Turn secure web proxy on

networksetup -setsecurewebproxystate Ethernet 1

Turn secure web proxy off

networksetup -setsecurewebproxystate Ethernet 0

To get more interesting things about network automation read this note.

Sniffing network traffic on Android ≥7

2019-06-20T00:00:00+01:00

Introduction

Today I want to write about sniffing network traffic on Android. Let me show you what we had before and what we have after Android Nougat:

In other words, Google removed the feature to sniff a network traffic out of the box in Android Nougat. Furthermore, it means, that you won’t be able to sniff a network traffic in apps from Play Market. Thereby, we should install network security config manually to resurrect sniffing feature. There are two ways to do it:

add network_security_config.xml in the source
inject network_security_config.xml in an .apk file

training/articles/security-config

Addition network_security_config.xml in the source

default

<?xml version="1.0" encoding="utf-8"?>
<manifest ... >
    <application android:networkSecurityConfig="@xml/network_security_config"
                    ... >
        ...
    </application>
</manifest>

custom

<?xml version="1.0" encoding="utf-8"?>
<network-security-config>
    <domain-config>
        <domain includeSubdomains="true">example.com</domain>
        <trust-anchors>
            <certificates src="@raw/my_ca"/>
        </trust-anchors>
    </domain-config>
</network-security-config>

But there is a high probability, that Google may reject apps have network_security_config.xml, because it’s a «care» about peace, rock ‘n’ roll and safety.

Injection network_security_config.xml in an .apk file

Here we are able to build a release .apk, duplicate it and make a network_security_config.xml injection. So, let’s start:

Build an .apk

Clone the repo:

 git clone https://github.com/levyitay/AddSecurityExceptionAndroid.git

Run a script from the repo:
```
 ./addSecurityExceptions.sh yourApp.apk
```
- the script unpacks your .apk via apktool
- injects network_security_config.xml into AndroidManifest.xml
- signs and repacks your .apk
Your app is ready for test, install it and enjoy the sniffing

Conclusion

Hope it was interesting or at least a bit useful (:

Thanks for the reading and see you!

Pull out whole information about any App Store apps

2019-06-14T00:00:00+01:00

Get an url of required app (e.g. YouTube)

https://apps.apple.com/us/app/youtube-watch-listen-stream/id544007664

Take an app identifier and set it in a new request

https://itunes.apple.com/lookup?id=544007664

Enjoy the result

version, rating, release date and much more, just look at the sample response:

{
  "results" : [
    {
      "primaryGenreName" : "Photo & Video",
      "artworkUrl100" : "https:\/\/is3-ssl.mzstatic.com\/image\/thumb\/Purple123\/v4\/4f\/e3\/6b\/4fe36b1b-e7ef-cc70-a3fb-4da243252d85\/source\/100x100bb.jpg",
      "sellerUrl" : "https:\/\/www.youtube.com",
      "currency" : "USD",
      "artworkUrl512" : "https:\/\/is3-ssl.mzstatic.com\/image\/thumb\/Purple123\/v4\/4f\/e3\/6b\/4fe36b1b-e7ef-cc70-a3fb-4da243252d85\/source\/512x512bb.jpg",
      "ipadScreenshotUrls" : [
        "https:\/\/is1-ssl.mzstatic.com\/image\/thumb\/Purple113\/v4\/67\/54\/68\/67546885-887f-5ea9-74e9-cafd26fabeb1\/mzl.xproroqw.png\/1024x768bb.png",
        "https:\/\/is4-ssl.mzstatic.com\/image\/thumb\/Purple113\/v4\/d0\/c2\/81\/d0c281a1-0d6a-2656-170f-66ccbd5b3b24\/mzl.zzfhxybc.png\/1024x768bb.png",
        "https:\/\/is2-ssl.mzstatic.com\/image\/thumb\/Purple123\/v4\/43\/4c\/72\/434c726b-c8a0-9319-5070-5a5555b493e9\/mzl.gpcziwoh.png\/1024x768bb.png",
        "https:\/\/is2-ssl.mzstatic.com\/image\/thumb\/Purple113\/v4\/f0\/d7\/f9\/f0d7f9d5-8c27-2138-075b-1b039a7a1175\/mzl.qczwltsx.png\/1024x768bb.png",
        "https:\/\/is3-ssl.mzstatic.com\/image\/thumb\/Purple113\/v4\/e8\/c3\/2f\/e8c32f0d-5a43-eda2-0967-58db5bb8740b\/mzl.dptqwucx.png\/1024x768bb.png"
      ],
      "fileSizeBytes" : "263431168",
      "genres" : [
        "Photo & Video",
        "Entertainment"
      ],
      "languageCodesISO2A" : [
        "IE", "..."
      ],
      "artworkUrl60" : "https:\/\/is3-ssl.mzstatic.com\/image\/thumb\/Purple123\/v4\/4f\/e3\/6b\/4fe36b1b-e7ef-cc70-a3fb-4da243252d85\/source\/60x60bb.jpg",
      "supportedDevices" : [
        "iPhone5-iPhone5",
        "iPadFourthGen-iPadFourthGen",
        "iPadFourthGen4G-iPadFourthGen4G",
        "iPhone5c-iPhone5c",
        "iPhone5s-iPhone5s",
        "iPadAir-iPadAir",
        "iPadAirCellular-iPadAirCellular",
        "iPadMiniRetina-iPadMiniRetina",
        "iPadMiniRetinaCellular-iPadMiniRetinaCellular",
        "iPhone6-iPhone6",
        "iPhone6Plus-iPhone6Plus",
        "iPadAir2-iPadAir2",
        "iPadAir2Cellular-iPadAir2Cellular",
        "iPadMini3-iPadMini3",
        "iPadMini3Cellular-iPadMini3Cellular",
        "iPodTouchSixthGen-iPodTouchSixthGen",
        "iPhone6s-iPhone6s",
        "iPhone6sPlus-iPhone6sPlus",
        "iPadMini4-iPadMini4",
        "iPadMini4Cellular-iPadMini4Cellular",
        "iPadPro-iPadPro",
        "iPadProCellular-iPadProCellular",
        "iPadPro97-iPadPro97",
        "iPadPro97Cellular-iPadPro97Cellular",
        "iPhoneSE-iPhoneSE",
        "iPhone7-iPhone7",
        "iPhone7Plus-iPhone7Plus",
        "iPad611-iPad611",
        "iPad612-iPad612",
        "iPad71-iPad71",
        "iPad72-iPad72",
        "iPad73-iPad73",
        "iPad74-iPad74",
        "iPhone8-iPhone8",
        "iPhone8Plus-iPhone8Plus",
        "iPhoneX-iPhoneX",
        "iPad75-iPad75",
        "iPad76-iPad76",
        "iPhoneXS-iPhoneXS",
        "iPhoneXSMax-iPhoneXSMax",
        "iPhoneXR-iPhoneXR",
        "iPad812-iPad812",
        "iPad834-iPad834",
        "iPad856-iPad856",
        "iPad878-iPad878",
        "iPadMini5-iPadMini5",
        "iPadMini5Cellular-iPadMini5Cellular",
        "iPadAir3-iPadAir3",
        "iPadAir3Cellular-iPadAir3Cellular",
        "iPodTouchSeventhGen-iPodTouchSeventhGen"
      ],
      "trackViewUrl" : "https:\/\/apps.apple.com\/us\/app\/youtube-watch-listen-stream\/id544007664?uo=4",
      "description" : "Make watching your favorite videos easier with the YouTube app. Explore brand new content, music, news and more with the official YouTube app for iPhone and iPad.\n\nDiscover even more with the official YouTube app. Subscribe to channels with your favorite content, share them with friends or upload your very own videos for everyone to see.\n\nGet more out of your video streaming app and discover even more with YouTube.\n\nYOUTUBE FEATURES:\n\nDISCOVER BRAND NEW CONTENT\n- Browse YouTube’s personal recommendations on the home tab\n- Watch the hottest videos, from music to viral sensations on the trending tab\n- Like something? Tap like to save to your personal list or share with your friends!\n\nSTREAM MUSIC\n- Find the videos of the latest hits! \n- Watch fresh releases from popular and trending artists\n\nLIVE STREAMS & MORE\n- Get the latest news or watch your favorite popular live streams right on your device!\n\nLEARN FROM THE BEST\n- Practice and learn alongside the best online\n- Watch videos and learn any skill you need \n- Explore thousands of videos for makeup, DIY, tips and more!\n\nCONNECT AND SHARE\n- Let people know how you feel with likes, comments, or shares\n- Directly share your friends favorite videos with your friends\n- Make a playlist that saves your favorite videos\n- See the latest from the creators you follow on the subscriptions tab\n\nDownload the official YouTube app and start watching your favorite videos today!\n\nWITH YOUTUBE PREMIUM, GET YOUTUBE AD-FREE, OFFLINE, AND IN THE BACKGROUND. PLUS, ACCESS TO ALL YOUTUBE ORIGINALS.\n- Watch the videos you love without ads!\n- Play videos in the background, even when you open another app\n- Download videos for when you're low on data or can't get online\n- Discover original series and movies from today's hottest talent\n\nIf you subscribe via iTunes:\n\nPayment will be charged to iTunes Account at confirmation of purchase. Subscription automatically renews unless auto-renew is turned off at least 24 hours before the end of the current period. Account will be charged for renewal within 24 hours prior to the end of the current period at the rate of the selected plan. Subscriptions and auto-renewal may be managed by going to Account Settings after purchase\n\nAny unused portion of a free trial period will be forfeited when purchasing a subscription.\n\nMonthly price: $15.99\n\nYouTube paid service terms: https:\/\/www.youtube.com\/t\/terms_paidservice\nPrivacy policy: https:\/\/www.google.com\/policies\/privacy",
      "version" : "14.23",
      "bundleId" : "com.google.ios.youtube",
      "artistViewUrl" : "https:\/\/apps.apple.com\/us\/developer\/google-llc\/id281956209?uo=4",
      "userRatingCountForCurrentVersion" : 86090,
      "appletvScreenshotUrls" : [],
      "isGameCenterEnabled" : false,
      "releaseDate" : "2012-09-11T07:23:19Z",
      "averageUserRatingForCurrentVersion" : 4.5,
      "genreIds" : [
        "6008",
        "6016"
      ],
      "trackId" : 544007664,
      "wrapperType" : "software",
      "minimumOsVersion" : "10.0",
      "formattedPrice" : "Free",
      "primaryGenreId" : 6008,
      "currentVersionReleaseDate" : "2019-06-09T23:07:29Z",
      "userRatingCount" : 7694122,
      "artistId" : 281956209,
      "trackContentRating" : "17+",
      "artistName" : "Google LLC",
      "price" : 0,
      "trackCensoredName" : "YouTube: Watch, Listen, Stream",
      "trackName" : "YouTube: Watch, Listen, Stream",
      "kind" : "software",
      "features" : [
        "iosUniversal"
      ],
      "contentAdvisoryRating" : "17+",
      "screenshotUrls" : [
        "https:\/\/is3-ssl.mzstatic.com\/image\/thumb\/Purple123\/v4\/4a\/a3\/3a\/4aa33a69-acd5-9779-3caf-c1f0f28e4ff7\/mzl.gpcntyew.png\/696x696bb.png",
        "https:\/\/is4-ssl.mzstatic.com\/image\/thumb\/Purple113\/v4\/da\/bb\/6a\/dabb6aca-31a4-fe77-9a92-312dafd5f323\/mzl.rnzdewny.png\/696x696bb.png",
        "https:\/\/is2-ssl.mzstatic.com\/image\/thumb\/Purple113\/v4\/fc\/a4\/4a\/fca44a97-0948-ada2-3e72-5d3127bad759\/mzl.pfcurnea.png\/696x696bb.png",
        "https:\/\/is2-ssl.mzstatic.com\/image\/thumb\/Purple123\/v4\/a9\/86\/2e\/a9862e97-afe3-40d8-4682-29e03404fa11\/mzl.qodnlftq.png\/696x696bb.png"
      ],
      "releaseNotes" : "Fixed bugs, improved performance, took out the garbage, mowed the front lawn, and now we need a little nap.",
      "isVppDeviceBasedLicensingEnabled" : true,
      "sellerName" : "Google LLC",
      "averageUserRating" : 4.5,
      "advisories" : [
        "Infrequent\/Mild Horror\/Fear Themes",
        "Infrequent\/Mild Sexual Content and Nudity",
        "Infrequent\/Mild Simulated Gambling",
        "Infrequent\/Mild Medical\/Treatment Information",
        "Infrequent\/Mild Profanity or Crude Humor",
        "Infrequent\/Mild Realistic Violence",
        "Infrequent\/Mild Cartoon or Fantasy Violence",
        "Infrequent\/Mild Alcohol, Tobacco, or Drug Use or References",
        "Frequent\/Intense Mature\/Suggestive Themes"
      ]
    }
  ],
  "resultCount" : 1
}

Opening the curtains (iOS/Android)

2019-06-12T00:00:00+01:00

Swiping, swiping, swiping. From dusk till dawn.

iOS

extension XCUIApplication {
    func openNotifications() {
        let up = self.coordinate(withNormalizedOffset: CGVector(dx: 0.1, dy: 0.001))
        let down = self.coordinate(withNormalizedOffset: CGVector(dx: 0.1, dy: 0.8))
        up.press(forDuration: 0.1, thenDragTo: down)
    }
}

extension XCUIApplication {
    func openControlCenter() {
        let down = self.coordinate(withNormalizedOffset: CGVector(dx: 0.5, dy: 0.999))
        let up = self.coordinate(withNormalizedOffset: CGVector(dx: 0.5, dy: 0.2))
        down.press(forDuration: 0.1, thenDragTo: up)
    }
}

Android

fun openNotifications() {
    val device = UiDevice.getInstance(InstrumentationRegistry.getInstrumentation())
    device.executeShellCommand("input swipe 0 0 0 500")
}

fun openLaunchPad() {
    val device = UiDevice.getInstance(InstrumentationRegistry.getInstrumentation())
    device.executeShellCommand("input swipe 0 500 0 0")
}

2019-06-01T00:00:00+01:00

Git Hooks are really useful guys that help us to control interactions with Git. I would like to show you couple of samples of their usage.

Precondition

First of all we should:

create .githooks folder near the .git folder
create a Makefile in the project’s root that will set githooks path and activate them
fill the Makefile like in the sample below:

init:
  git config core.hooksPath .githooks
  git config advice.ignoredHook false

To force Makefile works, just enough to execute this command in the terminal:

make

Commit message

If we are working in a feature branch workflow it would be pretty nice to have an issue name for each commit developer does. To realize it we could:

create commit-msg file into the .githooks folder
add a regex into the commit-msg file that will verify the commits messages:

#!/bin/bash

commit_regex="(ISSUE-[0-9]+|TASK-[0-9]+|merge)"
if ! grep -iqE "$commit_regex" "$1"; then
  echo "You have to add issue tag in the commit"
  exit 1
fi

If developer won’t set an issue tag in a commit message, he/she will get an error with request to add the issue tag.

User email

Imagine, we would like each developer to use only corporation emails. So, to realize it we could:

open the same commit-msg file in the .githooks folder
add a regex into commit-msg file that will verify developer’s email:

#!/bin/bash

USER_EMAIL=$(git config user.email)
CORP_DOMAIN="@corp.company.com"
if [[ $USER_EMAIL != *"$CORP_DOMAIN" ]]; then
  echo "Your email ($USER_EMAIL) has to contain this domain: $CORP_DOMAIN"
  exit 1
fi

If developer won’t set a corp email in the git project, he/she will get an error with request to add the correct email

Conclusion

Eventually I would like to highlight Git Hooks give us a soil (or better to say hooks?) for automation. Whether it be issue names collection for release or a simple censure in the commit messages.

Jenkins which deploys itself

2019-05-27T00:00:00+01:00

Jenkins - free, flexible, popular and gorgeous continuous integration server.
Jenkins Pipeline - plugin to build continuous delivery pipelines as a code (further Pipeline).
Jenkins Job Builder - tool to make jobs from templates in a human readable text format and keep them in a version control system (further JJB).

Purpose

minimize or exclude the relationships with Jenkins UI
provide CI which will deploying itself
make things easy and flexible

Precondition

make sure you have already installed Jenkins
install pipeline plugin
install JJB
create git repository for job templates and pipeline files

Provide access for JJB

go to user configuration page:

 http://<jenkins url>/user/<user name>/configure

generate API Token:
create jenkins_jobs.ini file in the root of repository and put API Token (password field) and User Name (user field) there:
```
 [jenkins]
 password=2433a61e9bc33fb4633de6d39d332b33
 user=a.alterpesotskiy
```

CI Server which is deploying itself

Our first job will be the most important. It will be essential, because of its influence. This job will have to update all Jenkins jobs after every commit with changes. We will call it «JenkinsDeploy». So, let’s go:

create JenkinsDeploy job via Jenkins web interface

This is an exception, because it will be only the one job we create via Jenkins UI
create JenkinsDeploy.yml template in the jobs/ folder

fill the template like this:

 - job:
     builders:
     - shell: |-
         jenkins-jobs --log_level debug --conf jenkins_jobs.ini update jobs/
     name: 'JenkinsDeploy'
     project-type: freestyle
     scm:
     - git:
         branches:
         - '*/master'
         credentials-id: bc71cef3-f7ca-4ec1-3325-9c13d871b92b
         url: git@gitlab.alteral.ru:sample/jenkins.git
         wipe-workspace: true
     triggers:
     - pollscm:
         cron: '* * * * *'
         ignore-post-commit-hooks: false

I’ll try to explain what is taking place in the job template:

Jenkins clones the repository with a project from git (via scm plugin)
JJB updates Jenkins jobs from templates (via jenkins-jobs cli)
It does it after each commit (checking for changes every minute)

Project contents

Sample: JJB + Pipeline

Finally I might to show how everything works here at the simple example. How to make it as simple as possible? We will build a job, that will create an empty folder. That’s all.

template (jobs/mkdir.yml)

- job:
    project-type: pipeline
    name: 'MKDIR'
    sandbox: true
    parameters:
    - string:
        default: 'test'
        description: 'folder name'
        name: FOLDER_NAME
    pipeline-scm:
        scm:
        - git:
            url: git@gitlab.alteral.ru:sample/jenkins.git
            branches:
              - '*/master'
            clean: true
            credentials-id: bc71cef3-f7ca-4ec1-3325-9c13d871b92b
        script-path: pipeline/mkdir.groovy
        lightweight-checkout: true
    wrappers:
    - build-name:
        name: "#$BUILD_NUMBER mkdir $FOLDER_NAME"

pipeline (pipeline/mkdir.groovy)

timeout(5) {
  node("master") {
    stage("Create folder") {
      sh("mkdir -p ${FOLDER_NAME}")
      echo("${FOLDER_NAME} was successfully created.")
    }
  }
}

So, this job will be automatically deployed after push via JenkinsDeploy job.

Conclusion

What opportunities does it open?

We might to describe whole CI/CD process as a code.
We might to combine many jobs and pipeline scripts in one template via JJB.
We might to relish of the resulting workspace.
We might to spend our obtained free time on useful things.

Also I would like to add that via JJB we can describe many Jenkins Plugins in .yml/.json representations and use many strategies to make our pipelines more flexible and expandable. If it was at least a little bit interesting, you can borrow much more interestingness from the official documentation:

jenkins-job-builder/latest/jenkins-job-builder.pdf

Thanks for reading (:

OCR or getting text from the screenshot

2019-04-07T00:00:00+01:00

Today I would like to consider one awesome thing — OCR (optical character recognition or optical character reader) and try to apply it in mobile automation.

Last time I told about snapshot testing in Android and iOS, but this is completely different feature.

Purpose

I’ll show you how to read text from a mobile screen. I know we could do it with XCTest or Espresso (or even UIAutomator) in our tests, finding elements via locators, but imagine the situation when you are not able to use automation frameworks or, and it will be better example, you have a Unity app.

Yeah, Unity is not the most testable platform. Unity tests run in the app, the result after (and no else) looks like this:

Fork in the road

Here I see at least three ways:

compare screenshots — umm, store referent images?
compare pixel colour — is it really your best decision?
read the text — perfect one! (:

So, let’s go with the last one.

Fulfilment

First of all we should take a screenshot:

Android

 adb shell /system/bin/screencap -p /sdcard/screenshot.png
 adb pull /sdcard/screenshot.png screenshot.png

iOS

 idevicescreenshot screenshot.png

Main hero here is a tesseract — cli for OCR
```
 brew install tesseract
```
Eventually we will read the text from screenshot and print it in the terminal
```
 tesseract screenshot.png stdout -l eng
```

Bonus

If you have a pretty small text size like in my sample, it could be really difficult for tesseract to recognise it. There is one thing we are able to do:

brew install imagemagick

Imagemagick is a gorgeous tool for working with images via terminal and has truly a lot of features. So, below you can see an example how to cut down the image to 500px wide and 100px high starting at pixel x==0 && y==0:

convert screenshot.png -crop 500x100+0+0 screenshot.png
tesseract screenshot.png stdout -l eng

Conclusion

Today we affect two awesome tools — tesseract and imagemagick. Suggest you to read more about them and put into practice. Perhaps it will be a pretty start to the love of working with images.

See you soon (:

Android apps under the microscope

2019-04-06T00:00:00+01:00

Would you like to look under the hood of any Android app?

You are welcome! (:

Precondition

dex2jar — tool for converting .dex file to .class files

brew install dex2jar

jd-gui — tool for decompiling and analysing Java byte code

brew cask install jd-gui

Let’s go

First of all we should get the .apk. If you haven’t one, you are able to download it from store (for more info read this note.
Then we need to convert our .apk into the classes (.jar):
```
 d2j-dex2jar test.apk -o test.jar
```
Eventually we should just decompile received .jar and look under the hood:
```
 open -a jd-gui test.jar
```

Conclusion

Today we affect two awesome tools — dex2jar and jd-gui. Suggest you to put them into practice. Perhaps it will be a deeply fascinating diving in reverse engineering.

Thanks and see you later (:

Bypassing iOS application verification error

2019-03-31T00:00:00+00:00

Sometimes, when you’d like to install a third-party iOS app outside of the App Store on your device you might come across with «Application Verification Failed» error.

It can be a rare case, because it’s really a concatenation of circumstances and means, that you received the app without your team’s provision profile. Anyways, we are able to solve this problem.

Plan

Get own provisioning profile
Resign the app
Install the app

Precondition

First of all, to make our life better and easier, we should install ideviceinstaller and fastlane:

brew install ideviceinstaller
gem install fastlane

Provisioning profile

Then we need to get a provision profile. If you already have it, you can jump to the next step, if not, let’s go on.

As Apple said:

Xcode is the easiest way to create an iOS provisioning profile. Connect your device to your Mac and click Use for Development in Xcode’s Organizer window. Sign in with the Apple ID associated with your Apple Developer Program membership and Xcode will generate a provisioning profile containing your certificate and device identifier.

So, in this case it doesn’t matter of signing type, will it be personal team or organisation team.

Resign the app

Here we need three things:

.ipa
.mobileprovision
iOS Developer Certificate

I hope you have got an .ipa, other ones we can get from Xcode:

So:

drag and drop iOS Team Provisioning Profile from Xcode tooltip into the folder with .ipa
copy iPhone Developer Certificate from the same tooltip
then resign the app via fastlane like this:

fastlane sigh resign testapp.ipa --signing_identity "iPhone Developer: a.alterpesotskiy@test.com (KJRTP5GJS9)" -p "test.mobileprovision"

Install the app

P.S.:

If you see this pop-up, you can go to Settings => General => Device Management on the device to trust the developer and allow the app to be run (see Apple documentation for more information):

Conclusion

To sum it up, I would say that «Application Verification Failed» is hardly a problem. Now we can install any .ipa on any devices (:

How to download .ipa file from App Store

2019-03-08T00:00:00+00:00

Introduction

Here are three ways to download iOS apps from Apple App Store for your attention:

Using Apple Configurator 2 (legal way)
- Here you need to have an app on your phone
Using ipatool (kinda legal way)
- Here you need appleId credentials
Using any iOS executable dumper (illegal way)
- Here you have to be Jailbroken

Using Apple Configurator

Download an app from App Store to your phone
Open Apple Configurator 2 on your Mac and log into your Apple account
Connect your phone to your Mac using a USB cable
Into Apple Configurator 2 select your phone
Then tap on the «Add» button in the top, then tap on Apps button
Choose the app and tap on «Add» button
Finally it could tell you an app already exists, but don’t worry — just be in a hurry to grab an .ipa using this path:

~/Library/Group\ Containers/K36BKF7T3D.group.com.apple.configurator/Library/Caches/Assets/TemporaryItems/MobileApps/

Using ipatool

Install the tool

 brew tap majd/repo
 brew install ipatool

Authenticate with the App Store

 ipatool auth login -e <appleId> -p <password>

Download the app
```
 ipatool download -b <bundleId>
```

Using any iOS executable dumper

As I said for this method you will need a Jailbreak. It can be illegal and provoke side effects, so do it at your own risk. When writing this article I used Checkra1n and frida-ios-dump for this purpose.

On device:

Make a Jailbreak
Open Cydia app
Add the source «https://build.frida.re»
Search for Frida and install it

On computer:

Install frida
```
 pip3 install frida-tools
```

Install frida-ios-dump

 git clone https://github.com/AloneMonkey/frida-ios-dump.git
 cd frida-ios-dump
 pip3 install -r requirements.txt --upgrade

Install usbmuxd
```
 brew install usbmuxd
```

Connect to your phone using ssh

 iproxy 2222 44 & sleep 3
 ssh -p 2222 root@localhost # default password: alpine

Open a new Terminal window and pull decrypted .ipa

 cd frida-ios-dump
 python3 dump.py "<your_bundle>" # default password: alpine

Conclusion

Downloading apps from Apple App Store can be a fairly simple process, either manual or automated.

Also you can find an answer how to download an .apk from Android Play Store here.

How to download .apk file from Play Store

2019-03-07T00:00:00+00:00

Introduction

There are three ways to download Android apps from Google Play Store for your attention:

Using third party websites (easiest, but not the safest way)
Using device on Android (the most secure and exact way)
Using google-play-cli (automation friendly way)

To download *.apks* following this method you have to know app's bundle identifier.

Using third party websites

As I said, this is not the provenest and the safest method, so use it at your own risk (:

Using device on Android

At first you need to install app from Play Store to your phone
Secondly you should enable «Unknown sources» in your phone’s security settings (Settings => Security => Unknown sources)
Then connect your phone to your PC using a USB cable/WiFi (adb needs to see it)
Make sure you have installed adb
Download an .apk:

bundleId="<your_bundle>"
apk=`adb shell pm path $bundleId`
apk=`echo $apk | awk '{print $NF}' FS=':' | tr -d '\r\n'`
adb pull $apk app.apk

Using Google-Play-CLI

The best way to make this process automated.

Install google-play-cli:

 brew install node
 npm i google-play-cli -g

Set environment variables in your bash.profile:
- GOOGLE_LOGIN - email address used on a mobile phone.
- GOOGLE_PASSWORD - The password used to access the Play Service.
- ANDROID_ID - the device GSF ID for Google. You can get the gsf id e.g., using the device id app.
```
 touch .bash_profile
 open .bash_profile
 export GOOGLE_LOGIN=sample@gmail.com
 export GOOGLE_PASSWORD=password
 export ANDROID_ID=a3n3d3r3o3i3d3i3d
 source .bash_profile
```
Download an .apk:
```
 gp-download "<your_bundle>" > app.apk
```

Conclusion

Downloading apps from Google Play Store can be a fairly simple process, either manual or automated.

Also you can find an answer how to download an .ipa from Apple App Store here.

MITM can be pretty easy with Mitmproxy and Python

2019-02-25T00:00:00+00:00

Introduction

First of all, I would like to link wiki MITM definition. If you are interested in my simple explanation, MITM is a hacker attack, in which there is a client, a server, and that man from the title in the middle. Man in the middle changes the original connection of the user and receives full access to his traffic with the ability to add or remove any stuff in requests and responses.

Mitmproxy — is your swiss-army knife for debugging, testing, privacy measurements, and penetration testing. This is about HTTP and HTPPS requests and touches on not only mobile (iOS/Android), but also OSX, Windows, and Linux apps. Especially nice that it is FREE.

In this note, I won’t go through the advantages of mitmproxy over other similar tools (Charles, scat!), but trust me, it would vanquish any tool out there almost single-handedly.

Install

brew install mitmproxy

A broad variety of UI

Mitmproxy has three interfaces for every taste:

Cli
```
 mitmproxy
```
Web (I personally prefer this one)
```
 mitmweb
```
Log dump
```
 mitmdump
```

Usage

Connect to the same Wi-Fi on computer and device
Start mitmproxy on computer
Open Wi-Fi settings on the device and set up a proxy with laptop IP and mitmproxy port (8080 by default)
Open mitm.it on device browser and choose your platform
Then install the downloaded certificate and trust it

Now you can sniff your traffic via mitmproxy.

More information about installing the certificates

Ignore hosts

A useful option when we don’t have to sniff traffic from some hosts, but if we sniff them we can receive a lot of problems. For example, if we don’t ignore:

android.clients.google.com, we will have problems with signing into the Play Store

init.itunes.apple.com and itunes.apple.com, we will have problems with signing into the App Store

ppq.apple.com, we will get this pop-up for some apps

So, let’s ignore these hosts. In order to do it we should set up a little regex at the startup of mitmproxy:

mitmweb --ignore-hosts '^(?:(?!android.clients.google.com|appldnld.apple.com|mesu.apple.com|ppq.apple.com).)*$'

What Python brings?

Python is the game-changer and the killer feature at the same time.

The main things to consider for the quick start are:

method request, if we want to modify our requests and/or create new responses
method response, if we want to modify server responses

from mitmproxy import http

def request(flow: http.HTTPFlow) -> None:
   print("here will be request modification via flow.request and its params")

def response(flow: http.HTTPFlow) -> None:
   print("here will be response modification via flow.response and its params")

We’re lucky guys ‘cause using these two methods we can control the whole traffic.

Attention! The following examples are really useful but don’t claim to be exhaustive, but only describe some of the possibilities of interaction with the mitmproxy API. Something heavier, and generally anything you want you can find in the official examples.

kill requests (e.g.: kill iOS requests for update):

from mitmproxy import http

block_hosts = [ "appldnld.apple.com", "mesu.apple.com" ]

def request(flow: http.HTTPFlow) -> None:
  if any(host in flow.request.host for host in block_hosts):
    flow.kill

change query/headers of requests/responses:

from mitmproxy import http

def request(flow: http.HTTPFlow) -> None:
  flow.request.query["mitm"] = "proxy"
  flow.request.headers["newheader"] = "foo"

def response(flow: http.HTTPFlow) -> None:
  flow.response.query["proxy"] = "mitm"
  flow.response.headers["newheader"] = "bar"

change response code (e.g.: return 503 for offline emulation):

from mitmproxy import http

def response(flow: http.HTTPFlow) -> None:
  flow.response.status_code = 503

emulate slow connection:

from mitmproxy import http
import time

def response(flow: http.HTTPFlow) -> None:
  time.sleep(5.0)

redirect requests:

from mitmproxy import http

def request(flow: http.HTTPFlow) -> None:
  flow.request.host = "127.0.0.1"
  flow.request.scheme = "http"
  flow.request.port = 4567

create own responses:

from mitmproxy import http

def request(flow: http.HTTPFlow) -> None:
  flow.response = http.Response.make(
    204,
    '{"foo":"bar"}',
    {"Content-Type": "application/json"}
  )

Don’t take seriously

What if we want to flip all the images in the server responses?

import io
from PIL import Image
from mitmproxy import http

def response(flow: http.HTTPFlow) -> None:
  if flow.response.headers.get("content-type", "").startswith("image"):
    s = io.BytesIO(flow.response.content)
    img = Image.open(s).rotate(180)
    s2 = io.BytesIO()
    img.save(s2, "png")
    flow.response.content = s2.getvalue()
    flow.response.headers["content-type"] = "image/png"

Conclusion

MITM attacks in real life can be a bit more complicated due to requirement of physical access to the device (hi https). The main thing that we should take from MITM is an analysis of ours, let me highlight, not another’s, but ours traffic. An interesting discovery may be the number of requests with our data flowing to other servers from our own devices and our favourite apps.

If you had any questions or clarifications after reading the note about the MITM or mitmproxy, I’ll be happy to answer them.

So, wish you happy Mondays (:

Screenshot testing in Espresso

2018-11-15T00:00:00+00:00

All below is worth to read, but if you request my humble opinion today on which exactly tool to use, I’d suggest going with paparazzi.

Introduction

At last time I talked about snapshot testing on iOS. Now I would like to turn to the second, no less popular mobile platform, — Android. Here we are very limited in the choice of tools for the snapshot verifying using native frameworks.

The popular tool from Facebook: Screenshot-Tests-For-Android will spend the biggest part of my note. Probably I should highlight at the beginning, that although this tool is the market leader and the trend for the forks, it grows rather slowly (v0.8.0 at the time of article publication) and it has several lacks, which I will discuss with a little comparative analysis with iOS.

We may have noticed the confusing difference between the type of the name of this testing on both platforms: Snapshot Testing on iOS and Screenshot Testing on Android. I propose to perceive it as a confrontation between a simulator and an emulator (:

Precondition

To pick up closer to Screenshot-Tests-For-Android project we need to make a few simple motions:

Install python dependencies:
```
 pip install mock
 pip install Pillow
```

Add permission for write in storage into AndroidManifest.xml:

 <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

Setup gradle plugin:

in root build.gradle

 buildscript {

    dependencies {

       classpath 'com.facebook.testing.screenshot:plugin:0.8.0'

    }

 }

in app build.gradle

 apply plugin: 'com.facebook.testing.screenshot'

Create custom Test Runner:

 // Set this test runner in your app build.gradle like this:
 // defaultConfig { testInstrumentationRunner "com.my.example.ScreenshotTestRunner" }

 class ScreenshotTestRunner: AndroidJUnitRunner() {
     override fun onCreate(arguments: Bundle) {
         ScreenshotRunner.onCreate(this, arguments)
         super.onCreate(arguments)
     }

     override fun finish(resultCode: Int, results: Bundle) {
         ScreenshotRunner.onDestroy()
         super.finish(resultCode, results)
     }
 }

Usage

In order to run screenshot testing along with simple Espresso tests, we will need to use gradle tasks:

· clean<App Variant>Screenshots - Clean last generated screenshot report
· pull<App Variant>Screenshots - Pull screenshots from your device
· record<App Variant>ScreenshotTest - Installs and runs screenshot tests, then records their output for later verification
· run<App Variant>ScreenshotTest - Installs and runs screenshot tests, then generates a repor
· verify<App Variant>ScreenshotTest - Installs and runs screenshot tests, then verifies their output against previously recorded screenshots

Available checks

Whole activity

@Test
fun test_Activity_and_View_Snapshot() {
    val activity = activityTestRule.launchActivity(null)
    Screenshot.snapActivity(activity).record()
}

Bloated view

@Test
fun test_View_Snapshot() {
    val targetContext = InstrumentationRegistry.getInstrumentation().targetContext
    val inflater = LayoutInflater.from(targetContext)
    val view = inflater.inflate(R.layout.search_bar, null, false)
    val tv = view.findViewById(R.id.search_box) as TextView
    tv.text = "This is a really long text and should overflow"
    ViewHelpers.setupView(view).setExactWidthDp(300).layout()
    Screenshot.snap(view).record()
}

Any element

@Test
fun test_View_Snapshot() {
    val activity = activityTestRule.launchActivity(null)
    val view = activityTestRule.activity.findViewById(R.id.fab)
    Screenshot.snap(view).setName("view").record()
}

Multiple checks

If we would like to snap many screenshots in one test we should set up a name of every screenshot:

@Test
fun test_Activity_and_View_Snapshot() {
    val activity = activityTestRule.launchActivity(null)
    val view = activityTestRule.activity.findViewById(R.id.fab)
    Screenshot.snapActivity(activity).setName("sample_activity_test").record()
    Screenshot.snap(view).setName("sample_view_test").record()
}

Multiple devices

In order to run screenshot tests on multiple devices, we will need to prescribe an extra property into app build.gradle:

screenshots {

    multipleDevices true

}

Our screenshots will be saved in the unique directory with a composite name, e.g.:

API_23_GP_XXHDPI_1080x1920_armeabi-v7a_ru-RU

Tips of record

The only way to collect screenshots is this gradle task:

record<App Variant>ScreenshotTest

I am sure we don’t want to spend so much time to run all our espresso tests to collect all screenshots. It is not only about laziness, but also about her (:

So, let’s divide our tests into a screenshot-test and not a screenshot-test via one «if» statement in app build.gradle:

android {
    // ...
    defaultConfig {
        if (project.gradle.startParameter.taskNames.toString().contains("record")) {
            Map<String, String> map = new HashMap<String, String>()
            map.put("package", "my.com.samplesnapshot.screenshots")
            setTestInstrumentationRunnerArguments map
        }
    // ...
    }
}

Reports

Here we came to the weakest part of Screenshot-Tests-For-Android. Yes, we can say that at the moment the report about failed tests is uninformative at all and does not show the visual differences with the actual snapshot and the reference one.

In pursuit of beautiful and understandable reports, we can come to one tool named Shot, which, inheriting Screenshot-Tests-For-Android, really «painted over» this problem. But finding awesome reports, we lose flexibility, because Shot can’t work with a lot of devices. Based on all of this, everyone can choose the extremes that will make his weekdays less painful.

Analysis

In this block I would like to make a small comparison between hero of the article and his elder brother on «antagonistic» platform — iOS-Snapshot-Test-Case:

Capability	Screenshot Tests For Android	iOS Snapshot Test Case
Birthday	8 Oct 2015	1 Oct 2013
Languages	Kotlin/Java	Swift/Objective-C
Multiple devices/simulators	+/-	+
File name options	+/-	+
Tolerance of Comparison	-	+
Objects of verification	Views/Activities	UIViews/CALayers
Test Runner	Custom	Custom
Reports	+/-	+/-

Conclusion

To sum it up, I would say that testing with screenshots on Android is pretty easy and useful. It is mighty speed up our automation testing and reduce sum of test cases.

If you had any questions or clarifications after reading the article about the screenshot testing, I’ll be happy to answer them. And you can find the completed project here:

sample_of_android_screenshot_testing_2048

So, good luck, have fun and see you later (:

Snapshot testing in XCTest

2018-10-24T00:00:00+01:00

All below is worth to read, but if you request my humble opinion today on which exactly tool to use, I’d suggest going with pointfreeco.

Introduction

«It’s not a bug, it’s a feature». Almost every member of the team has already learned this phrase. And yes, curiously enough, it can be said about some amount of defects. But not about badly displayed layout. This is especially important for mobile testing, given the time-cost of delivery to the store.

We have many similar tools for testing the layout in iOS applications (by the way, Android note is here). But the most part of them forked or took an idea from only one. So, in this note I would like to talk about that — iOSSnapshotTestCase.

This tool was originally created as an open source project under the leadership of Facebook. Over time, having got into the archive, the initiative passed to Uber and the project is still successfully developing. Most pleasantly, that iOSSnapshotTestCase is inherited from XCTestCase, which obviously means, that we don’t need to make any changes into our automation system at all.

Also I would like to notice we can use iOSSnapshotTestCase as in XCUITest as in XCTest. Awesome, let’s try.

Precondition

At the beginning we should:

add an additional pod in Podfile:

target "Tests" do
  use_frameworks!
  pod 'iOSSnapshotTestCase'
end

and set the environment variables in our test scheme:

Name	Value
FB_REFERENCE_IMAGE_DIR	$(SOURCE_ROOT)/$(PROJECT_NAME)Tests/Snapshots/ReferenceImages
IMAGE_DIFF_DIR	$(SOURCE_ROOT)/$(PROJECT_NAME)Tests/Snapshots/FailureDiffs

Usage

The functional of saving reference snapshots is regulated by the variable recordMode, which is located in the setUp():

recordMode = true  // saving reference snapshots
recordMode = false // verifying reference snapshots with actual

To make more than one check per test we should set «identifier» as an argument of FBSnapshotVerifyView and FBSnapshotVerifyLayer methods.

XCUITest implementation

import XCTest
import FBSnapshotTestCase

class TestExample: FBSnapshotTestCase {

    override func setUp() {
        super.setUp()
        recordMode = true
        XCUIApplication().launch()
    }

    func test_Fullscreen_and_Element_Snapshot() {
        let fullscreen = XCUIApplication().screenshot().image
        let element = XCUIApplication().images["image"].screenshot().image

        FBSnapshotVerifyView(UIImageView(image: fullscreen), identifier: "fullscreen")
        FBSnapshotVerifyView(UIImageView(image: element), identifier: "element")
    }

}

XCTest implementation

import XCTest
import FBSnapshotTestCase
@testable import Product_Module_Name

class TestExample: FBSnapshotTestCase {

    override func setUp() {
        super.setUp()
        recordMode = true
    }

    func test_View_and_Layer_and_Sublayer_Snapshots() {
        let view = SampleViewController().view!
        let sublayer = CALayer()
        sublayer.frame = CGRect(x: 33, y: 33, width: 50, height: 50)
        sublayer.backgroundColor = UIColor.blue.cgColor
        view.backgroundColor = UIColor.orange
        view.layer.backgroundColor = UIColor.red.cgColor
        view.layer.addSublayer(sublayer)

        FBSnapshotVerifyView(view, identifier: "view")
        FBSnapshotVerifyLayer(view.layer, identifier: "layer")
        FBSnapshotVerifyLayer((view.layer.sublayers?.last)!, identifier: "sublayer")
    }

}

File name options

These are capabilities that allow to add distinctive features to the names of the created snapshots, depending on the required filtering. Wow 😳. If explains it on fingers and a little bit easier:

Imagine that we have a mobile application opened on the screen with the text. We see difference between fonts on iOS 9 and iOS 12. We understand that these font differences are not a bug of the app, but when verifying snapshots, the test has been failing.

To avoid this problem, we can set the agnostic option, which will add the iOS version to the name of out test snapshot. Thus, we will need to create reference snapshots for each version of iOS, and during verification, the test will look for the snapshot it needs.

Currently out of the box we have 4 types of file name options:

FBSnapshotTestCaseFileNameIncludeOption.device

The file name should be agnostic on the device name, as returned by UIDevice.currentDevice.model.
FBSnapshotTestCaseFileNameIncludeOption.OS

The file name should be agnostic on the OS version, as returned by UIDevice.currentDevice.systemVersion.
FBSnapshotTestCaseFileNameIncludeOption.screenScale

The file name should be agnostic on the screen scale, as returned by UIScreen.main.scale.
FBSnapshotTestCaseFileNameIncludeOption.screenSize

The file name should be agnostic on the screen size of the current keyWindow, as returned by UIApplication.sharedApplication.keyWindow.bounds.size.

We can use the file name options in the same way as the recordMode, by assigning them in the setUp():

override func setUp() {
  super.setUp()
  recordMode = true
  fileNameOptions = [
      FBSnapshotTestCaseFileNameIncludeOption.OS,
      FBSnapshotTestCaseFileNameIncludeOption.screenScale
  ]
}

Tolerance

I mean, as far as we allow current snapshots to be different from the reference ones. It is calculated on a scale from 0 to 1, where tolerance = 0.05 is a difference in snapshots equal to 5%. By default: 0%. The tolerance is set as an argument of FBSnapshotVerifyView and FBSnapshotVerifyLayer methods. The library provide us with two realisations of this feature:

overallTolerance — how many pixels may not match
perPixelTolerance — how each pixel may not match

import XCTest
import FBSnapshotTestCase

class TestExample: FBSnapshotTestCase {

    func test_overallTolerance() {
        let image = XCUIApplication().screenshot().image
        FBSnapshotVerifyView(UIImageView(image: image), overallTolerance: 0.01)
    }

    func test_perPixelTolerance() {
        let image = XCUIApplication().screenshot().image
        FBSnapshotVerifyView(UIImageView(image: image), perPixelTolerance: 0.1)
    }

}

Tear off the «Status bar»

If we need to verify the snapshot of full screen via XCUITest, we may face the problem that we will have conflicts in the status bar (battery, time, etc.). It is possible to solve this problem, although in a slightly ridiculous way — by cutting off a few pixels that used for status bar, while taking a snapshot.

The main problem is that since «X» series of iPhone we have nonstandard size of status bar:

Device	Size of status bar (.pixels)
iPhone < X	22
iPhone X	44
iPhone XS	44
iPhone XR	48.6
iPhone XS Max	48.6

We can implement this crutch using UIImage extension:

import UIKit

extension UIImage {

    var removingStatusBar: UIImage? {
        guard let cgImage = cgImage else {
            return nil
        }

        let yOffset = 22 * scale // status bar height on standard devices (not iPhoneX)
        let rect = CGRect(
            x: 0,
            y: Int(yOffset),
            width: cgImage.width,
            height: cgImage.height - Int(yOffset)
        )

        if let croppedCGImage = cgImage.cropping(to: rect) {
            return UIImage(cgImage: croppedCGImage, scale: scale, orientation: imageOrientation)
        }

        return nil
    }
}

Thereafter, for the correct operation of this cutting on all iPhone models, we will need to learn how to determine it from our tests. To do it, we need to extend the UIDevice:

import XCTest

extension UIDevice {

    enum UIDeviceModelType : Equatable {

        case iPhoneX
        case iPhoneXR
        case iPhoneX_Max
        case other(model: String)

        static func type(from model: String) -> UIDeviceModelType {
            switch model {
            case "iPhone10,3", "iPhone10,6", "iPhone11,2":
                return .iPhoneX
            case "iPhone11,4":
                return .iPhoneX_Max
            case "iPhone11,8":
                return .iPhoneXR
            default:
                return .other(model: model)
            }
        }

        static func ==(lhs: UIDeviceModelType, rhs: UIDeviceModelType) -> Bool {
            switch (lhs, rhs) {
            case (.iPhoneX, .iPhoneX):
                return true
            case (.iPhoneX_Max, .iPhoneX_Max):
                return true
            case (.iPhoneXR, .iPhoneXR):
                return true
            case (.other(let modelOne), .other(let modelTwo)):
                return modelOne == modelTwo
            default:
                return false
            }
        }
    }

    var simulatorModel: String? {
        guard TARGET_OS_SIMULATOR != 0 else { return nil }
        return ProcessInfo.processInfo.environment["SIMULATOR_MODEL_IDENTIFIER"]
    }

    var hardwareModel: String {
        var systemInfo = utsname()
        uname(&systemInfo)
        let machineMirror = Mirror(reflecting: systemInfo.machine)
        let model = machineMirror.children.reduce("") { identifier, element in
            guard let value = element.value as? Int8, value != 0 else { return identifier }
            return identifier + String(UnicodeScalar(UInt8(value)))
        }

        return model
    }

    var modelType: UIDeviceModelType {
        let model = self.simulatorModel ?? self.hardwareModel
        return UIDeviceModelType.type(from: model)
    }

    var isIPhoneX: Bool {
        return modelType == .iPhoneX
    }

    var isIPhoneXR: Bool {
        return modelType == .iPhoneXR
    }

    var isIPhoneX_Max: Bool {
        return modelType == .iPhoneX_Max
    }

    var isXFamily: Bool {
        return modelType == .iPhoneX_Max
            || modelType == .iPhoneX
            || modelType == .iPhoneXR
    }
}

And then will use it like this:

import UIKit

extension UIImage {

    var removingStatusBar: UIImage? {
        guard let cgImage = cgImage else {
            return nil
        }

        var yOffset: CGFloat = 0
        if (!UIDevice.current.isXFamily) {
          yOffset = 22 * scale
        } else if (UIDevice.current.isIPhoneX) {
          yOffset = 44 * scale
        } else if (UIDevice.current.isIPhoneX_Max || UIDevice.current.isIPhoneXR) {
          yOffset = 48.6 * scale
        }
        let rect = CGRect(
            x: 0,
            y: Int(yOffset),
            width: cgImage.width,
            height: cgImage.height - Int(yOffset)
        )

        if let croppedCGImage = cgImage.cropping(to: rect) {
            return UIImage(cgImage: croppedCGImage, scale: scale, orientation: imageOrientation)
        }

        return nil
    }
}

Painting

Sometimes we may need to verify the snapshot of the full screen with the exception of a specific element.

Can anyone draw? And paint?) It is time to paint over, because we can solve this problem with the help of «colorings». I am not kidding 😉. For this we need:

to extend the UIImage once again by creating the* fill(element: XCUIElement)* method, which will paint over the chosen element on the snapshot:

 import UIKit
 import XCTest

 extension UIImage {

     func fill(element: XCUIElement) -> UIImage {
         UIGraphicsBeginImageContext(self.size)
         self.draw(at: CGPoint.zero)
         let context = UIGraphicsGetCurrentContext()!
         context.fill(
             CGRect(
                 x: Double(element.frame.minX),
                 y: Double(element.frame.minY),
                 width: Double(element.frame.size.width),
                 height: Double(element.frame.size.height)
             )
         )
         let myImage = UIGraphicsGetImageFromCurrentImageContext()
         UIGraphicsEndImageContext()
         return myImage!
     }

 }

to create a new method verifyScreen(without element: XCUIElement) near the previously created verification methods:

 func verifyScreen(without element: XCUIElement) {
   let paintedImage = XCUIApplication().screenshot().image.fill(element: element).removingStatusBar
   FBSnapshotVerifyView(UIImageView(image: paintedImage))
 }

 func verifyScreen() {
   let image = XCUIApplication().screenshot().image.removingStatusBar
   FBSnapshotVerifyView(UIImageView(image: image))
 }

 func verifyElement(_ element: XCUIElement) {
   let image = element.screenshot().image
   FBSnapshotVerifyView(UIImageView(image: image))
 }

Conclusion

To sum it up, I would say that testing with snapshots on mobiles is at least possible and at most mighty speed up our automation testing.

If you had any questions or clarifications after reading the article about the snapshot testing, I’ll be happy to answer them. And you can find the completed project here:

sample_of_ios_snapshot_testing_2048

So, good luck, have fun and see you later (:

Test multiple apps using bundle identifier in XCTest

2018-10-09T00:00:00+01:00

Introduction

The one who wrote XCTests/XCUITests at least once is perfectly familiar with the approach to writing tests in the development workspace, that is, literally side-by-side with the app’s source code.

This is convenient and doesn’t require censure, but should highlight, that this approach is not the only one. We received this superpower with Xcode 9.0 — creating a proxy for an application associated with the specified bundle identifier. And if we can run the tested app via bundle identifier, it means that:

we don’t depend on the app’s source code
we can run not only the tested app

That’s right, this feature gives us the ability to manage an unlimited number of apps on an iOS device at the same time (hello UIAutomator 🤖).

Launch tested app via bundleId

As I already said, this feature gives us the opportunity to run our tested app via identifier:

create xcodeproject with empty activity
add new Target — «iOS UI Testing Bundle»
create a variable in which we put an instance of our app with bundleId
run tests and enjoy, as if it has always been

let app = XCUIApplication(bundleIdentifier: "com.my.app")

override func setUp() {
    super.setUp()
    app.launch()
}

Launch other apps via bundleId

Yes, here I would like to throw in final exclamation that any app installed in the system is open to us. For this we only need to know its bundleId. Below is the pretty repo with a list of default apps and their identifiers:

joeblau/apple-bundle-identifiers

How to check that the expected app was opened?

In some cases, we may need to know which app is currently active. We can do this in such a simple, uncomplicated way:

func isOpen(app: XCUIApplication) -> Bool {
   return (app.windows.count > 0)
}

How to interact with system alerts?

With this feature, we can also contact with system pop-ups, because they are also the system app’s UI that called Springboard:

enum Alert: String {
    case left, right, ok
}

func alert(button: Alert) {
    let buttonIndex = (.right == button) ? 2 : 1
    let springboard = XCUIApplication(bundleIdentifier: "com.apple.springboard")
    let alertBtn = springboard.buttons.element(boundBy: buttonIndex)
    if alertBtn.exists { alertBtn.tap() }
}

How to find the app’s bundleId?

There are two quite simple ways to crank this venture:

If your app is from the App Store. Here we can make our life a little bit easier with jq:
```
 curl https://itunes.apple.com/lookup?id=310633997 | jq -r '.results[0].bundleId'
```
If your app is installed on the device. Here we can make our life a little bit easier with ideviceinstaller:
```
 ideviceinstaller -l
```

Conclusion

I think this superpower opens up many opportunities, but, as say, «with great power there must also come great responsibility».

Good luck and have fun. See you late (:

Android battery keeper

2018-09-25T00:00:00+01:00

Introduction

Own test stand with mobile devices is awesome. Especially, if there are continuously running tests. Might endlessly looking at fire, water and mobile auto-tests (but it is not exactly).

Unfortunately, everything has a shelf life and this we can’t escape. Someone chooses cloud solutions, trying to run down from this fact. I, in turn, will consider the opportunity of extending a lifetime of mobile devices from a solution with a stand. Maybe not for long, but still noticeable.

If iDevices on average show themselves from the good side in the severe conditions of the perpetual motion machine, then devices on Android quickly inflate their tummies like on the image above.

It is easy to guess that devices swell after permanent charging when live on a test stand. Whether there are plastic or metal cases, USB 2.0 or Type-C, a ventilator or air conditioner – doesn’t matter.

The first thing that comes to mind for saving hard-workers is to unplug them from charge. Yes, we might plug device to PC via Wi-Fi, but here we run into two problems:

Someday the device must be charged. Well, not without it.
Wi-Fi isn’t the most stable thing. UI auto-tests already have something to fall down without it.

Reflections

Developing our first thought, about unplugging devices — that would be great if it is possible to unplug the device, until it itself asked for recharging. Google offers us the opportunity to emulate the disconnection of the battery on the device out of the box, via adb:

adb shell dumpsys battery plug
adb shell dumpsys battery unplug
adb shell dumpsys battery

The main word here is «to emulate».

Good news

if we request the status of charging, it looks like not available
the charging icon isn’t lighting up on the device screen

Bad news

as such, unplugging isn’t occur
the device is still charging, despite everything

Solution

Actually, we can still implement this task, but for this we need Super User privileges. I won’t tell how to install SuperSu on Android, there are quite a lot of discussions and step-by-step tutorials about it in free access. The only thing I would add is that there is nothing terrible for testing here. This hack won’t affect on user cases at all. Android will be the same as at past, just on the steroids. So, let’s start hexing.

We will also perform basic manipulations via adb:

# unplug
adb shell "su -c 'echo 0 > /sys/class/power_supply/battery/charging_enabled'"

# plug
adb shell "su -c 'echo 1 > /sys/class/power_supply/battery/charging_enabled'"

The path to the file with the status of the battery can vary depending on the device model. I showed the most common value (hit about 95% on pure Android without shells).

Beautified solution

Now I would like to sculpt from it a small service that will support the (life or level? 🤔) of battery power on our devices, when it will necessary. Rumored that the optimal battery charge for the mobile devices is about 50%. The answer to the ultimate question of Life is somewhere near:

@max_battery_level = 42
@charging_file = '/sys/class/power_supply/battery/charging_enabled'.freeze

def devices
  `adb devices`.scan(/\n(.*)\t/).flatten
end

def unplug(udid:)
  charging = (`adb -s #{udid} shell "su -c 'cat #{@charging_file}'"`.to_i == 1)
  puts "🔋 Charging enabled: #{charging}"
  `adb -s #{udid} shell "su -c 'echo 0 > #{@charging_file}'"`
  puts "🔌 Unplug #{udid}" if charging
end

def plug(udid:)
  charging = (`adb -s #{udid} shell "su -c 'cat #{@charging_file}'"`.to_i == 1)
  puts "🔋 Charging enabled: #{charging}"
  `adb -s #{udid} shell "su -c 'echo 1 > #{@charging_file}'"`
  puts "🔌 Plug #{udid}" unless charging
end

def battery(udid:)
  battery_level = /level: (.*?)\n/.match(`adb -s #{udid} shell dumpsys battery`)
  raise("⚠️ Could not get battery level from: #{udid}") if battery_level[1].nil?
  puts "💡 Battery level: #{battery_level[1]}"
  battery_level[1].to_i
end

def battery_saver
  devices.each do |udid|
    next if udid.include?('emulator')
    puts "📱 #{udid}"
    battery_level = battery(udid: udid)
    battery_level > @max_battery_level ? unplug(udid: udid) : plug(udid: udid)
  end
end

battery_saver

Conclusion

The script has written in Ruby and claims a good mood. Putting it on our CI and running about once an hour, we can greatly reduce the load on the batteries of our test devices. Via it we request all the connected Android devices and get the current percentage of the charge. If it is less than 42 — we turn on the charge, if it is greater than 42 — we turn off the charge.

In order not to scare our auto-tests with a disabled screen, you can turn it on before starting them:

adb shell input keyevent KEYCODE_WAKEUP

I hope that it was interesting. So long, and thanks for all the fish. See you in the following notes (:

The easiest way to retry XCTests

2018-09-17T00:00:00+01:00

Introduction

It’s not a secret, that UI Tests are rather unstable thing. Pretty often, in practice, we have to come up with various ways to make them as stable as possible. One such way is to retry the failed tests.

In JUnit for such cases we have RetryTestHelper, in TestNG — IRetryAnalyzer, even in Espresso we’re able to create a custom rule for that purpose (e.g.: Barista). But in XCTest (and other XCTest extensions) we don’t have anything like that out of the box.

After a short googling, the first solution we can find — setup_fragile_tests_for_rescan — this is the Fastlane plugin. Fastlane is a gorgeous and elegant tool for automating everything around mobile apps (signing, building, testing, delivering, etc). This plugin is currently deprecated, but in its place came test_center.

I really like Fastlane, but this plugin was a bit buggy to me. I mean, if it’s working great for you, just go on using it. Otherwise - welcome to the party.

I, in turn, came up with an idea to create my own Fastlane plugin, that will just do its job without lots of features - retry tests if they fail.

Here we go

brew install fastlane
fastlane add_plugin try_scan

Usage

try_scan(
  workspace: "Example.xcworkspace",
  device: "iPhone 11 Pro Max",
  try_count: 3
)

Conclusion

That’s nice, isn’t it? But I should point out that retrying the tests, although a forced measure, is far from the best way to maintain stability and reliability of the tests, as your app becomes a target for heisenbugs.

Hope it’ll help, see ya (: