In the first part of the article, we established not only that UI tests can be used to write specifications for mobile apps, but that they can also be very expressive in doing so, allowing for accurate modelling of the behaviour of the system and leaving no room for overlooked specifications.

If you were though to use UI tests to write specifications, then you would also need to have a way to do so before the development even starts while still in the specifications phase. Let’s see how we can do this.

Before embracing the “UI tests as specifications” approach, the timeline of a feature release in our team, from the requirements phase to the actual release looked like that.

The product team outlined requirements, which business analysts refined through discussions with developers and stakeholders. They clarified designs, created documentation, and incorporated BDD scenarios. Development prioritised unit and snapshot testing, with UI tests addressed at the end, based on team availability.

UI tests were essentially a 2nd class citizen due to time constraints and maintenance challenges. With the goal of turning them to a 1st class citizen we shifted them to the far left, becoming part of the specifications phase and replacing the BDD scenarios.

Writing UI tests becomes now the responsibility of the feature’s tech lead, who should have the necessary Android and iOS skills to write them. This shift eliminates reliance on developer availability, as the UI tests are now written before the development begins. Specifications now become acceptance criteria and regression tests at the same time, making it easier to sync the implementations between Android and iOS.

Open the specs MR

Before any development starts — remember we are still in the specifications phase trying to write the specifications for our engineers — the feature’s tech lead opens the very 1st MR writing the specs of the feature as UI tests in either iOS or Android.

Apply code ownership

The specs MR contains only the the UI tests files themselves, all placed within the a folder that is under the code ownership of the feature’s tech lead.

And this is quite important because if ever the behaviour of the app was to deviate from the initial specs due to a code change, then the relevant UI tests would fail and we would have to update them. And these updates would need to be approved by the code owner, effectively the owner of the specs themselves, making him aware of any change in the specifications and mismatches between specification and implementation for free.

And with source control you even have a history of these changes to reason about if needed. The need to communicate to be in sync about any changes in specs vanishes into thin air 💨.

And then the feature’s tech lead starts writing the actual scenarios.

Define the preconditions (network stubs and feature flags)

We should start each scenario by leaving a note to the developer what should its preconditions be (the network stubs¹ of the API calls that we care about and any feature flag setup that you might have). In the scenario we described in part 1 of the article, we specifically care about the two login API calls and we require them to return successfully. All other calls are irrelevant (and are stubbed successfully by default by the way).

When writing specs you do not have to implement the stubs — the developer will. You just have to make it explicit for him in the comments section like above.

Likewise, in non-happy path scenarios, you should make it clear in the comments which API call fails and the relevant error code.

Write the scenario using the Page Object pattern

Then you can begin writing the scenario itself using the Page Object pattern which, as we have seen, is very close to the clarity of the natural language used in a BDD scenario.

Use empty stub methods

The methods that we define for each screen (like the LoginScreen in the above example) should be empty stub methods. And by empty I mean that you do not care if the tapLoginButton method is actually implemented. The developer will have to implement it when he starts working on this feature, trying to make the test pass. In a TDD fashion but with UI tests if I may say².

Do not leave your API calls unspecified (optional)

If you wanted to specify which API calls should be part of this scenario and in what order, you can add them by defining how the snapshot of the API call interaction should look like as we saw in part 1 of the article.

Address any question raised during MR code review

Writing specs that way we use a language the devs completely understand. In the tool they are most familiar with. When all scenarios are done, you can proceed with requesting feedback on the scenarios and the specs, just like when requesting feedback for a code review.

When all the discussions are resolved, the specs MR is merged and the development can start. Eventually, the developers will have to make all the tests pass, thus making sure specs are respected.

Copy the UI tests for Android as well

And in the native mobile development world where the duality between Android and iOS is omnipresent , syncing the Android implementation becomes as easy as copying the UI tests for Android and applying the same code ownership rules 🎉🎉.

It should be clear by now that writing specifications using UI tests before coding greatly reduces any communication flaws in the specifications phase, the first most important problems when writing specifications. The main benefits include:

• Leaves no ambiguity to the developers
• Specs that never go out of sync with the implementation
• Under source control and code ownership, we can keep track when a requirement is changed

No more “Oh I have to update the BDD scenarios” or “Is this specification scenario written here still valid? Who should I ask?” or “Does it behave the same in Android and iOS” moments. And then hope that actual implementation in both Android and iOS also follows the specification in question.

As a project lead, opt for specifications as acceptance criteria and regression tests at the same time, opt for peace of mind 🧘

P.S. If you wish to dig more into this exciting topic, you could register for the amazing Swift Craft conference (in Southern England between 21–24 of May), where I will be holding a 3 hour workshop on this matter on the 22nd of May.

References — Further reading

[1] Using network stubs and not the actual network gives us control of the responses and also eliminates the delay and flakiness introduced by the real network.

[2] People also refer to this as ATDD

Writing specs correctly is a hard thing to do, with insufficient/hidden specs and people communication flaws being the two most common problems¹. Moreover, people that have led mobile teams know by experience that it is an even harder problem in the native mobile world due to its duality nature, where engineers from both iOS/Android should understand and implement the exact same thing.

Can we remove some of the complexity involved by writing specs using UI tests and by shifting them to the far left at the specification phase before the development even starts? When can UI tests — or any tests in general — be considered as specifications and how can they reveal behaviours and specs not manifesting in the UI itself that remain hidden when they shouldn’t? Let’s try to find out.

Writing complete and correct specs is a hard thing to do. Why is that?

The lifecycle of a feature starts with the requirements. Written in natural language by the product team, they provide a description of the required behaviour and they usually live in a Product Requirements Document (PRD²).

The project tech lead picks up the requirements and transforms them to specifications, which are a more technical response to the requirements document. They are used to let the developers know what to build and the testers know what to test.

This process often fails, leading to time wasted in back and forth and frustatrion pilling up.

People communication flaws

Any process that involves people communication over multiple hops has an inherent communication complexity where people have to understand the exact same thing. A process I have seen failing too may times. To make things even worse, in mobile teams the last hop consists of two teams (iOS and Android), an extra complexity that manifests itself in the QA team quite often reporting implementation inconsistencies between the two platforms³.

Insufficient/hidden specs

Specs are often easy to be missed. Some of them will be discovered or decided only during the implementation phase. For instance, in mobile apps hidden states (loading/error states) are everywhere and often missed in designs. Or the designs themselves say nothing about what happens in the background in terms of API interaction with the backend, leaving room for implementation inconsistencies between Android and iOS to crawl in.

In the mobile world, UI tests are widespread. If we can demonstrate that they are expressive enough⁴ to effectively capture the various aspects of the mobile app’s behavior, then we could use them to write complete specifications before the implementation even starts, shifting them to the left to the specifications phase.

Acceptance criteria and regression tests at the same

Using UI tests to write specifications has the paramount benefit of them also acting as regression tests, ensuring that the implementation stays aligned with the specifications and leaving little room for divergence.

Under which conditions UI tests can be used as specifications? The 3 conditions

To qualify as specifications, UI tests (and all tests in general) must meet three criteria⁵:

1. They should not change after a refactor

2. They must be written in a clear, easily understandable format.

3. They should be expressive enough to uncover any hidden requirements.

To further explore if UI tests can meet the above conditions and thus be used as specifications, we will use a simple example taken from our application, where we had to migrate to a new backend, requiring when logging in to be authenticated in both old and new backend at the same time during the transition period.

In this scenario, the functional requirement can be expressed in natural language as follows:

When logging in, we should be authenticated at the same time to the old and the new API, allowing us to access endpoints in both systems.

And could potentially be further transformed to a specification in the following BDD scenario:

Scenario: User logs in

Given that an existing user launches the app

When he fills in his credentials in the login page and taps on the login button

Then he should login to both systems and be presented with the home page

How a UI test can meet the 3 conditions and act as a specification?

The above specification can surely be re-written as a UI test in the following way.

It is important to note that we should always choose to control the network using stubs (avoiding the flakiness of the real network is a must have for any robust UI test suite) and also any feature flag that might affect our scenarios.

Let’s see how each of the 3 conditions can actually be met:

✅ 1. They should not change after a refactor

It is quite obvious that a UI test changes if and only if requirements change. After all it is nothing more than a translation of the BDD scenario and thus does not depend on the underlying implementation nor does it change when refactoring it. So we can check condition #1 from our list.

✅ 2. They must be written in a clear, easily understandable format

Nevertheless, the problem with the way the above UI test is written compared with the BDD scenario, is that it is extremely more difficult for someone to understand and reason about it. A specification should be easily understood by everybody, non developer people as well.

Using the Page Object pattern (also known as the Robot pattern) we can refactor our UI test into a more declarative human readable format.

The UI test now achieves a level of clarity comparable to the BDD scenario, making it easily comprehensible. After all, as a specification it must be effectively communicated to others and be understandable when looking back at it sometime in the future. Thus we can check condition #2 as well.

✅ 3. They should be expressive enough to uncover any hidden requirements

It turns out that, when looked from a different perspective, UI tests really shine in expressing specifications that are missed or remain hidden in mobile apps.

Here are some examples of easy to miss requirements and how UI tests can help in revealing them:

Missed loading states

Well, one thing that is easily missed in the mobile world is defining exactly how the loading state looks like. No more guessing or asking around how loading should look like. We can explicitly define it when writing the spec inside the UI test. And also enforce it by the way when this is implemented!

Missed error states

The same applies for error states or different flows that are triggered by specific API error codes. You can simply control the network response by using stubs and create additional UI tests that define those specs as well. Like in this non-happy path scenario where the login call returns the noLongerSupported API error and the user is asked to force update the app.

Non-happy scenarios are notoriously easy to miss during implementation and extremely time consuming to test. Having a UI test as a specification (along with using network stub responses) is the only way to ensure that this non-happy behaviour will be implemented and will not regress in the future. I call that piece of mind for the whole team 🧘

It turns out that that there are specifications that will never show up in the UI itself. They will remain hidden in what happens in the background. Here are some examples.

Hidden interaction with the API

How our app interacts with the backend, which API calls are made and in what order is definitely a specification that needs to be agreed upon before implementation. And one that can never be exposed as part of a related BDD scenario.

Revealing this hidden spec is actually quite easy. We can record all the HTTP calls that are made during the test execution and use an inline snapshot that flushes and asserts all done up to that point in time.

In our backend migration example, the snapshot would need to assert that authentication is being made to both old and new backend.

Thus, when writing a UI test as a specification, one can also specify which specific endpoints are being called making it pretty clear to the developer what needs to be done in terms of API interaction to support this flow.

I think the above method is actually quite powerful for the following reasons:

• It enforces how the interaction with the backend is upon implementation, leaving no room for ambiguity or miscommunication (even for divergence between Android and iOS implementations)
• It asserts that nothing changes in the future. It catches unintended deletion or addition of API calls and stops duplicate calls crawling in our flows, which is something that can remain hidden for a long time if something critical is not broken.
• It serves as the perfect documentation of the interaction with the server. If at any point in time, somebody asks you what are the API calls involved in a specific flow, especially in complex flows like the login one in our app, you do not have to provide a guess by asking someone with context, or try to remember, or try to follow the code and find the answer, or even open a network proxy and see what is being recorded. And do it twice for Android and iOS… With this technique, you will quickly know the correct answer every time, with 100% certainty. And everybody in the team will know too 😎

Hidden analytics events / audit logs

Using the same method, you could specify and reveal any other business critical operations that happen in the background, like analytics and audit logs. Providing regression detection for those at the same time! P.S. This argument is so strong it never gets old 😈

Using the UI tests in the ways we presented above, it becomes obvious that the expressive power of the UI tests can go beyond the mere specification of user flows, revealing specifications that remain hidden and are difficult to be specified. We can finally check condition #3, the final condition from our list, establishing that UI tests can be used to write specifications for mobile apps.

There are other ways to expand the UI tests and reveal easy to miss or hidden specifications, but it should be clear by now that they can certainly add a lot of value to this dimension and thus become an efficient solution to the problem of missing specifications, the second most important problem when writing specifications.

In part 2 of the article we will explore how one can quickly write UI tests before any coding starts and how this methodology addresses communication flaws, the other important problem when writing specifications.

P.S. If you wish to dig more into this exciting topic, you could register for the amazing Swift Craft conference (in Southern England between 21–24 of May), where I will be holding a 3 hour workshop on this matter on the 22nd of May.

References — Further reading

[1] The NaPiRE Project (2016), Naming the Pain in Requirements Engineering: Contemporary Problems, Causes, and Effects in Practice

[2] This tweet from @GergelyOrosz introduced me to PRDs and why they are needed.

[3] I have often found myself triaging such issues and thinking if I should just ignore the lack of consistency between Android and iOS and accept that this might be an insolvable problem.

[4] Various methods are available for writing specifications, including natural language, input-output examples, behavior-driven development (BDD) scenarios, decision tables, state diagrams, flow charts and Harel statecharts. Each method is bounded by its expressive power with the more advanced ones allowing for more accurate modelling of the behaviour of the system, leaving no room for overlooked specifications. None of the above act as regression tests at the same time though. They remain disconnected from the actual implementation.

[5] I have to admit that I have come to these 3 conditions from hands-on experience. My impression is that treating tests as specifications is a subject that has remained under the radar of the tech community with lots of hidden potential. Maybe there is bibliography for this, maybe there isn’t but I would argue that at least these 3 conditions are a minimum subset of any condition set someone might come up with.

Shifting UI tests to the far left (part 1) was originally published in XM Global on Medium, where people are continuing the conversation by highlighting and responding to this story.

With part 1 focusing on describing how one should approach screenshot testing in mobile UI development, this final part illustrates how this can be done in practice. An iOS hands-on guide that I hope you will find useful.

Therefore, for a better understanding of the context, it is strongly recommended to read part 1 first before continuing.

P.S. Part 2 goes beyond UI testing and explores other interesting non-UI related applications. Backend people might find this interesting as well.

Before applying snapshot testing in our project, we need to define a device configuration matrix that lays out our strategy for testing against multiple device configurations (language, device size, theme, dynamic font size).

The device configuration matrix groups the configurations that one must test into two final configurations. A large one (common device size, light theme, and default font size) and a small one (small device size, dark theme, and XXXLarge font size). Let’s take a look at how this translates into code.

Device configuration matrix in practice

We will be using the core-testing¹ library, an iOS snapshot testing utility library that we created. It is built upon the excellent SnapshotTesting library provided by the PointFree team and extends it by providing structured snapshot configuration generation.

Testing screens

For testing how a UIViewContoller looks like when it covers the full screen, the library defines the following three public snapshot configuration generator functions.

SnapshotTestConfig.VC.large { config in ... }
SnapshotTestConfig.VC.small { config in ... }
SnapshotTestConfig.VC.all { config in ... }

where “all” generates both large and small configurations (the closure is going to be called twice). These functions are defined here.

The assertion happens inside the closure using the configuration that it is passed, as can be seen in the example below.

You can check the DocumentHistoryViewController snapshot tests example in core-testing found here.

Testing views

Also for testing a UIView in isolation, the library defines the aforementioned three public snapshot configuration generator functions.

SnapshotTestConfig.View.large { config in ... }
SnapshotTestConfig.View.small { config in ... }
SnapshotTestConfig.View.all { config in ... }

These generators are used for testing (full width, dynamic height) views (i.e. views that span the full width of the screen and their height is dynamically defined by its content), like a UITableViewCell. You can check the DocumentHistoryCell snapshot tests example in core-testing found here.

The library also provides the following two extra configuration generator functions for dealing with view specific scenarios that are not covered by the (full-width, dynamic height) configurations.

SnapshotTestConfig.View.fixed(fixedSize) { config in ... }
SnapshotTestConfig.View.free { config in ... }

The fixed configuration generator is used for testing (fixed width, dynamic height), (dynamic width, fixed height), or (fixed width, fixed height) views. Similar to the “all” generator, two configurations as defined in the configuration matrix (the light theme — default font size, and the dark theme — XXXLarge font size) are generated. The only difference is that the size is fixed in one or both dimensions.

The six subviews of the following example should be tested using a (fixed width, dynamic height) configuration generator. We need to fix the width to a specific size (1/3 of the screen width and not the full width) and allow it to grow in height. The XXXLarge font size configuration will reveal if the content fits inside the view nicely.

The free configuration generator is used for (dynamic width, dynamic height) views, that take as much space as needed to accommodate for their content, like a custom UIButton. Similar to the “all” generator, two configurations as defined in the configuration matrix (the light theme — default font size, and the dark theme — XXXLarge font size) are generated. The only difference is that the size always adjusts to the growing content. You can check the SupportButton snapshot tests example in core-testing found here.

Testing how the view reacts to changes in the environment

While writing the snapshot test, if you instantiate the view outside of the configuration generator closure, the same instance of the view is tested in both configurations. Thus, as a nice side effect, when the configuration is changed, you also get to test how your view adapts to interface style (dark/light theme) and dynamic text size changes in the runtime.

Applying screenshot testing in practice

Let’s assume that we would like to use screenshot testing to develop and test the account info screen in our Forex trading app. The account info screen is the screen where the user can see his balance, profit and loss, and other account-related information.

This is how it should look like (for a demo account) according to the designs.

Let’s see how we apply step by step everything we learned in part 1 using the snapshot configurations generated by the core-testing library¹.

Testing all device configurations

We must always start by testing how the view looks like in all device configurations. We will be using the core-testing library to help us set up the screenshot tests. Here is how the test case looks like.

1. Create a view model object (which is the dependency of the view that we would like to test)
2. Configure the view with the view model (in this case it is a view controller since we want to test the whole screen)
3. Generate the snapshot configurations (in this case we want to generate all device configurations for a view controller as defined in our configuration matrix)
4. Snapshot the view rendered with the provided configuration

If it is the first time the test is run (or the test is run explicitly in record mode), it will save the snapshot as a reference on the disk. Otherwise, the test will assert that the reference snapshot and the new one match.

The following two reference screenshots are generated.

In this way, we can check how the view looks like in both large and small configuration and as you can see, there is a problem with small width devices (like iPhone SE) when users have selected larger dynamic font size.

Important: To avoid flakiness, please remember to always run all snapshot tests of your project against the same iPhone simulator (e.g. iPhone 11).

Testing domain cases

After fixing how the view looks like in the small configuration, we now should check how it looks like in all other domain cases. That means we should at least write tests for

• a demo user
• a real user that has been validated
• a real user that has not been validated

And since device configurations have been already checked, we can perhaps now test the domain cases against only the large configuration.

And this is how the view looks like for a real user (validated or not).

Discovering missing domain cases

If you want to make sure that you have covered all possible domain cases, then you can check which parts of the view UI code are not covered by the code coverage 😀. Needless to say that having clean views that only deal with the UI and have no business logic really helps here.

Take a look at the example below. Using code coverage report integration with Xcode, we can quickly spot that we are probably missing a case where an extra label is presented as a warning if the margin drops below 50%.

This seems quite important and as you can imagine it should be covered under a separate screenshot test.

Having covered all device configurations and domain UI cases, the view has been fully tested and more importantly future-proofed against UI regression errors.

This hands-on guide concludes the 3 part posts series and everything you need to know about snapshot testing. You can follow me on Twitter and LinkedIn for more.

References — Further reading

[1] Core testing library. This is a fork of the original we use in XM, as I wanted to adjust it to the content of this posts series. Feel free to play with both and open a PR for feedback.

Snapshot Testing. Testing the UI and Beyond (Part 3) was originally published in XM Global on Medium, where people are continuing the conversation by highlighting and responding to this story.

While part 1 is focused on mobile UI development, this part goes beyond UI testing and explores other interesting non-UI related applications. For understanding what is snapshot testing and how it works, take a look at the intro of part 1. Part 3 is an iOS hands-on guide that illustrates how UI testing can be done in practice.

Beyond Testing the UI

Snapshot testing is by no means limited to testing the UI. As we will see, it has lots of other interesting use cases. By listing some examples in this part of the posts series, I will try to highlight the versatile nature of snapshot testing as a rather effortless testing tool that mainly fits well under the integration testing spectrum of the software verification methodologies. And hopefully, convince you that it can become a significant part of your testing tools.

Testing complex data structures

Testing complex data structures that have lots of properties usually requires writing lots of unit tests. This of course requires time and increases maintainability.

But if you can produce a textual representation of the complex data structure (like an HTTP request as the example shown below), you could use this as a snapshot and snapshot test it, and with a single assertion test all properties at once (the method, the URL, the headers, the body, etc).

Moreover, if the complex data structure can be easily converted to JSON (like an API response), then basically for free a JSON snapshot would also test all properties at once, avoiding even to come up with a textual representation for it.

Testing complex data formats

In a similar fashion, if you write libraries or programs that produce an output with a complex data format, snapshot testing can be proven really helpful.

Imagine that your program generates PDF files and you would like to test that. In the unit test world, you would have to write multiple assertions as the PDF is a very complex data format. Instead, you could just produce an image of the generated PDF and use that as a snapshot applying again a single assertion, while also revealing how the PDF looks like.

You can check how this Swift library for generating PDF files is actually using snapshot tests for verifying its correctness, and how people even build their own custom snapshot process for testing that their generated PDFs do not change.

The same is true for programs that produce for example XML, HTML, or even code, where in the latter case you could easily reason about the generated code by just looking at the snapshots (see here) or even use the compiler to further validate its correctness (see here). Pretty amazing stuff to be honest!

Testing log and analytics events

If you need to make sure that critical paths of the code (e.g. the login flow) are always covered by logs, you would of course need to write tests for that.

Testing with unit tests would mean asserting every log event in the flow along with its properties (valuable debug information that needs to be logged). Instead, you could avoid the hassle and just write one single snapshot test that prints all log messages, in the order they should appear, along with their respective properties.

And you also get to expose them in a very human-readable way.

Along the same lines, people are actually snapshot testing warnings and errors that their CLI tools produce (see here).

Likewise, you can snapshot test the sequence of the analytics events (along with their properties) in a business-sensitive funnel and make sure the funnel does not break⁷.

A way to explore your domain

It should be pretty evident by now that readability is an intrinsic characteristic of snapshot testing. It is also its greatest strength as its output is by far the easiest to reason about.

This is why snapshot testing is an ideal way to quickly expose and explore your domain that could otherwise remain hidden. Just take a look at how the output of a snapshot test could expose all API error logs related to a specific API endpoint in a very human-readable way.

And since it is quite often that you only discover that some extra debug information might be missing from the error logs only when already in production³, you also get to see how your logs look like before going to production and reason about them.

Also, take a look at how the output of your custom date formatter changes when the language changes. Snapshot test makes setting up something like that almost effortless.

Without the convenience of setting up a snapshot test for this, probably nobody would go the extra mile to check how dates are represented for different locales (something that users will see anyway). And that would remain hidden.

And as a nice side effect, snapshot tests’ output also acts as documentation for the developer and perhaps QA or business people.

A change detector

Snapshot tests are essentially a change detector. And as such, they are ideal for writing characterization tests⁴ (also known as Golden Master Testing) when refactoring legacy code.

When refactoring the legacy code, you want to lock the behavior to a golden master version and you are only interested in whether the output has changed and not about whether it is correct.

Thus, snapshot tests, as a low effort method that tests multiple properties at once using a single assertion, are ideal in this case. You can check here how Code Climate uses snapshot tests for writing Golden Master Testing.

They are also ideal when we want to track if something changed due to a change in an external dependency that we do not control, like an external system, or an external API or even external libraries that we would like to update from time to time.

Imagine that your program output depends on an external parser library that is currently at version 1. And we want to make sure the output remains unchanged when we update the parsing library to version 2. We can use snapshot testing to quickly capture the output when the library is at version 1, update the library and check if the output is the same and the test passes.

Testing accessibility

One of the most mind-blowing applications of snapshot testing is accessibility testing.

The AccessibilitySnapshot iOS library by CashApp produces an image snapshot of your view with all of the accessibility elements highlighted using different colors, along with an ordered list of the descriptions that VoiceOver will read for each of the elements⁶.

And if the accessibility elements of this view ever change (for example the grouping of the payment-related middle row elements breaks — see next image), the test will fail.

It is pretty clear in this case as well that snapshot tests not only protect you against regression errors but also manage to expose the domain in a very human-readable way. I cannot think of a better and faster way to reason about which elements of the screen are accessible and in what order other than a simple image. Once again 🤯 Also check this one

Potential pitfalls

Of course, there also are some pitfalls about snapshot tests that we should all be aware of.

1. Effortless to update but easier to accept new snapshots with bugs. This is why snapshots need proper and thorough code review.
2. Since the expected output is missing (hidden inside the snapshot artifact), the context of what you test is lost. This is why we need to write really solid test descriptions.
3. They might lead to fragile tests as they test multiple properties at once and a minor change in any of them (even ones that might not be of interest) will break the test. This is why we need to keep them small.
4. They increase the git repository size, especially when doing screenshot testing. Maybe git LFS could be a solution?

Tools

For iOS projects, one should check the excellent SnapshotTesting library provided by the amazing PointFree team that handles both text and iOS image snapshots.

Every iOS developer should also check the variety of snapshot strategies that this library supports. You will find strategies for things like CGPath, UIBezierPath, WKWebView, URLRequest and so much more. Really inspiring work by the PointFree 🥇.

For Kotlin projects, the options are the KotlinSnapshot by Karumi when it comes to text snapshots and the Shot library by Karumi or the most recent Paparazzi by CashApp when it comes to Android image snapshots.

For the Web, the leader and pioneer in snapshot testing is the Jest library.

You can follow me on Twitter and LinkedIn.

References — Further reading

[1] https://github.com/pointfreeco/swift-snapshot-testing

[2] https://kentcdodds.com/blog/effective-snapshot-testing

[3] Has happened to me multiple times and had to release a new version just to add some extra debug info that was missing.

[4] https://michaelfeathers.silvrback.com/characterization-testing

[5] For the sake of completeness, it is worth mentioning that some libraries also support an inline assertion mode, which avoids saving the reference snapshot on the disk and directly places it inline inside the test when the test is run in record mode. See here

[6] https://cashapp.github.io/2020-05-20/making-ios-accessibility-testing-easy

[7] https://dropbox.tech/mobile/how-we-ensure-credible-analytics-on-dropbox-mobile-apps

Snapshot Testing. Testing the UI and Beyond (Part 2) was originally published in XM Global on Medium, where people are continuing the conversation by highlighting and responding to this story.

Part 1 is focused on mobile UI development while part 2 goes beyond UI testing and explores other rather interesting non-UI related applications. Part 3 is an iOS hands-on guide that illustrates how this can be done in practice.

What is snapshot testing?

Snapshot testing is just a form of unit testing. In unit tests, you have a function, and given some input, you explicitly assert against the expected output.

The equivalent snapshot test would look like the following and as you can see the main difference is that there is no explicit expected output.

But then, how does this work? Well, you can run a snapshot test in two modes.

• When running the test in record mode, a snapshot of the output is produced and saved as an artifact on the disk (just a simple text file).
• When running the test in verify mode a snapshot of the output is produced, it then looks on the disk for a file of the previously saved reference snapshot, it compares the two and if they differ it fails with a nice error message.

When the test fails, either the change is unexpected, or the reference snapshot needs to be updated to the new version. Finally, the snapshot artifact should be committed alongside code changes and reviewed as part of your code review process¹.

Classic assertion-based tests are perfect for testing clearly defined behavior that is expected to remain relatively stable. Snapshot tests are great for testing less clearly defined behavior that may change often. UI components often change in small and trivial ways. The copy is changed, whitespace is added, a border color is modified².

Snapshot testing is by no means a replacement of unit testing, but rather an additional test mechanism³.

Testing the UI

Ιn the mobile community the most popular form of snapshot testing is screenshot testing.

Screenshot testing

Screenshot testing is a visual regression testing method where the snapshot that you produce is an image of a part of the UI (could be a whole screen or just a single view).

Comparing screenshots

When you run the test in record mode, a reference image is saved on the disk.

Then every time you run it in verify mode, a newly generated image is produced and compared to the previously saved reference image.

If they differ, then the test fails and the failure diff would look like something like that.

It is the same concept as before when snapshotting the output of a function, but instead of text as the produced artifact you now have an image, and instead of applying text comparison you now apply pixel by pixel comparison. If you can serialize your output into a format, like text or image, that can be saved to disk and you have a way to compare and produce a diff, then you can also apply snapshot testing.

Benefits

By adding screenshot testing into our mobile project, you have already gained quite a lot.

• Trivial to write, almost effortless to update
• A fast way to verify what users see and also check how your view looks while developing (No need to fire up the simulator and navigate to a specific screen just to see how the view you are developing looks like. You just take a look at the produced image snapshot)
• They provide far more coverage than a unit test normally allows (test multiple properties at once)
• Plug them in the CI and detect UI regression errors

Device configurations

These are all nice benefits but you should not stop here. To maximize what screenshot testing has to offer, you have to make sure that you test against all possible device configurations.

Language

You can quickly test how your view looks like in other languages and produce a screenshot for each language.

Or instead of supporting multiple locales and multiple screenshots for them, you could just use one single pseudo-locale⁴ to simulate strings with increased length and see how larger texts stress your UI.

Device size

You can just as easily test how your view looks like on other devices and check, for example, if something does not fit in smaller devices.

Theme

If your app supports a dark theme, no problem... This can be tested as well.

Dynamic font size

Finally, you can also see how your view supports dynamic font size.

Device configuration matrix

If you were to test each of the above configurations in a different snapshot test, you would inevitably end up with lots of generated reference screenshots. Instead, you can group different configurations and reduce the number of configurations that need to be tested. A configuration matrix helps you define your grouping strategy.

In this example, we grouped everything in two configurations.

• The large configuration tries to be as close as possible to what the majority of the users will see (using the most common device size, theme, and font size) and possibly also matches the designs⁵.
• The small configuration tries to check if everything still looks ok in small devices, using a dark theme and an XXXLarge font size.

Exploring the domain

More importantly, screenshot testing also allows you to quickly see how the same view looks like in all different cases of your domain.

Let’s take a look at the view we have been using as an example so far. Its business purpose is to inform the user about when the market is open during the day.

Just by mocking your data, you can quickly have a look at what the view looks like when the market is closed

or when there are more than one open intervals.

With screenshot tests requiring almost no effort to set up, exploring all the different UI domain cases has never been quicker and easier.

The biggest benefits

Testing all device and domain configurations

When you combine all the above, you will end up testing multiple configurations (localisations — theme — size — dynamic font size) all at once and be done with them early in the development phase.

In that way, you would have quickly tested cases that otherwise you would have never bothered to manually recreate and test. And most importantly if the CI has your back, you also avoid UI regression errors for all cases.

The perfect pull request

As a very nice side-effect, screenshot testing will change your workflow when developing the UI and how the PR would look like in that case.

• The view can and should be developed independently. You do not need to wait for the container of the view to be implemented first.
• Thus the PR becomes quite concise, as it only contains the code for the view along with screenshots of how it will like in all possible UI cases.
• More importantly, screenshots add a missing visual dimension to the code review, thus reducing cognitive effort for the reviewer. The code reviewer no longer has to try to imagine how the UI will look like, he can actually see it.

Other interesting use cases

In the scope of the Composable Architecture, the PointFree team presented in episode 86 their solution on e2e functional tests using SwiftUI and snapshot testing. It is absolutely mind-blowing 🤯 and a must-watch.

This talk (Testing and Declarative UI’s) by Nataliya Patsovska is about combining Xcode Previews and snapshot tests. Pretty interesting as well.

People have been using screenshot testing to build and test their app’s UI design system (see Playbook for iOS and Storybook for Web)

Testing animations is also possible. We have been using it to test Lottie animations by taking snapshots of the animated view. Moreover, Stagehand, a composable type-safe animation library from Cashapp, uses snapshot testing that can also generate animated PNG files to test its functionality. Similar concepts are described in this talk.

Finally, some people use as a snapshot the difference produced between two snapshots. In this way they can isolate what changed before and after you interacted with it and test just that. Check it out here.

Tools

For iOS projects, one should check the excellent SnapshotTesting library provided by the amazing PointFree team.

Every iOS developer should also check the variety of snapshot strategies that this library supports. You will find strategies for things like CGPath, UIBezierPath, WKWebView, URLRequest and so much more. Really inspiring work by the PointFree 🥇.

For the sake of completeness, I would also like to mention the iOSSnapshotTestCase library, which is the very first library available for iOS.

For Android projects, the options are the Shot library by Karumi or the most recent Paparazzi by CashApp.

You can also check out part 3 which is an iOS hands-on guide that illustrates how all the above are translated into code.

To go beyond UI testing and find out lots of examples of how snapshot testing could be used in a non-UI context, you can continue reading part 2.

You can follow me on Twitter and LinkedIn.

References — Further reading

[1] Jest Snapshot Testing

[2] https://benmccormick.org/2016/09/19/testing-with-jest-snapshots-first-impressions/

[3] https://www.thoughtworks.com/radar/techniques/snapshot-testing-only

[4] https://netflixtechblog.com/pseudo-localization-netflix-12fff76fbcbe

[5] As a next step, you could even produce a side-by-side comparison of the generated screenshots against the designs (exported directly from e.g. Figma). And using this to automatically augment the context of a pull requests for the code reviewer would be very nice indeed.

[6] https://www.stephencelis.com/2017/09/snapshot-testing-in-swift

[7] https://www.vadimbulavin.com/snapshot-testing-swiftui-views/

[8] https://osinski.dev/posts/snapshot-testing-self-sizing-table-view-cells/

[9] https://troz.net/post/2020/swiftui_snapshots/

Special thanks to Vagelis Koutkias for introducing screenshot testing to the team a while ago (lots of the screenshots are actually part of that very first PR), and mentioning [5] while reviewing.

Special thanks to Nikos Linakis for letting me use screenshots and terms from his initial article on Screenshot testing and always coming up with the best terminology (“the perfect pull request” is all his).

Special thanks to Natalia Chalkidou and Stelios Poulakakis for their ideas about how one device configuration should match the designs.

Snapshot Testing. Testing the UI and Beyond (Part 1) was originally published in XM Global on Medium, where people are continuing the conversation by highlighting and responding to this story.

Delivery on time, requirement analysis, roadmap planning, and people management are some of the things that a product owner or a manager has on his mind. However, the key to success for all the above is, without any doubt, being able to prioritize. This article presents a modular feature prioritization framework and acts as a self-fulfilling prophecy on my claim to treat priority as a priority.

Thinking strategically and roadmap planning for a product owner/manager will inevitably, at some point, require prioritizing features. Prioritizing tasks also appears as a necessity in technical teams whenever, for example, they need to plan technical debt tasks for the next sprints. This work is an endeavor of formalizing a process for doing so, and I hope it provides a consistent, standardized, and straightforward approach in the form of a framework. Hence the term Feature Prioritisation Framework.

As an engineer, and since I could not waste too much time on it, modularity and composability came as an essential requirement for it to be simple and extensible for future enhancements.

Value vs. Effort

We are going to apply the Value vs. Effort model¹, one of the most popular prioritization frameworks. It is essentially a cost-value analysis model that allows you to evaluate each feature based on how much business value it will bring and how difficult or complex it will be to implement it.

We assign initial (Value, Effort) value pairs for each of the features, and then we plot them on a scatterplot, as seen in the image below.

The scatterplot allows for a quick comparative overview of all the features’ priorities. We even divide the chart area into four regions, each one forming a bucket of features that further help with the prioritization.

High Effort, High Value: long-term strategy

Low Effort, High Value: quick wins

Low Effort, Low Value: pet projects

High Effort, Low Value: time sinks

Assigning initial weights

To start putting features to the scatterplot, we need a way to calculate the initial values for Effort and Business Value².

Choosing a scale

The absolute values we assign to a feature do not really matter. For example, it makes no difference if we use values from the scale of [0–100] or [0–200] or any other. We choose a range and stick to it since the only thing that matters is the relative difference between the values. For that purpose, it is more suitable to think of these values as weights.

For example, if feature #1 (𝔽1) has double the Effort compared to feature #2 (𝔽2), we could assign weights to them like the following: 𝔽1=100 and 𝔽2=50, or 𝔽1=200 and 𝔽2=100, etc.

Deciding on a methodology

The methodology we follow to derive those initial weights is arbitrary but must be consistent. Every team can come up with a process that suits its needs and follow it. There is plenty of room here for devising smart and consensus-aware methodologies.

Here is an example of a consensus-aware decision strategy about Business Value weights³. Let’s say that a team of product stakeholders has to prioritize six features. Each stakeholder is given a bag of 6 weight units, where each weight is a number of the Fibonacci sequence. In our example, each stakeholder’s bag will have the following weights {1, 2, 3, 5, 8, 13}. In secret, he assigns a weight to each feature, and when everyone is done, the total weight for each feature is calculated. Finally, we normalize these weights to our scale of choice (e.g., [0–100])⁴, making sure we preserve the relative differences between them.

The engineering team can, for example, follow a similar strategy when choosing initial Effort weights. Τhe weights themselves could be an estimation of the person-hours needed for task completion.

Once we decide on the methodology for calculating initial values for both Effort and Business Value, we can start adding features on the scatterplot.

Considering multiple factors

When calculating the Effort or the Business Value of a feature, it is often the case that the initial estimation of weights is not enough. A product owner or a team might also, and probably should, want to consider multiple factors.

For example, competition analysis (what features does the competition have or is missing), feedback analysis (e.g., comments on the App Store, feedback emails, questionnaires), user testing analysis (e.g., through interactive prototypes), etc. should affect the Business Value and should be taken into account.

Every factor is a module that transforms the weights

The proposed framework tries to do that in a simple, modular, and composable way. Every factor that we think should affect the weight of a feature is a stand-alone module that increases or decreases the existing weight. How the weight is changed, it depends on the module itself and the data we have in our possession.

For example, if we keep track of the user comments on the App Store asking for missing features, we could come up with a “user feedback module” that would increase the Business Value weight of all the features users are requesting.

It is entirely up to us to identify modules like the “user feedback module” that affect either the Business Value or the Effort, define how they change them in the form of a transformation function, and apply them. And since the transformation functions are of the form f: Value → Value (endomorphism⁵), they compose infinitely. That means we can use any number of modules we want and chain them one after another in a plug-n-play fashion.

The power of compositionality is that it allows us to modify at any time the chain of modules by adding any other module we would like to affect e.g the Business Value or by removing an existing one. We could even apply normalization² after each module application to keep the weights within the selected scale.

Let’s have a look at two examples of how modules could look like, one for Business Value and one for Effort.

An example module for Business Value

Competition analysis module

Let’s assume that we want to factor in the competition status when calculating the Business Value weights. We start by identifying the competitors and writing down all the features that our application is missing, but the competitors might have in a competition analysis matrix like below.

We note the presence of the feature in a competitor with an (X) and the absence with an (-). Features like 𝔽3 that do not exist in our competitors but are something that we might be interested in implementing ourselves, we call them competitive advantages. Features like 𝔽2 that appear in all competitors except our application, we call them competitive disadvantages.

For example, we could choose to promote competitive advantages by 20% and competitive disadvantages by 10% (based on some reasoning that we should catch up with the competition), while applying no change to all other features.

Applying the “competition analysis module” to our initial values will shift the Business Value weights in the scatterplot along the y-axis.

An example module for Effort

Implementation complexity module

For this example, we will assume that we would like to factor in the implementation complexity of a feature when calculating the Effort. First, we would have to agree on how to model complexity. We could divide the tasks into high, medium, low, and none complexity and increase the weights accordingly.

Again, it is essential to state that how we model complexity and how it transforms the Effort weights is entirely up to us. There is no magic number here; we only have to agree on something that suits our data and use that consistently. It is the same process as when defining a module for Business Value, but now the features on the scatterplot are shifted along the x-axis.

Below you can see an examples on how the features would be further affected when applying the “implementation complexity module” (we assume none complexity for 𝔽1 and 𝔽3).

Conclusion

The Effort-Value model makes prioritization of new features very clear and understandable by visualizing them on a scatterplot. The values that we feed the model, though, could be more than just once-off initial weights that we have chosen. What if there are multiple factors to consider when estimating the Business Value or the Effor? The current proposal extends the basic model to account for any other factor in a simple and plug-n-play fashion.

In this proposed framework, every factor is an independent module that can be plugged in and out whenever we want and can compose well with the other modules. The composable module is the core of this framework as it gives flexibility and increases our modelling capabilities.

This proposal is not about providing a concrete prioritization framework. It proposes something more powerful, a way to build a prioritization framework that can be extended and formed to meet the specific needs of each team. Thus, although I tried to present some module examples, every team should try to come up with their own based on intelligence and data they have in their possession. I cannot wait to see what you can come up with.

[1] https://www.mindtheproduct.com/enter-matrix-lean-prioritisation/ https://www.productplan.com/glossary/value-vs-complexity/

[2] Usually, choosing values for the Business Value is done by the product team, and selecting values for the Effort is done by the project manager and the engineering team.

[3] https://en.wikipedia.org/wiki/Planning_poker

[4] Every time we want to transform our values and keep them within an initially selected scale (e.g. [0–100]), we should normalize. We take the maximum weight, divide it with the maximum scale value, and compute a normalization factor. We then need to multiply all weights with this factor.

[5] In mathematics, an endomorphism is a morphism from a mathematical object to itself. https://en.wikipedia.org/wiki/Endomorphism. The input and output are of the same type, and in our case, it is a function that takes as input a value and produces a value again. That enables us to compose them easily by chaining one after another.

Priority first was originally published in XM Global on Medium, where people are continuing the conversation by highlighting and responding to this story.