Quality Assurance is composed of a complex set of steps, which could be divided into four main parts: Analyzing, Planning, Testing, and Reporting. I would like to highlight “Planning”, as this step paves the path for the whole testing process. In order to plan a test cycle, one of the most time-saving and guiding instruments is a test case. Well formulated test cases have a great impact on the testing phase and writing them is as important as testing itself. Test cases are usually based on PRDs, design specs, user experience, and flow charts. Each feature could be separated into positive and negative flows to cover as many edge cases as possible.

As the application grows and the list of product functionalities expands, the test case tree expands as well. Eventually, it becomes very complex. This is a milestone that should be achieved, but, at the same time, it comes with a pretty big downside: along with the test case tree, the release cycle expands and the time for regression testing significantly increases, which leads to a delay in getting testing feedback and shortens the time frame for potential issue resolution.

So how do you maintain a high quality of testing in constrained time, if automation is still under development? How do you minimize the release cycle and simultaneously not miss out on anything important?

1. Reduce the number of test cases in each testing suite by omitting low-risk scenarios

It’s good practice to assign the “type” to each test case. The most commonly used types are Sanity, Smoke, Functional and Regression. This approach helps to group the test cases into test suites and execute them based on priority. The testing process that I find most efficient is executed in the following order:

• Smoke & Sanity test suite (includes the most high-level scenarios and high-risk tests)
• Functional & Regression test suite (includes the main functionality of the application and regressions)
• Sanity test suite (includes only high-level scenarios, as this is the final check before the release)

Test cycle time can be shortened by reducing the number of test cases in each of the testing suites. It can be done by adding one more type called “Monthly”, the main purpose of which is to bundle all the test cases that won’t block the release if their status is “Failed” and execute them once a month separately from the release Cycle. Having fewer test cases in each test suite should significantly speed up the testing process while ensuring quality.

The test case tree needs to be closely monitored for changes in the product requirements and their corresponding impact on the functionality of the application. Adjustments should be done regularly in order to keep the process up-to-date.

2. Determine the combination of test devices and environments based on priority

The important thing to keep in mind is the variety of devices and environments on the market. For example, on the web, customers access the product in any number of browsers and browser versions. Also, there is a large variety of mobile devices, platforms and operating system combinations to consider. Comprehensive coverage means that the release cycle would be endless and inefficient.

Tools like Mixpanel and Google Analytics are used by QA Engineers to analyze the summary of devices customers use the most and set the priorities on what to test first based on customers’ impact.

If testing with a lot of different combinations is critical and necessary, the other useful approach would be to complete the Smoke & Sanity test suite by testing on the wider variety of devices or browsers and then move to the Functional & Regression test suite by testing only on the primary cohort. It should catch the device-specific or environment-specific issues in the early stage of the release cycle.

3. Pre-release testing process: Minimize the QA on RC build

Pre-release cycle testing helps to discover issues early in the process. The goal is to test new features along with the high-risk areas and provide the detailed feedback accordingly. Also, it is important to keep the queue of tickets which are ready for QA verification clear prior to the release cycle.

This approach significantly reduces the chances of critical issues slipping into the release candidate build and leads to a clean and blocker-free release cycle.

4. Use resources efficiently and remove the duplication of efforts

This method is largely related to efficient teamwork. It is very important to divide the work for the release between the QA Engineers since the more eyes there will be on the product, the fewer defects could end up in the released build.

A simple trick to try and avoid duplication of efforts in the early stage of the testing process is executing a different sequence of test cases at a given time. As an example, one team member could start from the bottom of the test suite and another one from the top. This approach will ensure that a potentially critical issue will be found way faster and the results will be provided in a timely manner.

Conclusion

The key benefit of minimizing the time of release cycle for QA Engineers is the additional time gained that can be directed towards the completion of other testing activities, thereby increasing the overall quality of the product, improving test documentation and expanding test coverage. The ability to manage the release cycle and keep the process up-to-date is a valuable skill set for any QA Engineer.

These methods allow Motiv to deliver more value to customers faster as well as save a lot of time for the company. Additionally, it significantly improved the performance of the test team and boosted confidence in a successful product launch.

Reducing time to release: minimizing the manual Test Cycle was originally published in Motiv Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

It starts with acronyms.

When I started iOS development in 2008, the best practice was something called Model-View-Controller (MVC). This was one of the first patterns to make a tsunami-sized wave in the global community. Before its adoption in iOS, MVC was the standard for desktop application design. It enabled the separation of critical aspects of the application lifecycle into discrete components whose responsibilities were clear. And there was much rejoicing.

By this time, Javascript was beginning to experience a huge resurgence in the web development community as a server-side language. The last ten years have been an extended golden age for Javascript, with new open source libraries popping up all over the place. During this same period, web browser users grew increasingly demanding of responsive features and reactive components. This ushered in the new hotness, called Model-View-ViewModel (MVVM).

With MVVM, components started reacting in a more fluid way to upstream data changes. With an in-memory store of a data model managing the state of your view hierarchy, browsers could offer unprecedented control to developers. There was suddenly an object in browser memory with all the properties of the view state based on the data model state. This allowed UI developers to build reactive components, where pending changes were assumed to succeed until the response data returned from the remote server provided new information. In this way, reactive design was born.

Drunk on its own euphoric rise to prominence, the MVVM community forked, and a new faction emerged to champion their new pattern, called Model-View-Presenter (MVP). As an engineer building systems with these patterns, I’ve had a hard time keeping them separate. Candidly, when MVP came along, I found myself wondering why it is any different from MVVM or MVC. At their core, the most innovative aspect is the isolation of key behaviors into discrete components with clear responsibilities. The specific nature of the delegation of responsibility seemed situational.

Then, a critical milestone was reached — Reactive Extensions (Rx) were introduced to the developer community. Before Rx was published, multi-threaded systems required ubiquitous consideration for execution context and tools to enable dispatched code to be executed on a specific queue, using some thread-safe mechanism. It was extremely difficult to find engineers with advanced skill in this area. Many excellent best practices emerged for managing task execution in a multi-threaded environment. With the advent of Rx, all that was turned on its head. Rx enables the development of synchronous multi-threaded systems whose components react to external influence. Moreover, Rx enforces a strict declarative approach to system behavior definition. In other words, reactive systems follow pre-defined pathways, triggered by specific notifications.

“When X happens, perform task Y.”

This is highly counterintuitive to industry veterans. Some of us have spent the last 20 years building things that work in a very different way, like this:

“Every N seconds, check the value of X asynchronously and conditionally perform task Y when X satisfies some criteria.”

Hilariously, even the English sentence required to describe the behavior is longer, just as the non-Rx code is typically more verbose than its Rx equivalent. Let that sink in…

Here at Motiv, we have encountered some very complex problems related to real-time data processing and we believe there is a better way than all of the patterns described above. We have devised a pattern — taking advantage of the Rx philosophy — to enable an unprecedented level of clarity in reactive design. We call this Provider-Model-View-Presenter (PMVP).

Overview

Provider

The Motiv ecosystem introduced a new component into the pattern. The Provider component is responsible for owning all lifecycle context for a group of data models. In its simplest form, you might think of it as a template, like Provider<T>, where T is the model type. This is highly flexible. You might have Provider<Array<T>> or Provider<Map<K,T>> or any arbitrarily complex data type. The key is the Provider object owns the lifecycle of all the objects it manages. We found the name “provider” the most universally clear and concise term. There have been flame wars across the internet about words like “manager” and “controller.” Adding fuel to the fire, the word “service” has special meaning on Android.

Model

In this context, Model refers to the ViewModel. The Provider owns the data model, so we can decouple our Model as a dedicated component responsible for view state. I want to be clear about one thing here. “View state” means the state of view components, but not anything like “label6 is visible.” If you envision the view components as being participants in a story, the Model is the storyteller.

View

In PMVP, the View components represent the standard elements used to compose a user interface, such as labels, images, etc. These components have no interactive behaviors associated with them. They are purely driven by presenters.

Presenter

In essence, the Presenter is responsible for updating objects in the view hierarchy. For simple views, this may require only one presenter. For more complex views, it may make more sense to separate groups of view objects into multiple presenters. Also, every Presenter must only be made aware of the ViewModel. In this way, Presenter is always a slave to ViewModel state. Presenters can only subscribe to observables exposed by the ViewModel. By definition, they shall never subscribe to external components; these subscriptions shall be defined in the ViewModel and exposed as derivative observables, if necessary. This guarantees the ViewModel is the ultimate source of truth for all behaviors defined in the UI layer.

PMVP in Action, a Tale of Two Implementations

Now that you’ve had an overview, let’s look at a real world example. We’re going to build a simple interface two different ways and compare them. This helps to highlight the benefits of using PMVP over traditional techniques. For our example, we will build a tamagotchi simulator. Let’s dive right in!

Requirements

Tamagotchi simulate the needs of a living creature as follows:

• Model shall track food as an integer property.
• Food shall decrease by one unit per tick.

Non-interactive components:

• Creature shall emote “HUNGRY” each tick when food is low (<10).
• Creature shall emote “DEAD” when food supply equals zero.

Interactive components:

• User can tap FEED button to add ten units of food, but not if creature is dead.

Initial conditions:

• Creature starts with 5 food.

View hierarchy:

view
- creatureImageView
- creatureStateLabel
- feedButton

Part 1: Traditional Approach

Using traditional approach, we would create a view controller to represent our tamagotchi state. We track the creature food state as a simple integer property on the view controller itself. Then we create a tick method to update the data model, and we invoke the tick method using a timer. This forces us to define logic in the tick method to handle the state of UI elements. It also forces our tick method to be invoked on main queue (because NSTimer needs a run loop to function properly).

class TamagotchiViewController: UIViewController {

  @IBOutlet private weak var creatureImageView: UIImageView!
  @IBOutlet private weak var creatureStateLabel: UILabel!
  @IBOutlet private weak var feedButton: UIButton!

  private var food: Int = 20

  override func viewDidLoad() {
    super.viewDidLoad()
    creatureStateLabel.isHidden = true
  }

  private func tick() {
    if food == 1 {
      food = 0
      emoteDead()
    }
    else {
      food -= 1
      if food < 10 {
        emoteHungry()
      }
    }
  }

  @IBAction func feed() {
    food += 10
  }
  
  private func emoteHungry() {
    creatureStateLabel.text = “HUNGRY”
    creatureStateLabel.isHidden = false
  }

  private func emoteDead() {
    creatureStateLabel.text = “DEAD”
    creatureStateLabel.isHidden = false
  }
}

Part 2: PMVP Approach

With PMVP, we introduce a view model and presenter to separate the responsibilities into discrete components. The food tracking properties move into the view model. Also note the view model exposes observables related to the UI components. Instead of publishing a “food” observable and forcing the presenter to define the “hungry” and “dead” states, we define them explicitly in the view model.

Provider

With PMVP, we must have a component responsible for managing the state of the data model. In this case, we define a CreatureProvider, which owns a Creature model and manages its lifecycle. Note: both public methods include a guard against invalid state; they only emit new state if the current state is valid.

struct Creature {
  var food: Int
}

class CreatureProvider {

  let creatureSubject = BehaviorSubject<Creature>(value: Creature(food: 5))

  private let disposable: Disposable

  init() {
    disposable = Observer<Int>.interval(1.0, scheduler: MainScheduler.asyncInstance)
      .subscribe(onNext: { [weak self] _ in self?.tick() })
  }

  func feed() {
    guard var creature = try? creatureSubject.value(), creature.food != 0 else { return }
    creature.food += 10
    creatureSubject.onNext(creature)
  }

  private func tick() {
    guard var creature = try? creatureSubject.value(), creature.food > 0 else { return }
    creature.food -= 1
    creatureSubject.onNext(creature)
  }
}

View Controller

The PMVP view controller has very similar structure to its traditional counterpart. It is responsible for owning references to all the view elements, as well as the model and presenter(s). Also, any interaction logic lives here. Note the inclusion of the tap observer, which delegates to the model.

class TamagotchiViewController: UIViewController {

  @IBOutlet private weak var creatureImageView: UIImageView!
  @IBOutlet private weak var creatureStateLabel: UILabel!
  @IBOutlet private weak var feedButton: UIButton!

  private let viewModel = TamagotchiViewModel()
  private var presenter: TamagotchiPresenter!
  private var disposeBag = DisposeBag()

  override func viewDidLoad() {
    super.viewDidLoad()
    presenter = TamagotchiPresenter(viewModel: viewModel,
      imageView: creatureImageView,
      label: creatureStateLabel,
      button: feedButton)
  }

  override func viewWillAppear() {
    super.viewWillAppear()
    registerObservers()
    presenter.registerObservers()
  }

  override func viewWillDisappear() {
    super.viewWillDisappear()
    disposeObservers()
    presenter.disposeObservers()
  }

  private func registerObservers() {
    feedButton.rx.tap
      .subscribe(onNext: { [weak self] _ in
        self?.viewModel.feed()
      })
      .disposed(by: disposeBag)
  }

  private func disposeObservers() {
    disposeBag = DisposeBag()
  }
}

Model

Now, the logic for tracking creature state is fully encapsulated into the provider. Changes to the creature data model are published via observable. The view model subscribes to the provider observable and reacts to changes. In this case, the view model tracks the creature’s food directly, binding it to a local BehaviorSubject. Furthermore, the model now also publishes observables specifically intended to drive behavior in the presenter.

class TamagotchiViewModel {
  private let foodSubject = BehaviorSubject<Int>(value: 5)
  private let disposeBag = DisposeBag()

  init() {
    CreatureProvider.instance.creature()
      .map({ $0.food })
      .observeOn(MainScheduler.asyncInstance)
      .bind(to: foodSubject)
      .disposed(by: disposeBag)
  }

  func hungry() -> Observable<Bool> {
    return foodSubject.map({ $0 < 10 })
  }

  func dead() -> Observable<Bool> {
    return foodSubject.map({ $0 == 0 })
  }

  func feed() {
    CreatureProvider.instance.feed()
  }
}

Presenter

This class is responsible for reacting to state changes in the model and updating view elements accordingly. Note: there is no interaction logic being defined here. The observer subscriptions are simple content bindings. All interaction observers are defined in the controller.

class TamagotchiPresenter {
  private let viewModel: TamagotchiViewModel
  private let imageView: UIImageView
  private let label: UILabel
  private let button: UIButton
  private var disposeBag = DisposeBag()

  init(viewModel: TamagotchiViewModel, imageView: UIImageView, label: UILabel, button: UIButton) {
    self.viewModel = viewModel
    self.imageView = imageView
    self.label = label
    self.button = button
  }

  func registerObservers() {
    Observable.combineLatest(viewModel.hungry(), viewModel.dead())
      .map({ !($0 || $1) })
      .bind(to: label.rx.isHidden)
      .disposed(by: disposeBag)

    viewModel.hungry()
      .filter({ $0 })
      .map({ “Hungry” })
      .bind(to: label.rx.text)
      .disposed(by: disposeBag)

    viewModel.dead()
      .filter({ $0 })
      .map({ “Dead” })
      .bind(to: label.rx.text)
      .disposed(by: disposeBag)

    viewModel.dead()
      .bind(to: button.rx.enabled)
      .disposed(by: disposeBag)
  }

  func disposeObservers() {
    disposeBag = DisposeBag()
  }
}

Comparison

Cons

• Rx Required. Using this pattern will require your engineers to learn Rx.
• Steep learning curve. While is it incredibly powerful, reactive design is very different from traditional techniques to software design. There are many rough edges, pitfalls, and gotchas, and it can be challenging to identify the source of strange behaviors. Learning Rx can be especially challenging for industry veterans, who must adjust to a “new normal” and think about things backwards and upside down.
• Verbose. For very simple examples like the one given above, PMVP feels like a little overkill. We must add two new classes, and each class has its own boilerplate requirements for managing disposable lifecycle.

Pros

• Reactive. The declarative nature of Rx leads to highly readable code, acting more like a recipe than a procedural sequence of commands. Instead of using method invocations to fetch state from external sources, the pattern is inverted; we have observers receiving new values and responding with specific behavior.
• Team-friendly. The decoupled component architecture enforces a natural integration boundary between components. This makes it easier to delegate development responsibility to team members. One person can build a view model, while another builds a presenter. Moreover, different functional areas within a complex UI can be split up into multiple presenters — one for each area — and built by different team members.
• Easy to review. Our engineers celebrate the elegance and simplicity introduced into our code review process. PMVP changes are easy to understand in context and decrease the cognitive load on the reviewers.
• Testable! The introduction of Rx-based view model and presenter make it easy to test the interaction between data and UI layers. Providers can be mocked into test fixtures to exercise the view model observable behavior in response to changes published upstream. This makes it easier to incorporate unit tests into your UI code.

Conclusion

We’re really excited about our new pattern, and we’re using it in all future versions of the Motiv app. It truly crystallizes development of robust components by forcing developers to focus on critical aspects of a design before building anything. It demands a consideration for state machine dynamics and compels engineers to define all the external triggers causing state transition. This helps your team members break down complex interactions into something simple and easy-to-understand.

We couldn’t be happier about PMVP, and we look forward to sharing more articles and screencasts over the next year!

Introducing PMVP, a Reactive Mobile Application Design Pattern was originally published in Motiv Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Every time I take more than a few days of vacation, I bring at least one book for professional development. (And try to finish reading it before I binge on sci-fi and beaches.) This time, I thought I’d share my learnings with you.

For this trip, I chose “Measure What Matters,” a recently released book by John Doerr, the influential venture capitalist and godfather to many successful silicon valley companies. “Measure” introduces the OKR system, a management tool for aligning, inspiring and stretching organizations to peak performance and unity of action.

The tagline is catchy: “How Google, Bono, and the Gates Foundation Rock the World with OKRs.” Wouldn’t you like your team to have the impact those groups have had on the world?

What’s in it?

The book is split into three main sections, each of which comprise several chapters, weaving stories of how organizations have implemented OKRs to great success, along with principles and lessons.

OKRs stand for “objectives and key results.” Objectives are your organization’s near-term goals to push you towards a vision, with key results being the indisputable yardstick of achievement.

The main principles, or superpowers, covered between the stories are: focus and commitment to priorities, align and connect for teamwork, track for accountability, and stretch goals.

There are two chapters on another useful and related management tool: CFRs, or conversations, feedback and recognition. This is worthy of its own book (even though much has been written on the topic), and I’m glad “Measure” introduced it and tied it in with the OKR system.

What it means to me

I was first introduced to OKRs during my time at Intel, where some parts of the organization used MBOs in classic Drucker fashion, and others used OKRs, which owe their genesis to Andy Grove. (There are several important distinctions which I’ll completely skip over.) Reading “Measure” gave me a deeper understanding of the principles and benefits of the OKR system.

“Measure” also subtly interweaves an important set of concepts: input, activity, output, outcome, and impact (see logic model). The best OKRs measure outcome and/or impact (settle for output if you have to), since inputs and activities can just waste resources (the “activity trap”). Input and activity goals are easy to avoid with a little bit of effort (and “Measure” has some practical tips).

Going from output- to outcome/impact-oriented can be challenging in particular for engineering teams, since we usually focus on completing projects within the context of known cross-functional programs and organizational goals, and take great pride in shipping. OKRs for maintaining infrastructure and product quality metrics are usually easier to frame. Practically, your OKRs will probably include a mix, but whenever possible, phrase the goal in terms of the impact you’re looking to get from achieving it, and you will measure up to your full potential.

I wish I had access to this book when I took on my first leadership role about a decade ago. The combination of narrative examples and principles created a powerful learning experience that I hope to draw on in my work. Go read it, or learn more at whatmatters.com, and let’s compare notes.

If you read it carefully and reflect, you’ll get a practical guideline to introducing structured goal setting in any organization to get more better results per unit of effort, along with some inspirational stories that convey nuances and options you have to customize it. If you’re already doing OKRs, read the book and re-review your current set of OKRs; you’ll probably see a lot of room for improvement. (I did!)

OKRs at Motiv

At Motiv, we use the OKR system (with a right-sized tool for our company size: a Google Sheet). It’s provided a level of transparency, accountability, and alignment that otherwise gets lost when you grow beyond a small team.

If you are interested in rocking the world with us and using OKRs, check out our careers page — we’re hiring!

Book review: “Measure What Matters” was originally published in Motiv Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Topics: iOS, Mobile Analytics, SQL, NLP, n-grams

Apps running in the background on iOS are subject to a set of rules, defined by Apple. In some circumstances, the operating system will force a backgrounded app to be terminated, usually to free up resources for an active app. The specific details of when this happens are not published by Apple. They simply advise developers to be aware their apps may be evicted in this way. In this scenario, the app lifecycle methods are not invoked, so there is no convenient callback we can use to log the eviction. We may receive a memory warning callback, but even that is not guaranteed. Instead, we must rely on our data warehouse and logging infrastructure to provide critical information regarding the state of the app in the wild.

Prerequisites

Before we can discuss the approach in depth, we need a clear picture of the tools we’ll need to achieve our goal.

Distributed Logging

This approach is predicated on existing logging infrastructure. We use Amazon Redshift to aggregate logging data. Our logger works based on unique strings representing significant events within the app. We use multiple logging levels to categorize logs, but for the purposes of this piece, we’re focused on the “info” log level, which has the following method signature:

protocol Logger {
  class func info(_ name: String)
}

Each time this method is invoked, we effectively add a new row in a Redshift table. This table is queryable via SQL. This is important because we’ll be using some advanced SQL techniques later.

App Lifecycle Logging

At an absolute minimum, we need to track the lifecycle of our app in the wild. To do this, we annotate our AppDelegate with launch and termination logs, like this:

func applicationDidFinishLaunching(…) {
  Logger.info(“app.launched”)
}

func applicationWillTerminate(…) {
  Logger.info(“app.terminated”)
}

Of course, other areas of your app will need to have similar logging for this approach to be useful. As long as you include logging for your critical systems, you can probably use this technique.

Methodology

Zooming out a bit, app logging manifests as a collection of streams, one for each user. We can take advantage of this naturally sequential phenomenon to study the flow of execution. If we include sufficient logging in the various components of our system, we can highlight sub-sequences where the normal flow is truncated. These represent moments within a user flow where the user encountered a crash or eviction. We can use SQL tools to identify and characterize these scenarios. Consider the following sequence:

app.launched
app.fetch_started
app.fetch_completed
ui.refresh
app.backgrounded
app.background_update_started
app.background_update_completed
app.foregrounded
ui.refresh
app.terminated

This simple example shows a sequence we might expect to loop over time, as the user interacts with their app. As long as the user remains active, this sequence or one similar to it will appear in the data warehouse, when querying for the given user and ordering by timestamp. If this stream is interrupted by a crash or eviction, we will abruptly see the first message in the sequence, in this case app.launched immediately preceded by a message other than app.terminated in the sequence. The following example shows a truncated sequence:

app.background_update_started
app.launched

In this instance, the last log we see before the app.launched indicates the app was processing a background update before it was unexpectedly killed. We can learn quite a lot from analyzing sequences of logs in context with each other. These anomalies can help guide in diagnostics. This can help avoid costly exploration and/or logic audit by prioritizing high-value options first.

Natural Language Processing with N-Grams

There is a rich set of tools available for analyzing sequences. Here, we will employ a powerful tool from natural language processing (NLP) to organize the logging data in our warehouse. Log data is a time-ordered sequence of “words,” represented by individual log messages. In NLP parlance, these are known as n-grams, or a sequence of n words in a row. For example, a trigram is a sequence of three words in a row, like “take the bus.” If we map this technique to our logging data, we will find lots of unique n-gram sequences. Filtering these n-grams down to those ending with the app.launched log, we can characterize the scenarios leading to early termination. This analysis results in a report showing the most popular eviction n-grams.

For our purposes, we will focus on trigrams. The general outline can be extended to support sequences of any arbitrary number. Let’s dig into the SQL query details.

SQL Lead/Lag

One very powerful suite of SQL functions makes this problem very easy to solve. Most SQL functions produce results for records matching conditional criteria. This is great for known conditions, but it doesn’t tell you anything about the context of the rows relative to the other rows. With LEAD/LAG, we can select adjacent rows and include them in the context of their neighboring rows. Here’s an example of a query to identify the number of rows matching our first log:

select
event_type,
count(*)
from logs
where event_type = ‘app.launched’
group by event_type

This query will count all instances in our data warehouse with the matching event type, which is very unhelpful for our purposes. However, adding just a little extra to this query will add substantial value. Here’s an expanded example:

select
event_type,
lag(event_type)
  over (partition by user order by event_timestamp)
  as previous_type
from logs
where event_type = ‘app.launched’

This will result in a list of all combinations of app.launched and their preceding log. It’s important to note the options in the OVER clause. These determine the constraints of the LEAD/LAG function. If we do not include the user partitioning, the results will be nonsense, as they will include records from potentially different users. By partitioning, we’re instructing the database to include results where the adjacent records are related by a common user. Now we have a tool to find all the instances of termination, but we’re not quite done. We need one more thing to support trigrams.

The LEAD/LAG functions work beyond the adjacent rows. They also support an offset parameter. Using this parameter, we can specify an arbitrary offset from the given reference row. This allows us to construct a simple query to organize our trigrams.

select
event_type as e0,
lag(event_type, 1)
  over (partition by user order by event_timestamp)
  as e1,
lag(event_type, 2)
  over (partition by user order by event_timestamp)
  as e2
from logs
where event_type = ‘app.launched’

Now we have a result set with e0,e1,e2 columns and rows representing all combinations of three events leading up to app launch. This collection includes the normal termination scenarios as well, since they also culminate in app launch. We need to wrap this in a second query to filter these out. This looks something like the following:

select
e0,e1
from (…)
where e1 != ‘app.terminated’

Finally, we modify this to include consideration for statistics, so we can order our results. Observant readers will note we’ve removed the e2 parameter from the result set, as it will always be app.launched. This simplifies the query and allows us to use the GROUP BY feature, resulting in the following:

select
e0, e1, count(*)
from (…)
where e1 != ‘app.terminated’
group by e0, e1
order by count desc

Now, we have a result set showing each eviction trigram and its corresponding frequency. The resulting table is immediately actionable, with the top offenders presented nicely along with their individual significance. The results might look something like this:

e0,               e1,                            count
app.backgrounded  app.background_update_started  987
app.backgrounded  ui.refresh                     658
app.foregrounded  app.backgrounded               276

In this case, the chief offender is a stalled background update; a process starts but never finishes. Next appears to be the dreaded “UI update on background queue” causing the app to use too many resources on a main queue while in a background state.

Accelerated Diagnostics

Using these techniques, you can begin to explore some of the undiscovered sharp edges of your user experience. This targeted approach allows engineers to reduce the total time spent solving hard problems. This shines a spotlight on a traditionally dark and devious area of multithreaded systems — deadlocks. When your app deadlocks in the wild, most likely the operating system will simply evict it in the background. The user may not notice, especially if your app refreshes its state on becoming active. This means thankfully no users report problems, but it also means tragically no users report problems, so the problems go unreported. By applying some creative multidisciplinary thinking, using simple tools, we can achieve a powerful result, reducing time to resolve challenging bugs.

Using Natural Language Processing Techniques to Identify iOS App Eviction was originally published in Motiv Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Prepare yourself. This is one of the biggest horrors of Motiv app engineers, the bane of our existence:

We call this the “infinite spinner.” That, to be fair, is technically incorrect, as none of us have waited past the heat death of the universe to see if it resolves. It’s much more likely to be starvation, deadlock, or livelock.

We almost never see this issue. That doesn’t negate the user pain of having to wait, then decide it’s not going to fix itself, kill the app, launch it (waiting for the OS to completely evict the prior process), wait for it to load, and manually sync the ring and wait for the data to transfer and process, since the starvation probably prevented automatic background syncs from occurring.

This leads us to problem two: since the user kills the app, and it is not detected by a crash reporter, we have zero visibility into the frequency or root cause of this problem. Zilch. Nada. No visibility means prioritizing it and debugging it is impossible!

But wait. I have a screenshot of the behavior, and know that it happened. It turns out, seeing this on an alpha build and knowing what to do are actually sufficient! We don’t have to know the size of the iceberg at first, just knowing that there is one helps us steer around it and bring other specialty tools to bear. Let’s look at two different doors.

Door 0: Avoid the problem

Some architectures lend themselves to reducing certain classes of human errors.

For example, typed languages catch at compile time many types of bugs that would be runtime bugs in dynamic languages. Some environments may offer thread safety analysis.

However, exhaustively checking that all possibility of these class of problems has been eradicated is most likely equivalent to the halting problem, so it’s computationally infeasible to prove. Additionally, rewriting an entire system to solve one bug is a tough and highly speculative investment to make. (What if it doesn’t solve the problem?) So let’s leave this door closed for now.

Door 1: Sysdiagnose

There is a very useful, but little known, system feature in iOS called sysdiagnose. You trigger it on recent devices by pressing both volume buttons and the home button at the same time. Your phone should vibrate, indicating sysdiagnose has done its job. There are articles on how to use sysdiagnose, but nothing about how to get value out of it as a developer.

Sysdiagnose writes some files named stacks-YYYY-MM-DD-TTTTTT.ips. Once you sync with iTunes, you’ll find them in ~/Library/Logs/CrashReporter/MobileDevice/DEVICENAME/. These files contain a record of every single process in the system, each thread for each process, and each stack frame in each process! Sounds useful.

Here’s the downside: they only contain program counter offsets. Useless!

However, if you’ve been a good iOS engineer, you’ve archived all of your dSYMs from every alpha and external build somewhere permanent (in our case, in an S3 bucket).

Enter symbolicate-stackshot: a simple Ruby command-line utility to transmogrify your stack file plus the dSYM into something human-readable! Now, visibility!

Here is one of several threads stuck in similar places:

*******************************************************************

Process 1150

========================================

Thread 843020

0x000000010097d7c4 (in Motiv)

0x0000000100802498 (in Motiv)

0x0000000100802e88 (in Motiv)

0x00000001007f4b44 (in Motiv)

1836ceb4c

@objc ReadableMotivModelContext.performAtomicallySerially(readBlock:) (in Motiv) (ReadableMotivModelContext.swift:0)

-[MotivModelController performReadOnly:] (in Motiv) (MotivModelController.m:214)

+[MVDailyTargetService weeklyTargetForWeekContaingDate:forType:] (in Motiv) (MVDailyTargetService.m:81)

In our case, the primary failure revealed an issue with our CoreData setup that caused starvation of the original main thread and temporary saturation of background threads. Embarrassingly, it also caused several operations that should have been durable/synchronous database updates to be asynchronous.

From the commit message from one of our developers:

A simple execution of the following code would cause a deadlock

for _ in 0…50 {

Dispatch.global().async {

MotivModelController.instance.performAtomically { (context) in

//insert some object

}

}

}

The awkwardly small diff:

- parentContext.perform {

+ parentContext.performAndWait {

Door 2: Cerberus, the watchdog

The venerable watchdog is old hat to embedded programmers. The concept is simple: part of your system (usually strictly separated from the other parts of your system) keeps time, and when it reaches a predetermined time value, it resets the system. When your system is operating normally, it “kicks” the watchdog. If something prevents normal flow through, your system automatically recovers.

The watchdog is usually implemented in hardware, though some systems have software watchdogs. Hybrid systems can be powerful; a software watchdog, triggered by a timer in ISR context, can save diagnostic info when it works so you have information to debug from, and a hardware watchdog is your safety net from total and complete lockup. But I digress.

Let’s apply the watchdog concept to a mobile app.

For complex or top-level user operations (example: taps on ‘week’ view), we could start an NSThread that sleeps, then logs or crashes. Completion of the regular code path (data loading and refreshing the screen) would then cancel the watchdog thread.

We have selectively deployed a limited version of a watchdog that only logs initially, as it will bring visibility to problems we may otherwise only occasionally hear a non-actionable (no data to debug) report from a customer.

What to do if you make the watchdog too sensitive, and your users experience crashes far too frequently? Consider having a server-controlled parameter (feature flag) that switches the watchdog between logging and crashing. Also consider setting the timeout relatively high, as your users may navigate faster through your app than you anticipated or tested for, leading to a backlog of operations to complete!

Wrapping up

Check out our basic sysdiagnose stackshot symbolicator over here on GitHub. Pull requests welcome!

I hope you’ve learned a bit more about failure modes of multithreaded programs, and now have some tools for diagnosing or working around some of the most difficult class of bugs.

Motiv is hiring — if you want to work with us, check out our careers page!

App Zombies: Feeding the iOS Deadlock was originally published in Motiv Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

First, a word about Daylight Saving Time

Most of the world has a sensible approach to time. Here in the USA, we suffer from a long-failed experiment to win back more daylight. This manifests in confusing three-character time zone codes (PST, PDT), encoding not only where you are on the planet, but also any silly local offset to which you might be subjected. Unsuspecting users may simply overlook the bizarre behavior of their mobile app when, twice a year, we abruptly redefine “now” as “an hour ago” or “an hour from what was otherwise now.” This nuance of US federal law requires everyone to introduce a different offset suddenly on pre-coordinated dates. We all “spring forward” or “fall back”, leaving our computers wondering what is going on.

Sure, but it’s still just a time zone offset, right?

So far, we’ve gotten lucky. It’s currently cost-prohibitive to travel faster along the surface of the planet than one time zone per hour. For the collective sanity of all my fellow engineers, let’s hope this continues long enough for compound interest in our pensions to distract us. Until then, we’re comfortable knowing there is very little chance of a user’s going for a run in San Francisco then jumping on a plane before their wearable has had a chance to sync, only to sync in New York, making it look like they ran three hours ago. Worse still, if there’s any sort of bug related to time, it quickly becomes unclear if values in the future are actually in the past.

Pitfall #0: Assuming Local Time / Ignoring Offset

Problem

You board a transcontinental flight in San Francisco at 12pm, flying east. When you land in New York, the local time is probably around 9pm. Without crystal clear agreement of the time zone offset, your phone syncs with your connected device, and data is recorded based on the time stamp from the originating time zone. Then, when your phone has a chance to sync with the cell tower network, it adjusts the clock accordingly. This wreaks havoc on your app’s ability to align itself in space-time.

If your mobile device is logging current time without clear and distinct policy surrounding time zone offset, all bets are off. Put another way, it’s critically important all the components of your ecosystem agree about the current time. This is magnified when you have users spanning multiple time zones.

Symptoms / Early Warnings

• Odd time shifts — Data recorded apparently from the future, but contextually pinned to the past. This means your ecosystem is potentially out of sync with itself. If you find yourself filtering out future timestamps in an SQL query, you may suffer this problem.
• Unique index constraint violations — If you use a database system where the time stamp is used as a primary key, as is common with time-series data, you may experience bizarre constraint violations when a device changes its system clock. Some records may have conflicting timestamps due to overlaps introduced by the clock change. Records received before the clock change may overlap with those received afterward.

Solution: Standardize on UTC (Zulu) time

If every device participating in your ecosystem exists on GMT, the only time offsets your systems will experience are relativistic in nature.

Reasoning

Let’s be honest — time zone confusion is a 20th century problem. If we’re looking forward to 21st century problems, relativistic time distortion seems par for the course.

Pitfall #1: Assumptions About Reference Times

Problem

As a general rule, software engineers love functions with inputs and outputs. It’s easy to think of things in these terms. We can provide a controlled set of input values and measure the output, comparing it against expectations. If the values match, the test passes. You could spend your whole career, building a rich and complex understanding of the systems you curate, and never encounter the scenario I’m about to share. Hopefully, this helps you to avoid what I suffered.

I once designed and built a dynamic rules-based event processor system. This system reacted to simple events with complex reactions, based on conditions defined by a downloaded file. One of the components of this system was a time filter. If an event occurred within some threshold of a reference time, the function would return true. Naturally, I created a test for this. Tests passed, and code review was positive, so I merged the changes. A few days later, one of the tests failed.

The triage team scratched their heads and thought “well, if it happens again, let’s re-evaluate.” The next day, the tests passed, so the triage team disregarded the issue as an isolated anomaly. A week later, the same test failed again. This time, the triage team escalated the issue directly to me, saying “we don’t know why, but it happened again.”

At this point, I had the benefit of perspective. When I saw the report, I thought there are very few reasons for this sort of problem. It could be concurrency failure. We had previously had concurrency issues with some C++ standard libraries. (There’s a special place in hell for std::localtime.) So, I checked for use of undesirable libraries, and I found we were doing everything right. Then, I saw the culprit. Deep inside a logic flow, I found this:

NSDate* now = [NSDate date];

On its face, this is innocuous. Surely, it’s normal to know when things happen. Yes. Normally, yes. However, in the world of unit tests, where we have prescriptive expectations, the consideration of “now” is weird. From the perspective of the test, I had written something like this:

assert(!isTuesday, “encountered Tuesday”);

Of course no one would write this, and I certainly didn’t write this intentionally. It ended up looking more like this:

assert(expectedDayOfWeek == actualDayOfWeek, “unexpected day of week”);

I’m sure we can all agree we the nuance is easily lost.

Symptoms / Early Warnings

• Repeating failures on specific days of the week.
• Assumptions about “now” — Any time you see some code attempting to measure the current time, take extra care in assessing the risks. This goes double for delayed invocation, dispatching blocks to asynchronous queues. It’s easy to inadvertently introduce an unexpected delay.

Solution: Injecting Time Values (Don’t assume “now”)

The simple solution is to move the “now” evaluation up one level to the calling component. In this case, it’s a simple matter of refactoring, like this:

// TimeCondition.swift
// determines if delta between date and reference is greater than
// a specified threshold

// old
func isSatisfied() -> Bool {
let date = Date() // ← assumption about “now”
return date.timeInterval(since: self.referenceDate) > self.thresholdValue
}

// new
func isSatisfied(date: Date) -> Bool {
return date.timeInterval(since: self.referenceDate) > self.thresholdValue
}

Reasoning

• Inversion of Control — Requiring an input to the method enables a wider variety of potential uses. This also allows you to craft your tests with specific values for known edge cases, rather than relying on hidden assumptions.
• Self-describing Design — When a developer encounters your method descriptor with a date value in its signature, they know immediately they will be required to provide a date.

Pitfall #2: Device System Clock

Problem

Mobile phones offer many luxuries to users. One such convenience is the ability to synchronize the system clock to the universally adopted definition of “now”. The vast majority of users have enabled this useful feature. However, there are some users who have adjusted their device system clocks, and their device does not agree with the internet about what time it is. This manifests in strange data, either from the future or the past.

When working with a distributed hardware ecosystem, where some computation happens on a Motiv Ring and some happens on a mobile device, time synchronization is even more critical to overall system health. When we’re measuring time-series data, it’s essential to have nearly perfect agreement between devices about the current time.

Everything in the universe experiences time slightly differently, due to motion relative to other things in the universe. For example, clocks at substantially different altitudes experience subtle drift in their values, when measured over long periods of time. To account for this, the clock in a Motiv Ring is synchronized with the mobile device each time we sync. If these values diverge, it can manifest in very unusual ways.

Symptoms / Early Warnings

• Negative or Large Time Offsets — We use a cloud-based analytics platform to receive application log data from devices in the wild. Our system includes consideration for both the time stamp when the event occurred and when the log was received by the web service. Large discrepancies between actual and received time stamps can indicate non-standard system clock.
• Unique index constraint violations — As noted above, using the system time as a primary key for records created at a specific time can lead to unique constraint violations.

Solution: Use Network Time Protocol (NTP)

Rather than relying on the user to specify the correct time, query an NTP server cluster on app launch. This way, you avoid any unexpected behavior due to user customization of their device.

Note: this only works for network-connected devices.

Lyft offers a convenient Swift library for integrating this functionality into your iOS app. (https://eng.lyft.com/freezing-time-6ebe8ffe3321)

Instacart also developed a similar library, called True Time. (https://tech.instacart.com/offline-first-introducing-truetime-for-swift-and-android-15e5d968df96)

Effective Unit Tests for Time Travel was originally published in Motiv Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

To start with, we wanted to share a little about how we think about teams and how we work together.

A few bold claims

The best teams share a common purpose: tactical execution towards a strategy to complete a mission to fulfill a vision. They work together to achieve this vision by excelling in their individual roles and collaborating effectively towards those shared goals.

In complex, uncertain, ambiguous environments (hat tip to Andy Grove¹), it’s only possible to be effective through shared culture — by that, I mean what kind of behavior is tolerated and encouraged.

One great way to build culture is intentional: by writing down what you want and having every team member embody and reinforce the team’s principles and standards. This helps create psychological safety² and cultural predictability.

What are some tools we use to encourage effective and positive human behavior? Let’s look at principles, standards, and autonomy.

Principles

Principles should be inviolate. Every action we take should be in accordance with our principles. Therefore, we should have a few, carefully chosen, memorable principles that are in natural harmony.

Motiv Engineering has chosen the following four principles, in no particular order:

• User-centric: Everything we do is centered around the users of products.
• Pragmatic: We balance the short term and the long term, acting in the best interest of our customers, teammates, and the company.
• Holistic: We embrace big-picture thinking, taking responsibility for the outcome, and owning our work end-to-end.
• Respectful: We treat our teammates with respect and empathy — healthy conflict is important to any functioning team; handling it well is critical.

So what?

How did we come to these principles? What do they mean, practically? Stay tuned for another post in this series and even more about standards and autonomy.

What principles and standards have made your teams effective and happy? What anti-patterns have made your teams ineffective or unhappy? Share your experiences.

Lastly, we’re hiring in a variety of engineering roles across the company. Apply at our careers site!

Citations

High Output Management, Andy Grove

What Google learned from its quest to build the perfect team

Engineering Principles was originally published in Motiv Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.