This is the last in an eight-part series on implementing data sharing in Shopping UK using CloudKit.

Shopping UK is a smart shopping list for UK shoppers. It knows almost every product in the supermarket and will arrange them by aisle. Lists can be shared with family or friends.

Last week, we looked at how to change or stop a share, what happens when a list is deleted and background maintenance. Today, we’ll wrap up the series with a look at error handling, merging data, and diagnosing problems.

The Path of Sadness

Throughout this series, the focus has been on the happy path — when everything goes to plan — but a synchronisation solution is only as good as its ability to handle unexpected situations.

We’ll start with a look at what can go wrong before examining how Shopping UK handles things.

I can’t guarantee I’ve thought of all possible problematic scenarios, but I will share what I built, what I learned from it, and some techniques I used to help diagnose problems when they do arise.

Why do things go wrong?

Synchronising data is hard for two reasons:

• Networking: data must be moved between physically separated systems — the user’s device and iCloud — using an unreliable network.
• Merging Data: data must be combined from different sources — between each user’s device and iCloud — while maintaining data integrity and striving to keep data consistent across all devices.

The problem with networks

CloudKit provides a clean interface for moving data between a device and iCloud, but it cannot change the nature of networks:

• The network is never reliable.
• Latency is never zero.
• Bandwidth is never infinite.
• Transport cost is never zero.
• The network is never homogeneous.

Every request sent from your app to iCloud must travel from the user’s device to iCloud across many networks. Each message will start on a Wi-Fi or a mobile (cellular) link before finding larger internet backbones, and eventually arriving at the datacenter that hosts Apple’s iCloud servers. The path taken to iCloud will be different for each user, and possibly for each message.

Wi-Fi will not always be available, mobile (cellular) data may be disabled (e.g. flight mode enabled or data roaming disabled) or have patchy reception, iCloud may be down or busy, and messages may be lost.

Nothing happens instantly. Each message will take several milliseconds, or longer, to arrive — this is millions of times slower than a message can be processed on the device. Many messages could be in-flight at the same time. Many responses may arrive at the same time.

Everything has a cost. Larger messages need more processing, use more iCloud storage, and eat up more of the user’s mobile (cellular) data plan.

Everyone will have a different experience. Some users have ultra fast broadband, some are limited to a weak or slow data signal. Some parts of the world are closer to a datacenter than others.

The problem with data merging

The aim of a synchronisation solution is to move data between devices to give each user the same up-to-date view of the world.

But the network won’t always be available, messages take time to move between devices, and users expect their app to continue to work when their mobile signal is patchy. The best we can hope for is for all devices to eventually show a consistent view of the data.

And this means we must consider what to do when the same data is changed by two people at the same time.

How should this be resolved?

1. Should the changes be applied in order?
2. Should they be somehow merged?
3. Should they both be rejected?

There isn’t a “right” answer to this — it depends on the nature of the app, the meaning of the data, and the expectation of the users.

Taming the network

When designing the sharing strategy for Shopping UK, I had several guiding principles in mind:

1. Synchronise changes with others as soon as possible.
2. Don’t lose any changes.
3. Minimise user’s data costs.
4. Use the battery efficiently.

These are not universal principles that work for all apps. If your app shares the user’s location in real-time, it may be ok for a single update to be lost. But lost items is not a good look for a shopping list app.

Because the network is unreliable, I chose a queue-based solution.

When the user adds an item, the app creates a JournalEntry record to represent the change. This is saved to a local queue and then the queue is uploaded to iCloud (using CloudKit).

If the network goes down nothing is lost.

The same thing happens when fetching changes from iCloud. Newly received JournalEntry records are saved to a local queue and the queue is processed, and each change is applied to the list in turn.

After a local change is uploaded, it is removed from the upload queue.

After a remote change is applied to the device, the entry is removed from the to-apply queue.

An upload may fail for many reasons. Look at the CloudKit errors for a full reference.

What does the app do when it receives an error?

Generally, Shopping UK will simply show a friendly form of the error in the Activity Log to let the user know synchronisation may be delayed.

For example, when Airplane Mode is enabled, a NetworkUnavailable will be returned:

Is this what happens for all errors?

Nope, just errors that cannot be fixed by the app.

CloudKit protects itself from huge messages.

• If a request is too large or contains too many records, CloudKit will reject it with a LimitExceeded error.

• If requests are made too quickly, CloudKit may reject some with a RequestRateLimited error. I’ve also occasionally seen a PartialFailure error with the message “will not be saved but can be retried as is”. This appears to happen when iCloud is under heavy load.

The app knows what to do about this type of error:

• When a request contains too many records Shopping UK splits the list in half and sends each half separately. If either half fails, it is split in half again. This halving continues until everything has been sent successfully.

• When the request is rejected due to rate limiting or when the “will not be saved but can be retried as is” message is returned, Shopping UK will wait a while and try again.

Taming the merge

Is a merge always necessary?

No. It can be avoided by never caching anything on device, and treating iCloud as the single authoritative source:

• When your app needs data, it fetches it from iCloud.
• To change data, your app immediately uploads the changes to iCloud.
• If the upload fails because another device made changes at the same time, a fresh copy is fetched, and the user makes the changes again.

This is the way websites worked in the very early days.

This may be ok for some types of app, but it doesn’t work for a shopping list. If you’re in a rural shop with no data signal it would be frustrating if your shopping list was offline and could not be viewed.

So we’re stuck with the reality that data on the device and the data in iCloud can diverge. And this means merges are necessary.

When do we merge?

The beauty of a using a write-only journal means merges are never needed when uploading changes to iCloud. Each change is simply appended to the set of JournalEntry records in iCloud.

But this doesn’t eliminate the merge, it just moves it elsewhere! This is why I love design. Nothing is free; everything is a trade-off :)

For Shopping UK, the merge happens when we need to apply the changes to the user’s device.

Here’s some examples.

Adding while offline

Alice and Bob are both offline, with no phone reception.

Alice adds ‘bread’ below ‘milk’ moments before Bob adds ‘eggs’ below ‘milk’.

What happens when their network returns — how do we resolve this conflict?

Shopping UK will apply the changes in order. This can result in Alice and Bob’s lists appearing in a slightly different order:

Remember this is an unusual event. It is not often that both users would add items to their shared list while both were offline. And, although each user has a different ordering, it won’t affect how the lists are used. Items are never lost, and when Alice and Bob start shopping, their lists will be automatically categorised and ordered to match the aisles in the supermarket.

Arguments could be made for a better approach, but in my mind, this is a reasonable compromise that has the benefit of allowing the app to continue to be used offline.

Renaming or recolouring the list while offline

Some merge situations are a lot simpler to resolve.

If two users were to change their shared shopping list’s colour or name while offline, Shopping UK would simply apply the changes in the order they were made — the last one wins.

For example, Alice and Bob are both offline. Alice changes the list’s name to “Weekly Shop” a second before Bob renames the list to “Alice and Bob’s List”. When the devices re-join the network, each device will upload the “rename” change, and both devices will fetch the other person’s “rename”. On both devices, the operations will be applied in the same order. This will result in the list being called “Alice and Bob’s List”, because Bob made his change after Alice.

Easily Diagnosing problems

Things often went wrong while I was building the synchronisation solution — due to my misunderstanding, bugs, or undocumented iCloud behaviour.

Diagnosing the cause of the issue was not always easy.

Here’s some things that helped me.

Logging to Console

Having detailed logs available was essential for diagnosing problems. Many bugs cannot be found simply by capturing the current state of the system. It is often necessary to see how the state changed over time. Most of CloudKit’s calls are asynchronous and this adds to the complexity.

When writing code, I added “write-to-log” messages everywhere — for almost every path through the code. In many cases, these log messages became a substitute for code comments.

My logging framework has configurable verbosity. While developing, I set it to DEBUG mode, to show every message in the console. This meant I could see exactly what was happening internally, and in real-time. The logs showed every decision, the contents of every request, the contents of every response, error messages, everything.

For the version released to the App Store, only ERROR messages and critical information is logged. This means logging won’t slow down normal operation.

Real Time Journal Diagnostics

In addition to console logging, I found it useful to add a view to show the state of the upload and to-apply queues. This floating window shows how many of each type of operation are waiting to be sent to iCloud and waiting to be applied to the device.

Tapping on the window will show the entries in the queue, and tapping an entry shows further detail:

This view is hidden for the App Store version of the app, but it is incredibly helpful while developing.

Diagnostic Log

Finally, I’ve extended the information included in the diagnostic log users can send me when they encounter a problem.

This log now contains details of the CloudKit configuration, and the contents of the upload and to-apply queues.

The information in this report should prove invaluable for tracking down the root cause if problems are reported by users.

The End

And that concludes the series. I hope it has been useful. Please send any errors, omissions, or feedback to @wheelies

Before I go, I’d like to share some CloudKit articles I found to be incredibly useful while adding CloudKit support to Shopping UK:

• First, there’s the official Apple CloudKit documentation. It’s rather good.
• Guilherme Rambo provides a wonderful overview of CloudKit.
• Kuba Suder has a very useful CloudKit Reference and Cheat Sheet.
• Jaanus Kase has written some great blog posts on using CloudKit to build a messaging app.
• Dan Griffin has five excellent tips for sharing data with CloudKit.
• Filip Němeček has a detailed guide to Configuring Subscriptions
• Bart Jacobs has a very useful Troubleshooting Guide for when subscription notifications don’t arrive.

If you get a chance, please try Shopping UK and let me know what you think at @wheelies

This is the seventh in an eight-part series on implementing data sharing in Shopping UK using CloudKit.

Shopping UK is a smart shopping list for UK shoppers. It knows almost every product in the supermarket and will arrange them by aisle. Lists can be shared with family or friends.

Last week, we looked at how data is synchronised between devices and some of the challenges to consider. Today, we’ll look how to change or stop a share, what happens when a list is deleted and background maintenance activities.

Managing the Share as the Owner

The “owner” of the list is the person that originally shared it.

You’ll remember from Part 4 how a UICloudSharingController is used to create the initial share. Well, the same controller is also used for viewing and managing the share after it has been created.

As an owner of the list, you can:

• View the list of participants.
• Invite new participants.
• Remove participants.
• Stop sharing the list.

Inviting new participants

Only the owner of the list can invite new participants.

You can invite up to 100 people. This limit is imposed by CloudKit, but it is more than adequate for our purposes. The UICloudSharingController has options for controlling who can accept an invite (anyone with the link or named accounts), and their access permissions (read-write or read-only).

Lists in Shopping UK can only be shared with named accounts and read-write permissions — the user doesn’t get to choose. This may change in the future, but I honestly couldn’t think of a use case for allowing a read-only shopping list or anonymous access at present (perhaps you have one?).

The app doesn’t need to do anything special when a user adds a new participant. The UICloudSharingController takes care of it all:

• Generating and sending invites
• Updating the cloudkit.share record with the new participant’s details
• Updating the cloudkit.share acceptance status

When the cloudkit.share status changes to accepted, the CKRecordZone for the shared list will automatically appear in the participant’s shared CKDatabase.

Removing participants

Only the owner of the list can remove a participant, but anyone can remove themselves.

As an owner, you remove a participant using the “Remove Access” button:

Note: If there is only one participant in the list, the list will stop sharing.

The app doesn’t need to do anything special to support this action. The UICloudSharingController takes care of everything — i.e. updating the cloudkit.share record to remove the participant’s details, which will hide the list’s CKRecordZone from the participant’s shared CKDatabase.

The removed participant will see this message before the list is deleted from their device:

This is handled by subscriptions and the CKFetchDatabaseChangesOperation described in Part 3 and Part 6. The fetch operation will return a Deleted Record Zone response and the app responds by showing this message.

Note: The list data is still in iCloud, in the owner’s Private database, and on the other participants’ devices.

Stop Sharing

The “Stop Sharing” button will remove sharing for all participants and remove the data from iCloud.

When this option is used, the app needs to do some housekeeping of its own to reset the local list to be a local-only list as it was before the list was first shared.

The app must do three things:

1. Remove the locally stored CKServerChangeTokens
2. Delete the CKRecordZone that contains the list data from iCloud
3. Remove the local reference to the iCloud list.

The app is notified of the “Stop Sharing” action by implementing the cloudSharingControllerDidStopSharing method in the UICloudSharingControllerDelegate

The zone is deleted using deleteRecordZoneWithID on the CKDatabase

When the zone is deleted, the participants will each receive a message like the one seen when a single participant is removed.

Managing the Share as a Participant

As a participant, the UICloudSharingController shows a different view. There are no options for inviting others, stopping the share or removing other participants. The only option is to remove yourself from the list:

The UICloudSharingController takes care of the mechanics of this by updating the cloudkit.share record.

What happens when a list is deleted

A shared list can be deleted by either the owner or a participant. The behaviour is different for each.

Deleting a list as an owner

When the owner deletes the list, everything is removed:

1. The list data from the owner’s device.
2. The list’s CKRecordZone from the owner’s private CKDatabase in iCloud.
3. The list’s CKRecordZone from all participants’ shared CKDatabase in iCloud.
4. The list data from the devices of all participants.

The owner’s app handles tasks 1 and 2, using the “Stop Sharing” logic to remove the zone.

Task 3 happens automatically because when the owner’s private zone is removed, the shared zones disappear too.

The participant’s app handles task 4, the same way as for “Stop Sharing”.

Deleting a list as a participant

When a participant deletes a list, things are simpler, because the original list is not deleted — it remains in iCloud, and on the owner’s device, and on the devices of other participants.

Only two things need to be removed:

1. The list data from the participant’s device.
2. The cloudkit.share CKRecord from the participant’s shared CKDatabase in iCloud.

The app handles both tasks itself. The cloudkit.share is deleted like any other CKRecord, using deleteRecordWithID, which provides a simple wrapper around CKModifyRecordsOperation.

Note: after deleting the list, a participant can re-join later using the same invitation they originally received.

What happens when a user signs into or out of iCloud?

It is important to understand that when a list is first shared, the shopping list is no longer considered to be a locally hosted list that lives on the device. It is now an iCloud-hosted list and the data on the local device is just a locally cached copy.

This means when a user signs out of iCloud, all iCloud-hosted lists must be deleted from the local device. If the app didn’t do this, it would have no way to guarantee their integrity because the user no longer has access to the iCloud database that contains list changes. Furthermore, permitting continued access to the list would violate the list’s security because the user is logged out of iCloud and no longer authorised to view the list’s contents.

When a user signs-in to iCloud, the opposite happens. The app will restore a copy of all iCloud-hosted lists from the Private and Shared databases of the newly available iCloud account.

The app detects a change of account status by observing the CkAccountChanged notification.

When this notification is triggered, the app sends an accountStatus message to CloudKit to check the account status.

If the response indicates a status change from NoAccount to Available, all iCloud-hosted lists are restored to the device.

If the response indicates a status change from Available to NoAccount, all iCloud-hosted lists are removed from the device.

The account change notifications have been reliable so far. But I like to have a manual option if things go wrong. That’s why I added an option to force a re-fetch of all iCloud-hosted lists in the “Troubleshooting” section of the Help menu.

Without this option, the user would have to sign-out then sign back into iCloud to refresh, and this can be time consuming.

Background Maintenance

Once a share is established, the user need not do anything. The app will happily synchronise changes all day long.

But, behind the scenes, nothing stands still.

If you’ve been following the previous posts, you’ll recall how Shopping UK doesn’t synchronise the state of the list. Instead, it synchronises the Journal Entry records that represent the history of changes made to the list (see Part 2 and Part 6 for a refresher). And from these changes, the current list can be built.

But, you’re probably wondering what will happen after a few months, when there are thousands of JournalEntry records?

Firstly, for devices already sharing the list, it doesn’t matter how many JournalEntry records there are because, as we saw in Part 3 and Part 6, the app uses CKServerChangeTokens to only fetch new records.

But this doesn’t let us off the hook entirely.

What would happen if the journal keeps growing?

Two problems:

1. Storage and Quotas.
2. New Participants joining the list.

Every CKRecord takes up space (albeit just a few bytes), and this counts towards the share owner’s iCloud storage quota. If the number of JournalEntry records were allowed to grow indefinitely, the owner’s iCloud storage would fill up.

Secondly, if a new participant joins the list later, the size of the journal becomes relevant. Remember how, in Part 5, a CKServerChangeToken is not used when retrieving the list initially. With a large journal, it could take a considerable time to fetch data from iCloud, and to apply the changes to the view in the user interface.

Neither of these situations is good.

What’s the answer?

The approach I chose for Shopping UK was to compress the journal by removing old JournalEntry records that aren’t necessary for reconstructing the current list.

For example, the two lists below are identical but the second was built using fewer JournalEntry records:

Admittedly, by removing rows 1 and 3, we have lost some of the list’s history, but the final state of both lists is the same.

Let’s look at another example. Are these two lists equivalent?

No, not quite. The first list includes “bread” in a marked-off state, but the second doesn’t show “bread” at all.

Does this matter? Maybe not.

If “bread” was bought a long time ago, does it need to be on the list? After all, the list’s purpose is to tell us what we need to buy not what was bought.

Then again, if my wife has just bought “bread”, I’d find it useful to see it crossed-off the list, so I know it was bought and not just deleted.

From these examples, we can infer some general principles:

1. Recent items should be left alone, and
2. The full history of changes should be preserved in an Activity view.

And this is how journal compression works in Shopping UK.

Compression is never applied to the full set of JournalEntry changes, only to those created before a cut-off time.

And every change made to the list is also recorded in an Activity view that won’t be affected when the journal is compressed.

When are JournalEntry records compressed?

Compression occurs in two places:

1. Before updating the user interface
2. In iCloud, at regular intervals

Before updating the user interface

When a device requests new changes from iCloud, many JournalEntry records may be returned. Updating the user interface rapidly with so many changes can make interaction sluggish and distract the user. To prevent this, the app will compress JournalEntry records older than three hours and only apply the remaining changes to the user interface.

Practically, this means if a user on another device added and marked-off “milk” more than 3 hours ago it won’t be shown in the list (because the “add” and the “mark-off” both happened more than 3 hours ago). But if the user added and marked-off “milk” within 3 hours the item would be shown in the list like this in a marked-off state, like this: milk

In iCloud

The journal in iCloud will also be regularly compressed.

Every device that has access to a shared list is responsible for compressing the JournalEntry records in iCloud. After each iCloud upload (see Part 6), the app will check if the journal can be compressed. But, to save battery and bandwidth, this won’t happen more often than every 30 minutes.

Compression is a three-stage process:

1. Fetch eligible JournalEntry records from iCloud.
2. Attempt to compress them on the app.
3. Delete the discarded JournalEntry records from iCloud and update the CompressedUntil date on the ShoppingList record

JournalEntry records older than 14 days are considered eligible for compression.

The app uses a CKQueryOperation to fetch the eligible JournalEntry records.

If the compression logic identifies records to be deleted, a CKModifyRecordsOperation is used to remove the records from iCloud and update the CompressedUntil date.

The default SavePolicy of IfServerRecordUnchanged is used, which means the entire operation will succeed or fail atomically.

Consequences of Compression

Journal compression is a housekeeping activity, and users should mostly be oblivious to its existence. The only time the effect of compression will be surfaced to users is if their device has not been used for more than 14 days. If this happens, the app will perform a full re-fetch of the shared lists from CloudKit to ensure they are fully up-to-date.

Incidentally, if you used the app before version 3.3 was released, you may have experienced an issue that caused me no end of stress trying to reproduce and diagnose. The old, home-grown sharing mechanism used a primitive form of journalling, and didn’t have a way to detect if a device hadn’t been updated recently. Frustrated users would send me messages asking why the sync stopped working after the app hadn’t been used in a while. If you were one of the users affected by this issue, I’m truly sorry it took me so long to find and fix.

Alternatives to compression

Journal compression is a convenient way of reducing the size of the app’s write-ahead journal. It works well for a shopping list because the state of items on a list follows a predictable pattern: they are Added then Marked-Off or Deleted. And each pair of start-end states (Add then Mark-Off) and (Add then Delete) can be neatly purged without affecting other items. For other types of app, a more appropriate clean-up strategy for a write-ahead journal may involve a low-water mark of some sort.

Next week, in the final part of the series, we’ll look at what happens when things go wrong: error handling, merging data, and diagnosing problems.

If you get a chance, please try Shopping UK and let me know what you think at @wheelies

This is the sixth in an eight-part series on implementing data sharing in Shopping UK using CloudKit.

Shopping UK is a smart shopping list for UK shoppers. It knows almost every product in the supermarket and will arrange them by aisle. Lists can be shared with family or friends.

Last week, we looked at what happens when another device accepts an invitation to collaborate on a shared list. Today, we’ll look at how data is synchronised between devices and some of the challenges to consider.

What is Synchronised?

In Shopping UK, we need to synchronise data from two Record Types:

1. cloudkit.share
   This tells us who joined or left the shared list
2. JournalEntry
   This contains all changes made to the shopping list

Here’s how they are related:

ShoppingList represents the list itself but from a synchronisation perspective, it is very dull. cloudkit.share and JournalEntry are where the action happens.

What is special about cloudkit.share?

Cloudkit.share is a built-in CloudKit Record Type It represents a CKShare object. It’s interesting because it contains a list of who has accepted the share.

The app doesn’t manage the list of participants directly (this is the job of UICloudSharingController (see Part 4 and Part 5 for details).

However, the app is interested in tracking when a participant joins or leaves the list so it can update the Activity view:

Whenever a change is detected in a cloudkit.share CKRecord, the app compares the share’s participants list with the previously cached list to identify what has changed:

In this example, Charlie and Diane joined the list and Bob left.

This list of participant changes is used to populate the Activity list.

What is special about JournalEntry?

For sharing purposes, every list is represented as a set of JournalEntry records, as discussed in Part 2. JournalEntry can be used to represent every type of change made to a list, and the full list of JournalEntry records is enough to reconstruct the entire list from scratch.

For example, this list:

Is represented as seven JournalEntry CKRecords (newest entry at the top):

In eight steps, we can see how these seven JournalEntry records are applied to a blank list to recreate the full shopping list:

Once a JournalEntry has been added to iCloud, it will never be modified — it is a read-only log of changes. The only way JournalEntry records are ever removed is when the app compresses the list, which will happen regularly to prevent the list becoming large and causing performance issues. List compression will be discussed in detail in part 7.

Every newly created JournalEntry record must be synchronised between each device and iCloud.

How does Synchronisation Work?

In Part 3, we looked at how data in iCloud can be changed using CKModifyRecordsOperation.

We also looked at three ways data can be read from iCloud:

1. CKFetchRecordsOperation
2. CKQueryOperation
3. CKFetchDatabaseChangesOperation with CKFetchRecordZoneChangesOperation

These operations form the basis of synchronisation. Now let’s look at the full cycle and how each device knows when a change has been made.

Sync Step 1: Upload Changes

When a change is made to a list on one device, a record representing that change must be uploaded to iCloud as soon as possible.

The device that made the change will send a CKModifyRecordsOperation request to iCloud using CloudKit:

Sync Step 2: Notify other devices

After CloudKit has updated the iCloud database with the new entry, it sends a CKDatabaseNotification message to all other devices running the app.

Sync Step 3: Other device requests change

Upon receipt of the notification, the other device will issue a CKFetchDatabaseChangesOperation followed by a CKFetchRecordZoneChangesOperation to request a list of all changes.

Sync Step 4: iCloud sends changes

iCloud sends a list of changes made since the last request. This list will include the newly created “Add potatoes” record:

How does iCloud know which devices to notify?

Shopping UK uses CloudKit Subscriptions to inform the app when another user makes a change to a shared list.

After the app has registered a subscription with CloudKit, the app will be notified when the watched data changes.

How are subscriptions registered?

Subscriptions are managed separately for each user. They are not global — they live in the user’s iCloud account.

There are three types of subscription:

1. CKQuerySubscription
2. CKRecordZoneSubscription
3. CKDatabaseSubscription

Shopping UK uses CKDatabaseSubscription, which is used to watch changes in a specific CKDatabase and it can be further limited to specific Record Types.

During app start up (willFinishLaunchingWithOptions), Shopping UK will attempt to register four subscriptions, one for each Record Type (cloudkit.share and JournalEntry) and one for each CKDatabase (Private and Shared)

CKModifySubscriptionsOperation is used to register the four subscriptions:

1. PrivateJournalEntryChanges
2. Privatecloudkit.shareChanges
3. SharedJournalEntryChanges
4. Sharedcloudkit.shareChanges

Receiving notifications from subscriptions?

Once a subscription has been registered, iOS will call the AppDelegate’s didReceiveRemoteNotification method to tell the app when a change happens.

This notification can be received while the app is active, in the background or even when it is not running. (source)

the system calls this method when your app is running in the foreground or background. In addition, if you enabled the remote notifications background mode, the system launches your app (or wakes it from the suspended state) and puts it in the background state when a remote notification arrives

The basics:

• Call the completion handler after processing the notification.
• Return the appropriate result: NewData, NoData or Failed. Be consistent with how you use these.
• You only have 30 seconds to process the notification. Be as quick and energy efficient as possible.

Important: A notification will not always be received

A notification is not guaranteed to be sent from iCloud. iOS uses heuristics for deciding when to send a notification.

Apple doesn’t document the heuristics. Presumably because it doesn’t want developers to game the system. The best official document I found was Apple TechNote TN2265 from 2016:

your app will receive the notification if iOS or OS X determines it is energy-efficient to do so. If the energy or data budget for the device has been exceeded, your app will not receive any more notifications with the content-available key until the budget has been reset. This occurs once a day and cannot be changed by user or developer action.

iOS tracks how long your app takes to process each notification and what result was returned by your app.

Example:

iOS uses this data to decide how to handle the next notification. Should it be sent to the app at all? And, if judged worthy, how soon should it be sent?

My guess, based on a lot of testing, is iOS groups these results by type, then calculates stats on each group:

• Median time for newdata was 2750ms
• Median time for nodata was 60ms

iOS appears to use this information, and other data (battery charge, battery decrease rate, number of notifications received) to decide how much budget to give your app for future notifications.

While testing, I learnt a couple of important lessons.

Lesson 1: Always return a result

I found a bug in my app, which meant a result was not returned when the app was in the background. When this happened, CloudKit would refuse to send notifications for a few minutes. My app wasn’t playing by the rules so it was being punished.

Lesson 2: Classify results correctly

Another bug meant my app would sometimes return nodata instead of newdata, despite actually fetching and processing genuine data. This would happen about 50% of the time. I’d add an item to a list on one device but the notification wouldn’t appear on the second device. I’d add a second item and this time a notification would be received on the second device.

Because the app was incorrectly reporting work as nodata, iOS reduced the number of notifications it sent by half. From iOS’s perspective this made sense — why waste resources on notifications that take the same time to process but don’t produce useful work.

It’s all about saving battery. If an app abuses the time given to process remote notifications, iOS will grant it less time in the future. If the app fails to respond at all, iOS will penalise the app. If the app says it hasn’t done anything useful with the time it was given, iOS will grant it less time in future. This can result in notifications being skipped, combined or delayed.

The best strategy is to be efficient with resources — don’t do more than you need to when a notification arrives — and be truthful with what you did. If your app used its time to fetch new data, return newdata. If your app didn’t need to do anything, return nodata. And if your app could not complete processing for whatever reason, return failed. This will allow iOS to classify the time taken in the correct dataset so it can draw the correct conclusions.

One final tip, when testing, if your notifications arrive on some devices but not others, try:

• Charging the device. iOS will try harder to save battery when it is low, and skipping notifications is one way to do this.
• Disable “Do Not Disturb”. I saw a situation when this prevented notifications from arriving. As soon as I disabled “Do Not Disturb” notifications began to arrive again.
• Wait a while. iOS seems to reset its stats after a while. Try waiting a few minutes or even a few hours. Or try a different device.

How does the app respond to subscription notifications?

After receiving the notification, Shopping UK fetches the list of changes using CKFetchDatabaseChangesOperation and CKFetchRecordZoneChangesOperation as described in Step 3 above, and Part 3 of this series.

It may be tempting to skip the “fetch” step and use the information provided in the notification message, but this is unreliable. iOS may decide to combine or drop notifications for individual changes so an explicit fetch is required to ensure no changes are missed.

How is the list of changes processed?

After the app has made the request, CloudKit will send several responses, which may include:

1. a set of CKRecords
2. a list of deleted RecordZone.IDs,
3. a list of deleted Record.IDs

See Part 3 for full details.

When the app receives changes, it stores them locally until it is the right time to apply them to the user interface.

The “right time” depends on which list is active, and on what the user is doing.

When the app is in the “Planning” screen, the JournalEntry operations can be applied to the current list straight away. But, if the app is in the “Shopping” screen, a different approach is needed. If items were added to the visible list immediately, it would disrupt the user’s flow (imagine if a set of new items were added to your list just as you were about to mark it off. It would be annoying to accidentally tap the wrong item because the app had decided to shuffle the list items around just at that moment).

To provide a better user experience, Shopping UK displays each “Add” as a visible notification at the top of the list. Then, when the user taps the notification, the queued items will move to the right category in the list.

When an item is “Marked-off” or “Deleted”, the app handles things differently. Since the item is already in the list, there is no need to display a separate notification. Instead, the item’s table cell is updated to reflect the state change.

This behaviour makes it a delight to use Shopping UK when family and friends are shopping together. Some people like to split up in the supermarket to save time. To make this work, it is vital that changes appear quickly and are visually obvious, so the same item isn’t purchased twice.

When to synchronise?

Most of the time, synchronisation happens automatically. If a user makes a change to the list, a JournalEntry record representing the change will be immediately uploaded to iCloud. If a CKDatabaseNotification is received, which indicates someone else has made a change, the app will immediately fetch changes from iCloud.

In addition, the app will fetch when the app is launched (in case any changes were made by other participants while the app was in the background).

But this is not always enough.

To handle unforeseen situations, such as CloudKit subscriptions not arriving on time, I also added a way to do invoke the full upload-and-fetch synchronisation process manually.

This can be triggered by pulling down on the “planning” screen (shown below) or the “shopping” screen (the one with checkboxes)

Another way to invoke manual sync is the “Synchronise Now” button on the Activity sheet:

I don’t normally like adding multiple ways of doing the same thing — I’d rather ensure there is a single reliable way to accomplish something — but when there are a lot of moving parts, some of which out of my control (i.e. iCloud, database subscriptions, and the network), it is reassuring to know users have a manual alternative available if needed.

Next week, we’ll look at how a share is changed or stopped, what happens when a list is deleted, and background maintenance.

If you get a chance, please try Shopping UK and let me know what you think at @wheelies

This is the fifth in an eight-part series on implementing data sharing in Shopping UK using CloudKit.

Shopping UK is a smart shopping list for UK shoppers. It knows almost every product in the supermarket and will arrange them by aisle. Lists can be shared with family or friends.

Last week, we looked at how sharing works and how to start sharing the list. Today, we’ll look at what happens when another device accepts an invitation to collaborate on a shared list.

Accepting a share

A sharing invitation is sent from the owner to one or more participants, typically via Messages or Email. The “owner” is the account that initiates the sharing.

The invitation itself is just a link to iCloud, like this:

https://www.icloud.com/share/0123456789abcdefghijklmno#Shopping

The fragment (i.e. the part after the #) is the title of the share, in this case, “Shopping”.

Step 1: User clicks sharing invitation link

When a user clicks on the invitation link, iOS first checks if the app is installed, and it is right version to accept shares.

The CKSharingSupported key in info.plist indicates this.

Before releasing v3.3 (the first version of Shopping UK to support CloudKit sharing), I released v3.2, which had sharing enabled. The idea was to improve the adoption of sharing. When v3.3 was published, all existing v3.2 users could accept shares immediately without having to first update their app.

Once the system completes its checks, the user sees a message:

Step 2: App Informs CloudKit that share has been accepted

After the user accepts the link, iOS calls userDidAcceptCloudKitShareWith.

A CKShare.Metadata object is provided as a parameter. This has everything the app needs to find the shared data in iCloud.

To accept the share, Shopping UK sends a CKAcceptSharesOperation message to the identified CKContainer.

CloudKit will change the participant’s status from Pending to Accepted.

Note: Before calling CKAcceptSharesOperation, Shopping UK checks the list hasn’t already been added to the device. If it already exists, the shopping list is opened in the app.

Step 3: App creates a new list and assigns the sharing details

The app creates a new empty list on the device using the info in CKShare.Metadata:

Why is the Database set to “Shared”?

This confused me initially, but this is what I learned:

When Alice shares a list, it is uploaded to her Private CKDatabase.

But, when Bob accepts the invitation to join Alice’s list, the same records are visible to him in his Shared Database. Both accounts can make changes to the underlying data via their respective database: Alice via her Private database and Bob via his Shared database.

Data appears in a different database depending on whether your iCloud account is the owner or a participant.

Step 4: App fetches data from iCloud and applies it to the device

Next, the app fetches a list of changes using Fetch Changes (described in Part 3). But the CKServerChangeToken is ignored to ensure all changes are fetched.

As changes are received, the app applies them to the new list then shows the list to the user.

Next week, we’ll take a closer look at how data is synchronised between devices and some of the challenges to consider.

If you get a chance, please try Shopping UK and let me know what you think at @wheelies

This is the fourth in an eight-part series on implementing data sharing in Shopping UK using CloudKit.

Shopping UK is a smart shopping list for UK shoppers. It knows almost every product in the supermarket and will arrange them by aisle. Lists can be shared with family or friends.

Last week, we looked at the three ways records can be retrieved from iCloud, and how the data can be added and changed. Today, we’ll look at how sharing works and how to initiate the sharing of a list.

Where does the data live?

Before we look at sharing, let’s review how data is segregated in iCloud.

Every iCloud account sees three separate CKDatabases:

When the app sends a CKModifyRecordsOperation to iCloud to add data, it chooses which of the three CKDatabases to use.

Public and Private databases are easy to understand:

• Data added to Public can be viewed or modified by anyone (access can be restricted to only users signed into iCloud, or made read-only for everyone except the creator)

• Data added to Private can only be seen and changed by the iCloud account holder. No-one else, not even app developers like me, can see your Private database.

The Shared database is more interesting because it doesn’t really contain data. It simply exposes it. Think of it like a database view.

What happens when a list is shared?

There are three key activities involved in sharing a list:

1. Preparing the data to be shared. For Shopping UK, the data includes the list’s items (e.g. “bread”, “milk”) and the attributes of the list itself (name, colour). The list will already exist on the “owner’s” device, but it must be uploaded to iCloud before it is available for other people.
2. Sending the sharing invitation. The invitation will be sent from the owner’s device to other people who will become “participants” in the share.
3. Accepting the invite. The other participants must open and explicitly accept the share before data can be exchanged.

Once setup, all changes to the shared list will be synchronised between devices.

Next week, in part 5, we’ll see how a sharing invitation is accepted. Later, in part 6, we’ll look at how data is synchronised between devices.

Today, we’ll look at the first two activities: preparing the data and sending the invite.

Step 1: User requests the list to be shared

List sharing begins when the user chooses the “Share Changes to List” option from the menu:

Step 2: The app checks the iCloud account and list state

Before data is exchanged, the app checks:

1. An iCloud account is available.
2. The list is not already being shared.

A message will be shown if either check fails.

The iCloud status is checked using accountStatusWithCompletionHandler available on the CKContainer.

The app knows the list is shared if a ShareId is set in the locally cached list record on the device. Later, we’ll see how this is set.

Step 3: The app creates a controller for managing the sharing activity

The app creates a UICloudSharingController to manage the interaction with the user. It is possible to manage the interaction with custom code but the UICloudSharingController’s user interface is familiar, widely used in Apple’s own apps, and it works well.

A UICloudSharingControllerDelegate is used with the UICloudSharingController to provide a title and thumbnail image . These are shown in the top-left of the sharing sheet and in the invitation message we’ll see in Step 6.

Step 4: PreparationHandler

The PreparationHandler is called when the user selects the method for sending data (e.g. Message, Email, AirDrop, WhatsApp).

The PreparationHandler is responsible for adding the shared records to iCloud.

Shopping UK first creates a new CKRecordZone in the Private database using SaveRecordZone.

The new CKRecordZone name is derived from the identity of the list (e.g. “List-db6a6e29-…”).

By placing all records for a shopping list in a dedicated CKRecordZone , it will be easier to delete the data atomically. We’ll revisit this later, in part 7, when we explore what happens when the user stops sharing a list or signs out of iCloud.

Next, the CKRecordZone will be populated with the records that represent the list.

Step 5: Upload Records

After the CKRecordZone has been successfully created, the app issues a CKModifyRecordsOperation to CloudKit:

Only two CKRecords are included in the upload message:

• A ShoppingList CKRecord. This represents the shopping list itself, but it doesn’t contain any information about the list’s Name, Colour, or Items. These will be sent later as JournalEntry CKRecords

• A CKShare. This is a special type of CKRecord (Record Type: cloudkit.share), which identifies the CKRecords to expose.

The CKShare acts as a window through which other share participants can view the shared records. There are two ways to use it:

1. Share a CKRecordZone
2. Share a root CKRecord (and its children)

Option 1: Share a CKRecordZone

If the CKShare is created with a CKRecordZone, the zone and its contents will all be shared.

Option 2: Share a root CKRecord (and its children)

If the CKShare is created using a CKRecord, this root record and its children (and its children’s children, etc. — all the way to the bottom of the hierarchy) will be shared.

In Shopping UK, there is a dedicated CKRecordZone for every shopping list, which means both options are available. But I chose to share via a root record (Option 2) rather than by zone (Option 1) because I thought it may be useful in the future to have records present in the zone that are not part of the share.

As well as defining the scope of visibility, the CKShare is also responsible for recording the owner of the shared records and the acceptance status of all participants. The “Owner” is the iCloud account of the person who initiated the share.

What happens to the list’s items?

So far, we’ve uploaded a list placeholder CKRecord and the CKShare record, but we haven’t uploaded the most important bit — the contents of the list — “bread” and “milk”.

I excluded the list’s items from the initial payload because I encountered problems when sharing very large lists (with several hundred items).

I discovered the upload would take a long time and trigger a timeout. No error was reported but the UICloudSharingController would give up without creating the share. This could be reproduced consistently, and the controller would close after about 10 seconds.

To fix this, I changed strategy to avoid uploading the items upfront. Instead, the items are now added to an internal upload queue, which is processed after sharing has completed (as part of Step 7 below).

This has a nice consequence — simpler logic. The logic to upload list items after the share has been initialised is now the same as the logic to upload list items during normal synchronisation (which will be discussed later, in part 6).

Once the data has uploaded, the app calls the PreparationHandler’s completion block to tell the UICloudSharingController to continue with the invitation workflow.

Step 6: Sending the invitation

After the PreparationHandler’s completion block has receives the success message from the app, it launches the chosen method of sharing (e.g. Messages)

The user will choose the message’s recipients in the usual way, add additional message contents if they choose, and hit the “Send” button.

Step 7: Finishing up

Once the invitation message has been sent, the controller calls the delegate’s cloudSharingControllerDidSaveShare method.

Note: I found the cloudSharingControllerDidSaveShare method was sometimes called more than once. This seems to happen when sharing is initiated multiple times in the same session, as if the event handler is being wired-up multiple times and never released. I don’t know if this is something I have done wrong, or a bug in the framework (it’s most likely to be something I’ve done), but I had to work around this by ensuring all operations are idempotent.

Now the data is safely in iCloud, the next task is for the app to link the on-device list to the list in iCloud.

The iCloud information can be read from the UICloudSharingController’s Share property, which contains the new CKShare created in step 5.

The following properties are sufficient to locate the iCloud data:

CloudKitDatabase  = "private"
CloudKitZoneName  = share.Id.ZoneId.ZoneName
CloudKitZoneOwner = share.Id.ZoneId.OwnerName
CloudKitShareId   = share.Id.RecordName

Finally, the user interface is refreshed so the user knows the list is being shared. The app also triggers an upload of the JournalEntry items (“bread”, “milk”) from the internal upload queue (as described in step 5).

Now sharing is complete:

Next week, we’ll see what happens on the other side, when another device accepts the share.

If you get a chance, please try Shopping UK and let me know what you think at @wheelies

This is the third in an eight-part series on implementing data sharing in Shopping UK using CloudKit.

Shopping UK is a smart shopping list for UK shoppers. It knows almost every product in the supermarket and will arrange them by aisle. Lists can be shared with family or friends.

Last week, we looked at CloudKit concepts and the design of the Shopping UK schema. Today, we’ll look at how CKRecords are uploaded to iCloud using CloudKit; and the three ways the app can fetch CKRecords.

Adding and Changing Data

To create, modify or delete a CKRecord in iCloud, the app must send a CKModifyRecordsOperation message to CloudKit.

Shopping UK uses CKModifyRecordsOperation to upload new JournalEntry CKRecords to iCloud so they can be fetched by other devices that share the shopping list.

Let’s look at what happens when a user adds two new items to the list.

Step 1. User adds items to the shared list

There are three ways a user can add items to the list:

1. Start typing and choose a suggestion.
2. Copy items from another list.
3. Paste from another app, such as Notes, using the clipboard.

It doesn’t matter how the user added “milk” and “bread” to their shopping list, the way they are uploaded to iCloud via CloudKit is the same.

Step 2. App saves changes to local journal

Shopping UK maintains a local journal of changes. All changes are first written to this journal before the app sends them to iCloud.

This step is not essential for uploading data to iCloud using CloudKit but, if your app must guarantee changes will be uploaded, it is wise to first store them on the device, even if only temporarily, before uploading the changes to iCloud.

By doing this, we can be sure nothing will be lost if the network is not available. We’ll examine how things can fail in depth in part 8.

Step 3. App sends CKRecords to iCloud using CloudKit

The app creates a CKModifyRecordsOperation message and attaches the two CKRecords that represent the changes from the local journal.

Note: The app can set a Quality of Service (QoS) value for the CKModifyRecordsOperation. A lower level QoS is used for operations that are not time critical. iOS may delay these operations when the device is in low power mode or low on battery.

Shopping UK uses the User-Initiated level for sending and fetching JournalEntry records. This provides nearly instantaneous messaging (a few seconds or less).

Step 4. CloudKit sends response back to app

CloudKit adds the CKRecords to the CKDatabase and returns three response messages.

All CloudKit operations are asynchronous and typically follow this pattern:

• The app sends the initial message to CloudKit
• CloudKit will return a response message for each completed unit (i.e. two responses, one for each CKRecord)
• CloudKit will return another response message when the entire operation is complete.

The happy path is shown here — when everything goes well — but, due to the nature of networks and remote systems, things can go wrong. In part 8, we will revisit the error scenarios and how Shopping UK handles each one.

Fetching Data

We looked at uploading new CKRecords to iCloud, but how is data fetched?

There are three options:

Shopping UK uses all three techniques in different places.

Fetch By Name is used to:

1. Retrieve a CKShare record to allow the user to manage the shared list (more on this in part 4 and part 7)
2. Retrieve the current ShoppingList record for compression (more on this in part 7)

Fetch By Query is used to fetch JournalEntry records eligible for compression (more on this in part 7).

Fetch Changes is the most interesting of the three. It uses a CKServerChangeToken to identify each version in the database’s history, which means the app can request just the records that changed since its last request. This is much more efficient than fetching everything every time, and it is more reliable than creating a custom mechanism for tracking changes.

For my first attempt at implementing sharing using CloudKit, before I understood how Fetch Changes worked, I tried to build my own change tracking mechanism using timestamps. I spent a long time on this dead-end, but it didn’t work well due to the precision of timestamps, and tiny differences in the clock setting of each device

Shopping UK uses Fetch Changes to retrieve changes made by other devices participating in a shared list.

Step 1. App sends a Database Changes request to CloudKit

Assume two CKRecords already exist in iCloud and this is the first time the white iPhone is requesting changes.

Since this is the first time data will be fetched, the device won’t have a Database CKServerChangeToken so it will send null for the token’s value. This instructs CloudKit to return all changes ever made.

Step 2. CloudKit returns changed Record Zones

CloudKit will return three response messages:

1. A Change Token Updated response containing the updated Database CKServerChangeToken (e.g. ABC123)
2. A single Changed Record Zone response (because both CKRecords are contained within a single CKRecordZone: abcd1234-…)
3. A final Changes Completed response

If multiple zones had changed, there would be a Changed Record Zone response for each zone that contained a change.

If any zones had been deleted, an additional Deleted Record Zone response would have been returned too.

Step 3. App saves database change token

The database change token is stored so it can be used next time CKFetchDatabaseChangesOperation is sent.

Step 4. App requests changes in each zone

Now we have a list of CKRecordZones that changed, but we don’t yet know which CKRecords. To find these, the app issues a CKFetchRecordZoneChangesOperation.

Since this is the first time this has been issued, the app doesn’t yet have a Zone CKServerChangeToken so it sends null.

Note: the Database CKServerChangeToken returned in step 2 is not the same as the Zone CKServerChangeToken

Step 5. CloudKit responds with changed records

CloudKit will send a separate Record Changed for every record added or modified since the Zone CKServerChangeToken. Because this is the first time this operation has been used, and the token was set to null, all records (milk and bread) will be returned.

A single Fetch Completed response will be sent at the end.

If a record had been deleted since the last fetch, a Record Id Deleted response would also be received by the app.

When the fetch operation is complete, a Fetch Completed response will be received by the app. This will contain the Zone CKServerChangeToken.

Step 6. App updates its local list and stores the Zone Change Token

The app uses the received CKRecords to update its local representation of the list, and it updates user interface with the new items.

The app stores the CKServerChangeToken for zone “abcd1234-…” on the device. This will be used next time CKFetchRecordZoneChangesOperation is used to fetch changes for the same zone.

CloudKit’s Fetch Changes mechanism provides an efficient way of synchronising the app’s local cache of the iCloud data. Instead of fetching everything every time, only the recent changes need to be fetched.

In part 6 we’ll look at how the Fetch Changes request can be triggered: when the app receives a remote notification from a CloudKit Subscription or when the user explicitly triggers the list synchronisation.

And, in part 7, we’ll look at situations when the app must fetch everything, such as when the user signs into a new iCloud account.

Next week, we’ll see how sharing works, and how to share a list for the first time.

If you get a chance, please try Shopping UK and let me know what you think at @wheelies

This is the second in an eight-part series on implementing data sharing in Shopping UK using CloudKit.

Shopping UK is a smart shopping list for UK shoppers. It knows almost every product in the supermarket and will arrange them by aisle. Lists can be shared with family or friends.

Last week, we introduced Shopping UK and discussed limitations of its old home-grown data sharing solution. Today, we’ll look at CloudKit’s basic concepts and how the Shopping UK schema is designed.

Concepts

We’ll start with a quick review of the core CloudKit concepts.

If you’re already familiar with the concepts, please skip to the “Creating a CloudKit Schema” section below, which explains how CloudKit is used in Shopping UK.

CloudKit is one of Apple’s key technologies for sharing data in the cloud. Apple offers three options for data sharing:

1. iCloud Document Storage for file or document synchronisation.
2. iCloud Key-Value Storage for small key-value pairs.
3. CloudKit for sharing complex objects.

CloudKit is a way to move structured data between your app and iCloud. It also has a web-based dashboard for viewing and manipulating the data in your own iCloud account (this is really useful for testing).

CloudKit is free to use. It is natively supported in iOS and it also has a JavaScript API so it can also be used from web sites and outside the Apple ecosystem.

Let’s look at three key concepts: Records, Record Types and Record Zones.

Records

Data is stored in a CKRecord. Think of these like records in a relational database system.

Each CKRecord has a Record Type — a template for defining what values the CKRecord can contain. A Record Type has a name and a list of Keys. Each key is assigned a data-type (e.g String, Date, Bool).

Think of the Record Type like a table in a relational database system, and the Keys as fields.

Record Types

To store books by an author, we might create two Record Types: Book and Author. A Book has a Title, a PublishedOn date, a number of Pages, an ISBN number and a link to an Author:

Together, the Record Type describe the “shape” of the data and how it links together — its schema. The data itself is represented by a CKRecord. This example uses a couple of my books from one of my favourite childhood authors, Roald Dahl:

Record Zones

A CKRecordZone is a grouping of CKRecords.

All CKRecords live in one and only one CKRecordZone.

A CKRecordZone lives in a CKDatabase. There are three databases: Public, Private and Shared (more on these later).

The Public and Private CKDatabases each have a Default CKRecordZone.

New, custom CKRecordZones can be created in the Private and Shared databases (but not in the Public database).

A CKDatabase lives in a CKContainer. There is typically only one CKContainer for the whole app.

Summary of Concepts

This shows how these concepts relate to each other:

Creating a CloudKit Schema

At first glance, the data structure for a shopping list would be similar to the Book/Author example above, with a parent “Shopping List” record and a set of child “ShoppingItem” records:

This is the way most tutorials approach the problem, and this is similar to the way Shopping UK stores data internally, but it isn’t the optimal structure for sharing a set of items on a list.

It is not enough to represent only the current state of the list. We need to see some history too, so we can answer questions like:

• I added ‘milk’ yesterday, but it has disappeared. Is there a glitch in the app or did my wife remove it from the list?
• I added ‘bread’ last week, but it is still on the list. Did my wife buy some and have we run out already?
• When did we last buy coffee?

So instead of transferring a snapshot of the current list, Shopping UK transfers the ordered list of changes that have been applied to the list.

This “journal-based” approach is similar to how a relational database works internally (see write-ahead logging). It is a well-established pattern, but it isn’t appropriate for all apps. If you have an app that doesn’t need to track every change, a simple state-based sharing model, like the one shown earlier, would probably be simpler to implement.

When all changes have been applied to another device, the lists will become the same.

The journal-based approach has a couple of useful consequences:

• Items don’t have an absolute position in the list. Instead, they know which item they follow. This means if my wife and I are both adding items to the list we won’t overwrite each other’s changes. Instead, each item will be added in turn.

• Changes can be queued rather than being applied immediately. This is important when the app is in “Shopping” mode. It would be annoying if the app added an item to my checklist while I was shopping in the supermarket. It could cause me to mark-off the wrong item. Instead, the app will show the newly added items as notifications, and I can tap on them when I’m ready to include them in my checklist:

You’re probably wondering what happens after the list has been shared for several weeks and the journal contains thousands of changes. We’ll look at how the journal is compressed in a later post.

Next week, we’ll look at the lifecycle of a CKRecord: how they are created, read, modified and deleted.

If you get a chance, please try Shopping UK and let me know what you think at @wheelies

I’m also running an A/B test of a new paywall.

When the app first starts, each user is randomly assigned a group: “A” or “B”, and this group stays with the app until it’s uninstalled from the device.

Users in “Group A” will see the old paywall and those in “Group B” will see the new one.

Using AppCenter, I’ll track how many times each paywall is viewed, and App Store Connect will show the number of sales for each.

I’m hoping the new paywall — with its clearer design, single call-to-action, and a free trial period — will outperform the old one.

I’m excited to find out if I’m right. But, nothing is certain and, to mudddy the water, the price is higher for Group B. Ideally, I should have only changed a single variable (the appearance or the price) but I’m keen to experiment with both variables so I took the risk of muddying the results in the hope of getting answers to both questions sooner.

(Day 30 of 87)

If you get a chance, please try Shopping UK and let me know what you think at @wheelies

But for anything other than a trivial app this is impossible.

Everybody is different. Some know their devices well, grasp concepts quickly and use your app all the time. These people need the least help. But many people are occasional users, less willing to experiment, and they like to know there’s somewhere to look when they get stuck.

For Shopping UK, my aim has been to keep the user interface simple and use iOS idioms and well-known UX patterns where possible. But I also offer in-app help pages to explain tasks and answer common questions because some users need more hand-holding, and some parts of the app are harder to understand.

I also provide the same content on the web because some users like to search Google for answers.

Many apps provide help in both places — inside the app and on the web — using a WkWebView control in their app to point at the help pages on their public web server.

This works well. Help pages can be published once yet still be corrected or improved later.

But there are downsides. Help is not available when the network is offline, and pages may be slow to load because they are fetched from a remote web server over a high latency or limited bandwidth connection.

For Shopping UK, I wanted help pages to appear instantly. And this meant bundling the help pages inside the app.

How does this work?

All help pages are written using Markdown.

This is converted to HTML using the static site generator Jekyll.

All Markdown files are stored in GitHub. When I commit a change, a GitHub Action runs to publish the help pages to an Azure Static Web App (this only costs pennies to run). You can view the live help pages here.

I use a slightly elongated process when publishing the same content to the app.

Jekyll takes the same Markdown source and converts it into HTML, but this is then passed through a custom script.

The script does two things:

1. Flattens the hierarchy of pages.
2. Creates a file containing navigation data.

I flatten the folder hierarchy and place all files and images in a single resource folder in the app because it makes it easier to link between pages without having to worry about absolute and relative paths.

And the navigation data file is needed so the app can present the help navigation in a UITableView, which is a lot easier to navigate than a HTML page full of hyperlinks.

Here’s the navigation data file that powers the UITableView:

Here’s the full process for producing the same content on the web and inside the app:

This approach won’t work for all apps. It won’t work if your help content changes frequently and outside the app’s release cycle. But, if you can use it, the user will appreciate how fast they can find answers when they have the least patience — when they need help.

(Day 28 of 87)

If you get a chance, please try Shopping UK and let me know what you think at @wheelies

• Part 1: Introduction: About Shopping UK and why its home-grown data sharing solution needed to be replaced. (1st May 2023)
• Part 2: What is CloudKit?: A review of CloudKit concepts and how the Shopping UK schema was designed. (8th May 2023)
• Part 3: Adding, Updating and Deleting Records: How to change data in iCloud and the different options for retrieving it. (15th May 2023)
• Part 4: Sharing Concepts and How to Begin Sharing: Introduces sharing concepts and how to begin sharing a list. (22nd May 2023)
• Part 5: Accepting a Sharing Invitation: How another device can accept an invitation to collaborate on a shared list. (29th May 2023)
• Part 6: Synchronising Data: How data is synchronised between devices and some of the challenges to consider. (5th June 2023)
• Part 7: Managing the Share and Background Maintenance: How to change or stop a share, what happens when a list is deleted, background maintenance. (12th June 2023)
• Part 8: When Things Go Wrong: Error handling, merging data, and diagnosing problems. (19th June 2023)

If you get a chance, please try Shopping UK and let me know what you think at @wheelies