Panorama and document stitching are both techniques for merging multiple images to create a single one. However, there are some key differences between them. Panorama stitching often uses spherical projection, which means that the images are warped to fit onto a sphere. This is useful for capturing a wide view of a scene, although the projection alone does not guarantee a seamless or distortion-free result. Document stitching, on the other hand, does not require sphere projection because documents are typically flat objects, and there is no need to warp the images to fit onto a sphere. The other key difference is that you almost always know what objects you are stitching — usually, it is a black-and-white document in the center of the photo. As a result, document stitching can be a simpler process that often produces better quality results, but it has its own challenges. This article addresses the issue of resolving computer vision problems with the experience of a mobile developer and provides a good flow to start. On the other hand, I believe that for an appropriate domain expert, this article may seem a bit basic.

As a mobile developer, with the help of document stitching, you can significantly enhance certain app experiences and provide impressive “magic” for your users — like scanning a long receipt, for example. So let’s define the task that we are going to resolve as “stitching multiple photos of a long receipt into a single image”.

Here Comes OpenCV

OpenCV (Open Source Computer Vision Library) is a powerful library of programming functions mainly used for real-time computer vision. It can be utilised for a wide variety of computer vision tasks and is available for various platforms. This means that you can use it to stitch images together on any platform you need — build a prototype in Jupyter Notebook, which is a popular environment for interactive computing; integrate it in iOS with C++ implementation, or in Android with OpenCV Kotlin. In this article, I am going to add C++ code snippets and attach a Jupyter Notebook at the end for you to experiment with.

If we are talking about moving forward with OpenCV, I am going to introduce you to the first concept— cv::Mat. Mat is a basic data structure in OpenCV C++ that represents a matrix of pixels. It is used to store images, image channels, and other multidimensional data. So please consider that when I refer to images in OpenCV, I am talking about Mat, not iOS UIImage. However, do not worry, as OpenCV 3 supports converting UIImage to Mat and Mat to UIImage out of the box with import of opencv2/imgcodecs/ios.h

Another important note is that OpenCV has a built-in stitcher with “SCANS” mode which you can use with:

Stitcher::create(cv::Stitcher::SCANS)

It provides some opportunities for customisation, but from my point of view, it has significant disadvantages. The major issue with it is the “lack of control” — you just give it the vector<Mat> (array of images) and it returns the stitched Mat as an output. It can fail with a limited amount of errors such as “Need More Images”, “Homography Fail”, and “Camera Params Adjust Failed”. But these errors do not always tell you what went wrong. Even worse is that the chances of encountering “Out of Memory” are quite high. So, my point is “If you are working on a production solution, you should be in control of your code”. That is why I decompose the process into steps with an understanding of what each step does and why I need it.

Stitching Process Steps

1. Image preprocessing
2. Feature detection
3. Feature matching
4. Homography estimation
5. Image stitching using a homography

Some of these terms may be unfamiliar. I am going to describe each step in simple language and provide more detail about what it does and why it is important.

Step 1. Image Preprocessing

The crucial step to achieve good results and avoid “Out of Memory” is image preprocessing. Image preprocessing in this context means image downscaling. But I put it in a separate step because it is not so simple. Image downscaling helps to achieve two important things here — reduce image size and suppress some of the noise in document photos. Otherwise, some random pixels that are produced by photo camera noise can participate in stitching and affect the result.

The key point here is that we should use “INTER_AREA” interpolation (pixel-area resampling) in:

cv::resize(image, output, new_dimensions, 0.0, 0.0, cv::INTER_AREA);

INTER_AREA resamples an image using pixel-area relationships and is generally a good choice for downscaling. To preserve the aspect ratio, calculate the target dimensions using the same scale factor for width and height. Downscaling can suppress some fine-grained noise, but reducing the image too much can also remove small text and details needed for feature matching.

Step 2. Feature detection

The next step is to find “features” on images. A feature is a point in an image that has a unique appearance. Features are often used for image matching, object recognition, and other computer vision tasks.

We use the SIFT feature detection algorithm, where SIFT stands for Scale-Invariant Feature Transform. It works by first detecting interest points in an image. Interest points are points in an image that are likely to be features, such as corners or blobs. Once the interest points have been detected, they are then described using a SIFT descriptor. For any object in an image, interesting points on the object can be extracted to provide a “feature description” of the object. This description, extracted from an image, can then be used to identify the object when attempting to locate the object in a test image containing many other objects.

To perform robust recognition, it is important that the features extracted from the image be detectable even under changes in image scale, noise, and illumination. Such points usually lie on high-contrast regions of the image, such as object edges.

SIFT can help identify objects even among clutter and under partial occlusion. Its features are designed to handle changes in scale and rotation, with some robustness to illumination changes and moderate viewpoint changes. Another important characteristic of these features is that the relative positions between them in the original scene shouldn’t change from one image to another. For example, if only the four corners of a door were used as features, they would work regardless of the door’s position; but the recognition would fail if the door became opened. The interesting thing here is that SIFT searches for keypoints across multiple scales. The 4×4 grid belongs to the descriptor: it contains 16 spatial cells with eight orientation bins each, producing 128 values. It is not a fixed 4×4-pixel search window. Downscaling helps us avoid processing unnecessarily large images, improving speed and memory consumption, but we still need enough detail for reliable feature matching.

Looks good, but…

One of the challenges of document images stitching is that the background of an image can also contain valid features. For example, if a document is being photographed on a textured table, the table texture could also be detected as features. This can lead to problems, as the algorithm may try to stitch the images together based on the background features, rather than the document features.

One way to address this problem is to use a mask to restrict the area of the image where features are detected. A mask is a binary image that defines the region of interest in the original image. Only pixels in the region of interest are considered when detecting features. OpenCV’s SIFT detectAndCompute allows you to pass a mask to the function. This will ensure that only features in the region of interest are detected.

To find a mask for a document image, you can use the following two steps: convert the image to grayscale and apply a threshold to the grayscale image to create a binary image with:

cv::threshold(image, mask, threshold, 255, cv::THRESH_BINARY)

You can start with a threshold of 127. Any pixel brighter than the threshold will be set to white, and the others will be set to black. Here, we use this as a simple way to select bright paper against a darker background. It does not detect the document itself: shadows can exclude parts of the paper, and a bright background can remain in the mask. You can experiment with different threshold values to find what works best for you. I finished with threshold = 150.

Step 3. Feature matching

After SIFT features are extracted from the two images, the next step is to find matches between the features in the two images. This is done using a Brute-Force Matcher. The Brute-Force Matcher compares each feature in the first image to all of the features in the second image. For SIFT descriptors compared using L2 distance, the matches with the smallest distances are considered to be the closest matches.

It uses function:

brute_force_matcher.knnMatch(image1_desc, image2_desc, matches, 2) 

This function takes four arguments: the descriptors from the first image, the descriptors from the second image, an output container for the matches, and the number of nearest neighbors to find. In C++, the output is a vector of vectors of DMatch objects. Each DMatch contains the descriptor indices and the distance between the descriptors. The parameter “2” in the knnMatch function call asks for the two nearest neighbors for each descriptor. Comparing these two candidates lets us check whether the closest match is clearly better than the alternative. This is what the ratio test below does; requesting two neighbors does not itself remove false matches. Before applying the test, we should check that two candidates were actually returned. The closest matches are candidates for stitching, but we still need to filter out ambiguous matches.

After finding all of the matches between the two images, the next step is to filter the matches a little bit more. This is done by using a ratio test. The ratio test compares the distance between the two nearest neighbors for each match. If the distance between the first nearest neighbor is less than a certain threshold times the distance between the second nearest neighbour, then the match is considered to be a good match. The ratio test is a simple but effective way to filter out false matches which passed knnMatch. The threshold value can be adjusted to control the number of good matches that are found. A lower threshold makes the test stricter and accepts fewer matches. A higher threshold accepts more matches, potentially including more false matches. I finished with threshold = 0.5.

Now we are ready to use the good matches to stitch the two images together.

Step 4. Homography estimation

Once the good matches have been found, the next step is to find the homography between the two images. Homography is a 3x3 matrix that maps points from one image to another. It can be used to stitch images together, warp images, and solve other problems in computer vision.

The homography can be found using a robust estimator called RANSAC. RANSAC stands for RANdom SAmple Consensus. It repeatedly selects a small random subset of point correspondences and estimates a homography from it. Each candidate is then checked against the other correspondences to see how many agree with it within a reprojection-error threshold. The candidate with the strongest consensus is selected and refined using its inliers.

RANSAC is a powerful tool for finding the homography between two images. It is robust to outliers and can be used to stitch images together even in the presence of noise and errors.

As always, the OpenCV function lets us set the threshold for RANSAC. This threshold controls how closely a correspondence must agree with the estimated homography to count as an inlier. A lower threshold is stricter; a higher threshold tolerates larger errors, but can also admit incorrect matches. When the point coordinates are measured in pixels, the threshold is also in pixels, at the resolution used for matching.

The optimal RANSAC threshold value depends on the data set and the application. A good way to find the optimal threshold value is to experiment with different values and see which value results in the best homography estimation.

Step 5. Image stitching using a homography

The homography can be used to stitch the two images together by warping the second image to the first one. Operations I implemented in C++ code are — finding the frame size of a new stitched image with a correction vector (the correction vector is used to offset the second image when it is warped to the first), warping the second image to the first one, and copying one image onto another.

To calculate the new frame size, we first transform the corners of the secondary image into the first image’s coordinate system using the homography. The transformed coordinates must be converted from homogeneous coordinates back to image coordinates. We then find bounds that include both the transformed corners and the first image.

If the minimum x or y coordinate is negative, the correction vector shifts the images by the corresponding amount so that they fit inside the frame. We need to account for this same shift when placing the first image and warping the secondary image.

The code for calculating the new frame size is a critical step in the image stitching process. The correct calculation of the new frame size ensures that the secondary image is stitched to the first image in the correct location. The correction vector is also important, as it ensures that the secondary image is not offset outside of the frame.

The job is almost done — get the warped secondary image and copy it on the stitched frame. That is it…

One more thing, let’s address the black areas that appear when the second image is overlaid on the previous one. There are a few techniques for doing this, the most common of which is known as alpha blending. However, I prefer a simpler approach. I preferred to copy pixel information from the background image if the stitched image has a black pixel.

The technique of alpha blending is a more sophisticated way of overlaying two images. It allows for the blending of the two images so that the edges of the stitched image are more seamless. However, alpha blending can be more computationally expensive than the simpler approach of copying pixel information from the background image.

To implement “copy pixel” approach, we need to do the following. After warping the second image onto the first using the homography matrix, we can identify these black pixels by checking if their color values are (0, 0, 0) in the RGB color space. Once we find a black pixel, we can simply copy the corresponding pixel from the background image onto the stitched image to fill in the black area. There is a catch with this simple approach: a black pixel can be part of the document itself, including its text, rather than an empty area left by warping. A safer improvement would be to warp a separate validity mask alongside the image and use it to identify empty areas. Blending is a separate choice for the overlapping areas where both images contain valid content.

The final result:

Jupyter Notebook:

Jupyter-Stitching-OpenCV/Stitching-OpenCV.ipynb at master · bigMOTOR/Jupyter-Stitching-OpenCV

References:

• Scale-invariant feature transform - Wikipedia
• Random sample consensus - Wikipedia

Let’s talk about app architecture and the approach I apply as an iOS software engineer in a few companies. My team and I were trying to build something solid without slipping into a dense swamp where following the rules distracts you from actual business domain code. As a result, we got something that works for us and is worth sharing, in my view.

Previously I wrote a short article on how iOS engineers can adopt Clean Architecture in the SwiftUI world. After that, I realized that not many developers understood the ideas and abstractions I was talking about. So I decided to write this as a prequel to explain how I apply Clean Architecture in my everyday UIKit world.

The architecture of an entire application is the most important thing you should care about if you are building a solid, reliable, scalable product. Nowadays, you will be asked about MVC, MVVM in almost every job interview or VIPER, and RIBs if an interviewer added some creativity to a century-old checklist. However, here’s the problem — most of these cover the presentation layer and leave us alone when we step out from “presentation” boundaries. But can we build some abstractions that will work on every layer and across multiple applications no matter what?

My answer is yes. This recipe includes Clean Architecture, MVVM, RxSwift, and Dependency Injection. The glue that holds it together is composition. Composition is one of the essential things that help developers deal with complexity and changing requirements. As long as you make things composable, your abstractions will face all bumps and turns.

Let’s take a look at a simple example — the user screen. This screen should display the current user name and have a “Sign Out” button. On the domain level, we have an entity — User, which contains all user info.

struct User {
  let id: String
  let firstName: String
  let lastName: String
  let permissions: [Permission]
}

As I mentioned before, we already have a bunch of well-known patterns for the presentation layer. I consider myself a big FRP (Functional Reactive Programming) fan, so my choice was quite obvious — MVVM with RxSwift.

struct UserScreenViewModel {
  let name: Observable<String>
  let signOut: () -> Void
  ...
}

It was the easiest part. That is why I don’t want to talk about a presentation based on MVVM-RxSwift. There are a lot of good articles about it, while I prefer to stay focused on the upcoming part. This is the part I’d like to really talk about and explain it as clearly as possible. Sorry for my drawings.

Thus we have ViewModel on the one side and some source of our User on the other. How will we connect them? The nasty dirty hack that comes to mind is putting all Stores, Managers, and Repositories we need into the initializer. Oo

init(userStore: SomeUserStore, locationManager: SomeLocationManager, …, justAnotherManager: BullshitManager)

But how does this make things composable? Where should I put a clean User’s domain code? How do we control exactly which operations this particular object can perform? How to reuse it in some other place?

Instead, we aimed to decompose our application into multiple “pieces” and abstract them from one another with low coupling, inspired by the idea of Clean Architecture. Clean Architecture, like the related Onion Architecture, separates an application into layers, with source-code dependencies pointing inward toward business rules. In this article, I am talking about my real-life implementation and our example. So we put away digging deep into Clean Architecture itself.

We are going to abstract all the things our ViewModel needs into small pieces called UseCases. In this approach, each UseCase is a protocol with one particular function. It will provide us granularity for operations we are about to give our ViewModel and precise semantics because we can easily understand what this specific model can do at one glance.

You can see that you can compose as many UseCases as you like and anywhere you need. The initializer just expects an object conforming to those protocols. We will return to that object a little bit later. Now I’d like to focus on our business domain code. In UseCases, we deal with a tiny decomposed piece of our program and don’t care about the rest of the world. In other words, we can do some Protocol Oriented Programming and define the following statement: “Any object which has Repository A can do something.”

It’s cool. We don’t care where the data are coming from or what purpose they serve. We are just dealing with a very tiny problem that can easily fit in our minds. That’s a UseCase.

Let’s transform the diagram above into something closer to our app.

And here it comes, the Context. Context is the “bridge” between all the stuff that our ViewModels require and managers, services, repositories, etc. Usually, each screen has its Context, which is constructed using a Dependency Injection container. > The primary idea here is that Context doesn’t implement business logic of its own. It holds the repository properties and gets all the code it needs via the composition of UseCases.

That’s it. We achieved very low coupling and made things composable. We can adapt to new requirements by adding repositories and composing UseCases. We can write UnitTests with mocked UseCases or Repositories. We can even rearrange our ViewControllers, and as long as we have the required objects for its initialization in Dependency Container, they will be instantiated.

Furthermore, to put everything together, I have a straightforward repository that I use for occasional workshops for newcomers and on other occasions. The “master” branch is a starting point from where I explain everything I said above. After that, we connect the new repository, and the result is in the “workshops/ComposableContextsArchitecture” branch. I hope it helps.

GitHub - bigMOTOR/CaseStudies

Thank you for your time!

In this article, I’d like to introduce our approach to deal with Table and Collection views quick and easy way. To be short I am going to talk about UITableView, but all of this works for UICollectionView as well. As the result, we will be able to build TableView as easily as tableView.rx.bind(sections: sections).disposed(by: bag) for any table with any cells inside. It is all the code that you need in your UIController.

Cool, isn’t it? Let’s get started.

The only notice that I have to say that all of this based on RxSwift. In case you are not familiar with this, I highly recommend taking a look. Otherwise, it could be a little bit difficult to catch.

The standard approach to managing table views with data sources has several flaws:

• Repeated boilerplate code — data source and delegate methods, cell registration, etc.
• Tangled follow the flow of control for TableView data source and delegate methods since they are often placed in a different order, far from each other, or even located in different files.
• Violation of dependency inversion principle. The knowledge about which cells are attached to a table view and how these cells are instantiated (nib or class) leaks to corresponding view controllers. View controller becomes dependent on the module of the lower level (tableview cell).
• Leaves lots of room for mistakes, since data source methods must be consistent with each other. For example, if numberOfRows(inSection:), numberOfSections(in:) and tableView(_,cellForRowAt:) are inconsistent, it results in an unwanted behaviour or even crash. Cells in these methods is a generic UITableViewCell type which usually should be type casted to a concrete class.
• Table/collection view data source protocol implementation is imperative and does not feel Swift way.

Since the moment I got carried away with RxSwift, I prefer to build code that is driving by events and has no room for numberOfRows(inSection:). I used to feel very frustrated when I found myself creating just another table for my app. Strong Rx community attempted to build tables (remember, when I am talking about tables, I am talking about collections as well) rx-way with RxDataSources.

This library helped us to solve some of “standard approach” problems with an elegant data binding mechanism and powerful AnimatableSectionModel & SectionModel abstractions but still left us on our own with repeatable cell registration and violation of dependency inversion principle.

After building a few tableviews with RxDataSources you maybe noticed some repeatable stuff that is screaming about being encapsulated. We made this logical step and made our own DataDrivenRxDatasources library which brings another level of the abstractions and lets us address these issues and design a data-driven, reusable, and declarative table view component which depends only on its ViewModel.

The core of our library is based on abstracted protocols that are supposed to provide binding sections and cells but don’t care what these sections and cells really are. Here is the example of a cell that assigns some value and provides xib to register itself:

The cell has to adopt the ModelledCell protocol that’s it. Additionally, we have some other helpers protocols such as Selectable, Editable, etc which allows us to provide selection or “swipe to delete” actions.

ViewController’s view model takes some reactive data provider and transforms its events (data) to cell view models wrapped into sections. Here is the word “DataDriven” stands.

As you could notice we can mix different cells as long as they are satisfied ModelledCell protocol without touching ViewController’s code.

Summary

Loving this level of abstraction we are successfully applying DataDrivenRxDatasources in our projects and production for multiple companies. I wrote this because I’d like to introduce our library to the community, maybe you will like it too and find a place to apply it in your projects or if you have some ideas about improving it ⏤ welcome to join, PRs are welcome. I believe it worth some audience.

We placed our code on GitHub with available integrations via Swift Package Manager and Cocoapods. Also we included examples for an easy start. I strongly believe that it will help you to build tables and collections with less code easily way.

bigMOTOR/DataDrivenRxDatasources

I was building an iOS app with a reactive approach before it became mainstream. UI is the function from data, unidirectional data flow, Onion Architecture, and all other words that used to drive some of my colleagues crazy. 2019 has changed this. SwiftUI with Combine are powerful tools that Apple brought to us and nowadays more and more developers find themselves on this road.

I don’t want to dive into a discussion about the Clean Architecture, reactive approach, or MVVM — all of this just works for me. Maybe it will work for you too, maybe won’t. But we should talk in case we’re in the same boat.

As early adopters of new technologies each of us faced some issues walking on the uncharted road, but exactly this feeling makes the journey more enjoyable. This article is not a silver bullet but a step towards the discussion about where I am now and a hope that as a strong community we will find a good approach together. Let’s get started.

The idea of Clean Architecture is dividing the software into layers. (Clean Architecture by Robert C. Martin aka Uncle Bob). In case you are not familiar with this, I highly recommend taking a look. Otherwise, it will be a little bit difficult to catch all of the abstractions I will talk about.

View Model takes the composition of use cases for initialization. Use Case ⏤ is a protocol with only one concrete purpose method (e.g. observe user, save the document, etc); works with Domain Entities. Repositories are the implementation of Use Cases, operating with Platform Entities.

Usually, it is convenient to implement the Use Cases in Protocol Oriented way and don’t care what actual implementation of repositories are. Whatever responsible for store the document will be able to save it.

It works for me in the UIKit world with the stack: MVVM-RxSwift-DIP. DIP is a simple Dependency Injection Container, it provides rich functionality for DI containers such as auto-wiring, scopes, storyboards integration, etc. All of this helps me to be focused on a particular piece of responsibility and don’t care about the big picture.

Since SwiftUI rolled out I’d like to adopt this approach and keep all its benefits. But what is different in the SwiftUI world?

In addition to a bunch of different changes, the existence of Environment property wrappers was introduced. It allows us to inject arbitrary values into the environment so we don’t need storyboard integration anymore. It sounds like I don’t need third-party DI at all. Let’s try to take advantage of this.

We will define AppEnvironment, which will hold DIContainer with repositories as a property and make this DIContainer EnvironmentKey to get its value whenever we need it.

The implementation of Repositories is just a basic example here. You free to go implement it in any way to achieve weak retaining to destroy the instances you don’t need and all other improvements.

Next step, I’d like to generalize the boilerplate of initialization of ViewModel and subsequent initialization of View with this ViewModel something like this:

Now the creation of the View body looks short, easy to read, and brings nothing annoying to our sight. Cool.

Summary

As I said before this article isn’t a silver bullet with a unique approach rather an invite to the discussion in attempting to build a clear approach to build SwiftUI views. Because all I saw before looks quite messy, usually limited with Single State App approach and violate the separation of concern principle sometimes.

I put my code on GitHub. So if you have some ideas about improving it ⏤ welcome to join.

bigMOTOR/swiftui-clean-architecture

I like the power of strong typing which Swift language provides for us as the developers. Bring it together with a functional reactive programming approach helps me to build responsive iOS applications for different business needs. RxSwift used to help me with it, but after Combine was introduced I decided to dig deep into it and write a new pet project within it.

Combine is a newborn baby and I miss some tools I used to have in RxSwift. What if we have an events stream which can throw ‘nil’ values:

But on the other hand, we won’t deal with ‘nil’ in some method: func doSomething(_ value: Int) Of course, we can use filter or flatMap or whatever:.filter { $0 != nil } But what type will we get downstream? Anyway, we should deal with an Optional Type. The knowledge of "not possible nil in there" forces us to use some technics, which I don't really like in my code.

The strong RxSwift Community gives us an extension RxOptional, which allows us to filter ‘nil’ values and unwrapped type in the downstream. Inspired by this extension I wrote the similar for Combine and made it available via Swift Package.

It’s simple but allows me to avoid copy-paste across all my projects. It’s my first article on Medium, but I think that it can be useful for Combine’s newcomers. Here is my CombineOptional package. Welcome to join if you need something like that.

And when I have finished this article some guy reminded me about compactMap… 🤦‍♂️😊

https://github.com/bigMOTOR/CombineOptional