Panorama and document stitching are both techniques for merging multiple images to create a single one. However, there are some key differences between them. Panorama stitching typically requires sphere projection, which means that the images are warped to fit onto a sphere. This is because panoramas are typically used to capture a wide view of a scene, and sphere projection ensures that the image will be seamless and distortion-free. Document stitching, on the other hand, does not require sphere projection because documents are typically flat objects, and there is no need to warp the images to fit onto a sphere. The other key difference is that you almost always know what objects you are stitching — usually, it is a black-and-white document in the center of the photo. As a result, document stitching can be a simpler process that often produces better quality results but has some specific. This article addresses the issue of resolving computer vision problems with the experience of a mobile developer and provides a good flow to start. On the other hand, I believe that for an appropriate domain expert, this article may seem a bit basic.

As a mobile developer, with the help of document stitching, you can significantly enhance certain app experiences and provide impressive “magic” for your users — like scanning a long receipt, for example. So let’s define the task that we are going to resolve as “stitching multiple photos of a long receipt into a single image”.

Here Comes OpenCV

OpenCV (Open Source Computer Vision Library) is a powerful library of programming functions mainly used for real-time computer vision. It can be utilised for a wide variety of computer vision tasks and is available for various platforms. This means that you can use it to stitch images together on any platform you need — build a prototype in Jupyter Notebook, which is a popular environment for interactive computing; integrate it in iOS with C++ implementation, or in Android with OpenCV Kotlin. In this article, I am going to add C++ code snippets and attach a Jupyter Notebook at the end for you to experiment with.

If we are talking about moving forward with OpenCV, I am going to introduce you the first concept — cv::Mat. Mat is a basic data structure in OpenCV C++ that represents a matrix of pixels. It is used to store images, image channels, and other multidimensional data. So please consider that when I refer to images in OpenCV, I am talking about Mat, not iOS UIImage. However, do not worry, as OpenCV 3 supports converting UIImage to Mat and Mat to UIImage out of the box with import of opencv2/imgcodecs/ios.h

Another important note is that OpenCV has a built-in stitcher with “SCANS” mode which you can use with:

Stitcher::create(cv::Stitcher::SCANS)

It provides some opportunities for customisation, but from my point of view, it has significant disadvantages. The major issue with it is the “lack of control” — you just give it the vector<Mat> (array of images) and it returns the stitched Mat as an output. It can fail with a limited amount of errors such as “Need More Images”, “Homography Fail”, and “Camera Params Adjust Failed”. But actually, you did not really know what went wrong. Even worse is that the chances of encountering “Out of Memory” are quite high. So, my point is “If you are working on a production solution, you should be in control of your code”. That is why I decompose the process into steps with an understanding of what each step does and why I need it.

Stitching Process Steps

1. Image preprocessing
2. Features detection
3. Matches finding
4. Homography finding
5. Images stitching with homography applying

Some of these terms may be unfamiliar. I am going to describe each step in simple language and provide more detail about what it does and why it is important.

Step 1. Image Preprocessing

The crucial step to achieve good results and avoid “Out of Memory” is image preprocessing. Image preprocessing in this context means image downscaling. But I put it in a separate step because it is not so simple. Image downscaling helps to achieve two important things here — reduce image size and get rid of possible noise on document photos. Otherwise, some random pixels that are produced by photo camera noise can participate in stitching and affect the result.

The key point here is that we should use “INTER_AREA” interpolation (bilinear interpolation) in:

cv::resize(image, output, new_dimensions, cv::INTER_AREA)

INTER_AREA is a resampling method used to resize images. It works by averaging the pixels in the source image to create new pixels in the destination image while preserving the aspect ratio of the image. This is important for document images as it ensures that the text in the image will not be blurry or pixelated. It also means that resized images will not contain noise because noise pixels are averaged out.

Step 2. Features Detection

The next step is to find “features” on images. A feature is a point in an image that has a unique appearance. Features are often used for image matching, object recognition, and other computer vision tasks.

We use the SIFT feature detection algorithm, where SIFT stands for Scale-Invariant Feature Transform. It works by first detecting interest points in an image. Interest points are points in an image that are likely to be features, such as corners or blobs. Once the interest points have been detected, they are then described using a SIFT descriptor. For any object in an image, interesting points on the object can be extracted to provide a “feature description” of the object. This description, extracted from an image, can then be used to identify the object when attempting to locate the object in a test image containing many other objects.

To perform robust recognition, it is important that the features extracted from the image be detectable even under changes in image scale, noise, and illumination. Such points usually lie on high-contrast regions of the image, such as object edges.

SIFT can robustly identify objects even among clutter and under partial occlusion, as the SIFT feature descriptor is invariant to uniform scaling, orientation, rotation, illumination changes, and partially invariant to affine distortion. Another important characteristic of these features is that the relative positions between them in the original scene shouldn’t change from one image to another. For example, if only the four corners of a door were used as features, they would work regardless of the door’s position; but the recognition would fail if the door became opened. The interesting thing about the keypoint locations search is that the algorithm starts with an area of 4×4 pixels. That’s why image downscale was an important step in preprocessing to avoid dealing with high-res images and positively affects the quality, speed and memory consumption.

Looks good, but…

One of the challenges of document images stitching is that the background of an image can also contain valid features. For example, if a document is being photographed on a textured table, the table texture could also be detected as features. This can lead to problems, as the algorithm may try to stitch the images together based on the background features, rather than the document features.

One way to address this problem is to use a mask to restrict the area of the image where features are detected. A mask is a binary image that defines the region of interest in the original image. Only pixels in the region of interest are considered when detecting features. OpenCV’s SIFT detectAndCompute allows you to pass a mask to the function. This will ensure that only features in the region of interest are detected.

To find a mask for a document image, you can use the following two steps: convert the image to grayscale and apply a threshold to the grayscale image to create a binary image with:

cv::threshold(image, mask, threshold, 255, cv::THRESH_BINARY)

You can start with threshold equal to 127. 127 is used to create a binary image that represents the region of interest in the original image. In this case, the region of interest is the document itself. A threshold value of 127 means that any pixel that is brighter than the threshold will be set to white, and any pixel that is darker than the threshold will be set to black. You can experiment with different threshold values to find what works best for you. I finished with threshold = 150.

Step 3. Matches Finding

After SIFT features are extracted from the two images, the next step is to find matches between the features in the two images. This is done using a Brute-Force Matcher. The Brute-Force Matcher compares each feature in the first image to all of the features in the second image. The matches with the highest scores are considered to be the most likely matches.

It uses function:

brute_force_matcher.knnMatch(image1_desc, image2_desc, matches, 2) 

This function takes four arguments: the descriptors from the first image, the descriptors from the second image, the matches result pointer as input, and the number of nearest neighbors to consider. The function provides a list of matches, where each match is a tuple of two elements: the index of the feature in the first image and the index of the feature in the second image. The parametr “2” in the knnMatch function call specifies that the function should consider the two nearest neighbors for each feature. This is done because the SIFT descriptor is a 128-dimensional vector, and it is possible that two features in different images could have similar descriptors. By considering the two nearest neighbors, the Brute-Force Matcher can reduce the number of false matches. The matches with the highest scores are considered to be the most likely matches and can then be used to stitch the two images together.

After finding all of the matches between the two images, the next step is to filter the matches a little bit more. This is done by using a ratio test. The ratio test compares the distance between the two nearest neighbors for each match. If the distance between the first nearest neighbor is less than a certain threshold times the distance between the second nearest neighbour, then the match is considered to be a good match. The ratio test is a simple but effective way to filter out false matches which passed knnMatch. The threshold value can be adjusted to control the number of good matches that are found. A higher threshold value will result in fewer good matches being found, but it will also reduce the number of false matches. I finished with threshold = 0.5.

Now we are ready to use the good matches to stitch the two images together.

Step 4. Homography Finding

Once the good matches have been found, the next step is to find the homography between the two images. Homography is a 3x3 matrix that maps points from one image to another. It can be used to stitch images together, warp images, and solve other problems in computer vision.

The homography can be found using a robust estimator called RANSAC. RANSAC is a RANdom SAmple Consensus algorithm that is used to find the best-fit model to a set of data points. RANSAC works by iteratively sampling a subset of the data points and fitting a model to the subset. The model that best fits the subset is then used to estimate the homography. This process is repeated a number of times, and the homography that is estimated most often is the best estimate of the true homography.

RANSAC is a powerful tool for finding the homography between two images. It is robust to outliers and can be used to stitch images together even in the presence of noise and errors.

As always the OpenCV function provides us an ability to set the threshold for RANSAC. The RANSAC threshold is a critical parameter, as it affects the accuracy of the homography estimation. A lower threshold will result in more inliers being included in the homography estimation, but it will also increase the chances of including outliers. A higher threshold will result in fewer inliers being included in the homography estimation, but it will also decrease the chances of including outliers.

The optimal RANSAC threshold value depends on the data set and the application. A good way to find the optimal threshold value is to experiment with different values and see which value results in the best homography estimation.

Step 5. Images Stitching with Homography Applying

The homography can be used to stitch the two images together by warping the second image to the first one. Operations I implemented in C++ code are — finding the frame size of a new stitched image with a correction vector (the correction vector is used to offset the second image when it is warped to the first), warping the second image to the first one, and copying one image onto another.

The code for calculating the new frame size first reads the size of the secondary image. Then, it creates a matrix that contains the coordinates of the corners of the secondary image. This matrix is then multiplied by the homography matrix to find the final coordinates of the corners of the image after transformation. The correction vector is used to offset the secondary image so that it is stitched to the first image in the correct location. The correction factor is calculated by finding the minimum and maximum values of the x and y coordinates of the corners of the image after transformation. If the minimum value is negative, then the correction factor is set to the absolute value of the minimum value.

The code for calculating the new frame size is a critical step in the image stitching process. The correct calculation of the new frame size ensures that the secondary image is stitched to the first image in the correct location. The correction vector is also important, as it ensures that the secondary image is not offset outside of the frame.

Job almost done — get the warped secondary image and copy it on the stitched frame. That is it…

One more thing, let’s address the black areas that appear when the second image is overlaid on the previous one. There are a few techniques for doing this, the most common of which is known as alpha blending. However, I prefer a simpler approach. I preferred to copy pixel information from the background image if the stitched image has a black pixel.

The technique of alpha blending is a more sophisticated way of overlaying two images. It allows for the blending of the two images so that the edges of the stitched image are more seamless. However, alpha blending can be more computationally expensive than the simpler approach of copying pixel information from the background image.

To implement “copy pixel” approach, we need to do the following. After warping the second image onto the first using the homography matrix, we can identify these black pixels by checking if their color values are (0, 0, 0) in the RGB color space. Once we find a black pixel, we can simply copy the corresponding pixel from the background image onto the stitched image to fill in the black area. It’s important to note that while this approach is simpler and computationally less expensive, it may not provide the same level of visual quality as alpha blending. Depending on the application and the level of seamlessness required in the final image, you may choose between the two techniques.

The final result:

Jupiter Notebook:

Jupyter-Stitching-OpenCV/Stitching-OpenCV.ipynb at master · bigMOTOR/Jupyter-Stitching-OpenCV

Used Links:

• Scale-invariant feature transform - Wikipedia
• Random sample consensus - Wikipedia

Let’s talk about app architecture and the approach I apply as an iOS software engineer in a few companies. My team and I were trying to build something solid without slipping into a dense swamp where following the rules distracts you from actual business domain code. As a result, we got something that works for us and good enough to be told from my point of view.

Previously I wrote a short article on how iOS engineers can adopt Clean Architecture in SwiftUI world. After that, I realized that not many developers understood the ideas and abstractions I was talking about. So I decided to write this as a prequel to explain how I apply Clean Architecture in my everyday UIKit world.

The architecture of an entire application is the most important thing you should care about if you are building a solid, reliable, scalable product. Nowadays, you will be asked about MVC, MVVM in almost every job interview or VIPER, and RIBs if an interviewer added some creativity to a century-old checklist. However, here’s the problem — most of these cover the presentation layer and leave us alone when we step out from “presentation” boundaries. But can we build some abstractions that will work on every layer and across multiple applications no matter what?

My answer is yes. This recipe includes Clean Architecture, MVVM, RxSwift, and Dependency Injection. The glue how we put it together is the Composition. Composition is one of the essential things that help developers deal with difficulty and requirements mutation. As long as you make things composable, your abstractions will face all bumps and turns.

Let’s take a look at a simple example — the user screen. This screen should display the current user name and have a “Sign Out” button. On the domain level, we have an entity — User, which contains all user info.

struct User {
  let id: String
  let firstName: String
  let lastName: String
  let permissions: [Permission]
}

As I mentioned before, we already have a bunch of well-known patterns for the presentation layer. I consider myself a big FRP (Functional Reactive Programming) fan, so my choice was quite obvious — MVVM with RxSwift.

struct UserScreenViewModel {
  let name: Observable<String>
  let signOut: () -> Void
  ...
}

It was the easiest part. That is why I don’t want to talk about a presentation based on MVVM-RxSwift. There are a lot of good articles about it, while I prefer to stay focused on the upcoming part. This is the part I’d like to really talk about and explain it as clearly as possible. Sorry for my drawings.

Thus we have ViewModel on the one side and some source of our User on the other. How will we connect them? The nasty dirty hack that comes to mind is putting all Stores, Managers, and Repositories we need into the initializer. Oo

init(userStore: SomeUserStore, locationManager: SomeLocationManager, …, justAnotherManager: BullshitManager)

But how it makes things composable? Where should I put a clean User’s domain code? How to provide the granularity of operations what this particular object can do? How to reuse it in some other place?

Instead, we aimed to decompose our application into multiple “pieces” and abstract them one from another with low coupling inspired by Clean Architecture idea. Clean Architecture, someone calls it Onion Architecture, is a decomposition of your application into layers and unidirectional data flow. In this article, I am talking about my real-life implementation and our example. So we put away digging deep into Clean Architecture itself.

We are going to abstract all the things our ViewModel needs into small pieces called UseCases. UseCase is a protocol with one particular function. It will provide us granularity for operations we are about to give our ViewModel and precise semantics because we can easily understand what this specific model can do at one glance.

You can see that you can compose as many UseCases as you like and anywhere you need. The initializer, just waiting for the object confirming them. We will return to that object a little bit later. Now I’d like to focus on our business domain code. In UseCases, we deal with a tiny decomposed piece of our program and don’t care about the rest of the world. In other words, we can do some Protocol Oriented Programming and define the following statement: “Any object which has Repository A can do something.”

It’s cool. We don’t care where the data are coming from or what purpose they serve. We are just dealing with a very tiny problem that can easily fit in our minds. That’s a UseCase.

Let’s transform the diagram above into something closer to our app.

And here it comes, the Context. Context is the “bridge” between all the stuff that our ViewModels require and managers, services, repositories, etc. Usually, each screen has its Context, which constructs from a Dependency Injection container. The primary idea here is that Context doesn’t have any code. It holds the repository properties and gets all the code it needs via the composition of UseCases.

That’s it. We achieved very low coupling and made things composable we are prepared for new requirements by adding new repositories mechanism and UseCase composition. We can write UnitTests with mocked UseCases or Repositories. We can even rearrange our ViewControllers, and as long as we have the required objects for its initialization in Dependency Container, they will be instantiated.

Furthermore, to put everything together, I have a straightforward repository which I am using from time to time on accidentally workshops for newcomers or some other occasions. The “master” branch is a starting point from where I explain everything I said above. After that, we connect the new repository, and the result is “workshops/ComposableContextsArchitecture” branch. I hope it helps.

GitHub - bigMOTOR/CaseStudies

Thank you for your time!

In this article, I’d like to introduce our approach to deal with Table and Collection views quick and easy way. To be short I am going to talk about UITableView, but all of this works for UICollectionView as well. As the result, we will be able to build TableView as easily as tableView.rx.bind(sections: sections).disposed(by: bag) for any table with any cells inside. It is all the code that you need in your UIController.

Cool, isn’t it? Let’s get started.

The only notice that I have to say that all of this based on RxSwift. In case you are not familiar with this, I highly recommend taking a look. Otherwise, it could be a little bit difficult to catch.

The standard approach to managing table views with data sources has several flaws:

• Repeated boilerplate code — data source and delegate methods, cell registration, etc.
• Tangled follow the flow of control for TableView data source and delegate methods since they are often placed in a different order, far from each other, or even located in different files.
• Violation of dependency inversion principle. The knowledge about which cells are attached to a table view and how these cells are instantiated (nib or class) leaks to corresponding view controllers. View controller becomes dependent on the module of the lower level (tableview cell).
• Leaves lots of room for mistakes, since data source methods must be consistent with each other. For example, if numberOfRows(inSection:), numberOfSections(in:) and tableView(_,cellForRowAt:) are inconsistent, it results in an unwanted behaviour or even crash. Cells in these methods is a generic UITableViewCell type which usually should be type casted to a concrete class.
• Table/collection view data source protocol implementation is imperative and does not feel Swift way.

Since the moment I got carried away with RxSwift, I prefer to build code that is driving by events and has no room for numberOfRows(inSection:). I used to feel very frustrated when I found myself creating just another table for my app. Strong Rx community attempted to build tables (remember, when I am talking about tables, I am talking about collections as well) rx-way with RxDataSources.

This library helped us to solve some of “standard approach” problems with an elegant data binding mechanism and powerful AnimatableSectionModel & SectionModel abstractions but still left us on our own with repeatable cell registration and violation of dependency inversion principle.

After building a few tableviews with RxDataSources you maybe noticed some repeatable stuff that is screaming about being encapsulated. We made this logical step and made our own DataDrivenRxDatasources library which brings another level of the abstractions and lets us address these issues and design a data-driven, reusable, and declarative table view component which depends only on its ViewModel.

The core of our library is based on abstracted protocols that are supposed to provide binding sections and cells but don’t care what these sections and cells really are. Here is the example of a cell that assigns some value and provides xib to register itself:

The cell has to adopt the ModelledCell protocol that’s it. Additionally, we have some other helpers protocols such as Selectable, Editable, etc which allows us to provide selection or “swipe to delete” actions.

ViewController’s view model takes some reactive data provider and transforms its events (data) to cell view models wrapped into sections. Here is the word “DataDriven” stands.

As you could notice we can mix different cells as long as they are satisfied ModelledCell protocol without touching ViewController’s code.

Summary

Loving this level of abstraction we are successfully applying DataDrivenRxDatasources in our projects and production for multiple companies. I wrote this because I’d like to introduce our library to the community, maybe you will like it too and find a place to apply it in your projects or if you have some ideas about improving it ⏤ welcome to join, PRs are welcome. I believe it worth some audience.

We placed our code on GitHub with available integrations via Swift Package Manager and Cocoapods. Also we included examples for an easy start. I strongly believe that it will help you to build tables and collections with less code easily way.

bigMOTOR/DataDrivenRxDatasources

I was building an iOS app with a reactive approach before it became mainstream. UI is the function from data, unidirectional data flow, Onion Architecture, and all other words that used to drive some of my colleagues crazy. 2019 has changed this. SwiftUI with Combine are powerful tools that Apple brought to us and nowadays more and more developers find themselves on this road.

I don’t want to dive into a discussion about the Clean Architecture, reactive approach, or MVVM — all of this just works for me. Maybe it will work for you too, maybe won’t. But we should talk in case we’re in the same boat.

As early adopters of new technologies each of us faced some issues walking on the uncharted road, but exactly this feeling makes the journey more enjoyable. This article is not a silver bullet but a step towards the discussion about where I am now and a hope that as a strong community we will find a good approach together. Let’s get started.

The idea of Clean Architecture is dividing the software into layers. (Clean Architecture by Robert C. Martin aka Uncle Bob). In case you are not familiar with this, I highly recommend taking a look. Otherwise, it will be a little bit difficult to catch all of the abstractions I will talk about.

View Model takes the composition of use cases for initialization. Use Case ⏤ is a protocol with only one concrete purpose method (e.g. observe user, save the document, etc); works with Domain Entities. Repositories are the implementation of Use Cases, operating with Platform Entities.

Usually, it is convenient to implement the Use Cases in Protocol Oriented way and don’t care what actual implementation of repositories are. Whatever responsible for store the document will be able to save it.

It works for me in the UIKit world with the stack: MVVM-RxSwift-DIP. DIP is a simple Dependency Injection Container, it provides rich functionality for DI containers such as auto-wiring, scopes, storyboards integration, etc. All of this helps me to be focused on a particular piece of responsibility and don’t care about the big picture.

Since SwiftUI rolled out I’d like to adopt this approach and keep all its benefits. But what is different in the SwiftUI world?

In addition to a bunch of different changes, the existence of Environment property wrappers was introduced. It allows us to inject arbitrary values into the environment so we don’t need storyboard integration anymore. It sounds like I don’t need third-party DI at all. Let’s try to take advantage of this.

We will define AppEnvironment, which will hold DIContainer with repositories as a property and make this DIContainer EnvironmentKey to get its value whenever we need it.

The implementation of Repositories is just a basic example here. You free to go implement it in any way to achieve weak retaining to destroy the instances you don’t need and all other improvements.

Next step, I’d like to generalize the boilerplate of initialization of ViewModel and subsequent initialization of View with this ViewModel something like this:

Now the creation of the View body looks short, easy to read, and brings nothing annoying to our sight. Cool.

Summary

As I said before this article isn’t a silver bullet with a unique approach rather an invite to the discussion in attempting to build a clear approach to build SwiftUI views. Because all I saw before looks quite messy, usually limited with Single State App approach and violate the separation of concern principle sometimes.

I put my code on GitHub. So if you have some ideas about improving it ⏤ welcome to join.

bigMOTOR/swiftui-clean-architecture

I like the power of strong typing which Swift language provides for us as the developers. Bring it together with a functional reactive programming approach helps me to build responsive iOS applications for different business needs. RxSwift used to help me with it, but after Combine was introduced I decided to dig deep into it and write a new pet project within it.

Combine is a newborn baby and I miss some tools I used to have in RxSwift. What if we have an events stream which can throw ‘nil’ values:

But on the other hand, we won’t deal with ‘nil’ in some method: func doSomething(_ value: Int) Of course, we can use filter or flatMap or whatever:.filter { $0 != nil } But what type will we get downstream? Anyway, we should deal with an Optional Type. The knowledge of "not possible nil in there" forces us to use some technics, which I don't really like in my code.

The strong RxSwift Community gives us an extension RxOptional, which allows us to filter ‘nil’ values and unwrapped type in the downstream. Inspired by this extension I wrote the similar for Combine and made it available via Swift Package.

It’s simple but allows me to avoid copy-paste across all my projects. It’s my first article on Medium, but I think that it can be useful for Combine’s newcomers. Here is my CombineOptional package. Welcome to join if you need something like that.

And when I have finished this article some guy reminded me about compactMap… 🤦‍♂️😊

https://github.com/bigMOTOR/CombineOptional