This post is from the multi-part series on ARKit, where we talk about designing for AR, building a demo app, exploring and testing many of the features of ARKit. We previously wrote on designing 3D models for AR apps .

Introduction

AR is the core technology behind amazing apps such as Pokémon Go, Snapchat’s animated emojis, and Instagram’s 3D stickers. Apple’s announcement of ARKit at WWDC in 2017 has already resulted in some impressive software, with a mix of fun and practical apps providing something for everyone. We wanted to have the opportunity to play around with it and see what incredible things we could build with it.

Over the past year Novoda have been investigating the features of ARKit, seeing what we could built and what were the limitations of the technology, we had tons of fun building things with it and wanted to share some of our findings.

Setting a house as a hat is best way to test location placement, they say

We will be using a custom 3D model created in the design part of this series for this demo. Even if you cannot create your own custom model, you could use the simple AR cube that Apple provides or download a model from SketchUp or Google’s Poly

The first thing to understand is how AR perceives the world through the device camera: it translates the camera input into a scene composed of planes, light sources, a virtual camera, and Feature Points.

ARKit recognizes notable features in the scene image, tracks differences in the positions of those features across video frames, and compares that information with motion sensing data. The result is a high-precision model of the device’s position and motion that also analyzes and understands the contents of a scene.

If you want a more in depth analysis I highly recommend you read this page About Augmented Reality by Apple or watch their WWDC 2017 talk on ARKit. I would aslo recommend to watch Understanding ARKit Tracking and Detection talk and ARKit2 video from WWDC 2018.

How a model with planes and light source looks on Xcode. This will be added to an AR scene

With this World Tracking and Plane Detection ARKit is able to create Feature Points, Feature Points are used in ARKit to place models on the scene and to have the models anchored to their “surroundings”. As Apple explains:

These points represent notable features detected in the camera image. Their positions in 3D world coordinate space are extrapolated as part of the image analysis that ARKit performs in order to accurately track the device’s position, orientation, and movement. Taken together, these points loosely correlate to the contours of real-world objects in view of the camera.

Using ARView and ARSCNView

To build the AR app we followed a series of tutorials AppCode ARKit Introduction, AppCoda ARKit with 3D objects, Pusher building AR with ARKit and MarkDaws AR by Example, as well as the documentation on AR classes Apple provides. Since most of the basic setup has already been covered by Apple and by other tutorials we will not post all the code here, just go through some of the logic, issues and solutions we found along the way. All the source code for this and all the following posts related to this project can be found on our GitHub.

The first decision to make when creating an ARKit project is whether to use a standard one-view app template or the AR template Apple provides. We have tried both and found little difference when it came to simple apps/demos. The AR template is set up to use storyboards and has a pre-configured ARSCNView with a model of a plane. If you like playing around with working code before you write your own, we would recommend the AR template, especially as it comes with some clear explanatory comments. Alternatively, if you like having control of every piece of code it is obviously better to start from scratch. For this demo we used the template and storyboards but even if you create the project from scratch you should be able to follow along.

There are some key points every AR app needs:

• You will need an ARSCNView. Most people name their instance sceneView. This is where all the AR magic happens. You can set it to occupy the whole screen or simply as a part of the UI.

• You need to implement the ARSCNViewDelegate protocol which includes the methods used to render the models into the View. The sceneView controller will implement this protocol and be the delegate of the View.

sceneView.delegate = self

• ARConfiguration needs to be set up with the type of plane tracking you want (horizontal is the default) and then added to the sceneView session run() method to actually start the AR scene. ARSession
• On viewWillDisappear we pause the sceneView session to stop the world tracking and device motion tracking the phone performs while AR is running. This allows the device to free up resources.

This is the basic configuration you need for every AR scene. None of this code will add any AR object just yet though, only set up the view.

Apple’s pre-made template then sets up a scene directly by initialising it with a model asset at launch. That is straightforward and works well if you simply want to open the app and have a model appear in the scene. If you want to let the user choose where to place the object (for example by tapping) then you’ll need to put in a little more work.

Before we move forward I highly recommend you add this to the viewDidLoad method of your view controller:

Enabling these options will allow you to see the recognised Feature Points and the XYZ axes of the AR scene. If there is any bug with your model these features are one of the few ways you can debug AR. We’ll dig deeper into how you can test and debug AR and ML applications in an upcoming article of this series.

With Feature points options on debug enabled you are able to see what ARKit is recognising as you move around your plane aka yellow dots

Now for the fun part: adding your 3D model to the sceneView! Instead of creating a scene with an asset you can create a SCNNode then place that node onto the sceneView at a specific point. We are using nodes here instead of SCNScene because a SCNScene object occupies the entire sceneView, but we want our model in a specific point of the scene.

To create the SCNode we first load a temporary SCNScene with an asset and then save the scene’s childNode as the node we are going to use. We do this because you can’t initialise a node with an asset but you can load a node from a loaded scene if you search for the node by name.

Note that AssetName here is not the fileName of the asset but rather the node name of the model itself. You can find what nodeName your model has just by opening the .dae or .scn file in XCode, and toggling the Scene Graph view, which will reveal the layer list of the file.

How to set up the name of the node on the scene

After getting the node, the next step is adding it to the scene. We found two different ways to do it, and choosing one or the other depends on how you want your app to work.

First, we need to know where to render our model within the 3D world. For our demo we get the location by getting the user tap CGpoint from the touchesBegan method:

Getting a location CGPoint and translating it into a float_4x4 matrix with the worldTransform method.

The location variable we are getting from the above example is a 2D point which we need to position in the 3D AR scene. This is where the Feature Points mentioned above come into play. They are used to extrapolate the z-coordinate of the anchor by finding the closest Feature Point to the tap location.

sceneView.hitTest(location, types: .featurePoint)

You can also use the cases .existingPlaneUsingExtent and .estimatedHorizontalPlane to get the positions of the planes when using planeDetection

This method gives us an array of the closest ARHitTestResult, sorted by increasing distance from the tap location. The first result of that array is therefore the closest point. We can then use the following
let transformHit = hit.worldTransform that returns a float4x4 matrix of the real world location of a 2D touch point.

Plane Detection

Now that we have the location of the touch in the 3D world, we can use it to place our object. We can add the model to the scene in two different ways, choosing one over the other depends on how we have set up our ARSession and if we have planeDetection enabled. That is because if you run your configuration with planeDetection enabled, to either horizontal or vertical detection, the ARSCNView will continuously detect the environment and render any changes into the sceneView.

When you run a world-tracking AR session whose planeDetection option is enabled, the session automatically adds to its list of anchors an ARPlaneAnchor object for each flat surface ARKit detects with the rear-facing camera. Each plane anchor provides information about the estimated position and shape of the surface.

We can enable planeDetection on viewWillAppear when adding a ARWorldTrackingConfiguration to the ARSession:

configuration.planeDetection = .horizontal

So while planeDetection is on we can add a new node into the scene by creating a new SCNode from our Scene object and changing the node's position, a SCNVector3, to where we want the model to be on the view. We will then add this node as part of the childNode of the sceneView, and since planeDetection is enabled the AR framework will automatically pick up the new anchor and render it on the scene.

You can use either .existingPlaneUsingExtent and .estimatedHorizontalPlane cases instead of .featurePoints when trying to find where to place the model. The results given would be different in each case and it depends on where and how you want to place your object. Existing Planes would give you a point fix on a plane, like a floor or a table, and feature points would give a more specific location around objects in being tracked in a real environment.

To get the correct node position we will need to use the finalTransform float4x4 matrix we created before and translate it to an float3. To do that translation we used an extension that translates our float4x4 matrix into a float3.

Tada 🎉 we just successfully added a 3D model into an AR Scene!

Anchoring

Having the app continuously detect the plane is quite resource heavy. Apple recommends disabling planeDetection after you are done detecting the scene. But as we mentioned before, if planeDetection is not enabled the ARScene won't pick up your newly added childNode and render it on to the sceneView.

So if you want to be able to add new nodes and models to a scene after you are done detecting planes you will need to add a new ARAnchor manually.

To create an ARAnchor from the tap location we will use the same transformHit float4x4 matrix we created before — without needing to translate it this time, since ARAnchors and ARHitResults use the same coordinate space.

By adding the new anchor by ourselves instead of relying on the Session Configuration we trigger the renderer() function from the delegate that will return the node to be rendered for a particular anchor.

We need to double check if the anchor triggering the render function is the anchor we just added and not an ARPlaneAnchor.

With this in place our model will be rendered at the tap location of the sceneView just as seamlessly as when we had planeDetection enabled.

Tada 🎉 we just successfully added a 3D model into an AR Scene!

Conclusions

To summarise, in this post we went through the basics of Augmented Reality and Apple’s ARKit. We applied the lessons learned and crafted an application that adds our 3D models to the world using two different methods.

The code for this demo can be found on Novoda’s GitHub and you can also check our ARDemoApp repo, where you can import your own models into an AR Scene without having to write a line of code.

If you enjoyed this post make sure to check the rest of the series!

Have any comments or questions? Hit us up on Twitter @bertadevant @KaraviasD

Originally published at blog.novoda.com on January 21, 2019.

This post is from the multi-part series on ARKit, where we talk about designing for AR, building a demo app, exploring and testing many of the features of ARKit. We previously wrote on Adding the finishing touches to 3D models in Xcode.

In the previous post in this series we played around with AR and added a model to a scene. While doing so we wondered if there was a way to add a model to a specific object of the world. So we turned to Machine Learning to see if by recognising the scene we could place a 3D model seamlessly into an environment. Furthermore we wanted to create a recipe for adding ML capabilities to an app that doesn’t require specialised knowledge of image recognition, Neural Networks, or Machine Learning in general.

We experimented on CoreML with a TensorFlow model, as well as the YOLO framework. Our goal was to find the most clearly identified object within the scene and automatically add a 3D model to it. Read on to find out about our experiences with the different approaches we tried, and each one’s comparative advantages.

For an easy to read introduction to Machine Learning check out this post by Alex Styl on understanding machine learning

Exploring our options

We decided to work with pre trained Machine Learning models and the 3D models created in the first part of this series Designing for ARKit.

The focus of our investigation for this demo was not training a machine learning model but rather seeing how easily you can implement machine learning into an AR app and how straightforward CoreML is to use on iOS.

Core ML delivers blazingly fast performance with easy integration of machine learning models enabling you to build apps with intelligent new features using just a few lines of code.

Another important distinction before we move on is the difference between image recognition and object detection. Image recognition is a machine learning term for when a model is trained to identify what an image contains:

Image recognition is the process of identifying and detecting an object or a feature in a digital image or video.

MathWorks

Object detection on the other hand is the process of a trained model detecting where certain objects are located in the image.

Object detection is a computer technology related to computer vision and image processing that deals with detecting instances of semantic objects of a certain class in digital images and videos.

Wikipedia

All the pre trained models Apple gives us for CoreML are built for image identification instead of object detection, so we knew that we had to convert an object detection model to CoreML.
For this exercise we also used Apple’s Vision Framework since it processes images and would prepare input from ARSCNView for the model.

Apply high-performance image analysis and computer vision techniques to identify faces, detect features, and classify scenes in images and video.

We decided to use CoreML because its implementation is straightforward once the model has been imported into the app. CoreML reads as a regular Swift class which means it can be implemented by mobile developers without having to worry about machine learning concepts. That makes it a really powerful framework, the caveat is that your trained model might end up being a large file that a user will have to download with the app (using disk space on the phone) but since the model lives on the phone it also allows the machine learning functionalities to be used on the device instead of through a cloud. Doing so allows for object detection to happen even without an internet connection.

During WWDC 2018 Apple announced that their Vision Framework could now recognise and detect objects on live capture camera, making it perfect for this tutorial without the need to use external machine learning models or conversions. Because of that we updated the code that goes with this blog to use Vision to test how it worked, but also left the existing code and documentation for how to use a converted Tensorflow model called Tiny YOLO.

You can check out the new Vision Model working on our github. For both machine learning models, the 3D model used and how is placed on the ARScene is the same.

Using a Tiny YOLO model

Apple provides different ways to convert a model to CoreML. Options include a Python package called CoreML Tools, maintained by Apple itself, which converts any python model to CoreML; Apache MXNet, maintained by Apache, which can train MXNet models and then convert them to CoreML, or the recently added TensorFlow converter, maintained by Google, which allows us to convert some of the best known TensorFlow models into usable CoreML.

We looked into converting a tensorFlow model into CoreML using the converter but found that the graph for the object detection model was not optimised to be understood by the converter. While considering different object detection models we found this article by Matthijs Hollemans in which he uses a YOLO framework, converts it to CoreML and implements it to an IOS app.

YOLO is a neural network made up of 9 convolutional layers and 6 max-pooling layers, but its last layer does not output a probability distribution like a classifier would. Instead the last layer produces a 13×13×125 tensor.
This output tensor describes a grid of 13×13 cells. Each cell predicts 5 bounding boxes (where each bounding box is described by 25 numbers). We then use non-maximum suppression to find the best bounding boxes.

The YOLO model converted to CoreML will get an input of an image and output the 13x13 cells of every objects it detects. We then need to convert this grid into CGPoint coordinates for our scene. In his post Matthijs goes over how to convert this 25 numbers to actual Floats values needed for the prediction. Read the post and take a look at his code if you want to understand YOLO and CoreML a little better.

We used his code, with his permission, to predict where our object is located.There are few differences between our implementation and his, notably that we used an ARScene as the input image, and that in our case the output is not an array of bounding boxes, but a single box marking the most prominent object in the scene.

Our first step was importing the TinyYOLO ML model into the app and setting up the TinyYolo class, this class converts the grid into a Prediction structure. Once we had this set up we started editing the View Controller code from the ARKit app. On the ARKit app we were using user tap events to see where to place the 3d object and now we want to use the point given by the machine learning model to place the object there.

The first thing we need to do is pass the current AR scene view to the model so it can analyse it and find the most prominent object on the screen.

let pixelBuffer = sceneView.session.currentFrame?.capturedImage

However if we pass that pixelBuffer straight into the YOLO model it will return an error, not a prediction. This happens because the model expects a specific size image as input, meaning we need to scale the image to fit the model input (which in this model is 416x416).

To get the input processed we could manually scale the image to a 416x416 or we could wrap the model objects into a Vision request. Vision, as I mentioned before, processes images for Machine Learning models and will scale our input image, pass it through the CoreML model and get the prediction results.

We set up a vision model as a wrapper of our CoreML model and then create a Vision Request at the start of the app. Once we want to perform the request to the model we create a VNImageRequestHandler with the pixelBuffer we want to use and perform the request.

let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer) 
try? handler.perform([request])

Before we can perform the request there are still a couple of things we need to set up. One is to perform the request on a background thread so it does not actually block the UI of the app and the second is to get the Dispatch Semaphore to wait. We initialised the semaphore with a value of 2 to avoid possible repeated calls to the model before we get the results.

let semaphore = DispatchSemaphore(value: 2) semaphore.wait()

You will need to create the dispatch semaphore at the beginning of the app and then set it to wait before performing the first request and before going into the background thread.

Once the model has finished processing the image we will get the result prediction on our visionRequestDidComplete; we can then get the results as a multi array value and pass that to the Yolo class to get the outline CGRect boxes of the objects detected.

computeBoundingBoxes is a method created by Matthijs Hollance that accompanies the TinyYolo model. He explains the transformation on his blog

Using Object Detection in Vision

We now need to change how we get the location of the model but can maintain the rest of the AR code. Instead of getting a CGPoint from the touch location we get a CGRect from the model of where the object is and use those points against ARKit's Hit Points to find where we should place the model on a 3D AR scene.

To do any of the AR code we will need to return to main thread and interact with the UI, but before we do that we should free the semaphore allowing for other requests to be completed if necessary: self.semaphore.signal()

Once we are back to the main thread and ready to place the model on the scene we will first need to scale back the CGRect prediction into the proportions of the sceneView. This scaling code was done by Matthijs Hollance on his github code assuming the scene where the image comes from is full screen of a phone.

Once we have the scaled CGRect position of the prediction we create the model, get a hit point of where the model is supposed to go and use that hit point to change the model position and add it to the sceneView.rootNode

And just like that you can place a box wireframe inside a potted plant. That in itself doesn’t seem very useful or impressive, but we can use the same project set up to calculate the space between two objects and place a model there only if there is enough space, or using the code to automatically place a new color on a wall or floor while avoiding the objects that get in the way, etc.

There a lot of possibilities you can get from combining both frameworks, not only making your AR apps smarter but also helping make your Machine Learning apps more visual.

Conclusions

To summarise, in this post we went through the basics of CoreML and expanded onto the usage of ARKit by having it place models around existing objects.

The code for this demo can be found on Novoda’s GitHub and you can also check our ARDemoApp repo, where you can import your own models into an AR Scene without having to write a line of code.

Have any comments or questions? Hit us up on Twitter @bertadevant @KaraviasD

Originally published at blog.novoda.com on February 19, 2018.