While I’m not able to live in my house that is located in country that started a war, I still need to do neural net training and cloud GPUs are not something I can afford right now.

Normally, I like to ramp up my fans in my workstation so that cards don’t overheat during multiple day training sessions. I used to be using GUI apps that NVIDIA ships, but now I can only access my PC via SSH connection.

NVIDIA has nvidia-settings console app which technically allows you to set any GPU settings using terminal. I started googling on how to set the fan speed for individual GPU using that.

First of all, you have to set a bitmask that will allow controlling cooling settings:

sudo nvidia-xconfig --cool-bits=28

Then, what most forum answers on the web suggest, you should use this command:

sudo nvidia-settings -a "[gpu:0]/GPUFanControlState=1" \
-a "[fan:0]/GPUTargetFanSpeed=90"

Which is pretty self-explanatory. First argument enables controlling fans in the first GPU, second one assigns target fan speed in percentage (0–100) for the first fan.

Note: Fans are enumareted continously across all GPUs, i.e. I have 2 fans on each GPU and in order to control first fan on second GPU I have to use fan:2

However, after running this command through you’ll likely see this:

Unable to init server: Could not connect: Connection refused

ERROR: The control display is undefined; please run `nvidia-settings --help` for usage information.

What this means is that apparently nvidia-settings cannot connect to the running X server. In fact it was built so that you can interact with your GPU only if you have a graphics interface operational!

I had spent quite some time coming up with creative google prompts to find something that is addressing this on the web but failed to. This article brings us closer to solution, but still doesn’t do the trick. Fortunately, I was able to solve this puzzle using a combination of facts that I was able to collect.

Our goal is to trick nvidia-settings app into thinking that it is being used under a GUI session on a display. We need to define two variable for this purpose: DISPLAY and XAUTHORITY.

First one is easy, we can simply assign to random “first” display, like this:

export DISPLAY=:0

The article I mentioned also suggests a value for XAUTHORITY but unfortunately it didn’t work for me and likely won’t work for you, because every system has it’s own authority for X server, that is also depends on OS you running and other factors. In order to find a correct value for this you should run this command:

ps a | grep X

The command filters out all processed that has a mention of X server. On my machine I get this output:

3067 tty1     Sl+    0:01 /usr/lib/xorg/Xorg vt1 -displayfd 3 -auth /run/user/121/gdm/Xauthority -background none -noreset -keeptty -verbose 3
 5121 pts/0    S+     0:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox

The part that we are interested in is -auth /run/user/121/gdm/Xauthority. Copy the value of -auth argument and use it to define XAUTHORITY bash variable, like so:

export XAUTHORITY=/run/user/121/gdm/Xauthority

Then we can try assigning settings again:

sudo nvidia-settings -a “[gpu:1]/GPUFanControlState=1”

And now we can see desired output:

Attribute 'GPUFanControlState' (server:0[gpu:1]) assigned value 1.

Using this approach you setup any options of nvidia-settings, you can examine the whole list by running:

sudo nvidia-settings -q all

I hope this article will help to avoid headache for those who use their workstation over SSH.

Peace ✌️

Why you should convert your Pods into Packages

SwiftPM is also going to bring a lot of neat things like static linkage by default and dedicated bundles for framework resources, so one should definitely consider migrating from CocoaPods to SwiftPM as it can make your app faster, smaller and also provide some nice code-generated things in your code, like Bundle.module. Conversion process is pretty straightforward for 99% of frameworks out there. Plus, there are plenty of articles on the web right now explaining how to achieve that in under 15 mins.

Development Pods experience

However, there is one feature that was particularly great in CocoaPods. They call it Development Pods. Essentially, they allow you to work on your frameworks simultaneously to the client side, iterating very quickly, being able to check your framework changes against client integration. This feature was not only very powerful, but also very easy to use. In order to achieve that you simple have to provide a path to a local copy of your Pods and reinstall pods. Dead easy.

pod ‘Grain’, :path => ‘../Grain’

This feature is so great it has literally been enabling me to actually work as a developer for all those years I’ve been using CocoaPods, especially given that I work on a highly modular engine that is spread across tens of frameworks.

Editing Packages from CLI

SwiftPM has a pretty convenient interface for package editing in its CLI. What you have to do is to run a single command inside your project:

swift package edit <package name> --path <path/to/dependency> 

A checkout of <package name> will be created if it doesn't exist at the given path. If a checkout exists, package manager will validate the package name at the given path and attach to it. The package manager will also create a symlink in the Packages/ directory to the checkout path.

Using Swift Package Manager for framework development with Xcode

In the early days of SwiftPM it was really tricky to achieve simultaneous development. It was possible, but it wasn’t something you’d want to call a good development experience. CocoaPods were way cooler.

The problem became even bigger when Xcode introduced native support for Swift Packages. While it was possible to use local URLs pointing to Packages stored in your local file system, it still required way more clicks and things to do to simply convert a framework into a development mode, hack on it and see how it affects a client app.

It’s been my go-to solution for those rare SwiftPM projects I’ve been working on. Things changes when I discovered that Xcode can actually provide very similar experience to what you used to have with Development Pods. It is kinda weird that this feature isn’t really exposed into any visible UI, or at least I couldn’t discover it anywhere.

The solution

So let’s imagine you have set up a Xcode project with some Packages from GitHub. In my case it will be a single library for Metal development, Alloy, that I recently converted from Pod to a Package. The setup looks very simply:

Note: from what I understand, it is important to start with a remote package setup, otherwise local development may not work for you.

So imagine you now want to change few things inside a library while continuing working on a hosting application. All you need to do is… drag and drop a folder with a local copy of your library into the project, like this:

That’s it! This local copy of your framework will now “override” the one that was connected via remote URL. You can now change your framework in whatever way you like and immediately observe the result.

Going back to remote

Once you’ve finished your work on a local copy and pushed your changes into a remote, you can go back to using a published version of your framework. In case of a CLI you have to call a single command, unedit:

swift package unedit <package name>

If you are using Xcode it is sufficient to simply remove that package you’ve dragged and Xcode automatically resolve the remote one without any effort from your side.

Final words

SwiftPM is really great for Swift community and Swift apps, but it also very important for it to provide convenient experience for developers to be able to use to it’s fullest. Hope it was helpful! Hit me up on twitter in case you have any questions, I will be happy to answer!

In this article we are going to quickly cover a trivial GitHub Actions setup I use for my open source Metal library Alloy to automatically publish new versions into CocoaPods trunk register.

First, we will need to grab a registration token from CocoaPods. You can get it by using the pod trunk register command.

pod trunk register your@email.com -- description "GitHub Actions Token"

You will receive an e-mail with a confirmation link to confirm you are the owner of that e-mail. Once you’ve confirmed the token you can obtain it from private files using the following command:

grep -A2 'trunk.cocoapods.org' ~/.netrc

You will see something like this:

machine trunk.cocoapods.org
 login <your@email.com>
 password <TOKEN>

Copy the token you’ve got and then go to Settings of your repo:

Next, open Secrets in the side menu:

Press “Add a new secret” and create a new secret named COCOAPODS_TRUNK_TOKEN with the value of token you’ve just copied.

Setting up a GitHub Action

Okay, once you’ve done with the CocoaPods you are ready to proceed onto writing your first GitHub Action. Go to Actions of your repo and press “Set up a workflow yourself”:

And then replace the script with the following:

name: CI

on:
  push:
    branches:
    - master
    
jobs:
  build:

  runs-on: macOS-latest
    
    steps:
    - uses: actions/checkout@v1
    - name: Publish to CocoaPod register
      env:
        COCOAPODS_TRUNK_TOKEN: ${{ secrets.COCOAPODS_TRUNK_TOKEN }}
      run: |
        pod trunk push <YOURLIBRARY>.podspec

What this means is pretty clear:

1. Our script will run on every push into master branch of a repo
2. We will have only one build job which will run on latest macOS (you can specify macOS-10.14 for example
3. At first we will checkout our repo using another built-in action called actions/checkout@v1
4. We are declaring a COCOAPODS_TRUNK_TOKEN that is initialized with a secret value that we specified earlier so that nobody can access publishing even though your library and actions may be open-sourced
5. Finally, we call our command to push library into CocoaPods register

I use this 3 steps process to publish my libs:

1. Create a tag in the branch you are going to merge
2. Bump a version number inside a podspec to match the latest tag you’ve created
3. Merge branch into master and action will be triggered

Final words

We’ve just created a minimal example on how to automatically publish your libraries safely and securely. Of course you shouldn’t stop there: you can add automatic unit-test launches or more complex release pipelines into that.

If you have any questions please hit me up on Twitter.

Automated CocoaPod releases with GitHub Actions was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

Metal API is great, it opens up a whole lot of possibilities on both mobile and desktop devices. However, GPUs can be sometimes a bit clumsy, especially in high-precision things like font rendering or curve drawing, so sometimes you really miss good old CPU things. It happens to me pretty often that I would love to use some tiny bits of CoreGraphics functionality, but keep my GPU pipelines fast and efficient, with absence of redundant conversions or CPU/GPU synchronization breaks.

The case that inspired me to write this article was to give users the ability to draw a mask overlay in order to identify zones where they want certain effects to appear. It is of course possible to implement drawing with Metal, but that will require a lot of work: keeping a path data, triangulation and rendering logic. For quick prototype it is much easier to just use CoreGraphics, however the most straightforward approach is to create a CGImage from UI canvas and then convert to a MTLTexture to feed it as a mask to some shaders.

Interdimensional portal

But what if we would love to avoid unnecessary conversions and draw CoreGraphics paths straight into MTLTexture? That’s appears to be totally possible!

In order to achieve that we have to share a memory buffer between some CGContext and MTLTexture. Metal has a possibility to create textures from MTLBuffer objects, and MTLBuffer in turn has an opportunity to point into already allocated memory.

So what we need to do is the following:

1. Allocate memory for CGContext
2. Create MTLBuffer pointing to the same memory
3. Create MTLTexture out of MTLBuffer
4. Voila!

The implementation

Disclaimer #1: In this demo I use Alloy, this is a Metal library that I use in all of my GPU projects on Apple platforms. It provides a very tiny, Swifty API over vanilla Metal

Disclaimer #2: I neither do any decomposition in the demo nor error handling. Everything is done in a view controller, force-unwrapped or fatal-errored. This article serves as a reference snippet.

Things are a bit tricky when it comes to allocating no-copy memory for MTLBuffer. To make it happy, it has to be perfectly aligned, usually to 4096 bytes. MTLTextures are also acting fussy when it comes to what MTLBuffer it can be created from: we have to align each texture row and the alignment is dependent on the pixel format we want to use. Luckily, both of these tasks are doable!

For the sake of the demo we will allocate a shared resource for a drawable single-channel mask over MTKView.

Every time our MTKView will change its size, we will reallocate the mask. It is of course not necessary, but it is the simplest approach.

First, we will need to know a RAM page size from the system, for this reason we will use getpagesize() function. Now we have to calculate amount of aligned bytes per row and the total allocation size. To allocate aligned memory we will use posix_memalign(_,_,_) system function.

Note that from this point we are responsible for deallocating this memory, however we will handle it a bit later

Now we are ready to create a CGContext from this memory, you probably have done this many times, so this should be pretty straightforward:

I use CGImageAlphaInfo.none here to indicate that this is a grayscale image, but you can also use .alphaOnly

Using the exact same memory we can now create a no-copy MTLBuffer, making it responsible for deallocating the memory. However you can of course manage this on your own by passing nil to the last argument.

Now, the final step: we are creating a texture from this buffer. Storage mode of the texture must be the same with the buffer. There are additional requirements for different platform/OS that you can read here.

Since we’ve initially allocated memory in a format that Metal should be happy with, everything should go just fine.

Demo

Okay, now we are all set! After caching CGContext and MTLTexture somewhere, you can do whatever you want with them.

We will create a simple app, that will allow you “scratch” over a blurred texture, the final result can be seen in the video:

Let’s start with finding a cute puppy image, loading it into a MTLTexture and then blurring using Metal Performance Shaders.

Simple Metal fragment shader will handle the blending of original, overlay and mask textures:

Then we just need to dispatch it whenever a MTKView wants to redraw itself.

You should take into account the memory synchronization that happens between CPU and GPU. You should not create a situation where this buffer can be simultaneously accessed by CoreGraphics and Metal, especially on macOS.

To prevent this article from bloating, I purposefully didn’t include render pipeline state creation snippets, vertex shaders and CoreGraphics drawing code using UIKit touch APIs, you can find the full source code here.

Conclusion

Mixing CoreGraphics and Metal can be really useful and can work both ways: draw text from CoreText to Metal or render 3d mesh into CGContext and process it somehow on CPU. The rest is your imagination!

When you work for AI company, you always do a lot of technical demos to play with a certain technology in the wild. Modern algorithms tend to have quite a lot of parameters you would want to experiment with and sometimes the result of the work can only be approved by human, especially when it is related to any kind of image processing/computer vision or AR/ML type of apps, where you have to check results with your eyes.

This often leads to the situation where you have to create a bunch of on-screen buttons/sliders so that you can parameterize core logic and see how new hyper-parameters affects the result. After a handful of such demos I decided to create a custom debug menu, where you can add several options in a declarative way and then observe changes.

I made a prototype in a day by forking an existing debug tool Dotzu. Project is still in a beginning stage, but it more or less fits my needs for now. You can play with it in a public repo. Below is the small demo of achieved results:

Implementation

In this small tutorial I’m going to show you a technique which I used to achieve dynamic cell type evaluation in a single UITableView. What I wanted to achieve is that I could create an array of Option objects all of which conforms to a single protocol. This array basically will be an array of models. Then I want my make my table view controller to understand what type of cell (view) it has to use for it representation. I also wanted to avoid using enums, switch, or long conditional chaining (if-else), so that it will be extensible from outside and adding new type of options wouldn’t require rewriting existing library/controller code. Another goal is to encapsulate configuration code for every cell inside its class definition to move it outside of controller.

I started with declaring a protocol for our options. Since it is a 1-day prototype I didn’t put too much of a thought into this protocol, since I’m not quite sure about whole design, but for now let’s require every option to have a name. It also should probably have an associatedtype, but let’s keep it simple for now.

Next, let’s declare a protocol for cells that can represent an option: OptionCell. It also a dead-easy protocol which basically requires an object to know how to configure itself for an option.

Alright, as we now have all needed protocols, let’s create first option type and corresponding cell view. We will start with BoolOption which is often used for feature-toggling.

Now, the interesting part. We create a UITableViewController subclass with no cell prototype in its storyboard:

All the magic goes into its implementation. Let’s go through:

1. At the very bottom we declare an array of models to be presented
2. Here is the most interesting part. Since first-class type objects are not comparable, we use ObjectIdentifier struct from Swift standard library to create Hashable/Equatable proxy object for each Type object. We declare a dictionary and two handy methods to register and obtain types from it.

Since it is a repetetive code part, it should probably be moved into separate class called TypeRegister or something like that.

3. In viewDidLoad() method we register cell types into table view using a small helper method.

4. Next we register associative Model-View pairs into our type dictionary.

5. In a cellForRowAt: we get a model from our array, determine a cell type for it and dequeue a cell of appropriate type.

6. Finally, we tell UITableView to use Auto-Layout to calculate height of cells, since it can very from one option to another.

That’s it! Now you have super-flexible 60 lines of code implementation of UITableView, that can handle unlimited amount of models in it.

Usage

Usage of the framework is super easy. All you have to do is to initialize debug menu once (usually in AppDelegate) by calling one method:

DebugMenu.sharedManager.enable()

And then declare your options in a declarative manner (usually in controller’s viewDidLoad):

The result of this call can be seen on the video in the beginning of the story

Since the repo is dedicated for debug purposes, you probably would want to cover all interactions with DebugMenu with #if DEBUG … #endif clauses.

Final words

Since UI engineering is not what I do on daily-basis I would love to hear your feedback on the technique and to hear any directions on how to do it even better. Hope you enjoyed the read!

Don’t forget to play with the repo on GitHub and follow me on twitter.

CoreML was one of the loudest reveals on WWDC 2017. Machine Learning is taking over the world, so Apple released it just on time to get the hype.

There are a lot of examples in the web on how to use CoreML for classification purposes. Most of them use classic models like Inception-v3 or MobileNet.

The problem is that you can barely imagine someone who needs classification neural net in 2017 in their app :) Today I would love to show you how to use CoreML for image-to-image neural nets.

For that purpose I’ve chosen Holistically-Nested Edge Detection, which has publicly available pre-trained caffe model, which is just what we need to dive in.

In this tutorial I’m not going to deeply touch the CoreML tools setup, I assume you to be already familiar with them.

Step 1. Convert model from Caffe to CoreML

The process of converting model is fairly simple, but if you want to concentrate on iOS/Swift parts, you can download pre-converted models in the accompanying repo and proceed to Step 2.

For those, who want to do everything themselves, first go to HED repository and download it. There in examples/hed folder you can find a file named deploy.prototxt. This file contains meta-information about the model, so that inference engine (Caffe in this case) can find appropriate weights for all layers. Next thing you need is pre-trained model, you can train one yourself, but that goes beyond the scope of this tutorial. You can find one here.

Okay, now you need the deploy file with meta information and pre-trained model.

Next, open your favourite Python editor and create a fairly simple script, which should look something like this:

Don’t get upset if you don’t understand those magic numbers of chanel-biases or why the input tensor should be BGR. All you need to know for now is that those requirements come from original Caffe model.

Call it from the same directory, where two previously downloaded files are landed:

python convert.py

However, it won’t work out of the box. You will see the following error:

RuntimeError: Unsupported option ‘0’ for the parameter ‘offset size’ in layer ‘crop’ of type ‘Crop’ during caffe conversion.

The problem is that CoreML can’t find enough information to setup Crop layer internally. To fix this, you will need to add the following parameter to every Crop layer in deploy.prototxt:

crop_param {
axis: 2
offset: 32
offset: 32
}

With those parameters the result image will not be offseted and will fit original one just fine.

However, this is not the only fix you will have to do. After you will try convert one more time, you will see something like this:

RuntimeError: Caffe layer ‘sigmoid-dsn1’ is defined in the .prototxt file but is missing from the the .caffemodel file

That is because CoreML doesn’t support “empty” layers. Good thing it only doesn’t work for Sigmoid layers, which are fairly easy to implement yourself. What you have to do is to comment out (with #) all Sigmoid layers in deploy.prototxt.

Now you should be just fine and after few seconds you will see HED_main.mlmodel file in the same directory with your caffe model. That file is everything we need to launch this neural net in iOS Application, simply drag and drop it in your application.

Note: HED is the model which contains “side-outputs”. Those are model calculation’s results on intermediate steps, but in this particular case they can be used as well. However, CoreML doesn’t support side-outputs if they are used by “upper” layers, so if you would love to grab those, you will have to modify deploy.protoxt file so that those layers are final ones. I included examples on how to do it in the accompanying repo. Best thing about side-outputs is that if they fit your requirements, then it could save a significant amount of size in your application, for example original model is around 59mb, while dsn3 output is only 7mb.

Step 2. Launch inference from Swift in your app

All you need to do to integrate the model into your application is to drag the .mlmodel file into your XCode project and add it into your application’s target.

CoreML models contain pre-generated code, which allow you to use models like classes in Swift or Objective-C. That said, to create and use the model you simply need to do the following:

let hedMain = HED_fuse()

Next thing you have to do is to convert your input image (which is likely to be UIImage instance) to models’ input format. In our case it is 500x500 CVPixelBuffer object. You can find conversion util extensions in the repo.

Next, you have to call .prediction(data:) to launch actual inference of the model on input data. What you will get in return is MLFeatureProvider, which contains model’s computation results. To retreive features you need to call .featureValue(for:) and pass the name of output layer you want to get features from. If you don’t want to hardcode things into your code, you can use something like this:

results.featureValue(for: results.featureNames.first!)

To know the shape of the output feature you should use shape property. In our case it will be 1x1x500x500 which corresponds to single-channel image.

At this points we have two problems:

1. The result features are in Double format, which is not really useful.
2. The result features are not normalized.

While first is trivial to fix, the second is actually pretty interesting. The thing is that before CoreML, every company who did their own ML had either hardcoded MPS (Metal Performance Shaders) models or runtime model builders (similar to what Apple released in MPS Graph API) with custom converters. The main challenge with those was to combine usage of MPS with the missing layers, which is actually the case here: CoreML can’t parse our Sigmoid layers properly (not really the Sigmoid itself is the problem here, but for the tutorial we will pretend like CoreML doesn’t contain Sigmoid layer), so we have to sort of plug it in.

Implementing custom layers in Metal for CoreML goes beyond this tutorial. Good thing for us is that our “missing” layer is final one. In case the layer is in the middle of computation process, you will have to split your CoreML model in two: before and after that layer. For the sake of this tutorial we will just normalize our result tensor with sigmoid on CPU.

Let me know if you would love to see the article on implementing custom layers.

Here you can the snippet of how to normalize our result features on CPU using sigmoid function and put it into UInt8 buffer.

Last thing you will need to do is to convert freshly-baked UInt8 buffer into single-channel CGImage and display it.

The results looks like this:

Today we learned how to integrate image-to-image models into iOS Applications. We also learned how to implement simplest missing layers on CPU and display results of tensor.

You can find WWDC Session that covers basics of CoreML: Integrating a Core ML Model into Your App.

And the accompanying repo here: https://github.com/s1ddok/HED-CoreML

Let me know if you are interested in more articles about CoreML!