MTLDoc

Optimizing Metal: Stitch several Metal functions into a single shader using MTLFunctionStitchingGraph

Fri, 18 Aug 2023 09:00:00 +0000

In this post I’ll show how to use Metal’s new feature MTLFunctionStitchingGraph to create an image processing pipeline in a single shader instead of chain of several GPU kernels. In terms of common GPU practices it is a new option for creating uber-shaders in runtime using precompiled Metal functions. And for large pipelines it can be much more efficient than compiling the whole shader from source in runtime and less error prone.

With this post I start a new series of articles about Metal optimizations. I’ll cover several topics which I found interesting and useful for Metal pipeline creation and encoding which can help you to process your pipelines faster. And I’ll try to explain them in details so that you can use them in your projects.

This technology is used inside Apple’s MPSGraph but MPSGraph doesn’t allow to inject custom shaders in it and works only with multidimensional arrays. I’ll show how to create MPSGraph like processing but with plain old Metal textures and of course your own shaders.

What is MTLStitchingGraph?

In most cases a single shader will be more efficient comparing to several shaders chained together. A long time ago Apple introduced function constants which is a good alternative to preprocessing macros which can remove unnecessary branching from shader code during processing. Metal Stitching Graph is another option to create uber-shaders in runtime. And it can be much more flexible. You’re not stuck to a single shader with compile time branching but can create shader from building blocks right in runtime. I’ll show how to do it for image processing but this API can be used in processing all kinds of data on GPU.

Usually if you want to create node based GPU processing engine you define several types of nodes and allow user or some external API (for example JSON encoded graph received from server) define the order of nodes execution. Each node in this architecture represents atomic processing task. For example one node applies blur filter to input texture, another one adjusts texture brightness and so on.

This is a very flexible architecture but it has some drawbacks. First of all each node will be encoded to separate shader and GPU will perform full texture processing for each of them. Second you need to create a separate output texture in memory to store the result of node processing which will be provided as input to the next node. And last but not least the full graph processing will be encoded on each call. For example you have a graph with 10 nodes and user adjusts brightness (one parameter for one node) the full pipeline will be encoded again. With MTLStitchingGraph you can avoid all of these drawbacks. But it will require to completely rethink your node engine architecture.

MTLFunctionStitchingGraph allows you to create a Metal function or several functions in runtime which will be executed for each pixel (or any data) and have all the processing steps defined by your graph. Actually it is just an API for creating directed acyclic graphs which will be executed on GPU. Good example of it is MPSGraph, Apple framework for neural network training and inference on GPU.

Stitchable Function

In the heart of Metal Stitching Graph is a [[ stitchable ]] Metal function which is a new function qualifier like [[ kernel ]], [[ vertex ]] or [[ fragment ]]. And the only purpose of this type of function is to be stitched with other functions to create a chain of executions on GPU inside a single function. Let’s try to build a simple graph which will apply grayscale filter to input image and then invert its color.

#include <metal_stdlib>
using namespace metal;

[[ stitchable ]]
half4 read_color(const texture2d<half, access::read> tex,
                 const ushort2 position) {
    return tex.read(position);
}

[[ stitchable ]]
void write_color(const texture2d<half, access::write> tex,
                 const half4 color,
                 const ushort2 position) {
    tex.write(color, position);
}

[[ stitchable ]]
half4 grayscale_color(const half4 color) {
    return half4(half3((color.r + color.b + color.g) / 3.0h), color.a);
}

[[ stitchable ]]
half4 invert_color(const half4 color) {
    return half4(half3(1.0h - color.rgb), color.a);
}

This is our Metal file which will be compiled to Metal library during Xcode build. It contains four [[ stitchable ]] functions. First two functions are used to read color from texture and write color. The last two functions are actual image processing functions. They are very simple and don’t do any complex processing. But they are enough to show how to use MTLFunctionStitchingGraph.

In regular image processing pipeline you will define two separate [[ kernel ]] functions each of which will receive source texture to read from and destination texture to write to. So on CPU side you’ll create input texture, separate texture to store grayscale result and another one to store inverted result which will be the output of your pipeline.

With Metal Stitching Graph you don’t need an intermediate texture to store grayscale result because both functions will be executed in a single shader one after another like if you wrote a separate [[ kernel ]] function for grayscale_and_invert.

Stitching Graph

Let’s create a graph which will execute these two functions in a single shader. We will do it in four separate steps which I’ll explain in details.

Create graph description with all the nodes and connections between them;
Compile graph to a separate Metal library with a single function which will execute our graph;
Create another Metal library from source with a single [[ kernel ]] function which will be used to execute our graph;
Create pipeline state with this shiny new kernel.

After this you can encode this kernel as any other kernel function, add it to a larger pipeline or use it as a standalone function. I’ll show how to do it in Swift but since all of the Metal API is available in Objective-C you can use it in your Objective-C projects as well or C++ projects with Metal-C++ header library or Objective-C++ wrapper.

Graph Description

let inputs: [MTLFunctionStitchingInputNode] = [
    MTLFunctionStitchingInputNode(argumentIndex: 0), // source texture
    MTLFunctionStitchingInputNode(argumentIndex: 1), // destination texture
    MTLFunctionStitchingInputNode(argumentIndex: 2), // pixel position
]

let readColor = MTLFunctionStitchingFunctionNode(
    name: "read_color",
    arguments: [inputs[0], inputs[2]],
    controlDependencies: []
)

let grayscaleColor = MTLFunctionStitchingFunctionNode(
    name: "grayscale_color",
    arguments: [readColor],
    controlDependencies: []
)

let invertColor = MTLFunctionStitchingFunctionNode(
    name: "invert_color",
    arguments: [grayscaleColor],
    controlDependencies: []
)

let writeColor = MTLFunctionStitchingFunctionNode(
    name: "write_color",
    arguments: [inputs[1], invertColor, inputs[2]],
    controlDependencies: []
)

let graph = MTLFunctionStitchingGraph(
    functionName: "process_graph",
    nodes: [
        readColor,
        grayscaleColor,
        invertColor,
        writeColor,
    ],
    outputNode: nil,
    attributes: []
)

Let’s go through this code line by line. First you define inputs for your graph. These are the actual arguments for Metal Shading Language (MSL) [[ visible ]] function which we will later compile to Metal library. Notice that you don’t define types of inputs even though MSL as a C++ based language requires types of arguments to be defined in function declaration. Types of these arguments will be found from the arguments of [[ stitchable ]] functions which later uses these inputs.

Next you define nodes of your graph. Each node is a MTLFunctionStitchingFunctionNode which represents a single [[ stitchable ]] function. You need to provide a name of this function from your Metal library, list of arguments and list of control dependencies.

Arguments of these nodes have to be the other nodes of the graph, so you can’t inject a single float or integer value as an argument which can be a good addition to current API (Apple, please :). You need to create a separate input node for each value you want to pass to your graph using MTLBuffer or if they are of a single type you can put them into a one buffer of course.

Control dependencies define the order of execution. Each directed acyclic graph can be converted into a sequence, such that for every edge the start vertex of the edge occurs earlier in the sequence than the ending vertex of the edge. This operation is called topological ordering and Metal Stitching Graph will do it for us. Graph can have several such orderings but will have at least one. For example a simple graph from this article cover has six orderings.

1, 2, 3, 4, 5, 6
1, 4, 5, 2, 3, 6
1, 2, 4, 3, 5, 6
1, 2, 4, 5, 3, 6
1, 4, 2, 3, 5, 6
1, 4, 2, 5, 3, 6

If you add to nodes 2 and 3 node 4 as control dependency that means that node 4 will be executed before node 2 and node 3 and the second or fifth ordering will be used. It can be useful if node 2 or node 3 have side effects which will modify data in node 1 and node 4 needs to receive unmodified data.

When we defined all the nodes we can create a graph defining the function name to be later used to compile the function.

You pass all the nodes to graph’s nodes argument and also you may select one of the node as the output node of the graph and the return type of this node will be used as return type of the graph function. If you don’t provide output node the graph will be compiled as a void function.

Graph attributes is an array of MTLFunctionStitchingAttribute which in iOS17/macOS12 API has only one option MTLFunctionStitchingAttributeAlwaysInline. This option will apply __attribute__((always_inline)) to resulting function.

Readers familiar with Metal pipelines may notice that all these initializers are not throwable or return optional types which is not common for Metal APIs. This is because MTLFunctionStitchingGraph is just a description of the graph and it doesn’t check if the graph is valid. It will be checked during compilation later.

Compiling the Graph

let device = MTLCreateSystemDefaultDevice()!
let library = device.makeDefaultLibrary()!

let functions = [
    library.makeFunction(name: "read_color")!,
    library.makeFunction(name: "grayscale_color")!,
    library.makeFunction(name: "invert_color")!,
    library.makeFunction(name: "write_color")!,
]

let stitchedDescriptor = MTLStitchedLibraryDescriptor()
stitchedDescriptor.functions = functions
stitchedDescriptor.functionGraphs = [graph]

let stitchedLibrary = try! device.makeLibrary(stitchedDescriptor: stitchedDescriptor)
let stitchedFunction = stitchedLibrary.makeFunction(name: "process_graph")!

Here we perform an actual graph compilation into a single function and you see a lot of force unwraps and tries here :) First we create MTLFunction objects for all our graph’s nodes functions and using a familiar descriptor pattern create a library with a single function which will execute our graph. And finally create an MTLFunction object with the name of our graph function.

If you look at the Apple’s example for stitching graph you’ll see there that next to [[ stitchable ]] functions they declare [[ visible ]] function inside Metal file to which the graph will be stitched. We’ll use it also in next section, but I found that it is not required and you can compile function graph without any declarations inside your precompiled library.

The resulting graph function may look like this:

[[ visible ]]
void process_graph(texture2d<half, access::read> arg1,
                   texture2d<half, access::write> arg2,
                   ushort2 arg3) {
    const half4 var1 = arg1.read(arg3);
    const half4 var2 = half4(half3((var1.r + var1.b + var1.g) / 3.0h), var1.a);
    const half4 var3 = half4(half3(1.0h - var2.rgb), var2.a);
    arg2.write(var3, arg3);
}

This is the main part of our uber-shader which was compiled in runtime using precompiled functions from existing Metal library without any preprocessing macros or function constants. Metal Stitchable API is rather verbose as you see but it can give you more flexibility and control over uber-shader creation.

[[ visible ]] is another MSL function qualifier which is used to mark functions which can be used to create MTLFunction object on CPU side. It is similar to [[ kernel ]] function but it can’t be called directly i.e. assigned to computeFunction in MTLComputePipelineState object.

And here comes a tricky part. How to call this function?

Calling the Graph (Predefined Arguments)

For now we have a hardcoded graph which has three arguments and we know the types of them. If you want to build graphs of this kind where inputs and output are already known you can add a wrapper kernel right inside your Metal library and call it directly. Add this to your Metal file right after [[ stitchable ]] functions:

[[ visible ]]
void process_graph(texture2d<half, access::read> src,
                   texture2d<half, access::write> dst,
                   const ushort2 pos);

[[ kernel ]]
void process_graph_kernel(texture2d<half, access::read> src [[ texture(0) ]],
                          texture2d<half, access::write> dst [[ texture(1) ]],
                          const ushort2 pos [[ thread_position_in_grid ]]) {
    process_graph(src, dst, pos);
}

Here we declare a [[ visible ]] function and implementation of it will be compiled during graph creation. And from [[ kernel ]] function with predefined arguments we simply call this function. Here is how to create pipeline state to encode this function.

let linkedFunctions = MTLLinkedFunctions()
linkedFunctions.privateFunctions = [stitchedFunction]

let descriptor = MTLComputePipelineDescriptor()
descriptor.computeFunction = library.makeFunction(name: "process_graph_kernel")!
descriptor.linkedFunctions = linkedFunctions

let pipelineState = try! gpu.device.makeComputePipelineState(descriptor: descriptor, options: []).0

And after this you can use this pipeline state in your processing.

The only difference comparing to “usual” kernel encoding in this case is that you need to add your graph function to linked functions of pipeline state so that kernel will be able to find it. Otherwise you will get a runtime error during pipeline state compilation.

With this approach there is no Metal library compilation from source in runtime. You only use precompiled [[ stitchable ]] functions to create a graph with Metal API. But it would be much more flexible if you could create a graph with arbitrary inputs and outputs and then compile it to a single function. Let’s see how to do it.

Calling the Graph (Arbitrary Arguments)

Unfortunately there is no Metal API to build shaders like we build graph function using node objects. So the only option for us is to compile it from source.

This option is vital if you want to build your own library for GPU processing and MPSGraph with its close architecture doesn’t suit your needs. I’ll show only the basics and provide some guidance how to achieve such functionality but won’t show you how to build general purpose node based processing library on top of Metal Stitching API. It is very much depends on your needs and current architecture of your pipeline.

For demonstration purpose we will compile the same kernel from source code which we had in our library with predefined arguments graph. The only difference is that we don’t have declaration of process_graph visible function so we need another way to call our graph from kernel.

Remember that we already have two Metal libraries. The first one is compiled default.metallib which contains only our [[ stitchable ]] functions. And the second one is a stitched library with a single [[ visible ]] function which executes our graph. Let’s add another library with the [[ kernel ]] function created in runtime.

let source = """
    #include <metal_stdlib>
    using namespace metal;

    using graph_function = void (
        texture2d<half, access::read>,
        texture2d<half, access::write>,
        ushort2
    );

    [[ kernel ]]
    void process_graph_kernel(texture2d<half, access::read> src [[ texture(0) ]],
                              texture2d<half, access::write> dst [[ texture(1) ]],
                              visible_function_table<graph_function> table [[ buffer(0) ]],
                              const ushort2 position [[ thread_position_in_grid ]])
    {
        table[0](src, dst, position);
    }
    """

let kernelLibrary = try! device.makeLibrary(source: source, options: nil)

You see that here we don’t use [[ visible ]] function declaration and call it from our shader. Instead we use visible_function_table which will contain a pointer to our graph function. This table is some kind of fixed size array of pointers to [[ visible ]] functions. And we need to create this table on CPU side and pass it to our shader.

let linkedFunctions = MTLLinkedFunctions()
linkedFunctions.functions = [stitchedFunction]

let descriptor = MTLComputePipelineDescriptor()
descriptor.computeFunction = kernelLibrary.makeFunction(name: "process_graph_kernel")!
descriptor.linkedFunctions = linkedFunctions

let pipelineState = try! device.makeComputePipelineState(descriptor: descriptor, options: []).0

let tableDescriptor = MTLVisibleFunctionTableDescriptor()
tableDescriptor.functionCount = 1

let table = pipelineState.makeVisibleFunctionTable(descriptor: tableDescriptor)!
let handle = pipelineState.functionHandle(function: stitchedFunction)!
table.setFunction(handle, index: 0)

The first part of this code is rather similar to the previous example. The only difference is that we add stitchedFunction to array of functions and not privateFunctions of MTLLinkedFunctions object because private functions can’t be added to visible function table and need to be called directly from shader code.

Then we create a visible function table, obtain a function handle from pipeline state and add it to table. And finally we can encode our pipeline state.

// Assume that we have source and destination textures and command buffer
let encoder = commandBuffer.makeComputeCommandEncoder()!
encoder.setComputePipelineState(pipelineState)
encoder.setTexture(sourceTexture, index: 0)
encoder.setTexture(destinationTexture, index: 1)
encoder.setVisibleFunctionTable(table, bufferIndex: 0)
encoder.dispatchThreads(...)
encoder.endEncoding()

Voila!

Notes about Nodes

First of all thank you for taking this journey inside this very specific Metal API with me. If you’re a node engine developer you may be impressed with it like I was and want immediately implement it in your project. But before you start I want to share some thoughts about this API.

To create something like an MPSGraph library which uses stitching graph will be a nontrivial task. In standard node based architecture nodes are independent from the graph itself. All they expose outside are inputs, outputs and parameters. If you’ll try to convert your current architecture to stitching graph you’ll find that nodes has to tell much more information about theirs behavior.

For example if node outputs a texture with different size than input texture at this point you’ll have to split the graph into separate graphs and each one will encode a kernel with different number of GPU threads. So if your pipeline consists of a lot of convolution operations you may end up with a lot of small graphs which will be inefficient to encode comparing to standard precompiled kernels.

Second you need to think how to pass parameters for each node inside a single kernel which contains several nodes processing. With separate encoder for each node you pass only parameters required for this particular node. With general purpose stitching API you need to pass them to GPU all in a single encoder. And since [[ stitchable ]] functions know nothing about the structure of your nodes you need to pass them inside an array of floats for example. And on CPU side you’ll need to encode graph which will be able to take required parameter from array with a proper index.

From my point of view this API is not ready enough to be easily used in arbitrary GPU processing. Though MPSGraph proves that it is an accomplishable task. Nevertheless it is a good option for encoding fixed pipelines with optional steps, so that you can compile the graph with only selected processing steps. For example postprocessing pipeline in game engines. For processing with arbitrary pipeline it would be better to use indirect encoding as for me which I’ll cover in the next articles.

Conclusion

Uber-shaders without branching in most cases will be a better solution in terms of performance but to maintain them is rather a difficult task. Metal provides several technics to write them: function constants, MSL uniform types and now MTLFunctionStitchingGraph. Which one to use depends on your needs and your pipeline architecture. There is no best solution for all cases. But if you want to find one for your particular case I hope this article and further ones about Metal optimizations will help you to make a decision.

If you’re a game developer or app developer and face issues with Metal performance feel free to contact me and I’ll try to help you. Also follow me on Twitter @mtl_doc to be notified about new articles.

Thanks for reading!

Code Signing Hell with XCFramework

Fri, 23 Dec 2022 09:00:00 +0000

If you’ve ever dealt with code signing issues in iOS or macOS apps, you know how challenging it can be. Code signing is typically something that just works, but when you encounter errors, it can be difficult to determine what went wrong. Seems that code signing problems are so unique that it can be hard to find solutions in Google, and you have to figure out the problem on your own.

This post is about one such unique issue that I encountered. I don’t know if anyone else has faced the same problem, as I couldn’t find any information about it. The issue involved signing an internal XCFramework with a C++ library that was distributed as a Swift Package, added as a dependency to another package, and only appeared on macOS.

Asset validation failed

Invalid Code Signature Identifier. The identifier “SomeFramework-8885494daf37cec1267d771a9cdwi75f0196184d1” in your code signature for “SomeFramework” must match its Bundle Identifier “SomeFramework” (ID: 909f33e1-f60b-4ed2-8f0b-574b0cd93e6f)

This error first arose when I tried to build a macOS app with a dependency that worked fine in iOS apps. This was a regular app without any macOS-specific capabilities such as hardened runtime, which might have stricter code signing rules. The app had the same build pipeline as our iOS apps, so the problem was specific to macOS and like any code signing issues only occurred in the release configuration.

XC++Framework

The dependency in question was a Swift package that contained another dependency, a prebuilt XCFramework from a C++ library with binaries for different Apple platforms and architectures, also wrapped in a Swift package. I won’t go into detail about the process of building a C++ XCFramework and distributing it as a Swift package, but I will mention that we used the swift-create-xcframework tool for building.

If you’re not familiar with XCFrameworks, they are essentially folders with an Info.plist file and separate folders for different platforms and architectures, such as iOS, macOS, watchOS, Mac Catalyst, Simulator etc.

For the purposes of this issue, we will focus on the iOS and macOS versions, as they had the most noticeable differences. One important thing to note is that the macOS version of the framework has a _CodeSignature folder, while the iOS version does not. This is why the problem only appeared on macOS.

Codesign for the Resque

There is a tool called codesign that is used to sign your app during the build process, and it also allows you to inspect the current signature of apps and frameworks. To see the signature of a framework, you can use the following command:

codesign -dv SomeFramework.framework

The -dv flag stands for the --display and --verbose options. You can find more information about the codesign command and its options here or by running man codesign in Terminal.

If you run this command for an iOS framework, the output will be:

SomeFramework.framework: code object is not signed at all

But for a macOS framework, you’ll see:

Executable=/path/to/executable
Identifier=SomeFramework-8885494daf37cec12e67d771a9cdwi75f0196184d1
Format=bundle with Mach-O universal (x86_64 arm64)
CodeDirectory v=20400 size=28550 flags=0x2(adhoc) hashes=886+3 location=embedded
Signature=adhoc
Info.plist entries=20
TeamIdentifier=not set
Sealed Resources version=2 rules=13 files=3
Internal requirements count=0 size=12

Notice that the Identifier here is the same as the one in the error message from the App Store above, and it definitely doesn’t match the framework’s bundle identifier, which is simply SomeFramework. This is where the source of the error becomes clear. Also, note that the TeamIdentifier is not set here, and the framework needs to be resigned during the app build. But why is it resigned with the same identifier?

To find out, I started inspecting the Xcode build logs for the app and found that Xcode was running the following command to sign the framework:

/usr/bin/codesign --force --sign - --timestamp=none --preserve-metadata=identifier,entitlements,flags --generate-entitlement-der /path/to/framework/embedded/inside/app

And here is the issue! When signing the embedded framework, Xcode updates the signature but preserves the identifier (using the --preserve-metadata option). Since this XCFramework is inside a regular Swift package, I can’t embed it without signing. This option is only available for frameworks, but not Swift packages. So I need to figure out how to sign it during XCFramework creation, but with a valid identifier.

The codesign tool allows you to set the signature identifier using the -i option, and the swift-create-xcframework tool has an --xcconfig option to pass a .xcconfig file with a list of Xcode build settings that will be used during XCFramework creation.

One relevant build setting is OTHER_CODE_SIGN_FLAGS, which is a list of additional flags for codesign.

OTHER_CODE_SIGN_FLAGS=-iSomeFramework

Finally, after three days of investigation, I rebuilt the XCFramework with the proper identifier using this one line of code. When Xcode signed the macOS app, it preserved this identifier, and the App Store validation process passed successfully.

I also tried removing code signing at all by using the CODE_SIGNING_REQUIRED=NO flag, which would prevent the macOS framework from being signed. However, this didn’t help, as Xcode still generated the same identifier with an additional hash when signing the app. The only solution was to sign the framework with the proper identifier.

Conclusion

I don’t know if this post will help anyone fix code signing errors, as they are always unique. It’s possible that the problem was caused by the C++ nature of the framework, and that a Swift XCFramework would have had the proper identifier from the start. However, there is very little information available about these kinds of developer problems on the internet, so I decided to write about it in the hope that it will help someone avoid spending three days troubleshooting and instead focus on more productive tasks.

If you have any questions, feel free to ask them via email at contact@mtldoc.com or connect with me on Twitter at @petertretyakov. Keep in mind that I am not a code signing expert and really appreciate it when these security things just work :) I hope your day is free of build errors!

Shaders Explained: Dithering

Sun, 20 Nov 2022 09:00:00 +0000

Three months ago I launched a small app on the App Store which allows users to edit photos in a stylish old-school way. The processing technology is called dithering. It is a very old technic from the days when computers were capable to show very limited amount of colors. But developers and artists always wanted to go further existing software and hardware limits so they invented such algorithms. Dithering makes images look like they use much extended color palette than they actually has and missing shades are emulated through placing pixels with allowed colors in proper way. Look at the image in the header. Central part of it uses only black and white pixels and not a single grey one but because the amount of black and white pixels is different in different areas the whole image looks like it uses much extended grayscale color palette. In this post I’ll explain how it works and how to implement it in Metal.

DISCLAIMER! I used God of War Ragnarök official art to show how dithering algorithms work. All the rights for it belongs to Sony and I hope Sony allows to use this image for demonstration purposes. Nevertheless it is a wonderful game!

Why Dithering?

You may wonder why do we need to mimic missing colors in 2022 when modern displays can show millions of colors? The first answer is of course old-school vibes which we can create using old algorithms for color management :) This is what my Dither app do. The second reason is to reduce file size. You actually see dithering a lot even today while laughing at memes on Twitter. GIF format uses only 256 colors to make files small in size and shading is done via ordered dithering.

Ordered dithering is way more popular then an unordered alternative. Because it requires less processing power. Also it can be easily parallelized and so we can run it on GPU in realtime for video frames or camera input without frame drops. Unordered dithering looks slightly better but it is an error diffusion algorithm. That means it is linear by design and for a long time it was considered impossible to run it in parallel. Because pixels are processed in order and on each step error is calculated and propagated to neighboring pixels. So theoretically color of the very last pixel of the image depends on the first one. But researches don’t like the word “impossible” and as GPU processing becomes standard for image processing they invented parallel algorithms for error diffusion dithering. And I implemented one of them in Metal shader.

Though error diffusion dithering is more interesting from engineering perspective lets start from the simple ordered algorithms!

Ordered Dithering

Ordered dithering was used on a variety of old computers and game consoles. It is perfectly suited for GPU processing where each pixel processed individually in a highly parallelized way.

To implement it we will need input image, color palette of allowed output colors and a matrix which we will apply to input image before trying to find the closest color from palette.

Matrix for ordered dithering is called index matrix or Bayer matrix and it can be constructed recursively from the root 2x2 matrix according to formula from this Wikipedia article.

For performance reason I used predefined matrices with 2x2, 4x4 and 8x8 sizes also from Wikipedia. Here they are:

enum Matrix {
    case matrix2x2
    case matrix4x4
    case matrix8x8

    var array: [Float] {
        switch self {
        case .matrix2x2:
            return [
                0, 2,
                3, 1,
            ].map { $0 / 4 - 0.5 }
        case .matrix4x4:
            return [
                0, 8, 2, 10,
                12, 4, 14, 6,
                3, 11, 1, 9,
                15, 7, 13, 5,
            ].map { $0 / 16 - 0.5 }
        case .matrix8x8:
            return [
                0, 32, 8, 40, 2, 34, 10, 42,
                48, 16, 56, 24, 50, 18, 58, 26,
                12, 44, 4, 36, 14, 46, 6, 38,
                60, 28, 52, 20, 62, 30, 54, 22,
                3, 35, 11, 43, 1, 33, 9, 41,
                51, 19, 59, 27, 49, 17, 57, 25,
                15, 47, 7, 39, 13, 45, 5, 37,
                63, 31, 55, 23, 61, 29, 53, 21,
            ].map { $0 / 64 - 0.5 }
        }
    }
}

The matrix is passed as an array of Floats to shader according with matrix size as ushort2 (SIMD2<UInt8> in Swift terminology). Palette is also an array of float3 (SIMD3<Float>) colors, each channel of which is in 0...1 range. Since it is a pointer to MTLBuffer with array in it we also need to tell the shader its size.

The full shader is rather simple:

[[ kernel ]]
void orderedDither(texture2d<float, access::read> source [[ texture(0) ]],
                   texture2d<float, access::write> destination [[ texture(1) ]],
                   constant float* matrixValues [[ buffer(0) ]],
                   constant ushort& matrixSize [[ buffer(1) ]],
                   constant float3* palette [[ buffer(2) ]],
                   constant ushort& paletteCount [[ buffer(3) ]],
                   const ushort2 tid [[ thread_position_in_grid ]]) {
    const float4 sourceColor = source.read(tid);

    const ushort2 matrixPosition = tid % ushort2(matrixSize);
    const float matrixValue = matrixValues[matrixPosition.y * matrixSize + matrixPosition.x];
    const float3 destinationColor = closestColor(saturate(sourceColor.rgb + float3(matrixValue)), palette, paletteCount);

    destination.write(float4(destinationColor, sourceColor.a), tid);
}

First we take the color of pixel from the source texture;
Then find the corresponding position from matrix;
Take the matrix value at this position;
Sum source color and matrix value, saturate it so it will be in 0...1 range;
Find the closest color to this summed value from palette;
And write it to destination texture.

There are several implementations of closestColor function available on the internet. For black and white dithering you may just use Metal’s step function and for color dithering the simple vector distance will work fine. Some developers suggest to convert colors to LAB color space and then calculate distance between current color and palette colors. I’ll omit implementation of this function here.

For black and white palette we will have God of War Ragnarök poster as it was printed on slow and loud matrix printer.

And what if we would have GameBoy HD console with its traditional greenish color palette which can run this state of the art game?

We are not limited in making only grayscale images with applied tint to it. Any color palette can be used. Here is the same poster which uses only black, white, red, green and blue colors (BWRGB).

Notice how matrix pattern is clearly visible in all these images. The same pattern you may see on GIF images. For smaller matrix size the pattern become so repetitive that it ruin a lot of image details. Here is the black and white dithered image with 2x2 matrix. For the previous images I used 8x8. Here you can see how the shapes of mountains on background become indistinguishable with the sky.

But enough for ordered dithering! It was really a trivial shader. Lets try to accomplish more challenging task and apply linear algorithm to GPU processing. But first we need to understand how Metal dispatching works.

Few Words About Dispatching

When you schedule tasks to GPU you need to tell Metal how much threads it need to run to process it. For vertex/fragment shaders in base case you use this function

func drawPrimitives(
    type primitiveType: MTLPrimitiveType,
    vertexStart: Int,
    vertexCount: Int
)

Metal will schedule as many threads as you declare in vertexCount variable for vertex shader and then after rasterization step schedule a thread for each pixel which is covered by your geometry.

When you program GPU pipelines you may think that all the vertices will be processed in parallel and then all the pixels also. That is what GPU was created for after all! But the reality is slightly different. Processors has certain number of cores and even the most powerful unit has its own limits.

For example Apple M1’s GPU have 8 cores, each of which contains 16 execution units and every execution unit has 8 arithmetic logic units (ALUs). So the total number of ALUs is 1024 but each ALU can process 24 threads simultaneously. It doesn’t mean that you can expect that all 24K threads will be scheduled for you task because iOS also uses GPU for UI rendering and some background work. But it clearly shows the difference between CPU and GPU performance for parallel execution tasks.

Render pipeline doesn’t give programmer ability to customize dispatching but compute pipeline does. MTLComputeCommandEncoder has two dispatch methods

func dispatchThreadgroups(
    _ threadgroupsPerGrid: MTLSize,
    threadsPerThreadgroup: MTLSize
)

func dispatchThreads(
    _ threadsPerGrid: MTLSize,
    threadsPerThreadgroup: MTLSize
)

Size of the grid is the amount of threads scheduled for execution. Grid can be of 1D, 2D or 3D size. For image processing we usually use 2D grids for width and height of processed image and run a separate thread for each pixel. Threads in grid are split into thread groups each of which is split into SIMD groups. These methods of compute encoder allow you to fine tune how much threads will be in each thread group and how much threadgroups will form the entire grid. You may even dispatch a single thread for the whole GPU task and execute a fully linear algorithm as you do it usually in Swift on CPU. But we won’t do this in our next shader :)

Combining threads to SIMD and thread groups has another advantage. You can initialize shared memory for thread group or SIMD group right inside shader code and it will be shared across threads in group. Also you may set a barrier for memory access so that every thread inside group will wait for other threads to come to this point. And they will continue only after all the threads will execute barrier function. This is waht we will use for unordered dithering.

Error Diffusion Dithering

As I told earlier error diffusion dithering algorithms or simply unordered dithering propagate error from one pixel to neighboring ones. There are plenty of these algorithms but the main difference between them is in how error is distributed to neighbors. You may find several examples of them in this article which I used as a starting point for my app.

We will implement the most popular algorithm named after its authors Floyd-Steinberg Dithering. To illustrate it lets consider we have grayscale image which we want to convert to black and white dithered image. So our pixel has only one channel and we need to convert its value to zero or one.

Here currently processing pixel is noted as (XX) and numbers around it is how much of error we propagate to neighbors.

      (XX)  7/16
3/16  5/16  1/16

Error is simply the difference between our pixel value and closest value from palette. In our case error will lie in -0.5...0.5 range. If pixel value is 0.4 the error will be 0.4 and if it is 0.6 the error will be -0.4. After calculating the error we propagate 7/16 of it to the right pixel and other 9/16 to three bottom pixels. Summing these values you’ll see that we propagate the whole error (16/16) to neighbors.

This algorithm can be described in 7 steps:

Take current pixel color;
Add error already propagated from previous pixels to it;
Find closest color from palette to this summed value;
Calculate the difference between this value and the closest color, it will be the new error;
Assign closest color to current pixel;
Propagate error to neighboring pixel according to algorithm;
Repeat from step 1 for next pixel.

As you see this algorithm is linear. The error propagated from first pixel to the next one is used to find the new color for the second pixel. We even can’t find all the errors first and then calculate the pixel colors from them in parallel. Because the error from previous pixel has impact on the error value of current one. But notice that error is propagated only to neighboring pixels, so we can wait when the first thread will go a little bit further and calculate all the required errors. And then we’ll start a second thread. Here (XX) is the first thread and (YY) is the second. And all the already processed pixels marked as (..).

(..)  (..)  (..)  (XX)  7/16
(YY)  (..)  3/16  5/16  1/16

When the first thread processed three pixels the second thread can start execution because the first one assigned all the errors for the right pixel of YY thread. So we actually can run them simultaneously! When YY thread will finish processing of the first pixel in a row the XX thread will go one pixel further and YY will have all the necessary data to continue processing.

I found this algorithm implemented in OpenCL here and if you want to have better understanding of it take a look at this article. I will focus on its Metal implementation but not the algorithm itself.

Lets dive right into it first and then I’ll explain the details.

[[ kernel ]]
void floydSteinbergDither(texture2d<float, access::read> source [[ texture(0) ]],
                          texture2d<float, access::write> destination [[ texture(1) ]],
                          device float3* errorsBuffer [[ buffer(0) ]],
                          constant float3* palette [[ buffer(1) ]],
                          constant ushort& paletteCount [[ buffer(2) ]],
                          const uint threadIndex [[ thread_position_in_grid ]],
                          const uint gridHeight [[ threads_per_grid ]]) {
    const int width = int(source.get_width());
    const int height = int(source.get_height());

    const int startX = int(threadIndex) * -3;
    const int startY = int(threadIndex);

    for (int x = 0; x < (width + int(gridHeight) * 3); x++) {
        const int2 position = int2(startX + x, startY);
        const bool isXInside = position.x >= 0 && position.x < width;
        const bool isYInside = position.y >= 0 && position.y < height;
        if (isXInside && isYInside) {
            const float3 readError = errorsBuffer[position.y * width + position.x];
            const float4 sourceColor = source.read(uint2(position));
            const float3 errorColor = saturate(sourceColor.rgb + readError);
            const float3 destinationColor = closestColor(errorColor, palette, paletteCount);
            destination.write(float4(destinationColor, sourceColor.a), uint2(position));

            const float3 writeError = errorColor.rgb - destinationColor;

            if (position.x < width - 1) {
                errorsBuffer[position.y * width + position.x + 1] += writeError * (7.0f / 16.0f);
            }

            if (position.y < height - 1) {
                const int baseOffset = (position.y + 1) * width + position.x;
                if (position.x > 0) {
                    errorsBuffer[baseOffset - 1] += writeError * (3.0f / 16.0f);
                }

                errorsBuffer[baseOffset] += writeError * (5.0f / 16.0f);

                if (position.x < width - 1) {
                    errorsBuffer[baseOffset + 1] += writeError * (1.0f / 16.0f);
                }
            }
        }

        threadgroup_barrier(mem_flags::mem_device);
    }
}

First of all we pass error buffer to shader. Notice device attribute near errorBuffer parameter in shader signature. It means that memory allocated for this buffer will be in the device address space and can be visible for all the threads no matter which thread group they are dispatched to. Other buffers are in constant address space and so can’t be writable.

Also notice that we use [[ thread_position_in_grid ]] and [[ threads_per_grid ]] modifiers here. Compute shaders can give you all the information about which SIMD group and thread group they are in as well as the size of these groups. So you can write any custom logic using this information if you want to make some kind of interaction between threads in different groups.

Here each thread process a single row of pixels of our image. In Metal we can’t simply put threads in waiting state (or maybe I don’t know how to do it), so all the threads start execution simultaneously but with -3 offset from previous one. So all the threads starting from the second will start execution outside image bounds.

Then we iterate in for-loop through all the pixels in a row and if we are inside image bounds perform error calculation and assign pixel color from palette to destination texture. Also we update error stored in the buffer so that the next threads will be able to find the final calculated error there when they start processing theirs pixels.

The most interesting part is the final line inside for-loop

threadgroup_barrier(mem_flags::mem_device);

We have a lot of logic inside if (isXInside && isYInside) statement. So for bottom rows of our image this part of code won’t be executed for a certain amount of time while they are outside the image bounds, these threads can get ahead of the previous ones and we can start processing bottom pixels before all the required error for them is calculated and stored in buffer. That’s why we use barrier here, and these out of bounds threads will wait when currently processing threads will finish all the error calculating. mem_flags::mem_device means that they need to wait while device memory will be written. Remember that errors buffer is lied in this address space.

And here is the result! God of War Ragnarök art processed with Floyd-Steinberg dithering. There are no patterns there because each pixel calculated with different amount of error propagated to it. Also you may see that in the third image which used BWRGB colors Atreus’ armor is more yellow than in ordered dithered image. But there is no yellow color there BOY! Check the last image if you don’t believe me :) Error diffusion algorithms preserve color information better because all the unused color data for pixel is stored in error and propagated to other pixels.

Performance Notice

If you followed carefully along this post you saw that Floyd-Steinberg algorithm is not very efficient. We do a lot of unnecessary work there while processing out-of-bounds pixels and wait for in-bounds pixels to finish processing. This is because error diffusion algorithms are not made for GPU and can perform better on CPU in a single thread with way more processing power. I didn’t make measurements though :)

But sometimes you need to put such a linear algorithms inside GPU pipelines and you have two options to do it:

Make a CPU stop inside GPU processing. You can make two command buffers with CPU step between them or use MTLEvent to stop GPU execution, perform calculations on CPU and then turn back to GPU execution;
Try to apply algorithm to GPU processing even in a not efficient way.

With CPU stop there may be a lot of troubles. First of all you pipeline architecture may not allow to make another command buffer. For example you use node architecture for processing and each node receives already created command buffer and schedule tasks to it. You’ll need to change all your architecture to make second buffer just for one edge case. MTLEvent on the other side while allow you to use single command buffer, can make a huge impact on pipeline execution time.

Since GPU programming is all about performance you need to measure any single case and choose the best option for you. But sometimes you will find that not efficient GPU algorithm performs better than highly efficient CPU one just because you don’t need to make a switch between processors.

That’s all for now! Hope that you enjoy this long post and it will help you to make stunning results in you apps using Metal. If you have any questions feel free to ask them via email contact@mtldoc.com or connect via Twitter @petertretyakov.

Thank you for reading!

SwiftUI Gradient Builder

Fri, 19 Aug 2022 09:00:00 +0000

In the last post about gradient shader I covered GPU part of gradient rendering in Metal but if you’ll use it in your apps it won’t make enough sense without giving user the ability to build a gradient in UI. I’m a big fan of SwiftUI and in my opinion current state of this new UI framework including iOS16 API gives all the necessary tools to build apps of any kinds and complexity. Today I’ll show you how to create this fancy control to construct gradients purely on SwiftUI.

TL; DR

I know that a lot of you folks prefer Swift over English ;) I’m also this kind of guy so in order not to torture you with a lot of text here is a repo with all the code from this post and you can go through the implementation by yourself. The repo also contains the CPU part of gradient rendering which I skipped in previous post.

Decomposing Control

For our gradient control there will be four separate views packed in a single root view.

GradientView will contain only the linear gradient with some UI modifications;
StopsView will have ForEach loop with all the stops placed on top of GradientView;
StopView is a host view for a single stop;
SelectedStopView is the bottom view for current stop color description and modification.

Why view decomposition in SwiftUI matters? It is not only about readability and reusability of your code. SwiftUI view lifecycle has two main stages. Like any Swift struct it has init which will be called each time you create a view in other view’s or scene’s body and it has this body which will be fired each time view’s state changes. So your main goal as a developer is to limit unnecessary body calls when view state doesn’t actually changed but SwiftUI thinks it does. How it can happen?

Actually any change in @State or @Binding in view or @Published variable of ObservableObject will require body call on next frame. That means that if you have a huge view described in a single view’s body in all this fine declarative way the whole body will be called each time you change the title of one tiny button. So proper decomposition not only allows you to maintain code easily but makes it faster!

The last thing before diving into views’ implementation we need to define the data model.

Gradient Stop Model

We won’t use SwiftUI Gradient.Stop as data model for two reasons. First since this control is made for gradient shader from previous post we need color to be defined as SIMD4<Float> but not SwiftUI Color. Second we need to support selection of particular stop to allow user to change the color. And to preserve selection while changing color or location we need our model to conform to Identifiable protocol and has some static unique ID.

struct GradientStop: Identifiable {
    let id: UUID
    var location: Float
    var color: SIMD4<Float>

    init(location: Float, color: SIMD4<Float>) {
        self.id = UUID()
        self.location = location
        self.color = color
    }
}

⚠️ The full Gradient model contains gradient type and rotation angle but I omit them in this post. Also we need to implement conversion between SIMD color and SwiftUI color which I also won’t show here to keep focus on UI related code. You can find the full implementation in repo.

GradientView

This view is pretty straightforward. It just renders linear gradient inside RoundedRectangle with semitransparent border overlay. And on tap gesture the new stop will be added to stops array between last two elements. It is the only problem with current implementation since TapGesture in SwiftUI doesn’t provide the location of tap. In iOS16 there is a new SpatialTapGesture added which provides the location but I wanted to keep iOS15 as deployment target so that you’ll be able to run the code from repo on your current devices.

struct GradientView: View {
    init(stops: Binding<[Gradient.GradientStop]>) {
        self._stops = stops
    }

    var body: some View {
        RoundedRectangle(cornerRadius: 8, style: .continuous)
            .fill(
                LinearGradient(
                    stops: self.stops.map(\.ui),
                    startPoint: .leading,
                    endPoint: .trailing
                )
            )
            .overlay(
                RoundedRectangle(cornerRadius: 8, style: .continuous)
                    .stroke(lineWidth: 2)
                    .foregroundColor(.white.opacity(0.7))
            )
            .frame(height: 32)
            .onTapGesture {
                let stops = self.stops.suffix(2)
                let location = stops.reduce(0) { $0 + $1.location } / 2.0
                let color = stops.reduce(.zero) { $0 + $1.color } / 2.0

                self.stops.insert(
                    Gradient.GradientStop(
                        location: location,
                        color: color
                    ),
                    at: self.stops.count - 1
                )
            }
    }

    @Binding private var stops: [Gradient.GradientStop]
}

StopsView

Above gradient view there is a container view for all the stops. It is also rather easy to implement. Just ForEach loop on all the stops. But since our model define stop location in normalized coordinates between 0 and 1 we need to provide the parent view width to this view to place stops in proper locations. Also this view receives binding to selected stop ID and proxies it to all the StopViews to support selection.

struct StopsView: View {
    init(
        width: CGFloat,
        stops: Binding<[Gradient.GradientStop]>,
        selectedStopID: Binding<UUID?>
    ) {
        self.width = width
        self._stops = stops
        self._selectedStopID = selectedStopID
    }

    var body: some View {
        ForEach(self.$stops) { stop in
            StopView(
                stops: self.$stops,
                stop: stop,
                selectedStopID: self.$selectedStopID,
                width: self.width
            )
        }
    }

    private let width: CGFloat
    @Binding private var stops: [Gradient.GradientStop]
    @Binding private var selectedStopID: UUID?
}

This is another reason why we need to conform GradientStop to Identifiable protocol. ForEach supports iteration on array binding and inside its content stop will also be a binding. But to be able to call this initializer your data for iteration needs to be Identifiable.

StopView

This view is just a circle with current stop color, border and shadow. But it supports stop dragging and deletion. To place the stop on proper position it also receives the same width as StopsView and to support stop removing and prevent user from deleting the last two stops we need access to full stops array. Also we need full array to sort it during drag gesture because SwiftUI requires stops to be properly ordered based on location while rendering GradientView.

struct StopView: View {
    init(
        stops: Binding<[Gradient.GradientStop]>,
        stop: Binding<Gradient.GradientStop>,
        selectedStopID: Binding<UUID?>,
        width: CGFloat
    ) {
        self._stops = stops
        self._stop = stop
        self._selectedStopID = selectedStopID
        self.width = width
    }

    var body: some View {
        Circle()
            .foregroundColor(Color(self.stop.color.cg))
            .frame(width: 24, height: 24)
            .overlay(
                Circle()
                    .stroke(lineWidth: 2)
                    .foregroundColor(self.selectedStopID == self.stop.id ? .accentColor : .white.opacity(0.7))
            )
            .shadow(color: .black.opacity(0.1), radius: 2, x: 0, y: 1)
            .position(x: self.width * CGFloat(self.stop.location))
            .gesture(self.dragGesture)
            .contextMenu {
                if self.stops.count > 2 {
                    Button("Delete") {
                        self.stops.removeAll { $0.id == self.stop.id }
                    }
                } else {
                    EmptyView()
                }
            }
            .onTapGesture {
                self.selectedStopID = self.stop.id
            }
    }

    private let width: CGFloat

    @Binding private var stops: [Gradient.GradientStop]
    @Binding private var stop: Gradient.GradientStop
    @Binding private var selectedStopID: UUID?
    @GestureState private var initialLocation: Float?

    private var dragGesture: some Gesture {
        DragGesture()
            .updating(self.$initialLocation) { _, state, _ in
                guard state == nil else { return }
                state = self.stop.location
            }
            .onChanged { value in
                let location = (self.initialLocation ?? 0) + Float(value.translation.width / self.width)
                self.stop.location = location.clamped(to: 0 ... 1)
                self.stops.sort { $0.location < $1.location }
            }
    }
}

There is a nice hint which I found while implementing this view. You may already know that order of view modifiers matters in SwiftUI because every modifier returns some kind of View and the next one receives already modified view. If you’re not familiar with this behavior try to make Rectangle in SwiftUI and add .foregroundColor and .padding in different order you’ll get different results. And for me it was clear with UI related modifiers. But I was surprised when I got infinite loop, 100% CPU load and stuck UI during drag gesture when .gesture modifier was placed before .position one. I’m still not sure why this happened but since view’s body is called every frame while dragging there was some kind of position conflict I guess. Gesture’s .onChanged action was infinitely called as well as body after it. Maybe it was because gesture updates position but the actual update comes only on the next frame and .position modifier places the view on previous position. So the resulting view doesn’t receive any position update and gesture tries to update it again and again. If you can explain this strange behavior from engineering perspective but not from magical one please add it to comments I would really appreciate it :)

In this view I also used @GestureState to store initial location of the stop when drag begins. This wrapper is very useful when you work with gestures and need to store some transient variable during gesture recognition. You need to implement .updating method for each @GestureState. Its closure receives current state of the whole gesture with initial location, translation and so on, current state of @GestureState variable as inout parameter and animation transaction. So you update this inout parameter inside closure and it is stored in state variable. Pretty weird API as for me but it works. Another advantage of @GestureState is that it automatically reset to initial value when gesture ends.

And on tap gesture we simply assign current stop’s ID to selectedStopID variable. Pretty easy. The final view before combine them all together is the view which shows selected stop color value and allows to change it.

SelectedStopView

struct SelectedStopView: View {
    init(
        stops: Binding<[Gradient.GradientStop]>,
        stopID: Binding<UUID?>
    ) {
        self._stops = stops
        self._stopID = stopID
    }

    var body: some View {
        HStack {
            ForEach(self.channels, id: \.title) { channel in
                Text("\(channel.title): \(channel.value)")
            }

            ColorPicker(
                selection: self.color,
                label: EmptyView.init
            )
            .disabled(self.stopID == nil)
        }
        .font(.system(size: 12))
        .foregroundColor(self.stopID == nil ? .gray : .black)
    }

    @State private var placeholder: Color = .clear
    @Binding private var stops: [Gradient.GradientStop]
    @Binding private var stopID: UUID?

    private var stop: Binding<Gradient.GradientStop>? {
        self.$stops.first { $0.id == self.stopID }
    }

    private var color: Binding<Color> {
        if let stop = self.stop {
            return stop.color.ui
        } else {
            return self.$placeholder
        }
    }

    private var channels: [(title: String, value: String)] {
        func channelValue(_ value: Float?) -> String {
            value.map { String(describing: Int($0 * 255)) } ?? "-"
        }

        let color = self.stop?.wrappedValue.color

        return [
            ("R", channelValue(color?[0])),
            ("G", channelValue(color?[1])),
            ("B", channelValue(color?[2])),
            ("A", channelValue(color?[3])),
        ]
    }
}

I think everything’s clear here. We just show integer value for each color channel and SwiftUI ColorPicker. Since color picker doesn’t allow optional color we need placeholder binding with clear color for the case when no stop is selected. Let’s combine all these four views into a single one root view.

GradientBuilderView

A warm welcome to GeometryReader! I’m not a big fan of it but sometimes we need plain old frames like in ancient UIKit times :) Here we need to know the view width to place stops properly inside it. If you’ll have fixed width for this builder you don’t need to wrap it in GeometryReader of course.

struct GradientBuilderView: View {
    init(gradient: Binding<Gradient>) {
        self._gradient = gradient
    }

    var body: some View {
        VStack {
            GeometryReader { proxy in
                GradientView(stops: self.$gradient.stops)
                    .frame(width: proxy.size.width, height: 32)

                StopsView(
                    width: proxy.size.width,
                    stops: self.$gradient.stops,
                    selectedStopID: self.$selectedStopID
                )
                .frame(width: proxy.size.width, height: 24)
                .offset(y: 16)
            }
            .frame(height: 32)

            SelectedStopView(
                stops: self.$gradient.stops,
                stopID: self.$selectedStopID
            )
        }
    }

    @State private var selectedStopID: UUID?
    @Binding private var gradient: Gradient
}

This view also contains segmented control for gradient type selection and rotation angle slider but since they are default SwiftUI views there is no need to show them here.

And that’s it! The whole view with all the logic and UI took about 200 lines of code! The full implementation of this control and gradient shader you can find on GitHub particularly here. Thank you for reading! If you have any questions feel free to write me an email to contact@mtldoc.com or connect via Twitter @petertretyakov. Next time there will a post about Metal, I promise ;)

Shaders Explained: Gradients

Thu, 04 Aug 2022 09:00:00 +0000

Write functional and performant shaders is the basic skill for GPU developer no matter what framework or engine he or she uses. But there are not much information about it on the web especially if we talk about Metal. Though shading languages are pretty similar and hours spent on ShaderToy can give you some inspiration and knowledge about OpenGL shaders and you can easily convert them to Metal I personally feel a huge lack of structured information about shader writing. So in this article and others under the title Shaders Explained I’ll go through different shaders in detail, teach you some shader technics and explain my motivation under each line of code. Hope that it will make you and me better shader programmers.

For the first article I chose a pretty basic gradient shader which I implemented for one of my personal projects. Gradients are at the very heart of GPU programming and every shader tutorial starts with the triangle colored with gradient from red, green and blue. But we won’t build yet another Hello Triangle tutorial. Instead I’ll show you how to make gradient shader to fill texture with arbitrary gradient stops with different locations and colors for three basic gradient types. Additionally we’ll add support for rotation. Actually we will implement the most part of CAGradientLayer or SwiftUI Gradient from scratch with broad focus on its GPU side.

CPU

Draw a gradient in iOS is pretty simple. All the available implementations have the similar API. You define an array of gradient stops which contains location in normalized 1D coordinate space (from 0 to 1) and a color. CAGradientLayer asks for separate arrays with locations and colors and SwiftUI uses Gradient.Stop struct with CGFloat location and SwiftUI Color for it. But we won’t use SwiftUI color or CGColor or other UI related colors available in iOS since they are not available on GPU and Metal Shading Language does know nothing about them. Instead we will use array of four Float values packed in a single SIMD register because Metal has the same type float4. Also our colors need to be defined in 0 to 1 range for each channel and not from 0 to 255. Metal has support for unsigned integers for color definition but the common practice is to use floating points.

struct GradientStop {
    let location: Float
    let color: SIMD4<Float>
}

Colors between gradient stops are interpolated linearly. So in the middle between stop with location 0 and opaque black color (0, 0, 0, 1) and the next stop with location 1 and white color (1, 1, 1, 1) there will be gray color with RGBA values 0.5, 0.5, 0.5, 1. Besides stops we need support for three types of gradients and rotation angle so the whole gradient structure will look like this.

struct Gradient {
    enum GradientType: UInt8 {
        case linear
        case radial
        case angular
    }

    struct GradientStop {
        let location: Float
        let color: SIMD4<Float>
    }

    let type: GradientType
    let stops: [GradientStop]
    let rotationAngle: Float
}

Instead of rotation angle we can add support for the full range of transform matrices with scale and translation but this is outside of our topic for now. I won’t cover the full CPU side implementation here with render pipeline creation and encoding because it is highly dependent on how you work with Metal in your app. Also it would require to explain the huge part of Metal’s Swift API. You can find it in Metal by Example blog or Ray Wenderlich’s Metal by Tutorials. And I want to focus on shader implementation here.

GPU

To make a gradient fill on GPU we need to write two render shaders. The first one will use stops and produce an actual gradient on 1D texture (texture with the height of 1 pixel) and the second will stretch it to final texture according to gradient type and rotation angle. The main problem which took me quite a while to solve is how to wrap a gradient to the final texture differently based on type.

But first things first. Lets write the gradient shader.

1D Gradient Texture

struct GradientTextureVertex {
    float4 position [[ position ]];
    float4 color;
};

[[ vertex ]]
GradientTextureVertex gradientTextureVertex(constant float* positions [[ buffer(0) ]],
                                            constant float4* colors [[ buffer(1) ]],
                                            const ushort vid [[ vertex_id ]]) {
    return {
        .position = float4(fma(positions[vid], 2.0f, -1.0f), 0.0f, 0.0f, 1.0f),
        .color = colors[vid]
    };
}

[[ fragment ]]
float4 gradientTextureFragment(GradientTextureVertex in [[ stage_in ]]) {
    return in.color;
}

For those of you familiar with Metal Shading Language this shader doesn’t require further explanations. For Metal newcomers I’ll briefly explain some details. But if this article is the first about Metal for you I highly recommend to read first two parts of Up and Running with Metal from Warren Moore here and here before continue. It will give you Metal basics required to understand what is happening here. Also he has the best book about Metal available so check it out.

[[ vertex ]] and [[ fragment ]] modifiers tell the compiler what these functions are for. The first one will be called as many times as you asked in render encoder’s drawPrimitives function. Usually once per vertex in your scene geometry but in our case it will be called once per gradient stop. Vertex function has to return something with vertex position in homogeneous coordinate space. It can be only float4 value or any other struct with position in it. You need to tell the compiler where position is placed in this struct using [[ position ]] modifier.

All the values passed from Swift code to GPU shader need to be packed in Metal buffers. They are just boxes for raw data without any type information. It is your responsibility to put properly aligned data in buffers and use the corresponding type in shaders. If you put UInt8 in buffer but shader expects ushort you will have a crash. In vise versa situation shader will run but you won’t get the expected result and these types of bugs are really hard to find and debug, so be careful. In our shader we pass all the stops’ locations in the first buffer with index 0 and all the colors in second one. But since our locations are in range from 0 to 1 we need to make two additional modifications to them.

First we need values for Y, Z and W in homogeneous coordinates. We use zero for Y because we render texture with one pixel height and zero for Z because there is no depth in our scene, we just render in 2D. W needs to be one because rasterizer which converts geometry data to pixels on screen will divide XYZ by W before passing the final values to fragment function. You definitely need to understand this concept when you’ll write shaders for 3D but in our case it is not required.

Second step is to convert normalized 0...1 coordinate space to Metal clip space where X and Y axes has range from -1 to 1 and Z axis from 0 to 1. We use fma function for it which is a shortcut for Fused Multiply-Add. It has three parameters (x, y, z) and returns x * y + z. Of course we can use positions[vid] * 2.0f - 1.0f instead of fma but fma makes it in one processor instruction with a single rounding after the whole operation.

Fragment function will be called once per pixel in rendered texture. In our case we just return the color associated with that pixel. The value returned from vertex function will be passed to fragment function in [[ stage_in ]] parameter. The magic is happening between these two functions. Metal rasterizer will interpolate all the colors and pass the interpolated value to each fragment function call. That means if we have two gradient stops with locations 0 and 1 and black and white colors respectively and render them to texture with 100 pixels width and 1 pixel height fragment function will be called 100 times and in each call the color will be interpolated between float4(0, 0, 0, 1) and float4(1, 1, 1, 1). On the 10th pixel the color passed to fragment function will have value float4(0.1, 0.1, 0.1, 1), on 50th the value will be 0.5 for RGB channels and so forth. As you see we don’t need to make interpolation by ourself, GPU will do it faster and with less errors :)

After this step we have a shiny new 1D texture with gradient properly colored according to stops. Lets wrap it to the final texture.

2D Gradient Texture

All three types of gradient which our shader will support follow the same implementation pattern. We will use UV coordinates to wrap gradient to the final texture and apply rotation to UV while keeping position in place. I’ll explain later why. In fragment shader we will use sampler with clamp_to_edge address mode and all we need to do is to properly convert UV to position on 1D gradient texture according to gradient type.

struct GradientVertex {
    float4 position [[ position ]];
    float2 uv;
};

enum GradientType: uchar {
    kLinear,
    kRadial,
    kAngular
};

constant float2 positions[4] {
    { -1.0f,  1.0f },
    { -1.0f, -1.0f },
    {  1.0f,  1.0f },
    {  1.0f, -1.0f }
};

constant float2 uvs[4] {
    { 0.0f, 0.0f },
    { 0.0f, 1.0f },
    { 1.0f, 0.0f },
    { 1.0f, 1.0f }
};

[[ vertex ]]
GradientVertex gradientVertex(constant float2x2& rotationTransform [[ buffer(0) ]],
                              const ushort vid [[ vertex_id ]]) {
    return {
        .position = float4(positions[vid], 0.0f, 1.0f),
        .uv = (uvs[vid] - float2(0.5f)) * rotationTransform + float2(0.5f)
    };
}

[[ fragment ]]
float4 gradientFragment(GradientVertex in [[ stage_in ]],
                        texture2d<float, access::sample> gradientTexture [[ texture(0) ]],
                        constant GradientType& gradientType [[ buffer(0) ]]) {
    constexpr sampler s(filter::linear, coord::normalized, address::clamp_to_edge);

    float2 texCoords;
    switch (gradientType) {
        case kLinear:
            // TODO: Assign `texCoords` for Linear
            break;
        case kRadial:
            // TODO: Assign `texCoords` for Radial
            break;
        case kAngular:
            // TODO: Assign `texCoords` for Angular
            break;
    }

    return gradientTexture.sample(s, texCoords);
}

Why we can’t use just position and need separate UV coordinates here? The reason is not only because positions are in -1...1 range and UVs are in 0...1 range. If it was the only issue we could just use the same fma function to convert values. Actually positions in fragment shader will be with absolute values. Since it runs once per pixel you’ll have the position of that pixel during execution. The second reason is that pixels outside of our rotated geometry won’t be rendered. Here on the first image we rotate vertex positions and on the second UVs. Also since rotation is performed from center of coordinates we need to subtract 0.5f from UVs first, perform rotation and add it back again. Without this shift UVs will be rotated around top left corner.

So we create proper geometry in vertex shader, let’s move to fragment. By now we have only 1D texture with gradient and we need to stretch it to final texture no matter what size it will have. Here is how we will stretch it based on gradient type for linear, radial and angle gradients respectively. The colored line is our 1 pixel height gradient texture, I made it larger for illustration purposes.

For linear gradient it’s pretty straightforward, we just assign UV to texCoords. Because of clamp_to_edge address mode for all the UVs outside 0...1 range sampler will return closest edge value. It is not an issue without rotation since all the pixels on final texture will have UVs in proper range but if we apply rotation UVs on corners will have values below zero or above one. With edge clamping every pixel will be filled with proper color.

texCoords = in.uv;

Radial gradient runs uniformly from the center so we need to calculate how far we are from it in current pixel. Since UVs are normalized the center is always at float2(0.5f, 0.5f). To find the distance between two points we need to put the center of coordinates to the second one by subtracting the second from first and calculate vector length. Inside the gradient circle this length will be from 0 to 0.5 so we need to multiply it by 2 to receive the full 0...1 range for sampling. And you already understand that pixels outside of the circle will be filled with the color from last pixel of gradient texture because of clamping.

texCoords = length(in.uv - float2(0.5f)) * 2.0f;

With angular gradient things are a bit more complicated. We need to calculate an angle between two vectors. The first one will be the same vector from center of texture to current pixel and the second one will be the horizontal vector with positive X direction. When we will find the angle we need to normalize it and take the sample from gradient texture with this normalized position. To find an angle there is a popular atan2 function. You may use it in CoreGraphics while manipulating with CGPoints or in any other graphics frameworks. atan2 returns value in -PI...PI range and normalization is done by dividing by PI and our old friend fma.

const float2 offsetUV = in.uv - float2(0.5f);
const float angle = atan2(offsetUV.y, offsetUV.x);
texCoords = float2(fma(angle / PI, 0.5f, 0.5f), 0.0f);

And that’s it! You’re now fully prepared to implement CAGradientLayer or SwiftUI Gradient from CPU API up to GPU rendering. Here is the final Gist with the full shader implementation. If you have any questions feel free to shoot me an email to contact@mtldoc.com or connect via Twitter @petertretyakov.

And one more thing :) Besides gradient shader I’ve also made this nice Gradient builder purely on SwiftUI. I covered it in the separate post here. Also you can find a link to demo app there with full implementation of this shader.

Build Swift Executable with Metal Library

Sat, 18 Jun 2022 09:00:00 +0000

When you build a Swift Package Xcode will compile all your .swift files to single executable file and process all other resources which includes Assets catalogue, Core Data files, Storyboards and many more including all .metal files. All these resources will be put in package Bundle and you have access to them through Bundle.module if you add them to resources array in package’s target.

But it will work only for library product type. If you want to build a Swift executable product resources won’t be processed or copied to Bundle because there is no actual Bundle there. So Swift executable is limited to only .swift files so you can’t build executable which will run you Metal shaders. Let’s try to fix it!

Metal allows offline library compilation and library initialisation from URL with MTLDevice instance method

func makeLibrary(URL url: URL) throws -> MTLLibrary

This method requires a valid URL to .metallib file so we need a way to build Metal Library from our .metal files, put it somewhere near our executable file and create library on the fly from it.

Let’s do it using shell script! You can do it manually on you Mac or run shell script on your CI assuming that CI is also a Mac server.

You can get the whole script here or go step by step with me.

Create intermediate representation from Metal files

First of all we need to locate all our .metal files and create .air file from each of them

#!/bin/bash

METAL_FILES=`ls /path/to/metal/files/*.metal`

for path in $METAL_FILES
do
  FILE_NAME=${path%%.*}
  xcrun -sdk macosx metal -c $path -o ${FILE_NAME}.air
done

FILE_NAME here will be the name of Metal file but without .metal extension. xcrun command will build .air file from each .metal file and put in the same folder where all your Metal files exist. We will remove them after we compile the whole library.

If you want to put .air files somewhere else just update -o flag to /path/to/air/files/${FILE_NAME}.air

Also if you have only one Metal file (assume it is named Shaders.metal) you can omit for-loop and run only

xcrun -sdk macosx metal -c Shaders.metal -o Shaders.air

Archive AIR files to single MetalAR file

The next step will be to archive all .air files to single .metalar file which will be used to compile the .metallib file later on. You can also omit this step if you have a single .metal file which you already compiled to single .air file. The Metal Library can be created from .air or .metalar file it doesn’t matter.

This step will look pretty like the previous but we will iterate through .air files this time

AIR_FILES=`ls /path/to/air/files/*.air`

for path in $AIR_FILES
do
  xcrun metal-ar r default.metalar $path
done

I used default.metalar name here like Xcode do when compiles Metal Library during package or app building. But you can use the name more appropriate for you of course. And you can put it anywhere you want if you replace default.metalar with /path/to/metalar/file/default.metalar.

metal-ar is the name of the tool which will archive .air files and r command will add the second file ($path in our case) to the first one and create it if needed.

So now we have a single .metalar file with all our Metal shaders in intermediate representation. Let’s build a Metal Library file from it.

Build Metal Library from MetalAR archive

This step is pretty straightforward. Again we need to run xcrun command to do the thing

xcrun -sdk macosx metallib default.metalar -o default.metallib

So now we have a .metallib file which is a good candidate to create MTLLibrary from. But the last but not least step is to be good citizens and clean up the shaders folder where we created all our intermediate files.

Clean up

rm default.metalar

for path in $AIR_FILES
do
  rm $path
done

Create MTLLibrary from `.metallib` file

When you archive you Swift executable Xcode will export it as a folder with your single executable file in Products/usr/local/bin folder inside. You can move it anywhere you want and run it from shell using path to executable file or if you add the executable location to $PATH you can run it from anywhere in your system. But we need a way to tell the executable file where our .metallib is located.

There are two ways to do it. If you create Swift executable you are already familiar with Apple’s Swift Argument Parser framework which is must-have dependency for Swift executables. So you can create an @Option for Metal Library file location which will be required for you script or if you build an executable for internal usage you can assume that .metallib file will be placed near your executable file and create library from it.

There is one thing you need to notice if you use the second approach. If you use

FileManager.default.currentDirectoryPath

in your Swift code it will return you not the executable’s location but the current location from where you run your executable. So if you open the Terminal and run

/path/to/your/swift/executable

your current directory will be your user’s home directory and FileManager won’t be able to locate .metallib file which is placed near your executable but not in your home folder.

To get access to executable’s location you need to get process info from inside your run() method in Swift file. So the valid approach to locate .metallib file which is placed near executable will be

func run() throws {
    guard let execPath = ProcessInfo.processInfo.arguments.first,
          let device = MTLCreateSystemDefaultDevice()
    else { return }

    let execURL = URL(fileURLWithPath: execPath)
    let execRootURL = execURL.deletingLastPathComponent()
    let libraryURL = execRootURL.appendingPathComponent("default.metallib")
    let library = try device.makeLibrary(URL: libraryURL)
    // PROFIT!
}

execPath and execURL here will be the full path to executable itself so we need to remove its name from the URL to get the path where it is located and where we put our .metallib file.

Conclusion

Now you’re not limited by Swift only files inside Swift executable and can build an executable which can run Metal Shaders right from the command line.

I personally used this flow to create the Swift Package with two products. The first one is a plain old Swift library which I can add as dependency to iOS apps and the second one is executable which has the first library as dependency and can be used to run the whole library functionality from command line. We use it to simplify library tests and to make the library available to run from Python scripts for example.

Here is the gist with all the previous shell code. I hope it will help you in creating awesome Swift packages with CLI availability.

If you have any questions feel free to shoot me an email to contact@mtldoc.com or connect via Twitter @petertretyakov.

MTLDoc

Optimizing Metal: Stitch several Metal functions into a single shader using MTLFunctionStitchingGraph

What is MTLStitchingGraph?

Stitchable Function

Stitching Graph

Graph Description

Compiling the Graph

Calling the Graph (Predefined Arguments)

Calling the Graph (Arbitrary Arguments)

Notes about Nodes

Conclusion

Code Signing Hell with XCFramework

Asset validation failed

XC++Framework

Codesign for the Resque

Conclusion

Shaders Explained: Dithering

Why Dithering?

Ordered Dithering

Few Words About Dispatching

Error Diffusion Dithering

Performance Notice

SwiftUI Gradient Builder

TL; DR

Decomposing Control

Gradient Stop Model

GradientView

StopsView

StopView

SelectedStopView

GradientBuilderView

Shaders Explained: Gradients

CPU

GPU

1D Gradient Texture

2D Gradient Texture

Build Swift Executable with Metal Library

Create intermediate representation from Metal files

Archive AIR files to single MetalAR file

Build Metal Library from MetalAR archive

Clean up

Create MTLLibrary from .metallib file

Conclusion

Create MTLLibrary from `.metallib` file