I had the pleasure of chatting with the CEO of ATC / Enhanced Radar recently. They built an iOS app that lets you listen to live and past Air Traffic Control audio. It’s been charting well in the Travel section of the US App Store.

The app is really cool. But the onboarding bothered me. So here are my thoughts on how to improve it. I’m still learning and these are my humble opinions.

Steal Like An Artist

Before critiquing any app, it’s nice to get a feel for the space first. What does world class look like? Who’s the niche cult favorite? What’s the design language? Basically what does good even mean.

For ATC, I looked at other apps charting in Travel with a focus on planes and Air Traffic Radio apps. That led me to four:

1. Flighty (Ranked #87 at the time)
2. Flightradar24 (Ranked #45 at the time)
3. FlightAware (Ranked #126 at the time)
4. PlaneFinder

Flighty was my north star. Cult following, Apple Design Award, pretty and easy to use.

The other three were interesting in their own right. Great reviews, press, and real value for users. They all did well enough to rank well, but I felt like their designs were a product of 2012 that no one has touched since. Yet, users didn’t seem to mind.

Findings

Anyways, after studying those apps, I had data to back up my feeling about ATC’s onboarding. It felt long. The app itself has a really cool and unique opening with live recordings playing and dropping you into a demo. That part feels great and fun. The gap between that and the paywall is my gripe.

Onboarding Screen Counts

1. Flighty = 0
2. Flightradar24 = 2 (Privacy + Paywall)
3. FlightAware = 4 (Privacy + Paywall + Tracking + Account)
4. PlaneFinder = 4 (Marketing + Notifications + Tracking + Paywall)
5. ATC = 20 (Demo + Notifications + Tracking + Paywall + Data Collection)

Based on the numbers, a good average is around three or four onboarding screens.

What I would change

I think ATC can collect the same data it already collects and present it more concisely.

The biggest issue is that the app is asking users to personalize the app before they’ve even tried it. This makes more sense after the paywall as the narrative shifts from “I downloaded this and now have to configure it” to “I own this and want to make it mine.” So I’d pull personalization out of onboarding entirely for ATC specifically.

Onboarding should not punish users for wanting to use the app. Half interested users should be able to judge your work on its merits, not on the front door.

Moreover, I’d combine the tracking and notification prompts into one screen.

Those two changes take ATC from 20 screens to just 4 screens.

Two of the remaining four are already hero moments that could be made even cooler. I don’t want to explain those flows in an article so I’d recommend you download the app and try it with the context of what I’ve written below.

Put It All Together

The app’s core idea is great, the opening demo is fun, and the modern UI for Air Traffic Control radio is appreciated, even for a non pilot like myself. The onboarding is just a cover for a book that doesn’t match its content. As of June 5, 2026, they’re also hiring for an iOS dev role if that’s your thing. Just email them or apply from the app.

Also, I have a Substack now. Where I write about my experience building Setto and other cool things. Setto is basically tinder for music recommendations. You pick a few artists, swipe on songs and the DJ figures out what you’re into. Last I checked, it was on the US Best New Apps list and Canada’s Hot This Week for both iPad and iPhone.

carlosmbe.substack.com

If you’d like me to analyze your app, share some advice or write about a specific topic. Send me an email or comment on this post. My details and calendar are on carlosmbe.com.

In case I don’t see you again, good afternoon, good evening, and good night.

Hello There! Ever wanted to quickly create 3D Models of Objects in your room using only the iPhone (or iPad) camera? If the answer is yes, then this is the article you’ve been looking for.

Quick note, this is an update of an article I wrote on my personal Hashnode Blog. I’ve decided to cross publish it here on Medium due to popular demand.

Alright, so what is Object Capture? In the olden days, also known as WWDC23, Apple released and announced a new feature in Reality Kit. Object Capture. It allows developers to create AR Experinces using 3D Models that users can create with their iPhone.

In other words, instead of using Blender or other 3D Workstations, we can now create objects for rendering in AR or VR or other Computer Graphics pipelines with the iPhone camera. Pretty neat for the “era of Spatial Computing.”

One approach to do this is taking multiple pictures of the object we’re scanning and use the Lidar sensor for depth. This graphic from the Apple Documentation illustrates it well.

Alternatively, you can also put the object on a turn table.

Anyways, now that you know the vision. Let’s talk about code.

Introduction

Here’s a smooth demo video of the final implementation. The gif below is just for illsutration.

To implement Object Capture in your app, target devices must be an iPhone or iPad with the following:

• A LiDAR Scanner
• An A14 Bionic chip or later
• iOS or iPadOS 18 or later

References

When building my implementation, I referenced the following articles, projects and videos.

SuperSimpleObjectCapture

Japanese Article Breaking Down Super Simple Object Capture

WWDC23: Meet Object Capture for iOS

Apple Official Sample Project — Scanning objects using Object Capture, Capturing photographs for RealityKit Object Capture

Show Me The Code!

Firstly, we need to request access to the user’s Camera before we can start scanning objects. To do this, we want to navigate to our Target’s Settings in Xcode.

Here’s Apple’s guide on doing that.

1. Navigate to your Project’s target
2. Go to the Target’s Info
3. Add Privacy - Camera Usage Description and write a description for it

Awesome!

Now we’ll create a few Swift files and I’ll highlight what each file does and how you can modify it for your use case. I’m using the SuperSimpleObjectCapture’s project structure as that’s the most malleable. If you want something that’s a better reference for shipping code, use Apple’s project that can be found above.

ScanningView.swift

This is the entry point for our scanning logic. In the SuperSimpleObjectCapture, it’s the Content View. I’ve refactored the code a bit to make it a bit more compacted. You should get an error regarding the CreateButton and ARQuickLookView. But we’ll implement those in due time.

//Shoutout to the SimpleObjectCapture project crated by @littleossa
  import SwiftUI
  import RealityKit

  struct ScanView: View {

      @State private var session: ObjectCaptureSession?
      @State private var imageFolderPath: URL?
      @State private var photogrammetrySession: PhotogrammetrySession?
      @State private var modelFolderPath: URL?
      @State private var isProgressing = false
      @State private var quickLookIsPresented = false

      var modelPath: URL? {
          return modelFolderPath?.appending(path: "model.usdz")
      }

      var body: some View {

          ZStack(alignment: .bottom) {
              if let session {
                  ObjectCaptureView(session: session)

                  VStack(spacing: 16) {

                      if session.state == .ready || session.state == .detecting {
                          // Detect and Capture
                          CreateButton(session: session)
                      }

                      HStack {
                          Text(session.state.label)
                              .bold()
                              .foregroundStyle(.yellow)
                              .padding(.bottom)
                      }

                  }
              }

              if isProgressing {
                  //I'd suggest creating a view that shows the user that the model is processing in a more straight forward
                  Color.blue.opacity(0.4)
                      .overlay {
                          VStack {
                              Text("Processing Model")
                              ProgressView()
                          }
                      }
              }


          }
          .task {
              guard let directory = createNewScanDirectory()
              else { return }
              session = ObjectCaptureSession()

              modelFolderPath = directory.appending(path: "Models/")
              imageFolderPath = directory.appending(path: "Images/")
              guard let imageFolderPath else { return }
              session?.start(imagesDirectory: imageFolderPath)
          }
          .onChange(of: session?.userCompletedScanPass) { _, newValue in
              if let newValue,
                 newValue {
                  //MARK: Take Note of this dear reader
                  // This time, I've completed one scan pass.
                  // However, Apple recommends that the scan pass should be done three times.
                  session?.finish()
              }
          }
          .onChange(of: session?.state) { _, newValue in
              if newValue == .completed {
                  session = nil

                  Task {
                      await startReconstruction()
                  }
              }
          }
          .sheet(isPresented: $quickLookIsPresented) {

              if let modelPath {
                  ARQuickLookView(modelFile: modelPath) {
                      guard let directory = createNewScanDirectory()
                      else { return }
                      quickLookIsPresented = false
                      // TODO: Restart ObjectCapture
                  }
              }
          }
      }
          func createNewScanDirectory() -> URL? {
              guard let capturesFolder = getRootScansFolder()
              else { return nil }

              let formatter = ISO8601DateFormatter()
              let timestamp = formatter.string(from: Date())
              let newCaptureDirectory = capturesFolder.appendingPathComponent(timestamp,
                                                                              isDirectory: true)
              print("Start creating capture path: \(newCaptureDirectory)")
              let capturePath = newCaptureDirectory.path
              do {
                  try FileManager.default.createDirectory(atPath: capturePath,
                                                          withIntermediateDirectories: true)
              } catch {
                  print("Failed to create capture path: \(capturePath) with error: \(String(describing: error))")
              }

              var isDirectory: ObjCBool = false
              let exists = FileManager.default.fileExists(atPath: capturePath,
                                                          isDirectory: &isDirectory)
              guard exists, isDirectory.boolValue
              else { return nil }
              print("New capture path was created")
              return newCaptureDirectory
          }

          func getRootScansFolder() -> URL? {
              guard let documentFolder = try? FileManager.default.url(for: .documentDirectory,
                                                                      in: .userDomainMask,
                                                                      appropriateFor: nil,
                                                                      create: false)
              else { return nil }
              return documentFolder.appendingPathComponent("Scans/", isDirectory: true)
          }

          func startReconstruction() async {
              guard let imageFolderPath,
                    let modelPath else { return }
              isProgressing = true
              do {
                  photogrammetrySession = try PhotogrammetrySession(input: imageFolderPath)
                  guard let photogrammetrySession else { return }
                  try photogrammetrySession.process(requests: [.modelFile(url: modelPath)])
                  for try await output in photogrammetrySession.outputs {
                      switch output {
                      case .requestError, .processingCancelled:
                          isProgressing = false
                          self.photogrammetrySession = nil
                          // TODO: Restart ObjectCapture
                      case .processingComplete:
                          isProgressing = false
                          self.photogrammetrySession = nil
                          quickLookIsPresented = true
                      default:
                          break
                      }
                  }

              } catch {
                  print("Error!!!", error)
              }
          }

  }

Awesome! At this point, if you try building the app it will complain about ARQuickLookView and CreateButton in the Scan View. Let’s sort that out

ARQuickLookView.swift

So this view uses the QuickLook library from Apple and is called after we’re done creating a 3D Model in order to preview the result.

You may be tempted to rewrite this code for a custom User Interface to handle your model file (at least I was) but that might not be the best approach. In true SwiftUI fashion, this view works very well for what it was designed for. Modifying it will be very sad and painful.

Thus, what I opted to do is focus on how I save and manage the file produced. Please drop some suggestions in the comments, I would greatly appreciate them.

//Shoutout to the SimpleObjectCapture project crated by @littleossa
  import SwiftUI
  import QuickLook

  struct ARQuickLookView: UIViewControllerRepresentable {

      let modelFile: URL
      let endCaptureCallback: () -> Void

      func makeUIViewController(context: Context) -> QLPreviewControllerWrapper {
          let controller = QLPreviewControllerWrapper()
          controller.previewController.dataSource = context.coordinator
          controller.previewController.delegate = context.coordinator
          return controller
      }

      func updateUIViewController(_ uiViewController: QLPreviewControllerWrapper, context: Context) {}

      func makeCoordinator() -> Coordinator {
          return Coordinator(parent: self)
      }

      class Coordinator: NSObject, QLPreviewControllerDelegate, QLPreviewControllerDataSource {
          let parent: ARQuickLookView

          init(parent: ARQuickLookView) {
              self.parent = parent
          }

          func numberOfPreviewItems(in controller: QLPreviewController) -> Int {
              return 1
          }

          func previewController(_ controller: QLPreviewController, previewItemAt index: Int) -> QLPreviewItem {
              return parent.modelFile as QLPreviewItem
          }

          func previewControllerWillDismiss(_ controller: QLPreviewController) {
              parent.endCaptureCallback()
          }
      }
  }

  extension ARQuickLookView {

      class QLPreviewControllerWrapper: UIViewController {
          let previewController = QLPreviewController()
          var quickLookIsPresented = false

          override func viewDidAppear(_ animated: Bool) {
              super.viewDidAppear(animated)

              if !quickLookIsPresented {
                  present(previewController, animated: false)
                  quickLookIsPresented = true
              }
          }
      }
  }

CreateButton.swift

 // You can honestly create your own Create Button, this is the one from the SimpleObjectCapture project.
  // Estenially the label for the create button changes depending on the state of the project
  // And where we are in the Object Capture pipeline

  import SwiftUI
  import RealityKit

  @MainActor
  struct CreateButton: View {
      let session: ObjectCaptureSession

      var body: some View {
          Button(action: {
              performAction()
          }, label: {
              Text(label)
                  .foregroundStyle(.white)
                  .padding()
                  .background(.tint)
                  .clipShape(Capsule())
          })
      }

      private var label: LocalizedStringKey {
          if session.state == .ready {
              return "Start Detecting"
          } else if session.state == .detecting {
              return "Start Capturing"
          } else {
              return "Undefined"
          }
      }

      private func performAction() {
          if session.state == .ready {
              let isDetecting = session.startDetecting()
              print(isDetecting ? "Start Detecting" : "Did Not Start Detecting Session")
          } else if session.state == .detecting {
              session.startCapturing()
          } else {
              print("Undefined")
          }
      }
  }


  extension ObjectCaptureSession.CaptureState {

      var label: String {
          switch self {
          case .initializing:
              "Initializing"
          case .ready:
              "Ready"
          case .detecting:
              "Detecting"
          case .capturing:
              "Capturing"
          case .finishing:
              "Finishing"
          case .completed:
              "Completed"
          case .failed(let error):
              "Failed with Error: \(String(describing: error))"
          @unknown default:
              fatalError("Unknown Default: \(self)")
          }
      }
  }

Conclusion

That’s it, with 3 Swift Files you can now scan and create 3D Models in your iOS Apps. Congratulations! So the user expereince is as follows:

1. User opens the app and grants Camera permission
2. They click the start detecting button when their object is in the viewfinder
3. They confirm the detected proportions
4. They scan the 3D Object
5. The model is generated and can be previewed with QuickLook

Please note that this app is a starter and Apple has some other things they’d recommend you implement with this framework, such as having 3 iterations of the scans. Here’s their relevant project.

WWDC23: Meet Object Capture for iOS

Apple Official Sample Project — Scanning objects using Object Capture

Thanks for reading. I really appreciate it and hope that your project compiles bug-free.

Hope you have a nice one! And in case I don’t see you again. Good afternoon, good evening and good night!

In this article, I’ll be sharing what I learned, break down how I set up my project, a few challenges I faced along the way and more.

I also have a new blog on Hashnode, where I’ll be writing about more cool tech, so check that out here: https://carlosmbe.hashnode.dev/.

An updated version of this article is also on Hashnode: https://carlosmbe.hashnode.dev/running-speech-models-with-swift-using-sherpa-onnx-for-apple-development

Update from September 2025: There now exists an open source library that makes Speech Diarization easy to implement across the Apple ecosystem called FluidAudio. For most people, I’d recommend using that. However, its essentially an abstraction of the Sherpa code. Hence, if you’re doing some advanced or custom work, this article may still be relevant as you have direct access to the C/C++/Swift code.

Feel free to share any tips and suggestions you may have as I’m still learning.

Speech diarization is the process of segmenting and labeling an audio stream by speaker to identify "who spoke when," enabling a clear understanding of who is speaking at any given time.

Part 1: Research and the Motive Behind my Approach

After numerous hours of Googling, I found a promising model, Pyannote’s Speaker Diarization 3.1.

It works well.

If you’re planning to have a python server run your models, then I’d probably suggest using that as it’ll save you a fair amount of work.

What work? Well, my objective was to have an on-device speech diarization, thus, I needed to convert this model to a format my Xcode project would recognize.

My first instinct was to use CoreMLTools to convert the model. Unfortunately, this didn’t work. If you attempt to retrace my debugging steps, you shall eventually find yourself at certain GitHub Issue in CoreMLTools and realize that you’re repeating the journey of a few developers before you.

The alternative? Use the C++ version and have a bridging header for your Swift and C++ code. This may sound intimidating, especially if you’re a native Swift developer who’s never worked with C++.

Don’t worry, I’m here to make this as easy as possible.

Part 2: Sherpa-Onnx

By using Sherpa-Onnx we can bypass a significant amount of boilerplate and porting.

They even have starter projects you can use as a reference for how to structure your project.

However, I did find it a bit overwhelming and created my own Starter Project. It’s oriented towards Speech Diarization but you can use it for Sherpa-Onnx’s other models, such as voice activity detection, audio tagging and more.

If you don’t want to clone my Starter Project, you can follow the Sherpa-Onnx build instructions and compile the framework by yourself. Just clone k2-fsa/sherpa-onnx and follow these build instructions

Please note, if you’re aiming to build for macOS, it has its own seprate build.sh file and that visionOS’s build script doesn’t exist (officially).

I built a branch of Sherpa-Onnx that supports the VisionOS Simulator and should theoretically support the device with a few line changes.

How Do I build Sherpa-Onnx?

Alright, now that we’ve covered the motivation for my approach and I’ve disclosed any potential issues. Let me walk through the technicals, starting with building the Frameworks.

Building the Frameworks for iOS

1. Clone Sherpa-Onnx

git clone https://github.com/k2-fsa/sherpa-onnx.git

2. Run the Build Script (Either the iOS, macOS or visionOS script)

./build-ios.sh

It will download Onnx Runtime and build Sherpa-Onnx for iOS. You should see a new folder titled build-ios

3. Either use:

• My Starter Project
• The Sherpa-Onnx examples found in folders ios-swift and ios-swiftui
• Create a new project and the copy the frameworks ios-onnxruntime and sherpa-onnx.xcframework to your project from the newly created build-ios folder, create a bridging header and copy the Sherpa-Onnx Swift functions to your new project.

If you used the Starter Project or the Sherpa examples. You should be good to download Models and just experiment.

However, if you’re opting to create your own project from scratch, we have to a few more steps to follow. In particular, we need to set up a Bridging Header and add some linker flags.

Essentially its some boiler plate code that allows our Swift and C++ files to communicate. If you’ve worked with Metal, then this should feel like home.

I’ll write another article in the future documenting this process as its fairly involved.

Using My Starter Project for Speech Diarization

If you opt to use My Starter Project, you can easily start performing Speech Diarization on clips as soon as you download and add the Onnx-Runtime framework as highlighted in the ReadMe. To test it out, just add a file titled “Clip.mp4” and run the app.

Show Me The Code

Here’s a breakdown of what’s happening in the background.

1. Firstly, using AudioKit, we convert the clip to a format the Model accepts. Specifically a 16 kHz, mono, 32‑bit float WAV file. If you want your app size to be smaller and rely on less frameworks, you could probably do this with Apple’s built in AVFoundation. AudioKit abstracts a lot of the code.
2. Afterwards, we pass the newly converted file to my runDiarization function. This is where all the C++ and Swift interactions occur. Sherpa-Onnx also abstracts this so it looks somewhat Swifty, but be fully aware that pointers are being passed around in the background.

• The runDiarization function expects a requires that we have a Speech Diarization model in the project alongside an embedding Extractor Model. The Starter Project has these preloaded, so we’re mainly creating a config for the model. So we pass an expected number of speakers.
• This is the ViewModel with all of this code:

import AVFoundation
import Foundation

class SDViewModel : ObservableObject{
    
    let segmentationModel = getResource("SpeechModel", "onnx")
    let embeddingExtractorModel = getResource("3dspeaker_speech_eres2net_base_sv_zh-cn_3dspeaker_16k", "onnx")
 
    
    func runDiarization(waveFileName: String, numSpeakers: Int = 0, fullPath: URL? = nil)  async -> [SherpaOnnxOfflineSpeakerDiarizationSegmentWrapper] {
       
        let waveFilePath = fullPath?.path ?? getResource(waveFileName, "wav")
        
        var config = sherpaOnnxOfflineSpeakerDiarizationConfig(
                segmentation: sherpaOnnxOfflineSpeakerSegmentationModelConfig(
                pyannote: sherpaOnnxOfflineSpeakerSegmentationPyannoteModelConfig(model: segmentationModel)),
                embedding: sherpaOnnxSpeakerEmbeddingExtractorConfig(model: embeddingExtractorModel),
                clustering: sherpaOnnxFastClusteringConfig(numClusters: numSpeakers)
        )
        
        //The SherpaOnnx functions are defined in SherpaFiles/Swift/SherpaOnnx.swift
        let sd = SherpaOnnxOfflineSpeakerDiarizationWrapper(config: &config)

        let fileURL: NSURL = NSURL(fileURLWithPath: waveFilePath)
        let audioFile = try! AVAudioFile(forReading: fileURL as URL)

        let audioFormat = audioFile.processingFormat
        assert(Int(audioFormat.sampleRate) == sd.sampleRate)
        assert(audioFormat.channelCount == 1)
        assert(audioFormat.commonFormat == AVAudioCommonFormat.pcmFormatFloat32)

        let audioFrameCount = UInt32(audioFile.length)
        let audioFileBuffer = AVAudioPCMBuffer(pcmFormat: audioFormat, frameCapacity: audioFrameCount)

        try! audioFile.read(into: audioFileBuffer!)
        let array: [Float]! = audioFileBuffer?.array()
        
        let segments = await Task.detached(priority: .background) {
               return sd.process(samples: array)
           }.value
        
        for i in 0..<segments.count {
          print("\(segments[i].start) -- \(segments[i].end) speaker_\(segments[i].speaker)")
        }
        
        return segments
    }
    
}

• We also have a few extensions to make accessing the Audio File easier.

import AVFoundation

extension AudioBuffer {
    func array() -> [Float] {
        return Array(UnsafeBufferPointer(self))
    }
}

extension AVAudioPCMBuffer {
    func array() -> [Float] {
        return self.audioBufferList.pointee.mBuffers.array()
    }
}

For reference, the Sherpa-Onnx official examples have a get resource function and this is its implementation.

import Foundation

func getResource(_ forResource: String, _ ofType: String) -> String {
  let path = Bundle.main.path(forResource: forResource, ofType: ofType)
  precondition(
    path != nil,
    "\(forResource).\(ofType) does not exist!\n" + "Remember to change \n"
      + "  Build Phases -> Copy Bundle Resources\n" + "to add it!"
  )
  return path!
}

If you’re using the starter projcet as a reference for your own custom project. Please take note of my Build Settings. Specifically:

• Swift Compiler — General: Objective C Bridging Header
• Search Paths: Header Search Paths
• Linking — General: Other Linking Flags: Set to -lc++

Pretty Cool Links:

• Check out my GitHub for some other cool projects, I’ve built.
• Connect with me on LinkedIn.

Very Important shoutout to Koedeco and Caroline Begbie. They have an awesome article talking about this. Link To Article.

Most of the work we’ll do in this part is refactoring our logic from part 4 so that its a bit more open ended and capable of rendering data from files containing 3D Models such as but not limited to .obj and .usdz files.

Show Me The Code

Firstly, we need to be able to read a .obj file loaded into the app. To do this, we can add some logic in our Renderer’s init function that loads our file as a MDL Asset.

class Renderer : NSObject, MTKViewDelegate {
 //... Old Properties ..
    
    //New Properties
    let allocator : MTKMeshBufferAllocator
    let asset : MDLAsset
    let mdlMesh : MDLMesh
    let mesh : MTKMesh

    init(_ parent : ContentView) {
//... Old Code ...
  //New logic that intitalises our new properties
        allocator = MTKMeshBufferAllocator(device: device)
        
        asset = MDLAsset(url: Bundle.main.url(forResource: "Your File Here", withExtension: "obj")!,
                         vertexDescriptor: .defaultLayout,
                         bufferAllocator: allocator)
        
        mdlMesh = asset.childObjects(of: MDLMesh.self).first as! MDLMesh
        
        mesh = try! MTKMesh(mesh: mdlMesh, device: device)

        super.init()
    }

If you need a file to test out your new rendering pipeline. You can look up “Free 3D Models” on Google and download a .obj file or anything really.

Before we modify our draw call. I would like to refactor and simplify our shaders to better align with the one used in Kodeco’s content. The content I wrote in Part 3 still applies. We’re just changing our struct definitions in the bridge and making our Vertex only handle positions and temporarily removing the color component.

#include "bridge.h"
#include <metal_stdlib>
using namespace metal;

vertex VertexOut vertexMain(
                            VertexIn v [[stage_in]],
                            constant Uniforms &uniforms[[buffer(2)]])
{
    VertexOut data;
    float4 position = uniforms.projectionMatrix * uniforms.viewMatrix
    * uniforms.modelMatrix * v.position;
    data.position = position;
    return data;
};

half4 fragment fragmentMain(){
    return half4(1, 1, 1.0, 1);
};

In bridge.h, we have some new definitions.

#ifndef bridge_h
#define bridge_h
#include <simd/simd.h>

struct Vertex{
    vector_float4 position;
    vector_float3 color;
};

struct VertexIn {
    vector_float4  position [[attribute(0)]];
};

struct VertexOut {
    vector_float4  position [[position]];
};


struct Uniforms{
  matrix_float4x4 modelMatrix;
  matrix_float4x4 viewMatrix;
  matrix_float4x4 projectionMatrix;
};

#endif /* bridge_h */

Ok great. We’re almost done. I promise. Now we need to make one change before we call our draw call. We’re going to add a vertexDescriptor to our pipeline builder. They’re two ways we could do this. We can create one manually or add an extension to handle this. In this case, I am going to use an extension written by Caroline Bagbie from Kodeco.

//Extensions. You can put these wherever you feel fit

extension MTLVertexDescriptor {
  static var defaultLayout: MTLVertexDescriptor? {
    MTKMetalVertexDescriptorFromModelIO(.defaultLayout)
  }
}

extension MDLVertexDescriptor {
  static var defaultLayout: MDLVertexDescriptor {
    let vertexDescriptor = MDLVertexDescriptor()
    var offset = 0
    vertexDescriptor.attributes[0] = MDLVertexAttribute(
      name: MDLVertexAttributePosition,
      format: .float3,
      offset: 0,
      bufferIndex: 0)
    offset += MemoryLayout<float3>.stride
    vertexDescriptor.layouts[0] = MDLVertexBufferLayout(stride: offset)
    return vertexDescriptor
  }
}

With these extensions in place, we can add a vertexDescriptor to our pipelineBuilder.

import Metal

func buildPipeline(device: MTLDevice) -> MTLRenderPipelineState {
    
    let pipeline : MTLRenderPipelineState
    
    let pipelineDescriptor = MTLRenderPipelineDescriptor()
    let library =  device.makeDefaultLibrary()!
    
   
    pipelineDescriptor.vertexFunction = library.makeFunction(name: "vertexMain")
    pipelineDescriptor.fragmentFunction = library.makeFunction(name: "fragmentMain")
    pipelineDescriptor.colorAttachments[0].pixelFormat = .bgra8Unorm
    //New Line
    pipelineDescriptor.vertexDescriptor = MTLVertexDescriptor.defaultLayout
    
    do{
        pipeline = try device.makeRenderPipelineState(descriptor: pipelineDescriptor)
        return pipeline
    }catch{
        print(error.localizedDescription)
        fatalError()
    }
    
}

That’s pretty much all the changes you need to make to draw 3D Models. Once more, you can find various free models on the internet to experiment with. To utlize all of these changes. Your draw call will look like this:

 func draw(in view: MTKView) {
....   
      renderEncoder.setRenderPipelineState(pipeline)

        //Draw our new Mesh from the Asset
        renderEncoder.setVertexBuffer(
          mesh.vertexBuffers[0].buffer,
          offset: 0,
          index: 0)
        
        for submesh in mesh.submeshes {
          renderEncoder.drawIndexedPrimitives(
                                  type: .triangle,
                                  indexCount: submesh.indexCount,
                                  indexType: submesh.indexType,
                                  indexBuffer: submesh.indexBuffer.buffer,
                                  indexBufferOffset: submesh.indexBuffer.offset
          )
        }
         renderEncoder.endEncoding()
        
        //Commit and present the state of the buffer
        commandBuffer.present(drawable)
        commandBuffer.commit()
        
    }

To demonstarate this, I went to free3D.com and downloaded @printable_models’s Cat Model. If we want to see the texture and lighting applied, you’ll need to come back and read future parts.

• Kodeco’s (Ray Wenderlich) Metal Course Taught by Caroline
• Old but FREE Kodeco Course on YouTube Taught by Caroline
• Metal by Tutorials book Source Code on GitHub
• Metal By Tutorials Book (Kodeco)

Show Me The Code

Rendering in 3D isn’t that much different than rendering in 2D. Essentially, we need to modify our initial project in two ways to achieve it. Our set of vertices need to represent a 3D Obeject and we need a set of matrices telling our GPU how to make a 3D Object on a 2D Screen.

1. Vertices

We can get the vertices needed to draw a 3D object in a handful of ways. But for now, I will add a “Make Cube” function in our Mesh Builder.

 func makeCube() -> Mesh {
        let s = Float(0.5)
        let vertices: [Vertex] = [
            Vertex(position: SIMD4<Float>(-s, -s,  s, 1), color: SIMD3<Float>(1, 0, 0)),
            Vertex(position: SIMD4<Float>( s, -s,  s, 1), color: SIMD3<Float>(0, 1, 0)),
            Vertex(position: SIMD4<Float>( s,  s,  s, 1), color: SIMD3<Float>(0, 0, 1)),
            Vertex(position: SIMD4<Float>(-s,  s,  s, 1), color: SIMD3<Float>(1, 1, 0)),
            Vertex(position: SIMD4<Float>(-s, -s, -s, 1), color: SIMD3<Float>(1, 0, 1)),
            Vertex(position: SIMD4<Float>( s, -s, -s, 1), color: SIMD3<Float>(0, 1, 1)),
            Vertex(position: SIMD4<Float>( s,  s, -s, 1), color: SIMD3<Float>(1, 1, 1)),
            Vertex(position: SIMD4<Float>(-s,  s, -s, 1), color: SIMD3<Float>(0, 0, 0))
        ]
        let indices: [UInt16] = [
            0,1,2,2,3,0,
            1,5,6,6,2,1,
            5,4,7,7,6,5,
            4,0,3,3,7,4,
            3,2,6,6,7,3,
            4,5,1,1,0,4
        ]
        let vb = device.makeBuffer(bytes: vertices, length: vertices.count * MemoryLayout<Vertex>.stride, options: [])!
        let ib = device.makeBuffer(bytes: indices, length: indices.count * MemoryLayout<UInt16>.stride, options: [])!
        return Mesh(vertexBuffer: vb, indexBuffer: ib, indexCount: indices.count)
    }

To draw this, it will be quite literally identical to drawing our quad or triangle from the last part but with a different Mesh. In your Renderer’s Draw call, we’ll have:

func draw(in view: MTKView) {
...
    renderEncoder.setVertexBuffer(cube.vertexBuffer, offset: 0, index: 0)
    renderEncoder.drawIndexedPrimitives(
        type: .triangle,
        indexCount: cube.indexCount,
        indexType: .uint16,
        indexBuffer: cube.indexBuffer,
        indexBufferOffset: 0)
...
     renderEncoder.endEncoding()
   
}

However, if you run this, you’ll surely be disappointed and see your screen filled with the cube’s colors or nothing at all. But why? There’s a short and long answer to both of this question but essentially, its a lot of math. @GetIntoGameDev, @Caroline Begbie and others have very detailed explanations of this. But for now, just now we need to create some matrices.

2. The Matrix

Firstly, let’s add Struct to our bridging header. We’ll call it “Uniforms.” This is how we will send our matrices to the GPU and create them on the CPU side.

//In your .h file
#include <simd/simd.h>

struct Uniforms{
  matrix_float4x4 modelMatrix;
  matrix_float4x4 viewMatrix;
  matrix_float4x4 projectionMatrix;
};

In the Renderer class. We want to add a property called Uniforms and then initialize it in the class’s init. Afterwards, we will create the projection matrix once we know the View’s size but before passing it to the GPU. At this point, we transition to the draw function and send the Uniforms.

class Renderer : NSObject, MTKViewDelegate {
    //Other properties 
    var uniforms : Uniforms
    ...
init(_ parent : ContentView) {
    ...
    uniforms = setUpUniforms()
    ...
    }

 func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
          //Create the Projection Matrix Using the CGSize from the View
          let aspect = Float(view.bounds.width) / Float(view.bounds.height)
          let projectionMatrix =
            createFloat4x4Projection(
              projectionFov: Float(70).degreesToRadians,
              near: 0.1,
              far: 100,
              aspect: aspect)
          uniforms.projectionMatrix = projectionMatrix
    }

  func draw(in view: MTKView) {
  ....
  // Send the Uniforms to the Shaders
  renderEncoder.setVertexBytes(&uniforms,
                               length: MemoryLayout<Uniforms>.stride,
                               index: 1)
...
}

//Basic and Hardcoded Math Functions


func createFloat4x4Projection(projectionFov fov: Float, near: Float, far: Float, aspect: Float) -> float4x4{
    let y = 1 / tan(fov * 0.5)
    let x = y / aspect
    let z = far / (far - near)
    let X = SIMD4<Float>( x,  0,  0,  0)
    let Y = SIMD4<Float>( 0,  y,  0,  0)
    let Z = SIMD4<Float>( 0,  0,  z, 1)
    let W = SIMD4<Float>( 0,  0,  z * -near,  0)
    return float4x4(columns: (X, Y, Z, W))
}

func setUpUniforms() -> Uniforms{
  
    var uniforms = Uniforms()
    
    let translation = float4x4(
        SIMD4<Float>(1, 0, 0, 0),
        SIMD4<Float>(0, 1, 0, 0),
        SIMD4<Float>(0, 0, 1, 0),
        SIMD4<Float>(0, 0, 0, 1)
    )

   
    let angle = Float.pi / 6
    let rotation = float4x4(
        SIMD4<Float>(cos(angle), 0, sin(angle), 0),
        SIMD4<Float>(0, 1, 0, 0),
        SIMD4<Float>(-sin(angle), 0, cos(angle), 0),
        SIMD4<Float>(0, 0, 0, 1)
    )

    let modelMatrix = matrix_multiply(translation, rotation)
    let viewTranslation = float4x4(
        SIMD4<Float>(1, 0, 0, 0),
        SIMD4<Float>(0, 1, 0, 0),
        SIMD4<Float>(0, 0, 1, 0),
        SIMD4<Float>(0, 0, -2, 1)
    )

    let viewMatrix = viewTranslation.inverse
    
    uniforms.modelMatrix = modelMatrix
    uniforms.viewMatrix = viewMatrix

    return uniforms
}

For convenience purposes, I will use a simple function called Create Uniforms which is very basic and has values hardcoded. However, in production, you’d likely create a set of functions to do common matrix graphics math for you. Both Caroline and GetIntoGameDev have publicly availably Swift functions showcasing this. I’d honestly opt for this if you’re making a personal project. If you want an deep understanding of what’s going on. I’d say write your own. You’ll most likely use Apple’s SIMD Library.

Here’s the updated Vertex Shader:

VertexOutput vertex vertexMain(const device Vertex* vertices [[buffer(0)]],
                               uint vertexID [[vertex_id]],
                               constant Uniforms &uniforms [[buffer(1)]])
{
    
    Vertex v = vertices[vertexID];
    VertexOutput data;
    float4 position = uniforms.projectionMatrix * uniforms.viewMatrix   * uniforms.modelMatrix * v.position;
    data.position = position;
    data.color = half3(v.color);
    return data;
}

At this point. We should have a cube but it looks kinda weird.

To fix this we need to enable back culling (and possibly depth testing)

 func draw(in view: MTKView) {{
....   
 let renderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor)!     
//After creating the render encoder, we set Cull Mode to back
 renderEncoder.setCullMode(.back)
....
}

Congratulations! You now have a 3D Cube. If you wanted to have a 3D Model. We’d use a very similar process. But instead of using the 3D Cube we created. The vertices we load would come from a .obj file or some other source.

Now we’re focused on connecting the CPU and GPU to draw things somewhat efficiently. So from after this part, theoretically, you’d mainly be doing math for projections, modifying shader code, optimization through things like Instanced Rendering, but all of that is on a project by project basis. If I write any more articles, they will be focused on things I want to try doing in Metal for fun. In particular, I would like to render the University of Kansas Mascot, Big Jay with Metal.

Show Me The Code!

We’re now looking into Buffers. This should allow us to no longer hard code values and send it directly to the GPU from the CPU.

Metal has a cool things called a Bridging Header that allows us to make this connection from the GPU and CPU fairly seamless.

Step 1: Create a new Header file in your project directory

Similar to how we create new Swift files. We want to create a Header File.

In the new Header. We will create a struct containing all the information a vertex should hold.

#ifndef bridge_h
#define bridge_h
#include <simd/simd.h>

struct Vertex{
    // You'll probably expand this to suit your personal needs such as having Normals 
    vector_float4 position;
    vector_float3 color;
};

#endif /* bridge_h */

We’ll include the simd type and give our struct 2 properties for the position and color of a vertex.

Then go to Build Settings and assign the Objective-C Bridging Header to our new file. In my case, the path to the new header is “MacMetalExp/bridge.h” following the structure, “projectName/headerFileName”, so I enter that path in the Swift Compiler’s bridging header section. I’ve labelled the screenshot below to help you identify where in Xcode this section is.

The steps shown in the screenshot are as follows:

1. Navigate to Project Settings → Build Settings
2. Filter down results to show only options with the word “header”
3. Scroll down the Swift Compiler — Genreal section
4. In the “Objective-C Bridging Header” field, enter the path to your Header file.

To confirm that everything is well set up. Try running and building your app. Also remember to clean the build folder if you’re having strange issues despite knowing your code is perfect.

Vertex Buffer

The Vertex Buffer is essentially a list of things we need to at draw a single point. In our current project, our vertex only has position and color assigned to it. However, with time we may have more attributes.

By Creating a Bridging Header, we make it possible for the CPU and GPU to communicate. So now, instead of hard coding the vertex information in the Shader. We’ll create a Vertex Buffer at creation and pass it over to the GPU.

Let’s update our .shader file to reflect this.

#include "bridge.h"
#include <metal_stdlib>
using namespace metal;


struct VertexOutput {
    //Similar to how we qualify the Vertex and Fragment shader, we can qualify variables, hence the [[___]]
    //Here, we're telling Metal that this varibale is the actual position and some cool things happen under the hood
    float4 position [[position]];
    half3 color;
};

//Now we have a pointer to our Vertex Buffer as part of the Vertex Shader. Also, we qualify it as a buffer at Index 0
VertexOutput vertex vertexMain(const device Vertex* vertices [[buffer(0)]],
                               uint vertexID [[vertex_id]])
{
    //Instead of just loading the values from an array, we're now reading from the Vertex Struct we created earlier
    Vertex v = vertices[vertexID];
    VertexOutput data;
    data.position = v.position;
    data.color = half3(v.color);
    return data;
};

//Fragment Shader Receives the Output from the Vertex Shader as input
half4 fragment fragmentMain(VertexOutput frag [[stage_in]]){
    //Similarly we're qualifiying the input as a [[stage_in]]
    return half4(frag.color, 1.0);
    //Return Color as is, with an alpha of 1
};

Drawing A Triangle With A Vertex Buffer

Continuing on the renderer code we wrote in Part 2. We can now make a Vertex Buffer with our triangles vertices. The process is as follows:

//Make Vertex Buffer With Triangle Information
 func makeTriangle() -> MTLBuffer {
        
        let vertices: [Vertex] = [
            Vertex(position: [-0.75, -0.75, 0.0, 1.0], color: [1,1.0, 1.0]),
            Vertex(position: [0.75, -0.75, 0.0, 1.0], color: [1.0,1.0, 1.0]),
            Vertex(position: [0.0, 0.75, 0.0, 1.0], color: [1,0.3, 0.7])
            ]
        
        return device.makeBuffer(bytes: vertices,
                                 length: vertices.count * MemoryLayout<Vertex>.stride)!
        
    }

Then when drawing, it would be as follows

// In Renderer Class That Calls Draw Function
....
        renderEncoder.setRenderPipelineState(pipeline)
        
        //Add This line. We initizalie triangle to be the value of makeTriangle
        renderEncoder.setVertexBuffer(triangle, offset: 0, index: 0)

        renderEncoder.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 3)
        renderEncoder.endEncoding()

Mesh Builder Class and Index Buffer

However, we can also make a mesh builder class. So if we want to make and render a complex shapes we can easily expand. Likewise it’d be nice to put our make Triangle function there. And while we’re at it. Let’s introduce Indexed drawing by drawing a quad behind our triangle.

Mesh Builder

import Metal

struct Mesh{
    //Makes it easier to bundle together data
    let vertexBuffer: MTLBuffer
    let indexBuffer: MTLBuffer
    let indexCount: Int
}

class MeshBuilder {
    
    let device: MTLDevice
    
    init(device: MTLDevice) {
        self.device = device
    }
    
    func makeTriangle() -> MTLBuffer {
        
        let vertices: [Vertex] = [
            Vertex(position: [-0.75, -0.75, 0.0, 1.0], color: [1,1.0, 1.0]),
            Vertex(position: [0.75, -0.75, 0.0, 1.0], color: [1.0,1.0, 1.0]),
            Vertex(position: [0.0, 0.75, 0.0, 1.0], color: [1,0.3, 0.7])
            ]
        
        return device.makeBuffer(bytes: vertices,
                                 length: vertices.count * MemoryLayout<Vertex>.stride)!
        
    }
    
    
    func makeQuad() -> Mesh {
        
        //Points in our quad
        let vertices: [Vertex] = [
            Vertex(position: [-0.75, -0.75, 0.0, 1.0], color: [1,1, 1]),
            Vertex(position: [0.75, -0.75, 0.0, 1.0], color: [1,1, 1]),
            Vertex(position: [0.75, 0.75, 0.0, 1.0], color: [1,1, 1]),
            Vertex(position: [-0.75, 0.75, 0.0, 1.0], color: [1,1,1])
            ]
        
        //Order in which we want to draw lines between points
        let indices: [UInt16] = [ 0, 1, 2, 2, 3, 0 ]
            
        
        //Making and combining the buffers for both the Vertices and indices
        let vertexBuffer = device.makeBuffer(bytes: vertices,
                                 length: vertices.count * MemoryLayout<Vertex>.stride)!
        
        let indexBuffer = device.makeBuffer(bytes: indices,
                                            length: indices.count * MemoryLayout<UInt16>.stride)!
        
        
        return Mesh(vertexBuffer: vertexBuffer,
                    indexBuffer: indexBuffer,
                    indexCount: indices.count)
    }
    
    
}

Final Render Code With Quad/Mesh Drawing

import MetalKit

///Put metal code here
class Renderer : NSObject, MTKViewDelegate {
    var parent : ContentView
    var device : MTLDevice!
    var commandQueue : MTLCommandQueue!
    
    let triangle: MTLBuffer
    let quad : Mesh
    
    var pipeline: MTLRenderPipelineState
    
    init(_ parent : ContentView) {
        self.parent = parent
       
        if let device = MTLCreateSystemDefaultDevice() {
            self.device = device
        }
        
        self.commandQueue = device.makeCommandQueue()
        
        //Adding our Pipeline Builder
        pipeline = buildPipeline(device: device)
        
        let meshBuilder = MeshBuilder(device: device)
        triangle = meshBuilder.makeTriangle()
        quad = meshBuilder.makeQuad()
        
        super.init()
    }
    
    
    func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
    }
    
    ///Drawing Pass
    func draw(in view: MTKView) {
        guard let drawable = view.currentDrawable else {    return  }
        
        let commandBuffer = commandQueue.makeCommandBuffer()!
        let renderPassDescriptor = view.currentRenderPassDescriptor!
        //Render Pass - Clear - Set Clear Colour
        renderPassDescriptor.colorAttachments[0].clearColor = MTLClearColor(red: 0.7, green: 0.3, blue: 0.6, alpha: 1.0)
        renderPassDescriptor.colorAttachments[0].loadAction = .clear
        renderPassDescriptor.colorAttachments[0].storeAction = .store
        
        let renderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor)!
        
        //Updates so draw calls are made
        renderEncoder.setRenderPipelineState(pipeline)
        
        renderEncoder.setVertexBuffer(quad.vertexBuffer, offset: 0, index: 0)
        renderEncoder.drawIndexedPrimitives(type: .triangle, indexCount: quad.indexCount, indexType: .uint16, indexBuffer: quad.indexBuffer, indexBufferOffset: 0)
        
        renderEncoder.setVertexBuffer(triangle, offset: 0, index: 0)
        renderEncoder.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 3)
        
        
        
        renderEncoder.endEncoding()
        
        //Commit and present the state of the buffer
        commandBuffer.present(drawable)
        commandBuffer.commit()
        
    }
    
}

Congratulations, you have a triangle on top of a quad. If it’s crashing right now, try cleaning your build folder.

Yes, if you copied and pasted my complete functions, you may have noticed I changed the background color again after part 2. Why? The answer is simple, Why not?

At this point, you can build a lot of things. Assuming you have a background in Computer Graphics. Now that you know how to write shaders and send data from the CPU side. All that’s left is to let your imagination go wild and do a lot of math.

In the next installment, I’ll attempt to render a 3D Mesh and possibly introduce a basic lighting model to our shader.

So now, instead of hard coding the vertex information in the Shader. We’ll create a Vertex Buffer at creation and pass it over to the GPU.

Special shout out to @GetIntoGameDev’s YouTube channel and work.

Show Me The Code

After setting up your Metal Kit Views, you’ll want to create shaders and make draw calls to actually render something.

Creating Shaders

In your Xcode Project, you’ll want to create a new “.metal” file. Feel free to name it whatever you want. Unlike OpenGL, we can put both the Vertex and Fragment Shader in the same file. We just need to create their functions and give them the qualifier “vertex” or “fragment” accordingly. One more thing, is that we also need to create a struct that contains the Vertex Shader’s data.

Putting all of this together, our .metal file should look like this.

#include <metal_stdlib>
using namespace metal;

struct VertexOutput {
    //Similar to how we qualify the Vertex and Fragment shader, we can qualify variables, hence the [[___]]
    //Here, we're telling Metal that this varibale is the actual position and some cool things happen under the hood
    float4 position [[position]];
    half3 color;
};

//Vertex ID is the index of the current Vertex
VertexOutput vertex vertexMain(uint vertexID [[vertex_id]]){
    VertexOutput data;
    data.position = positions[vertexID];
    data.color = colors[vertexID];
    return data;
};

//Fragment Shader Receives the Output from the Vertex Shader as input
half4 fragment fragmentMain(VertexOutput frag [[stage_in]]){
    //Similarly we're qualifiying the input as a [[stage_in]]
    return half4(frag.color, 1.0);
    //Return Color as is, with an alpha of 1
};

Drawing A Triangle

To draw a Triangle with our new Shaders, we need to modify our starter code in 3ways. Firstly, we need to add the vertices and colors of the triangle we’re drawing. Secondly, we need to combine all the aspects of our rendering pipeline together. Lastly, we need to make a draw call to render our triangle.

Step 1: Adding Our Triangle’s Information

For educational purposes, we’ll just hard code these in the .metal file. However, in practice, we’d load this from elsewhere. Possibly a mesh.

#include <metal_stdlib>
using namespace metal;

//Hard coded data for learning purposes, don't this in real projects
constant float4 positions[] = {
    //X -1== Left, 1 == Right, 0 == Centre
    float4(-0.75, -0.75, 0.0, 1.0), //Bottom Left
    float4(0.75, -0.75, 0.0, 1.0), //Bottom Right
    float4(0.0, 0.75, 0.0, 1.0), //Centre Top
};

constant half3 colors[] = {
    //RGB
    half3(1,0.0, 1.0), //Bottom Left
    half3(0.0, 0.0, 1.0), //Bottom Right
    half3(1.0,1.0, 1.0) //Centre Top
};

//Shaders Come Here ....

Step 2: Build Our Pipeline

At this point, we’ll create a function that builds a pipeline using the shaders and Metal Device. In practice, this function would be more generic but for now we’ve hardcoded the names of the shader.

import Metal

func buildPipeline(device: MTLDevice) -> MTLRenderPipelineState {
    
    let pipeline : MTLRenderPipelineState
    
    let pipelineDescriptor = MTLRenderPipelineDescriptor()
    let library =  device.makeDefaultLibrary()!
    
    //We're hardcoding these, but realistically in a more complex app, we'd accept these as and map it accordingly
    pipelineDescriptor.vertexFunction = library.makeFunction(name: "vertexMain")
    pipelineDescriptor.fragmentFunction = library.makeFunction(name: "fragmentMain")
    pipelineDescriptor.colorAttachments[0].pixelFormat = .bgra8Unorm
    
    do{
        pipeline = try device.makeRenderPipelineState(descriptor: pipelineDescriptor)
        return pipeline
    }catch{
        //Since we're hardcoding things right now, we know that is should work, otherwise in practise, our code would handle this better
        //If it does crash, try cleaning your build folder
        fatalError()
    }
    
}

Step 3: Draw Calls

This Renderer class is mostly the same as the one from the previous part. The main difference is that we now have a pipeline variable of type MTLRenderPipelineState we initialize it with the function from step 2 and use it in the draw function with renderEncoder.

import MetalKit

///Put metal code here
class Renderer : NSObject, MTKViewDelegate {
    var parent : ContentView
    var device : MTLDevice!
    var commandQueue : MTLCommandQueue!
    
    var pipeline: MTLRenderPipelineState
    
    init(_ parent : ContentView) {
        self.parent = parent
       
        if let device = MTLCreateSystemDefaultDevice() {
            self.device = device
        }
        
        self.commandQueue = device.makeCommandQueue()
        
        //Adding our Pipeline Builder
        pipeline = buildPipeline(device: device)
        
        super.init()
    }
    
    
    func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
    }
    
    ///Drawing Pass
    func draw(in view: MTKView) {
        guard let drawable = view.currentDrawable else {    return  }
        
        let commandBuffer = commandQueue.makeCommandBuffer()!
        let renderPassDescriptor = view.currentRenderPassDescriptor!
        //Render Pass - Clear - Set Clear Colour
        renderPassDescriptor.colorAttachments[0].clearColor = MTLClearColor(red: 0.8, green: 0.3, blue: 0.6, alpha: 1.0)
        renderPassDescriptor.colorAttachments[0].loadAction = .clear
        renderPassDescriptor.colorAttachments[0].storeAction = .store
        
        let renderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor)!
        
        //Updates so draw calls are made
        renderEncoder.setRenderPipelineState(pipeline)
        renderEncoder.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 3)
        
        renderEncoder.endEncoding()
        
        //Commit and present the state of the buffer
        commandBuffer.present(drawable)
        commandBuffer.commit()
        
    }
    
}

Closing Words

Congratulations, you have a triangle. If it’s crashing right now, try cleaning your build folder.

Also, if you copied and pasted my complete functions, you may have noticed I changed the background color from the one in Part 1. Why? The answer is simple, Why not?

Once more, special shout out to @GetIntoGameDev’s YouTube channel. You can find their GitHub repo documenting a similar set up here: https://github.com/amengede/getIntoMetalDev

Alternatively, you can follow this GitHub Repo that I will push my code to as I develop it and also have links to all the future articles I write in this series.

If I’ve made any errors, or you have a really cool way of implementing something, feel free to leave a comment.

Unfortunately, from my research, I believe Metal and Metal Kit Views are such a framework. Thus, setting up a View to see your rendering pipeline’s work has a few more steps then I’d like. But don’t worry, I got you, however, first I would like to give a special shout out to @GetIntoGameDev’s YouTube channel.

I’m not delving too deeply into the theory of 3D Graphics Rendering, I’m a not professor, just an enthusiast who likes writing Swift. For those interested in learning Metal, I found these resources.

• @GetIntoGameDev’s YouTube channel
• (Paid) Kodeco Course
• Old but FREE Kodeco Course on YouTube

Show Me The Code

I’m assuming that you’ve already created a new project on Xcode. First things first, we need to modify the Content View to show our rendered content. If you come from an OpenGL and C++ background and not iOS and Swift, you can think of this portion as somewhat similar to using GLFW to create a window for our OpenGL rendering. Here we’re using SwiftUI (really UIKit) to do something similar for Metal.

Content View

import SwiftUI
import MetalKit

struct ContentView: UIViewRepresentable {
    ///Create UI Kit View to Set Up Metal Kit View
    
    func makeCoordinator() -> Renderer {
        // We are going to create this class elsewhere, so don't be shocked if you're getting a few errors here
        Renderer(self)
    }
    
    func makeUIView(context: UIViewRepresentableContext<ContentView>) -> MTKView {
        // We're using the Metal Kit library to create a Metal Kit view. You can customize this section with your own parameters
        let mtkView = MTKView()
        mtkView.delegate = context.coordinator
        mtkView.preferredFramesPerSecond = 60
        mtkView.enableSetNeedsDisplay = true
        
        
        if let metalDevice = MTLCreateSystemDefaultDevice() {
            mtkView.device = metalDevice
        }
        
        mtkView.framebufferOnly = false
        mtkView.drawableSize = mtkView.frame.size
        return mtkView
    }
        
    func updateUIView(_ uiView: MTKView, context: UIViewRepresentableContext<ContentView>) {
        // Place Holder function for now. We'll implement this later but for now, we need it to meet the UIViewRepresentable Prototype requirements 
    }
    
}

Renderer Class

Next we create a class to handle our drawing. For this section, all we’re really doing is setting a background color. In the future, we will make our drawing calls from here.

import MetalKit

///Put metal code here
class Renderer : NSObject, MTKViewDelegate {
    var parent : ContentView
    var device : MTLDevice!
    var commandQueue : MTLCommandQueue!
    
    init(_ parent : ContentView) {
        self.parent = parent
       
        if let device = MTLCreateSystemDefaultDevice() {
            self.device = device
        }
        
        self.commandQueue = device.makeCommandQueue()
        
        super.init()
    }
    
    
    func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
    }
    
    ///Drawing Pass
    func draw(in view: MTKView) {
        guard let drawable = view.currentDrawable else {    return  }
        
        let commandBuffer = commandQueue.makeCommandBuffer()!
        let renderPassDescriptor = view.currentRenderPassDescriptor!
        //Render Pass - Clear - Set Clear Colour
        renderPassDescriptor.colorAttachments[0].clearColor = MTLClearColor(red: 0.0, green: 0.5, blue: 0.5, alpha: 1.0)
        renderPassDescriptor.colorAttachments[0].loadAction = .clear
        renderPassDescriptor.colorAttachments[0].storeAction = .store
        
        let renderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor)!
        renderEncoder.endEncoding()
        
        //Commit and present the state of the buffer
        commandBuffer.present(drawable)
        commandBuffer.commit()
        
    }
    
}

Closing Words

In the next article, I’ll do the Graphics equivalent of Hello World, draw a triangle and some other cool things as I learn the Swifty way doing them. This portion was focused on setting up your project. Once more, special shout out to @GetIntoGameDev’s YouTube channel. You can find their GitHub repo documenting a similar set up here: https://github.com/amengede/getIntoMetalDev

Alternatively, you can follow this GitHub Repo that I will push my code to as I develop it and also have links to all the future articles I write in this series.

If I’ve made any errors, or you have a really cool way of implementing something, feel free to leave a comment.

Options: Virtualization? Porting?

Before, diving into my exact set up, I’d like to share some context on my choices, in case you’d like to further research.

Porting over newer OpenGL to MacOS

Initially, I hoped that despite Apple’s discontinued support for OpenGL, there would be a dedicated group of developers unofficially porting it over. I would like to acknowledge the OpenGL 4.6 on Metal project on GitHub for doing exactly that. Unfortunately, this was not a good fit for my use case but could be perfect for yours.

Virtualization and Emulation

As a result, I concluded that it would be better for me to have an easily accessible Linux distro on my computer instead. Which involved its own configuration quirks, especially on Apple Silicon Macs.

After doing some research, I found that popular methods of running Linux VMs included:

• Parallels (Paid)
• VM Fusion (Free if for individual use)
• UTM (Free)

Personally, I ended up using VM Fusion with ARM Ubuntu Server with Ubuntu Desktop GUI installed onto said server. Why? Because, as far as I know, there’s no ARM version of Ubuntu Desktop ISO but there is one for Ubuntu Server. Thus, having Apple Silicon Macs have a few extra steps to follow. Please feel free to leave a comment, if I have made any mistakes or if there’s an easier way to get this set up.

For setting up Ubuntu Server after installing it with one of the options above. You can follow this tutorial. It covers the UTM process in great detail, however, VM Fusion users should also follow step 4 onwards.

Once you get the Virtual Machine running, shared folders set up and can confidently build your OpenGL project on the Linux side. I would strongly recommend making a snapshot of your VM here.

Running Modern OpenGL on macOS: A Virtualized Ubuntu Approach was originally published in CodeX on Medium, where people are continuing the conversation by highlighting and responding to this story.

Recently, Meta released LLAMA 3 and allowed the masses to use it (made it open source).

Wanting to test how fast the new MacBook Pros with the fancy M3 Pro chip can handle on device Language Models, I decided to download the model and make a Mac App to chat with the model from my Menu Bar at all times. In this article, I will you teach you, YES YOU, how to deploy your own language model locally onto your Mac.

Setting Up Ollama / Deploying a Local Model

This part is relatively simple. Thank you Ollama developers!

First step, ollama.com and follow their download and set up instructions.

Namely, you will download the Ollama App, after opening it, you will go through a set up process that installs Ollama to your Mac.

After installing Ollama, we can download and run our model. For this article, we will use LLAMA3:8b because that’s what my M3 Pro 32GB Memory Mac Book Pro runs the best.

To do that, we’ll open the Terminal and type in

 ollama run llama3

This will download the model and start a Text Interface where you can interact with the model via the terminal.

Since I have run this command before, using this command simply starts the model and does not cause it download once more.

You can exit this interface with the command:

Control + d

Show Me The Code / Making A Mac Menu Bar App To Chat With The Local Model

By the end, our app will look like this:

If you want to see the full project, it’s on my GitHub, here https://github.com/carlosmbe/MyMacLLAMA

I will be updating the project there with new features every once in a while. So this article will focus on the basics and essentials.

The Data Model and Fun Logic:

Things to Note:
1. The address, http://127.0.0.1:11434/ is the local host at port 11434. Which the default port that Ollama runs on. What we’re doing here is doing an API call to our own device as the server.

2. Furthermore, since this is an API Call, we need explicitly request permission for incoming and outgoing network calls in the App capabilities.

import Foundation

// Struct to decode the JSON response
struct Response: Codable {
    let model: String
    let response: String
}

// Class for managing application data and network communication
class DataInterface: ObservableObject, Observable {
    
    // Store the current prompt as a modifiable string
    @Published var prompt: String = ""
    // Store the response to the prompt as a modifiable string
    @Published var response: String = ""
    // Track whether a network request is currently being sent
    @Published var isSending: Bool = false

    // Function to handle sending the prompt to a server
    func sendPrompt() {
        print("Started Send Prompt")  // Log the start of sending a prompt
        // Prevent sending if the prompt is empty or a request is already in progress
        guard !prompt.isEmpty, !isSending else { return }
        isSending = true  // Mark that a sending process has started
        
        // Define the server endpoint
        let urlString = "http://127.0.0.1:11434/api/generate"
        // Safely unwrap the URL constructed from the urlString
        guard let url = URL(string: urlString) else { return }
        
        // Prepare the network request with the URL
        var request = URLRequest(url: url)
        request.httpMethod = "POST"  // Set the HTTP method to POST
        request.addValue("application/json", forHTTPHeaderField: "Content-Type")  // Set the content type to JSON
        let body: [String: Any] = [
            "model": "llama3",  // Specify the model to be used
            "prompt": prompt,  // Pass the prompt
            "options": [
                "num_ctx": 4096  // Specify context options
            ]
        ]
        // Encode the request body as JSON
        request.httpBody = try? JSONSerialization.data(withJSONObject: body)
        
        // Start the data task with the request
        URLSession.shared.dataTask(with: request) { data, response, error in
            defer { DispatchQueue.main.async { self.isSending = false } }  // Ensure isSending is reset after operation
            if let error = error {
                DispatchQueue.main.async { self.response = "Error: \(error.localizedDescription)" }  // Handle errors by updating the response
                return
            }
            
            // Ensure data was received
            guard let data = data else {
                DispatchQueue.main.async { self.response = "No data received" }  // Handle the absence of data
                return
            }
            
            let decoder = JSONDecoder()  // Initialize JSON decoder
            let lines = data.split(separator: 10)  // Split the data into lines
            var responses = [String]()  // Array to hold the decoded responses
            
            // Iterate over each line of data
            for line in lines {
                if let jsonLine = try? decoder.decode(Response.self, from: Data(line)) {
                    responses.append(jsonLine.response)  // Decode each line and append the response
                }
            }
            
            print(responses)  // Log all responses
            
            DispatchQueue.main.async {
                self.response = responses.joined(separator: "")  // Combine all responses into one string
                print(self.response)  // Print the full response
            }
        }.resume()  // Resume the task if it was suspended
    }
}

The User Interface

The Menu Bar:

We will go to the App file to make the app operate solely in the Menu Bar. For more information on the Menu Bar Section. Check out this article, its what I used as a foundation.
https://sarunw.com/posts/swiftui-menu-bar-app/

import SwiftUI

@main
struct MyMacLLAMAApp: App {
    var body: some Scene {
        
        //Create instance of Data Interface for the app
        @StateObject var appModel = DataInterface()
        
        //Create a Menu Bar for our app and have it have the brain icon
        MenuBarExtra("My Mac LLAMA Bar", systemImage: "brain"){
            //When Clicked The Menu Bar Will Show the content View
            ContentView()
                .environment(appModel)// Pass the appModel environment object to ContentView

        }
        //This modifier lets us show a Window in the Menu Bar
        .menuBarExtraStyle(.window)
        
        
    }
}

Content View:

To make the user interface, I will make a simple Content View with a TextField, a Submit Button and a Text View for the Response.

struct ContentView: View {
    
    // I will use the EnvironmentObject property wrapper to share data between this view and others 
     @EnvironmentObject var appModel: DataInterface
    
    var body: some View {
        VStack {
            
            // TextField for the user input .
            TextField("Prompt", text: $appModel.prompt)
                .textFieldStyle(.roundedBorder) 
                .onSubmit(appModel.sendPrompt) // Send the prompt to Ollama and get a response
            
            // Divider draws a line separating elements
            Divider()
            
            // Use an if statement to conditionally display a view depending on if appModel.isSending.
            if appModel.isSending{
                ProgressView() // Display a progress bar while waiting for a response.
                    .padding() 
            }else{
                Text(appModel.response) // Display the response text from appModel if not currently sending.
            }
            
           
            HStack{
                
                // Button to send the current prompt. It triggers the sendPrompt function when clicked.
                Button("Send"){
                    appModel.sendPrompt()
                }
                .keyboardShortcut(.return) // Assign the return key as a shortcut to activate this button. Cause Mac.
                
                // Button to clear the current prompt and response.
                Button("Clear"){
                    appModel.prompt = "" // Clear the prompt string.
                    appModel.response = "" // Clear the response string.
                }
                .keyboardShortcut("c") // Assign the 'c' key as a shortcut to activate this button. So Command + C
                
            }
        }
        .padding()
    }
}

Links:

Final Project: https://github.com/carlosmbe/MyMacLLAMA

Ollama: https://ollama.com

Tutorial on Menu Bar Mac Apps: https://sarunw.com/posts/swiftui-menu-bar-app/

Building A Local OLLAMA App for your Mac with Swift was originally published in CodeX on Medium, where people are continuing the conversation by highlighting and responding to this story.