For those of us that have been using it from the start, it can be hard to believe that Direct3D 12 has been around for ten years now. Windows 10 was released on July 29th 2015, and D3D12 has been with us ever since. While it’s true that this is the longest we’ve gone between major D3D version updates, it’s also not fair to say that the API has remained static. On the contrary, D3D12 has received a steady stream of new interfaces, functions, and shader models. These updates have included some very major features that have gathered lots of press and attention, such as DXR or Work Graphs. But there’s also been a long line of smaller features and quality-of-life updates that can make a developer’s life easier. In addition, HLSL as a shader language has also seen steady updates, with some of them being rather transformative.

Given the recently-passed 10 year anniversary, I thought it would be a good time to look back at what’s changed in the world of D3D12 since it was released. After that, I thought it would also be interesting to reflect on what’s changed for me personally as my coding and development style has evolved along with APIs and programming languages. In some cases I’ve been able to streamline things quite a bit thanks to API enhancements, and in others I’ve just embraced new ways of doing things.

Let’s dive In! Feel free to use the TOC below if you want to skip around.

Additions To the Core D3D12 API

Programmable Sample Points

Spec Link

This is not a commonly used feature, since its use cases are pretty niche and MSAA is no longer popular. In short, this adds a new API that lets you provide your own custom MSAA subsample positions instead of using the rotated grid pattern that the “standard” sample provides:

The API allows you to specify the positions over a 2x2 quad of pixels rather than just a single pixel, which you could potentially use to get a more irregular sampling pattern that changes aliasing characteristics. Or at least, you can do that if you still care about MSAA. :)

The other main use case is for software implementations of variable rate shading, which is something that Activition has presented in the past on more than one occasion. It’s also something I’ve dabbled in myself in the past. The reason programmable sample points are useful is so that you can generate a “half-res” depth buffer with 4x MSAA and have the depth subsample positions exactly match what was used for the full-res pass. Of course these days you also have hardware VRS as another option (which we’ll get to soon!), which renders this use case fairly niche.

View Instancing

Spec Link

Another less-commonly-used feature is View Instancing. This provides API and shader mechanisms that try to make it easy and optimal to render in stereo for VR or 3D. The way it works is your draws will get instanced N times (for up to 4 views), and in any shader you can take SV_ViewID as an input to know which view you’re drawing to. This allows you to choose and apply the appropriate view + projection matrices for that view. Then on the app side of things you specify whether you want each view to get mapped to a different viewport, or to a different slice of an array texture. You can also setup a bitfield mask on the app side, which allows you to cull separately for each view if you want to.

In theory this API could benefit hardware that has some native functionality for broadcasting to multiple views if used correctly, which as far as I know is really just Nvidia with its “viewport broadcast” feature of the GS. In practice I’m not sure it really provided any serious benefit, since I imagine most apps would consume SV_ViewID in the vertex shader rather than trying to use it later in the pipeline. AMD also has never exposed more than Tier 1 support, which means there’s some sort of “draw-call level looping” going on rather than anything fancy. If that’s the case, you’re likely better off by just doing it yourself with regular instancing + SV_RenderTargetArrayIndex/SV_ViewportArrayIndex being output by the vertex shader.

Depth Bounds Test

Spec Link

The depth bounds test is old…so old it was initially added to accelerate stencil shadows. Remember when Doom 3 was new? It’s been exposed in other APIs in various ways, but somehow it never got officially added to D3D until D3D12. Either way it’s now exposed if you have a good use for it. Basically how it works is on the command list you specify a min and max depth value which serve as the “bounds”. Then when you draw some triangles, each pixel checks the existing depth value (not the depth of the triangle being rasterized) and passes the depth test if the value is within the bounds that were provided.

I’m not sure what use cases are still out there for this feature…I know some renderers have used it for old-school style deferred rendering where light bounding volumes were rasterized one at a time. Deferred decals that are rasterized as bounding boxes could also potentially benefit in the same way. There’s also an old trick from the PS3 days for rendering deferred cascaded shadows one cascade at a time, where you would set the depth bounds to the cascade min/max depth and then draw a full-screen triangle.

Variable Rate Shading

Spec Link

Variable Rate Shading, or VRS for short, is all about trying to reduce expensive pixel shader threads for places where you don’t need it. The idea is that you choose a “shading rate”, and then the pixel shader will only run for 1 / N pixels. The nice part is that you still get full-resolution depth testing, which you won’t get if you draw things to a separate half-res render target. VRS also gives you very fine granularity for choosing the shading rate, which makes it much easier to mix lower shading rate content with full shading rate content. That can be particularly important for particles and transparents if not using OIT, since you need to draw them in back-to-front order. A separate half-res pass with a composite makes draw-based sorting impossible without introducing many low-res passes, which would end up negating any performance benefits.

The API allows you to specify the shading rate in 3 ways:

• Per-draw, from the command list
• Per-primitive, by emitting SV_ShadingRate from a vertex shader/mesh shader/geometry shader
• Per-tile in screenspace, by generating a shading-rate texture (with maximum 16x16 tile size)

In particular the tile-based shading rate is useful if you want to try to analyze the previous frame contents and use that to automatically generate an optimized shading rate based on the variance of the content. Whereas per-draw is probably more useful for manually tagging half-res transparent or particle draws.

There’s two main downsides to watch out for with VRS:

• The performance gains may not be as high as you’d expect, because the hardware is still going to need to launch pixel shaders in 2x2 quads and is also still going to need to fill full waves before it can execute.
• Any low-res draws or tiles will broadcast the coarse pixel output value to the actual pixels, effectively point sampling the results. So you probably want to add some extra spatial filtering and/or temporal reconstruction to get better quality.

Relaxed Format Casting

Spec Link

This is not so much a new API feature as it as a relaxing of the rules around creating multiple views with “compatible” formats. Initially in D3D12 the rules were the same as D3D11, which required you to create a texture resource with a TYPELESS format if you were going to view it through multiple compatible formats. So for example you would use R16_TYPELESS if you were going to view the texture as both R16_UINT and R16_UNORM. This is a bit inconvenient, since you have to know in advance if you’re going to view the resource with multiple formats. It can also result in pessimizations from the driver, since it may not be able to use certain compressions or optimizations if it has to assume that you will view the texture as both integer and floating-point data.

With relaxed format casting this is no longer necessary: you can create the resource fully “typed”, and then create compatible views all you’d like. The catch is that you can’t cast between floating-point and non-floating point formats, which avoids the situation you can get with TYPELESS where the driver might disable optimizations. This is especially nice for depth buffers, which previously required you to use TYPELESS if you were ever going to read the depth texture through a SRV.

Directly Writing To Block-Compressed Formats

Spec Link

Historically, the only way for the GPU to write to a texture with a block compressed format was to copy to it. To allow for performing block compression on the GPU, there was a “backdoor” added in D3D10 that allowed you to copy from a 1/4 width and 1/4 height texture with an equivalent UINT format (R32G32_UINT or R32G32B32A32_UINT) into a BC-formatted texture. This is obviously not ideal, since it would be faster to just write directly to the final texture instead of requiring an intermediary copy. As part of Enhanced Barriers (which I’ll cover later), D3D12 caught up with Vulkan and gained the ability to write directly to the BC texture through a UAV. The way it works is similar to Vulkan: when you create the texture you pass a “castable formats” array, and that allows you to include the equivalent UINT format that you’ll use for writing:

• R32G32_UINT (8 bytes per block) for BC1 and BC4
• R32G32B32A32_UINT (16 bytes per block) for BC2, BC3, BC5, BC6, and BC7

Writable R9G9B9E5 Textures

Spec Link

DXGI_FORMAT_R9G9B9E5_SHAREDEXP is an interesting format: it’s similar to R11G11B10_FLOAT in that it allows for storing positive floating point RGB values with 4 bytes per texel, and provides the same effective range as fp10/fp11/fp16 by virtue of having same number (5) of exponent bits. However unlike fp10/11, with R9G9B9E5 each channel gets 9 mantissa bits instead of the 6 or 5 that you get with R11G11B10. Therefore it can provide more precision and less visible banding in some cases. The catch is that the exponent is shared among the 3 channels, so it’s going to work best when the overall magnitude is highly correlated among the 3 channels.

Traditionally this was a read-only format, where GPUs could automatically decode from it when sampling as a texture but had no means of automatically encoding to it for RTV or UAV writes. Your only option would be to write to it through a UAV that uses R32_UINT, and encode the bits yourself. With this new feature, capable GPUs can indicate that they support writing to DXGI_FORMAT_R9G9B9E5_SHAREDEXP through render targets or typed UAVs.

At the time of writing, support for this feature seems to be limited to AMD GPUs that are RDNA2 or newer.

WriteBufferImmediate and OpenExistingHeapFromAddress

Spec Link

If you write D3D12 code, you’re eventually going to crash the GPU. When this happens, it’s nice to have even some vague information about what the GPU was doing when it gave up the ghost. WriteBufferImmediate and OpenExistingHeapFromAddress are a pair of APIs that aim to let you gather this sort of information by implementing “breadcrumbs”. Basically you use WriteBufferImmediate to write some sort of signal value at various points in your command list recording. Then when your device is removed, you use OpenExistingHeapFromAddress to access the breadcrumb buffer on the CPU and inspect it to figure out the last-written breadcrumb.

This functionality is also used under-the-hood by DRED, which I’ll cover in more detail later on.

GPU Upload Heaps

Spec Link

One of the early pain points for beginner D3D12 programmers is dealing with the fact that dedicated video cards have their own physical pool of VRAM that’s separate from the “SysRAM” that’s used by the CPU. Traditionally VRAM has not been directly accessible to the CPU, aside from small 256 MB region of VRAM accessible through the Base Address Register (BAR). This leads to tough choices about where to place resources, and how to use the GPU to copy from SysRAM to VRAM.

New systems and video cards have something called “Resizable BAR” which allows the OS to effectively map all of VRAM into the CPU address space. D3D12 exposes this functionality through GPU Upload Heaps. This is a new D3D12_HEAP_TYPE_GPU_UPLOAD enum value that you can use when creating a heap or committed resource, which can then be mapped into the CPU virtual address space using Map(). That means you can have the CPU directly update the contents of a resource that lives in VRAM, without having to use GPU copies. For CPU-updated buffers this a no-brainer: the data layout is linear and so you can just memcpy away. For textures it’s a bit more complicated…they use an opaque layout, and so you have to use WriteToSubresource if you want to update the contents.

AMD started supporting ReBAR for RDNA2, and Nvidia started supporting it for Ampere. Intel’s dedicated GPUs all support it. Note that you typically need to enable ReBAR in your UEFI settings before it can be used. GPU Upload Heaps are also unfortunately only available on Windows 11, since it required kernel changes.

ExecuteIndirect Incrementing Constant

Spec Link

ExecuteIndirect is really cool. It wraps up indirect drawing, indirect dispatches, and multi-draw/dispatch through a unified interface. But one major deficiency relative to Vulkan has been the lack of a “DrawID” that the shader can use to access its draw-specific or mesh-specific data. ExecuteIndirect is flexible enough to let you do this yourself by setting a root constant for each draw as part of your command signature, but it turns out that this requires some rather fancy footwork from drivers to make this run as expected. Whereas generating a DrawID is something a lot of hardware can do natively.

Incrementing constant (mostly) closes this gap by letting you have an implicit constant buffer value passed to your shader that increments for each draw or dispatch. While it’s not quite as nice as a SV_DrawID would be, it gets the job done.

At the time of writing this, Nvidia is the only vendor that reports support for Tier 1.1 execute indirect support with incrementing constant (which it does for all hardware going back to Pascal). I assume AMD and Intel will catch up at some point.

Sampler Feedback

Spec Link

Those of you who were around for the launch of Nvidia’s RTX 2000 series (AKA Turing) might remember some of their claims of support for “texture-space shading”. That sure sounded really cool, but it turns what they actually had was sampler feedback. Basically this is a feature that lets you figure out which texels (within some granularity) were accessed given a UV and texture description. This gives you some rather precise data that you could use as part of feedback system to determine residency for a sparse virtual texture (or which texels you should shade for texture-space shading).

The big downside is that the API is pretty complicated. You have to setup a resource for storing the initial opaque feedback data, you have to bind an extra descriptor, you have to add the feedback generation to the shader, you have to “resolve” the feedback data into a readable format, and then finally you have to copy back to the CPU if you’re going to read it there. After all of that, you can access your info about which regions were sampled. In my experience it’s a lot more unwieldy than simply doing an atomic write from your shader, and it’s also more expensive. So I probably wouldn’t recommend it unless you need very exact feedback.

Nvidia supports this for Turing or newer, while AMD supports it for RDNA2+. Intel supports it for Xe or newer.

Tight Resource Alignment

Spec Link

From the start, D3D12 made the decision to impose fixed alignment requirements for resources. This is simpler compared to Vulkan, where alignments are reported by the device properties at runtime. The big downside is that the D3D alignments can be much larger in some cases, which leads to a lot of waste. In particular the dreaded D3D12_DEFAULT_RESOURCE_PLACEMENT_ALIGNMENT is 64 KB for buffers, which can quickly bloat memory usage if creating many placed resources for tiny buffers.

With “tight placed resource alignment”, you can now pass a new D3D12_RESOURCE_FLAG_USE_TIGHT_ALIGNMENT flag that supports much smaller alignment values. For buffers the range is now 8 to 256 bytes, while textures have 8 byte to 64 KB alignment for non-MSAA (8B to 4 MB for MSAA). This makes small placed buffer resources much more practical.

As of writing, this feature is still in preview and is only available in the 1.716.1 preview Agility SDK.

Render Passes

Spec Link

If your target GPU is a tiler, then you really want render passes. These replace calls to OMSetRenderTargets with pairs of BeginRenderPass and EndRenderPass calls. Importantly, BeginRenderPass takes a bunch of flags that indicate the apps expectations about the lifetime of the render target data as well as whether previous data should be preserved. These flags allow tilers (like the one in Qualcomm’s Adreno line) to make optimal decisions about how to handle on-chip memory, and allow them to potentially skip loading data in and out of on-chip memory which consumes considerable bandwidth.

Enhanced Barriers

Spec Link

D3D12 and Vulkan ended up diverging quite a bit in how they initially exposed barriers in their respective APIs. D3D12 opted for a “resource state” abstraction, where barriers would indicate “transitions” from one state to another which the driver could use to infer which steps are required for ensuring proper data visibility and synchronization. Vulkan opted for a more explicit approach where the thread synchronization, cache flush requirements, and texture data transformations were all represented as separate pairs of before/after access points.

While it was perhaps an interesting idea, D3D12’s approach ended up being rather confusing and suboptimal in practice. Representing every visibility change as a transition gave some programmers the wrong mental model of what GPUs were actually doing under the hood, and in other cases the drivers were forced to insert excess flushes to ensure safety. By far the worst part was the concept of resource state “promotion” and “decay”. The underlying concept is actually fairly simple if you look at it from hardware level: when you submit a batch of command lists through ExecuteCommandLists, the runtime enforces an implicit synchronization and cache flush after that batch finishes. This means there’s no need to synchronize because of a dependency that spans multiple ECL batches: there was already a global sync/flush after the first ECL batch, so no need to do it again. There’s no implicit layout changes for textures since that can’t be done without knowing the future usage of the texture, so that still needs to be done with barriers as usual. The problem is that exposing this implicit sync through the legacy barrier system was awkward and messy: it simply doesn’t map cleanly to an abstraction that uses “resource states” for everything. This is why the documentation around implicit promotion and decay is so convoluted: it has to somehow describe how certain implicit state transitions are allowed in certain situations, while excluding anything that would cause a layout change for textures.

Enhanced Barriers essentially takes the Vulkan approach, where you have sync, flush, and layout options as part of the barrier. This is perhaps more initially challenging to wrap your head around as a beginner compared to legacy barriers, but in the long run it ends up being much simpler IMO. The implicit sync/flushes from ECL batches is now very easy to account for with Enhanced Barriers: you can skip any barriers that would only do one of those things if the dependency crosses ECL batches, but you still have to issue layout transition barriers. There’s also a few more benefits spelled out in the Enhanced Barrier spec, such as avoiding cases where previously the driver would require an additional flush. You can also now discard a resource as part of the barrier, which is another very welcome convenience that can save you additional commands and barriers.

Ray Tracing

Spec Link

I imagine I hardly need to explain DXR here, as this is perhaps the most marketed and well-known addition to D3D12. In case you’ve somehow missed all of that, DXR brings hardware ray tracing support to D3D12. It’s exposed in two ways:

1. The DXR 1.0 DispatchRays pipeline, where you get a very structured setup of providing RayGen, ClosestHit, Miss, AnyHit, or Intersection shaders as part of a big “RTPSO”. With this setup you trace rays from your RayGen shader (which is like a very simplified compute shader), and then based on what that ray hits (or doesn’t hit) the GPU will automatically invoke the right shader for you.
2. The DXR 1.1 RayQuery pipeline, also known as “inline tracing”, which much more directly exposes the hardware ray tracing functionality to existing shader types. With this setup there’s no “RTPSO” or “shader tables”, you just issue the query as part of a loop and then get the intersection info. From there it’s completely up to you what you do in response to a hit, and there’s no automatic calling of hit shaders or anything like that.

For either approach you need to build an “acceleration structure” for your scene, which contains the BVH or whatever format the hardware is using to accelerate ray/triangle intersections. These AS builds happen on the GPU, and you have to initiate them by providing a list of meshes along with pointers to their position and index data. These “bottom level acceleration structures” (BLAS) can then be instanced into a “top level acceleration structure” (TLAS), which can allow you to replicate the same mesh or meshes in multiple locations without duplicating the memory. You can also move meshes around by only rebuilding the simpler TLAS rather than having to fully rebuild the BLAS. Animated/deformable meshes are much trickier though…these need to have their BLAS constantly rebuilt as the positions move from frame to frame. For these you have the option of doing a more expensive BLAS rebuild, or doing a cheaper BLAS refit that reuses the same BVH tree structure. For the latter you need to be careful though, as the quality of the tree can become worse with more refits if the vertex positions drift too far from the original positions that were used for building the BLAS. If this happens, ray traces will take longer against that mesh since more traversal steps will be needed. In such cases it can be helpful to periodically perform a full rebuild of the BLAS after N refits. In practice there’s quite a few trade-offs that you can make regarding AS build time, trace performance, and memory consumption.

A future DXR 1.2 update is going to add Shader Execution Reordering, which allows some GPUs to re-sort and re-pack waves based on some user-provided sort key. The intention there is that you can reduce divergence for hit shading by re-packing your waves based on a material ID or similar, and that way you end up with more complete waves executing a single shader path.

DXR is fully supported on AMD RDNA2 or newer, Nvidia Turing or newer, and Intel Xe-HPG or newer. Nvidia does also support the more limited Tier 1 on certain Pascal GPUs, which only supports DXR 1.0-style RayGen tracing and not RayQuery/inline tracing. This hardware actually has no build-in hardware acceleration for ray tracing, so personally I wouldn’t recommend bothering with that unless it’s your only option.

Mesh Shaders

Spec Link

Mesh Shaders are another well-known addition that originally came from Nvidia’s Turing architecture. Mesh Shaders actually have two separate parts: the Mesh Shader, and the Amplification Shader. The latter is optional, so we’ll talk about the Mesh Shader first.

When you use Mesh Shaders, you no longer use the well-known Vertex Shader, Hull Shader, Domain Shader, or Geometry Shader phases for processing your vertices prior to rasterization. Instead, Mesh Shaders expose a more compute-like setup where you have a thread group that produces a batch of vertices and local indices that together form a batch of triangles. The main thing you gain is significant flexibility: there’s no longer any fixed-function index buffers or fixed-function vertex fetch, it’s completely up to your code to decide how to produce the batch of triangles. Since you’re working with a thread group that can make use of groupshared memory, you can also coordinate between threads in a group to work together to produce the output data. This allows for interesting possibilities of procedurally generating geometry, calculating per-triangle data, or de-compressing vertex data.

The typical way that Mesh Shaders get used is for meshlets, also known as clusters. Basically you split your meshes into chunks of up to N vertices and M triangles (say 64 vertices and 128 triangles), and you can then cull those meshlets individually to get fine-grained culling. These map very well to mesh shaders, since you can have one thread group for each meshlet. Inside of a mesh shader it’s then possible to do things like de-compressing the quantized and normalized vertex data based on the meshlet bounds. Or you can even cull at the individual triangle level if you want to, and only output the triangles and vertices that pass culling. Care is needed though when generating meshlets and writing the mesh shader. With mesh shaders you forego any automatic “caching” of vertex shader outputs, and it’s up to you to reduce duplicate vertex loads and transformations by minimizing the number of “shared” vertices between meshlets. Different GPUs also have different performance characteristics around vertex/triangle output counts for mesh shaders, or culling individual triangles.

Amplification Shaders are the other half of what you get with Mesh Shaders. As I mentioned these are optional, as you can invoke Mesh Shader groups directly if you’d like. If you have an Amplification Shader, this is an additional shader stage that runs right before the Mesh Shader phase. Amplification Shaders are also compute-like in that they are dispatched as thread groups, but their special ability is that they can dispatch Mesh Shader groups from the shader with DispatchMesh(). When you do this you pass a payload along, which the Mesh Shader can then use to figure out which meshlet (or similar) needs to be process. The natural use case is to do culling and/or LOD selection right in the Amplification Shader, and then only dispatch mesh shader groups for meshlets that pass culling. This can be super convenient since you don’t need to setup separate compute shader dispatches, output buffers, or indirect arguments: it all happens right in the graphics pipeline. On some hardware this may also provide some benefits in terms of keeping more of the data “on-chip”. But on the other hand, for other hardware it might be a less optimal than doing a compute + indirect dispatch approach since the latter may do a better job of completely filling large GPUs with work. In particular AMD can struggle with Amplification Shaders that cull too much (or Mesh Shaders that don’t do much work), since they use a ring buffer through the command processor that can fill up.

Work Graphs

Spec Link

Work Graphs are another rather new and well-publicized addition to D3D12. They’re also a bit complicated, but I’ll try to give a very brief overview. In short, Work Graphs are a new paradigm for having the GPU feed work to itself. Rather than relying on ExecuteIndirect, you can setup a graph of compute nodes where one node can launch groups of threads for another node that it’s connected to. So for example you might have a node that classifies a tile of pixels, and then it chooses one of N specialized nodes based on that classification. The first node can then launch a grid of threads using that selected node, effectively choosing the appropriate shader to use on-the-fly without any intermediate steps. There are definitely some limitations…in particular the topology of the graph has to be statically known when building the PSO before executing a graph. There’s is also no recursion allowed across groups of nodes, you can only recurse into the same node. But it’s still quite flexible and powerful compared with the traditional way of launching indirect dispatches.

Since Work Graphs are quite new, there’s still multiple big question marks around them. In particular, performance. On the RDNA3 hardware that launched Work Graphs, it was definitely quite easy to create simple graphs that were handily outperformed by an ExecuteIndirect equivalent. However AMD has also provided some more complex counterexamples where they were able to (modestly) beat the performance of ExecuteIndirect while also using far less memory. With performance improvements not being guaranteed, it’s harder to judge if Work Graphs will truly take off as the future way of scheduling GPU-driven compute. It’s something you sort-of need to go all-in on if you’re going to get the benefits, and it’s not clear yet if the benefits justify such a drastic change.

Work Graphs also have preview support for mesh nodes, which are essentially just Mesh Shader thread groups. This means you can directly create triangles and rasterize them from a Work Graph, which potentially allows building a GPU-driven mesh pipeline within a single large graph. This mesh nodes feature has yet to be released at the time of writing, and is available in the 1.715.0 or newer preview Agility SDK.

Currently, Work Graphs are supported on RDNA3 or newer for AMD, and Ampere or newer for Nvidia. Intel has no support across their product line.

New Shader Features

Wave Ops

Doc Link

Wave operations are a powerful and difficult-to-use addition to HLSL that came along when DXC/DXIL was first released. These intrinsics allow for simple communication between threads within a “wave” (also known as a warp, wavefront, or subgroup), which can be much faster and simpler than communicating through groupshared or global memory. The simplest of these are the broadcast operations, which allow one thread to see the value of a variable in a particular neighboring thread. The two intrinsics that can do this are WaveReadLaneFirst and WaveReadLaneAt, the latter of which allows you to specify the index of the thread/lane from which the value will be broadcast. Another common operation is the “ballot” (WaveActiveBallot) operation, which gives you a bitmask indicating if a condition is true or false across the entire wave. This can then be used to build higher level operations to determine if a condition is true for all threads in the wave, or if it’s true for any thread in the wave. These two operations are conveniently exposed as WaveActiveAllTrue and WaveActiveAnyTrue, and can be useful for crafting branches that never have divergence. Finally, there’s the reduction and scan intrinsics. The reductions, such as WaveActiveSum, perform some mathematical operation across all threads in the wave and return the result. Whereas the scan intrinsics, such as WavePrefixCountBits, will perform a prefix sum or prefix product across the wave and leave each thread with the cumulative result up to that lane.

Wave operations are part of many interesting low-level optimizations, such as scalarization. However they have to be used with considerable care to not produce undefined results. Wave sizes vary across different hardware vendors, or even across different shaders on the same hardware. It’s very very easy to write some wave-aware code that happens to work on the one wave size you’ve tested, but breaks badly on a different wave size. In general the HLSL language was also designed around the idea of threads being completely independent, and wave operations completely break those assumptions in similar ways to groupshared memory. This again makes it very easy to introduce undefined behavior, particularly when it comes to branching and inactive lanes.

AMD hardware exclusively used Wave64 up through Vega, but all RDNA+ GPUs support both Wave32 and Wave64. Nvidia exclusively supports Wave32 across its entire product line. Intel historically has been quite different with their wave sizes, and has used some combination of 8, 16, and 32-sized waves depending on the architecture.

Quad Ops

Doc Link

Quad operations are very similar to wave operations, except that they operate on the 4 pixel shader threads that form 2x2 quads during rasterization. A lot of these quad operations could already be done in earlier versions of HLSL through clever usage of ddx_fine() and ddy_fine(), but the quad operations make things a lot simpler. In particular you can pretty easily implement a quad overshading visualizer using a few quad operations combined with the input coverage mask.

16-bit floats and integers

Doc Link

DXC and Shader Model 6.2 (re-)introduced lower-precision types and math operations to HLSL. In particular, 16-bit float, integer, and unsigned integer types. If nothing else, these are great for packing data more tightly into buffers without having to write manual conversion and bit-packing code. But if used carefully, it’s possible to get improved ALU throughput and/or reduced register usage for a shader. However this is pretty tricky to pull off in practice when retrofitting existing code, since it’s very easy to accidentally introduce hidden implicit conversions between fp16 and fp32. Additionally, some hardware (AMD notably) can only achieve higher math throughput through use of “packed” instructions that perform the same fp16 math operation on two independent sets of inputs and outputs.

16-bit types are supported on AMD Vega or newer, Nvidia Turing or newer, and all DX12-capable Intel GPUs.

Barycentrics

Doc Link

Barycentrics are essentially coordinates on the surface of the triangle that can be used to interpolate the per-vertex attributes from the 3 triangle vertices. Most modern hardware does this interpolation with some degree of software at the beginning of the pixel shader, and thus they can expose the barycentrics used for this purpose to the shader author. This allows you to implement your own interpolation if you wish, potentially compressing and packing the attributes more compactly than they otherwise would be if using automatic interpolation. In addition, you can also do neat things like anti-aliased wireframe rendering directly in the pixel shader without requiring an additional pass (or a dreaded geometry shader!)

Barycentrics are supported on AMD RDNA 2 or newer, Nvidia Turing or newer, and Intel Xe2 or newer.

Shader Libraries & Linking

The HLSL lib_6_x profile for compiling shader libraries was originally added as part of Shader Model 6.3 for use with ray tracing state objects. While it’s not a well-advertised feature, they can also be used for general offline shader linking of standard graphics and compute shaders. Basically you compile your libraries first with the entry points tagged with the appropriate shader stage attribute, and then you can use DXC to link the entry point with functions defined in a library. It seems DXC then optimizes across the link boundary, so in theory it shouldn’t result in worse DXIL codegen compared to compiling without linking. This makes it a potentially useful option for reducing compilation times.

Helper Lane Detection And Control

Spec Link

Helper lanes are special pixel shader threads that are spawned in order to “fill out” missing entries from the 2x2 quads that pixel shaders are launched in. These lanes do not produce any kind of output, but still perform the usual calculations alongside its neighbors so that derivative operations work correctly when differencing across the quad. The standard way to get these is at triangle edges: all pixel shader threads within a quad have to belong to the same triangle, and so a partially-covered quad will have helper lanes spawned to fill out the quad (even if that pixel isn’t covered by any triangle). However it can also happen dynamically during the execution of a pixel shader. In particular a discard will “demote” a pixel shader thread to a helper, so that the remaining “alive” threads in the wave can continue calculating derivatives and sampling textures as usual. In either case, the IsHelperLane() intrinisic can be used to identify these threads and perform special case logic in the presence of them.

Dynamic Resources (ResourceDescriptorHeap)

Spec Link

The first release of D3D12 supported bindless resources, but you had to do so with some somewhat clunky usage of unbounded resource arrays in HLSL combined with some root signature boilerplate. On top of that you also had to make sure you bound your “one big descriptor table” to the root signature on each command list every time you changed the root signature. Shader model 6.6 and dynamic resources makes all of that go away. Instead you just get a magic ResourceDescriptorHeap array that you can access from anywhere, and it gives you the resource that lives at the provided index of the bound descriptor heap. With this you can drop nearly everything from your root signature if you want, and instead just provide a few root constants or root CBVs/SRVs that the shader can use to grab whatever descriptor indices it needs. While not technically required for bindless, it makes things much much simpler and nicer. This is particularly true for StructuredBuffer, which in the old way required you to declare a separate unbounded resource array for every possible struct type that you used as an element.

To go along with ResourceDescriptorHeap, there’s also SamplerDescriptorHeap which works exactly the same except for the sampler descriptor heap. This makes bindless samplers just as simple if that’s what you need.

WaveSize

Spec Link

As I mentioned earlier, wave ops are really tough to use effectively in a way that works across all possible wave sizes. This is especially true since the wave size is not known at compile time, so any logic for separate wave sizes would have to expressed as conditionals on WaveGetLaneCount(). The [WaveSize(N)] intrinsic from Shader Model 6.6 aims to help here by allowing compute shader to specify that can only work with the specified wave size. This is enforced at runtime, where PSO creation will fail if the wave size is not supported by the current device and driver. While this helps, the situation is still tricky since you will want to compile the same shader N times for each possible wave size you want to support. You also likely want to have some sort of fall back that doesn’t use wave operations if you don’t support every possible wave size that D3D allows.

Derivatives In Compute

Spec Link

Traditionally quads were something that only existed in pixel shaders, which meant that by extension pixel shaders were the only stage that could use derivative intrinsics or texture sampling functions with implicit LOD mip level selection. This Shader Model 6.6 feature brings quads to compute shaders as well, by specifying some rules about how certain threads within the thread group end up getting mapped to quad indices. If you carefully respect these rules when writing your shader, you can then use the regular Texture2D.Sample function to get mip selection without having to pass gradients or calculate the mip level yourself. Note that this spec still does not fully specify how threads get mapped to waves within a thread group, so care must be taken not to make assumptions about waves are arranged in compute.

64-Bit Atomics and Atomic CAS For Floats

Spec Link

Prior to this feature, atomic operations could only be performed on 32-bit integers. Shader Model 6.6 changes this, and now allows you to perform atomics on 64-bit integers in buffers and textures. This is surely handy if you’re going to the nanite route of software rasterizing into a visibility buffer. In addition, SM 6.6 adds very limited support for atomics on fp32 values. In particular, it introduces InterlockedCompareStoreFloatBitwise and InterlockedCompareExchangeFloatBitwise. These essentially treat the float bits as a uint32_t, and perform a regular atomic CAS.

Raw Texture Gather

Spec Link

Raw Texture Gather is an interesting Shader Model 6.7 feature that allows you to perform a 4-texel gather on certain integer texture formats. If you have a R16_UINT, R32_UINT or R32G32_UINT format texture, you can gather a 2x2 quad of texels and have them returned to the shader as raw unfiltered bytes. This bypasses previous restrictions on using Gather with an integer texture format.

Writable MSAA Textures

Spec Link

As the same suggests, this Shader Model 6.7 features allows shaders to directly write to MSAA textures through the new RWTexture2DMS and RWTexture2DMSArray HLSL resource types. Previously you could only write to them through a render target view, or through copy/resolve operations.

SampleCmpLevel

Spec Link

SampleCmpLevel is like SampleCmpLevelZero, except that it lets you pass a specific mip level instead of only being able to sample the 0th mip level.

ByteAddressBuffer Types Load and Store

Doc Link

This is not a shader model feature, but just something that DXC now supports at the language level. Instead of only being able to load a uint/2/3/4 from a ByteAddressBuffer, there’s now a templated Load<T> method that allows you to load any arbitrary type or struct at an offset (with 4-byte alignment, or 2-byte alignment if 16-bit types are supported). There’s also a matching Store<T> function on RWByteAddressBuffer that works the same way, except for storing. To accompany these new methods, DXC also added support for sizeof(). This allows you to load from arrays of arbitrary types, similar to StructuredBuffer:

MyStruct s = byteAddressBuffer.Load<MyStruct>(index * sizeof(MyStruct));

SV_StartVertexLocation + SV_StartInstanceLocation

Spec Link

Many people are caught off guard when they find out that the BaseVertexLocation and StartInstanceLocation parameters of DrawIndexedInstanced do not affect the SV_VertexID and SV_InstanceID system value inputs for the vertex shader. Instead, they only affect fixed-function vertex fetch that occurs in the input assembler. This is especially confusing since GL and Vulkan have not worked this way, and have mechanisms to provide the vertex and instance offsets to the vertex shader. This new feature closes the gap by providing new SV_StartVertexLocation and SV_StartInstanceLocation system value inputs that contain the value of the BaseVertexLocation and StartInstanceLocation parameters. This can be particularly useful with GPU-driven rendering pipelines that make use of ExecuteIndirect, since it provides additional way to pass data along to the resulting draw.

Enums

Weirdly, HLSL never supported C-style enums until DXC added support for HLSL 2017. Now you can do the usual enum and enum class declarations, just like you would in C++.

Templates

Announcement Link

HLSL 2021 added limited support for template types and functions to HLSL. This is obviously a major change with huge implications on how the language is authored, both good and bad. However it is unquestionably powerful and flexible, allowing pure templated HLSL code to be used in many places where either macros or code duplication had to be used previously. For example you could make wrappers around bindless resource descriptors, or go nuts and make templated spherical harmonics math libraries. Just be sure to watch out for compile times and inscrutable compilation errors.

Operator Overloading

Announcement Link

Along with templates, HLSL 2021 allows for overloading a limited subset of operators for user-defined types. The usual arithmetic, bitwise, comparison, function, and array subscript operators all supported. The main exception are pointer-related operators like * and &, and operators that require returning a reference to this like +=.

Bitfields

Announcement Link

Bitfields are often useful for tightly packing lots of data into a limited number of bits, which is something we tend to do a lot on GPUs. With bitfield support, you no longer thave to write manual masking and bit shifting to stuff values into smaller than 16 bits.

C++-Style Function Overloading and For Loop Scoping

Announcement Link

One of the weird quirks of HLSL was that it had wacky overload resolution. Rather than overloading purely based on the number and type of the arguments like C++, it would essentially “destructure” custom types into their primitive components and overload based on that. This could be very surprising, to say the least. HLSL 2021 fixes this and uses the usual overload rules that you’re probably used to.

Another odd quirk that was resolved in HLSL 2021 was for loop scoping for anything declared in the for block. Pre-2021, anything inside of the for lived in the parent scope of the loop. This meant that if you had for (uint i = 0; i < length; ++i) { } the i variable would persist after the loop was finished. With HLSL 2021, i would only exist in the loop body instead which is how C++ and C99 handles it.

Developer Workflow Enhancements

Agility SDK + Independent Devices

As you’ve no doubt surmised from reading this far, D3D12 has changed an awful lot since it was first released. However in the early days this changes could be rather slow to roll out, since they had to be tied to major Windows 10 updates. That’s not great for an API that’s constantly evolving, and it’s really not great for developers who need to ensure a minimum available feature set across their potential user base. The Agility SDK aimed to fix both of these issues by decoupling major DX12 releases and additions from the OS release cycle, and instead bundling them into a D3D12Core.dll that could be shipped to end users. The way it works is a bit interesting: rather than an app only using the D3D12Core.dll that it shipped with, the D3D loader will also look at the installed OS version of D3D12Core.dll and use whichever is newer. That way apps can still get bug fixes and updates, but still ensure they have a floor for available features.

The developer story for the Agility SDK was originally a bit clunky. It was easy enough to get a D3D12Core.dll, the debug layer dll, and the corresponding d3d12.h header. You had to be careful with your include paths to make sure you’re including the newer d3d12.h rather than the one from the Windows SDK, but that’s not too hard to do. However the really annoying part was setting up the “magic” exports on your exe that let the D3D loader know that you want to use an Agility SDK. These required exporting a string containing a relative path to where D3D12Core.dll could be found, as well as an integer representing the intended Agility SDK version. This is a bit annoying for packaging as well as development builds…for various reasons you didn’t want D3D12Core.dll to live in the same folder as your executable, and so you had to place it in a known subfolder relative to the executable. However the real dealbreaker was for app plugins that wanted to use a D3D12 device. Since the “magic” exports have to be on the executable, there’s no way for a plugin to specify the Agility SDK that it wants. So it’s stuck with whatever the base app specifies…which is likely no Agility SDK at all. The Independent Devices feature fixes both of these problems by providing an API that can specify the Agility SDK details. Doing this required D3D12 to change its “one device per process” rule that it used previously, which is why the feature is called “Independent Devices”.

Debug Layer Callback

Spec Link

The debug layer is obviously super valuable and useful for catching programmer mistakes and undefined behavior before it happens. It also has a handy feature where you can instruct the layer to break into the debugger when an error occurs, which makes it more useful for catching the line of code that actually generated the error. However the error message itself was always output to the debugger output stream, which is displayed in Visual Studio but is otherwise very hard to interpret yourself. This could be especially frustrating if you come from Vulkan, where the validation layer allows you to install a callback that gets passed the details of the error right when it’s generated. D3D12 finally caught up with the release of Windows 11, which adds a new API that allows you to install a callback that gets called whenever the debug layer generates a message. You can then assert, print, log, etc. however you wish.

Device Removed Extended Data (DRED)

Spec Link

Overall the story for debugging and resolving GPU crashes is pretty terrible on PC. When it happens, you’re often on the other side of multiple abstractions and black boxes which makes it very very difficult to find the root cause. DRED aims to help a bit with this near-impossible task by providing some information about what the GPU was doing (or rather has already done) when it crashed on you. Basically it uses WriteBufferImmediate to automatically generate “breadcrumbs” for various commands recorded on the command list, such as PIX markers or draw calls. Then when the device is removed, you can use the DRED APIs to dump out the state of each command buffer with an indicator of which breadcrumbs were reached by the GPU. With any luck, this can point you to the particular pass or dispatch that caused the crash. In the case of a page fault DRED can also provide a list of resource debug names that correspond to the page fault address…although in practice I’ve found that this information is often missing for unknown reasons.

What’s Changed For Me Personally

All-In On Bindless and “User-Space Bindings”

Bindless might seem like a fun party trick at first, or something you only use in situations where you truly need to the shader to decide which resources it wants to use. But once you go all-in, it can have a transformative effect on your codebase and the way you work. Even in cases where you know up-front exactly which textures and buffers a shader will need, in my experience it’s an unquestionably nicer experience to do things through bindless rather than using old slot-based APIs or clunkily assembling descriptor tables. This is especially true with SM 6.6 type bindless with ResourceDescriptorHeap. Essentially every resource descriptor you can access just turns into a simple uint32_t that you can put wherever you want. I like to call it “user-space bindings” since that uint32_t is no longer opaque, and doesn’t need to live in a descriptor heap or a descriptor set or some other place where you don’t actually own the memory. You can put in structs, arrays, textures, anything you want.

On the host side of things, the potential for simplification can be enormous. Root signatures in particular can be sidelined almost completely, it’s completely viable to have just a handful or even one global root signature that gets used for everything. All you need to care about in your root signature is how you pass your initial “root” data to a shader, which might be some root constants, root CBVs, or root SRVs. Once you have that, all of your other resources can be accessed by pulling in descriptor indices. With a non-bindless setup you might find yourself doing things like reflecting shaders to generate an “optimal” root signature, or creating many root signatures to handle many different cases of ways that you would organize various resources into descriptor tables that are bucketed by the frequency by which the table bindings change. With bindless, that’s all gone! Root signatures can get stuffed into some corner of your codebase that deals with booting up the D3D12 device, and you may never have to think about it again. My “universal” root signature that I use in my home codebase looks like this, and I create it during startup:

D3D12_ROOT_PARAMETER1 rootParameters[NumUniversalRootSignatureParams] = {};

// Constant buffers
for(uint32 i = 0; i < NumUniversalRootSignatureConstantBuffers; ++i)
{
    rootParameters[URS_ConstantBuffers + i].ParameterType = D3D12_ROOT_PARAMETER_TYPE_CBV;
    rootParameters[URS_ConstantBuffers + i].ShaderVisibility = D3D12_SHADER_VISIBILITY_ALL;
    rootParameters[URS_ConstantBuffers + i].Descriptor.RegisterSpace = 0;
    rootParameters[URS_ConstantBuffers + i].Descriptor.ShaderRegister = i;
    rootParameters[URS_ConstantBuffers + i].Descriptor.Flags = D3D12_ROOT_DESCRIPTOR_FLAG_DATA_STATIC;
}

// AppSettings
rootParameters[URS_AppSettings].ParameterType = D3D12_ROOT_PARAMETER_TYPE_CBV;
rootParameters[URS_AppSettings].ShaderVisibility = D3D12_SHADER_VISIBILITY_ALL;
rootParameters[URS_AppSettings].Descriptor.RegisterSpace = 0;
rootParameters[URS_AppSettings].Descriptor.ShaderRegister = AppSettings::CBufferRegister;
rootParameters[URS_AppSettings].Descriptor.Flags = D3D12_ROOT_DESCRIPTOR_FLAG_DATA_STATIC;

// Static samplers
D3D12_STATIC_SAMPLER_DESC staticSamplers[uint64(SamplerState::NumValues)] = {};
for(uint32 i = 0; i < uint32(SamplerState::NumValues); ++i)
    staticSamplers[i] = GetStaticSamplerState(SamplerState(i), i, 0, D3D12_SHADER_VISIBILITY_ALL);

D3D12_ROOT_SIGNATURE_DESC1 rootSignatureDesc = {};
rootSignatureDesc.NumParameters = ArraySize_(rootParameters);
rootSignatureDesc.pParameters = rootParameters;
rootSignatureDesc.NumStaticSamplers = ArraySize_(staticSamplers);
rootSignatureDesc.pStaticSamplers = staticSamplers;
rootSignatureDesc.Flags = D3D12_ROOT_SIGNATURE_FLAG_CBV_SRV_UAV_HEAP_DIRECTLY_INDEXED;

DX12::CreateRootSignature(&UniversalRootSignature, rootSignatureDesc);

A “bindful” engine might have code that carefully manages the lifetime of descriptor tables, maybe even using hashes or similar to determine whether a table can be re-used or a new table has to be built. And perhaps also you have hooks for your texture streaming or CPU-updated buffers to know when to invalidate your descriptor sets, since the SRV descriptor had to change. Once again, all gone! Poof! You do still need to have a double-buffered descriptor heap and make sure you don’t update descriptors that are still being accessed by the GPU, but that’s straightforward and also something you can likely manage in one centralized place by having a queue of updates. In my home codebase, binding some resources and constants looks like this:

MeshConstants meshConstants =
{
    .World = world,
    .View = camera.ViewMatrix(),
    .WorldViewProjection = world * camera.ViewProjectionMatrix(),
    .CameraPos = camera.Position(),
    .Time = data.ElapsedTime,

    .VertexBuffer = model->VertexBuffer().SRV,
    .MeshletBuffer = model->MeshletBuffer().SRV,
    .MeshletBoundsBuffer = model->MeshletBoundsBuffer().SRV,
    .MeshletVerticesBuffer = model->MeshletVerticesBuffer().SRV,
    .MeshletTrianglesBuffer = model->MeshletTrianglesBuffer().SRV,
};

camera.GetFrustumPlanes(meshConstants.FrustumPlanes);

DX12::BindTempConstantBuffer(cmdList, meshConstants, URS_ConstantBuffers + 0, CmdListMode::Graphics);

It’s dead simple: I init a struct with a bunch of constants as well as SRV descriptor indices, which can live together side-by-side instead of having to be kept separate. Then in BindTempConstantBuffer I grab some temporary frame-lifetime CPU-writable buffer memory, copy in the struct contents, and bind it to the root signature as a root CBV in one go. I could also trivially have nested structures with more descriptor indices if I want, or have this live in a persistent buffer instead of creating one on-the-fly. Either way it’s all something I can control and handle however I want, which is incredibly freeing.

On the shader side it’s perhaps less-transformative, but bindless is still cool. One aspect that can cause some friction at first is that you don’t just have the resources available as globals in HLSL. Instead you explicitly grab the resources using ResourceDescriptorHeap and then do what you want. In the long term though I think this is a very positive change vs. requiring resources to be globals, and can make the code a whole lot cleaner and composable once you lean into it. If you really want you can also load your HLSL resources into static global variables to mimic the old style, but I don’t particularly recommend that unless you’re adapting a large existing codebase. And of course once you have bindless, you can also do very fancy things in your shaders like traversing your own data structures or sampling arbitrary material textures based on a material ID.

For ray tracing in particular bindless is a lifesaver, and a massive simplification over juggling local root signatures and shader binding tables. In fact I wrote a whole chapter about that in Ray Tracing Gems II!

The main downside with “bindless everywhere” tends to be in tooling and introspection. When bindings are known statically at draw/dispatch time, it’s trivial for tools like RenderDoc and PIX to intercept those bindings and display them for you later when you’re viewing a capture. On a similar note, validation layers also have a much easier time with old-style bindings since they need to know which resources a particular operation accessed. With bindless there’s no way to know which resources actually got accessed without running the shader, which is basically what these tools now have to do if you want accurate information. They will patch your shader bytecode to insert new instructions that can store out introspection data to a buffer, and then gather that data after running your draw or dispatch to know which resources got accessed. This does actually work, but adds significant complexity on the tooling side that didn’t exist previously. On a similar note shader reflection APIs can also be rendered nearly useless for determining which resources a shader uses, so you may need a new approach if you’re relying on that information.

HLSL Templates

Templates always inspire mixed feelings in me. They are undeniably useful and powerful, but come with an endless list of ways they can blow up in your face. Whether it’s bloated compile times, indecipherable error messages, or insane SFINAE gymnastics that cause programmers to consider taking up farming: the potential downsides are numerous and well-documented. But again, undeniably useful. Having carefully introduced templates into my own shader code, I am of the opinion that adding them was the right call and we’re better off having them than not. But I try to be at least as cautious with them as I am with my C++ code, in that I generally reach for simpler lower-tech solutions whenever I can even if it adds a bit of copy/pasted code.

So what have I used them for? The list is pretty long at this point, but here’s some highlights:

• They’re immediately useful for functions that you want to be able to use with both fp32 and fp16 data types. No need to duplicate the function, just write it once. You can also make functions vector-size agnostic, but that’s a little clunkier. For an example of both of these things you can have a look at my SHforHLSL library that implements types and functions for working with low-order spherical harmonics. In fact it’s probably also a good case for trying to figure out what’s “going too far” with templates. In particular I think templating on the monochrome vs RGB variants hurts the readability of the code in some places. For a library where you don’t really touch the code this might be fine, but in more actively-developed code I would probably be annoyed with it. If you compare against the non-templated Lite version, the non-templated code can end up looking a good bit cleaner.
• Similarly, templating on the texture type can be nice for implementing things like filtering functions where the code could easily handle float/float2/float3/float4 types.
• For some code that I’ve worked on, I ended up making an algorithm that used a templated “writer” type. This let me use the same code and have it write its outputs to a buffer, or alternatively have it log its results using a shader printf.
• Speaking of shader printf…my own implementation uses templates for handling the format string as well as packing the arguments into the debug buffer.
• I never actually used this, but I did sketch out a proof-of-concept for a buffer wrapper that could emulate Buffer Device Address functionality that you have in Vulkan. It’s pretty clunky compared to actual language support for such a thing, but it’s nice that you can do it.

Just as with C++ I think templates are potentially a liability, and it’s very easy to over-use them. But at the same time I still think they have be a large net positive if used judiciously. There’s definitely room for very reasonable arguments that HLSL should have chosen a different direction here instead of following C++…Slang’s generics are probably the most usable counter-example at the moment. Personally I’m generally a pragmatist and think there’s a large value in using existing paradigms that are well-understood, and I’m not sure if Slang’s divergence is a win in the long run. But time will tell, and I’ll definitely pay attention to what happens in that space!

More Code Sharing Between HLSL and C++

A lot of engines invest considerable amounts of engineering and code into trying to automatically manage the boundaries between C++ and shader code. These tend to be some combination of shader reflection, code generation, and/or runtime layers that make it possible for engine code to provide the data and bindings that the shader needs. This can be a rather useful investment: it allows engines to hide specific details of how bindings work, or even transparently account for wacky HLSL constant buffer alignment rules. But this can be a double-edged sword: it can result in esoteric peculiarities specific to an engine, and can require considerable maintenance to keep such layers working through new features and/or APIs.

Lately I find myself drawn to the much simpler option of simply sharing structs, functions, and other code between my HLSL and C++ where it makes sense. This is helped along a bit by the latest HLSL language features, where it’s now fine to use enums, templates, and methods as much as you’d like. But in particular, using bindless for everything makes it even more viable. Like I mentioned earlier, with bindless you can effectively represent any resource descriptor as 32-bit integer or handle that you store freely in structs. This means you can trivially store all of a shader’s various constants and resources in one more structs that you share between C++ and HLSL, and both C++ and HLSL will also stay up-to-date without the need for codegen or reflection. One major wrinkle here is constant buffers, which unfortunately after 10 years of D3D12 still have the annoying legacy packing rules for vectors and arrays. For such cases you would have to manually account for alignment issues, or ignore constant buffers altogether. Fortunately the latter approach is increasingly viable on modern hardware.

In reality this “just share” approach is probably going to be more viable for simpler scenarios. Larger engines will likely have more complex binding mechanisms to deal with, and may have to deal with weird platforms where sharing is less viable. So it’s certainly not a panacea, but I do think it’s increasingly becoming a worthwhile option.

Designated Initializers

This is not strictly related to graphics or APIs, but I do rather like designated initializers in C++20. They’re nicer than having to set struct members one at a time, and they have less compiler + non-optimized runtime overhead than “builder” patterns that use method chaining. I generally like to pair them with structs that have default values specified through member initializers, and then the designated initializers can “override” the default value. With this pattern you can even emulate named arguments, which can be much easier to read than a soup of parameters. For example:

struct RawBufferInit
{
    uint64_t NumElements = 0;
    bool32 CreateUAV = false;
    bool32 Dynamic = false;
    bool32 CPUAccessible = false;
    const void* InitData = nullptr;
    ID3D12Heap* Heap = nullptr;
    uint64_t HeapOffset = 0;
    const char* Name = nullptr;
};

spotLightClusterBuffer.Initialize({
    .NumElements = numXYZTiles * AppSettings::SpotLightElementsPerCluster,
    .CreateUAV = true,
    .Name = "Spot Light Cluster Buffer",
});

In my experience this is great for resource initialization, and also for barriers. I like to have helpers that set you up for common barrier situations, and then use designated initializers to override the specifics:

struct DepthWritableBarrierDesc
{
    bool FirstAccess = false;
    bool Discard = false;
    D3D12_BARRIER_SYNC SyncBefore = D3D12_BARRIER_SYNC_ALL_SHADING;
    D3D12_BARRIER_ACCESS AccessBefore = D3D12_BARRIER_ACCESS_SHADER_RESOURCE;
    D3D12_BARRIER_LAYOUT LayoutBefore = D3D12_BARRIER_LAYOUT_DIRECT_QUEUE_SHADER_RESOURCE;
    uint32_t StartArraySlice = 0;
    uint32_t NumArraySlices = uint32_t(-1);
};

DX12::Barrier(cmdList, depthBuffer.DepthWritableBarrier({ .FirstAccess = true }));

This is where I now get on my soapbox to rant about the specific implementation of designated initializers in C++, which requires that the order of your initializer match the order in which the members are declared. It’s setup this way to avoid nasty footguns where the initialization/construction of one member depends on another member…but it’s deeply unfortunate that this particular edge case hurts the usability of the feature for the common 99% of cases where you don’t really care about the init order. Clang does at least have a way to demote this into a warning, but MSVC does not as far as I’m aware.

Printf and Other Shader Debug Features

Once again this is not something that requires the latest D3D and HLSL toys to pull off, but if you are on the latest and greatest it becomes quite a bit easier to do. In particular:

• SM 6.6-style bindless makes it possible to make writable debug buffers globally available to any shader without any modifications for app code. This means if you have shader hot reloading, you can just add some debug printing code to your shader without restarting or recompiling the app. In my own code I achieve this by reserving a particular known index in the descriptor heap for the debug buffer, and hard-code that index into the .hlsli file that shaders include for debug functionality.
• HLSL templates and RWByteAddressBuffer typed stores make it much more convenient to stuff heterogenous data into your debug buffer (useful for things like printf arguments)
• While not an official feature, it’s possible to (ab-)use DXC’s template functionality to work with actual string literals for printf-like functionality, which is much simpler and more natural than the alternatives.

Having a reliable working shader printf can be a game changer, and can almost make you forget how awful the debugging situation is on PC. What I like to do is have some utility functions that allow you to filter the debug functionality down to a particular thread, potentially by filtering output to the thread/pixel under the mouse cursor. Then I will use helper macros that print out the state of local variables, which serves as a sort-of live debugger for a thread. Once you have this setup you can make changes to shader, and continue inspecting the values of the intermediate variables to see the end result of your changes.

Another useful trick that I now get a lot of mileage out of is drawing debug wireframe shapes directly from a shader. The process for this starts out rather similar to shader printf: you have some globally-accessible debug buffer, and your shader functions dump the shape transform + properties into that buffer. Then towards the end of the frame, you perform an indirect draw for each shape to consume the data from the debug buffer and draw the shape to the screen. It takes a bit of plumbing to get it all setup, but once it works it can be incredibly useful for debugging. It can also be a rather convenient way of implementing dev-only debug visualizations that you otherwise would have had to do on the CPU, or with bespoke shaders. For example, if you’re performing culling in your a compute shader you can draw colored bounding spheres to indicate the result of the culling operation.

Here’s an example of using both printf and debug drawing together for debugging a ray generation shader, so you can see what I mean:

ShaderDebug::FilterIfCursorOnPos(pixelCoord);
DebugPrint_("RayGen Pixel XY: {0}", pixelCoord);
DebugPrintVar_(ray.Origin);
DebugPrintVar_(ray.Direction);
DebugPrintVar_(payload.HitT);

float3 arrowColor = payload.Radiance;
arrowColor/= max(max(max(arrowColor.x, arrowColor.y), arrowColor.z), 0.0001f);
float3 arrowEnd = ray.Origin + ray.Direction * payload.HitT;
ShaderDebug::DrawArrow(ray.Origin, arrowEnd, float4(arrowColor, 1.0f));

Conclusion

Thanks for reading this far! Hopefully you got an idea of the scope of things that have changed in D3D12 over the past 10 years, and also found it interesting to read about my own experiences. See you next time!

One of the things we often take for granted on GPUs is the idea of early Z testing. It’s the main reason why Z prepasses exist at all, and it’s one of the things that has allowed forward rendering to remain viable without being completely overwhelmed by pixel shader overdraw (instead it merely gets overwhelmed by quad overshading, but I digress). Despite its ubiquity and decades of use, I find that early Z can still be confusing and is often misunderstood. This is for a few reasons, but personally I think it’s because it’s usually deployed as a “magic” optimization that’s not really supposed to be noticeable or observable by the programmer (aside from performance, of course). That lack of explicit-ness tends to make it unclear as to whether early Z is actually working unless you look at the performance and/or pixel shader execution statistics. In light of that, I’m going to use this post to give an overview of what “Early-Z” is, how it works, and when it doesn’t work. At the end, I’ll finish with a handy table to summarize the useful information.

While explaining things, I’ll include insights and screenshots from my DX12 Early-Z Tester that you can find on GitHub. This testing app can create many of the scenarios that we’ll discuss below, while also displaying the number of pixel shader invocations as a way to determine whether or not Early-Z culling occurred.

I would also like to make it clear up-front that this article is primarily focused on desktop-class graphics hardware, and not mobile. While much of what’s here likely applies to mobile GPUs, I am not familiar enough with them to confidently explain how they may or not diverge from their desktop cousins.

Depth In The Logical Rendering Pipeline

Graphics APIs such as D3D, Vulkan, and GL are specified in terms of what’s called a logical pipeline for rendering. This pipeline is intentionally abstract and not strictly tied to a specific hardware implementation, and instead dictates the behavior that the programmer can expect to see when the draw a triangle. This necessarily gives IHVs and drivers a lot of latitude to structure their hardware in different ways, as well as to change their implemention over generations of hardware. For example, the D3D logical pipeline looks like this:

This pipeline hasn’t fundamentally changed for D3D12 (aside from the addition of mesh shaders) and is very similar in GL/Vulkan, albeit with different terminology. In all cases the actual depth buffer operations are specified to happen at the very end of this logical pipeline after the pixel/fragment shader has run. D3D calls this stage the Output-Merger or OM for short, and it’s the logical stage where writes/blending to render target occurs and also where depth/stencil operations logically occur. While it might seem weird that depth operations are the end if you’re looking at things from a modern viewpoint, if you apply a historical context it really makes sense:

• The main purpose of depth/stencil operations is to control whether anything gets written to the render targets, so it makes sense that it would be in the same logical stage where write/blend operations occur
• The pixel shader can do certain operations, such as discard/alpha test, that also affect depth operations
• The earliest GPUs with hardware depth buffers performed depth operations at the end, since it was originally a visibility algorithm and not so much an optimization

The important bit here is that this “logical” structure fits what you see in your results, regardless of how the GPU actually produced them. The GPU might have had an explicit binning phase, or may have combined your VS + HS + DS all in to one hardware shader stage, but either way you see that depth testing has produced the render target and depth buffer values that you would expect.

Looking through the lens of the logical pipeline, it should also be apparent why writes to UAVs/storage buffers/storage textures from a pixel/fragment shader are unaffected by the depth test by default. After all, depth operations come after the pixel shader and so there’s no reason to expect otherwise. In practice things are of course more complex when it comes to UAVs, but more on that later.

Where Does Early-Z Fit In?

Ok, so obviously we know that the logical pipeline does not tell the full story here since you’re reading an article about Early-Z. So how does it work, if the pipeline says depth operations happen at the very end? The answer is that drivers have enough wiggle room to be “sneaky” and silently move depth operations up before the pixel shader actually runs so that it can cull the pixel shader entirely:

Basically the driver will look at the current pixel shader and render states, and make a call as to whether it would be safe to cull pixel shader executions and still produce the same observable results that the logical pipeline dictates. If we’re talking about bog-standard opaque rendering that only writes to a render target this actually really simple: culling the pixel shader is fine since the only observable output is the render target writes, and so there’s no visible difference between culling the render target write and culling the entire pixel shader before it even runs. This works even if depth writes are enabled: the early Z operations can test the existing Z buffer value first and commit the Z write if the test passes, while also allowing the pixel shader to run. Later on if some other pixel/fragment ends up having a depth that also passes the depth test due to being closer, that’s fine since the render target results will just get overwritten. In that regard the order of the drawn triangles doesn’t matter, you get the same result either way in the end. There’s of course a performance difference from the ordering: with depth writes enabled you ideally want to order triangles from closer to further so that you have fewer “wasted” pixel shader threads that ultimately don’t contribute. Or alternatively you can use the popular approach of a depth-only prepass, which effectively guarantees that the depth buffer is pre-seeded with the closest depth and thus the pixel shader will (mostly) only run for those visible pixels. Either way, it’s pretty clever that hardware and drivers can do this sort of thing to improve performance while still conforming to the logical pipeline!

Let’s try a simple test case in our test app to verify this. For this test case and others shared here, I’ll be running them on my AMD RX 7900 XT. First, let’s draw two triangles in back-to-front order with the red triangle being further from the camera:

You can see that in this case we end up with 648000 pixel shader invocations for these triangles. Let’s now reverse the draw order so that we draw with the more optimal front-to-back ordering:

This time we only shaded 440640 pixels, which is substantially fewer! This is Early-Z in action, working the way we would expect.

One thing to keep in mind is that even for cases where Early-Z is considered “safe” to do, it’s still ultimately up to the driver and the hardware to make that call unless you force its hand (more on that later). The driver may decide not to based on heuristics, since the hardware might have to take extra steps to be able to perform Early-Z culling. The granularity at which pixels are culled as part of Early-Z can also vary depending on the hardware in question. The earliest PC hardware to use Early-Z did so with “hierarchical depth buffers” (Hi-Z for short), where the hardware would maintain a separate buffer containing the min and max depth values for an NxN region of the depth buffer. Such a structure is useful, since it allows for fast and coarse z cull without requiring the bandwidth of reading the full per-pixel depth buffer values. During the D3D10 era both ATI/AMD and Nvidia gained the ability to perform fine-grained early Z tests as well, but hierarchical depth buffers are still used to coarsely cull before performing the more expensive fine-grained tests.

When Does Early-Z Have To Be Disabled?

In the standard opaque rendering case we mentioned earlier, we discussed how it was “safe” for the driver to cull the pixel shader if the Early-Z test failed. But clearly there are cases where the driver can’t do this since it would violate the rules of the logical pipeline. Let’s now go through the most common cases.

Discard/Alpha Test

This is probably the most common one to encounter, and probably also mentioned the most in the context of “things that break Early-Z”. As a quick recap, if the pixel shader executes a discard operation (or uses it indirectly through the clip() intrinsic) then all depth, stencil, and render target writes skipped for that pixel. This clearly has implications for Early-Z if depth writes are enabled, since the depth write can’t actually be committed until the pixel shader runs to completion. At first thought, it may seem like the driver has no choice but to fall back to “Late-Z” in the case of pixel shader discard + depth writes. While this is certainly a possibility, many GPUs are still able to perform some degree of early depth testing to reject and cull pixel shader executions even if it has to defer the depth write until afterwards. The reason GPUs can do this is because discard doesn’t change the z/depth of the pixel that was produced by interpolating the triangle’s vertex positions: it merely makes the write of that depth conditional. Therefore it is still “safe” to cull a pixel shader thread with discard if the rasterized depth fails the depth test based on the current depth buffer contents. You can see this called out in this older slide from GDC 2013, which shows how the coarse hierarchical depth tests still occur while deferring depth writes until after the pixel shader runs:

While “splitting” the depth test in this way can allow Early-Z culling to still occur for discard scenarios, the fact that depth is written later in the pipeline can potentially have adverse effects on our Early-Z culling rate. To see this in action, let’s revisit the front-to-back test case we looked at earlier. Except this time, in the pixel shader we’ll have a discard that never actually gets executed:

With this scenario we get about 582000 PS invocations, with the final value fluctuating slightly from frame-to-frame. This cull rate is not nearly as great was what we achieved in the no-discard case, since a lot of pixels from the further triangle are going to get accepted by the Early-Z test before the Z buffer has been updated by the closer triangle. We can try to verify this hypothesis by inserting a global barrier between the draws to force the first draw to finish before starting the second:

With the barrier (which to be clear, you wouldn’t want to do in practice!) we now get an Early-Z culling rate that’s much closer to what we got without the discard. We can also confirm with the testing app that clearing the depth buffer to the closest possible value results in no pixel shaader invocations. This suggests that it can be good practice to try to render all of your non-alpha-test opaques first, or at least ones that have a good chance of occluding things. Only performing discard in a depth prepass followed by a full opaque pass with EQUAL depth testing can also be effective for ensuring that the heavier opaque pixel shader doesn’t execute for occluded pixels.

Meanwhile, if depth writes are not enabled then Early-Z is totally compatible with shader discard. For that case the depth buffer doesn’t need to be updated, so it’s completely safe to perform the depth test early and cull the pixel shader before it runs. This means that discard can be used freely as an optimization in particles and other transparent draws without worrying about affecting Early-Z.

One very important thing to note is that the implications to Early-Z apply as long as your compiled shader instruction stream contains any discard at all! This is still true even if no pixel ever ends up being discarded, since the driver must make decisions about Early-Z up-front before actually running your shader. Therefore a runtime branch on a constant or uniform to avoid a discard can still have Early-Z implications even if that branch is never taken.

It’s also worth noting that exporting coverage from the pixel shader via SV_Coverage/gl_SampleMask has exactly the same implications as performing a discard, so the same rules apply in terms of Early-Z.

Pixel Shader Depth Export

Typically the depth value used for depth testing is generated by interpolating Z from the triangle vertices. However pixel/fragment shaders also have the ability to fully “override” this depth value if they choose to. In HLSL this is done with the SV_Depth attribute, whereas in GLSL you make use of the gl_FragDepth global. Depth export can be a useful tool, but it naturally creates issues for Early-Z. Short of being able to see into the future, there’s simply no way for the hardware to know what the depth value is going to be without executing the pixel shader. Thus the hardware has no choice but to fall back to Late-Z for any shaders that manually export a depth value. Notably the situation is the same regardless of whether or not depth writes are enabled: in both cases Late-Z is the only safe option that respects the logical pipeline. We can confirm this behavior with our testing app:

Conservative Depth Export

D3D11 later added variants of depth export that are collectively known as conservative depth export. These variants allow you to add a constraint on the depth value output by the pixel shader in the form of an inequality. For example, SV_DepthGreaterEqual allows you to generate a depth value with the constraint that it must be greater than or equal to the “natural” pixel depth that was calculated by interpolating the triangle vertex positions. The reason these exist is that this constraint can potentially allow the hardware to safely enable a degree of Early-Z functionality despite the final depth not being known until the pixel shader executes. For that to occur, the inequality has to “oppose” the inequality used as part of the depth render state. Returning to our SV_DepthGreaterEqual example: let’s say we have depth writes disabled but depth testing enabled with a LESS_EQUAL depth function. In this scenario the driver and hardware can determine that the pixel shader can only increase the natural depth from interpolation, which means that if the natural depth fails the depth test the pixel will always fail no matter the results of the shader. Therefore the pixel shader can be safely culled with no visible effect on the outputs. This is very similar to the situation for shader discard, where Early-Z can also be enabled for this case since the shader can’t invalidate the results of the Early-Z test.

Just as with discard, things get more complicated when depth writes are combined with conservative depth export. In this case depth writes can’t be committed before the pixel shader runs, since the final depth value isn’t known yet. Therefore it must be postponed until the Late-Z stage. Potentially the hardware can still perform an Early-Z cull as it does for the discard case, assuming it’s capable of such a thing. On such hardware conservative depth export retains an advantage over arbitrary depth export even when depth writes are enabled, with the same caveats about reduced Early-Z culling rates that apply for discard. We can confirm this in our testing app:

Here we very early end up with the full 648000 shader invocations, which is actually even worse than what we encountered with discard. Forcing a barrier once again gives us an almost-perfect culling rate:

UAVs/Storage Textures/Storage Buffers

Buckle up everyone, because this is where things get even more complicated. As I alluded to earlier, pixel shaders that make use of arbitrary writes to textures or buffers will also cause the hardware to switch to Late-Z. The underlying assumption made by drivers that enable Early-Z is that the pixel shader is a “pure” function with well-defined outputs and no side effects. Writes to UAVs/storage buffers/storage textures violate these assumptions, since by nature they are side effects of running the shader. Faced with such a shader, drivers have no choice but to fall back to using Late-Z so that depth testing does not affect the values written to the UAVs.

…unless Early-Z is forced on, which is a feature that was added to APIs alongside shader-writable resources. So let’s talk about that next.

Forcing Early-Z

While the “UAVs ignore depth testing” behavior makes sense through the lens of the logical pipeline, it’s clearly not a useful behavior for a lot of things you can do with arbitrary pixel shader writes. As an example straight from 2009, consider an Order Independent Transparency algorithm that appends transparent fragments to a linked list so they can be sorted later. Appending transparent fragments that are occluded by opaque surfaces makes no sense, and manually performing depth testing in the shader doesn’t save you the full cost of executing the pixel shader the way that Early-Z cull does. The Early-Z hardware is right there, we should just be able to use it!

D3D11 addressed this need by letting you tag your pixel shader entry points with a then-new [earlydepthstencil] attribute. While a bit of a mouthful, this attribute does exactly what it says it does: it forces all depth and stencil operations to happen before the pixel shader executes, and requires that the pixel shader be culled entirely if the test fails. This gives us exactly what we want for the OIT case described earlier: we get both the results and the optimizations we want from the depth test, and we can still do fancy UAV stuff. In fact you don’t even necessarilly need to use this on pixel shaders with UAVs, it can be applied to any pixel shader where you want to force the behavior.

Now this is clearly great and useful, but before slapping that attribute on every shader one has to consider what it means to fully move all depth reads and writes to before the pixel shader. For example, any kind of depth export is completely incompatible with this attribute¹ and those shader depth outputs are completely ignored. More subtly, it also breaks discard behavior if depth writes are enabled: the discard will still prevent render target writes, but the depth buffer write will have already happened before the discard occurs and thus the new depth will still be written. Here’s what it looks like in practice:

Notice how the leaves in the right image are still applying alpha testing, but they’re occluding each other as if there were no alpha testing. This could be quite unexpected if you didn’t think it all the way through, and is probably not ever what you want!

Here’s a simpler example from the testing app that shows what’s happening even more clearly:

You’re generally fine if you have depth writes enabled with [earlydepthstencil] as long as you’re not using discard, depth exports, UAVs, or anything else that we already mentioned as causing issues for automatically enabling Early-Z. In such cases performing the depth test + write early is fine since there’s no possibility for the pixel shader to modify the depth in any way, and render target writes already have implicit ordering guarantees.

Forced Early-Z With UAVs And Depth Writes

Ok so let’s say we really wanna get crazy and use UAVs together with depth writes and forced Early-Z. What happens in this case? The short answer is that you can still get a race condition between overlapping pixels, just like any other usage of UAVs from the pixel shader. While the Early-Z depth test + update can cull an already-occluded pixel before its shader has a chance to run, this will only happen if the triangle containing the culled pixel was submitted after the triangle that occludes it. If the triangles are not submitted in this order (or they intersect), then the closer and further pixels can run concurrently and will end up racing each other.

Here’s two examples of that this ends up looking like:

This is a bit of a bummer, since it means you’re essentially stuck with regular fixed-function render target writes if you need to also enable hardware depth writes². This may force you to require a depth prepass, and it’s not ideal to have such a requirement for correctness of our results. However there is one more option that we have yet to explore….

Rasterizer Order Views/Fragment Shader Interlock

Before we wrap up, I wanted to touch on how ROVs/FSI factors into the Early-Z + depth write situation. ROVs guarantee two things:

1. That pixel shaders mapping to the same pixel coordinate will not have overlapping access to the ROV resource, and can safely perform read/modify/write operations without requiring atomic operations
2. That accesses to ROV resources happen in API submission order, similar to how render target writes/blending works

If you’re not using depth writes, then the results you get from ROVs + [earlydepthstencil] are straightforward: you’ll only get writes from pixels that pass the depth test, and those writes will happen in submission order. This is very handy for OIT and other similar techniques. But what about if depth writes are enabled? In this case it seems like we should again get results that match old-school render target writes/blends, where the last visible write is from the closest pixel that ultimately passes the Z test. Getting these expected z-order results assumes that Early-Z tests happen in order of API submission, such that you get one of two results when two pixels overlap:

• The “closer” pixel performs Z operations first, and so the “further” pixel shader thread is culled before it even runs
• The “further” pixel performs Z operations first, but the “top” pixel is guaranteed for its write to happen second (and thus be visible) because of ROV rules

If we think about it, this really the same situation that we have if we just use regular render target writes instead of ROVs: the writes happen in submission order, and so the Z operations must also happen in submission order to get the correct results.

From some simple tests I’ve performed on my AMD RX 7900 XT, it does seem that this works! But like anything mentioned in this article, you should try to verify yourself on the hardware that you care on. I’ve tested successfully in more complex scenes, and it also works as expected in the testing app:

Regardless, this approach would still be incompatible with discard or depth export just like any other pixel shader that forces Early-Z. And of course there’s the inherent performance overhead of ROVs, which varies depending on the hardware, the amount of parallelism, the amount of overdraw, and the size of the critical section in the shader.

Summary and Conclusion

To sum things up, here’s a table listing the expected implicit behavior that a driver will use for Early-Z if you don’t force it:

And now let’s also sum up the expected behavior when forcing Early-Z using [earlydepthstencil]:

Hopefully these tables and the article itself gave you some intuition for understanding when the driver is sneaking in Early-Z automatically, while also giving you an understanding of what exactly happens when you force Early-Z on for your pixel shader. Thanks for reading!

Unless you’re fortunate enough to to be working exclusively in Cuda, debugging GPU shaders is still very much “not great” in the year 2024. Tools like RenderDoc and PIX are amazing and do provide the ability to step through a shader and inspect variables, but they’re fundamentally tied to a “capture” workflow. This means that you need to run your game or app from RenderDoc/PIX in capture mode (which adds overhead), and then capture at least a single frame. Then after the capture is finished you can analyze it, find your draw or dispatch, and finally select which thread you’d like to debug. Once you’re debugging things generally work, but it’s still possible you’ll encounter issues due to the way that debugging works in these tools. They are are not simply inspecting the state of an executing (suspended) thread like you would expect from a CPU debugger, instead they work by either emulating shader instructions on the CPU or by patching the shader bytecode to emit intermediate program state to a buffer that can be read by the CPU. Don’t get me wrong, these tools are great to have and it’s fantastic that they can be used to debug shaders at all. But it’s still not enough, and generally more debugging tools are needed for unusually tough problems. Even if vendor-agnostic GPU debuggers were as good as they are on CPU, you would still probably want to reach for some other tools depending on the situation.

In CPU land, the venerable printf and all of its related functions are commonly used for debugging and diagnostics. However we have not yet had a cross-vendor/cross-API way to do the same for shaders. To be fair it’s a lot more complicated on a GPU! Shaders run in batches of thousands or even millions of threads, and run on a completely separate processor than what the OS and all of its facilities run on. Despite those issues, Vulkan/SPIR-V does actually provide a printf that’s available from both GLSL and HLSL¹. It comes with caveats though. In particular it’s setup so that the messages can be intercepted by a validation layer or by RenderDoc, which makes it more difficult for the engine/app itself to obtain the messages and process them. And of course this doesn’t help if you’re target D3D12 or other APIs and you’d like your prints to work on those platforms as well.

In this article I’ll walk through how to build your own shader printf completely in software, using HLSL and D3D12 as the target language and API. The concepts here can be adapted to any API and shader language, although the shading language and compile can (significantly) impact how you handle strings in your shaders,

Overall Approach

The printf implementation I’m going to describe is going to work like this:

• Every frame, have the GPU clear a big buffer that we’ll use to store the print strings and arguments
• Make a debug info buffer SRV available at a “magic” known descriptor index, which effectively makes it globally available to all shaders through a #include
  • This buffer will have a descriptor index for the print buffer, along with some additional info useful for debugging
• When a shader wants to print, it will use an atomic to allocate some space in the big print buffer and stuff the string + data into it
• After recording every frame, copy the print buffer to a CPU-accessible readback buffer so that the data can be read back. Then every print can be decoded from the buffer and logged somewhere.

Pretty simple in concept! But as always the devil is in the details. Lets go though them one by one.

Setting Up The Print Buffer

For our print buffer we don’t need anything fancy at all, just a “big enough” buffer that’s writable by the GPU. In my sample framework I do it like this, and also create a pair of matching readback buffers:

PrintBuffer.Initialize({
    .NumElements = 1024 * 1024  * 4,
    .CreateUAV = true,
    .Name = L"Shader Debug Print Buffer",
});

for(ReadbackBuffer& buffer : PrintReadbackBuffers)
    buffer.Initialize(PrintBuffer.InternalBuffer.Size);

PrintBuffer in this case is a RawBuffer which ends up being a RWByteAddressBuffer in the shader. Every frame it gets cleared to all 0’s, which I do with a utility function that just uses a compute shader to do the clear (since ClearUnorderedAccessViewUINT is painful to use in D3D12):

DX12::ClearRawBuffer(cmdList, PrintBuffer, Uint4(0, 0, 0, 0));

DX12::Barrier(cmdList, PrintBuffer.InternalBuffer.WriteToWriteBarrier());

void ClearRawBuffer(ID3D12GraphicsCommandList* cmdList, const RawBuffer& buffer, const Uint4& clearValue)
{
    cmdList->SetComputeRootSignature(UniversalRootSignature);
    cmdList->SetPipelineState(clearRawBufferPSO);

    Assert_(buffer.UAV != uint32(-1));

    ClearRawBufferConstants cbData =
    {
        .ClearValue = clearValue,
        .DescriptorIdx = buffer.UAV,
        .Num16ByteElements = uint32(AlignTo(buffer.NumElements * buffer.Stride, 16) / 16),
    };
    BindTempConstantBuffer(cmdList, cbData, URS_ConstantBuffers + 0, CmdListMode::Compute);

    uint32 dispatchX = DispatchSize(cbData.Num16ByteElements, clearRawBufferTGSize);
    cmdList->Dispatch(dispatchX, 1, 1);
}

That’s it for the print buffer setup! Next we’ll look at our “magic” debug info buffer.

The “Magic” Debug Info Buffer

In my sample framework I use shader hot-reloading quite frequently, since it provides a super-quick iteration loop that doesn’t require re-starting the app. Often one of the reasons for hot-reloading a shader is to add some temporary debugging code in order to figure out why something isn’t working correctly. To that end, I wanted to have the ability to add debug prints to any shader without having to change the bindings and re-compile the C++ code. This could potentially be done by either always adding an extra descriptor index to binding structs or by ensuring it was always bound to the root signature, but instead I opted to lean on Shader Model 6.6 bindless by just placing a buffer SRV descriptor at a “known” static index shared between C++ and GPU code. That makes the buffer available to anyone regardless of what root signature they use or what else is going on in the shader, which is nice. On the C++ side it’s simple: I just added a way to allocate a specific descriptor index from my DescriptorHeap type, create an SRV in that descriptor slot, and free the original SRV descriptor:

DebugInfoBuffer.Initialize({
    .NumElements = sizeof(DebugInfo) / 4,
    .Dynamic = true,
    .CPUAccessible = true,
    .Name = L"Debug Info Buffer",
});

const PersistentDescriptorAlloc alloc = DX12::SRVDescriptorHeap.AllocatePersistent(MagicDebugBufferIndex);
DX12::SRVDescriptorHeap.FreePersistent(DebugInfoBuffer.SRV);
DebugInfoBuffer.SRV = alloc.Index;

for(uint32 i = 0; i < ArraySize_(alloc.Handles); ++i)
{
    const D3D12_SHADER_RESOURCE_VIEW_DESC srvDesc = DebugInfoBuffer.SRVDesc(i);
    DX12::Device->CreateShaderResourceView(DebugInfoBuffer.Resource(), &srvDesc, alloc.Handles[i]);
}

Then every CPU frame we just need to fill that buffer with fresh data:

DebugInfo debugInfo =
{
    .PrintBuffer = PrintBuffer.SRV,
    .PrintBufferSize = uint32(PrintBuffer.InternalBuffer.Size),
    .CursorXY = { cursorX, cursorY },
};
DebugInfoBuffer.MapAndSetData(&debugInfo, sizeof(debugInfo) / 4);

In the shader it’s no fuss at all to get the buffer through ResourceDescriptorHeap:

// In a header file shared between shaders and C++:
struct DebugInfo
{
    DescriptorIndex PrintBuffer;
    ShaderUint PrintBufferSize;
    ShaderUint2 CursorXY;
};

SharedConstant_ DescriptorIndex MagicDebugBufferIndex = 1024;

// In ShaderDebug.hlsli
DebugInfo GetDebugInfo()
{
    ByteAddressBuffer debugBuffer = ResourceDescriptorHeap[MagicDebugBufferIndex];
    return debugBuffer.Load<DebugInfo>(0);
}

In addition to the index of the print buffer descriptor, I have the mouse cursor position in there as well since that’s useful for “print data for the pixel under the cursor” scenarios. But you could add other things too: debug flags or floats that you can use in the shader without having to explicitly add new ones, frame indices or other program state, whatever you want.

Dealing With The String Problem

Now that we have access to our magic debug info buffer which then gives us access to our print buffer, we can start building up the functionality needed to write data into that buffer. This is where things unfortunately get rather dicey in HLSL, which has no native support for working with strings or even a char type. The SPIR-V printf kinda works around this, since you only pass a literal to a printf intrinsic and then the compiler handles the rest. But this does not help us at all if we need to do our own processing of the string, which we need to do for our home-grown printf implementation. One workaround I’ve used myself and seen used elsewhere is to declare uint arrays of character literals, and pass those around. This works nicely enough with existing language support, but it’s about as ugly as you’d imagine:

const uint printStr[] = { 'T', 'h', 'i', 's', ' ', 'i', 's', ' ', 'a', ' ',
                          't', 'e', 'r', 'r', 'i', 'b', 'l', 'e',
                          ' ', 'w', 'a', 'y', ' ', 't', 'o', ' ',
                          'w', 'r', 'i', 't', 'e', ' ', 'a',
                          ' ', 's', 't', 'r', 'i', 'n', 'g', };
DebugPrint(printStr);

While this is not unworkable, it’s very ugly and takes significantly longer to write strings this way. You could potentially use editor tooling to help alleviate this… in fact I used to have a Sublime Text add-in that I wrote which could convert back and forth between a string literal and an array of char literals. But that’s at best a band aid.

Another alternative would be to write or use some kind of custom pre-processor that can extract the string literals from the shader and replace them with something else. This could seamlessly replace a literal with that uint array monstrosity above. Or you could potentially take it even further and replace the string with some kind of hash or token, and then store the string somewhere accessible on the CPU side so that it can be looked up when the print buffer is read back and resolved. That last approach could potentially be quite a bit more efficient on the shader side by drastically reducing the amount of data that needs to be written to the buffer, but it is more complicated. In particular adding any kind of shader pre-processing is a heavy-handed step, and the decision to take on the associated burdens should not be treated lightly. Some engines have already chosen to do this for other reasons, in which case adding string literal handling would be a smaller incremental cost.

Really neither of these solutions are ideal, particularly for a smaller codebase being used for experimenting or messing around. I filed a GitHub issue on the DXC repo with a request for better string support back in October 2021, but as of January 2024 it still has not had a resolution.

A Cursed Path

Update 05/04/2025: as of the February 2025 release of DXC, the compiler will no longer let you pass a string literal as an argument for a templated function. This new error message was a fix for the below code pattern causing crashes in SPIR-V codegen. That means the StrLen function in the code below no longer works. As a workaround, the for loop on the character count can be replaced with a while loop that checks for the null terminator. Either way, please comment on this feature request to indicate your support if you care about strings in HLSL. Original article contents follow.

Before I go any further I’ll warn that this code is going to be ugly, and is almost certainly relying on unintended functionality in the compiler that just happened to be exposed through templates. It’s very possible this won’t work in the future, so I would not rely on this for anything critical and perhaps it would be safest to avoid it altogether. You’ve been warned!

While experimenting with HLSL 2021, I stumbled upon a way to use a combination of template and macro hacks to work with string literals and extract the characters as integers. It turns out that literals can be passed to templated functions that expect an array of type T, where T is ultimately char even though that type is intentionally not exposed. While indexing into that array doesn’t seem to work, we can make a simple strlen implementation that just uses the size of the array to determine the length of the string literal:

template<typename T, uint N> uint StrLen(T str[N])
{
    // Includes the null terminator
    return N;
}

Nice! As for indexing into the characters of the literal, while we can’t do that from a templated function we can do it by indexing into the literal itself. For example, "Hello"[2]. This means we have to resort to a loop in a macro, but for me this is an acceptable cost of doing business. But we still have one more problem: because of the way that char has been disabled inside of DXC, you can’t actually do useful things with it. This includes any arithmetic, or even casting it to an int or uint. However it turns that there is one thing you can do with it, which is compare it to another character literal. Therefore "Hello"[2] == 'l' will evaluate to true. While this isn’t immediately useful, it does mean we can write the world’s most cursed CharToUint function:

template<typename T> uint CharToUint(in T c)
{
    if(c == 'A')
        return 65;
    if(c == 'B')
        return 66;
    if(c == 'C')
        return 67;
    if(c == 'D')
        return 68;
    if(c == 'E')
        return 69;
    if(c == 'F')
        return 70;
    if(c == 'G')
        return 71;

    // ...and about 90 more cases to handle
  }

Putting it all together, we can finally make a macro that can process the string literal one at a time so that it can be packed into a buffer:

#define DebugPrint(str, ...) do {                            \
    ShaderDebug::DebugPrinter printer;                       \
    printer.Init();                                          \
    const uint strLen = ShaderDebug::StrLen(str);            \
    for(uint i = 0; i < strLen; ++i)                         \
        printer.AppendChar(ShaderDebug::CharToUint(str[i])); \
    printer.StringSize = printer.ByteCount;                  \
    printer.AppendArgs(__VA_ARGS__);                         \
    printer.Commit(ShaderDebug::GetDebugInfo());             \
} while(0)

Amazing? Terrible? Helpful? Morally dangerous? I’m honestly not sure, you’ll have to decide for yourself.

Packing It All Into A Buffer

Now that we’ve covered strings, we’re ready to look how the string and any arguments get packed into our print buffer. You may have noticed that in my DebugPrint macro, I’ve got a DebugPrinter type that’s doing most of the heavy lifting. Let’s hook at how that’s written:

struct DebugPrinter
{
    static const uint BufferSize = 256;
    static const uint BufferSizeInBytes = BufferSize * sizeof(uint);
    uint InternalBuffer[BufferSize];
    uint ByteCount;
    uint StringSize;
    uint ArgCount;

    void Init()
    {
        for(uint i = 0; i < BufferSize; ++i)
            InternalBuffer[i] = 0;
        ByteCount = 0;
        StringSize = 0;
        ArgCount = 0;
    }

    uint CurrBufferIndex()
    {
        return ByteCount / 4;
    }

    uint CurrBufferShift()
    {
        return (ByteCount % 4) * 8;
    }

    void AppendChar(uint c)
    {
        if(ByteCount < BufferSizeInBytes)
        {
            InternalBuffer[CurrBufferIndex()] |= ((c & 0xFF) << CurrBufferShift());
            ByteCount += 1;
        }
    }

Basically we have an internal buffer of uint that we use to store the converted characters, packed as 1 of the 4 bytes in the uint. As we loop over the string literal, we just keep incrementing ByteCount and packing in the data. However for a printf we also want to be able to handle arguments, that way we can print integers and floats and the like. For those we will also pack the data into the internal buffer, but we will also store a special code before each argument to provide the CPU code with the type and size of the argument:

enum ArgCode
{
    DebugPrint_Uint = 0,
    DebugPrint_Uint2,
    DebugPrint_Uint3,
    DebugPrint_Uint4,
    DebugPrint_Int,
    DebugPrint_Int2,
    DebugPrint_Int3,
    DebugPrint_Int4,
    DebugPrint_Float,
    DebugPrint_Float2,
    DebugPrint_Float3,
    DebugPrint_Float4,

    NumDebugPrintArgCodes,
};

template<typename T, uint N> void AppendArgWithCode(ArgCode code, T arg[N])
{
    if(ByteCount + sizeof(arg) > BufferSizeInBytes)
        return;

    if(ArgCount >= MaxDebugPrintArgs)
        return;

    AppendChar(code);
    for(uint elem = 0; elem < N; ++elem)
    {
        for(uint b = 0; b < sizeof(T); ++b)
        {
            AppendChar(asuint(arg[elem]) >> (b * 8));
        }
    }

    ArgCount += 1;
}

To make things work nicely with the arguments passed to the macro, we have some trampoline functions that append each argument individually and pass the correct code to AppendArgWithCode:

void AppendArg(uint x)
{
    uint a[1] = { x };
    AppendArgWithCode(DebugPrint_Uint, a);
}

void AppendArg(uint2 x)
{
    uint a[2] = { x.x, x.y };
    AppendArgWithCode(DebugPrint_Uint2, a);
}

void AppendArg(uint3 x)
{
    uint a[3] = { x.x, x.y, x.z };
    AppendArgWithCode(DebugPrint_Uint3, a);
}

// More of these for floats, signed integers, etc.

void AppendArgs()
{
}

template<typename T0> void AppendArgs(T0 arg0)
{
    AppendArg(arg0);
}

template<typename T0, typename T1> void AppendArgs(T0 arg0, T1 arg1)
{
    AppendArg(arg0);
    AppendArg(arg1);
}

template<typename T0, typename T1, typename T2> void AppendArgs(T0 arg0, T1 arg1, T2 arg2)
{
    AppendArg(arg0);
    AppendArg(arg1);
    AppendArg(arg2);
}

// More of these for higher arg counts

This allows the macro to just do printer.AppendArgs(__VA_ARGS__); and it all works.

Finally we have one last Commit() method on DebugPrinter that actually stores everything into the RWByteAddressBuffer along with a special header:

void Commit(in DebugInfo debugInfo)
{
    if(ByteCount < 2)
        return;

    // Round up to the next multiple of 4 since we work with 4-byte alignment for each print
    ByteCount = ((ByteCount + 3) / 4) * 4;

    RWByteAddressBuffer printBuffer = ResourceDescriptorHeap[debugInfo.PrintBuffer];

    // Increment the atomic counter to allocate space to store the bytes
    const uint numBytesToWrite = ByteCount + sizeof(DebugPrintHeader);
    uint offset = 0;
    printBuffer.InterlockedAdd(0, numBytesToWrite, offset);

    // Account for the atomic counter at the beginning of the buffer
    offset += sizeof(uint);

    if((offset + numBytesToWrite) > debugInfo.PrintBufferSize)
        return;

    // Store the header
    DebugPrintHeader header;
    header.NumBytes = ByteCount;
    header.StringSize = StringSize;
    header.NumArgs = ArgCount;

    printBuffer.Store<DebugPrintHeader>(offset, header);
    offset += sizeof(DebugPrintHeader);

    // Store the buffer data
    for(uint i = 0; i < ByteCount / 4; ++i)
        printBuffer.Store(offset + (i * sizeof(uint)), InternalBuffer[i]);
}

In Commit() we’re assuming there’s a counter at the start of the print buffer that indicates how many bytes have been written to it. By atomically incrementing that counter we can safely “allocate” some space for our print data, and also ensure that enough space is left so that we can early-out if the buffer is full. Once the counter is incremented we fill out our small header and write that first, then write the contents of the internal buffer one uint at a time.

While there’s some complexity and ugliness here that we need to hide in an included file, actually doing a print in the shader code is about as easy as it gets:

float3 color = ComputeColor();
DebugPrint("The color is {0}", color);

Since we’re embedding the argument type as a code in the buffer, there’s no need for printf-style format specifiers and we can instead use argument IDs like in std::format or C#’s String.Format.

Reading Back On The CPU

In order to have the CPU process all of the print data that’s been generated on the GPU, we need to issue some commands that copies the print buffer to a CPU-accessible readback buffer:

void EndRender(ID3D12GraphicsCommandList7* cmdList)
{
    PIXMarker marker(cmdList, "ShaderDebug - EndRender");

    DX12::Barrier(cmdList, PrintBuffer.WriteToReadBarrier( { .SyncAfter = D3D12_BARRIER_SYNC_COPY,
                                                             .AccessAfter = D3D12_BARRIER_ACCESS_COPY_SOURCE } ));

    const ReadbackBuffer& readbackBuffer = PrintReadbackBuffers[DX12::CurrentCPUFrame % DX12::RenderLatency];
    cmdList->CopyResource(readbackBuffer.Resource, PrintBuffer.Resource());
}

Doing the copy to the readback buffer allows us to map and read the oldest generated print buffer data every frame so that we can process it:

if(DX12::CurrentCPUFrame >= DX12::RenderLatency)
{
    const ReadbackBuffer& readbackBuffer = PrintReadbackBuffers[(DX12::CurrentCPUFrame + 1) % DX12::RenderLatency];
    DebugPrintReader printReader(readbackBuffer.Map<uint8>(), uint32(readbackBuffer.Size));

    while(printReader.HasMoreData(sizeof(DebugPrintHeader)))
    {
        const DebugPrintHeader header = printReader.Consume<DebugPrintHeader>(DebugPrintHeader{});
        if(header.NumBytes == 0 || printReader.HasMoreData(header.NumBytes) == false)
            break;

        std::string formatStr = printReader.ConsumeString(header.StringSize);
        if(formatStr.length() == 0)
            break;

        if(header.NumArgs > MaxDebugPrintArgs)
            break;

        argStrings.Reserve(header.NumArgs);
        for(uint32 argIdx = 0; argIdx < header.NumArgs; ++argIdx)
        {
            const ArgCode argCode = (ArgCode)printReader.Consume<uint8>(0xFF);
            if(argCode >= NumDebugPrintArgCodes)
                break;

            const uint32 argSize = ArgCodeSizes[argCode];
            if(printReader.HasMoreData(argSize) == false)
                break;

            const std::string argStr = MakeArgString(printReader, argCode);
            ReplaceStringInPlace(formatStr, ArgPlaceHolders[argIdx], argStr);
        }

        GlobalApp->AddToLog(formatStr.c_str());
    }

    readbackBuffer.Unmap();
}

The processing involves a loop that iterates until all of the print data in the buffer has been processed. For each iteration of the loop, we pull out a DebugPrintHeader that marks the beginning of a single DebugPrint call from a single thread of a shader. That header is then used to gather the actual format string, as well as the number of argument. Each argument is extracted using the code we embedded on the shader side, and a stringified version of the argument is then inserted into the original print string to replace the argument index. The final expanded string with arguments is then passed on to my simple log system, but at that point you could do whatever you’d like with it. In my case the log outputs to a simple ImGui text window, but you can get as fancy as you want.

Going Beyond Printf

Having a shader printf is already hugely valuable and worthwhile, but once you have this sort of infrastructure in place there’s all kinds of things you can do with it. For example you could build a higher-level assert mechanism on top of DebugPrint that logs information when a condition isn’t met. The __FILE__ and __LINE__ predefined macros work in DXC, so you could include that along with the condition itself in the log message. Unfortunately there’s no way I know of to get any kind of “stack strace”, so you’re on your own for that. There also isn’t a standard __debugbreak() intrinsic or anything like that for shaders. The best you can do is enter an infinite loop or force a deliberate page fault, and hope that your GPU crash reporting systems catch it appropriately.

Another common trick is to implement shader-driven debug rendering. If you have a CPU-based debug renderer then it’s pretty straightforward to add special “print” code that can be detected on the CPU, which can then extract and forward the draw arguments to the debug renderer. The main downside of doing it this way is that you’ll need to accept a few frames of latency between when the shader draws and when it shows up. For a lot of cases this is totally fine, but it can be a dealbreaker for certain debugging scenarios. A zero-latency alternative is to have a separate buffer for debug draws, and run a compute shader to process that and convert it into indirect draw commands.

Once you take it to the extreme, your print system can basically be a deferred message passing interface between your shaders and the engine. This could potentially be used for all kinds of powerful or wacky things if you’re sufficiently motivated. For example, you could replicate a subset of the Dear ImGui interface in your shader, and give your shaders the ability to draw their own debug UI. The sky is the limit!

CR LF

That’s it for the article! Hopefully what I’ve described is helpful for people implementing similar systems, or provides some ideas for how to improve existing ones. Good luck, and happy printing!

When the monkey’s paw granted our wish for lower-level/explicit graphics APIs, one of the consequences was that we were much more directly exposed to the fact that GPUs can have their own separate set of physical memory. Anybody that’s ever had to purchase a dedicated video card for their PC can probably tell you that such a video card has its own set of physical RAM chips on the board that are reserved for the GPU. Meanwhile, the integrated GPUs often used in laptops and mobile devices will typically share one set of RAM with the CPU. This reality can make things complicated for a graphics programmer, since you potentially have two different physical pools of “GPU memory” that will have different characteristics depending on how you’re using it. Earlier APIs such as D3D11 and OpenGL sort of exposed these pools by abstracting away GPU memory altogether behind different resource types, but in D3D12 and Vulkan you are now more-or-less directly exposed to lower-level allocation and management of GPU memory. Altogether it’s complicated enough to cause confusion and headaches for beginners or even experienced graphics programmers, and it tends to come up rather early since you usually need to grapple with it before you can create your first texture.

In this article we’re going to dive in on this topic, and in particular cover the following things:

• The basics of GPU memory
• How GPU memory works in D3D12
• Common patterns in D3D12
• Some timing results gathered from a D3D12 test app

Ultimately I’m going to cover a lot of things that were already covered in some form by Adam Sawicki’s excellent talk from Digital Dragons 2021 about optimizing for GPU memory pools. I would recommend watching that talk either way, but I’m hoping that this article can complement that presentation by adding some extra details as well as some real-world benchmark results.

To keep this article somewhat focused, I’m also only going to talk about D3D12 here and ignore other APIs. I would suggest reading Adam’s post about Vulkan memory types if you’re looking for information specific to Vulkan. You can also check out his later post about D3D12 memory types and pools for a great overview of how memory pools are exposed through the D3D12 API.

Basics of GPU Memory

As I briefly mentioned in the intro, a particular GPU may or may not have its own physically separate pool of memory to use for storing whatever data it needs to access. In general this sort of “split” memory pool setup is something you see on the discrete GPUs found on video cards, whereas the integrated GPUs that are built onto the same die as the CPU will typically use the same physical memory as the CPU. To start, let’s look at the latter case which is arguably simpler.

Integrated/UMA GPUs

So-called “integrated” GPUs are what you’ll find built into many of the CPU offerings from Intel as well as the APU offerings from AMD, and are called that because the GPU is literally integrated into the same die as the CPU:

These types of GPU are not a monolith, especially if you look at what you might see in phones, consoles, and Apple/Mac devices. However if you’re looking at the Windows/PC world you can usually assume that the following is true:

• The CPU and the GPU share the same physical memory, and often the same memory controller
• The RAM being used is of the DDR variety typically used for CPUs, as opposed to the more exotic GDDR chips found in video cards
• The amount of raw bandwidth available to the GPU is perhaps an order of magnitude lower than what you might find on high-end dedicated graphics cards. As an example, the Gen12 GPUs found in recent Tiger Lake and Alder Lake CPUs from Intel will range from about ~50 GB/s to ~76 GB/s. Meanwhile an AMD RX 6800 can achieve 512 GB/s to its dedicated Video RAM (VRAM), while an Nvidia RTX 3090 peaks at a whopping 936 GB/s!

Since the CPU and GPU share the same pool of physical memory, this setup is often referred to as a “Unified Memory Architecture”, or UMA for short. While it might seem at first that this is purely a disadvantage relative to a faster dedicated memory pool, this isn’t entirely true. Sharing the memory pool and mechanisms for accessing that memory can make it easier for the CPU and GPU to work together: both processors can directly read and write to the same physical memory addresses, and perhaps even have a shared level of cache hierarchy. This can improve performance for CPU/GPU communication, and also remove certain cases of bugs. The other main advantage is that the system and its applications are free to partition the single pool of memory however they see fit, without having to instead manage two separate pools of memory that can only be used for a single processor.

Dedicated/NUMA GPUs

For a PC that has its own dedicated video card, the situation is of course very different: these video cards have their own bank of physical RAM chips right on the board. Since this RAM pool is physically separate from the CPU RAM, this setup is usually referred to as a “Non-Unifed Memory Architecture” or “NUMA” for short. Often this separate pool is accessed through its own memory controller that’s present on the GPU, and can even have its own separate page tables and virtual address space.

As we mentioned before, dedicated VRAM is capable of achieiving significantly higher bandwidth than what you’ll see for the DDR modules used by PC CPUs. The secret weapon here for VRAM is that they typically use exotic GDDR RAM chips that are specifically designed for high-bandwidth use cases, and these chips are then wired onto very wide 256-bit or even 384-bit buses. The chips, bus, and the GPU itself are all explicitly designed to handle the sort of wide bursts of coalesced memory accesses that you can expect from the 32-wide SIMD arrays that shader programs execute on.

Since the memory pools are physically split on a dedicated GPU setup, sharing data between the CPU and GPU is naturally much more complicated. For sharing to happen, everything needs to happen on the PCI-express bus that the video card is connected to. The GPU is capable of reading from CPU memory (SysRAM) over this bus, which it can do at around 16 GB/s for PCI Express 3.0 and around 32 GB/s with the newer PCI Express 4.0. 16 GB/s really isn’t very much compared with the GPU’s bandwidth to VRAM or even the CPU’s bandwidth to SysRAM, which makes it a resource that has to be utilized rather carefully. A video card will usually have a dedicated DMA unit on board that is optimized for asynchronously “uploading” large chunks of texture or buffer data from SysRAM to VRAM. PCIe can also service general texture and buffer reads issued by the shader core, which can work out well for small buffers that are updated frequently and fit comfortably in caches. The CPU also has the ability to directly write to VRAM over the PCIe bus, through a mechanism known as “Base Address Register”, or BAR for short. Until recently the amount of VRAM accessible through BAR was limited to around 256 MB, limiting its usefulness to special case data that was carefully managed by drivers. More modern systems however can support a feature known as “Resizable BAR”, or ReBAR, which can make the entire pool of VRAM accessible from the CPU.

The fact that the CPU and GPU have their own memory controllers and cache hierarchies can cause further pain and complexity for CPU/GPU communication. PC video cards can typically snoop the caches of the CPU when the CPU writes to GPU-visible SysRAM, however CPU and GPU cache hierarchies are still effectively seperate. This leads to something of a dilemma for the CPU when writing to memory that only the GPU will read from: does it make sense to leave caching enabled and “pollute” the cache with data from GPU, or would it make more sense to bypass the CPU cache entirely? The latter has historically been the default on PC, while making use of a hardware feature known as write combining. This feature effectively buffers nearby writes together so that each individual write doesn’t need its own separate uncached bus transaction, which can work decently as long as you’re doing fairly boring memcpy-style sequential writes. However this does nothing at all for you if you are unlucky enough to try to read from GPU-visible memory: the reads are completely uncached, and they will be quite slow. Things are even worse if you somehow read from VRAM over PCIe, which can be catostrophically slow. Enabling caching can avoid the performance penalty from accidental reads, however it may cause adverse performance for subsequent code by pushing important CPU data out of cache.

How It Works In D3D12

D3D12 applications have to allocate their own memory, so how is all of this exposed to us? Adam Sawicki already gave a great overview of how everything works on his blog, and there is also information in the official programming guide for D3D12. For completeness I’ll provide a brief overview here as well, along with some additional information based on my own experiences.

In D3D12 you have two different ways to allocate GPU-accessible memory for your device: by creating a committed resource, or by creating a heap. Heaps are the more direct way of allocating and utilizing memory, where you must create a heap first with various properties and then “place” resources within that heap via CreatePlacedResource. Committed resources allocate memory indirectly: it’s roughly the same as if the runtime/driver created a single resource-sized heap for you behind the scenes, and then placed the resource in that heap for you. For that reason, we will focus on heaps here instead of committed resources.

Once you’ve created your device, you can query some feature bits in order to get some information in advance about the architecture and behavior of the GPU that you will be allocating memory for. In particular there are two fields of interest on D3D12_FEATURE_DATA_ARCHITECTURE, which you obtain by calling CheckFeatureSupport with D3D12_FEATURE_ARCHITECTURE:

typedef struct D3D12_FEATURE_DATA_ARCHITECTURE
{
    UINT NodeIndex;
    BOOL TileBasedRenderer;
    BOOL UMA;                 // < -- 1
    BOOL CacheCoherentUMA;    // < -- 2
} D3D12_FEATURE_DATA_ARCHITECTURE;

1. The UMA flag tells you if the GPU uses a Unified Memory Architecture where a single physical pool of RAM is shared between the CPU and GPU, like what you’d normally find on an integrated GPU
2. If UMA is true and CacheCoherentUMA is also true, then the CPU and GPU caches are fully coherent with each other. In this situation using uncached writes for filling GPU-visible SysRAM does not make sense, since the GPU can benefit from data being resident in cache.

In a bit we’ll talk about how these flags affect the behavior of a device and ways an app can respond to them, but first let’s look at the lowest-level mechanism for creating a heap: creating a “custom” heap. Custom heaps allow the app to fully specify all of the supported attributes of a heap, which is done through these two structs:

typedef struct D3D12_HEAP_DESC
{
    UINT64 SizeInBytes;
    D3D12_HEAP_PROPERTIES Properties;
    UINT64 Alignment;
    D3D12_HEAP_FLAGS Flags;
} D3D12_HEAP_DESC;

typedef struct D3D12_HEAP_PROPERTIES
{
    D3D12_HEAP_TYPE Type;   // Must be D3D12_HEAP_TYPE_CUSTOM
    D3D12_CPU_PAGE_PROPERTY CPUPageProperty;
    D3D12_MEMORY_POOL MemoryPoolPreference;
    UINT CreationNodeMask;
    UINT VisibleNodeMask;
} D3D12_HEAP_PROPERTIES;

First, there is SizeInBytes which is exactly what it sounds like. There is also an Alignment needed for satisfying alignment requirements for specific kinds of resources (in particular, MSAA textures). There is also a set of D3D12_HEAP_FLAGS that you can configure, but they are not generally relevant to our focus on memory pools and so we will ignore this. Instead, let’s look at two of the interesting enums that you can specify in D3D12_HEAP_PROPERTIES:

typedef enum D3D12_MEMORY_POOL
{
    D3D12_MEMORY_POOL_UNKNOWN = 0,  // Not relevant for custom heaps
    D3D12_MEMORY_POOL_L0  = 1,
    D3D12_MEMORY_POOL_L1  = 2
} D3D12_MEMORY_POOL;

typedef enum D3D12_CPU_PAGE_PROPERTY
{
    D3D12_CPU_PAGE_PROPERTY_UNKNOWN = 0,    // Not relevant for custom heaps
    D3D12_CPU_PAGE_PROPERTY_NOT_AVAILABLE = 1,
    D3D12_CPU_PAGE_PROPERTY_WRITE_COMBINE = 2,
    D3D12_CPU_PAGE_PROPERTY_WRITE_BACK  = 3
} D3D12_CPU_PAGE_PROPERTY;

Now things are getting interesting! D3D12_MEMORY_POOL is exactly what it sounds like: it allows us to choose the physical pool of RAM that the memory will live in. On NUMA devices (dedicated video cards) L0 maps to SysRAM AKA CPU memory, while L1 maps to VRAM located on the video card. However on UMA devices there is only physical memory pool available, which is LO. L1 is not available on such devices, and attempting to use it will cause the CreateHeap call to fail. This choice of memory pool must also be carefully paired with an appropriate D3D12_CPU_PAGE_PROPERTY in order to play nice with the CPU’s ability (or lack thereof) to access the memory pool. In practice it ends up working something like this:

if(UMA)
{
    if(MemoryPoolPreference == D3D12_MEMORY_POOL_L0)
    {
        // Memory will live SysRAM pool shared between CPU and GPU
        if(CPUPageProperty == D3D12_CPU_PAGE_PROPERTY_NOT_AVAILABLE)
        {
            // Not accessible to the CPU at all
        }
        else if(CPUPageProperty == D3D12_CPU_PAGE_PROPERTY_WRITE_COMBINE)
        {
            // Uncached for the CPU. Write combined for CPU writes, very very slow CPU reads.
            // Ok for passing data from CPU -> GPU.
        }
        else if(CPUPageProperty == D3D12_CPU_PAGE_PROPERTY_WRITE_BACK)
        {
            if(CacheCoherentUMA)
            {
                // CPU pages are cached, fast for writes and reads.
                // GPU potentially benefits from data being in cache as well.
                // Good for all purposes.
            }
            else
            {
                // Cached for CPU. May not have ideal CPU performance when
                // compared with uncached/write-combined. Good for
                // reading back data from GPU -> CPU
            }
        }
    }
    else if(MemoryPoolPreference == D3D12_MEMORY_POOL_L1)
    {
        // Error, can't use L1 on UMA!
    }
}
else
{
    // NUMA device
    if(MemoryPoolPreference == D3D12_MEMORY_POOL_L0)
    {
        // Memory will live in the SysRAM pool
        if(CPUPageProperty == D3D12_CPU_PAGE_PROPERTY_NOT_AVAILABLE)
        {
            // Lives in SysRAM but the CPU can't access it, don't do this!
        }
        else if(CPUPageProperty == D3D12_CPU_PAGE_PROPERTY_WRITE_COMBINE)
        {
            // Uncached for the CPU. Write combined for CPU writes, very very slow CPU reads.
            // GPU will read this data over PCIe at low speeds relative to VRAM.
            // Good for passing data from CPU -> GPU
        }
        else if(CPUPageProperty == D3D12_CPU_PAGE_PROPERTY_WRITE_BACK)
        {
            // Cached for the CPU with full speed reads and writes.
            // GPU will access this data over PCIe at low speeds relative to VRAM.
            // Good for reading back data from GPU -> CPU.
            // Can be used for CPU -> GPU traffic as well, but may pollute
            // the CPU cache with data that will only be read by the GPU.
        }
    }
    else if(MemoryPoolPreference == D3D12_MEMORY_POOL_L1)
    {
        // Memory will live in the high-bandwidth VRAM pool
        if(CPUPageProperty == D3D12_CPU_PAGE_PROPERTY_NOT_AVAILABLE)
        {
            // The CPU can't access this memory at all, only the GPU can.
            // Good for data that only the GPU will ever read or write, such as render targets.
            // Good for persistent, read-only resources but filling the data requires
            // a two-step upload process.
        }
        else if(CPUPageProperty == D3D12_CPU_PAGE_PROPERTY_WRITE_COMBINE)
        {
            // Error, not valid for a NUMA device
        }
        else if(CPUPageProperty == D3D12_CPU_PAGE_PROPERTY_WRITE_BACK)
        {
            // Error, not valid for a NUMA device
        }
    }
}

This is naturally a bit complicated to manage all of this yourself, so to make things a bit easier D3D12 provides “built-in” heap types that are optimized for common use cases. These types are accessed by using the D3D12_HEAP_TYPE_DEFAULT, D3D12_HEAP_TYPE_UPLOAD, or D3D12_HEAP_TYPE_READBACK values of D3D12_HEAP_TYPE when creating the heap, and setting the MemoryPoolPreference to D3D12_MEMORY_POOL_UNKNOWN and CPUPageProperty to D3D12_CPU_PAGE_PROPERTY_UNKNOWN. Here’s how they work:

• DEFAULT is intended for resources that the GPU primarily accesses at full bandwidth. On NUMA this maps to L1 with no CPU access, while on UMA it maps to L0 with no CPU access.
• UPLOAD is intended for CPU -> GPU communication where the CPU writes the data and the GPU reads it. On NUMA this maps to L0 with WRITE_COMBINE CPU access, while on UMA it maps to L0 with WRITE_COMBINE if CacheCoherentUMA is false, and WRITE_BACK if CacheCoherentUMA is true.
• READBACK is intended for GPU -> CPU communication where the GPU writes the data and the CPU reads it. On both UMA and NUMA devices this maps to L0 with WRITE_BACK CPU access.

One thing you may have noticed about this setup is that CPU-accessible VRAM is not currently exposed through any of these options. While the driver can use the small amount of BAR VRAM if it decides it’s worthwhile, applications have no direct way of allocating from that special bit of memory. This is true even on newer machines with ReBAR support that makes the entire pool of VRAM accessible to the CPU. However, here is an NVAPI extension that allows you to query and allocate from CPU-accessible VRAM if the device was created on a supported Nvidia GPU. Meanwhile, AMD’s driver can automatically promote allocations to CPU-accessible VRAM based on heuristics and ReBAR support. Hopefully a future version of D3D12 adds support for this functionality, since it is certainly a useful option.

It’s not necessarily obvious from reading the official documentation, but these various built-in and custom heap types come with limitations on resource usage that are either implicit or explicit depending on the exact use case. In general buffer resources are not limited: you can place them in any of the built-in heap types as well as any possible configuration of a custom heap, provided that the heap wasn’t created with D3D12_HEAP_FLAG_DENY_BUFFERS. Buffers have an implied linear layout of the data that is similar to an array in C/C++, which makes it trivial for both CPU and GPU to access these resources. Textures however are special: these typically do not use a linear layout¹, but instead use a hardware-specific swizzle pattern that’s designed for maximum spatial locality and cache efficiency. This isn’t an issue if the GPU accesses the texture since it knows how to read/write to these special patterns, but it’s a major issue if you wanted to access such a texture from the CPU: your app doesn’t know the swizzle pattern, which means only the driver knows how to encode or decode it. Even if you could access the texture you likely wouldn’t want to read it from SysRAM on a NUMA device, since the slow PCIe bandwidth could degrade performance. The built-in heaps use a heavy-handed means of guarding you from these issues by flat-out forbidding you from creating texture resources in an UPLOAD heap. Instead you must use a “two-step” upload scheme where the CPU writes un-swizzled data to a buffer in UPLOAD memory, and then the GPU itself copies and swizzles the data to a texture resource in DEFAULT memory.

With custom heaps, you can potentially bypass the texture/heap limitations but it must be done carefully. D3D12 will let you place a ROW_MAJOR texture resource in a CPU-writable L0 heap even on a NUMA device, which you can then fill from the CPU. But you should be prepared for excruciatingly poor performance if you try to do this, due poor spatial locality coupled with slow PCIe bandiwdth. A somewhat better option is to use the special WriteToSubresource function, which allows the driver to fill an UNKNOWN-layout texture using a hardware-specific swizzle pattern. This would likely still be too slow on NUMA hardware that reads from L0, however it can be compelling option for UMA devices that share a memory pool anyway. WriteToSubresource allows you to forego a GPU-side copy to initialize your resources, which is appealing since integrated GPUs often lack an asynchronous DMA for performing said copies. WriteToSubresource might one day be relevant for dedicated video cards if ReBAR becomes the norm an D3D12 exposes CPU-accessible VRAM, however it ultimately may not be desirable for apps to burn a CPU core on swizzling texture data.

If you want to keep an eye on how much memory your app is using from the two pools, QueryVideoMemoryInfo can be called from a IDXGIAdapter3 to get the current usage of L0 and L1. To do this you just need to know that DXGI_MEMORY_SEGMENT_GROUP_LOCAL maps to L1, while DXGI_MEMORY_SEGMENT_GROUP_NON_LOCAL maps to L0. The OS-assigned budgets returned by this function are also useful to keep an eye on. In particular if you start to execeed the L1 budget, the OS can start to automatically “demote” some of your heaps from L1 to L0 in order to leave enough free VRAM for other processes on the system. If this demotion happens to the wrong resource it can destroy your performance, and so you really want to avoid that situation if you can help it. PIX for Windows can also show you this information when attached to your app, which is handy. If demotion does occur you can mitigate the damage by assigning a residency priority on your heaps/resources. This can let you keep your more critical resources in VRAM by allowing the OS to demote less-important things.

Common Patterns in D3D12

After discussing some of the specifics of the API, let’s now talk about how D3D12 apps and samples typically work with memory pools.

Textures And The Two-Step Upload

To start, let’s discuss textures since they are somewhat special. As we covered earlier, textures tend to be swizzled in an opaque hardware-specific layout while also being quite large and bandwidth-hungry. Therefore the common choice is to throw these guys in a DEFAULT heap, which maps to the high-bandwidth L1/VRAM pool on NUMA devices (and as we mentioned earlier, the runtime won’t even let you create a texture in an UPLOAD heap). This puts us in our desired physical memory pool on both UMA and NUMA devices, but introduces a separate problem: the CPU cannot directly initialize the contents of a texture living in DEFAULT. Instead we must rely on what I call the two-step upload process, which goes something like this:

1. Some memory from a buffer resource living in CPU-accessible memory (typically a UPLOAD heap) needs to be reserved, enough to fit the texture data being uploaded. This is sometimes called a “staging” buffer.
2. The CPU loads or copies un-swizzled texture data into the CPU-accessible buffer
3. CopyTextureRegion is called on a command list, one for each subresource in the texture (mip level or array slice)
4. The command list is submitted on a queue where it executes on the GPU, and proper synchronization is used to ensure tha the copy completes before another command attempts to read from the texture

This process has the GPU effectively accomplish two goals: swizzling the texture data from a linear layout into a hardware-specific texel order, and getting the texture data into fast VRAM. Both are critical for performance, since textures are bandwidth heavy and rely on caching and 2D locality to keep things from slowing down too much. Neither of these are stricly recquired, since it’s possible to allocate textures in SysRAM via custom heaps and you can also specify that a texture resource should use D3D12_TEXTURE_LAYOUT_ROW_MAJOR when creating it. However both of these can come with severe peformance penalties, so it is highly recommended to place textures in VRAM with D3D12_TEXTURE_LAYOUT_UNKNOWN (or D3D12_TEXTURE_LAYOUT_64KB_UNDEFINED_SWIZZLE for tiled resources).

The two-step upload process might sound rather simple at first, but make no mistake: there are many potential paths to take and a lot of subtleties to consider when it comes to allocating staging memory and scheduling the actual copy/upload. We’ll dig into that in a little bit, but first let’s discuss buffers.

Should We Upload Buffers?

The two-step upload is a de-facto requirement for textures, but what about buffers? Unlike textures, buffers can be created in UPLOAD heaps and can be read directly from SysRAM by shaders and copy operations. But wouldn’t they also benefit from being uploaded to VRAM? The answer to that is “yes, sometimes”. For large, read-only or rarely-updated buffers allocating them in VRAM and uploading their contents makes a ton of sense, and you can likely use the same mechanisms that you would use for initializing textures. The situation is slightly easier since you don’t need to worry about CopyTextureRegion and RowPitch alignment requirements, but is otherwise mostly the same.

Things are different though for smaller, frequently-updated buffers. For example, a 64-byte constant buffer containing a per-draw world transform that you update every frame. In these cases the data can fit nicely in cache, so you’re probably not losing very much performance (if any) by letting the GPU read it over PCIe. Also if you’re only reading the constant buffer once, you’re liable to actually make performance worse by copying from SysRAM to VRAM first. Therefore you can usually save yourself the headache for these kinds of small dynamic buffers, and just let them live in SysRAM. If the data is truly fire-and-forget and doesn’t need to persist longer than the current frame being recorded, you can use really fast linear allocator patterns to quickly grab a bit of temporary UPLOAD memory that’s tied to frame lifetime.

Where things get trickier is buffers that are not small, are updated fairly frequently, and are accessed either more than once or with addresses that are unique for every thread of a draw or dispatch. For these cases you probably do want the full bandwidth of VRAM, but uploading their contents every frame can add additional costs and headaches that need to be considered. The simplest option of uploading the data syncronously via a CopyResource or CopyBufferRegion call on the main DIRECT/graphics queue gets the job done, but that introduces additional GPU time on that queue that has to be offset with the improved bandwidth afforded by VRAM. In the next section we’ll talk about more advanced scenarios for uploading data that can potentially result in better overall performance.

Once CPU-accessible VRAM is ubiquitous through ReBAR, it seems plausible that nearly all CPU-updated buffers could live in VRAM and be directly updated from the CPU. This would simplify the situation considerably compared to what’s required now. For integrated UMA GPUs that situation already exists: there’s really no reason to do a two-step upload for buffer resources on these systems, and it would benefit engines to detect this case if they want to optimize for these GPUs.

Working With The COPY Queue

When discussing the two-step upload we mentioned how there are actually many choices and trade-offs to consider when setting up upload paths in your engine. In particular you need to choose how exactly you allocate/aquire the UPLOAD buffer used as the source of the copy, and you also need to decide how and when you submit your command list on a command queue. The official D3D12 samples use a very simple scheme where an UPLOAD buffer is created as a committed resource and the copy is immediately submitted on a DIRECT command queue (via UpdateSubresources from the d3dx12.h helper header) followed by an immediate CPU wait for the submission to finish. That’s totally fine for simple use cases, but what if you have many uploads to do and you don’t want to spend a ton of time creating and destroying committed resources? What if you need to stream in these textures in background while most of your CPU cores are busy and your GPU is already executing giant batches of rendering work on the DIRECT queue? What if there’s a texture or buffer in VRAM that needs to be updated every single frame, and you don’t want updating it to stall out one of your rendering threads? For these situations you probably want a more complex approach that better fits the use case.

In many cases it’s ideal to use the dedicated COPY queue type to perform this kind of uploading. On NUMA devices this queue typically feeds to a dedicated DMA unit on the video card, which is specifically optimized for performing asynchronous copies of data over the PCIe bus. This not only means it may be more efficient than a copy performed on a DIRECT queue, but it also means that the actual copy operation can potentially run in parallel to other graphics and compute work running on the GPU! For first-time engine initialization this is not a big deal, but for background streaming/loading it’s definitely a better option than forcing your critical per-frame drawing to wait for a bunch of uploads to finish executing (which is what will happen if you submit your copies to your main DIRECT queue). Async copies can still slow down your graphics work a bit by consuming some bandwidth, but this is generally still a better proposition than forcing your graphics work to wait for a bunch of slow PCIe copies to finish. To illustrate the difference, the images below are from two different GPU Trace captures taken with Nvidia Nsight graphics:

The top image is capture where a 16 MB buffer is uploaded every frame from SysRAM to VRAM using CopyBufferRegion on the main DIRECT queue, which is followed by a compute dispatch that immediately reads from the uploaded buffer. The bottom image shows what things look like if the CopyBufferRegion is instead submitted to a COPY queue, allowing it to overlap with the previous frame’s work on the DIRECT queue. Notice how the frame time is 1.2ms shorter when using the COPY queue, despite the dispatch taking slightly longer when overlapped with the upload.

The trade-off here is that you will take on some significant complexity by using a separate COPY queue instead of re-using your primary queue. Submitting on an async queue means it is now up to you to properly synchronize your submissions across multiple queues: you will need to signal a fence on your COPY queue after submitting a command list and then have another queue wait on that fence before commands on the latter queue can read from the resources being uploaded. This is exactly like “async compute” setups that are common for overlapping compute work with graphics work, and has some of the same headaches. Like async compute, you need to carefully manage the submissions and synchronization between your queues if you’re hoping for them to overlap. You also need to take care not to delete or otherwise re-use your staging buffers until the CPU has fully waited for the COPY fence to be signaled, otherwise you will get incorrect results and/or a GPU crash.

One extra bonus of using a COPY queue is that under the existing (and soon-to-be-legacy) barrier setup there are special rules carved out for resource decay and promotion when a COPY queue is involved. When a resource is copied to or from on a COPY queue, that resource automatically decays back to a COMMON state after the COPY command list finished executing. Since COMMON resources get auto-promoted on their first use, this means that you can typically completely skip issuing any barriers for read-only resources that you initialize this way! This is extremely convenient, and can even save some performance as well.

Before moving on, I should point out that all of the above really only applies for NUMA/dedicated devices. A UMA/integrated device often doesn’t have a dedicated DMA engine, and so any submissions to COPY queues will end up getting serialized/flattened onto a single hardware queue. The typical DEFAULT/UPLOAD setup will still work correctly on ths type of hardware, but if you want to optimize for integrated devices then you may want to consider adding a path that uses custom heaps to allow the CPU to directly initialize your textures.

Two COPY Queues Are Better Than One?

Let’s imagine you’ve setup your engine to be able to submit SysRAM -> VRAM uploads on a COPY queue so that they run asynchronously to your main graphics and compute work that’s performed every frame. You hook this up to the buffer/texture initialization uploads that occur when new resources are created, and it seems to work great. In fact it works so well that you decide to re-use the same path for when the CPU needs to perform per-frame updates of large buffers that live in VRAM. Whenever you peek at a timing capture you see the uploads overlapping with your graphics work, so you consider your work done and move on. But then one day you get reports that the game’s frame time is spiking erratically in areas where there is heavy background streaming, and the spikes appear to be coming from the GPU. You peek at the engine’s timestamp queries surrounding your main graphics queue submission, and confusingly everything looks fine. It’s not until you take a full multi-frame timing capture that you notice the problem:

It looks like your uploads from the background streaming are effectively hogging the COPY queue, preventing your more-critical buffer update upload from running. Since your graphics queue waits for the buffer upload to complete, it remains idle for a while which makes the overall frame time longer. This is pretty unfortunate and clearly needs to be fixed, but what should you do? One option is to move your critical upload onto the main DIRECT queue, effectively giving up on getting any overlap for that copy and letting your background uploads have the COPY queue all to themselves:

Things are much improved with this setup. In the fabricated scenario I setup for these captures we’ve gone from a worst-case of 19.39ms down to about 12ms per frame. You’re tempted to submit this and mark the JIRA task as complete, but as a performance-oriented programmer you can’t shake the nagging feeling that we can still do even better than this. It’s just not ideal to require the graphics queue to have to eat the cost of that critical buffer upload, and it’s really not ideal that the critical upload is actually running slower than usual since it’s sharing the PCIe bandwidth with the background uploads (in my test case it’s taking about 2.43ms instead of 1.2ms for a 16 MB buffer, roughly double the time). Seeing how the two submissions end up sharing bandwidth, you think to yourself “wouldn’t it be great if the graphics work, background uploads, and critical uploads could all run asynchronously to each other and the uploads could split the bandwidth?”. Figuring you have nothing to lose, you decide to add a second COPY queue to the engine and redirect all of your critical uploads to that queue instead. Much to your surprise and delight, this works!

It turns out that dedicated video cards from Nvidia and AMD support this kind of thing, which is advertised by the fact that they expose two transfer queues in Vulkan. What ends up happening is the background uploads and critical upload still share bandwidth which slows down the critical upload, but since the critical upload is able to execute much earlier it doesn’t hold up the main graphics work. This won’t scale up forever: at some point if there are too many critical uploads (or the main graphics work completes too quickly) then the critical upload COPY queue will hold up the main DIRECT/graphics queue. But it’s still absolutely a compelling option for these kinds of advanced scenarios, and likely justifies the additional complexity.

Allocating Staging Memory

As I alluded to earlier, another complicating aspect of the two-step upload is that you need to reserve a sufficient amount of staging memory from somewhere. This memory then needs to remain allocated and untouched until the copying queue completely finishes executing the submission that pulls from the staging buffer. The simple approach seen in some samples is to create a committed buffer resource every time some data needs to be uploaded, and then destroy it after the submission is finished. While this certainly works, creating those committed resoures is definitely not free (especially if you omit the D3D12_HEAP_FLAG_CREATE_NOT_ZEROED flag!), and you could slow down your resource loading and creation by constantly creating and freeing like that. Another tricky issue is memory usage: if you keep allocating more staging buffers, you won’t be able to free them until the GPU consumes them. This can result in unbounded memory usage if many uploads are being batched into a single queue submission during engine initialization. A single big submission with tons of uploads can also be bad on integrated GPUs that don’t have an async DMA unit that you can use, in which case you’re just going to stall the main graphics queue until all of the uploads finish.

As an alternative, one might consider sub-allocating from a single large staging buffer instead of constantly creating new ones. Since the allocation is tied to the GPU progress of copy operations, a ring buffer can be a pretty natural fit. The overall flow could go something like this:

1. An upload is requested for N bytes
2. Check if there is enough space in the ring buffer, if not stall and wait for previous submissions to finish
3. Allocate space in the ring buffer by moving the head back N bytes
4. Copy resource data into the ring buffer at the right offset
5. Record a copy command on a command list
6. Submit the command list on a queue
7. Tell the queue to signal a fence once the command list finishes executing
8. Periodically check the fence value of previous submissions, if they’ve completed move up the ring buffer tail to free up more space

This is but one possibility for an allocation scheme, and even within such a setup there are more choices to be made. For instance, should it be thread-safe? If multiple threads can use it, which operations can happen in parallel? Is the calling code responsible for copying into the staging buffer and issuing copy commands, or is the uploader mechanism responsible? When should you submit to the GPU, and when should you poll for completed submissions? Should you batch multiple uploads into a single submission? How does this interact with the “main” per-frame submissions on DIRECT and COMPUTE queues? How big should your ring buffer be, and what should happen if an upload exceeds the size of that ring buffer? These questions don’t necessarily have one-size-fits all answer, so it really depends on your engine and use case.

One example of the ring buffer setup can be found in my sample framework. It uses locks to ensure thread-safety by serializing allocations from the ring buffer as well as submissions to the COPY queue, but it allows multiple calling threads to copy into staging memory and record copy commands in parallel once they’ve successfully allocated. It also allows for uploads to happen on background streaming threads that are decoupled from frame rendering. This is achieved by having a 1:1 relationship between uploads and submissions, which is definitely a non-optimal choice for uploading many small resources in succession. In that case the uploads can become bottlenecked by the max number of submissions rather than the ring buffer itself, and overhead from queue submission could start to dominate. The async COPY queue submissions are also waited on by the next frame’s DIRECT/COMPUTE queue submissions, which is always safe (the calling code does not need to wait before using the resource to record commands) but usually too conservative. My framework also pairs this with a FastUploader path that’s intended for “critical” per-frame buffer updates. This path uses its own dedicated COPY queue, and batches all copies from a frame into a single command list and COPY queue submission that’s performed once a frame. It also does not use the staging ring buffer, and instead allows calling code to either aquire some temporary per-frame staging memory or provide its own staging buffer.

As an example of another intriguing approach that makes different trade-offs, consider the uploader that Alex Tardiff wrote for his D3D12Lite framework. Rather than using a ring buffer, the uploader allocates linearly from a staging buffer and fits as many uploads as it can until the staging buffer is exhausted. Every frame the uploads that were staged are submitted in a single command list, and the remaining uploads that do not fit are queued to be uploaded in a subsequent frame. This provides great batching behavior, and effectively enforces a fixed budget in terms of upload bandwidth that will consumed every frame. In comparison to my framework, the main tradeoff is that the uploader handles copying into staging memory and recording D3D12 commands (as opposed to the calling code handling it), which requires the memory containing the source data to be persistent until staging memory frees up. Calling code must also potentially wait several frames until the upload completes, although this kind of synchronization is nicely simplified by tying it to the main frame submission fence. Overall I would say it’s very compelling, and could work well for a lot of scenarios!

For another possible example, check out ResourceUploadBatch from DirectXTK. This helper class still allocates a committed resource for use as staging memory, but allows (manually) batching multiple uploads together into a single submission. It also makes use of std::future as a way of tracking when an upload actually completes, which is a nice touch.

What About DirectStorage?

Some of you may have been asking yourself “hey, how does DirectStorage fit into this?” as you were reading through this section. DirectStorage is capable of fully managing the full journey from disk to staging memory to VRAM all on its own, which is pretty awesome. Unfortunately I’m not personally familar with the inner details of how DirectStorage manages these things, nor do I have much experience with using it. I would assume they are biasing for high throughput of upload operations, since the main goal of that API is to improve loading times. Based on the docs, it sounds like they always suballocate staging memory from a single upload buffer and the operation will fail if the request is larger than the configured staging buffer size. The buffer is then flushed periodically after it fills up to trigger the actual upload to VRAM.

Results From My Testing App

While it’s useful to look at things in terms of theoretical max bandwidth, it’s helpful to gather some actual performance numbers from real working hardware. To that end, I created simple D3D12 application that can and gather timing data from a synthetic performance test. The test itself is nothing fancy: a buffer has its contents updated from the CPU, and then a compute shader is dispatched to read from that buffer. There are options to specify what kind of heap the buffer is allocated from, the size of the buffer, the number of compute shader threads, and the access pattern used by those threads. By timing the CPU update, we can gather some data indicating how quickly or slowly the CPU can fill the buffer with different memory pools and caching protocols. Optionally, we can also time how long it takes to read from the buffer in order to get an idea of how badly this will destroy performance. When the app is run, it presents some real-time results along with a simple interface that looks like this:

In the compute shader itself, a starting address is computed for each thread. The shader then runs an N-iteration loop, where each iteration reads and sums a single 16-byte element of the source buffer. The settings for controlling how the addresses are calculated are a bit obtuse, but together they allow setting up some commonly-seen access patterns to see how they fare with different memory and buffer types. The shader itself is quite short so I’ll post it here for reference:

const uint ThreadGroupSize = 256;

[numthreads(ThreadGroupSize, 1, 1)]
void ComputeJob(in uint3 dispatchThreadID : SV_DispatchThreadID,
                in uint3 groupThreadID : SV_GroupThreadID,
                in uint3 groupID : SV_GroupID)
{
    const uint localThreadIndex = groupThreadID.x;
    const uint globalThreadIndex = dispatchThreadID.x;
    const uint groupIndex = groupID.x;

    #if RawBuffer_
        ByteAddressBuffer inputBuffer = RawBufferTable[AppSettings.InputBufferIdx];
    #elif FormattedBuffer_
        Buffer<float4> inputBuffer = InputBuffers[AppSettings.InputBufferIdx];
    #elif StructuredBuffer_
        StructuredBuffer<float4> inputBuffer = InputBuffers[AppSettings.InputBufferIdx];
    #endif

    float sum = 0.0f;

    // Values with a trailing underscore are compile-time constants, allowing full
    // loop unrolling and dispatch-uniform/wave-uniform buffer access where applicable
    uint elemIdx = (ElemsPerThread_ * ThreadGroupSize * GroupElemOffset_ * groupIndex) +
                    (ElemsPerThread_ * ThreadElemOffset_ * localThreadIndex);
    for(uint i = 0; i < ElemsPerThread_; ++i)
    {
        const uint loadIdx = elemIdx % NumInputBufferElems_;

        #if RawBuffer_
            const float4 data = asfloat(inputBuffer.Load4(loadIdx * 16));
        #elif StructuredBuffer_ || FormattedBuffer_
            const float4 data = inputBuffer[loadIdx];
        #elif ConstantBuffer_
            const float4 data = CBuffer.Elems[loadIdx];
        #endif

        sum += dot(data, 1.0f);

        elemIdx += ThreadElemStride_;
    }

    // Should not actually execute, this is just here to keep the
    // compiler from optimizing away the entire shader
    if(sum < 0.0f)
        OutputBuffer.Store(globalThreadIndex * 4, sum);
}

I’ve used my desktop PC for gathering the data below, which has a Nvidia RTX 2080 with an AMD Ryzen 9 5900X CPU. The GPU has a peak bandwidth of 448 GB/s to VRAM, and as a PCIe 3.0 part it has a max bandwidth of about 16 GB/s for reading memory over that bus. Meanwhile the CPU is connected to 32 GB of DDR4 3600, which has a peak bandwidth of around 28 GB/s per channel, resulting in a total of 56 GB/s for my dual-channel configuration.

CPU Write Performance

Here we’re measuring the time it took for a single 5900X core to fill 16 MB buffer. For this test I only used CPU-accessible memory, since non-CPU-accessible VRAM requires additional work to schedule a GPU copy for the upload. What we see here is that the uncached and cached paths are basically identical, meaning that write combining is doing its job for our simple access pattern we’re using (which is just a memcpy). For the special CPU-accessible VRAM path available through NVAPI, the time grows by about 40% reflecting the fact that we are writing over a slow PCI express bus. For SysRAM this works out to about 15.5 GB/s, while for VRAM it’s about 11.6 GB/s.

CPU Read Performance

For this test case, we want to see how quickly (or slowly) the CPU is able to read from CPU-visible GPU memory. For uncached SysRAM it takes about 4ms to read the entire 16 MB buffer, which is about 4x more time than it took to fill the buffer with write combining. Meanwhile for cached SysRAM it’s about 60x faster, no doubt because the buffer is still present in the cache hierarchy after being filled just prior to the reading step. In real-world scenarios we might expect read performance to be worse for this case since the data may not be present in cache. As for CPU-visible VRAM, we have officially entered “oh no, what I have done?” territory at a whopping 26.66ms. Remember kids: friends don’t let friends read from VRAM on the CPU!

GPU Read Performance, Normal Access

For this test, we are now looking at GPU performance by measuring how long it takes for the GPU to complete the single big compute shader dispatch that reads from the buffer. While this doesn’t serve as a comprehensive bandwidth or cache performance test, it tells us roughly the overall effect that the GPU memory pool will have on GPU performance which is primarily what we are interested in. To gather the data, the test app was configured to use a 64 MB buffer and launch 16 * 1024 * 256 = 4M threads that each access a unique contiguous 16-byte element from the input buffer. This effectively reads from the entire buffer, giving us a rough idea of low long it takes to read the whole thing.

As you would expect for a dedicated GPU, there’s a significant disparity between reading from SysRAM (L0) and VRAM (L1). Reading from SysRAM causes the compute shader to take about ~32x longer to execute relative to VRAM, reflecting the significantly lower theoretical bandwidth for reading over PCIe vs VRAM. This is very close to what we would expect given that my RTX 2080 is rated at 448 GB/s for bandwidth to VRAM, and PCIe 3.0 tops out at 16 GB/s.

To help confirm things, let’s invert these numbers to estimate what the minimum required bandwidth would be in order for the dispatch to complete in the reported time:

While our calculated numbers fall a bit short of the expected theoretical bandwidth, this is probably expected given that our shader may not be hitting the exact conditions required for achieving peak bandwidth on the hardware. However they still clearly reflect the extreme bandwidth deficit that’s experienced when reading from SysRAM.

GPU Read Performance, Non-Coalesced Access

For the previous test, each thread accessed a buffer element that was exactly contiguous with the element being accessed by neighboring threads. GPUs can typically accelerate this case by issuing coalesced memory requests. This pattern is also good in general for cache efficiency: all data from a cache line ends up getting utilized, as opposed to unused neighboring data getting inadvertently pulled into cache. In this test we use a less efficient access pattern where each thread reads a 16 byte element that is 64 bytes away from what their neighboring threads read. If the cache line is 64 bytes, this means that 48 of those 64 bytes are wasted which will slow things down considerably. This is roughly reflected in the test numbers, where the dispatch is about 3-4x slower for all memory types when compared with the previous test case.

GPU Read Performance, Various Buffer Sizes

Earlier in the article we discussed how the size of the buffer being read is something that should factor into our decision of whether to place the buffer in VRAM or SysRAM. The general rule of thumb is “small” buffers are ok being in SysRAM, particularly if they are only read once and/or many threads end up reading the same data from that buffer (constant buffers typically fit this pattern). Let’s try to verify that assumption by running a test where we run a smaller dispatch of 1M threads that access various sizes of smaller buffers. The setup is such that each thread will mod their address with the buffer size, effectively causing the reads to “wrap around”. Therefore if 1M threads each reads a 16-byte element from the input buffer and that buffer is 512 KB, we would expect each byte of that buffer to be read (16 * 1024 * 1024) / (512 * 1024) = 32 times. Let’s now have a look at the results:

At the leftmost end of the chart we see that the gap between SysRAM and VRAM is almost 0 for a tiny 64 KB buffer. This suggests that at typical constant buffer sizes there is little benefit to VRAM, and the speed of the GPU’s cache hierarchies will dominate. As we increase in size though see the SysRAM times pull away from the VRAM times, the latter essentially appearing to have a flat slope at these scales. This suggests that the limited bandwidth of PCIe is bottlenecking the SysRAM dispatch more and more as the buffer size increases, but for VRAM we are likely bottlenecked elsewhere (perhaps on the output writes). Let’s zoom out a bit more and see what we get for 1 to 16 MB buffer sizes:

Up to 4 MB we continue our trend of SysRAM pulling away from the VRAM results, with SysRAM being about 10x slower at that point. But then at 5 MB things jump way up to 1.285 milliseconds, and it stays up there! What gives? What’s so special about 4 MB that things take a nosedive after that point? Well it turns out that 4 MB is the size of the L2 cache on my RTX 2080! Essentially at <= 4 MB the entire buffer was able to fit in L2 cache, which meant that the dispatch time roughly reflected the amount of time it would require to fill that <= 4 MB from memory. If you compute the minimum required bandwidth (buffer size divided by execution time) for these < 4 MB timings, they come out to about 10 GB/s which is fairly close to the 12.5 GB/s I measured from SysRAM in other tests. Meanwhile for the >= 5 MB data, that 1.285 millisecond number that repeats for each test is exactly what you get if you calculate how long it takes to read a 16 MB buffer with 12.5 GB/s bandwidth. 16 MB is what you get if you sum the size of all reads from every thread regardless of the buffer size (1M threads, each reading 16 bytes), which means we’re basically pegged at 12.5 GB/s “effective” bandwidth for these measurements. Since our data set no longer fits comfortably in L2 cache, we can’t count on the data to still be in cache the Nth time an element is accessed by a later thread.

Conclusion

Having to manage separate physical memory pools for GPU programming can be rather complicated, and there are definitely performance pitfalls that we need to watch out for. Integrated GPUs that share a memory pool with the CPU can be simpler, but we still need to interact with them using APIs that accomodate dedicated video cards. In most cases we also need to run the same code on both dedicated and integrated GPUs, which means we need to try to handle both cases gracefully. To minimize the pain, I would recommend the following high-level practices:

• For dedicated video cards:
  • Make sure you’re using VRAM for all textures as well as buffers that do not need to be updated from the CPU. The bandwidth difference between SysRAM and VRAM can be nearly 100x on high-end video cards.
  • Try to use a COPY/Transfer queue to initialize VRAM resources asynchronously to your graphics work.
  • For larger buffers that need to be updated from the CPU, consider allocating them in VRAM and using a staging buffer + COPY/Transfer queue to update them. Consider using two queues if you have both background uploads and critical per-frame uploads.
  • On APIs and systems that support it (Vulkan and NVAPI), consider allocating large CPU-updated buffers in CPU-writable VRAM so that you can update them directly. Watch out for potentially slower CPU write performance. Do not read ever from this memory on the CPU!
• For integrated/mobile GPUs:
  • Skip using the COPY/transfer queue to initialize or update buffers, you can just make them CPU-writable with no GPU performance penalty.
  • With D3D12 you can still use DEFAULT heaps if you like, but it’s still going to be located in SysRAM.
• In general:
  • Be careful not to read from memory that is uncached for the CPU, it can be quite slow!
  • Try to stick to a simple memcpy when updating GPU memory from the CPU. This will ensure that you play nicely with write combining by writing to the buffer sequentially, and it will also avoid implicit reads that can sometimes occur (for example by summing a value located in GPU memory).

Finally, I’d like to thank the following people for reviewing this post early and providing great feedback:

• Adam Sawicki (extra-special thanks for the suggestion to try using two COPY queues)
• Alex Tardiff
• Jeremy Ong
• Pete Brubaker
• Jiayin Cao
• Brian Collins

That’s all for now, good luck with wrangling those GPUs and their memory pools!

Update 01/04/2024: I’ve uploaded the memory bandwidth testing app to a public GitHub repo in case anybody else would like to try it out. I also updated it to use GPU Upload Heaps from the preview agility SDK, which should be available if you’re running on a system with ReBAR support.

If people are making memes about an issue then you know it’s pretty bad, and the shader permutation situation can indeed be pretty bad. It’s frustrating because collectively we have put a lot of effort into fixing or mitigating the problem, and yet it still hasn’t really gone away. It’s also frustrating because I’ve seen people trivialize it with suggestions like “just don’t make so many shaders!” or “just use branching!” which I think misses a lot of the nuance and context of the problem. To help with those latter two, I thought I would spend some time talking about why permutations continue to be a problem and also give a little bit of history on how we got to this point. In part 2 I’ll then talk a bit about the various options we have available that can help us to get out of this mess, and expand on what’s good and bad about them.

What’s A Few Shaders Among Friends?

To start, let’s go over a bit of background on the problem. When I talk about “shader permutations” (also sometimes called “shader variants”), what I’m referring to is taking a pile of shader code and compiling it N times with different options. In most cases these permutations are tied directly to features that are supported by the shader, often by writing the code in an “uber-shader” style with many different features that can be turned on and off independently. As a super-simple example let’s look at a little HLSL pixel shader that uses preprocessor macros to turn different features on and off:

// ENABLE_NORMAL_MAP, ENABLE_EMISSIVE_MAP, ENABLE_AO_MAPPING are
// macros whose values are defined via compiler command line options
static const bool EnableNormalMap = ENABLE_NORMAL_MAP;
static const bool EnableEmissiveMap = ENABLE_EMISSIVE_MAP;
static const bool EnableAOMap = ENABLE_AO_MAPPING;

float4 PSMain(in PSInput input) : SV_Target0
{
    float3 normal = normalize(input.VtxNormal);
    if(EnableNormalMap)
        normal = ApplyNormalMap(normal, input.TangentFrame, input.UV);

    float3 albedo = input.Albedo;
    float3 ambientAlbedo = albedo;
    if(EnableAOMap)
        ambientAlbedo *= SampleAOMap(input.UV);

    float3 lighting = CalcLighting(albedo, ambientAlbedo, normal);

    if(EnableEmissiveMap)
        lighting += SampleEmissiveMap(input.UV);

    return float4(lighting, 1.0f);
}

We have 3 features that can be enabled or disabled here, and if all of those features are independent we need to compile 2^3 = 8 permutations. You can hopefully see the problem already: each new feature we add means we double the permutation count! We can also generalize this a bit to non-binary features that have more than 2 different states, in which case the formular for computing the total number of permtuations looks something like this:

$$ NumPermutations = NumStates_{featureA} * NumStates_{featureB} * NumStates_{featureC} … $$

With that kind of exponential growth it’s not hard to imagine how the total permutation count can get up into the thousands or millions as you add more and more features. That’s pretty obviously not great when you start thinking about the impact this could have on developer workflow. As you add more and more features and more and more shader code to implement those features, you get this kind of cascading effect where you’re asking the compiler to churn through this big monolith shader program and over and over again. That’s a lot of wasted work of parsing and optimizing the same code over and over again, and unless you’ve got threadrippers and/or a local network of machines to distribute compilation across it can turn into hours of time for each change to the shader code. This is really not where you want to be if you’re trying to rapidly iterate on optimizations or new features, and you especially don’t want to pass that on to content creators that need to cook all of the necessary data for the game to run.

Speaking of content creators, yet another source of unbounded shader growth comes from engines that allow users to create their own custom shaders that are bound to materials. Some expose this by allowing users to hand-author shaders in a shader language, while others allow you to create “shader graphs” using a visual node-based interface. Depending on how the renderer is setup, these can potentially serve as an additional multiplier on top of the permutation count that comes from the engine itself. For example a shader graph might only allow you to generate shading parameters (such as color, normals, roughness, etc.) while the engine combines that with hand-written shader permutations that implement the actual lighting calculations. If you have 100 permutations for the lighting phase and 100 shader graphs for a project…well guess what, you now have 10k shaders.

The thing is that the time spent compiling isn’t even the entire problem, it’s just one part of it. Compiling all of those shaders can start to generate a lot of binary data, especially if you generate debug info. You may start running into issues with storing all of that data, or loading it and keeping it in memory at runtime. But wait, it gets worse! Recent PC and mobile graphics APIs like D3D and Vulkan actually implement a two-step compilation pipeline. Some people are surprised to learn this when they start with graphics, but offline shader compilers like dxc, fxc, and glslc actually generate some kind of intermediate bytecode rather than machine code that can be run on the GPU shader cores. GPUs actually have wildly different ISAs under the hood, with the instructions often changing from one hardware generation to the next even within the same vendor. To handle this setup, the driver will employ some kind of JIT compiler that takes your DXBC, DXIL or SPIR-V and converts it into the final ISA that the shader core can execute. The full pipeline typically goes something like this:

• The shader source code is compiled offline by a shader compiler, such as fxc, dxc, or glslc
  • The output of this step is a hardware-agnostic intermediate bytecode format, either DXBC (fxc), DXIL (dxc), or SPIR-V (dxc or glslc)
• The compiled bytecode is loaded into memory by the engine
• The engine creates a Pipeline State Object (PSO) by passing the required state as well as the compiled bytecode for any required shader stages
  • During this step, the driver runs its own JIT compiler that converts the bytecode into its own hardware-specific ISA that runs on the physical shader cores
• The engine binds a PSO to a command buffer, and issues draw/dispatch commands that used the shaders referenced by the PSO
• The command buffer is submitted to the GPU, where the shader program actually gets executed

It’s easy to focus on that first step when thinking about the downsides of permutation explosion, but in reality those other remaining steps can all be affected as well. Having more permutations often means invoking that JIT compile step more times, which can really add up. Keep in mind this is usually not a simple translation that’s happening here: most drivers will boot up a full-fledged optimizing compiler (often LLVM) and then run your bytecode through that in order to produce the final ISA. This can really catch you off guard if you’re just expecting to quickly create all your PSOs in the background as you stream in a new chunk of the world: you have to anticipate that creating those PSOs is going to consume considerable time and CPU resources which in turn affects your your effective load times. Even worse, the amount of time can vary depending on the vendor, exact GPU architecture, the exact driver version, the other states and shader stages you’ve provided in the PSO description, and even whether or not this is the first time creating that exact PSO on the system!¹ It’s true that D3D12 and Vulkan both provide mechanisms for manual caching of PSOs by the application, however these APIs tend to be complex and still don’t help your “first boot” loading times. Valve has even gone so far as to integrate a distributed caching system into Steam that works for OpenGL and Vulkan, which actually says a lot about how bad of a problem this can be!

Even once you’ve got your PSOs created, the downsides of unchecked permutation explosion continue. To use those PSOs, you need to bind them prior to a draw or dispatch on a command buffer. This binding step consumes CPU time: issuing many draws with a single PSO can be significantly cheaper than switching PSO between each draw. The end result is that the CPU time required for generating command buffers goes up as you increase your shader/PSO count. Having many PSOs can also prevent you from using techiniques such as instancing or batching to combine things into a single draw or dispatch, which also increases CPU time. But that’s not all: unless you’re using Nvidia-specific extensions, both D3D12 and Vulkan are incapable of changing PSOs from the GPU in the middle of a ExecuteIndirect or DrawIndirect/DispatchIndirect call.² This means that if you’re looking to use fancy GPU-driven techniques for culling and computing LOD on the GPU, you will need at minimum a number of ExecuteIndirect/DrawIndirect calls that is equal to the number of PSOs used by those draws. Too many PSOs/shaders can also pose issues for ray tracing, since large shader tables can lead to divergence and inefficient hardware utilization.

Finally, there’s the execution of the shader program on the GPU itself. Modern GPUs have a lot of tricks in them to keep performance acceptable even in the presence of many draw calls, and many shader switches. However, in the general case they are still going to achieve higher performance when you can feed them larger batches of work to do that don’t require switching states and shaders. As a consequence, your GPU performance can be reduced as a result of switching shaders too often. The same goes for your CPU performance: switching shaders/PSOs often involves more expensive operations in the driver that are required to reconfigure the necessary state bits on the GPU, and also handle downstream impacts of switching shaders (such as the low-level representation of the resource bindings used by the shaders). Therefore if you really want to burn through lots of draw calls in a few milliseconds of CPU time, you’ll want to aim for as few PSOs and PSO switches as you can.

The Monolithic Shader Program

In my earlier example where I shared a simple pixel shader with permutations, you may have noticed how each permutation required compiling the entire shader program all the way through. This idea is probably a bit weird if you’re not used to shader programming: after all, that’s not really how we do things for our CPU code. For a language like C or C++ we will typically compile much more modular sections of code into separate compilation units, and then link everything together through function calls and global variables. Or we might go even further and compile our code into completely separate executables (DLLs or SOs) that can be dynamically loaded and called into at runtime, giving us further flexibility in how we structure things. Why can’t we do things like that with shaders? Why do shaders have to be so monolithic instead of being modular? To answer that, let’s step back in time a bit.

If you look back to the start of shader programs, they started out small. Not in the metaphorical sense of “small ideas and limited use cases” (which is true, but not what I’m talking about here) but in the very literal sense of “they were very very short”. The original Shader Model 1.0 introduced with D3D8 had a hard limit of 128 instructions for vertex shaders, and the pixel shaders were basically just directly exposing the (very limited) register combiner and texture functionality of the hardware of that time. At that point you would have been crazy to think of a shader as anything other than a tiny little self-contained program or routine: they were only a handful of instructions! It wasn’t even until D3D9 and later that people authored them in an actual high-level shader language instead of hand-writing “assembly”.

The thing about shaders is that we never really shook that kind of “small self-contained program” mentality. For years we basically just extended it further and further: first with higher instruction counts, then with more features, then with fancier compilers. But all along the way we kept the model where a shader program is monolithic and completely self-contained. This is not entirely just legacy decisions either: along the way we benefited from some of the halo effects that come from this really simple model. Shaders never needed a linker since there was nothing to link, and there was never a need to have a standard ABI for calling between separately-compiled binaries. In fact there’s historically been no function calls at all within a compiled shader program! Why do that when you can instead just ruthlessly rip through the shader code and inline everything until you’re left with nothing but a linear stream of instructions. If you’re coming from a C++ background you’re probably used to seeing the optimizer pull some heroics in certain places, but shaders can be on another level due to the simplicity of the language: after all there’s no constructors, no side effects, and general memory writes weren’t even added until Shader Model 5.0! Let’s look at another simple HLSL pixel shader to show you what I mean by this:

float3 ComputeLighting(in float3 albedo, in float3 normal, in Light light)
{
    return saturate(dot(normal, light.Dir)) * albedo * light.Color;
}

float PSMain(in float3 albedo : ALBEDO, in float3 normal : NORMAL) : SV_Target0
{
    return ComputeLighting(albedo, normalize(normal), CBuffer.Light).x;
}

If you were expecting the compiler to emit instructions for ComputeLighting independently, you would probably expect it to perform the chain of multiplication using float3 vectors, resulting in 6 total multiplications after the saturated dot product is computed. You would also expect PSMain to receive that float3 result, ignore the Y/Z components, and then return the X component. In reality shader compilers have never worked this way: instead they would fully inline/flatten the function call and dead-strip like their life depended on it, resulting in generated assembly that was more equivalent to this code:

float PSMain(in float3 albedo : ALBEDO, in float3 normal : NORMAL) : SV_Target0
{
    return saturate(dot(normal, light.Dir)) * albedo.x * light.Color.x;
}

This is not really magic or anything, and you can see a C or C++ compiler do the same thing in similar places. But with shaders it’s different because you’re almost guaranteed to get that optimized and inlined result due to the simplicity of the programming model. This is really really great if you once again go back to the earlier SM 2.0 era of shaders: you really needed to penny pinch your instructions to avoid hitting instruction limits, and when you were running these programs across thousands of pixels each little add or multiply could really add up quickly. The “flatten everything!” style of code-gen also suited earlier programmable GPUs particularly well, since they either couldn’t do dynamic flow control or they were very very bad at it. Thus it was to your advantage to be able to take shader code written in a branchy or loopy style and convert that into something completely flattened and unrolled whereever possible.

Modern GPUs can still benefit from these kinds of agressive micro-optimizations. Sure a recent GPU can do an awful lot more MADDs per second than a GPU from 2005, but extra math can still add up when you’re doing it for millions of pixels every frame. They’re also significantly better at actual branching and loops then they used to be (at least as long as the flow control is “uniform”, which means that all threads within a wave/subgroup take the same path), but flow control is never 100% free. Once again a few extra instructions for iterating a loop control variable and checking the condition can add up if it’s done many times, and you might be better off letting the compiler unroll that for you. Flow control can also inhibit certain kinds of optimizations that compilers like to do where they overlap memory or math instructions from unrelated code in order to extract additional instruction level parallelism (or ILP for short), which can potentially decrease the latency of a wave in isolation.

Even if you ignore the optimization aspect, there’s still the fact that GPU shader cores are traditionally not really setup to handle true function calls or dynamic dispatch. For various reasons, these shader cores tend to use a simple register-only SIMD execution model with no true stack. This naturally makes it trickier to have the concept of a standard ABI, and things get especially tricky if you want to support diverging function calls within a wave. Shader cores also typically use a model where waves must statically allocate all of the registers they will ever need for the entire duration of the program, as opposed to a more dynamic model that might involve spilling to the stack. Keeping that register count low is good for occupancy (which is yet another reason why aggressive micro-optimization can help), and so it’s important you only allocate what you truly need. This means you have a problem to solve if you want to let a program jump out to arbitrary code: the wave has to statically allocate sufficient registers before it can call into some function. Even if you know all of the possible places that the shader might jump to, everybody will be limited to the occupancy of the worst-case target. ³ These are not necessarily unsolveable problems, but they help explain why it’s been easier not to rock the boat and just stick with a monolithic shader program.

Are We Too Focused On Micro-Optimization?

If we pull back the camera a bit, one of the broad themes we see in the rush to over-permute our shaders is a focus on micro-optimization. If your goals include “I want to let the compiler generate its idea of the best possible shader program for a given feature set”, then you’re pretty much guaranteed to end up with permutation hell once you have a non-trivial feature set. In a way this is pretty natural: gating a new feature behind a permutation lets you look at things more in isolation, and give you peace of mind that whatever crazy thing you do in that shader is going to have zero impact on the 99% of materials that don’t use it. In my experience that freedom can significantly lower the barrier of entry for new features: I don’t mind saying “yes” to an artist’s feature request if I know it’s wrapped up in a preprocessor macro. This kind of thinking can also naturally come out of situations where you’re optimizing some existing problematic material that your artists came up with, where you end up wanting to add a permutation that unrolls some loop or strips out a pointless texture fetch.

Broadly speaking, I think that breaking out of permutation jail requires a shift in mindset. Instead of looking at things one material at a time and focusing on lost cycles here and there, you need to pull back and look at GPU performance holistically. No wave of a shader program runs in isolation: we need to keep occupancy high and I$ pressure low while also reducing stalls from having too many shader switches or draw batches. We can’t overlook techniques that require spilling to memory or splitting things up into multiple passes if in the aggregate those techniques end up boosting utilization and throughput. Perhaps it’s ok if our simplest material is a little bit slower if we can ensure that all of our shaders end up faster overall. We also need to think more globally about the effects of our shaders on the full development pipeline: what’s the cost to productivity if we add another permutation and double the shader count?

It’s funny too because the micro-heavy approach is in some ways very counter-productive to the goals of getting “better” generated programs. Having so many permutations can actually make it much more difficult for a programmer to do actual hand-optimization of the shader source code, both because of the added iteration times but also because there’s just too many possible permutations to focus on! Some of the heavy optimizations performed by compilers can also be surprisingly bad for overall performance. In particular aggressive unrolling and overlapping of instructions tends to eat up registers very quickly, which can result in reduced occupancy. In other words these optimizations might have a small reduction to the latency of a single wave but can be actively harmful to the GPU’s efforts to boost overall throughput by cycling through different active waves. This can be quite frustrating to deal with, since sometimes it feels like you have to trick the compiler into generating code that uses fewer registers. You can’t even totally blame the compiler really since it doesn’t really have the full picture: it doesn’t know what else will be happening on the GPU at the time the shader executes, and it also doesn’t know the expected latency of a memory operation (maybe it was in cache, or maybe it wasn’t!).

With all that said, it’s a lot easier to just say “take a global approach to GPU performance” than it is to pull it off in practice. You likely need excellent tooling and test cases to make sure you’re getting data that isn’t misleading, and you also need to analyze quite a few stats to make informed decisions. Think for a moment about what you might need to consider in terms of your global GPU performance when deciding whether or not to split off another permutation of a shader, or perhaps unroll a loop in that shader. Off the top of my head, I can come up with a non-comprehensive list of considerations that’s pretty long:

• Time Spent Compiling From Source
  • How fast or slow is the compiler itself?
    • Do various compile options affect compilation time?
    • Does the speed change over time with newer versions?
  • Can we compile on multiple threads?
    • If so, how many cores are available on the local machine?
  • Can we distribute across multiple machines?
    • If so, how many remote machines are typically available?
    • What is the latency of sending over code and spinning up compilation on a remote machine?
    • Are those remote machines available to people who aren’t on in a central office on the same LAN?
  • How does compilation and optimization time scale with shader code length and number of emitted instructions?
  • How much does shader compilation negatively effect the productivity of graphics programmers?
    • Can programmers compile only a subset of permutations when testing a change, or do they need to compile all of them?
    • How quick is a change + compile + measure loop for optimizing shaders?
  • How much does shader compilation affect the productivity of non-programmers?
    • Do material artists need to recompile many shaders when creating or editing materials?
    • Are compilation results cached across the team, or does everybody have to recompile the same shaders?
• Time Spent Generating Hardware ISA
  • Can this happen ahead-of-time (consoles), or do we need to JIT compile at runtime (PC)?
  • How fast is the JIT compile?
    • Does that speed change over time or with different drivers?
    • What kind of caching is available?
• Number of shaders and PSOs loaded at runtime
  • How much memory do they consume? Can you even measure it?
  • How quickly can your code, the runtime, and the driver switch PSOs in terms of CPU time?
    • Does that time vary based on the bindings used for the shader?
  • How does switching shaders affect GPU performance?
• Length and size of individual shaders
  • How does this affect runtime memory consumption?
  • How does it affect PSO creation time?
  • Does it increase pressure on the shader instruction cache, degrading performance?
  • Does having long shaders that sample many textures increase L1/L2 cache thrashing, reducing performance?
  • Does having a longer shader give the compiler more opportunities to dead-strip unnecessary instructions, and interleave memory and math operations?
• Number of individual passes required to render a frame
  • How much memory bandwidth do we consume by writing out intermediate results vs. keeping them resident in registers?
• Register pressure and occupancy
  • How many registers do our shaders consume, and what is the resulting occupancy?
  • Is lower occupancy reducing performance by not giving the hardware the ability to hide long latency memory access?
  • Is high occupancy reducing performance by causing L1 cache thrashing?
  • Can register usage and occupancy be calculated with offline tools, and are there any tools available for analyzing and optimizing register usage?
• ILP of the shader programs
  • Can the shader program hide latency of memory access by executing ALU-heavy instructions while waiting for memory operations to complete?
  • Can multiple ALU instructions execute in a pipelined manner, increasing IPC?
  • How many registers are utilized for latency hiding and pipelining?
  • Does hiding latency require unrolling loops, increasing the size and register usage of the shader program?
• Number of features required for creating content
  • What functionality do artists need to create content quickly and at a satisfying quality level?
  • What features are needed for gameplay functionality? For example, changing colors based on team or building certain kinds of UI into the material.
  • Do content creators have the ability to write their own shaders for game-specific purposes, either by hand-writing HLSL or using a shader graph?

It’s…a lot. This list doesn’t even consider the non-technical aspects of working with and art team to explain to them the trade-offs in terms of their workflow and what kind of features they can use. Don’t let anybody ever tell you that this stuff is easy!

Coming Up in Part 2

In the final part of this series, I’m going to look at a list of techniques we can employ that potentially help us to dig ourselves at out of the hole that we created.

If you’ve read the previous article then you hopefully have a decent understanding of how shader permutations ended up being such a common issue. The good news is that there is some hope for the future: if we look across recent game releases as well as the features available in the latest GPUs and APIs, we do see some promising avenues for digging ourselves out of our self-imposed avalanche of compiled bytecode. In my opinion nothing I’m going to mention here is a silver bullet on its own: each technique comes with a set of trade-offs to be carefully evaluated in the context of an engine and the games that run on it. Regardless, it’s inspiring to see smart and resourceful people come up with clever approaches that help to sidestep some of the issues that I’ve brought up the previous article.

Only Compile What You Need

This is the simplest, oldest, and perhaps least-effective way to reduce permutation counts. The general idea is that out of your 2^N possible permutations, some subset of them are either redundant, invalid, or will never actually get used. Therefore you will reduce the set of shaders that you need to compile and load if you can strip out the unnecessary permutations. In many cases the reduction in shader permutation count can be substantial, and can be the difference between “completely untenable” and “we can ship this”. Ideally this process is something you would do offline, perhaps as part of a content processing pipeline that has knowledge of what meshes and materials are going to be used in each scene. But there have also been games/engines that have done it at runtime, essentially deferring compilation of a permutation until the scene is loaded. Either way there are some pretty obvious downsides:

• Determining your set of shaders offline requires your offline process to have a pretty complete understanding of both the content and how it interacts with your shader pipeline. Making changes to how that works may also require you to recompile and invoke the content processing pipeline again, as opposed to recompiling the runtime code and running the app again
• Offline approaches may make editors and other tooling more complicated, since you now have to deal with an on-the-fly combination of mesh + material
• On a related note, some engines are setup where they treat their shaders more like code and less like content. For instance, they might want to compile shaders using the same build system used for C++ code since it already handles dependencies and includes correctly. Moving to a system where the shaders are compiled as part of the content pipeline can potentially be a large shift.
• If you wait until runtime to compile your shaders, you now have to either make the user wait for compilation to complete or do something else to hide the compilation time. This might involve having QA generate a cache of shaders to ship with the game, or it might even involve using a slow and generic shader until your specialized permutation finishes compiling in the background. Platforms that don’t allow you to invoke the shader compiler at runtime can also make this approach a non-starter, at least without some kind of two-step process that discovers the shaders and then compiles an offline cache to ship with the app.
• The amount of shaders that you generate is content-dependent, and the count could vary wildly depending on the scene/game/material setup/etc.
• In the worst case this degenerates to compiling your full set of permutations, except that you may end up with a more complicated pipeline for generating those permutations.
• It doesn’t do anything to reduce your loaded shader/PSO count at runtime. If you need to rely on a small number of PSOs to implement some technique (for example, GPU-driven rendering) then this approach won’t help you.

For our engine at Ready At Dawn, we have used the offline variant of this approach with some success. When our content processing pipeline encounters a mesh, it looks at the assigned material as well as properties of the mesh itself to figure out the final shader permutation that will be required. A request to compile that shader is then kicked off, and the final bytecode is ready to load in the game. In our case the reduction is quite substantial: just our “generic” base material + shader set supports an unfathomable number of possible permutations, and the vast majority go unused. For the entirety Lone Echo 2 we end up compiling around 5000 pixel shaders out of a possible set of millions, and typically have around 900-1000 PSOs loaded at runtime (with about 50% of them being used in any particular frame). Unfortunately the full set of shaders to compile is still quite large, so a global change can take quite some time to churn through before the full game is rebuilt. To mitigate this we make use of distributed shader compilation that is run on idle machines. One silver lining is that it’s quite simple to build only the set of shaders required for a particular area of the game, since the shaders are compiled as part of processing a particular chunk of the game world. This can make iteration times lower for global shader changes, but still quite far from instant. To handle tooling issues where the mesh + material combination is not known at runtime, we use “generic” un-permuted versions of our shaders that branch on constant buffer values for all features. As a bonus these shaders also support full real-time parameter editing, and have additional dev-only features for debugging. These shaders are huge though since they support every possible feature, and can be quite slow!

Run-Time Specialization

Vulkan and Metal both support an interesting feature called specialization constants, or “spec constants” for short (Metal calls the feature “function specialization”). The basic idea goes like this:

• You compile your shader with a global uniform value (basically like a value in a constant buffer) that’s used in your shader code
• When creating the PSO, you pass a value for that uniform that will be constant for all draws and dispatches using the PSO
• The driver somehow ensures that the value you passed is used in the shader program. This might include:
  • Treating the value as a “push constant”, basically a small uniform/constant buffer that gets set from the command buffer
  • Patching a value into the compiled intermediate bytecode or vendor-specific ISA
  • Treating the value as a compile-time constant when the driver does its JIT compile and performing full optimizations (including constant folding and dead code elimination) based on that value

It’s a pretty neat concept, since it potentially lets you avoid having to do a lot of your own compiling and instead rely on the driver’s backend compiler to do a lot of the work for you. If you have a lot of specializations it won’t necessarily allow you to reduce your PSO count, but it can be pretty nice in a similar way to the “Compile What You Need” strategy if you don’t use your full set of possible permutations at runtime. A good example is using them to implement quality levels based on user settings: you’re not going to need both the low-quality and high-quality PSOs at once, so you can use a spec constant to choose when you create the PSO. It should be noted though that spec constants also share one of the main issues with “Compile What You Need”: it won’t reduce your runtime PSO count. You can potentially ship and load fewer shader binaries which is an improvement, but you can still run into the same problems that PSOs can cause with batching and ray tracing.

The other main issue with this approach (aside from lack of support in D3D12 and consoles) is that you don’t really know which approach the driver is going to take when it handles your spec constant. Sure it makes sense that it will dead-strip as much as it can since it’s doing that as part of its JIT compile anyway, but drivers have complicated trade-offs to deal with when it comes to doing their back-end compilation. If they take too long when initially creating your PSO then games can have slow loading times, and both devs and customers will complain. Doing further optimization on a background thread is possible, but drivers are limited in how much of the CPU they can use for this (particularly in D3D12) and it surely makes things more complicated for both the driver and developer. The worst part is that you could end up having wildly varying results (and thefore performance) based on the vendor and driver version. Sure in some ways you just have to be zen about hoping that PC drivers will do whatever they need to in the background to make things fast, but do you really want to set up your entire shader pipeline on the assumption that you’ll get as good results as if you compiled the shader permutations beforehand? Do you really want to do that on mobile where drivers are even more constrained in their CPU usage?

One interesting variant on this approach could be to have the app manually invoke the shader compiler (or some slimmed-down version of it) to patch in the spec constant and optimize based on its value. I’m not sure if this would be significantly faster than invoking the full compiler toolchain again, but perhaps it could be speedier since you wouldn’t need to parse anything. This approach would work on any API as long as the compiler can run on-target, and you would have the peace of mind of knowing that the optimization and dead-stripping is for sure happening. It would also put any trade-offs regarding background compilation in the hands of the app developers, which would certainly be more consistent with the overall spirit of the “explicit” APIs.

Cached Material Evaluation

A lot of what goes into our shader permutations is typically related to combining and building shading parameters in various ways. It’s quite common to build the “look” of one material by sampling multiple textures with different UV values, either to add varying levels of detail or to create meta-materials from simpler “building blocks”. A broad feature set for doing these operations can naturally result in a lot of shader code, which can in turn lead to shader permutations. Or alternatively an engine might allow technical artists to completely define their own logic by authoring shader graphs, in which case there are potentially unlimited numbers of compiled shaders that could be generated. If you can somehow pull this process out of your pixel shader, then you can perhaps reduce the number of shader variants that are needed for your draws. There are a few ways you might do this:

• Use Substance Designer or a custom material pipeline to do offline compositing and generating of your textures
  • Even with this you generally still want to combine tiling maps together at runtime since “flattening” them would consume considerably more memory
• Use an offline system combined with runtime virtual texturing to full generate a unique set of composited textures for all surfaces in the game (basically the Megatexture approach)
  • There are plenty of well-documented issues with this approach, such as disk space, streaming/transcoding cost, lack of fine texture detail, etc.
• Use a runtime virtual texturing system that composites/generates pages on-the-fly
  • More complex, you may still need permutations for the process that does the VT page generation

The end goal of these techniques is to end up with simpler pixel shaders that can just sample one set of textures without needing to do anything fancy with them. This won’t help you for the portions of the shader that don’t deal with generating shading parameters, but it can potentially cut things down quite a bit.

Replace Permutations with Branching and Looping

On modern GPUs flow control is no longer the problem that it used to be. Having divergent branching and looping within a wave is still a problem due to the SIMD execution model used on GPUs, but if all threads go down the same path it can be quite efficient. Therefore if you’ve got a material flag that enables a feature for an entire draw call, it could make a lot of sense to stick that value in a uniform/constant buffer and branch on it in the shader at runtime. This kind of branching is usually referred to as “uniform branching” or “static branching” since it guarantees that you’re not going to have any divergence. The idea here is that we can rethink a lot of the permutation decisions that we made 10 years ago and (hopefully) reduce our total shader count after sprinkling in a lot of these static branches. If that branch is totally uniform then the shader process will legitimately skip over the instructions in the branch when the flag is disabled, and it may not even cost very many cycles to do this if the code has a dedicated scalar unit for performing those operations.

The poster child for this approach is the past two Doom games from id: Doom 2016 and Doom: Eternal. In their presentations they mention that they use a forward-rendering ubershader that only results in ~100 shaders and ~350 PSOs. This is of course more than a handful but a distributed compilation system can probably tear through that many shaders in no time, making it very attractive from an iteration point of view. Personally I get quite jealous when thinking of how much simpler my life would be with only that many shaders to deal with, and I really doubt I’m the only one! Their games run really well and look great so it’s a pretty strong proof-of-concept that this approach can work.

With that said, I personally still think it can be quite difficult to achieve a low permutation count in the general case. In part 1 we discussed why this is tricky to get right, so I won’t go into detail again. But the main gist of it is that it can be difficult to know where you should leave a branch or loop and where you should permute/unroll. On consoles you can make pretty informed decisions by looking at a single GPU in isolation (and make use of the excellent tooling), but on PC and mobile you are in a much much tougher spot. But on the positive side having fewer permutations means fewer total programs to optimize by hand, which can potentially pay off in big ways. In particular it’s a lot more reasonable to try to make sure that all of your permutations have a reasonable register count and occupancy…at least assuming that you have the tools available to determine these things. There are also some things that really do require permutations since they are static properties of the shader itself, and thus can’t be replaced with branching. In particular this includes the set of inputs or outputs of a shader (interpolants, render targets), usage of discard, per-sample execution, and forced early-z.

I think what would truly help in using this approach is to have a lot of discipline and focus in your material/shader feature set. The less features you need to support, the easier it’s going to be to make sensible decisions about where you should permute. I would imagine you would have to get used to saying “no” to a lot of feature requests, since every feature added can potentially affect the performance of your ordinary baseline materials unless it’s gated behind a permutation. For those of us with very broad material feature sets it seems difficult to get to that point without being very disruptive to existing workflows, and/or without giving up things that make a significant contribution to the look of a game.

Another thing that could really help is good automated processes for helping to figure out which permutations are truly worth it vs. just leaving in a branch. Unfortunately the testing space for this is quite large: you can have N different materials and M different video cards with a huge amount of permutation options, so it’s not feasible to test exhaustively. But perhaps reasonable subsets can be derived that give a decent level of confidence.

One last thing I would like to note is that you have to watch out for drivers taking your handful of shaders and permuting them behind your back. Drivers have been known to peek at values from constant/uniform buffers and use them to silently flatten branches and unroll loops. This can lead you to think that your shaders are well-optimized…until you run them on a driver that doesn’t do this and the performance is worse. It’s tougher for drivers to do this in D3D12 and Vulkan since there are more avenues to load data into shaders, but it’s still feasible if the driver is sufficiently motivated.

Deferred Rendering, and Other Techniques For Splitting Up Shading

When people first learn about deferred rendering, there’s usually a lot of focus on how it helps to achieve greater dynamic lighting counts by decoupling the lighting process from your geometry. While this was true at the time it became popular, IMO the real secret weapon of deferred is that it’s a way of chopping up a mesh’s rendering process into at least two different distinct phases. In effect this is really a way of modularizing your shader code, despite the fact that individual shader programs are still monolithic. You can conceptually think of most deferred techniques as splitting up a single gigantic pixel shader function into two separate functions: one that calculates the parameters that feed into the lighting phase (surface normals, albedo, roughness, etc.) and another that takes those parameters and computes how much light reflects off the surface given a collection of lighting sources. With this mental model you can even think of the G-Buffer as being a sort of “ABI” that allows your first phase to pass its arguments to the second, except in this case it does so by writing out many megabytes of per-pixel data to a collection of render target textures.

The effect on permutation count can be significant, since it effectively causes us to split our pixel shader into two feature groups instead of one. Let’s see what our permutated pixel shader from part 1 looks like before and after converting to deferred:

$$ PShader(NormalMap[Off, On] * LightType[Point, Spot, Area] * Shadows[Off, On]) $$

$$ GBuffer(NormalMap[Off, On]) + Lighting(LightType[Point, Spot, Area] * Shadows[Off, On]) $$

Splitting up our shader turned one of those multiplies into an add, causing us to drop from 12 pixel shader permuations down to 8. Saving 4 shaders might not sound like a lot, but keep in mind the delta can be significantly larger as the number of features grow. In practice this can be a huge win for permutation counts, and the modularization can have all kinds of other halo effects. For example you can potentially optimize the code for your G-Buffer and Lighting phases separately without having to consider interactions, and the peak register usage will often be lower as a result of splitting things into smaller shaders with fewer steps. Shader compilers may also be able to chew through your shaders more quickly since they are smaller, since optimization time can grow non-linearly with code length.

In practice there all kinds of flavors of deferred, and even more ways of splitting things up the final rendering pipeline into separate phases with distinct shader programs. For instance, it’s completely possible to only pull out shadow map evaluation into its own deferred passes that only requires a depth buffer, while keeping everything else in a combined forward pixel shader. Or going in the other direction, with deferred you have the option to either do all lighting in a single combined draw or dispatch, vs. further splitting the lighting passes into multiple sub-steps. Doing 1 pass per light source can allow for trivial shader specialization based on the properties of the light source while avoiding combinatorial issues, while doing separate passes for things like sampling ambient diffuse lighting can also provide opportunities for using async compute.

But of course, everything is a trade-off. One thing that most deferred renderers need to do at some point in the frame is write out results to memory and then read them back into registers¹. In the more classic deferred setups this happens when the G-Buffer is written out to render target textures, and then gets read back in during a lighting pass. If you were to compare that to a modern forward renderer you would likely see that the “G-Buffer” still exists in the forward pipeline, it just ends up sitting in registers that store the values passed between two phases of the same shader program. When you think about things in those terms deferred might not sound so appealing! After all, isn’t it better to keep data in ultra-fast registers rather than spilling them out to memory and then reading them back in again? But of course things are much more complicated than that when it comes to performance and renderer design, as we discussed in part 1 when we talked about taking a global approach to GPU performance. Perhaps using more bandwidth and memory ends up being a net negative, or perhaps the shorter/higher-occupancy shader programs enabled by modularization win out in the end. Perhaps letting the compiler optimize across a giant forward pixel shader enables some serious wins, or perhaps the I$ misses add up and limit your performance. Or perhaps you end up designing your deferred renderer in a way such that you can analyze the properties of your G-Buffer on per-tile basis, and use that analysis to manually propogate optimizations into the deferred shader, thus gaining back many of the benefits that the compiler’s optimizer might have afforded you. Or maybe your forward renderer completely dies due to pixel shader quad overdraw caused by tiny triangles, whereas the deferred renderer handles it more gracefully. It’s clearly not a straighforward proposition, and the continued success of deferred certainly suggests that it can win out in many cases.

Offline Linking

In part 1 we discussed how the monolithic shader compilation model contributed to permutation explosions. If you can’t compile and optimize portions of a shader program individually, then permututations that only affect that portion end up requiring the entire program to be recompiled. What if there was some hot new technology from the 1970’s that would let us compile separate binaries and then “link” them together into one combined shader program? That would be be pretty neat, right? 😉

The good news is that DXIL and SPIR-V support this sort of thing! For D3D12/DXIL all of the functionality to compile and link libraries is present in dxc, either via the command line or through its API. See my recent post about it for more details on how it works in practice. The story for SPIR-V though is a little more complicated. SPIR-V Tools has a utility for doing the linking, provided you have some compiled SPIR-V that’s been generated with the appropriate linkage type. At the time of writing however, neither dxc nor glslang are capable of generating SPIR-V with the appropriate declarations. Therefore if you want to write HLSL or GLSL and link the results together, there doesn’t seem to be a working option for doing that. However I’ve been told that other projects that generate SPIR-V through other means (such as rust-gpu) have been able to successfully link using the SPIR-V tools. Either way it’s cool that in both cases the linking happens in open-source toolchains using documented IL representations. ²

There’s a big question mark when it comes to linking: for compiling a zillion permutations, would it actually reduce the compile times? I don’t have any data to draw even some basic conclusions from, so I think we’ll have to wait for someone to do some more comprehensive testing. Ultimately the situation might not be straightforward depending on the inputs to compiling and linking. There’s also the potential for a linking step to make the compilation pipeline more complicated, just like it does for C/C++. In particular it might convert what was once a straightforward parallel operation into a tree with dependencies and sync points.

The other thing that’s both good and bad is that linking doesn’t really change the runtime situation at all. Since the linking can happen fully offline and outside of the API, the API and driver don’t really have to know or care that things were linked together (although there’s always some possibility that linking generates new bytecode patterns that can cause drivers to choke). This means there’s no need to wait for new API or driver support, which is a good thing. It also means there’s no reason to expect a significant delta in performance between a linked and fully-compiled program, at least assuming there’s a post-link optimization step (not doing one would introduce some interesting trade-offs). But it also means that it won’t do anything to reduce the number of shaders and PSOs that have to loaded at runtime, which is not-so-great. Therefore it won’t be a cure if your engine has a case of PSO-itis.

True Function Calls and Dynamic Dispatch

Having a runtime function call combined with dynamic dispatch is potentially more interesting than a linking step, but it’s also much more of a radical change. While linking can happen offline with no driver input, dynamic dispatch for sure requires the driver and the hardware to be on board. The “stuff everything in a statically-allocated chunk of registers” model used by most GPUs certainly doesn’t lend itself to true dynamic dispatch, and it’s easy to imagine that various constraints on the programming might be necessary to keep performance from falling off a cliff.

The good news is that on PC we sort-of have this right now! The bad news is that it’s very constrained, and only works for ray tracing. I’m less familar with the specifics for Vulkan, but in D3D12/DXR it works by essentially having a sort of “run-time linker” step. Basically you compile a bunch of functions into a DXIL binary using a “lib” target (just like you would for offline linking), and then at runtime you assemble all of your pieces together into a combined state object. Later on when you call DispatchRays the driver is able to dynamically execute the right hit/miss/anyhit/etc. shader since it was linked into the state object. There is a callable shaders feature that can be used without any actual ray tracing, however it still needs to be used from within a ray generation shader that was kicked off from DispatchRays. In other words: it’s usable right now for compute-like shaders, but currently can’t be used within the graphics pipeline. With any luck this could change in the future! ³

Currently I’m not aware of anyone that has tried using DXR callable shaders to replace permutations, or at least that has published their findings. Personally I have not experimented with them, so I’m not yet sure how they might work out in practice. With that said, we can still probably make some reasonable assumptions based on what we know about how they work and what kind of implications they might have on usage and performance:

• They currently require a runtime “link” step where all possible callables are assembled into a single state object and associated shader table, either by creating a new one or by calling AddToStateObject to append additional shader binaries. This has implications on streaming, since loading in new callable shaders may require modifying a global state object being used for rendering. Either way you need to be careful where you interact with state objects since it can involve a JIT compile, just like creating a normal PSO.
• We probably should not expect any magic workarounds for static register allocation: if a callable shader requires many registers, we can likely expect for the occupancy of the entire batch to suffer. It’s possible that GPUs could diverge from this model in the future, but that could come with all kinds of potential pitfalls (it’s not like they’re going to start spilling to a stack when executing thousands of pixel shader waves).
• The way the state object spec is setup gives drivers some room to do transformations if necessary. Since the full set of call targets is known when creating or modifing a state object, it’s feasible that the driver might make a decision to “flatten” a CallShader into a branch with inlined function calls.
• There are some subtle implications depending on whether this kind of functionality is implemented as a function call from the same wave, or whether it implies that a wave is launched that runs the function being called. In particular the DXR model implies the latter, since hit shaders can set their own resource context and bindings via a local root signature. There can also be implications around how divergence is handled (different threads in the same wave calling different functions), and whether or not threads are allowed to be “re-packed” into a full wave. For replacing existing permutations I would expect the function calls to be completely uniform, and thus a simpler “run the function in the same thread” approach is sufficient and probably more desirable. Having a resource context is also not necessary as long as bindless is being used.

We’ll have to watch this area closely in the future. With any luck this kind of functionality can be extended to graphics pipelines and standard compute shaders, which could be a very compelling option for those of us that are juggling too many permutations. Metal is also currently ahead of the curve on this front by offering function pointers that can be called from any shader stage. Perhaps this can serve as inspiration for PC and Android!

Summary

Final Thoughts

That’s it for the series! I hope it was an enjoyable read, and that it helped to give some additional context regarding the ongoing battle against shader combinatorial explosion. Please leave a comment or reach out on Twitter if you have any additional thoughts or insights, or if you take issue with anything I’ve written here.

I’d also like to quickly think everyone that read these posts early and provided valuable feedback and insight:

• Jasper St. Pierre
• Nathan Reed
• Jedd Haberstro
• Alex Tardiff
• Rolando Caloca
• Jeremy Ong
• Sebastian Sylvan
• Gokhan Avkarogullari
• Joel de Vahl
• Sam Martin
• Richard Schreyer
• Stephen Hill

As I mentioned above, if you’re going to link things then you’re going to want to compile to a library target rather than one of the shader stage targets (ps_6_x, vs_6_x, etc.). Let’s say we want to create a full pixel shader where we can call into an external function that we can link in later. You would first want to setup an entry point like this in a file:

// Entry.hlsl

// This is the function we're going to link to
float4 LibFunc();

// This is the entry point for our pixel shader
[shader("pixel")]
float4 PSMain(in PSInput input) : SV_Target0
{
    return LibFunc();
}

You would then want to compile this with the lib_6_6 target (or the equivalent of whatever shader model you’re targetting) in order to produce a binary DXIL blob containing the compiled bytecode. For the implementation of LibFunc() we’ll now want to setup another file with this code:

// Func.hlsl

export float4 LibFunc()
{
    return float4(0, 1, 0, 1);
}

Note the export keyword. By default the compiler won’t give your functions external linkage when compiling to a lib, so you’ll want to add that keyword for any functions that you want to link to (you can also pass -default-linkage external when compiling but that’s quite a hammer). You would once again compile this to the lib_6_6 target to get another DXIL binary.

To link these two binaries together, you’ll want to create an instance of IDxcLinker, register your libraries with it, and then instruct it to create a new linked binary containing a complete pixel shader:

IDxcBlob* entryLib = CompileLib("Entry.hlsl", "lib_6_6");
IDxcBlob* funcLib = CompileLib("Func.hlsl", "lib_6_6");

IDxcLinker* linker = nullptr;
CheckHResult(DxcCreateInstance(CLSID_DxcLinker, IID_PPV_ARGS(&linker)));

CheckHResult(linker->RegisterLibrary(L"EntryLib", entryLib));
CheckHResult(linker->RegisterLibrary(L"FuncLib", funcLib));

const wchar* names[] = { L"EntryLib", L"FuncLib" };

IDxcOperationResult* result = nullptr;
CheckHResult(linker->Link(L"PSMain", L"ps_6_6", names, ArraySize_(names), nullptr, 0, &result));

IDxcBlob* fullShader = nullptr;

HRESULT status = S_OK;
CheckHResult(result->GetStatus(&status));
if(SUCCEEDED(status))
{
    result->GetResult(&fullShader);
}
else
{
    IDxcBlobEncoding* linkerErrorMsg = nullptr;
    result->GetErrorBuffer(&linkerErrorMsg);

    OutputErrorMessage(linkerErrorMsg);
}

That’s really it! If the operation succeeds, then fullShader will contain a complete DXIL binary that you can use to create a PSO the usual way. I haven’t done any testing yet to see if linking is actually any faster than doing a full compile, but if it is faster then it could potentially be an avenue for reducing the compile times for excessive shader permutations.

One major limitation to be aware of is that resources can not be a parameter or return value for exported functions. This is a real bummer, since it will require workarounds for real code. The only way to coordinate on resources right now is to either use the fancy new dynamic resources functionality from Shader Model 6.6, or have the lib that uses the resource declare it globally. You can also declare the same resource in both libs, in which case things will merge correctly as long as they use the exact same register assignment:

// Entry.hlsl

Texture2D MyTex : register(t0);

float4 LibFunc();

[shader("pixel")]
float4 PSMain(in PSInput input) : SV_Target0
{
    return LibFunc() + MyTex[uint2(1, 1)];
}

// Func.hlsl

Texture2D MyTex : register(t0);

export float4 LibFunc()
{
    return float4(0, 1, 0, 1) * MyTex[uint2(2, 2)];
}

Happy linking!

Ray Tracing Gems II was released a few weeks ago, and like the first one it’s free to download as a PDF from the publisher’s website. There are 50 articles in this thing written by over 80 authors, so I think it’s safe to say that this book is absolutely jam-packed with useful knowledge and techniques related to ray tracing.

Yours truly contributed a chapter called “Using Bindless Resources With DirectX Raytracing”, which as you’d expect is all about using bindless techniques in the context of DXR. I had already written a simple open source path tracer in DXR which uses bindless for all SRVs, so it seemed like a natural fit for me to write full article that dives into all of the details required to make that happen. I also wasn’t aware of a comprehensive article or tutorial explaining the ins and outs of bindless in D3D12, and I thought my article could help fill that niche. Hopefully I was successful! If you do end up reading it, feel free to let me know if you have any thoughts or have any lingering questions that I can help clear up.

I also wanted to use this blog post as an opportunity to emphasize a few things in an informal way, just in case they don’t fully come through in the article:

• Bindless is great, it really is! There are downsides of course (mostly on the tooling and debugging side of things), but I firmly believe the pros outweigh the cons. I don’t really see any future for the older binding models.
• It’s hard to oversell how bad of an experience it is dealing with multiple root signatures and descriptor tables/ranges compared to bindless. Once you go bindless a lot of that complexity just disappears, which could end up being a good thing for people that are new to D3D12/Vulkan or graphics programming in general.
• On a related note: just don’t bother with local root signatures in DXR, at least for binding descriptor tables. Ray tracing is inherently global, and it makes so much sense to have global access to all of your resources/descriptors.
• In the article I mention how the indirection offered by using descriptor indices can let you do some really cool things. Basically as long as you keep your descriptor index persistent, you’re free to update the descriptor located at the index without having to also go and update whatever data structures have that index in it. Got a ring buffer system where you swap to a different buffer when the contents are updated by the CPU? Just update the descriptor when you do that and everything keeps working great! Same goes for updating textures due to streaming in new mips, or swapping between aliased render target textures as part of an adaptive resolution system.
• Bindless is awesome for CPU performance and really helps unlock the original dream of low CPU overhead that was promised by D3D12 and Vulkan. Once you’ve moved all of your “binding” out of API functions and into your own user-managed memory you can do things much more quickly, and you can really fly through your draw calls. Combine that with multithreaded command buffer recoding and you can do a ton of draws and dispatches with very low latency.

Finally, I want to finish with a note addressing Shader Model 6.6 and the new “dynamic resource binding” feature. When I started working on the article it was March of 2020, since RTG II was originally supposed to be released that year. At the time the idea of a new HLSL feature that makes it simple and trivial to access bindless descriptors was little more than some hopes and dreams in a GitHub issue, and I thought the situation would be the same by the time the release date rolled around. I did not expect the book release to be pushed back, I really did not expect for Shader Model 6.6 to be announced before the year end, and I definitely did not expect for the D3D team to release an awesome new Agility SDK that allowed for SM 6.6 to be deployed to end user machines without waiting for Windows updates. But of course all of those things happened, when meant by the time the chapter came out people were already using dynamic resources in their code. It’s too bad, but it was of course bound to happen eventually. For those not using the latest-and-greatest the article should still help with spelling out the right HLSL and root signature incantations required for pre-SM6.6 bindless, and those that are on the bleeding edge can simply ignore that part and make use of the other content. On the plus side: dynamic resources are really really great. They fix a lot of little rough edges that show up when doing bindless the old way, and take an even further step towards simplifying and reducing the number of root signatures.

Thanks for reading!

If you’re skilled in the dark arts of approximating complex integrals with simpler functions, you could probably see yourself doing that for the lookup texture generated by a single diffusion kernel (and like I mentioned last time Angelo Pesce already took a stab at this for the kernel used in the original implementation of preintegrated SSS). However you would probably be less enthusiastic about doing this for a whole set of lookup textures. Instead, wouldn’t it would be much nicer if we could work with a more analytical approach that could approximate a wide range of scattering profiles without any precomputation? The rest of this post will talk about one possible way of achieving this using spherical gaussians.

Blurring A Light With an SG

Unfortunately I haven’t come up with any magical solutions for how to analytically compute the result of irradiance being convolved with a diffusion profile, even with the constraints used in the original preintegrated subsurface scattering technique (point lights, and making the assumption that the point being shaded is on the surface of a sphere). However, it turns out that we can achieve this somewhat indirectly by instead blurring the incident radiance and computing the resulting irradiance from that. It’s not the same thing conceptually, but it does produce similar results to what we’re after. The most obvious choice to do this kind of operation is probably spherical harmonics (SH): with SH we store spherical signals in the frequency domain, which makes it trivial to perform convolutions on the data (this is in fact how irradiance is traditionally computed from a set of SH coefficients representing incoming radiance). However this frequency domain representation means that we are unable to represent high-frequency components without storing and working with very large numbers of coefficients. In other words an SH approach might work well for something like an environment probe used for ambient lighting, but it can be pretty inaccurate when used for a punctual light source. To show you what I mean, here’s a comparison showing what the irradiance from a simple directional light looks like when computed analytically versus being projected onto L1 SH with 4 coefficients, and L2 SH with 9 coeffecients:

You can see quite clearly that we’re already getting a filtered, lower-frequency version of the lighting response just by going through low-order SH. This means that the SH approach can be suitable for relatively large/wide diffusion kernels, but it’s not a great fit in terms of being able to seamlessly scale from small scattering amounts to larger mean scattering distances.

As an alternative to SH, we can instead consider using our old friend the Spherical Gaussian (SG). SG’s have five properties that are of interest to us when it comes to trying to approximate the response from a filtered light source:

1. They can be oriented in any direction, which means they can be lined up perfectly with a punctual light source
2. They have an aribitrary sharpness parameter ( \( \lambda \) ) that lets them represent a range of filtering parameters, including very narrow kernels
3. An SG is trivial to normalize (when making some reasonable approximations), which ensures that we will not add or remove any energy
4. An SG convolved with a delta gives you another SG. This means we can use an SG to represent a filtering kernel, apply it to a punctional light source, and get another SG
5. We have a good approximation for computing the resulting irradiance from an SG light source

With this in mind, the approach looks fairly straightforward: first, we come up with an SG that represents the filtering kernel we will use to approximate our scattering profile. Wider kernels will have a lower sharpness parameter, while more narrow kernels will have a higher sharpness. We can then normalize the SG to ensure that it integrates to 1 over the entire sphere, which is critical for ensuring energy conservation in the filtering process. Once we have a normalized SG representing the filtering kernel, we combine it with our light source by setting the axis ( \( \mu \) ) parameter of the SG equal to the direction to the light source, and multiplying the SG’s amplitude ( \( a \) ) by the attenuated intensity of the light source. This effectively gives us an SG light source, for which we can compute the diffuse response in the usual way by using Stephen Hill’s fitted irradiance approximation and multiplying the resulting irradiance by \( \frac{albedo}{\pi} \). For scattering profiles that vary by wavelength (which is typically the case) we extend this process by coming up with a separate SG for the red, green, and blue channels and evaluating each separately. When doing this the light intensity can actually be factored out if we like, which means we can leave the SG with its normalized amplitude and multiply with the attenuated light intensity afterwards. Here’s what this all might look like in code form:

float3 diffuse = nDotL;
if(EnableSSS)
{
    // Represent the diffusion profiles as spherical gaussians
    SG redKernel = MakeNormalizedSG(lightDir, 1.0f / max(ScatterAmt.x, 0.0001f));
    SG greenKernel = MakeNormalizedSG(lightDir, 1.0f / max(ScatterAmt.y, 0.0001f));
    SG blueKernel = MakeNormalizedSG(lightDir, 1.0f / max(ScatterAmt.z, 0.0001f));

    // Compute the irradiance that would result from convolving a punctual light source
    // with the SG filtering kernels
    diffuse = float3(SGIrradianceFitted(redKernel, normal).x,
                     SGIrradianceFitted(greenKernel, normal).x,
                     SGIrradianceFitted(blueKernel, normal).x);
}

float3 lightingResponse = diffuse * LightIntensity * DiffuseAlbedo * InvPi;

For this snippet we used a very simple mapping from an arbitrary ScatterAmt parameter to an SG sharpness, where \( \lambda = \frac{1}{ScatterAmt} \). This doesn’t directly map to a diffusion kernel that’s specified in terms of distance from the shading point, but we’ll come back to that latter. First, let’s see what this looks like when we apply this algorithm for a simple directional light with various values of ScatterAmt:

From these results, it looks like this gets us pretty close to our goal of approximating the subsurface scattering response of a point light with aribtrary scattering widths. And we’re doing all of that with pure shader math, no lookup textures required! The first image in the top-left also shows that we can scale the filtering all the way down to the “no scattering” case just fine, which is convenient.

If you wanted to optimize this further you could probably start by pulling out the portions of SGIrradianceFitted that aren’t dependent on the SG sharpness, since the normal and axis are the same for all 3 SG’s. However in practice you will probably want to combine this with the normal map prefiltering technique described in the preintegrated subsurface scattering article, since that will simulate scattering through small-scale bumps. If you do this you’ll end up with separate normal values for each channel, in which case you won’t be able to make the same optimization.

Accounting For Curvature

If you’re familiar with how preintegrated SSS works, you may have already realized a problem with our proposed technique: it’s not accounting for the curvature of the surface. If we were to just use the same set of sharpness values everywhere on an arbitrary mesh the results would not look correct, because the falloff/wrap caused by the SG would be same regardless of how flat or curved the surface is. Another way of saying this is that since our SG works purely in an angular domain, we’re effectively getting larger or smaller scattering distances across the mesh. Fortunately this problem is solvable with a bit of math. Recall that an SG has the following form:

$$ G(\mathbf{v};\mathbf{\mu},\lambda,a) = ae^{\lambda(\mathbf{\mu} \cdot \mathbf{v} - 1)} $$

or alternatively:

$$ G(\mathbf{v};\mathbf{\mu},\lambda,a) = ae^{\lambda(\cos(\theta) - 1)} $$

Since it only depends on the cosine of the angle between the sample point and the SG axis direction, the “width” of the SG is fixed in the angular domain but variable in terms of Euclidean distance if we were to vary the radius ( \( r \) ) of the sphere that it’s defined on. Therefore we need to redefine our SG in terms of Euclidean distance ( \( d \) ) between the SG center on the surface of the sphere ( \( c \) ) and the shading point ( \( s \) ), where \( \theta \) is the angle between the surface normal and SG axis (which is aligned with the light direction):

We can then use the following to get the distance between the points:

$$ d = 2r\sin(\frac{\theta}{2}) $$

If we flip that around to get the value of \( \theta \) in terms of distance, we get this:

$$ \theta = \sin^{-1}(\frac{d}{2r})2 $$

We can then plug that into the second form of the SG definition and get the following:

$$ G(d;\lambda,a,r) = ae^{\lambda(\cos(\sin^{-1}(\frac{d}{2r})2) - 1)} $$

This looks pretty gnarly, but fortunately we can simplify this by making the substitution \( \cos(2\sin^{-1}(x)) = 1 - 2x^{2} \), which gives us something much nicer once we clean it up a bit:

$$ G(d;\lambda,a,r) = ae^{\frac{-\lambda d^2}{2r^2}} $$

In case that’s too hard to read, this is the exponent part on its own:

$$ \frac{-\lambda d^2}{2r^2} $$

This is now really close to a standard 1D Gaussian form, which is pretty neat! It’s also now quite clear that the effective width of the SG will increase as the radius increases, which is not what we wanted for subsurface scattering. This is confirmed if we graph this out for a few values of \( r \):

The good news is that we can fix this very easily: we just need that \( r^2 \) term to drop out, which will happen if we multiply \( \lambda \) by \( r^2 \). Doing that gives up exactly a 1D Gaussian, except with a \( \lambda \) in the numerator of the exponent instead of the usual \( \sigma \) in the denominator:

$$ G(d;\lambda,a) = ae^{\frac{-\lambda d^2}{2}} $$

This final form gives us the fixed width kernel that we’re after, as long as we can compute the effective radius from the curvature of the mesh. We can do that using pixel shader derivatives as described in the preintegrated SSS article, or we can precompute it and store it in a texture or per-vertex. This also gives another interpretation of our ScatterAmt parameter that we used earlier: it’s the \( \sigma \) parameter of a Gaussian centered around the point \( c \).

Applying To Ambient Lighting

In most real-time scenarios we’re going to be combining lighting from punctual light sources with precomputed GI stored as either spherical or hemispherical probes in 3D space. For SH probes, I already outlined a workable approach earlier in this article. Really you just need some kind of function for generating the SH coefficients for a Guassian, and then we can multiply that with the SH lighting environment to pre-filter it. Something similar to what Ravi Ramamoorthi proposed for approximating a Torrance-Sparrow NDF would probably work pretty well:

$$ \Lambda \rho \approx e^{-(\sigma l)^{2}} $$

For probes encoded as a set of SG’s, we can make use of Kei Iwasaki’s forumula for approximating the convolution of two SG’s (brought to my attention by Brian Karis’s blog post), which looks like this once implemented in HLSL:

//-------------------------------------------------------------------------------------------------
// Approximates the convolution of two SG's as another SG
//-------------------------------------------------------------------------------------------------
SG ConvolveSGs(in SG x, in SG y)
{
    SG convolvedSG;
    convolvedSG.Axis = x.Axis;
    convolvedSG.Sharpness = (x.Sharpness * y.Sharpness) / (x.Sharpness + y.Sharpness);
    convolvedSG.Amplitude = 2 * Pi * x.Amplitude * y.Amplitude / (x.Sharpness + y.Sharpness);

    return convolvedSG;
}

The approach is then similar to what we do for punctual lights: come up with the normalized SG’s for red/green/blue, convolve them with each SG in the lighting probe, and compute the resulting irradiance.

Limitations and Future Work

Compared with standard Lambertian diffuse, this approach is much more math-heavy. I have implemented and shipped this technique in two games already, so it’s definitely usable. But obviously you’ll want to be wary of the extra ALU and register pressure that it can add to a shader program.

In terms of the visual results, it mostly suffers from the same major visual drawbacks that you get with preintegrated subsurface scattering. In particular there’s no great solution for handling shadow edges, and your results may suffer if your technique for computing curvature has errors or discontinuities. It also does nothing to account for scattering through thin surfaces like the ears, so that has to be handled separately. It also shares another limitation with preintegrated SSS: it assumes punctional light sources. For area lights you would need to do something more complicated, perhaps using the LTC framework.

I also have not taken the time to come up with a more principled approach for applying the normal map prefiltering based on the ScatterAmt parameter. Currently I use a simple ad-hoc mapping from that parameter to lerp factors between the detail normal map and the base normal map, which works well enough but could certainly be improved. If anybody has thoughts on this, please let me know in the comments or on Twitter!

SSS Basics

First of all, what do we mean when we say “subsurface scattering”? In reality subsurface scattering is happening even on our more “standard” materials like plastic or leather, which we typically model with a diffuse BRDF like Lambertian. This helpful image from Naty Hoffman’s 2013 talk on the physics of shading is great for understanding what’s going on:

In this case you want to imagine that the green circle with the red outline is the shading area for a single pixel. The big orange arrow coming from the right is some incoming light from a particular direction, most of which reflects off the surface causing specular reflections. However some of that light also refracts into the surface, where it encounters particles within the medium. The light then scatters among the particles causing certain wavelengths to get absorbed, resulting in the light taking on the characteristic albedo color of the material. Eventually the light that isn’t completely absorbed refracts its way back out of the surface in a random direction, where it may or may not make it back to eye causing you to see it. The part that makes this easy is the fact that the scattered subsurface light all comes out within the pixel footprint. This lets us simplify things by only considering the light hitting the center of the pixel, and then estimating the amount of light that comes back out. In effect we do something like this, where we assume all of the scattered light comes out right where it initially entered the surface:

However this is not usually what we’re talking about when we mention subsurface scattering in the context of games or graphics. Usually that term is only brought up in cases where the material is particularly translucent, allowing the scattered light to bounce around even further than in more typical materials. Like in this diagram:

In this case the scattered light is no longer coming out within the pixel footprint, it’s coming out in areas that could possibly be covered by neighboring pixels. In other words we’ve crossed a threshold where we can no longer consider only the incoming lighting to a pixel, and we instead need to globally consider lighting from elsewhere in order to model the visible effect of light diffusing through the surface. As you can imagine this really complicates things for real-time renderers, but we’ll talk about that more in a bit.

For a moment, let’s assume we’re doing an offline render with access to ray tracing and other fancy features. How do we compute the contribution from subsurface scattering in that case? If we wanted to be really accurate, we would use volumetric path tracing techniques to compute how lighting scatters and refacts about within the medium. This probably sounds expensive (and it is!), but it’s becoming an increasingly viable option for high-end animation and VFX studios. Either way this approach lets you simulate the full path of light along with all of the various scattering events that occur under the surface, which can give you a relatively accurate result that accounts for complex geometry and non-uniform scattering parameters.

Diffusion Approximations

For games and real-time we’re obviously a lot more constrained, so instead we typically work with a more approximate diffusion-based approach. Most of the work in this area stems from Jensen’s paper published back in 2001, which was the first to introduce the concept of a Bidirectional Surface Scattering Distribution Function (BSSDF) to the graphics world. That paper spawned all kinds of interesting follow up research over the past 18 years, spanning both the worlds of offline and real-time graphics. Covering all of that is outside of the scope of this blog post, so instead I’m going distill things down to the basic approximations that we work with in games.

They key to Jensen’s approach is the simplifying assumption that the light distribution within a highly-scattering media tends to be isotropic. If you’re not familiar with volumetric rendering terms, what this means is that once light into the surface it’s randomly likely to end up coming out in any direction, rather than being mostly aligned to the incoming light direction or something like that. This effectively makes the scattering sort of a “blurring” function, which results in a uniform distribution of reflected lighting (note that this is basically the same assumption that we make for Lambertian diffuse). Jensen’s BSSDF was actually based on two parts: an exact method for computing the single-scattering contribtution (light that goes into the surface, hits something, then bounces right out again) and a “dipole” method that approximates the multiple-scattering contribution (light that goes into the surfaces, has multiple scattering events, and then finds its way out) by evaluating two virtual point light sources: one above the surface, and one below:

Even if you have analytical approaches for computing the scattering from an incoming point on the surface to an outgoing point, the problem is that your rendering integral just got even worse than it usually is. Evaluating an arbitrary BSSDF means computing the value of this integral:

$$ L_{o}(o, x_{o}) = \int_{A} \int_{\Omega} S(x_{i}, i; x_{o}, o) L_{i}(x_{i}, i)(i \cdot n_{i}) didA $$

That’s one mean integral to evaluate: it basically means we have to go over the entire surface of the mesh, and at every point on that surface sample every direction on the upper hemisphere and apply the scattering function \( S(x_{i}, i; x_{o}, o) \) to the incoming lighting. If we want to simplify this a bit, we can make the assumption that the scattering function is radially symmetric and really only depends on the distance between the incoming point and the outgoing point. That brings us to a somewhat less complicated integral:

$$ L_{o}(o, x_{o}) = \int_{A} R(||x_{i} - x_{o}||) \int_{\Omega} L_{i}(x_{i}, i)(i \cdot n_{i}) didA $$

Our new integral now boils down to “for every point on the surface, compute the irradiance and then multiply that with a scattering function that takes distance as a parameter”. This is less complicated, but doesn’t fix the whole “to shade single point we need to shade every single other point on the surface” problem. However it does give us a useful framework to build on, and also a simplified mental model that we can use. The “scattering only depends on distance” assumption basically turns the scattering function into blur/filter kernel, except it’s applied to an arbitrary mesh surface instead of a 2D image. Thus you’ll often see the \( R(r) \) part referred to as as a diffusion profile (or sometimes as a diffusion kernel), which is the terminology that I’ll often use myself for the rest of this article. If you want to picture what the diffusion profile would look like for something like human skin, you’ll want to imagine that you’re in a completely dark room and could shoot an extremely narrow completely-white laser beam at your skin:

Technically speaking, you can use whatever diffusion profile you’d like here. For instance you can use a Gaussian kernel, smoothstep, cubic function, etc. However you’ll really want to work with something that’s “normalized” in the sense that it doesn’t add or remove energy (in other words, the kernel integrates to 1 for all possible configurations). There’s also no guarantee that the kernel you pick will actually be a plausible match for the scattering behavior of a real-world behavior such as skin. To do that, you have to carefully choose your diffusion profile such that it closely matches the measured (or accurately computed) response of that material. Disney did exactly this for human skin, and they created a kernel based on the sum of two exponentials that they called “normalized diffusion”. The neat thing about their model is that not only is it simple and intuitive to work with, they also showed that they could accurately fit to both the single and multi-scattering components with a single kernel:

Rendering With a Diffusion Profile

So how we do we actually render something with a BSSDF? Even with the simplifications that we get from using a radially-symmetric diffusion profile, we still have the problem that the shading for any particular surface on a mesh potentially depends on every other surface on the same mesh. In practice we can improve the situation by exploiting the fact that a well-behaved diffusion profile should always decrease in value as the distance gets larger. This lets us determine a “maximum scattering distance” where the contribution beyond that distance is below an acceptable threshold. Once we have this \( r_{max} \), we can limit our sampling to within a sphere whose radius is equal to \( r_{max} \). This definitely helps things quite a bit by letting us constrain things to local neighborhood around the shading point.

Most early uses of sub-scattering the offline world was built using around using point clouds to sample the neighboring irradiance. Point clouds are a hierachical structure that allows the illuminance to be cached at arbitrary points in the scene, typically located on the surfaces of meshes (also known as surfels):

They were popularized by Pixar, who integrated them into RenderMan in order to compute global illumination. Later on as point clouds fell out of favor and path tracing began to take over, monte carlo methods began to appear. This presentation from Solid Angle and Sony Imageworks sums up 4 such approaches, each based around the idea of distributing ray samples around the sample point in such a way that their density is proportional to the diffusion profile. The fourth method (shooting rays along within a sphere from orthogonal directions) has been used by Disney, and perhaps many others:

while the first method (uniformly shooting rays from a point just below the surface) has actually been used in real-time by EA SEED’s Halcyon rendering engine:

These methods can both be very effective as long as you have the resources to perform ray intersections against your scene geometry. Despite the heavy expense involved, there are still making some assumptions being made here that can be violated depending on your particular scene. In particular, each point that is hit by a ray is assumed to be “connected” to the original shading point such that it would be plausible for light to scatter all of the way there. This issue is mitigated by restricting the ray queries so that they only intersect with the same mesh being shaded, which has the added bonus of reducing the amount of surfaces that need to be considered. But even with that restriction it’s easy to imagine a scattering path that would pass right through an empty gap of air. Disney has a diagram from their course notes showing such a case:

If light were to scatter all the way from the entry point to the exit point in that example, it would have to scatter all the way around the crevice which is quite a long way to go. An approach that merely consideres the distance between those points will over-estimate the scattering here, as opposed to a proper volumetric path tracer that could actually account for the gap.

Rendering Diffusion in Real-Time

So it’s great that offline renderers can just shoot a bunch of rays around in order to sample the irradiance in a local neighborhood, but until very recently it wasn’t viable to do this in real-time (and even with the latest RTX hardware you’re still only going to get a handful of rays per-pixel every frame). This meant that games generally had to fall back to even cheaper approximations in order render the skin on their hero characters. There have likely been all kinds of ad-hoc approaches used over the past 15 years or so, many of which were never presented or described anyhere. So instead of covering them all, I’m going to give a quick overview of some of the more commonly-used and referenced techniques that you’ll find out in the wild.

Texture-Space Diffusion

This approach was popularized by the incredible Human Head demo for the GTX 8800, (which you can still download on Nvidia’s site!) and described in detail in GPU Gems 3 as well as in the paper from Eurographics 2007. The technique actually comes from the film world: it was first used and documented as part of the production of The Matrix Reloaded way back in 2003. Eugene d’Eon and David Luebke were able to show that it could also work in real-time on (at the time) high-end GPUs, and in the process they were able to create some visuals that are still impressive today. It’s called “texture-space” because all of the scattering/diffusion simulation happens in 2D using the UV parameterization of the head/face mesh, as opposed doing it per-pixel in screen-space. A simplified explanation of the algorithm would go something like this:

• Draw the mesh using a special vertex shader that sets output.position = float4(uv * 2.0f - 1.0f, 0.0f, 1.0f); and a pixel shader that outputs the irradiance from dynamic + baked lighting, effectively computing an irradiance map in texture-space
• Blur the irradiance map with bunch of Gaussian kernels, with a horizontal + vertical pass for each kernel (they approximate skin’s diffusion profile as a sum of separable gaussians)
• Draw the mesh for real using the camera’s projection matrix, then in the pixel shader sample the convolved irradiance maps and apply the diffuse albedo map to get the diffuse reflectance with subsurface scattering.

Obviously their results speak for themselves in terms of quality, so I don’t think that would be the main issue preventing adoption. It’s also fairly flexible in that in can handle many different diffusion profiles, provided that they can be represented as a sum of Gaussians. However there are definitely some issues that become obvious pretty quickly if you think about it:

• Shading in texture-space can be difficult to tune in terms of performance and memory usage. Shading in a pixel shader means that your shader invocations naturally drop as the mesh gets further from the camera, since the mesh will occupy less are a in screen-space. However with texture-space shading you will shade however many texels are in your off-screen irradiance buffer, so you need to manually reduce the buffer size based on the distance to the character. Ideally you would want to match the mip level that will be sampled by the mesh during the primary rasterization path, but there could potentially be a wide range of mip levels used depending on the complexity of the mesh and the camera’s current position. Budgeting enough memory for the irradiance buffer can also be difficult if you have many characters on-screen at once, especially if sudden camera cuts will cause a particular character to suddenly be in a close-up. And don’t forget that you lose z-buffer occlusion and backface culling when shading in texture-space, which means that the naive approach could potentially result in shading a lot of surfaces that never contribute to the final image!

• In general, texture-space diffusion makes sense because of the assumption that the local neighborhood of a surface will also be local in the UV mapping. Most of the time this is correct, but any UV seams will break this assumption and cause ugly artifacts without somekind of workaround. It also fails for thin surfaces like the ear, since both sides of the ear are probably not going to be close in UV space. The demo accounts for the forward-scattering through thin surfaces by rendering a translucent shadow map that records the UV coordinate of the closest depth for each texel, which is then used by the surface on the “other side” to sample the blurred irradiance map at the point where the light ray hit the mesh.

• Typically the UV mapping of something like a face will have lots of stretching and warping, which would also result in a warping of the diffusion profile itself once the results are mapped back to world space: The Nvidia demo works around this by rendering out a “stretch map” in texture-space that contains the U/V distortion factors, which can then be used to modify the diffusion profile weights. You can pre-compute this map, but it will be incorrect in the case of extreme deformations from skinning or blend shapes. In those cases computing the stretch map on the fly would produce better results.

• The demo used 6 large Gaussian blur kernels to approximate the skin diffusion profile, which is a lot of passes! There was a Shader X7 article where John Hable described a simplified version were all 6 passes were collapsed down to a single 12-tap pass that could be done in the pixel shader when sampling the irradiance map. Unfortunately there’s no online version of that article, but he describes it a bit in this presentation from SIGGRAPH 2010.

• Specular shading is done in the pixel shader during the primary rasterization pass, which means that light sources need to be effectively sampled twice. An alternative would be to compute the specular in texture-space, and output it to a separate render target that’s sampled during the rasterization pass.

• The additional complexity of integrating a special-case texture-space rendering path for characters could potentially add a high maintenence cost, and may not play nice with other passes that are typically done in screen or camera space (for instance, binning lights in a frustum-aligned grid).

I’m not sure if any game actually shipped with this approach (or something) similar at runtime. Uncharted 2 used the approach described by John Hable for offline-rendered cinematics, but at runtime they fell back to something much simpler and cheaper. There was an old ATI demo that did texture-space diffusion 3 years before d’Eon and Luebke published their work at Nvidia, but what they describe sounds much less sophisticated. Overall the lack of adoption isn’t terribly surprising given the large list of potential headaches.

Update 10/7/2019: Vicki Ferguson was kind enough to point out that FaceBreaker shipped with texture-space diffusion way back in 2008! Angelo Pesce then explained how this tech made its way into the Fight Night series, where it eventually evolved into pre-integrated subsurface scattering! See Vicki’s GDC presentation and Angelo’s “director’s cut” slide deck for more info on the tech used for Fight Night Champion.

Screen-Space Subsurface Scattering (SSSS)

Once it became clear that texture-space diffusion would be both difficult and expensive to use in a real-time game scenario, a few people started looking into cheaper options that make use of lighting data stored in screen-space buffers. The idea was popularlized by the work of Jorge Jimenez, thanks to a very popular article that was published in the first edition of GPU Pro. Morten Mikkelsen (of MikkT fame) also did some early work in this area, and Jorge continued to publish additions and improvements in the following years. The state-of-the-art is probably best represented by Evgenii Golubev’s presentation from SIGGRAPH 2018, where he explained how Unity adopted Disney’s Normalized Diffusion profile for real-time use.

At a high level you’re making a similar bet that you make with texture-space diffusion: you’re hoping that 3D locality of a sample point’s neigboring surfaces is preserved when it’s rasterized and projected into 2D. When this is the case (and it often is) then you merely have to do a few texture fetches from neighboring texels in order to convolve the lighting with your diffusion profile. This is quite a bit more appealing than shading a surface in unwrapped UV space (especially on 2009-era hardware and APIs), since you can borrow from typical deferred approaches in order to output per-pixel irradiance to a secondary render target. This spares you from having to figure out the appropriate shading rate to use for the surface, and also prevents you from inadvertantly shading pixels that end up being occluded by other opaque surfaces.

With those advantages it’s not surprising that many engines ended up adopting SSSSS and using it in shipping games. But of course it’s not all roses, either. Let’s go through some of the issues that you can run into:

• Since you’re working in screen-space, you can only gather irradiance from surfaces that end up getting rasterized. This means that anytime you have some complex occlusion going on (for instance when the nose occludes parts of the cheek) you’re going to end up with missing information. It’s really the same problem that you can get with SSAO or SSR.

• Generally you’re going to be limited in the number of samples that you can gather, which is going to lead to undersampling. This will manifest as banding or “blobiness” if re-using the same sample pattern across all pixels, or as noise if using stochastic sampling. The latter can be mitigated with denoising and/or TAA.

• Doing things in screen-space necessitates doing the rendering in multiple passes, meaning you can’t just completely shade the surface within a single pixel shader. This is unfortunate if everything else is forward rendered, especially since it means you may need to eat the memory of a few extra render targets that are only used for a subset of materials.

• If you only want to run the SSSSS pass on pixels that require it, then you need some kind of system for tagging those pixels and selectively running the shader. Jorge’s original implementation used stencil, but I’ve never met a graphics programmer that enjoyed using stencil buffers. As an alternative you can branch on a per-pixel value, or run a pass to generate a list of screen-space tiles.

• Similar to texture-space diffusion, you can’t really account for light that transmits through thin surfaces like the ear. There might be some cases where both sides of the surface are visible to the camera, but in general you can’t rely on that. Instead of you have account for that separately, using shadow map depth buffers and/or pre-computed thickness maps. I would suggest reading through the slides from this presentation by Jorge for some ideas.

Pre-integrated Subsurface Scattering (SSSS)

The previous two techniques were all about turning diffusion into a gathering problem: one gathered local irradiance in texture-space, the other in screen-space. There’s some key differences that we already covered, but fundamentally they’re trying to do the same thing, just in different domains. Pre-integrated subsurface scattering takes a complelely different approach: it makes no attempt at all to look at the lighting in the local neighorhood. Instead, it tries pre-compute the scattered results, and use the correct result based on a few properties of the point being shaded. The key insight that Eric Penner presented was that subsurface scattering isn’t actually visible in cases where the surface is totally flat and the incoming lighting is uniform across the surface. It’s like opening up an image in photoshop where every pixel has the same color and then running a Gaussian blur filter on it: you won’t see any difference, since you’re just blurring the same color with itself. We can confirm this pretty easily by running ray-traced diffusion on a plane lit by a single direction light, and comparing it with the result you get without subsurface scattering:

Totally worth all of the cost all of those rays, am I right? If we know that you don’t see subsurface scattering when the the light is uniform and the surface is flat, then we can deduce that the scattering will be visible whenever those two things aren’t true. A quick test with our ray-tracer can confirm our hypothesis. Let’s first try a sphere to see what happens when the surface isn’t flat:

In this case the difference is huge! Now let’s look at an example where the lighting changes very quickly due to a shadow being cast by another mesh:

This time we see the lighting diffusing its way over into the shadowed pixels, which is very noticeable. Really these are both two sides of the same coin: the incident irradiance is varying across the surface, in one case due to the changing surface normal and the other due to changes in light visibility.

Eric decided to tackle these two things separately in his presentation, so let’s do the same and start with the case where scattering is visible on a non-flat surface. When a surface is curved, the incident irradiance will decrease on that surface as \( N \cdot L \) decreases which is what gives you that characteristic diffuse falloff. However with subsurface scattering effectively “blurring” the incident irradiance, the appearance of that falloff will change. This is particulaly noticeable at the point where \( N \cdot L = 0 \), since in the example with SSS enabled the lighting actually “wraps” right over that line where it would normally terminate. In fact, an old trick for approximating subsurface scattering is to deliberately modify the \( N \cdot L \) part of the lighting calculations in order to produce a result that resembles what you get from SSS. Eric took this idea and made it more principled by actually precomputing what the falloff would be if you used the diffusion profile predicted by the dipole model. The way you do this is to take a circle, and then compute the resulting falloff for every point on that circle by integrating the incident irradiance multiplied with the diffusion profile:

This ends up being a kind of convolution on the surface of a circle, which fits well with our mental model of SSS being a filtering operation. Doing this for the whole circle would give you a result for every value of \( \theta \) in the range \( [ -\pi, \pi ] \), but the result is symmetrical so we can discard half of the results and just keep everything from \( [ 0, \pi ] \). We can also switch from parameterizing on \( \theta \) to parameterizing on \( cos(\theta) \) in the range \( [ -1, 1 ] \), which gives us a nice 1D lookup table that we can index into using any result of \( N \cdot L \). Visualizing that would give us something like this:

This is immediatly usable for rendering, but with a major catch: it only works for a sphere with a particular radius and diffusion profile. However we can easily extend this into a 2D lookup table by computing the resulting falloff for a range of radii:

At this point you might be thinking “this is neat for spheres, but how do we use this for an arbitrary mesh?”, which is a valid question. To use this new 2D lookup texture, we need to basically take every shaded point on the mesh and map it to a sphere of a particular radius based on the local curvature of that point. Once again this was not a new idea at the time Eric presented: there was previous work from Konstantin Kolchin and Hiroyuki Kubo that had made the obvervation that subsurface scattering appearance was tied to surface curvature. Really the idea is pretty simple: figure out the local curvature, map that to a sphere radius, and then use that to choose the appropriate row of the preintegrated falloff lookup texture. Penner proposed a way to do this on-the-fly in a pixel shader by making use of pixel quad derivatives and little bit of geometry:

Very cool! This works…at least for curvature that’s broad enough to be captured accurately by the pixel quad derivatives. A normal map used for a human face will typically have many small bumps and other high-frequency details, and we need special-case handling if want them to look correct. Penner took inspiration from a paper called Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination, which effectively pre-filtered subsurface scattering into the normal maps it produced by generating separate normals for red, green, and blue channels:

You can think of it this way: by “flattening” the red channel more than the blue, the red light will appear to have more diffusion through the bumps. Instead of requiring three seperate normal maps, Eric instead proposed sampling from the same normal map 3 different times with 3 separate mip bias amounts (one for each color channel). This provides a similar result without requiring seperate per-channel normal maps, and also allows you to use the same normal map for specular shading.

With large and small-scale curvature accounted for, we now have to look at how we handle cases where the incoming lighting changes due to shadowing. Penner’s approach to solving this was to try to determine a pixel’s location relative to the start and end of the shadow penumbra, and us that location to index into another lookup texture containing pre-integrated scattering values for various penumbra widths:

One catch here is that you can’t just apply this penumbra scattering in only the penumbra region: the scattering should actually extend past the point where the penumbra reaches zero. This is why the above diagram remaps the penumbra into a smaller area (essentially narrowing it), and then has the red “scattering” zone where the penumbra reaches zero.

With all of these cases handled Penner was able to achieve some great results! This is quite remarkable, since this can all be achieved within the local context of a single pixel shader. This makes it possible to implement as a special shader permutation within a forward renderer, without having to implement additional passes to handle diffusion in texture-space or screen-space. It was for this reason that we shipped with a variant of this technique in both The Order: 1886 and Lone Echo, both of which use a forward renderer:

While the results speak for themselves, we also encountered many challenges and limitations:

Please note that I’ve spent a lot of time working with pre-integrated SSS, since it’s what we’ve always used in our engine at RAD. So it’s likely that the length of this list is a reflection of that, and that the list of downsides for the other techniques would be longer if I had more hands-on experience with them.

• In practice, we found that using quad derivatives to compute the surface curvature produced poor results in some cases. Eric mentioned this in his presentation, but quad derivatives will always produce values that are constant cross a triangle (you can actually see this quite clearly in the image that I used earlier). This means that you get a discontinuity at every triangle edge, which can manifest as a disconinuity in your diffuse shading. It’s not always noticeable, but when it is it can be quite distracting. You can actually see some of these artifacts in the images from the powerpoint slides if you look closely: These artifacts could also sometimes become even more noticeable due to vertex deformations from skinning and/or blend shapes, which was a real problem for using this on human faces. For these reasons we almost always ended up using pre-calculated curvature maps instead of trying to compute the curvature on the fly. The results become incorrect under deformation or scaling, which is unfortunate. But we found it was more desirable to have smoother and more consistent results.

• The original presentation and GPU Pro 2 article only explains how to handle lighting from point light sources. This means you’re on your own when it comes to figuring it out how to integrate it with baked lighting from lightmaps or probes, with the exception of the normal map prefiltering which works for any light source. While working on The Order I came up with a solution that worked with our (at the time) L2 spherical harmonics probes, which we explained in the course notes from our SIGGRAPH 2013 presentation. There’s also things you can do with spherical gaussians, but I’ll go into more detail about that in a follow-up post. Either way screen-space and texture-space diffusion can handle this naturally without any special considerations.

• I haven’t thought about it too much, but as of right now I have no idea how you would integrate this technique with area lights using LTC’s or similar approaches.

• Relying on a lookup texture can potentially be undesirable from a performance and shader architecture point of view. On some hardware it can definitely be preferable to do a pile of math instead of a texture sample, but you can’t do that without fitting some kind of approximate curve to the pre-computed data. Angelo Pesce took a stab at doing this after the original SIGGRAPH presentation, and he shared his results in two blog posts.

• For any particular diffusion profile, the technique requires two 2D lookup textures (or two fitted curves). This makes it difficult to support arbitrary diffusion profiles, since it would require generating custom lookup textures (or curves) for each material with a unique profile. Supporting a spatially-varying profile is even more challenging, since you would now need to interpolate between multiple textures or curves. Storing them as a bunch of 2D tables within the slices of a 3D texture is a doable (but expensive) option for achieving this. In a follow-up post I’ll talk a bit about a technique I came up with that side-steps this particular issue.

• I’ve found the proposed solution for handling scattering within shadow penumbras to be difficult to work with in practice. In particular I’m not sure that there’s a reliable way to estimate the penumbra width. Quad derivatives tend to produce poor results due to the reasons mentioned earlier, and the surface slope isn’t enough without also knowing the size of the “penumbra” that’s generated by filtering. And of course there’s not even a guarantee that you’ll be using a box filter for your shadow filtering, since it’s totally reasonable to use a Gaussian or some other kernel. Techniques that rely on dithering and TAA resolve would also completly break this, as would any techniques that approximate translucent shadows from things like hair (we’ve always done some tricks with VSM/MSM to approximate translucent shadows from hair). Even if you are able to perfectly reconstruct the penumbra size and the pixel’s position within it, you’re not even guaranteed that the penumbra will be wide enough to give you the appropriate scattering falloff. The Order was filled with close-up shots of faces with tight, detailed shadows.

• Using pre-integrated diffusion based on curvature and \( N \cdot L \) means you’re making the assumption that the visibility to the light source is constant over the sphere. This is obviously not always true when you have shadows. In fact it’s not even true for the simple case of a sphere, since a sphere will always shadow itself where \( N \cdot L \) is less than 0! Wide shadow penumbras and the penumbra diffusion approximation can hide the issues, but they’re always there. The worst case is something with relatively high curvature (or scattering distance) but a very tight penumbra. Here’s an example: What’s really ugly about this particular scenario is that the artifacts reveal the underlying topology of the mesh. We normally shade a virtual curved surface formed by smoothly interpolating vertex normals, but the depth rasterized into a shadow map follows the exact surface formed by the triangles. We often ran into this artifact on our faces when using tight shadows, particularly if the curvature map had too high of a value.

• The proposed technique for handling small-scale scattering through bumps in the normal map is implicitly making some assumptions about the size of the features in said normal map. In general it works best when the normal map only contains small pore-level details, but in games it’s common to also bake larger-scale curvature into the normal map when transfering from a high-resolution sculpt to an in-game mesh. You may find that you get better results by separating these two things as “base” and “detail” normal maps, and only applying the pre-filtering to your detail map. We switched to doing this for Lone Echo, and while it’s more expensive we prefer the higher-quality results. It’s also possible to use a smaller tiling detail map instead for noisy pore-level details, which can certainly save some memory.

• Like the other techniques mentioned in this article, you’ll need additional tech if you want to handle diffusion through thin surfaces like ears. Our artists would sometimes try to fake this with pre-integrated SSS by making the curvature abnormally high, but this has the effect of flattening out the front-facing lighting (since it essentialy assumes the light has diffused everywhere on the sphere).

Conclusion

Hopefully this post has helped give you a basic understanding of how subsurface scattering works, and how it can be approximated in real-time graphics. Perhaps it will also give you enough information to make a preliminary decision about which technique to use for your own real-time engine. If you were to ask me right now to choose between the 3 techniques that I discussed, I would probably go with the screen-space option. In general it seems to have the least corner cases to handle, and most closely resembles a ray-tracing approach. But that could also be a bit of “the grass is always greener” syndrome on my part. I also fully expect a few games to start using ray-traced SSS on high-end hardware at some point in the near future, since you can use that for high-quality results while keeping a screen-space path as a fallback for weaker hardware. You could also do a hybrid approach similar to what DICE did for reflections in Battlefield V, where you use screen-space for most pixels and only use ray-tracing in areas the necessary data isn’t present on the screen.

Please feel free to reach out in the comments or on Twitter if you have questions, or if you spot something incorrect that I should fix!

…or not. It turns out that fp16 is still useful for the reasons it was originally useful back in the days of D3D9: it’s a good way to improve throughput on a limited transitor/power budget, and the smaller storage size means that you can store more values in general purpose registers without having your thread occupancy suffer due to register pressure. As of Nvidia’s new Turing architecture (AKA the RTX 2000 series), AMD’s Vega (AKA gfx900, AKA GCN 5) series¹ and Intel’s Gen8 architecture (used in Broadwell) fp16 is now back in the desktop world. Which means that us desktop graphics programmers now have to deal with it again. And of course if you’re a mobile developer, it never really left in the first place. But how do you actually use fp16 in your shader code? That’s exactly what this blog will explain!

Before we get into the exact details, you should know that there’s actually two parallel paths to using fp16 math in your shaders. This article will cover both, so you can decide which works best for your situation.

Flexible Precision (AKA Minimum Precision, AKA Relaxed Precision)

This is the older of the two fp16 paths, and has actually been around in D3D11 since Windows 8. OpenGL ES has also had a variant of this available for a long time. On this path, what you’re basically doing is giving hints to the driver that say “it’s okay if you do these operations at less than full 32-bit precision, but it’s also okay if you do them at full precision”. This basically gives you runtime polymorphic shaders: your final shader only exists as 1 blob of compiled bytecode, but depending on the hardware and driver you may or may not get fp16 ops at runtime. The nice part is that this spares you the pain of having to compile your shader twice, which is great! The downside is that you’re not really sure what’s going on with the hardware unless you check device caps, and/or check the actual ISA generated by the driver.

When working with HLSL, the way you get this behavior is by using the min16float type and its variants (min16float2, min16float3x3, etc.) ². By declaring this type for a variable you’re providing a hint specifying that it’s okay for the driver to store it with less precision, as long as it’s greater than or equal to 16 bits. In general it works exactly as you’d expect: you can downcast from fp32 to to fp16 by casting with min16float(), and you’ll get warnings if an assignment causes an implicit truncation:

float fp32 = DoSomething();
min16float fp16_x = fp32;               // truncation warning
min16float fp16_y = min16float(fp32);

HLSL allows you to overload functions based on the input type being float or min16float, which means you can create dual versions of your utility functions for fp16 or fp32 when necessary without having to rename them:

float DoSomething(in float x)
{
    // do some stuff in fp32
}

min16float DoSomething(in min16float x)
{
    // do some stuff in fp16
}

If you enable warnings as errors (which you probably should!) you can start out by converting a few high-level values to fp16 and then letting the compiler point out all of the places where you need to explicitly cast to min16float. Unfortunately it won’t catch places where you go from fp16 to fp32, since that doesn’t truncate. So you’ll need to watch out for those on your own. You’ll also need to make sure that you’re not inadvertantly using the half datatype, since by default this is still mapped to fp32 in HLSL! I would recommend making some defines in a common header file that map half to min16float, since that lets you avoid that potential issue. It also makes your code cleaner to look at IMO, and makes it easy to globally disable fp16 should you need to:

#define half min16float
#define half2 min16float2
#define half3 min16float3
#define half4 min16float4
#define half3x3 min16float3x3
#define half3x4 min16float3x4
// keep going if you need more matrix types

You’ll also need to be careful with literals, which are a little weird when working with the min16float types. In HLSL you have 3 floating point suffixes: f, d, and h. These map to the float, double, and half types respectively. Sadly there’s no literal for min16float, and the h suffix maps back to fp32 (since half is mapped to float internally). So for literals you’ll instead want to use an unadorned literal (leave off the suffix entirely), and let the compiler sort it out on assignment. The unfortunate side effect of doing this is that calls to overloaded functions can resolve to the fp32 version if you pass a literal:

float DoSomething(in float x, in float y)
{
    // do some stuff in fp32
}

min16float DoSomething(in min16float x, in min16float y)
{
    // do some stuff in fp16
}

min16float x = 2.0;
min16float y = DoSomething(1.0, x);     // truncation warning!

Fortunately you can work around this by explicitly casting the literal to 16-bit type:

min16float y = DoSomething(min16float(1.0), x);     // no warning

There’s one more big gotcha with flexible precision: you can’t store them in buffer resources like constant buffers and structured buffers. It makes sense if you think about it: it’s okay for temporary values stored in registers to be runtime polymorphic, but it would be really bad if the size and packing of your constant buffer changed depending on the video card and driver you were using! So you’re stuck using 32-bit types for those cases. Ideally you would want to store fp16 values in your buffers, since that gives you more compact data and also lets you avoid any cost there might be from converting fp32 to fp16 in the shader core. But your only real option for doing that is to pack the fp16 values in a 32-bit type, and then convert back and forth in the shader:

struct CBLayout
{
    uint xy;    // two fp16 values packed in the upper and lower 16 bits
};

ConstantBuffer<CBLayout> CB;

min16float x = min16float(f16tof32(CB.x & 0xFFFF));
min16float y = min16float(f16tof32(CB.y >> 16));

Really you’re asking the compiler to convert from fp16 -> fp32 and then truncate back down to fp16, and hoping that the driver’s JIT compiler is smart enough to remove all of that when it generates the final ISA. Tom Hammersley’s post on GPUOpen suggest that AMD’s drivers are capable of recognizing this pattern, but I have no idea how reliable that is in practice across all vendors.

If you’re using the open source DirectX Shader Compiler (DXC) to compile your HLSL, then all of this works in both D3D12 and Vulkan. When compiling to DXIL, you’ll see the compiler emit code that uses the half data type. It will also mark the shader as requiring the “minimum-precision” feature. When compiling to SPIR-V, you’ll see that ops involving min16float are decorated with RelaxedPrecision, which is the SPIR-V version of minimum precision. FXC also supports the minimum precision types for D3D11 and D3D12, if you’re unfortunate enough to still be using that (which probably applies to most of us).

If you want to check what the hardware supports at runtime, in D3D11 you can query the device’s minimum precision support by calling ID3D11Device::CheckFeatureSupport and passing D3D11_FEATURE_SHADER_MIN_PRECISION_SUPPORT to get back a D3D11_FEATURE_DATA_SHADER_MIN_PRECISION_SUPPORT struct with the caps. In D3D12 it’s similar, except you want to ask for D3D12_FEATURE_D3D12_OPTIONS and look at the MinPrecisionSupport member of the returned D3D12_FEATURE_DATA_D3D12_OPTIONS structure. Unfortunately these values aren’t really a guarantee: the driver is free to choose what precision to use for any particular operation even if it reports fp16 support. So you really need to use an IHV disassembly tool if you want to be 100% sure of what your GPU is doing. As of right now this value reports 16-bit support for Vega or later on AMD for both D3D11 and D3D12. Intel also reports that it supports fp16 on my Gen9 integrated GPU. Meanwhile for my Turing-based RTX 2080 the driver reports full fp16 support through D3D11, but not through D3D12. Strange!

Unfortunately for Vulkan there’s no caps or extensions to indicate how the hardware will interpret RelaxedPrecision operations. This means that IHV tools or documentation are your only means of determining whether or not your operations will actually execute at fp16 precision.

When targetting Vulkan you also have an additional path for targetting fp16 through spirv-opt, which is the standard SPIR-V optimizer used by both glslc as well as DXC. Very recently a new --relax-float-ops pass was added to spirv-opt, which automatically tags everything with RelaxedPrecision. More details are provided in this whitepaper. This pass can also be enabled when using DXC by using the -Oconfig command line argument to invoke additional spirv-opt passes. I’m not sure how actually useful this is in practice, since you’re generally going to want to be doing certain things at fp32 when they actually require that amount of precision.

Explicit FP16

Like the name suggests, with this path you’ll be writing code that explicitly works with fp16 data types instead of polymorphic types. Being explicit has its advantages: you know for sure that it’s going to be run at exactly 16-bit precision without having to query device caps. But the major downside is that your shader will now only work on devices that support the related features and extensions. This means you’ll most likely need to compile shaders with and without fp16 types, at least until fp16-capable hardware is ubiquitious on desktop. The other major advantage of being explicit is that you can actually use fp16 data types in your resources, which means you can pack fp16/uint16 data in your constant buffers and structured buffers without needing a pile of ugly code to unpack and convert from 32-bit types. That’s not only convenient, it also makes it easier for you and the driver to avoid unnecessary conversions when performing fp16 math.

Explicit fp16 is only supported in DXC through Shader Model 6.2, which means there’s no support for FXC or D3D11. To compile your shader for explict fp16 you’ll want to pass -enable-16bit-types as an argument and make sure that you’re using one of the *_6_2 profiles (or higher). Once you flip this switch, the half type stops behaving as a float and instead acts as a true fp16 type. They’ve also added a new float16_t type that you can use as well, along with matching float32_t and float64_t types. Here’s simplified version of the table from their wiki showing how each type behaves with and without the switch:

As you can see you also get 16-bit signed/unsigned integers with this flag, which is great for packing more data into your buffers. The compiler also convieniently maps the min16float/min16int/min16uint minimum precision types to their native 16-bit counterparts, which can simplify porting older code. Just be aware that the compiler will emit a warning in this case, which is meant to remind you that you’re no longer getting the “flexible precision” behavior that those types normally provide.

Like I mentioned earlier, in this mode half is back to representing true fp16 values instead of being sliently mapped to the float type under the hood. This also means that the h suffix for literals actually works the way you want it to, which lets you avoid the amiguities that unadorned literals cause with overload resolution. In my opinion this results in cleaner and easier to understand code with less surprising behavior. Here’s the example that I showed earlier with functions overloaded by return and parameter types, except this time we’ll use explicit fp16:

float DoSomething(in float x, in float y)
{
    // do some stuff in fp32
}

half DoSomething(in half x, in half y)
{
    // do some stuff in fp16
}

half x = 2.0h;
half y = DoSomething(1.0h, x);  // this is fine!

With explicit fp16 we know that we’re giving up the convenience of polymorphic data types, which means our compiled shader will only run on hardware that actually supports fp16 operations. So how do we check this in our graphics APIs? In D3D12, we do this by calling ID3D12::CheckFeatureSupport and passing D3D12_FEATURE_D3D12_OPTIONS4 to obtain a D3D12_FEATURE_DATA_D3D12_OPTIONS4 structure, and checking the value of the Native16BitShaderOpsSupported member. Note that the DXIL produced by the compiler will look very similar to what you would get when using the minimum precision types. The major difference is that it will be marked with the UseNativeBit16 flag in the metadata, which tells the runtime and driver that the fp16 ops need to be natively supported.

On Vulkan things are unfortunately a bit more complicated, since you need to deal with extensions. First, you’ll want to check if your GPU and driver support the VK_KHR_shader_float16_int8 extension. If it does, you’ll have two extended device properties that you need to check: shaderFloat16 for fp16 support, and shaderInt8 for 8-bit integer support. If shaderFloat16 is set, then you can use native fp16 math operations in your shaders. However this only applies to math operations, and not anything involving I/O with resources! For that you need to check the VK_KHR_16bit_storage extension, which includes 4 new properties that you need to check: storageBuffer16BitAccess, uniformAndStorageBuffer16BitAccess, storagePushConstant16, and storageInputOutput16. These 4 flags correspond to the SPIR-V capabilities outlined in the SPV_KHR_16bit_storage extension, and basically tell you what classes of resources can use fp16 types in them. Nvidia Turing-based hardware currently reports support for shaderFloat16, shaderInt8, storageBuffer16BitAccess, uniformAndStorageBuffer16BitAccess, and storagePushConstant16, but not storageInputOutput16. Meanwhile AMD Vega and Navi-based hardware reports support for shaderFloat16, shaderInt8, storageBuffer16BitAccess, uniformAndStorageBuffer16BitAccess, and storageInputOutput16, but not storagePushConstant16. So basically you’ll want to avoid using half types as inputs or outputs from your shader entry points since storageInputOutput16 isn’t universally supported, and you’ll also want to avoid using 16-bit push constants since storagePushConstant16 isn’t universally supported.

That’s a lot of details, but we’re not quite done yet! It turns out that the original SPIR-V spec had a bunch of instructions included as “extension instructions”, which are documented here. These are mostly transcendental functions like Pow and Sin, as well as the FClamp instruction that’s commonly used for implementing the saturate() intrinsic. The original spec for these defined them as only taking 32-bit floating point inputs, which meant it was illegal to use them with fp16 values. The SPV_AMD_gpu_shader_half_float extension from AMD lifted this restriction, allowing AMD hardware to support these instructions with fp16 values. Fortunately the SPIR-V spec was revised to add full fp16 support sometime after that AMD extension was released, which means the extension is no longer necessary. Or rather I should say it’s almost unnecessary, since the interpolation instructions still only support fp32 (the AMD extension registry has been updated to reflect this). ³

When using DXC to compile HLSL to SPIR-V with -enable-16bit-types you should see ops generated that use types declared like this:

%half = OpTypeFloat 16
%v3half = OpTypeVector %half 3

…as opposed to seeing ops with the RelaxedPrecision tag. You’ll also see ops indicating which extension features are required, such as OpCapability Float16 for general fp16 math support and OpCapability UniformAndStorageBuffer16BitAccess for using 16-bit types in uniform or storage buffers.

Conclusion

Hopefully this gives you enough information to decide on how to move forward with fp16 in your HLSL codebase. As of right now using the flexible precision path seems to be the only reasonable choice for targetting PC hardware, since fp16 math is only supported on very recent video cards. However it seems clear to me that explicit fp16 is going to be future, since it’s much nicer to work with once you can use it. The only question is how long it will take until we can safely ignore all hardware that can’t do it. I’m not the kind of person to predict the future, so I’ll leave that part up to someone else. 😄

For the past 11(!) years I’ve hosted my blog on wordpress.com, which had a very low barrier to entry. But for a year or two now I’ve been itching to move on to something that I actually enjoy working with, and that doesn’t leave me locked in to the WordPress platform and hosting services. Over the past two weeks or so I’ve been working on setting up this new blog, which is hosted on GitHub pages and is generated as a static site using Hugo. If you’re not familiar with the term “static site”, what this basically means is that I use the Hugo executable to convert source markdown files into pure HTML files that I can load onto whichever host I want, without any crazy scripting or server-side stuff to generate that HTML on-the-fly whenever a user loads the web page. From my point of view I get the following benefits:

• I write the posts using markdown in my text editor of choice instead of fighting with a WYSIWYG editor (with fast live reloading provided by the Hugo local server)
• The site that people see is clean and fast to load/render, with minimal javascript
• I can put the generated HTML on whichever host I please
• I can choose from a wide selection of themes with cool features, and customize the theme and individual posts by dropping down to custom HTML/CSS/Javascript if I need to
• I don’t have to pay money just to keep my blog from serving viewers with ads that have potentially malicious code embedded in them

From now on, all new posts will be hosted here on GitHub pages. While I’m sure it’s annoying for everyone to update their RSS feeds and such, hopefully you all can appreciate the benefits on my end. I have no plans to shut down the old blog, so any old links and such will continue to work until WordPress decides to shut down their free hosting. The only thing that will change over there is that I’m going to stop paying for the ad-free version since I don’t want to keep paying for that indefinitely.

For the rest of this post I’m going to describe my experiences with migrating my blog to Hugo and GitHub Pages. If you’re not interested in that, feel free to move on. Just make sure to update your RSS feed first!

Setting Up Hugo and GitHub Pages

Hugo is actually really easy to work with, at least if you’re a programmer like me. It’s all contained in a single executable that you can download from the releases page on GitHub, so there’s no need to compile things from source or go through a multi-step installation process. Hugo has comprehensive docs that you can read through, and also has a quick start guide that can get you up and running with minimal fuss. I set things up so that I have two git repositories hosted on GitHub:

• A private repo that contains all of the source markdown/theme/image files that Hugo uses to generate the static site HTML
• A public repo that has the generated HTML files, using GitHub Pages to host

The private repo isn’t strictly necessary. I like it because it gives me version control on my source content, and it makes it easy to work on my blog posts across multiple computers. The public repo is necessary for using GitHub Pages, but that could easily use any kind of generic hosting service instead. GitHub Pages is nice because it’s free, and because it uses standard git functionality to push new changes to the live site. Really it’s up to you to decide how you want to manage your source content and host the final generated files.

For LaTeX/math content, I integrated KaTeX by referencing their auto-render extension. This was pretty straightforward despite me knowing next to nothing about JS or web development: the theme that I’m currently using lets you add a partial HTML file that gets added as an extra footer for every post, so I used that to reference the KaTex script. In fact I even got it working so that the script is conditionally included when the post enables a “math” config variable in the front matter, which means that the script only gets loaded for posts that actually need it. LaTeX syntax within markdown has some well-known issues that can make it a bit clunky to work with, but fortunately it’s very easy to have Hugo use the Mmark parser instead of the default parser. Mmark lets you avoid having to escape special characters like _ that frequently pop up in LaTeX, which can save some hair-pulling.

Migrating WordPress Content

This was the hard part of getting things going on my new blog. I strongly considered not converting the old posts at all, and just leaving them on the WordPress-hosted site. But ultimately I decided that it would be for the best to have all of those posts converted to a format that was more easily movable to any kind of hosting. My first attempt at converting my old posts was to run the exitwp-for-hugo python script on the exported XML from my wordPress blog. This script caused a segfault when I ran it (not a Python exception, an actual segfault!), so I quickly gave up on it. I had more luck with the blog2md script, which actually ran and produced some markdown files that I could use. blog2md uses node.js, which I previously had zero experience working with. Fortunately I had an Ubuntu installation that I could access through WSL, which made it relatively easy to install node.js and npm through Ubuntu’s Advanced Packaging Tool (apt). Despite some warnings about a deprecated package, the script produced 80 separate markdown files that roughly converted the content present in the exported XML file that WordPress produced for me. While it was great that this got me started, I unfortunately had to do a lot of manual fixing and cleanup to get things looking presentable. In no particular order, these are the main issues that I had to deal with:

• The WordPress export seemed to be wildly inconsistent with how it handled newlines. I have no idea if this is because of something I did when I typed the original blog posts, but most of the time the exported XML would just contain a newline character in the text instead of an HTML line break (<br>). The library being used internally by blog2md would just straight-up ignore newlines, collapsing many of my posts into gigantic single paragraphs. I worked around this by writing a pre-process script to add a <br> wherever a newline character was found within the text body, which sometimes added more line breaks than needed. But this was better starting point than having no newlines at all.

• WordPress [code] blocks did not get converted by the script, so I went through and manually converted those to markdown code blocks with c/cpp formatting.

• The script was helpful in trying to proactively add the escape character (\) in cases where it would cause issues with special markdown characters like [, _, and *. While this was fine (and desirable) for normal text, it unfortunately applied this to all of my code blocks as well. This meant I had to manually go in and remove the escape characters. This was rather tedious since I couldn’t just blindly search and replace, and could only do it within code blocks.

• Tables from the WordPress export weren’t handled at all, so I had to manually convert them using the extended syntax for tables in markdown.

• Footnotes weren’t handled, but these were pretty easy to fix up by converting them to use the markdown syntax for footnotes.

• All of the images work in the converted posts, but they just point to the images hosted on the old wordpress.com site. For now I’m just going to leave this since I don’t think it will cause any problems in the short term, but ideally I could somehow scrape all of those images so that I can re-host them and make it all easy to migrate in the case that wordpress.com ever shuts down. If anybody knows a thing or two about doing this, please let me know!

• Speaking of images, a few of the posts from my series about GPU barriers used a WordPress slideshow to allow the reader to flip through the synchronization examples. This obviously can’t be converted to markdown, but fortunately I was able to quickly integrate hugo-easy-gallery by Li-Wen Yip. It also helped that I still have local copies of all of the source images that I generated for the slideshow. 😃

Comments

A functional comments system is probably one of the strongest benefits of an out-of-the-box blogging platform like WordPress or Blogger, and I’ve always liked having them for when people share additional info or find a mistake in my post. Disqus is probably the most popular option for other blogs using Hugo, but I had some strong reservations about embedding that into my site. For a while I considered simply having no comments on the new blog. These days most of the discussion tend to happen on Twitter anyway, but I still wanted to have some sort of option for people who don’t like going on Twitter¹. For now I’m going to try using utterances and see how that goes. utterances basically uses a bot to create an issue on a backing GitHub repo, and then uses the GitHub API add comments to that issue and the contents right in the blog post. This means you get the ability to post rich text, and the backing data is all stored somewhere where anyone can access it and where I can also moderate it if necessary (comment spam was huge issue on WordPress). The downside of course is that you need a GitHub account to actually comment, which is unfortunate. But hopefully that already accounts for a decent percentage of people who might want to read a graphics programming blog. 😄

That’s It

I hope you all like the new site! I’ve got a few posts on the backlog to catch up on, so hopefully I should have some actual new content coming up soon. Don’t forget to update your RSS feed if you’re using one!

When I was growing up, the SNES was easily my favorite console. How could it not be, with top-tier games like Super Metroid and Mega Man X available? I had always wanted to learn how to program for the SNES so that I could develop my own silly games for the hardware that ran my favorite games as a kid. I decided to use this past holiday break as an excuse to finally get started, and I had a great time doing it! I love learning about old hardware and development environments, so this was really right up my alley. However it was a bit tough getting started, so I thought I would share some of my experiences in case it helps to spare others a bit of pain.

Just for context, I really knew very little about any of this before I dove in. I started learning programming around 2005-2006, and my first forays into graphics programming used DX9-capable hardware. This was a full two decades after the 65816 was first released, and well after the era of sprite and character-based display chips that the SNES graphics hardware was spawned from. My only prior experience in hand-writing assembly programs was a single micocontrollers class that I took in college, so I’m hardly an expert in that domain. The point here is that if I can learn how to program this crazy machine, I’m sure that you can too!

Hardware Basics

The SNES has a few main pieces of hardware that are relevant to the programmer:

• The main CPU, which is a clone of a 65816

• 128 KB of RAM, referred to as “Work RAM” or WRAM for short

• Anywhere from 256 KB to 4 MB of ROM data mapped directly into the CPU’s address space, depending on the cartridge.

• The “Picture Processing Unit” (PPU), which is the dedicated sprite and background-drawing processor

• 64 KB of VRAM, which holds tile and map data used by the PPU for drawing sprites and backgrounds. Not directly accessible to the CPU!

• A small bit of memory for storing sprite attributes used by the PPU (position, flipping, tile index, etc.),  known as Object Attribute Memory (OAM). Also not mapped to the CPU’s address space.

• Another small bit of memory for storing the 16 color palettes used by sprite and background tiles, known as Color Generator RAM (CGRAM). Each palette contains 16 15-bit colors.

• A DMA unit that can be used to copy data between memory banks, including  VRAM/OAM/CGRAM. Using this DMA freezes the CPU (they share the same bus), so it’s effectively synchronous.

• A custom 8-bit Sony CPU used as a sound processor, called the SPC700. This guy is hooked up to an 8-channel DSP and 64 KB of its own RAM.

A more complete overview of the hardware can be found on the Super NES Programming Wiki. A full dump of all of the system’s internals can be found in the Nocash fullsnes hardware specification, which is definitely a link you’ll want to bookmark. It has a full list of the system’s memory-mapped registers used for I/O with the custom hardware, and it also has the complete mapping of the CPU’s address space. In my experience it has the most complete description of how the custom hardware works, which is vital for understanding the specifics of how to work with the PPU.

If you look around, you can also find a scanned copy of the original SNES development manual out there. While it’s fun to look at from a historical perspective, it’s honestly not really all that useful for actual development. It doesn’t do a very good job of explaining the more esoteric parts of the hardware, and its filled with mostly-useless “examples” that take the form of of large flow charts. The Nocache docs are a much better reference in my opinion, and much easier to navigate than a bloated PDF file.

The CPU and its Address Space

The main CPU is a variant of the 65816, which itself is essentially a 16-bit upgrade of the 6502. The 6502 really got around in the 70’s and 80’s, so you might be familar with it if you’ve done any programming for the Atari 2600/5200, NES, Apple II or Commodore 64. If you’ve done 6502 programming then you’ll feel right at home, since the operations and addressing modes are largely the same. If you’re not familiar with either of them (I wasn’t) then don’t worry too much, since these processors are fairly easy to understand. To get myself up to speed, I read through large parts of the 65816 Programming Manual. I found it to be thoroughly interesting and useful, since it starts out by introducing concepts for the 6502 and then works its way up through the new additions for the 65816. So if you already know the 6502 you can skip ahead, but if you don’t know either you can get the full package. It’s also written in a way meant to be accessible to a newbie programmer from the 80’s, so it’s pretty good at explaining even the basic concepts that an experienced programmer might take for granted.

In terms of programming 65816 assembly for the SNES, it’s pretty straightforward. All of the normal 65816 opcodes work, so general assembly programming guides like the one I linked are almost totally applicable. Probably the most annoying part of working the 65816 is the fact that you can switch its accumulator (A) and index registers (X & Y) between 8-bit and 16-bit modes by setting or clearing two bits in the status register. These not only affect the math operations that you can do on these registers, but they also affect the width of loads an stores performed with that memory. I tended to find myself having to switch to 8-bit mode fairly frequently in order to store a value to an 8-bit variable in memory, or an 8-bit I/O register. The fact that its a persistent state makes things extra confusing and error-prone, since you can’t tell which kind of operation is being done just by looking at a piece of assembly code. It also makes it really easy to screw things up by accidentally leaving the processor in the incorrect state expected by a macro or subroutine, leading to subtle bugs. I’ve found that some good documentation and conventions can go a long way in avoiding those issues. I feel like having explicit instructions for 8-bit and 16-bit operations would have been much easier from a programmer’s point of view, but c’est la vie.

To get started with writing your program you just need to know where your WRAM is located in order to store and load from memory, and also where your ROM data is located in the address space. Working with ROM is awesome: you just read from it like normal memory, with nothing special required to access it. This might be a big shock coming from modern game development where you typically need to use OS API’s to copy data from disk into RAM before you can actually use. Your code is also stored in ROM and usually you just execute it right from there, which lets you freely intermingle code and content.

To see where things are placed in the address space, you can look at the memory map. However the assembler you use can typically help you out here by letting you name different segments, and then specify which segment variables and data should be placed in when you declare them. One funky thing about the address space is that you’ll see the “system area” (which contains the I/O registers and first 8KB of WRAM) mapped in there multiple times. This is called “mirroring”, and its done to let you access those things without having to constantly switch the data bank register (DBR) or use long addressing mode. The basic gist is that its faster and more flexible for the processor to access things using a 16-byte offset from the 8-bit DBR (forming a full 24-bit address), so ideally you want to work within a single 64 KB bank of memory for as long as you can. The mirroring helps with this, since you can flip through different 32 KB chunks of LoROM and still be able to access the stuff in the system area:

+------+-------------+-------------------------+
| Bank |   Offset    | Contents                |
+----------------------------------------------+
| 0x00 |   0x0000    | System Area             |
|      |   0x8000    | First 32KB of LoROM     |
+----------------------------------------------+
| 0x01 |   0x0000    | System Area             |
|      |   0x8000    | Second 32K of LoROM     |
+----------------------------------------------+
| 0x02 |   0x0000    | System Area             |
|      |   0x8000    | Third 32K of LoROM      |
+----------------------------------------------+
| 0x03 |   0x0000    | System Area             |
|      |   0x8000    | Fourth 32K of LoROM     |
+----------------------------------------------+
| ...  |    ...      |    ...                  |
+------+-------------+-------------------------+

For assembling code and linking together the ROM, I used ca65. ca65 is a macro assembler, which means you can define macros that are functionally similar to C/C++ macros. They can also be used to implement unrolled loops, which is handy for cases where you don’t want to hand-write a dynamic loop for performance or convenience reasons. I’m sure there are other assemblers out there that could do the job, but ca65 seemed to work fine for me. It’s actually part of the cc65 package, which is a full C compiler and linker targeting the 6502 and friends. So while I’m sure you could somehow use that to write C code that runs on the SNES, I’m not sure of the specifics of how that would work in practice. Either way the limited register set and banked memory architecture of the 65816 doesn’t seem to lend itself particularly well to a C compiler implementation, so I steered clear of that. Besides, assembly is what the pros used back in the day! :P

The PPU

The PPU is what you would consider to be the “GPU” of the console, at least in a very rough sense. Its whole job is to make your sprites and backgrounds show up on the screen, and that’s really all it does. If you’re coming from the world of modern GPU’s (like me), some aspects of it will probably seem very foreign:

• There are no “command buffers” or “commands” really. The only way you interact with the PPU is by setting its registers via memory-mapped IO (located in the system area), and by changing the contents of VRAM/OAM/CGRAM.

• You can only interact with the PPU’s registers and memory during VBlank, or when the screen is entirely disabled (force blank). This means that you only have about 1.4ms to update your sprite/background state every frame during gameplay.

• There’s no frame buffer! The PPU will draw your sprites and backgrounds on-the-fly as it’s generating each scanline. This is why you can only modify PPU state during VBlank, and also why there’s restrictions on the number of sprites per scanline.

Since the PPU only has a very limited set of functionality, it’s very particular about the data that you feed it. You need to strictly adhere to what the PPU expects, otherwise you’re just going to get a bunch of garbage on the screen. This can make it quite difficult to work with, especially when you’re just starting out. The PPU section of the Fullcash specs is going to be your best friend for getting this right, since it describes everything in full detail down to each individual bit.

PPU graphics are divided into two parts: backgrounds, and sprites. We’ll talk about backgrounds first.

Backgrounds

Backgrounds are essentially 2D grids of “tiles”  that typically cover the entire screen, where each tile is either 8x8 or 16x16 pixels in size. You’ll also see these tiles referred to as “characters” in the official SNES documentation, which I suspect is a vestige of old text-mode display processors found in early PC’s. The way it works is that you have a “map” stored in VRAM, which is a 2D array of integers that specify the index of the tile that you want to use for each slot of the map. The idea is that you can re-use the same tiles (often called a “tile set”) many times across a given map, which is critical when you consider that 256x256 worth of 4bpp tiles would consume 32 KB of VRAM! The ability to re-use tiles is aided by the fact that you can also flip the tile horizontally or vertically for each map entry, and also specify the palette to use. Careful use of flipping and palette switching could create the impression of a much larger tile set than what you can actually fit in VRAM.

The tiles pointed to by a BG map are either 8x8 or 16x16 in size, and either 4bpp (16 color) or 2bpp (4 color) indexed color. For the 8x8 tile size, things are pretty simple in terms of how the tile index works in the map. Each tile is stored in a swizzled bit-plane format, where a single tile is all contiguous is memory. So you’ll have 32 bytes for tile 0, then 32 bytes for tile 1, and so on. So your map just specifies the index of the tile it wants, and that’s it. For the 16x16 BG tile size, it’s more complicated. A 16x16 tile has to be split up into 4 8x8 sub-tiles, one for each corner. The BG map then specifies the index of the upper-left 8x8 sub-tile. The trick is that these sub-tiles can’t be located contiguously in VRAM, which makes VRAM management rather annoying. Instead, the PPU treats the tile memory as if it were a 16x64 2D array of sub-tiles. The 16x16 meta-tile then needs to be stored so that the top two sub-tiles are on the one row of the 16x64 table, while the bottom two sub-tiles must be located on the next row. Effectively this means that there’s a gap of 14 sub-tiles between the top and bottom portions of the meta-tile, which works out to be 448 bytes (224 words) . Trust me when I say that this is easy to screw up.

Here’s what the placement of a 16x16 tile in VRAM looks like if you were to put its top-left corner in the first tile entry in VRAM:

At the bottom you can see that the bottom-left sub-tile needs to be placed 14 tiles after the previous sub-tile so that it ends up on the next row of the VRAM tile matrix. Therefore you would want to DMA it to an offset of 0x0100 words from the start of the tile table in VRAM (VRAM generally deals with memory in terms of 2-byte words).

Each background layer, of which there are up to 4 at once depending on how you’ve configured the PPU, will typically use a map of 32x32 BG tiles. For an 8x8 tile size this covers 256x256 screen pixels, which is more than enough to cover the entire 256x224 screen resolution. Each background layer has a 10-bit pixel offset that will nudge the whole grid of tiles by specified X and Y values, which allows for smooth scrolling. The map will wrap around in areas exposed by the BG offset, which means if you have a tiled background you can just scroll it forever. In practice most games would constantly be updating the outer edges of the BG map in order to display a level much larger than what’s implied by a 32x32 grid of tiles. They would also scroll some of the layers at a slower rate than the “primary” background layer, which would approximate a parallax effect for things far off in the distance. Another background layer could also be used for a HUD and/or status bar.

Sprites

As we just covered, backgrounds are made up tiles that are implicitly laid out in a 2D grid. So you can’t just specify an arbitrary screen position for any single tile, or have one tile overlap another tile. Sprites (also called “objects”) are different: you directly tell the PPU where to draw the tiles by writing X and Y positions into Object Attribute Memory. So instead of a 32x32 map containing just tile indices and a few attributes, you have a 128 element array containing X/Y positions, tile indices, and a few attributes. Basically sparse tiles instead of a grid. Sprites can be 8x8, 16x16, 16x32, 32x32, 32x64, or 64x64 depending on the value of the OBSEL register, with only 2 sizes available for a given frame. The sizes large than 8x8 are set up as meta-tiles in the same way that BG tiles work: subsequent rows in the meta-tile must be located 16 tiles apart in VRAM. So a 32x32 sprite would have its sub-tiles placed in VRAM as 4 rows of 4 8x8 tiles, with a gap of 12 tiles because the end of one meta-tile row and the start of the next meta-tile row. This can make it annoying to move a single sprite into VRAM, since it means that you need to do it as 4 separate DMA operations.

Development Environment

To get a jump start on making a working SNES ROM, I used libSFX, which was developed by David Lindecrantz. This library is a collection of basic tools combined with a bunch of headers for ca65 that are full of macros and defines that correspond to the particulars of the SNES. It definitely saved me a lot of time that I would have otherwise spent going back and forth between my code and the SNES specs to get the right I/O addresses and bit offsets. It also does all of the magic incantations to boot up the system in the expected state (which is something you have to do on a system with no O/S!), which is pretty helpful. On top of all of that it has a makefile template and config system that makes it pretty simple to get your code assembled and linked into the layout expected for an SNES cartridge.

libSFX also includes a tool called SuperFamiconv, which is a command-line program that convert .PNG’s into a binary format that can be (almost) directly consumed by the PPU. It will auto-generate a palette, carve up your image into 8x8 tiles, and munge those tiles into the 2bpp or 4bpp bitplane format used in VRAM. For backgrounds it can also attempt to automatically de-duplicate the tiles in the image to generate a tileset, and will spit out a BG map in the required binary format. By linking the outputs right into your ROM, they’re immediately available to be DMA’d into VRAM at runtime, which is pretty sweet. The tool is also tied into the makefile template for libSFX, essentially forming a basic content processing pipeline. However you still need to be careful about making sure that your sub-tiles end laid out correctly in VRAM, since SuperFamiconv will end up packing all of its data contiguously. One option might be to pre-assemble atlases of tiles/sprites in a PNG in a way that’s aware of the weird “16 columns of tiles” setup and then copy large blocks at level transitions. For backgrounds larger than 32x32 you’ll also need to do your own on-the-fly updating of the BG map in VRAM like I mentioned earlier. I would also imagine that for any non-trivial game you would want to use a proper tile-based map editor to assemble your levels, rather than trying to rely on SuperFamiconv to de-duplicate things for you.

One wrinkle with libSFX is that it’s really only set up to work on Unix-like systems. Getting something this to work on Windows would traditionally involve Cygwin, which is an experience I normally don’t subject myself to in my free time. However for Windows 10 there’s now an alternative called the Windows Subsystem for Linux (WSL). This essentially allows you to install a real Linux distro and have it interact with the Windows file system and executables, with no VM’s required! I decided that this was as good an opportunity as any to try it out, so I went ahead and installed Ubuntu from the Microsoft Store (wild!) and within minutes I was in a bash shell! From there I just ran sudo apt install gcc/g++/make, and I was ready to clone and build libSFX! I was constantly surprised by how well everything worked, especially for someone like me that has very little Linux experience. What’s really cool is that you can invoke Linux from Windows command prompt or batch file by running “wsl.exe”, effectively allowing you to go in and out Linux from within your scripts. In the end I had one batch file that ran “wsl.exe make” to build the ROM, then another batch file that would launch snes9x with that ROM.

In the CPU section I mentioned that the 65816 can be switched between 8 and 16-bit register sizes, and that it can be a source of much pain. libSFX has a series of “RW” macros that to alleviate some of that pain by tracking the current register state in preprocessor variables. This lets it try to filter out redundant size changes, and also lets you print out whichever state is currently set (or more specifically, the state that the preprocessor thinks is set). Unfortunately being a preprocessor that’s part of an assembler, it can only track state as it assembles your code in top-down fashion. This means that the internal state can and will be wrong any time a branch or jump occurs, and its up to you to give it enough information to keep it from screwing up. In particular you need to strategically use RW_assume in order to inform the assembler of the expected state, and possible also deploy RW_forced as your nuclear option. In practice it kind of feels like this tracking system is more trouble than its worth, but I think I would need to use it longer to form a better opinion. Either way, just make sure that you take some time to familiarize yourself with it before diving in! Many of the helpful macros provided by libSFX will use the RW macros internally, so you’ll need to at least play nice with it at those points in your code. Also, be aware that libSFX will set the processor to a8i16 mode before handing control off to your Main label, and also when calling your VBlank handler.

Debugging

While it’s a bit of a bummer that you can’t really run your game on real SNES hardware without jumping through some serious hoops, the upside is that emulation allows for some really cool debugging functionality. You can do a lot when the entire state of the machine is visible to you! In particular, I would recommend using no$sns, by Nocash. It allows you to…

• View the disassembly of the running code
• Set breakpoints
• Inspect CPU registers and memory
• Inspect I/O register states for the PPU/DMA/etc.
• Visualize tiles in VRAM
• Visualize the BG map
• Visualize sprites in OAM
• Visualize palettes in CGRAM
• Probably more things that I didn’t get around to using!

The tile visualization in particular is really helpful for making sure that you get the layout right. Just make sure that you switch the disassembly mode to the Native 65XX mode, otherwise it will look very odd!

Gotchas and Insights

There were definitely a few things that caused me to lose a bunch of time before I could figure out what’s wrong. I also noticed a few ways to speed up my programming. Here they are, in no particular order:

• I mostly covered this already, but always keep track of your register sizes! This was perhaps my most common source of bugs, at least when I was first starting.

• This was also covered earlier, but make sure that your sub-tiles are in the right location in VRAM!

• Be careful not to omit the “#” when you want to load an immediate. There’s no type-safety to help you if you forget it, in which case you’ll load from that address instead.

• .proc is helpful for scoping constants and labels for a subroutine, ensuring that they don’t clash with constants/labels from your other suboutines. There’s also “proc” from libSFX, which does the same thing but also tries to ensure that the RW state is initialized appropriately. Just be aware that you may need to prefix global defines with “::” when you’re inside of a .proc in order to let ca65 know that you’re referring to something in the global scope.

• Use the direct page! It’s easy to feel cramped with only 1 true register to work with, but working with values in the direct page is fast because the operands are smaller. So you generally want to use it for storing temporary variables that you’re dealing with in a subroutine. Just make sure that you prefix the address with “z:” to make sure that ca65 emits the zero-page version of the instruction. libSFX also sets up a “ZEROPAGE” segment for you in the default config, which you can use for declaring variables that live in the direct page.

• Memory isn’t slow relative to the CPU! I had to keep reminding myself of this, since I’m so used to modern systems where a cache miss might mean thousands of cycles in latency.

• Make sure that you always clear the carry before an adc, and set the carry before a sbc! libSFX defines “add” and “sub” macros that will do this for you, so feel free to use them.

• Enable the debug variable in your makefile you have the linker output a map file that you can use to see the actual addresses of your variables and labels.

Things I Didn’t Cover

• Mode 7 - the infamous Mode 7 is very different from the “normal” background modes, and has plenty of its own weird quirks. Be sure to carefully read through Nocash specs if you’re interested in using it.

• Audio - I haven’t spent any time learning how to program the audio processor, so I would be of no help in that area. I did manage to get some music playing, however. An awesome co-worker made a quick looping track in Schism Tracker, and I was able to convert it into an SPC file using SNESMOD. I was then able to link that into my ROM, and start up the SPC program to play that music!

Links

• fullsnes - nocash SNES hardware specification
• Super Famicom Development Wiki
• Super NES Programming WikiBook
• SNES Development Forum on nesdev.com
• SNES Graphics Information
• 65816 Assembly Programming Guide
• libSFX GitHub Repository
• ca65 User’s Guide
• ld65 User’s Guide
• SNESMOD
• Schism Tracker
• SNES Game Sound System

Good luck, fellow retro developers!

Comments:

matt@pharr.org - Feb 4, 2019

Fun writeup! What did you end up getting running? Any screenshots?

Hey Matt! I basically had a very simple side-scrolling shooter working, where you could shoot a few infinitely-spawning enemies. It featured the main character “Jack” from our Lone Echo game series (I called my game “Super Lone Echo, naturally), so I had been meaning to clear it with my company’s owner before posting images or video. I guess I should actually do that at some point! :P

Part 1 - What’s a Barrier?
Part 2 - Synchronizing GPU Threads
Part 3 - Multiple Command Processors
Part 4 - GPU Preemption
Part 5 - Back To The Real World
Part 6 - Experimenting With Overlap and Preemption

In the previous art_icl_es we took a look at how barriers typically work on GPUs, and then we examined how multiple hardware queues can help with preemption and overall throughput. We then spent some time looking at the support for preemption and multiple queues in actual desktop video cards, and also exploring how they end up getting exposed via DX12 and the Windows 10 driver stack. To help tie everything together, I’m now going to show some results that I gathered from running a test DX12 application on a GPU from each of the major desktop IHV’s. Before I begin, I’d like to urge anyone reading this to take the results with a grain of salt. My test application gathers its timings in a rather unorthodox manner (which I’ll explain shortly), and so I can’t guarantee their accuracy with 100% certainty. It’s also very possible that some of the results here could change with future hardware or driver revisions, so please consider the full context when examining the data.

The Test App

Before I wrote these articles, I decided that it would be fun and educational to examine how different GPU’s respond to the explicit synchronization and multi-queue functionality offered by DX12. Unfortunately, I found it was rather difficult to do this in practice. The IHV’s have some vendor-specific tools for gathering timing data from an offline capture, but they’re not really set up to easily show you the degree to which different workloads were overlapping on the GPU. PIX for Windows has the ability to show overlapping draws or dispatches in its timeline view, but at the time it also wasn’t available yet. It also doesn’t currently show overlapping work from compute queues, and it can only show you timing during an offline analysis process. (AMD now has a really awesome GPU profiler that only shows you overlapping work on the GPU, but also shows you what the GPU is doing in response to barriers! Unfortunately I wrote the test app and blog post back in January of 2017, and RGP didn’t come out until July of that year).

After getting frustrated at my available options, I decided to try my hand at building my own timing system. D3D12 does have a built-in mechanism for tracking GPU execution timings via timestamp queries, which are useful for getting very coarse-grained timing data (for instance, overall GPU frame time, or the time required to complete all opaque draw calls). However they’re pretty useless when it comes to determining thread-level parallelism on the GPU. This is because it’s not specified how timestamps behave with regards to barriers and synchronization. The documentation says nothing about whether a timestamp should wait for all previous draws/dispatches to complete, or whether they should be affected by transition barriers. This means you wrap a draw call with timestamps, you don’t know if the second timestamp will record as soon as the command processor kicks of the draw or if it will record after the draw call is finished. In practice I’ve seen varying behavior from different vendors: in some cases a timestamp would cause a sync point, in others it wouldn’t. This all lead me to my “screw it, I’ll build my own timers” moment.

Fortunately, making my own shader timer wasn’t too difficult…or at least for my use case, which was a toy application where I was willing to bend the rules a bit. Really all I needed was for a shader to write to some bit of CPU-accessible memory at the start and end of the program, and I could have the CPU monitor that memory and record the timestamp. To do this I needed to use a custom heap that had the exact properties that I wanted:

D3D12_HEAP_DESC heapDesc = { };
heapDesc.SizeInBytes = 128 * 1024;
heapDesc.Flags = D3D12_HEAP_FLAG_ALLOW_ONLY_BUFFERS;
heapDesc.Alignment = 0;
heapDesc.Properties.CPUPageProperty = D3D12_CPU_PAGE_PROPERTY_WRITE_COMBINE;
heapDesc.Properties.CreationNodeMask = 0;
heapDesc.Properties.MemoryPoolPreference = D3D12_MEMORY_POOL_L0;
heapDesc.Properties.Type = D3D12_HEAP_TYPE_CUSTOM;
heapDesc.Properties.VisibleNodeMask = 0;
DXCall(DX12::Device->CreateHeap(&heapDesc, IID_PPV_ARGS(&workload.ReadbackHeap)));

The important bits here are  the CPUPageProperty, and the MemoryPoolPreference. Setting the page property to WRITE_COMBINE ensures that the memory is uncached from the CPU point of view, which is important since the GPU will be writing to that memory and not a CPU core. Using MEMORY_POOL_L0 also ensures that the heap will live in system memory (as opposed to device memory on the video card), which means that the CPU can access it. It also makes it more likely that the heap will be uncached from the point of view of the GPU. If the GPU writes to an internal cache instead of directly to memory, it’s likely that the CPU wouldn’t see the write until the cache line was evicted. In the end this hopefully gives us coherent, but slow communication between the GPU and CPU.

The other key component of making this work was figuring out a way have the first thread write its value to the “start” buffer, and having the last thread write its value to the “end” buffer. The API doesn’t make any guarantees about the order in which threads are executed for a dispatch or draw call, so I didn’t want to rely on SV_DispatchThreadID or SV_Position to determine the first or last invocation. Instead, I used atomics on a global counter buffer to figure out an ordered index for each thread:

uint prevValue = 0;
CounterBuffer.InterlockedAdd(0, 1, prevValue);
if(prevValue == 0)
    ShaderStartBuffer[0] = CBuffer.FrameNum;

// Do actual shader workload

if(prevValue + 1 == CBuffer.NumWorkloadElements)
    ShaderEndBuffer[0] = CBuffer.FrameNum;

I wrote the frame number in the start/end buffers so that they would be monotonically increasing counters, thus avoiding the need to clear them frame.

The general process for timing a single frame’s workload goes like this:

• Issue a command to clear the CounterBuffer to 0 using ClearUnorderedAccessViewUint
• Issue draw/dispatch command
• Draw UI to the back buffer
• Tell the direct queue to wait on a fence that we’ll use to tell the GPU to wait for our signal before starting the GPU work from our commands
• Submit command list on the direct queue
• Present
• (Next frame)
• Wait for the last present to finish using a swap chain waitable object (this lines up the CPU thread with the end of VSYNC)
• Signal the fence that lets the GPU start the work that was submitted last frame
• Loop and check the value of the start buffer, and record the time when it changes
• Loop and check the value of the end buffer, and record the time when it changes
• Issue commands for the next frame

The good news is that it worked! After implementing and testing, the resulting timing values seemed plausible and also roughly matched the values measured by timestamp queries (I also checked against PIX later on once it was available, and it confirmed my results). From there, I added a simple framework for adding a series of dispatch and draw calls on both the DIRECT and COMPUTE queues. I also added some settings that allow the user to enable/disable workloads, configure the amount of threads launched per workload as well as the amount of per-thread work performed, and also specify dependencies that end up getting expressed as transition barriers. Here’s what the UI ends up looking like:

Under the per-workload settings, the “Num Groups” slider controls how many threads are launched per workload, in increments of 1024. For compute workloads this just controls the dispatch size, while for graphics workloads it adjusts the viewport size to indirectly specify the number of pixel shader invocations. The “Num Iterations” slider specifies the inner loop count inside of the shader program. The loop body simulates an actual workload by computing some perlin noise using an input and output buffer.

Over by the application settings window, there’s a few global settings available. VSYNC can be enabled or disabled, and a slider lets you zoom in on the timeline. The “Use Split Barriers” setting will cause the app to use split transition barriers instead of standard transition barriers to resolve dependencies. The “begin” barrier is issued immediately after issuing the draw/dispatch that’s being depended on, and the “end” barrier is issued just before the draw/dispatch that has the dependency (if split barriers are disabled, the single transition barrier is issued immediately before the draw/dispatch with the dependency). There’s also a setting to toggle a stable power state to get consistent GPU clocks, as well as a setting the causes compute queue workloads to be submitted on a queue that was created with D3D12_COMMAND_QUEUE_PRIORITY_HIGH.

Results

I tested my app on the 4 GPU’s that I have available on my home PC: and Nvidia Titan V (Volta), an Nvidia GTX 1070 (Pascal), an AMD RX 460 (Polaris, GCN Gen 4), and the Intel HD Graphics 530 that’s integrated into my Core i7 6700K CPU (Gen9, Skylake). I also tested on a GTX 980 in my work PC. For each GPU I adjusted the number of threads launched per workgroup to attempt to account for differences in the number of execution units on each GPU. All Nvidia GPU’s used 8k threads per workgroup, the RX 460 used 4k, and the Intel GPU used 2k. These are the scenarios that I used for testing:

• Default - default settings for the app. All workloads enabled, with dependencies between sequential workloads on the same queue. This shows how well DIRECT/COMPUTE submissions can overlap with separate dependency chains.
• No Barriers - all barriers are disabled, which shows how well workloads from the same queue can overlap
• Three Dispatches - three dispatches are submitted on the DIRECT queue, with the third dispatch depending on the first dispatch. The second dispatch is longer than the first dispatch, which shows sync granularity from barriers.
• Three Dispatches, Split Barrier - same as the above, except a split barrier is used. This can show whether the driver/GPU can use take advantage of split barriers to increase workload overlap for this particular case.
• High-Priority Compute - same as Default, except that the compute queue workloads are submitted on a high-priority COMPUTE queue.

Default

In all cases here we see what we would expect from having barriers between each workload: none of the individuals draws/dispatches overlap with other draws/dispatches on the same queue. However we get very different behavior in terms of how each GPU handles simultaneous submissions on both the COMPUTE and DIRECT queue. The 980 appears to serialize the work from the two queues, completely executing the COMPUTE submission before the DIRECT submission. Meanwhile the 1070 has some overlap, but with curiously large gaps between the dispatches on the COMPUTE queue. We also see that workloads with overlap end up taking longer, suggesting that overlapping dispatches or draws are competing for shared resources. The Titan V shows similar results to the 1070, albeit with some smaller gaps. Intel’s GPU also serializes the two submissions, which we would expect based on the lack of a COMPUTE queue being exposed in GPUView. Finally we have the RX 460, which seems to be capable of completely overlapping both submissions with no weird gaps. This is definitely what we expect based on AMD’s marketing material and developer advice regarding “async compute”.

One thing to note from these results is that the Nvidia and AMD GPU’s all have a noticeable gap following the graphics workload, which is performing a draw to a render target. This suggests that render target barrier results in more post-shader work than a dispatch, perhaps due to ROP and cache activity.

No Barriers

The 980 and 1070 GPU’s show similar behavior in this case: the first 3 dispatches on each queue overlap with each other very well, but the dispatches on the GRAPHICS queue refuse to overlap with the draw call in the graphics workload. This is unfortunate, because it suggests a cost for switching between compute and graphics operations. I tried many situations on both GPU’s, and ultimately I was never able to get a dispatch and a draw to overlap on the GRAPHICS queue. Multiple draws overlap each other just fine (as you would expect), but there seems to always be an implicit sync point when switching from compute to graphics (or vice versa). The Titan V manages better overlap for the 3 first 3 dispatches on the compute and graphics queues, but still has the same issue where the draw won’t overlap with any dispatches. Intel shows similar behavior in this test, with the dispatches overlapping some but no overlap between draws and dispatches. Meanwhile, AMD maintains its title as the “King of Overlapping” by showing no visible sync points whatsoever. Also of note is that the total execution time for AMD went from ~2.5ms to ~1.5ms by removing all of the barriers!

Three Dispatches

This time, the test results are very similar for all of our GPU’s: inserting a transition barrier on the result of dispatch A right before dispatch C causes C to wait for dispatch B to complete execution. This suggests that write->read barrier is implemented using a full pipeline flush on all 3 vendors.

Three Dispatches, Split Barrier

This time we have a split barrier surrounding the second dispatch. So we dispatch workload A, issue the start of a transition barrier, dispatch workload B, then end the transition barrier. Ideally this would allow A to overlap with B and B to overlap with C, but with no overlap between A and C. Unfortunately this doesn’t seem to happen on any of the hardware that I tested. On Nvidia and Intel the split barrier doesn’t seem to make any difference at all, giving the same behavior as issuing a non-split barrier following dispatch B. AMD is the only one that shows any change, but it looks like they just put a full sync point where the start of the transition barrier is issued (right after dispatch A). UPDATE 12/9/2018: as of driver version 18.12.1.1, my 460 now shows the exact same behavior for split barriers as it does for a standard transition barrier. So it looks like now the driver syncs on the “end” of the split barrier instead of at the “begin”.

High-Priority Compute

Using the PRIORITY_HIGH flag on the compute queue doesn’t seem to make much of a difference on AMD and Nvidia. On Intel however it causes the COMPUTE submission to get executed before the DIRECT submissions, indicating that it affects the order in which submissions get serialized. This makes sense given what we know about this flag, which only influences the OS scheduler in cases where the OS is serializing submissions from multiple software queues so that they can be consumed by a single hardware queue.

I put the test app on GitHub, so feel free to play around it or fork for your own experiments!

UPDATE 12/15/2018: I just ran the overlap tests on my Nvidia RTX 2080, and the results are very similar to what I was getting from the GTX 1070 and Titan V.

Preemption Analysis

To wrap things up, I also wanted to include a quick analysis of preemption support for the GPU’s I have available to me. Analyzing preemption is naturally tricky: it mostly comes into play when multiple applications are using the GPU simultaneously, and the exact behavior can vary depending on what those applications are doing with the GPU, and also how “important” they are in the eyes of the OS. These issues lead me to set up a set of very simple experiments where I could observe the preemption behavior using both tools as well as my own eyes. The basic setup worked like this:

• First, I would start up a modified version of DX12 Deferred Texturing demo, and disable VSYNC so that it would run un-throttled. This would saturate the GPU with a constant load, with no gaps between submissions. This gives us a chance to observe whether DWM (the desktop composition engine in Windows) can preempt in order to re-composite and present to the display.

• Next, I would put on my Oculus Rift headset, which would cause the “Oculus Home” app to start submitting 3D rendering commands to the GPU (it runs on Unreal Engine). These submissions would compete with the deferred texturing demo for GPU time, showing how the OS and GPU handle scheduling in the case of two applications performing rendering commands.

• Putting on the headset means that the Oculus compositor service starts up, which is what actually presents frames to the headset. This service can make high-priority submissions to the GPU, which is intended to allow the compositor to continue presenting frames at 90 Hz even if the source VR application is failing to do so (this is very important for maintaining smooth updates from head movements). Thus we can also observe how the GPU handles preemption for a very specialized, high-priority scenario.

First, we’re going to look at how things worked on my primary GPU at the time I ran these experiments, which was an Nvidia GTX 1070. Here’s what the GPUView capture looked like when running Oculus Home at the same time as my unmodified, un-throttled deferred texturing app:

This is what I call the “baseline” scenario, where I haven’t done anything to really screw with preemption. We can see here that the submissions from Oculus Home and BindlessDeferred.exe actually end up on separate hardware queues, with one called “3D” and one called “Graphics_1”. The deferred texturing app is cruising along, generally taking anywhere from 5 to 10 milliseconds to complete a frame. Meanwhile Oculus Home is is taking about 12 milliseconds to complete, which is causing the Oculus service to drop into what they call Asynchronous Spacewarp (or ASW for short). When ASW kicks in the VR app is throttled to 45 Hz, but the Oculus compositor continues to present frames at 90Hz using interpolation techniques to generate the “in between” frames. In other words my GPU doesn’t have enough resources to keep both apps running at 90Hz, but the Oculus compositor is successfully using preemption to ensure that the headset is staying responsive to user input. In this particular case it looks like the Oculus service is submitting its commands to a compute queue, labeled “Compute_0”.

So how about DWM? If we expand that image a bit, we can see the CPU side of DWM’s submissions down below:

So we can see DWM submitting a command buffer, which ends up sitting enqueued for quite some time before it ends up actually executing on the 3D hardware queue. It looks like DWM commands end up getting submitting on a normal graphics/DIRECT queue as opposed to a compute queue, and it also looks like that process is getting bullied out of using the GPU as often as it would like. I was running this on a 144 Hz monitor, and here we see DWM flipping at a rate of around 72 Hz.

For my next scenario I wanted to really turn up the heat to stress the system and see how everything responded. To do this, I modified the deferred texturing demo to over-submit its draw calls by a constant factor. This caused the GPU frame time to increase to about 100 milliseconds, making preemption crucial for maintaining anything resembling responsiveness. Here’s the GPUView capture:

Despite our best efforts at hogging the GPU, preemption and multiple queues seem to be working they way you’d expect in this scenario. The Oculus Home application is still failing to run at 90Hz due to sharing the GPU, but the Oculus compositor is ticking regularly every 11ms or so. This is all good news, but let’s see what happens if we try a different approach:

This time, I modified BindlessDeferred to launch way more threads than it actually needed to when executing the deferred shading dispatch. This caused the frame time to balloon to about 100ms, and from the capture we can see that it took the Oculus Home app down with it. Even worse, the Oculus compositor app also seems to be unable to get the GPU to complete its small amount of work. The compositor submissions seem to be scheduled on the GPU’s compute queue, however it sits there for nearly 100ms before completing. This suggests that our “Big Dispatch” is hogging the entire GPU, and the other submissions are unable to preempt this dispatch while its threads are executing. We see similar results in my final test scenario, which uses a normal-sized dispatch but artificially increases the execution time of a single thread by looping in the shader program:

Once again everyone suffers in this scenario, and I can tell you first-hand that you do not want to put on a VR headset under these kinds of conditions.

To finish things off, let’s take a look at how the same scenarios played out on my AMD RX 460. Here’s the baseline capture:

The AMD captures are a little harder to parse, since the D3D11 and D3D12 submissions end up on the same hardware queue (named “3D” here). This also suggests that this hardware may be worse at allowing Oculus Home and BindlessDeferred to share the GPU, since they’re pretty much taking turns on the graphics queue instead of executing concurrently. However, we do see something rather interesting here if we look at the submissions from the Oculus compositor. The compositor work is executing on a hardware queue named “High Priority Compute”, which suggests that it’s perhaps making use of the “Quick Response Queue” functionality mentioned in AMD’s marketing material. Let’s see how things shake out in our “Many Draws” scenario:

As we feared, the AMD GPU holds up worse (compared to the Nvidia GPU) when it comes to letting the Oculus Home app (darker purple) share the GPU. BindlessDeferred (lighter purple) is taking over 100ms per-frame to complete, and it appears that Home needs to wait for the (single) command buffer submission to completely finish before it can start executing. The Oculus compositor, on the other hand, is executing very regularly. The compute submission comes in every 11ms, and consistently takes about 1.8ms to finish. In other words: Home is nowhere near making 90Hz, but the compositor is making sure that a new time-warped frame comes in every 11.1ms. This allowed things to remain comfortable when I put on the headset…provided I didn’t rotate my head to the sides very quickly. Next, let’s look at the “Big Dispatch” scenario:

As expected, the frame rate of Oculus Home (purple) remains very bad in this scenario. The submission from BindlessDeferred (light green) continues to hog the single graphics queue, forcing Home to wait 100ms or so before it gets a turn. On the bright side, AMD’s High Priority Compute queue continues to deliver the goods! Despite the massive number of compute threads swarming the GPU from the big dispatch, the compositor’s submission continues to run at a steady rate of 90 Hz. Very impressive! But will it hold up in the final “Slow Shader” scenario? Let’s find out!

Indeed, AMD’s high-priority compute keeps working even in the case where very slow shader programs are running on the GPU. The only noticeable difference is that the compositor submission now takes about 2.4ms to complete, but other than that things look good. Ultimately it looks like Nvidia and AMD hardware do a better job in different scenarios, so I don’t think we could say that one is inherently “better” than the other. In the future I’m sure we’ll see new innovations from both IHV’s in this area, so I wouldn’t get too hung up on these specific results.

End of Pipe

That’s it for the series! Thank you to anyone who stuck around through the whole thing, which has now spanned 9 months! I learned quite a bit myself while doing research for these articles, so I’m glad that I kept it going. But I think for my next post I’m going to do a shorter write up that doesn’t take me two years to finish. :)

Comments:

- Dec 2, 2018

Really good article, as always! I think there’s a small typo in “but no overlap between A and B. “, should be C instead of B?

Indeed, that was a typo! Thank you for pointing that out.

Excellent series!

very helpful to me, thank you!

Great articles. I’ve learned a lot about how GPU works in parallel.

[…] Part 1 – What’s a Barrier? Part 2 – Synchronizing GPU Threads Part 3 – Multiple Command Processors Part 4 – GPU Preemption Part 5 – Back To The Real World Part 6 – Experimenting With Overlap and Preemption […]

[…] Part 1 – What’s a Barrier? Part 2 – Synchronizing GPU Threads Part 3 – Multiple Command Processors Part 4 – GPU Preemption Part 5 – Back To The Real World Part 6 – Experimenting With Overlap and Preemption […]

Great series! Very in depth and would love to see more :)

Part 1 - What’s a Barrier?
Part 2 - Synchronizing GPU Threads
Part 3 - Multiple Command Processors
Part 4 - GPU Preemption
Part 5 - Back To The Real World
Part 6 - Experimenting With Overlap and Preemption

Welcome to part 5 of the series! If you’ve read all of the articles so far, thanks for hanging in there! In the last article, we spent a good deal of time talking about how GPU preemption might work for an imaginary GPU. We also talked about how having multiple command processors capable of kicking off GPU threads could potentially be helpful for reducing preemption latency, which in turn can improve the overall responsiveness of a GPU that needs to serve multiple applications.  For this article, I want to get back to discussing real-world GPU’s, and how they deal with preemption in the context of an actual OS and driver infrastructure. I’m mostly going to restrict myself to discussing Microsoft Windows and its associated Windows Display Driver Model (WDDM) and native 3D graphics API’s (D3D11 and D3D12), since these are what I know best and have the most experience in. So I’m afraid you’ll have to look elsewhere if you need comprehensive information about Linux, MacOS, OpenGL, or Metal. I will talk just a bit about Vulkan in a few places, but I’m pretty far from an expert on that topic. I’m also going to only talk about desktop GPU’s, since these are what I deal with almost exclusively in my line of work.

The first part of this article is something of a retrospective on pre-Windows 10 technology, so feel free to skip that if you’re not interested. However I think that it provides a lot of valuable context for understanding why D3D12 is designed the way it is (which is of course one of the central themes of this series!), which is why I included it in the first place. It’s also possible that my understanding of the Windows driver model is incomplete or inaccurate in a few ways, since I’ve only ever been on the application side of things. If you’re a seasoned driver developer or OS engineer and you read something that causes you to make a face, please feel free to reach out and/or correct me in the comments.

The Past: XDDM and WDDM 1.x

When I first started learning the very basics of graphics programming, I was in my junior year of college back in 2005. Naturally I used the very latest in 3D graphics API’s to draw my excessively-shiny spheres, which at the time was D3D9 running on Windows XP. Things were rather…different back then. 3D rendering on the GPU still wasn’t really mainstream in terms of general application/OS usage, and was generally only used by games that expected to run in exclusive fullscreen modes. Technically it was possible to have two different programs simultaneously use D3D9 to render to two different windows, but the extent to which those two programs would cooperate and play nicely with each other could vary depending on the particular video card you were using, as well as the implementation details of its driver¹. This was due to the fact that the Windows 2000/XP display driver model (known as XDDM, or XPDM) mostly let display drivers handle complex issues like scheduling command buffers for execution on the GPU, or managing GPU resource memory from multiple processes. Things were particularly rough on the memory/resource side of things, since the OS and drivers were working through the aging DirectDraw infrastructure.  On top of that you still had the entire legacy GDI pipeline with its own driver stack, which was supposed to interoperate with D3D and DirectDraw.

All of this makes sense when you consider that it all grew organically from a world where 3D GPU’s were still a novelty intended for playing games, and 2D graphics accelerators were ubiquitous. However it was also quite clear that the landscape was rapidly changing by the time that Windows Vista was nearing release. 3D-capable video processors were now found even in low-end machines (thanks to “integrated” GPU’s such as the GMA series from Intel), and the growing programmability of GPU’s was causing intrepid programmers to explore the possibility of using those GPU’s for things outside the realm of 3D graphics for games. OS developers like Microsoft were also considering the idea of using the GPU’s 3D rendering capabilities as a core component of how they displayed UI on their desktops. In other words, GPU’s were really starting to grow up, which meant it was time for the OS/driver stack to grow up along with them. This is where WDDM comes in.

WDDM, which is short for “Windows Display Driver Model”, was the new driver stack that replaced XDDM/XPDM in all Windows versions starting with Vista. WDDM was quite a departure from the earlier models, in that it really started to treat the GPU and its on-board memory as a shared resource arbitrated by the OS itself. For instance, with WDDM the OS was now in charge of submitting command buffers to a GPU, while the driver merely provided hooks for building command buffers in its hardware-specific format, and then executing a command buffer scheduled by the OS. This allows the OS internal scheduler to decide when a particular program’s packet of work should be executed, potentially deciding that one program’s commands are more important than another program’s commands. In a similar vein, a global video memory manager now “owned” the on-board memory of the video card, with the driver again providing the necessary hooks that would allow the OS to move data from system memory to video memory (or vice versa). This helped to unify GPU memory management across all vendors, and removed the need for band-aids like the old MANAGED resource pool from D3D9 by effectively virtualizing all resources. The general flow of commands through the WDDM driver stack looked something like this:

If you haven’t been a windows driver developer for the past 10 years, there’s probably a few steps in there that would make you raise an eyebrow. For instance, what the heck is a “DMA buffer”? It sounds like something fancy that might be related to hardware DMA units, but in reality it’s just what the OS calls command buffers on the kernel side of the user-mode/kernel-mode boundary. I suspect that the “DMA” part of that name comes from the fact that the GPU can directly read from a DMA buffer through a physical address, but I don’t actually know for sure. The step that you might find confusing is the one that’s labeled “Patch DMA Buffer with Physical Addresses”. At the time of WDDM 1.0, GPU’s had simple memory controllers that could only access their memory using physical addresses. This is rather unlike CPU’s, which have the capability to work with virtual addresses that are mapped to physical addresses using a page table. Having the vendor-supplied drivers work directly with physical addresses when dealing with resources like buffers or textures would generally be a Bad Idea for a few reasons, but under WDDM it’s a non-starter due to the fact that the OS’s video memory manager has the ability to move resources in and out of device memory as it sees fit. So to make things work, WDDM requires that the driver submit allocation and patch lists alongside DMA buffers. The allocation list tells the OS all of the resources that referenced by the commands inside the DMA buffer, which lets the memory manager decide which resources need to be resident in device memory. The patch list then tells the OS where all of the resources are actually referenced within the DMA buffer, which the OS then uses to patch the DMA buffer with physical addresses right before that buffer is executed on the GPU.

It should be obvious at this point that there was a whole lot going on behind the scenes whenever an app made a few draw calls! Fortunately D3D application developers only really had to worry about that left column, since everything in the other columns was generally hidden from you. In fact, the way that the D3D10/D3D11 API’s were written didn’t even directly expose the fact that GPU’s were parallel processors being fed by buffers full of command streams. For the most part you could pretend that your Draws and Dispatches were happening synchronously with your CPU commands, and you would still get correct results. The only time the abstraction really leaked through was if you looked at the actual timings of your calls, particularly if you hit a case that required the CPU to wait for the GPU to catch up. All things considered, writing D3D programs really got easier for application developers in the transition from D3D9 to D3D10/11, with some of the reasons why getting spelled out in this document. ² Driver and OS developers weren’t so lucky, and things were certainly quite rocky back during the initial transition from XDDM to WDDM 1.0. But after years of refinements it seems to have proved its worth, as we now take it for granted that the desktop composites its windows using the GPU, or that Chrome can use GPU acceleration for rendering webpages while you’re simultaneously playing Battlefield 1 on highest settings.

While there were a lot of minor updates to WDDM over various Windows versions and service packs, there was one major improvement in Windows 8 that’s worth calling out. Starting in WDDM 1.2, drivers could now specify whether or not they supported preemption of their command buffers, and also what level of granularity they could prempt at. Drivers indicating their supporting by returning two enum values, one for graphics and one for compute. Looking at the compute enum, the possible values seem quite similar to the premption granularities that we discussed in the earlier articles:

typedef enum _D3DKMDT_COMPUTE_PREEMPTION_GRANULARITY {
    D3DKMDT_COMPUTE_PREEMPTION_NONE ,
    D3DKMDT_COMPUTE_PREEMPTION_DMA_BUFFER_BOUNDARY ,
    D3DKMDT_COMPUTE_PREEMPTION_DISPATCH_BOUNDARY ,
    D3DKMDT_COMPUTE_PREEMPTION_THREAD_GROUP_BOUNDARY ,
    D3DKMDT_COMPUTE_PREEMPTION_THREAD_BOUNDARY ,
    D3DKMDT_COMPUTE_PREEMPTION_SHADER_BOUNDARY
} D3DKMDT_COMPUTE_PREEMPTION_GRANULARITY;

As we discussed in earlier articles, finer-grained preemption can make for much nicer user experience when multitasking, especially when a particular program is really trying to hog the GPU.

UPDATE: it turns out that you can actually view your video card’s reported premption granularity by using the old DirectX Caps Viewer! These days its included as part of the Windows 10 SDK, so you can typically find it in C:\Program Files (x86)\Windows Kits\10\bin\\x64\dxcapsviewer.exe. An Nvidia Titan V reports pixel-level for graphics and dispatch-level for compute, while an AMD RX Vega reports draw-level for graphics and DMA buffer-level for compute. Many thanks to Locuza for pointing this out to me!

Problems With D3D11 and WDDM 1.x

I think it’s fair to say that from a D3D application developer’s point of view, things generally got better and easier when comparing the Windows XP era to the Windows 7 era. I’m sure driver and OS developers had plenty to complain about, but certainly in my day job as a game developer I appreciated the increased orthogonality of the API’s as well as the fact that multi-tasking generally “just worked” across a variety of hardware (including multiple GPU’s in the same machine!). But you should never underestimate the ability of a game programmer to complain about something (especially when it comes to Windows!), and we certainly found some things to gripe about. Pretty much all of them revolved around performance and multithreading.

Back when D3D11 was first announced, there was a lot of hype around the fact that it allowed for multi-threaded rendering via so-called deferred contexts. While this turned out to be technically true (it’s totally possible to have multiple threads issuing Draw and Dispatch calls in stock D3D11), in practice it wasn’t as useful as everyone hoped it would be. Probably the biggest roadblock is something that we discussed way back in part 1: it’s really tough (or perhaps impossible) to make implicit dependencies work in an efficient way when multiple threads are generating the rendering commands. If you’ve got to run through the entire set of commands for a frame to determine when a depth buffer transitions from being writable to being readable as a texture, it quickly pushes you down the path of serializing your command lists before final generation of the hardware command buffer. This is definitely not great if your goal was to distribute the cost of command buffer generation across multiple cores, which is really the only reason to use deferred command lists in the first place. So that was a pretty big bummer, and led to games (mostly) sticking to single-threaded rendering on Windows.

Another issue that has sprung up in recent years is D3D11’s inability to leverage multiple command processors/hardware queues on modern GPU’s. As we learned in Part 3, submitting commands into multiple front-ends can potentially lead to higher shader core utilization, which in turn leads to higher overall throughput. We also saw that effectively utilizing these additional command processors requires identifying and submitting multiple chains of commands that aren’t dependent on one another. Having the driver do this automatically is a big ask, especially when data dependencies are implicit rather than explicit. If you think about it, it’s very similar to utilizing multiple CPU cores via threads: the compiler can handle detecting dependencies within a single thread, but it doesn’t extract out independent sequences of instructions that would long enough to justify having the OS spin up a whole new thread. The problem is made even more difficult by the fact that command buffer building and submission is also implicit in D3D11/WDDM 1.x, which means there’s no place for the app to give hints or more explicit input about which front-end a stream of commands should be submitted to. On top of that GPU’s typically have restrictions on which commands can be executed on which command processors (for instance, some can only consume compute-oriented commands), and D3D11 has no way of expressing these restrictions through its API’s.

One last issue that I’ll discuss is less technical in nature, and has to do with the abstraction presented by the API and how it relates to deferred command lists. As I mentioned earlier, D3D11 adopts the model where command buffer recording and submission to the hardware is implicit based on an app’s sequence of D3D11 commands, and is completely hidden behind the driver. This also effectively hides the asynchronous/parallel nature of the GPU, and lets you assume that draw/dispatch/copy commands happen synchronously and still get correct results. Or at least, that’s the case if you’re using the immediate context. Deferred contexts break this abstraction, since their semantics imply that the commands aren’t actually executed right away and are instead executed when ExecuteCommandList is called on the immediate context. This basically works, but it’s also pretty odd when you consider that the immediate context isn’t really “immediate”, and is actually “deferred” with implicit submission. So really a deferred context was more of a double-deferred context! On top of that weirdness, deferred command lists also had some odd restrictions as a result of their user-facing execution model. In particular, the Map operation was restricted to write-only operations, and disallowed anything involving reading. This stems from the fact that commands on a deferred command list can’t be expected to execute (from an app point of view) until ExecuteCommandList was called, which in turn meant that you couldn’t expect to read the results of prior commands. This is actually similar to how things work now in D3D12 (more on that in a minute), but it’s a bit strange that those semantics only applied when doing multi-threading in D3D11. Finally, I suspect that the automatic memory management/versioning happening behind the scenes when using D3D11_MAP_WRITE_DISCARD really stretched things to their breaking point when having to support that in a multi-threaded, deferred context scenario. At some point it just makes more sense to let apps manage their memory and submission using the app’s own knowledge of its particular needs, which leads us to…

The Present: Windows 10, D3D12, and WDDM 2.0

With D3D12 and WDDM 2.0, the API and driver stack has pulled back much of the curtain around how work actually gets submitted to the GPU. Instead of abstracting away the fact that commands are getting recorded to buffers and executed asynchronously by a parallel processor, D3D12 (almost) directly exposes those details to applications. The API gives you an interface for recording commands (ID3D12GraphicsCommandList), another interface for storing the memory into which those commands are recorded (ID3D12CommandAllocator), and a function that allows you to queue a completed command list for execution (ExecuteCommandLists). Of course with this newfound power comes new responsibilities. Since the app is in the driver seat now, it means it has to now be aware of when those command buffers are in-flight so that it can avoid writing to them. This leads to a typical double-buffered “submission loop” where the application waits for the GPU to finish processing an old command buffer before re-using it:

You generally deal with similar patterns all over the place when working with D3D12, since you’re now responsible for all lifetime management when it comes to memory and resources. No more MAP_DISCARD for you when update a constant buffer, instead you have to craft your own mechanisms for grabbing a safe bit of memory. And if you want to read back results from the GPU, you had best make sure that you’ve submitted the relevant command buffer and waited for it to complete before attempting to read that memory on the CPU. Like the barriers that we discussed in the first article, it’s quite a large burden that’s been shifted from the driver to the application. And the shift was mostly done for the same exact reason: to enable better efficiency and multithreading. The natural way to divide up command generation is to have each thread or task generate its own command buffer, and then submit them all as a “chain” when they’re all been recorded. When you combine this with explicit barriers and a single API for specifying pipeline state via PSO’s, you get an API that truly allows you parallelize command buffer generation across multiple CPU cores.

Providing more explicit control over submission also gives apps the ability to take advantage of GPU’s that have more than one front-end for queuing or processing commands. In Part 3, we talked about how adding an additional command processor to a GPU can help improve overall utilization for cases where an application submits multiple distinct dependency chains (commonly referred to asynchronous compute). If we were stuck in the old D3D11 model where dependencies and submission were implicit, it would be very difficult (or impossible) for a driver to be able to extract out a non-dependent chain of dispatches to execute on a second command processor. Fortunately with D3D12 we’re actually in a position to explicitly say “this list of commands can be executed separately, and depends on this other list of commands”, which can be done via command queues and fences. Command queues, like their name suggest, represent a single queue into which completed command buffers can be submitted for running on the GPU. When you create a command queue, you have to specify 1 of 3 different command list types to run on that queue:

1. COPY - can execute CopyResource commands
2. COMPUTE - can execute CopyResource and Dispatch commands
3. DIRECT - can execute CopyResource, Dispatch, and Draw commands

These command list types can potentially correspond to different “engines” on the GPU, which is what D3D/WDDM like to call a command processor. Each engine can have its own distinct set of commands it can run, hence the 3 types of command lists. Typically a discrete GPU will have at least 1 graphics command processor capable of executing all possible commands, and at least 1 DMA unit that can issue copy commands that are optimized for transferring over the PCI-e bus. Many recent GPU’s also have additional compute-only command processors, but we’ll get into that in more detail later in this article.

As an example, let’s return to the post-processing case that we were discussing in Part 3:

To submit our work in such a way that the two dependency chains could be executed on both command processors, we would do something like this in D3D12:

• Create a DIRECT command queue, which we’ll call GfxQueue
• Create 3 DIRECT command lists, which we’ll call GfxCmdListA, GfxCmdListB, and GfxCmdListC
• Create a COMPUTE command queue and command list, which we’ll call ComputeQueue and ComputeCmdList
• Create two fences, FenceA and FenceB
• Each frame:
  • Record all rendering commands for the Main Pass to GfxCmdListA
  • Record all bloom rendering commands to GfxCmdListB
  • Record all tone mapping and subsequent rendering commands to GfxCmdListC
  • Record all DOF dispatches to ComputeCmdList
  • Submit GfxCmdListA to GfxQueue
  • Signal FenceA from GfxQueue on completion of GfxCmdListA
  • Submit GfxCmdListB to GfxQueue
  • Tell ComputeQueue to wait for FenceA
  • Submit ComputeCmdList to ComputeQueue
  • Signal FenceB from ComputeQueue on completion of ComputeCmdList
  • Tell GfxQueue to wait for FenceB
  • Submit GfxCmdListC to GfxQueue

Since these D3D12 commands are very explicit about the dependencies between command lists as well as the kind of queue that the command lists should be submitted to, the OS and driver now have enough information to actually schedule the command lists on separate command processors. This is a major improvement over D3D11, whose implicit submission model didn’t really allow for this to occur.

Based on how the queue and submission API’s work, you may be tempted into thinking that submitting a command list to an ID3D12CommandQueue will directly submit the corresponding command buffer(s) to a hardware queue for a particular GPU front-end. But this isn’t really true in D3D12 and WDDM 2.0. When we were going over the details of WDDM 1.x, we discussed how the OS actually had its own software scheduler that was responsible for deciding when a command buffer should actually run on a GPU, and potentially arbitrating submissions from multiple applications. This is very much still the case in D3D12/WDDM2.0. Consequently you’re not submitting command buffers to a hardware queue, you’re instead submitting them to the OS’s scheduler. In addition to letting the OS handle shared resources between multiple applications, it also allows the OS to “virtualize” the queues for hardware that doesn’t actually support concurrent execution on multiple front ends (in D3D/WDDM terminology we would say that the GPU only has a single engine). If you look at the descriptions of the queue types that I listed above, you may have noticed that they’re set up such that each queue type’s list of supported commands is actually a subset of the next queue type’s supported commands. In other words, the DIRECT type can do everything you can do on a COMPUTE or COPY queue, and COMPUTE can do everything that you can do on a COPY queue:

With the queue/engine functionality being specified as subsets of each other, the OS can take a submission from a COPY queue and redirect it to a DIRECT engine if it needs to. This saves applications from having to change how they submit their command buffers based on the supported functionality of the end-user’s hardware and drivers. However to pull this off, the OS has to be able to take command buffers that were intended to be concurrent and “flatten’ them down to a serial stream of command buffers. As an example, let’s return to the DOF/bloom case study that we used earlier. If we were to submit that sequence of command buffers on a GPU that only has a single DIRECT engine, the OS will have to flatten the submissions into something like this:

Since the bloom and DOF command buffers are independent of each other, they can be executed in any order with regards to other and still give the same results. However you can tell the OS scheduler to favor one queue’s submissions over the other by creating the queue with a higher-than-normal priority. We should also note here that the OS needs to be fully aware of inter-queue dependencies in order to do this kind of flattening operation, which is why fences are heavyweight OS-level constructs instead of being lightweight sync operations. This is also why you can’t perform cross-queue synchronization within a command list, which would be convenient for applications but would break the ability of the scheduler to flatten multi-queue submissions. One benefit gained from giving the OS global knowledge of submission dependencies is that it can detect and prevent certain deadlocks in advance., which is nicer than allowing the system to choke.

If you’re ever curious to see what the OS and scheduler are actually doing with your command buffers behind the scenes, the best way to do so is to use GPUView (which is most easily done by using Bruce Dawson’s invaluable UIforETW tool). Here’s an example capture from an submitting a bunch of work to both a DIRECT and COMPUTE queue running on an AMD RX 460:

This screenshot is showing about 4 frame’s worth of capture data, which you can tell by looking at the activity in the flip queue. The darker area shows where the GPU is idle while waiting for VSYNC, so we can use that denote where the frame begins and ends. Towards the bottom we have the “device contexts”, which is showing command buffers that have been submitted on the CPU side of things by a single process. The top one shows the compute submissions, while the bottom one is the graphics/direct submissions. All of these command buffers end up in the hardware queues at the top of the image, which is what we’re interested in. Here we see there’s two hardware queues, one called “3D” and the other called “COMPUTE_0”. By looking at the activity on these queues we can see that they’re both processing submissions simultaneously, which suggests that the work from each queue will overlap. Ultimately what we’re seeing here is pretty similar what I was illustrating in Part 4 with the GPU/Driver timelines, which means we can also use this tool to inspect preemption behavior from multiple applications.

For those of you using Vulkan, be aware that their queue/submission model does not work like D3D12’s! In Vulkan, every physical device can be queried to discover which queue “families” it supports (a queue family is analogous to an engine type in D3D12), as well as how many of each queue family is available. You can then pass handles to those exposed queues when creating your logical device, which then allows you to submit command buffers to those queues at runtime. This means that if a GPU only has a single front-end, you will probably only see a single queue exposed on the physical device.

There’s one more improvement in WDDM 2.0 that I wanted to quickly mention: remember how I described the process for patching command buffers with physical addresses in WDDM 1.x? That’s no longer necessary in WDDM 2.0 thanks to its added support for GPU virtual memory. With the new driver model, each process gets its own virtual address space for GPU memory, just like they do for CPU memory. The app and driver can freely embed these virtual addresses directly into command buffers or ExecuteIndirect argument buffers without having to go through an expensive patching process right before execution. This change also allows the user-mode driver to directly build and submit a GPU-accessible command buffer instead of requiring the kernel-mode driver to create a DMA buffer. This is a nice performance improvement, and I would imagine it’s also an improvement from a security point of view. When using the D3D12 API you often work directly with GPU virtual address for things like creating vertex buffer views or setting root CBV’s/SRV’s using a single pointer, which is pretty convenient for apps. Vulkan on the other hand doesn’t expose GPU virtual addresses (likely because it has to run on platforms and hardware without full virtual memory support), so it will instead work with allocation/offset pairs.

A Quick Look at Real GPU’s

Before we wrap up this article, I wanted to also spend just a bit of timing looking at real-world GPU’s and their support for multiple command processors. In Parts 2, 3, and 4 we used my made-up GPU to look at how the basics of preemption and overlapped processing worked. But I wanted to also go over some real-world examples to see how they compare to my overly-simplified example, and also how they relate to D3D12’s support for multi-queue submission.

Let’s start out by looking at GPU’s from AMD’s. Take a look at this slide from one of AMD’s presentations:

In this diagram there’s one “GFX” command processor, which is is the only command processor capable of issuing Draw commands that use the primitive and pixel pipes. It can also issue Dispatch calls that use the CS pipeline to access the shader cores. In the top left there are a few units labeled “ACE”, which stands for “Asynchronous Compute Engine”. These guys are basically an array of simplified, independent command processors that can only process compute-related commands. This means that they can issue Dispatch calls and perform synchronization-related commands, but they can’t execute Draw commands that utilize the graphics pipeline. In other words, they exactly fit the required feature set of a COMPUTE engine in D3D12, which is obviously not a coincidence. In terms of functionality they’re also rather similar to the dual command processors of my fictional MJP-4000, and they can provide the same benefits for preemption, multitasking, and increased utilization from overlapping multiple workloads. In fact, if you read some of AMD’s documentation you’ll see them explain some of the same concepts that I was discussing in my earlier articles.

In practice the ACE’s are a bit more sophisticated than what was discussed in my examples, particularly due to the fact that each ACE contains up to 8 hardware queues for feeding command buffers to the ACE’s command processor. AMD’s processors support up to 8 ACE’s on a single chip, potentially allowing for 64 different command streams to be in-flight simultaneously! The separate queues support various scheduling and synchronization operations, effectively allowing them to serve as a very simple hardware task scheduler. These ACE’s have been present through every revision of the GCN architecture, including the recent RX Vega series. AMD has also mentioned a few improvements over the course of 4 architecture revisions. Most notably for their semi-recent Polaris architecture, the ACE’s were augmented with what they call a “Quick Response Queue”.  Their documentation suggests that this allows ACE’s to submit high-priority workloads that take priority over work submitted from other command processors, which should allow for the kind of thread-level preemption that I showed in own preemption examples. Polaris also introduced what they called a “Hardware Scheduler” (HWS), which is described as additional independent processors that can be used to create a layer of virtualization around the physical hardware queues.

UPDATE: It turns out that the Quick Response Queue has been around since GCN2, and HWS has been around since Fiji/GCN3. Thanks again to Locuza for correcting me on that.

This information suggests that we should see at least 1 compute engine exposed to Windows scheduler, as well as at least 1 queue with COMPUTE_BIT set being exposed on the physical device through Vulkan. The D3D12 side can be verified through experimentation by using ETW captures and GPUView (see the image that I posted above showing a COMPUTE submission being executed on a hardware compute queue), but for Vulkan we can verify very easily using the Vulkan Hardware Database. Here’s what we get when looking at the “Queue families” tab of an RX Vega:

It looks like AMD chose to expose their ACE’s to Vulkan as 8 compute queues, which is plenty for an individual application. We can also see 2 queues with TRANSFER_BIT specified, which is Vulkan’s version of a COPY queue. These correspond to the hardware DMA units that are present on an AMD Vega, as well as on virtually every discrete PC video card in the wild. Like I mentioned earlier, DMA units on GPU’s are specifically optimized for transferring batches of data across the PCI-e bus into on-board GPU memory. The typical use case is read-only textures and buffers, which need to live in on-board memory in order for the GPU to be able to read from them with full bandwidth. Drivers from the pre-D3D12 days would use the DMA to perform initialization of resources with app-specified memory, since the DMA unit is optimized for transferring from system memory, and also because it can execute concurrently with graphics operations. DMA units can also transform textures into hardware-specific layouts, which typically involves using some variant of a Z-order curve to obtain better 2D cache locality. With D3D12 and Vulkan you’re of course on your own when initializing resources, and a COPY queue is the natural choice for doing this on discrete GPU’s.

As for Nvidia, their hardware also appears to feature functionality that allows for simultaneously submitting multiple workloads to their GPUs. Their Kepler architecture introduced what they refer to as Hyper-Q, which sounds pretty similar to the ACE’s on AMD GPUs: