This blogpost will cover how we set up our dynamic diffuse global illumination system in our rendering framework at Traverse Research. It’s completely ReSTIR based, but we won’t be focussing as much on how ReSTIR math works, instead we’ll focus more on the tricks, techniques and things we tried with setting up this system.

Performance breakdown

Timings were measured on an RTX 3080 running at a native resolution of 1440p while running the scene Flying world — Battle of the Trash god.

Camera position: [7.8699145, 2.2485592, -7.960048]
Camera look at position: [7.840394, 2.066986, -8.942983]

In total we use the following number of rays:

• Indirect diffuse: 0.25 diffuse rays/pixel + 0.25 shadow rays/pixel + 0.25 validation rays/pixel
• Irradiance cache: An upper bound of 65535 * (4 diffuse + 4 validation + 8 shadow rays)

Indirect diffuse — 1.97ms

Denoiser — 697μs

Irradiance cache — 473μs

Candidate rays

For estimating indirect diffuse, we start off by tracing diffuse rays at half resolution. If we look at the ReSTIR GI paper, we should consider sampling the hemisphere uniformly rather than cosine-weighted. This will avoid low sampling probabilities on big lighting contributors at grazing angles, which can result in a much higher variance.

As for generating the diffuse rays, it’s generally a good idea to use blue noise. Though combining other noise techniques or combinations of them can also produce decent samples, if for example you want to abuse sampling patterns during spatial/temporal reuse or filtering.

If we’ve hit a surface, we sample the indirect lighting and direct lighting at the hit point. The indirect lighting comes from the irradiance cache in form of L1 spherical harmonics, while the direct lighting is estimated by tracing a shadow ray towards light sources.

Once we measured the incoming radiance from our candidate rays, we store the following information:

• Sample direction
• Hit distance
• Incoming lighting
• Sample hit surface normal

This may seem like a lot of data to store for each candidate, however we can compress this data down to a total of 4 uints. We encode the sample direction and hit normal as octahedron, for the incoming lighting we use R9G9B9E5_SHAREDEXP and finally we store the hit distance as full float.

NOTE: Since we are tracing rays uniformly over the hemisphere, we essentially get “free” ray-traced ambient occlusion, which may be useful to use in subsequent restir and denoising passes!

Temporal reuse

Temporal reuse is the pass where we need to be the most careful with sampling, reprojection needs to be accurate and rejection heurstics need to be relatively strict as all of our subsequent passes and frames rely on this pass. If our reprojected history is valid for reuse, we merge it with the candidate reservoir.

The output is still pretty noisy though as we clamp our reservoirs to a maximum sample count of 12.

A technique that we can use to reduce variance further is permutation sampling, the idea is pretty simple; we use a global jitter and afterwards xor the reprojected pixel index by a value, which results in an ordered spatial exchange of history reservoirs.

This is kind of the same idea as running a spatial search in temporal reuse. However if we were to use naive spatial resampling here, we would get massive correlations between pixels as multiple pixels would be resampling the same neighbor reservoirs, resulting in “clumps” of bright samples. The most ideal pattern seems to be a global xor value that changes every frame in a range of [1..3], though a naive xor of 2 or 3 works fine as well.

We chose to gather multiple permutation samples each temporal reuse pass, while exponentially decreasing the sampled reservoir M clamp. While this does initially make the temporal reservoirs less accurate, it helps a lot with disocclusion. Detail is recovered by using guiding masks and ReSTIR validity frames. Alternatively you can also choose to only run a set of permutation samples when the center reservoir is starved for samples (for example a newly disoccluded pixel).

Since the permutation samples came from a different origin we must make sure that we apply a jacobian here.

We don’t know if the merged reservoirs had a visible sample, naively merging these reservoirs in this case results in loss of contact shadows. The correct way of preventing this would be to simply trace the visibility of these samples, but this is too expensive as we have a very limited ray budget. Instead we can use an indirect shadow guiding factor generated by the previous frame, and reject permutation samples if their “shadow” values differ too much, more on this in the spatial reuse section.

Smaller details and shadows are better preserved temporally with indirect shadow guiding. Not being careful here can result in much more visible disocclusion/movement artifacts in dynamic scenes, which are very difficult to filter.

As for guiding/rejection techniques we’ve tried to use an average sampling direction from the selected reservoirs, SSAO masks or even denoise the RTAO we got for free earlier on. All of which have their advantages and disadvantages.

There may be other ways of steering sampling/rejecting samples, visualizing specific sampling data produced by resampling may give some ideas on what you can (ab)use to refine rejection heuristics!

Spatial reuse

For spatial reuse we run two separate kernels, the first kernel does a large spatial search with a radius around 4% (42 pixels) of the rendering resolution. The second kernel runs the same logic but instead takes less samples and runs at roughly 2%(20 pixels) of the rendering resolution, though this can be smaller depending on how low the sample count of the center reservoir is.

Both these spatial reuse passes don’t do anything fancy besides merging reservoirs that survive the rejection heuristics. In order to reject “bad” reservoirs, we compare normals and measure the tangent plane distance between the center and picked reservoirs. Finally we also apply the jacobian, if the reservoir has survived these checks and does not have a rediculous jacobian, we go ahead and merge the picked reservoir.

Using the tangent plane distance here is essential to avoid excessively rejecting neighbor samples. When only using depth rejection, we tend to get a lot of “boiling” noise on surfaces that have a grazing angle with respect to the view point.

So what happened to all the indirect shadows and details? We do not use guiding masks here to reject reservoir reuse, nor do we test visibility for each reused reservoir. Testing visibility for each reuse candidate is way too expensive if we were to use ray tracing as we would need to trace an additional 10–15 rays per pixel, though there have been some alternatives using screen-space ray marching.

Instead we keep the spatial reuse simple and light-weight. Afterwards we only test visibility for the final surviving reservoir. If the sample is occluded, the result sample will simply have 0 contribution. This is obviously biased and introduces some noise, but in reality provides a nice tradeoff between quality and performance.

Now that we’ve tested the visibility of all of the surviving reservoirs, which tend to be specifically towards indirect light sources with the highest contribution, we may find the results of these visibility tests to be useful for more than just validating reservoir visibility. Let’s see what this actually looks like if we were to treat it like a shadow mask:

This seems to exactly mark the regions where we would need to be extra careful with spatial exchanges and filtering! We definitely don’t want to “smear” out our indirect shadows in subsequent passes and frames, now we can just measure the shadowing difference between pixels and reject based on that.

Ideally we would have a denoised version of these indirect shadows:

However doing an extremely cheap spatio-temporal blur with adding some bias towards shadow hits that have a shallow angle actually appears to be more than enough. We only care about the difference between pixels rather than actually modulating the final indirect lighting.

Additionally, in the future we’d like to explore this idea and try storing the occluded direction instead. Doing so might prove useful as a way to estimate visibility during resampling. This way we can be more specific on what samples get shared rather than only comparing their indirect shadow factor.

Consider the following situation:

In this scenario we would reject sharing reservoirs between green and red, as their guiding factor differs too much. This is a problem since the green sample is actually visible from the red marked surface.

If we were to make the guiding factor consider directionality here, we would still reject any samples that would roughly come from the occluded direction, while being able to reuse samples that are more likely to be visible from other directions.

Resolving to full resolution

Now that we have validated our reservoirs after spatial reuse, we can resolve them to full resolution. We do this by taking 4 samples per pixel in a small screenspace radius. Furthermore we reject samples that differ too much in the indirect shadowing factor, this helps to preserve sharp indirect shadows.

We observe quite a lot of noise still being present here, which is the result of us being cheap about testing visibility. If we compare this to the resolved reservoirs that actually had all their visibility traced during the spatial exchange passes, we may be able to be less aggressive during denoising, however we decided that the additional cost of tracing many more shadow rays is not worth it.

For denoising we use a very basic spatio-temporal filter. We initially tried filtering in YCbCr here, but this gave some undesired “boiling” artifacts in the indirect shadow regions. Instead we we decide to use AABB color clipping to avoid hue shifts when lighting condition changes. Doing so resulted in much more stable indirect shadows.

Once again we use the indirect shadow guiding mask here to manipulate the accumulation rate and history rejection; indirect shadows from occluded visibility rays tend to be significantly more noisy compared to those which were not occluded, we accumulate at different rates based on those properties. If the indirect shadow factor shows a very large change, the accumulation rate gets changed accordingly to be more reactive.

Most remaining noise is then cleaned up with a small bilateral filter.

Once we have the denoised indirect lighting, we simply modulate it with the albedo, let TAA take care of any remaining noise and we’re done!

A quick note on handling alpha tested geometry, we found that masking out alpha tested materials for all the diffuse traces, and enabling alpha testing for the final visibility test produced the best quality vs performance. Limiting the number of alpha tests a ray can evaluate may help with performance.

ReSTIR validation

To keep the scene responsive to light changes, once every three frames a ReSTIR validation frame is executed to re-measure the selected reservoir’s data. If the hit position or incoming radiance of the sample is different, we scale the reservoir M down accordingly. This will make the indirect diffuse reservoirs much more responsive.

However if we would only trace validation rays during such a validation frame, newly disoccluded pixels would not get traced at all. More importantly we would be wasting validation on pixels that possibly have gone off-screen. To solve this issue, we create a split in the tracing kernel. We start by reprojecting the history pixel location to the current frame, if the reprojected pixel location falls outside of the current screen bounds, we simply trace regular diffuse rays for that pixel, otherwise we proceed with tracing validation rays, as we have a history available that will be re-used in the current frame.

Irradiance cache

To estimate “infinite” bounces we will need a world-space irradiance cache that can be queried from any point in our rendering pipeline. Initially we went for a clipmap that holds an indirection to the underlying data. The clipmap irradiance cache is largely inspired by the irradiance cache used in kajiya, with some minor differences.

To give a quick recap: each entry in the irradiance cache places a probe at some distance away from its initial hit position. The probe is represented as a 4x4 octahedron map, where 4 diffuse rays, 4 validation rays and 8 shadow rays total are being traced from every single frame. Each ray samples the irradiance cache that it’s constructing, which leads to infinite bounces.

As an experiment we tried using a very lossy but cheap way to approximate shadow rays for all rays that query an irradiance cache entry. We did this by tracing a single shadow ray per irradiance cache entry and use use this result for all rays that would query it. This way we hoped to get around tracing additional shadow rays towards light sources from our diffuse bounces, but turned out to be extremely biased as we now mark the entire cache entry as lit or unlit. In the end it really was not worth it at all as we would only be tracing a relatively small amount of shadow rays less per frame in total, while getting significantly less accurate indirect lighting.

For each entry in the 4x4 octahedron map we store the same information as our diffuse candidate rays:

• Incoming radiance
• Reservoir
• Sample direction
• Hit distance

Just like in our indirect diffuse kernel, all of this info is again compressed the same way to a total of 4x4x4 uints per cache entry.

After tracing all the candidate rays, we apply temporal reuse to keep track of the most important samples. During each frame we also select 4 reservoirs to perform ReSTIR validation on, these reservoirs are the most frames away from getting new candidate samples. In a 4x4 octahedron map, this would be the set of samples of frame index (currentFrameIdx + numberOfTraceFrames / 2) % numberOfTraceFrames where numberOfTraceFrames is 4.

Let’s assume we are at frame 3, in the upcoming frame we would provide new diffuse candidates for frame 0. In the previous frame we provided diffuse candidates for frame 2, which means that the last frame to receive any updates is frame 1. This way we keep all of the 16 samples in our entry from not being stale for longer than 1 frame total.

Once we are done tracing the diffuse and validation rays, we build the spherical harmonics of the cache entry out of all valid samples.

For sampling the irradiance cache, we only have to load a packed L1 spherical harmonic, which is stored in a compressed format of 128 bits. Alternatively if bandwidth is an issue, you can also choose to go for YCoCg spherical harmonics which can be compressed to 96 bits.

You may notice the irradiance cache entries are “wobbly”, this is not an artistic choice but actually has some purpose behind it. Imagine the following situation:

When multiple irradiance cache entries are located at exactly the same distance to a vertex(red), which entry will we get when we query the position of the geometry? The answer to this question is yes..

To get around this issue we can modify the lookup function of our cache entry by applying a periodic shift. What this does is drastically minimize the size of intersections between a given triangle and cache entries.

There are problems with naively running a clipmap irradiance cache, one of the major ones being light leaking. There’s nothing stopping a ray from both the front and back of a surface from querying the same irradiance.

A good solution to this issue is a ranking system, which assigns a rank to the ray that queries the irradiance cache at a given position. Rays that originate from the screen have a ranking score of 1. Irradiance cache entries that were allocated with this rank will trace rays with a ranking score of 2 etc. New tracing origins will only be considered if the voting ray has an equal or lower score compared to the rank the entry currently has.

Let’s take a closer look at the scenarios where these light leaks happen though, we can observe that they mainly appear on places where the ray towards the query position is smaller than the size of a cache entry.

As an experiment, we tried to just reject queries that meet this condition to see if this would be sufficient as well. This way we can let the indirect diffuse ReSTIR samples prioritise light that comes in from an angle that would exit the enclosed area, as the weight of those close hits would be 0.

Looking at the results we notice quite a bit of darkening happening in several locations, where light otherwise would be bouncing around. This works around the light leaking problem decently, but still would not stop location votes from ending up outside of the geometry.

Future work

Visibility caching

One of the “holy grails” of ReSTIR would be to efficiently cache and estimate visibility so we would not have to explicitly trace shadow rays for somewhat reliable resampling, DDGI does something along those lines with radial gaussian depth but lacks directionality which is essential for sample visibility.

Visibility bitmasks sounds promising if we could somehow convert them into spherical bitmasks and/or store a bit more information besides boolean visibility information, but this quickly would consume more memory than we might hope for. Additionally updating visibility with canonical rays may not be ideal; let’s assume we were to use spherical bitmasks at a resolution of 16×16 slices, we would need to test 256 directions to fully fill in the visibility mask. Perhaps using “spherical visibility masks” per N×N region of pixels might speed things up.

Denoising

Ideally we want resolve the reservoirs to spherical harmonics, and interpolate them to full resolution after the spatio-temporal denoiser in a separate pass. This should in theory greatly reduce blurriness in the final output, especially when the scene has aggressive normal mapping or geometric complexity. This might also allow for running the tracing and reuse passes at an even lower resolution, as interpolating the coefficients is trivial.

Irradiance cache

A clipmap irradiance cache works great, but quickly runs into memory footprint issues when trying to use a higher resolution. The main reason for this is that we store indices in a 3D texture, which wastes a lot of memory on empty space.

As an alternative, a GPU hash grid can be used as this opens up more potential for higher resolution irradiance/DI/path caches.

Hopefully this post gave you a bit more insight on what techniques and tradeoffs can be used when building a ReSTIR-based dynamic GI system. Trying to preserve micro detail while still keeping a clean and responsive image has proven to be extremely difficult without tracing an excessive amount of rays, but not completely impossible.

If you have any questions on specifics of this blog post that I did not cover, don’t hesitate to reach out on mastodon or X.

Finally I highly recommend checking out the links below as they provided all the foundations to build a dynamic GI system!

Resources

https://github.com/EmbarkStudios/kajiya/blob/main/docs/gi-overview.md

https://www.nvidia.com/en-us/on-demand/session/gtcsj20-s22699/

https://research.nvidia.com/publication/2021-06_restir-gi-path-resampling-real-time-path-tracing

https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9985-exploring-ray-traced-future-in-metro-exodus.pdf

https://gpuopen.com/download/publications/GPUOpen2022_GI1_0.pdf

https://research.nvidia.com/labs/rtr/publication/wyman2021rearchitecting/

https://gpuopen.com/download/publications/Efficient_Spatial_Resampling_Using_the_PDF_Similarity.pdf

https://www.youtube.com/watch?v=Uea9Wq1XdA4

https://www.youtube.com/watch?v=2GYXuM10riw

Dynamic diffuse global illumination was originally published in Traverse Research on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this blog post, I’ll write about how I recreated the OIDN U-Net DNN using Rust and HLSL compute shaders, and about my time as an Intern at Traverse Research.

For some background, I’m Matt, a 2nd year Games programming student at the Breda University of Applied Sciences (BUAS). I’ve been programming for a couple of years, but this was the first time I’ve properly dipped my toes into machine learning, compute shaders, and working with a professional codebase.

What is OIDN

For brevity and clarity, I’ll quote Intel’s description of OIDN, which is “an open-source library of high-performance, high-quality denoising filters for images rendered with ray tracing”. The specific part I’ll be writing about is what the network is, the parts that are required for it to work, and how I went about optimizing my implementation.

The Network

Simplified down, the OIDN U-Net is just a series of weighted convolutions with extra steps, almost exactly like how you would implement a basic Gaussian blur. It’s called a U-Net because we downsample as we get further along (moving down), and then start upsampling halfway through (moving up), incorporating previous layers to maintain detail, which ends up making a U-shaped network if viewed as a diagram, otherwise it is just a regular neural network.

In total, the OIDN U-Net has a whopping 43 operations, but of those, there are only 4 different types, convolution, max pooling, upsampling, and ReLU activation.

I’ll now go over each of the unique operations, in order of complexity.

ReLU

ReLU or “Rectified Linear Unit” is way simpler than the name makes it out to be, a ReLU activation simply returns the input value if it’s positive, otherwise returns 0 or the common max(0, input). That’s it.

Max Pooling and Upsampling

In our case, pooling is synonymous with downsampling to half resolution, so we take a 2x2 area of our image (a window if you will), select the highest value in the window (Max), and output that as our max pooled value. For upsampling we do something similarly simple, we just take each of our input values, and create a 2x2 area of the same value, doubling our resolution.

Convolution

Convolutions in the context of images work slightly differently than they do in neural networks. Traditionally, you have a window with weights, that you slide over the image and multiply all the pixels in the area the window covers by their appropriate weights, and then sometimes add a value (called bias). With image filtering most of the time you apply the same weights to all channels of the image, and you usually only have 3 channels (RGB).

In our network, there are different sets of weights for each convolution (of which there are 15 total) and each convolution has a different amount of in and out channels (160 in and 112 out at its peak) which also have their weights. So we have up to 160x112x3x3 (161,280) unique weights to use at one time. Luckily for us, this does not complicate the convolution code much, just makes executing it a bit more time-consuming. We iterate over the entire image using a different weight kernel for each channel, but we also have a different set for each output channel, which we then sum up.

So for example at the start is a 4x32x3x3 kernel, where we apply a different 3x3 kernel to each of the 4 input channels from our image, and then sum up the resulting values to form one output channel value, we then do it 32 times total, taking the next set of weights each time and we have our new convolution output. One thing to mention is that the convolution should use zero-padding since we want to maintain resolution throughout the entire U-Net.

Weights

One thing I haven’t explained yet is where we get our weights from. Well observant reader, you are in luck, because as well as providing the source for their denoising neural network, Intel provides a set of float16 (and float32) weights to be used with it. Unluckily the header part of it is in a proprietary format, but is fairly simple to deconstruct with the source provided.

Optimizing

With all of the operations made, and weights parsed and loaded, we should now be able to denoise images at will, as long as our will is to wait 200 milliseconds for a 512x512 denoised image. The neural network is made up of 43 individual steps, but with a bit of profiling it seems that Convolutions take up most of the time (a whopping 97.7% of total time), which is of no surprise because we are doing a couple hundred million operations per convolution operation and that does not include the IO of loading the weights or input image.

Early exits

Since we are doing a convolution in 4 dimensions, we end up doing a lot of looping, where some of those end up being outside the input image and we pad with a zero value. However, since we are summing up the results, we can simply skip the current loop, as adding 0 and simply skipping the add yields the same result.

Preloading input and weights into shared memory

Another thing to consider is that since we are using convolutions to go over a 3x3 area of the image, some of our input pixels might get read multiple times (up to 9x), which if we are accessing them from global GPU memory is slower than it needs to be. So to fix that we load an 18x18 tile of inputs (16x16 but we also need neighbors for the convolution) into shared memory for one input channel at a time, and convolve over that (which is 7.1x less global reads). For our weights, since they are also shared for the channel, we can do the same so we don’t have to read them per thread.

Removing branching

Earlier I mentioned that we can simply early exit a loop if we find ourselves outside the image, but branching on the GPU can be expensive at times and in our case replacing the branching, and multiplying our weighed result by the condition of the if statement instead, we save ourselves 16ms.

Implementation in the breda-nn framework

Luckily for me, I didn’t have to implement any of the neural network foundation code, just the compute shaders, their “operations” and the U-Net itself. I was provided with a framework for creating neural networks, which was very easy to use once you started to understand how they work.

I could simply create a new Operation by implementing the respective rust trait and then creating functions for the shape of data we’re inputting, outputting, and which shaders to execute.

impl Operation for Upsample2D {
    fn create_inference_resources(
        &self,
        input_shape: &Shape,
        persistent_store: &mut RenderGraphPersistentStore,
    ) -> OperationResources {
        OperationResources {
            input_shape: *input_shape,
            auxiliary: vec![],
            output: Tensor::empty(
                "upsample2D_forward_output",
                self.output_shape(input_shape),
                false,
                persistent_store,
            ),
        }
    }

    fn inference_forward(
        &self,
        input: &[&Tensor],
        resources: &OperationResources,
        _parameters: &[Tensor],
        shader_db: &dyn ShaderDatabase,
        render_graph: &mut RenderGraph,
    ) {
        let input = input[0];
        let forward_output = &resources.output;

        let dispatch_size = resources.input_shape.w * resources.input_shape.h;

        let constants = [
            forward_output.shape().w,
            forward_output.shape().h,
            forward_output.shape().c,
        ];

        ComputePass::new("upsample2d-forward", render_graph)
            .constants_buffer(&constants)
            .read(input)
            .write(forward_output)
            .dispatch(
                &shader_db.get_pipeline("upsample2d-forward"),
                dispatch_size.div_ceil(GROUP_SIZE),
                1,
                1,
            );
    }

    fn create_training_resources(...) -> OperationResources {...}

    fn create_trainable_parameters(...) -> Vec<Tensor> {...}

    fn training_forward(...) {...}

    fn training_backward(...) {...}
}

Once I implemented each operation (pooling/upsampling/convolution) I could then create an operation that encompassed all of the aforementioned ones, executed in the right order, and the input/output shapes being passed through so each operation automatically gets resized to the correct size.

I think that without this neural network foundation, and some very handy built-in support for comparing our results with PyTorch by loading and running ONNX models exported from PyTorch directly in breda-nn, it would’ve taken me way longer to get this up and running.

info!("upsample inference:");
run_onnx_inference_test(
    "./apps/breda-nn-test/assets/models/upsample_inference.onnx",
    &*shader_db,
    device.as_ref(),
);

info!("max pooling inference:");
run_onnx_inference_test(
    "./apps/breda-nn-test/assets/models/maxpool_inference.onnx",
    &*shader_db,
    device.as_ref(),
);

Test on G-Buffer data

OIDN also can enhance image denoising by utilizing auxiliary g-buffer data, including normals and albedo. This approach significantly improves the reconstruction quality compared to only supplying a noisy image.

The result below is from one of the examples shown on the OIDN website (Mazda scene by Evermotion) and denoised in our framework.

Conclusion

Overall I’d say that OIDN is an interesting bit of machine learning, and getting the opportunity to implement it has taught me a lot about neural networks and compute on GPU. I am happy with the result I’ve achieved, but in hindsight, I think I could’ve done better.

The internship at Traverse has been great, even at the interview, the people here were nice and welcoming. The atmosphere in the office is always great and asking for help and/or advice is easy and encouraged. The codebase is well structured and there's always someone ready to help out if you get stuck or are unsure about something. In conclusion, I’d say if you ever get the opportunity to be at Traverse, I wholeheartedly recommend it.

I’d also like to extend a special thanks to Max Liani for the articles about his journey implementing OIDN in his raytracer. They are well-written and were fundamental in my understanding of machine learning and OIDN.

Denoising raytraced images using OIDN was originally published in Traverse Research on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hi there, I’m Jason de Wolff. Currently, I am in my 2nd year at Breda University of Applied Sciences, I was offered to do an internship here at Traverse Research during the summer break. During these 8 weeks, my objective would be to recreate the rendering/inference of a NeRF inside the Breda framework. Obviously, I said yes.

This blog post goes over the basics of rendering a Neural Radiance Field (NeRF) inside the Breda rendering framework. It will also give an inside into the life of a traverse intern.

NeRF was first introduced here. Later NVidia published their Instant-NGP paper which significantly sped up the training process, NGP standing for neural graphics primitive. Our implementation is primarily based on the Instant-NGP paper.

Multi-resolution hash encoding

The main contributor to speeding up the training process is the multi-resolution hash encoding, which allows the neural network to learn details relatively well and quickly. The encoding consists of a sparse voxel representation, where each corner of the voxel grid is stored in a flat one-dimensional buffer. The encoding receives as input a 3D position which is used to sample and interpolate 8 vertices from the sparse 3D grid. This is done for L amount of layers, in our case 16. Each layer holds N parameters, in our case between 2¹⁴ and 2²⁴, depending on the size of the scene. Each layer has an increasing voxel size, determined by the per_level_scale parameter, in our case set to 1.5. The resolution of layer 0 is defined by base_resolution, set to 16.

The indexing into these parameters is done linearly when the resolution of the current layer doesn’t exceed the number of parameters it holds. Otherwise, a spatial hash function is used. Hash collisions are not treated as the neural network is responsible for countering these artifacts. Concatenating the outputs of the positional encoding reduces the artifacts further.

Every corner holds multiple values referred to as features, we use n_features = 2. These features are the actual learnable parameters. The interpolated corner values of every layer and its features are concatenated together and form the input of a dense Multi-Layer Perceptron (MLP).

Model setup

The NeRF from Instant-NGP consists out of 2 MLPs, a density, and a RGB MLP. The density MLP takes the encoded position as input and outputs 16 values, the first of which represents log-space density. The input of our RGB MLP takes the 16 output values from the density MLP and concatenates it with the spherically encoded ray direction (NeRFs are rendered using ray marching). This results in 32 input values for the RGB MLP. In our setup, we use 1 hidden layer of 64 neurons for the density network and 2 hidden layers of 64 neurons for the RGB network. The RGB network outputs an RGB color which uses a sigmoid activation function.

Rendering

A NeRF can be encapsulated by a bounding box. We march rays through this box with an exponentially increasing step size. At every step, we use an occupancy grid to get a rough idea if we need to take a sample there or not. If the occupancy grid hints that there’s something at this position, we query our entire model with the 3D position as input. The total query looks the following: (positional encoding -> density MLP + dir encoding) -> RGB MLP.

The occupancy grid helps speed up performance by a lot, as each query is very expensive. All samples along a ray are blended together to form the final output color for the corresponding pixel.

Because the rendering is done with classical rays, effects such as depth of field or motion blur can be applied if desired.

Breda framework

All of this is implemented inside the Breda framework. This framework uses a bindless approach to improve usability. It can be run either on our Vulkan or DX12 backend. To give an example of how easy it is to get a compute shader running, the following code executes a shader that plots all payloads from a buffer onto the render target texture.

All compute and render passes are compiled in the render graph, which optimally reorders all passes for execution. After the render graph has been compiled it can be executed. And all of this, without the user needing to know any specific knowledge about the rendering API being used.

Conclusion

There’s a lot left to be done. We still want to implement training from 360 images and speed up rendering performance. By this time recent papers have also shown the potential for relighting NeRFs in real-time. For now, this serves as a solid foundation to work from.

As an intern, I had a lot of fun working at Traverse. The company culture is very nice and the codebase is a joy to work with. If you get the chance to work with these smart people, you should ;)

Neural Radiance Fields was originally published in Traverse Research on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this post, we’d like to walk you through the steps of creating a Directed Acyclic Graph, or DAG, that we use at Traverse Research for our in-house rendering framework, Breda, to do mirco-polygon rendering. This is achieved by creating meshlets out of a larger mesh and iteratively joining and simplifying them. It can be used to stream, cull and render meshes at different levels of detail. While the general idea is similar to Unreal’s Nanite, their implementation is tailored to rasterization, while our focus will be primarily placed on raytracing. However, you will find that the steps as explained in this post are equally applicable to both.

This post assumes that the reader is somewhat familiar with mesh rendering on different levels of detail and rendering nomenclature, has a very rough idea about what Nanite tries to achieve, but otherwise requires no in-depth knowledge about Nanite or similar systems.

Introduction: Level of Detail, and the DAG

There are several techniques to render meshes at several levels of detail, or LODs. Discussing them is beyond the scope of this post, but suffice to say is that most of them work on meshes as a whole. This isn’t particularly convenient for very large meshes such as landscapes and structures. You can split them up into smaller meshlets, but you’ll need to take extra care that any adjacent polygons sharing an edge between the meshlets will not cause any visible seams. This typically means that you somehow ensure that the polygons of either side use binary identical vertices, and that the edges between the verts are not more subdivided for one side than the other.

One way to solve this problem is to never simplify the boundary of a meshlet. You could separate your mesh into a list of meshlets, and then recursively take two adjacent meshlets, combine them, and simplify them. You will end up with a tree structure, where each node in the tree is a simplified version of its two children combined. The collection of all leaf nodes represents the mesh at the highest level of detail, while the root node is the entire mesh at the lowest possible resolution.

But herein lies a problem. Remember how we never simplified the boundaries of meshlets. Of course, when pairing up adjacent meshlets, those edges that were considered a boundary at a higher resolution are now on the interior of the combined pair, so the edges do eventually end up simplified. However, if we consider the root node of the tree and its two children, the edges adjoining the two children have never been simplified. They are still at the resolution of the original mesh.

This is an inherent problem of tree structures. Because of their nature, you will always be able to draw a line separating the left side of the tree from the right.

What we need is some data structure where we can have connections between the nodes that straddle this line of separation. This is where the Directed Acyclic Graph, or DAG, comes in. It is a type of graph where the edges always point in a certain direction (hence “directed”), and when walking the edges from an arbitrary starting position you will never end up at the node where you started (hence “acyclic”). This would then look something like this:

Strictly speaking, a DAG is a generalization of a tree — all trees are DAGs, but not all DAGs are trees. But a DAG allows for more connections between the nodes than is possible in a tree. Where a non-root node in the tree points to a single parent, a non-root node in the DAG can point to multiple “parent” nodes. This means diamond shapes are now possible. For example, in the picture above, notice how node 0 connects to nodes 5 and 7, which in turn both connect to node 18. Any border that 5 and 7 share, has been simplified before as there is no way to draw a line separating their child nodes.

But how to we achieve multiple parents? What does this mean in the context for our mesh? Well, remember how a connection of multiple child nodes to a parent means those nodes got merged into a parent. Having multiple parents for the same set of child nodes would therefore imply that they got merged into both parents, but the parents are somehow distinct. We achieve this by not only merging the nodes, but also splitting the result.

If we target a specific triangle count for each meshlet after the split, we need to merge more than two meshlets, the exact number depending on the simplification factor. For example, with a simplification factor of 0.5 (meaning a meshlet after simplification will have half the number of triangles), we need to merge 4 adjacent meshlets. To put it in other words, in order to target a fixed number of triangles per meshlet at each LOD, the simplification factor s is a function of the number of input i and output o meshlets: s = o / i

We feel that 4 inputs and 2 outputs, with a simplification factor of 0.5, is a decent balance. Your mileage may vary.

We should also consider that some nodes might never merge. This might be because of their spatial separation or other metrics you might use to consider nodes for merging. The DAG should be able to accommodate for this situation, and that means it should support multiple root nodes. Similarly, the number of meshlets will unlikely be always divisible by the amount of meshlets you want to merge, so the number of children for a non-leaf node in the DAG should not be a constant.

Creating the DAG

Now that we have a general idea of what our DAG is going to look like, we need to define some parameters. We’ve already discussed that we like to merge 4 meshlets and then split them into 2. Furthermore, we need to decide on a target triangle count per meshlet. For our particular use case at Traverse Research, we have chosen 1024. This value is not set in stone and still very subject to change. Since our framework heavily relies on raytracing, we need to generate bottom level acceleration structures (BLASes) to be able to ray-trace the mesh, so we need to find a balance between triangle count, memory overhead and runtime performance. A low triangle count results in a finer LOD granularity, but more BLAS memory overhead. It also results in a larger DAG, which might affect performance of the determination of the active cut. However, for other purposes, fewer or more triangles might be a better fit. Again, YMMV.

The algorithm to create a DAG out of a high-resolution mesh consists of the following steps:

// first, transform the mesh in a list of meshlets
let meshlets = generate_meshlets(mesh)

// and put them as leaf nodes in the dag
dag.add_leafs(meshlets)

// track all the borders between the meshlets
find_borders(meshlets)

// iteratively merge-simplify-split
while we can group meshlets:
  for group in partition(meshlets):
    // merge the meshlets in the group
    let meshlet = merge(group)

    // remove parts of border that is now on the inside of the merged meshlet
    update_border(meshlet)

    // simplify the merged meshlet
    simplify(meshlet)

    // split the simplified meshlet
    let parts = split(meshlet)

    // split the borders
    split_borders(parts)

    // write the result to the dag
    dag.add_parents(group, parts)

Third party tools

For our implementation, we have opted to use two open-source libraries.

The first is Meshoptimizer, which is a C library (Rust bindings available) to simplify the mesh, generate meshlets, and other useful mesh processing tools. Beware that its meshlet generation is specifically tailored to store the meshlets using very compact data format, and it therefore only supports at most 126 tris and 64 verts, which unfortunately is too low for our own goals.

For our simplification needs, we need to be able to specify which edges to lock to prevent the meshlet border from being simplified. A PR has been submitted to add this to the API, but unfortunately has not yet been merged at time of writing (Rust bindings exposing this feature available here)

We also use METIS, a C library (Rust bindings available) to partition a graph. We use this library to partition the list of meshlets into groups of N meshlets. In our experience, using METIS hasn’t always seemed very intuitive. In the last chapter of this post, we will go more in-depth on the relevant API functions and how to use them.

Generating the initial list of meshlets

Given that the meshlet generator of the Meshoptimizer is not suitable for our triangle count, we use METIS here as well to partition the initial mesh into groups of 1024 triangles. One downside is that METIS only works on the topology of the graph; it has no knowledge about special coordinates of the vertices. This is usually fine for one large continuous mesh of roughly equally sized triangles, but if your input has an irregular shape, you might want to use a different algorithm. This is especially true if you want to group triangles in meshlets based on other properties, such as face normals.

Special care should be taken with parts of the mesh that are disjoint, especially when using METIS. You can merge disjoint parts into a single meshlet, but as METIS has no knowledge about spatial separation, our suggestion is to keep those separate and only use METIS to work on continuous meshes.

As we want to partition the graph of meshlets using METIS, we need to keep track of meshlet connectivity. We also need to track the borders shared between the meshlets, as this is both important for generating an optimal partition and for locking the border during simplification.

Tracking meshlet borders

Tracking borders might not be a straightforward problem, as we need to distinguish between the original mesh border that is allowed to be simplified, and the borders that we create by partitioning the mesh into meshlets.

For our algorithm, we define an edge as the connection between two vertices. As we’re working with vertex indices, it’s convenient to encode an edge simply as a tuple of two vertex indices: (i0, i1), where i0 < i1. The latter condition ensures that we can always uniquely identify the same edge, regardless of the order of vertex indices in the triangle. This way we can use it as a key in some kind of associated data structure such as a hash map.

Furthermore, we define a border as a list of edges that is shared between a pair of meshlets. A meshlet can have multiple borders (as it connects to multiple adjacent meshlets), but two meshlets only share a single border.

Initial border computation

To generate the initial borders for all meshlets:

// create a map of edges that maps an edge to a list of meshlets
define edge_map: map of edge -> list of meshlet
for meshlet in meshlets:
  for edge in edges(meshlet):
    edge_map[edge].add(meshlet)

// create a map of borders that maps a meshlet pair to a list of edges
define border_map: map of (meshlet, meshlet) -> list of edge
for (edge, meshlets) in edge_map:
  for meshlet_pair in all pairs in meshlets:
    border_map[meshlet_pair].add(edge)

// now that border_map contains a list of edges per pair of meshlets,
// we can add it to the meshlets
for (meshlet_pair, edges) in border_map:
  let b = new Border(edges, meshlet_pair)
  meshlet_pair[0].add_border(b)
  meshlet_pair[1].add_border(b)

With this code, we have generated an initial list of borders for each meshlet. As you might have spotted, both meshlets refer to the same border object. The reason for this becomes apparent as we split the meshlets after simplification. We also use these border objects as connectivity information; the meshlet pair of a border are connected to each other in the graph. So, in a sense, these borders themselves form the edges of the meshlet connectivity graph.

Updating the borders during merge-simplify-split

In the next phase of our algorithm, we iteratively find meshlet groups by partitioning the meshlet graph, and then for each group we merge the meshlets into one bigger meshlet, the merged meshlet, simplify the resulting meshlet into the simplified meshlet, and then split that meshlet into two parts, the split meshlets.

During merging, the inner border disappears. For the merged meshlet, we can collect all borders from the individual meshlets in the group, and remove those borders where both of the meshlet pairs are within the group.

define merged_borders: list of Border
for meshlet in group:
  for border in meshlet.borders:
    let (a, b) = border.meshlet_pair
    if not group.contains(a) or not group.contains(b):
      merged_borders.add(border)

After simplification, the outer border of the simplified meshlet remains untouched, as we locked it for simplification. However, as we split the simplified meshlet, we are very likely split an existing border! We need functionality to split a border, while keeping intact the connectivity information of the new split meshlets with its original adjacent meshlets. Also, we need to generate a whole new border for the split between the split meshlets.

The general idea to solving this problem is to generate two sets of edges, A and B, one for each split meshlet. We can then walk over the borders of the merged meshlet, calculating the intersection of the border edges with either edge set. If either 0 or all of the border edges are in set A, the border remains unsplit. If some are in set A and some in set B, we need to split the border and update the references. The new border between the split meshlets is simply the intersection of A and B.

// first, let’s define some supporting functions

// this function checks whether one of the pair of meshlets is
// contained in the group, and if so, it updates its reference to
// the new meshlet
fn update_meshlet(border, group, new_meshlet):
  if group.contains(border.meshlet_pair[0]):
    border.meshlet_pair[0] = new_meshlet
  else:
    border.meshlet_pair[1] = new_meshlet


// this function splits the border into two, based on a set of edges belonging to
// split_meshlets[0].
// it also requires a group of meshlets that contains one of the elements
// of the original border’s meshlet pairs, and a new pair of split meshlets
fn split_border(border, edges, group, split_meshlets):
  // remove all edges not contained in the ‘edges’ set, and move them to a separate list
  let other_edges = border.edges.retain(edges)
  let other_border = new Border(other_edges, border.meshlet_pair)
  update_meshlet(border, group, split_meshlets[0])
  split_meshlets[0].add_border(border)
  update_meshlet(other_border, group, split_meshlets[1])
  split_meshlets[1].add_border(other_border)


// update all borders
let edges_a = set of edges in split_meshlets[0]
let edges_b = set of edges in split_meshlets[1]

let b = new Border(intersection(edges_a, edges_b), split_meshlets)
split_meshlet[0].add_border(b);
split_meshlet[1].add_border(b);

for border in merged_borders:
  let i = intersection(edges_a, border.edges)
  if count(i) = count(border.edges):
    // border is completely in A
    // this function checks which of the meshlet pair is in the group, and then
    // updates that reference accordingly
    update_meshlet(border, group, split_meshlets[0])
    split_meshes[0].add_border(border);
  else if count(i) = 0:
    // border is completely in B
    update_meshlet(border, group, split_meshlets[1])
    split_meshes[1].add_border(border);
  else:
    // this function splits the border based on an edge set, and
    // updates internal references. After this call, the adjacent meshlet not
    // part of the group will contain both pieces of split border
    split_border(border, a, group, split_meshlets)

After all groups of meshlets have been merged, simplified and split, the borders are all updated and will point to the new split meshlets.

And there we have it. After we have determined there are no more meshlets to group, we have filled the entire DAG, and the root nodes are now heavily simplifed representations of the mesh.

Determining the cut of the DAG

Now that we generated the DAG, we need to select which nodes we’re interested in rendering. These node are what’s called the cut of the DAG. We need some kind of metric to determine which DAG nodes are below the quality threshold (0) and which are above (1). But remember, this is not a tree. Because of the nature of the DAG, there are many paths from a root to a certain node, so we cannot simply recursively walk the DAG and evaluate nodes as we encounter them. For an efficient (and parallelizable) implementation, we should be able to walk over all connections between parent and child in arbitrary order, and evaluate whether that child should render, without considering any other information.

To generate a correct cut, our evaluation function should adhere to some conditions:

• Given an arbitrary path from a root to a leaf node, the evaluation function should only flip from 0 to 1 once. All ancestors above the flip should report 0, all descendants below the flip should report 1. In other words, the evaluation of nodes should always be monotonically increasing.
• All parents of a single node should always evaluate the same. Remember that these parents represent the split parts of a merged & simplified piece of mesh. If the evaluation function determines that parent A should render but parent B should not, and all the children of parent B render instead, then there will be visible overlap between parent A itself and parent B’s children.

Given these specifications, the cut is then simply defined as each child node that evaluates to 1 where its parents evaluates to 0.

Finding a good and correct metric for the evaluation function is hard. For rasterization, you could determine the projected on-screen error given an error metric calculated during simplification. With raytracing, this is a harder problem because you can’t easily evaluate an object’s projected screen size. So for us this is still an open question.

Conclusion

This concludes this blog post. We went over the DAG creation algorithm, together with some more in-depth techniques to track mesh borders, and how you would go about to selecting a cut.

Here’s a little demo of what we did at Traverse:

This post has been written by Sylvester Hesp at Traverse Research. If you have any questions or remarks, feel free to reach out through our website.

Creating a Directed Acyclic Graph from a Mesh was originally published in Traverse Research on Medium, where people are continuing the conversation by highlighting and responding to this story.

Beyond SAH — Building Optimal BVHs

At Traverse Research we are always looking to improve ray-tracing performance. A large factor for optimizing this is building the right and best acceleration structure, usually a BVH nowadays. In the past year, we looked into which way we can built the best-performing BVH in the form of a master’s thesis. In this blogpost we will summarize some of our findings, but first, let’s briefly recap the most common method. The entirety of the research can be found here, which includes the collected raw-data for what we discuss here.

In order to follow along, we expect you to be familiar with the concept of BVHs and along with that know what a top-level acceleration structure (TLAS) and bottom-level acceleration structure (BLAS) is. Having some understanding of how BVHs and similar acceleration structures are constructed helps, but is not strictly required.

BVHs and SAH — A recap

A common method for constructing a high-quality BVH is using the Surface Area Heuristic. One of the earliest source for using this is from MacDonald & Booth in “Heuristics for Ray Tracing Using Space Subdivision”. The idea is based on two main principles:

1. We want to minimize the probability of intersection for BHV nodes that are expensive to test (i.e. contain many primitives)
2. The probability of intersection of a node is proportional to its surface area, under certain conditions.

From this, a cost function can be derived that helps find a BVH with minimum cost. As such, we have a Surface Area Heuristic (SAH) to minimize, which looks something like this:

Most sane people do this in a greedy manner¹ where already established costs will not be re-evaluated as we go down the hierarchy. This would require evaluating all potential combinations of BVH nodes, which for any useful mesh is infeasible. The minimum cost BVH is in practice rather a local than global minimum in practice.

If you want to read more on how to bring this into practice, a detailed approach to implementation can be followed from the excellent blog series by Jacco Bikker.

[1] Our linked research also looks into the not-so-sane side: computing the global minimum cost and the implications compared to greedy computations.

Limitations of SAH

Using SAH to optimize the construction of BVHs is widely used as it can boost trace performance significantly for the cost it adds during in construction. However, the relation between the surface area and probability of intersection exploited in SAH assumes the following:

1. Ray origins and directions are uniformly distributed.
2. Ray origins lie outside the scene’s bounding box.
3. Rays do not terminate inside the scene.

The first assumption is a crude simplification of how we perform ray tracing. Our camera or primary rays follow a perspective distribution with a single origin, diffuse materials generally produce cosine-distributed rays and effects like ambient-occlusion also limits the distributions and directions. Only shadow rays will arguably not follow such a standard pattern and can be considered to be uniformly random to some extent. This generally better approximations than SAH may exist based on the distribution one particular type of ray follows.

The second assumption is generally true in practice for a BLAS, but with the exception of primary rays, ray origins will generally lie inside of a TLAS rather than outside of it, so again, using SAH won’t necessarily give us the optimal BVH here.

Then finally, rays not terminating in the scene is rarely the case in practice, as it is would imply all our rays are multi-hit rather than first-or-any hit rays. This affects the intersection probability for obstructed geometry, as rays starting outside the scene are much less likely to hit them or cannot hit those at all.

Alternatives

Clearly, there is a potential for building even better BVHs than with SAH for the cost function. Let’s briefly discuss some alternatives that were researched over the years:

Scene-interior Ray Origin Metric (Fabianowski et al., 2009)

Remember how SAH assumes that the ray origins lie outside the scene’s bounding box? Fabianowski et al. changed heuristic with the assumption that ray origins always lie inside the scene bounding box. This heuristic’s definition is however more complex and the resulting integral needs to be approximated for our discrete scenes. The two approximations they suggest both split the scene around the BVH node and approximate the solid angle (i.e. the visible surface area) from a set of vantage points in the divided space, as illustrated below.

This is a relatively simple solution that can theoretically eliminate one of the incorrect assumptions made by SAH for the majority of rays cast. The used form of measurement in the results is not ideal as it is in FPS, but the estimation is that ray-tracing performance can be increased by up to 7%.

Preferred Ray Distributions (Havran & Bittner, 1999)

Instead of assuming that all our rays are uniformly distributed in origin and direction, the different types of rays can be broken down and a BVH can be fitted to its (estimated) distribution. Havran & Bittner did exactly this for various ray distributions, such as for primary rays or with spherical distribution².

[2] This spherical distribution can be seen as the basis for the above mentioned scene-interior ray origin metric; the latter will generally be superior for any practical use-cases.

Let’s take some observations in mind when we think of what casting rays looks like if we have a perspective-projection camera that shoots rays into the scene:

1. Objects closer to the camera have a higher probability of being intersected than those far away, as the camera rays diverge.
2. Any geometry outside the frustum around the camera rays will never be intersected and therefore has an intersection probability of zero.

These observations are used to adjust SAH by assuming the probability of intersection is proportional to the surface area of the geometry or BVH node once projected on the camera plane. To get this, simply project the vertices of the AABB of the BVH node onto the camera plane using the perspective projection matrix. Clip off any of the surface area that lies outside of the frustrum (or camera plane when projected), and you have the probability of intersection for that BVH node.

Using this for as cost function reduces the authors’ intersection tests and traversal steps by 29% and 16% respectively, though their scenes are rather dated and do not really match modern day working sets.

Ray Distribution Heuristic (Havran & Bittner, 2009)

Monte-Carlo is the bread and butter for the average graphics programmer that does anything ray-or-path-tracing related, so not why use it for building the BVH? After all, we are looking to minimize the probability of intersection. Assuming we know which rays will be cast and given the geometry, the probability of intersection can be calculated for a candidate BVH node by using the rays that are to be cast to determine how often it will be intersected. This forms the Ray Distribution Heuristic (RDH) from Bittner & Havran. The probability of intersection is then the ratio of rays passing through the respective child node relative to the parent node, similar to the surface area ratio for SAH.

The rays used for construction don’t need to be the exact rays we use later on during ray-tracing, as that would also lead to enormous construction time for the BVH. Subsampling patterns (e.g. subsampling the primary rays by emulating a lower resolution) or subsampling the rays over time can both still yield speed-ups, assuming the rays that will be cast do not drastically change.

Applying this heuristic to kd-tree construction yields estimated speed-ups of up to 15% for tracing primary rays, but adding diffuse, shadow and other rays showed no gains. It’s not unlikely this comes from an “overfitting” problem, as the distribution becomes more uniform as more distributions are combined and SAH uses amore accurate estimate of the combined ray distribution.

Occlusion Surface Area Heuristic (Vinkler et al., 2012)

Aside from use-cases like shadow rays and translucency, rays are generally not traced further than their first intersection, which diminishes, as illustrated, the probability of intersection for geometry behind the first hit. Vinkler et al. therefore modified the SAH-based BVH build to include a factor of occlusion: Occlusion Surface Area Heuristic (OSAH). Leaving numerous additionally formed conditions out, the core cost function basically boils down to estimating the intersection probability through a weighted blend between the ratio of occluded triangles for the node and its SAH cost.

Their reported performance increases require covering SBVHs and such, but the tracing performance increases by around 28% for their tested scenes.

A consequence of OSAH, similar to RDH, is that it requires tracing rays during the construction of the BVH, as rays are traced to estimate how occluded geometry is. This means we want to have a BVH to construct one; our personal approach here is to just build a BVH once using SAH which is then used for occlusion queries to build OSAH-based BVHs.

The Best Heuristic

So we know some alternatives to SAH for BVH construction, but what’s actually the best? One problem is that while SAH is almost always included in performance comparisons, there is none between the listed improvements. Each research tests different scenes, viewpoints and metrics, so it’s hard to compare them directly from the published works.

This led us to our research, which compares the above discussed heuristics in a single framework in 8 different scenes. Since performance characteristics can differ between different ray types, this is measured separately per ray type for primary rays, diffuse bounces, shadow rays and ambient occlusion. Note that while it would be interesting to distinguish between TLAS and BLAS performance, this is not something we yet researched.

Let’s move on to some results from testing this with a CPU path-tracer for a simple stanford dragon scene in which we move the camera around:

Evidently, the other heuristics are not much better. Only OSAH seems to steadily outperform SAH for all ray distributions here here, despite the occlusion being relatively low for this scene. RDH was meant to be the ideal case: we measure exactly what the probability of intersection is for a given set of rays and then we trace those rays, again, with the RDH-built BVH. The problem is that we have a scene with only little interesting going on. Only part of the scene is covered and it seems that RDH simply has some bad frames as the performance does seem pretty similar for the majority of it.

We can make some sense of these performance observations: the interior-scene ray origin, for example, is one we do not expect to excel in scenes where rays originate from outside the scene bounding box, which is the case here for primary rays. Still, evenfor the other ray types it’s slower than SAH.

The results change if we move to an indoor scene, such as the Cathedral Sibenik:

This scene has significantly more occlusion and is rendered in our fly-through (primarily) from the inside and as a result we get slightly better performance with the Interior-Scene Ray Origin Metric. Especially RDH works best here and SAH becomes a bit more middle of the pack.

The perspective adjustment with Peferred Ray Sets does not even seem viable for primary rays in contrast, which is interesting given the seeming soundness of its definition. This is discussed in more detail in the linked thesis, but the problem lies the impact of breaking the assumption of SAH that rays start outside the scene has a far greater impact than for SAH itself.

Note that the other tested scenes are discussed in detail in the linked thesis.

Conclusion

The take-away here is that the best heuristic is quite scene-dependent. We found that higher occlusion factors generally lead to better performance with OSAH, that SAH is generally a good default but rarely yields the best ray-tracing performance, and that there can be correlation factors that are not yet well defined that correlate the performance to a particular heuristic.

With OSAH often being a superior alternative in the tested scenes to SAH, why not use this by default? One not-yet-discussed aspect is the performance of constructing the BVH. RDH, OSAH and the Preferred Ray Sets method all depend on the camera viewpoint or even the exact set of rays that will be cast. This means that changing the viewpoint or set of rays requires a rebuild of the BVH. For a TLAS this can be acceptable, but this is undesirable for BLASes.

Even then a single build has the performance problem in that the operations are far more expensive than simply calculating the surface area, too expensive to build every frame. There is some potential of reducing the input set for RDH and OSAH, but while keeping in mind a minimum in optimization efforts, building with SAH is still much faster.

Is there no future in these alternative heuristics at all? We actually do think there is one. The Interior-Scene Ray Origin Metric, for example, can give a speed-up up to nine percent over SAH in some scenes and is rarely slower in ray-tracing, especially for rays other than primary rays, which we do not cast in Breda. Evaluating the heuristic is more expensive, but we think much can be optimized and the fast approximation of the heuristic is of comparable quality already. We highly recommend reading the paper by Fabianowski et al. as it is a relatively simple heuristic to understand and integrate in your own BVH builder.

Another possible way to benefit from these heuristics might even be to use separate BVHs for different ray types, as primary rays are usually faster for SAH than the Interior-Scene Ray Origin Metric. Other combinations may very well be worth it to deliver the, hypothetically, fastest tracer. Ideally we would be able to configure what heuristic or configure for which use-case a BVH is built when specifying this to the respective APIs, so hopefully some day this will be available.

There are many more directions to go with this. Which ray distributions do we use for construction with RDH and does it make a difference? Can we re-use the BVH from previous frames for cost evaluations that are view-or-ray dependent? Can we combine the heuristics in different ways such that they, as combination, counter some of the assumptions made in SAH? What about heuristics specific to a certain type of ray distribution, such as shadow rays? Several of these are also explored in the thesis, so if this caught your interest, we definitely suggest you give it a read! Ultimately we hope to dive into the others as well and explore new methods for boosting ray-tracing performance!

Beyond SAH — Building Optimal BVHs was originally published in Traverse Research on Medium, where people are continuing the conversation by highlighting and responding to this story.

Animating large crowds in a ray-traced environment on the GPU

What it takes to render crowds

Crowd simulations have been common in games for a while now, but they usually only had to work for rasterization. Now that ray tracing is becoming more popular, new challenges have appeared. This blog post covers how we implemented animations and crowds in our testing framework “Breda” here at Traverse Research.

There are various challenges that need to be tackled while implementing such a system. First of all, the large number of vertex animations means we need an optimal GPU implementation to update all the meshes into their animated poses. Since these crowds will be used in both our path tracer and hybrid renderer, we also must take acceleration structure building into account for the animated meshes. These require a persistent set of vertices in their animated pose, which means that we will need a duplicate vertex buffer for each animated instance!

Other than that we should keep in mind that we’ll need motion vectors for the animated geometry, and ideally have the ability to reuse as much of the animation state as possible to make the crowds perform well.

Our animation system

Our animation system consists of a few basic types and concepts. We import most of our animation data from glTF, and store it in our own asset format. This format still closely resembles glTF.

Animations

An animation in our engine is defined as a set of channels, where each channel has an output type (translation, rotation or scale). A channel defines the values for that output type for each key time-point in the animation. Every channel has an output index.

We apply animations to animation samples to sample one specific time point for that animation.

Animation Samples

Animation samples contain the state of an animation for one time-point. The channels in an animation directly write these values into the animation sample. A sample can contain an arbitrary number of outputs, where each output is the combination of a rotation, translation and scale. These can be combined into a transformation matrix later.

Note that not all outputs in the sample have every type of channel. In this example, if we were to apply output 1 to a transform, we would only override the rotation and translation. We keep the scale from the original transform.

Skeletons

A skeleton is defined by a set of bones that are parented together to form a hierarchy. Each bone has a default position, called the bind pose (or T-pose). Skeletons and their bones are in local space, because we will want to reuse the same skeleton for multiple meshes at different locations. This requires a conversion step from glTF, as that format defines them in world-space.

Skeleton Samples

These are similar to animation samples, in the sense that they contain the state of the bones in a skeleton for a specific time-point. In fact, each skeleton sample contains an animation sample! When we animate a skeleton, we first apply the chosen animation to this animation sample. Each transform output of the animation sample corresponds to a bone, and we apply the rotation, translation and scale overrides to that bone. We now have a set of bones in their animated state.

Next up we apply the hierarchy of the bones. This involves multiplying the bone transforms in the right order. E.g.: the hand bone is attached to the arm, which is attached to the shoulder.

Vertex Buffers

Now that we have the final position of each bone in our skeleton sample, we can apply the bones to the vertices in a vertex buffer. Because we need to build a bottom level acceleration structure, these vertices must be persistent. We can’t rely on the old rasterization trick of transforming the vertices during the draw calls.

Actually, there is still one thing we have to do before we can apply the bones to the vertices. We must first undo the bind-pose by applying the inverse-bind-matrix (ibm) to each vertex. To understand why, you have to realize that the vertices in their bind pose are already transformed by the bones. If we were to directly apply the new bone position, we would essentially have applied that bone twice. The ibm of a bone will transform that bone back to the origin of the world. If we apply it to a vertex, then we are moving that vertex back towards the origin (0,0,0), which means the vertex position is now relative to the bone. At this point, we can apply the new position of our animated bones to the vertices, and they will correctly end up relative to the bones.

In the example above you can see this happening. If we move the foot of the skeleton, then we want to move the vertices in the leg along the long green vector (the transform of the bone). However, the small red vector was already applied (the bind pose of the bone). First we undo the bind pose, then we can apply the new bone position.

In our implementation, we pre-apply the ibm to each bone after we chain the bones together.

Now, applying the bones to the vertices is fairly straightforward. First we read the vertices from the bind-pose vertex buffer in a compute shader, and then we multiply them with the corresponding bones. Each vertex can define up to four bone indices, and a weight for each of those bones.

Finally we write the animated vertex away to the animated vertex buffer.

World Infrastructure & Crowds

The previous section discussed what the animated data and update logic looks like, but it did not yet touch on how we actually set up these animations in the world. Most importantly, it did not involve anything about crowds yet.

Hierarchy

Central in our framework is a hierarchy of nodes that all have a transform. These represent the objects in our world. Each node can have an animated vertex buffer attached to it, but a node’s transform can also be animated directly.

Vertex buffers are not directly attached to nodes, but rather we attach a small container (we call them “meshes”) composed of references to a vertex buffer, skeleton sample, bottom level acceleration structure and materials. These “meshes” can be composed separately, and reused between multiple nodes.

Attaching a mesh to a node means we update our draw calls and add an acceleration structure instance to our TLAS.

Node Animation

As mentioned before, we can also directly animate nodes in the world. Think for example of rotating objects. We don’t have to apply vertex animation here, we just need to override a node’s transform in the hierarchy directly. This is why we can bind nodes in our hierarchy to a specific animation sample. We gather these, and apply the transforms in a compute pass.

Skeleton Parenting

We’d very much like it if our crowd actors could hold items in their hands, or wear hats. To do this, we should be able to parent nodes to bones in a skeleton. But this is a problem, because our skeletons are not actually a part of the hierarchy. Instead they are in local space so that we can reuse them. Fortunately this is simple to solve. We can provide nodes with a skeleton sample and bone index, and patch the node’s transform in a compute pass that reads and applies the right bone.

Crowd Instancing

A hierarchical approach like this allows us to reuse a lot of animation, and forms the backbone of our crowd system. For example, we can decide to load a single skeleton, and animate it ten times at different time points. We can then apply those ten skeleton samples to 100 unique vertex buffers. Those 100 vertex buffers can be combined with different sets of materials to create many unique “meshes”. Each mesh can then be added to one or more nodes to create large crowds with limited animation updates.

Because we need to make a full copy of the vertex buffer for each unique animated instance, reusing these also saves a lot of memory.

Double Buffering for motion vectors

Since we apply all sorts of graphics effects, we often need motion vectors so that we can retrieve temporal information from the previous frame. This goes for vertex animated geometry as well. Fortunately, we already need to keep a copy of the animated vertex data around for ray tracing. Double buffering the positions for those buffers is a small extra cost that allows us to then always retrieve the previous frames’ vertex positions for a specific mesh. Subtract those from the current positions, and you have your motion vectors.

BLAS building and refitting

Every time we update a vertex buffer in the animation system, the bottom level acceleration structure that uses the buffer must also be updated. This can be done by either rebuilding it completely, or by performing a refit. If the changes in geometry are small enough, a refit is often cheaper and still gives good tracing performance. In our framework we rebuild once every N frames, and do refits for the rest. We do this because the BLAS quality deteriorates over time while refitting. A heuristic can be used here to determine when a rebuild should happen.

The rebuilds and refits can be batched in most modern graphics APIs. This gives the driver the opportunity to limit the number of dispatches for better performance.

Other world stuff

This sums up the main tasks in our world update regarding animation. Of course, there’s more engineering around it. Sorting the draw calls and setting up our TLAS requires extra work. We have some logic in place that ensures that animation updates always run at least once so that we never have any uninitialized state.

Animating a crowd with many instances of the same mesh, but with different materials, also comes with some downsides. Draw call sorting depends on the material, so you’ll have to split up your instanced draw calls, and change pipeline state. BLASes require you to specify geometry flags that tell you whether a triangle is opaque or not, which means you’ll have to build new BLASes when the opacity of the materials between instances changes. Solving these is not in the scope of this blog post, but be warned that you will run into these!

Compute on the GPU

Now that we have all the theory down, we are still left with a considerable amount of work that the GPU has to do. How do we implement this optimally?

Compute work

Our animation system does a total of 7 compute dispatches, regardless of how many animated objects we have.

1. Read animation channels and write to animation samples.
2. Apply animation samples to bones.
3. Apply the skeleton hierarchy to the bones.
4. Apply the ibm to each bone, and calculate the normal matrix.
5. Apply bones to vertex buffers.
6. Apply animation samples to nodes.
7. Apply bones to nodes on the hierarchy.

Optimizing & bindless rendering

The main optimization that we can do is to batch as much work as possible. Launching a compute pass per skeleton to update its bones would be bad for performance because it comes with some overhead. Furthermore, for small skeletons it would mean that a large portion of the threads would be idle.

The same thing goes for all the other passes: we have many small buffers containing similar data. If all this data had been in a single large buffer, then a single update step would have been easy. Fortunately for us, we have a bindless rendering setup that allows us to dynamically access any buffer from the GPU. This means that we can launch a single compute dispatch that goes over all the buffers, and in turn all the data in those buffers.

If you would like to know more about our bindless implementation, check out our blog post series by Darius Bouma on setting up a bindless renderer. https://blog.traverseresearch.nl/bindless-rendering-setup-afeb678d77fc

Now, the technique I used to batch all these operations is by building a prefix sum. I’ve described a fast implementation of this algorithm in my previous blog post, which you can find here: https://blog.traverseresearch.nl/fast-cdf-generation-on-the-gpu-for-light-picking-5c50b97c552b

Quickly summarized, we want to know the total number of operations we have to perform. For example, if we have 50 skeletons in separate buffers with 10 bones each, then we have a total of 500 bones that need to be updated. We would need to launch a compute pass with 500 threads, each of which will update a single bone.

By calculating a prefix sum, we can use each thread’s global index to deduce the skeleton and bone index to update. This is done using a binary search in the prefix sum array. This array stores for each buffer how many elements came before it. For example, the first buffer has seen 0 elements. The second buffer has seen all elements in the first buffer, and so forth.

We can repeat this prefix sum implementation for each pass in our animation system. It can be done for the animation channels, the bone buffers and any other data that is spread out over many small buffers.

Shortcomings / Future work

Overall we are very happy with the implementation we came up with, though there are a few shortcomings still which I’ll list here.

Firstly, we don’t have access to the world space transforms of all our actors, because animations happen on the GPU. This was not a big problem for our demo, but in a more game-like scenario you’d quickly run into issues here. Proposed solutions are to mimic parts of the animation pipeline on the CPU depending on which transform you need to know, or to do a read-back (which would mean your transforms are always a frame behind).

We also didn’t tackle animation blending or bone labeling. These were simply not a priority for us, but are on the backlog. Perhaps we’ll release a blog post in the future to cover these.

Lastly, our crowds can still benefit from some changes. For example, adding LODs to not just the crowd actors, but also their skeletons and animations could improve performance. This was however not in scope for us at the time, as it would likely require a lot of manual tweaking by an artist. The same thing goes for adding more variation to the clothing worn and objects held by actors in the crowd. An interesting talk on these topics can be found here: https://www.youtube.com/watch?v=Rz2cNWVLncI.

I hope this was helpful! Feel free to leave a comment if you have any questions, and I’ll do my best to answer them.

Ray tracing animated crowds was originally published in Traverse Research on Medium, where people are continuing the conversation by highlighting and responding to this story.

Let’s start with some general stuff first; this will be an introductory post about the Render Graph developed at Traverse Research, intended for people already falmiliar with the basics of GPU programming. The main concepts used are (render or compute) passes, (texture and buffer) resources, and barriers.

Additionally, back in April 2022 our dear Manon wrote a blog post about our render graph too! You can find it here. Compared to her post, one main thing has changed: we changed the way in which we group the passes. For those who haven’t read that post yet I’ll do a quick recap of the render graph. For those who have you can skip straight down to “Using the implicit chronological order”.

To start things off, I’ll reiterate something you’re already familiar with: we need to place barriers between our passes in order to execute the passes in the correct order and avoid race conditions between the resource reads and writes. Ideally, this number of barriers is as little as possible to reduce the total time we’re waiting.

🔍 Our in house renderer

At Traverse Research we have our own in-house renderer. This renderer creates many different kinds of passes in a plethora of different systems which all result in a single frame being rendered. Since we’re rendering a single frame, we want to dispatch our passes to the GPU in one single go. Furthermore, each of these systems doesn’t know what other systems exist and what passes they create, nor do we even want to be concerned with that. Ideally we can provide some sub-system with our inputs and get all of its outputs back and ready for us to use in any of our following passes. This means we can’t make some smart manual decisions about where to place our barriers.

♥️ Making our life easier and faster

So, we need some kind of way to tackle these issues. Let us start with the simplest: being able to give all our passes to the GPU at once. This can be achieved by creating a centralized place to which we can add our resources and passes. We’ll call this our Render Graph (you’ll understand the name at the end of this). Various other renderers have a similar concept, although the name may vary. With everything in a single place, I’ll leave it up for the reader to imagine how we can then pass everything to the GPU in a single go.

At this stage we aren’t doing any work on the GPU yet, so we aren’t limited by the limitations of a GPU just yet. We can still reason about any chronological ordering between our passes. However we also mentioned that when we call a sub-system, we just want to provide it some input and get some output resource back, without having to reason about all of the passes that the sub-system might be creating. Creating a chronological ordering between the passes themselves is thus off the table. So what do we actually have that we can use?

We can do both reads and writes to our resources. Reading does not change the actual data of said resource, but we can safely assume writing always will (otherwise why are you saying you’re going to write to it?). What we can do is adding a version number to each resource. When we read a resource in a pass, we explicitly state which version we would like to read. When writing to a resource we do everything as we would for reading the resource, but afterwards we increment the version, after all this is now different data! Any passes using the resource afterwards will use the now incremented version number. As a more real-world example: we have a canvas on which we painted yesterday and we’re going to paint on it again today. That means tomorrow it’ll also be different than how we started today. Whilst what we have painted on the canvas is different throughout the three days (version 1, 2 and 3), the actual frame and cloth of the canvas stayed the same.

By adding these version numbers to our resources, each of the passes have specific versioned resource dependencies. These dependencies implicitly describe the ordering that we will need for all of our passes! If you’re not directly following: the dependencies can be followed to link the passes to one another. To know what passes need to be executed before a specific pass, we look at all of its versioned resources used and for each resource find the pass that did the actual version increment. If we were to look at all of our passes from a global overview, we could see that our passes form a large graph network, hence the name Render Graph.

In our graph, the starting and ending points are always resources. When a resource is created it’s version number is 0, even when no data is passed to initialise the buffer with. A pass filling this uninitialized resource is still writing to it and thus incrementing the version number. At the other end of our graph we also end up with some output resource, after all adding resources that only read but don’t output anything themselves don’t do much for our render.

To make things a bit more visual, let’s look at a simple case. We have got two separate systems creating some output texture. These textures are combined in a new pass, after which the output texture is blurred. This will give us the following graph:

Just as a small note: this graph cannot contain “loops”; a situation in which we can follow any number of arrows and end up at the same location. This is the same for any system interacting with a GPU that does not contain race conditions. The official term for our graph is a Directed Acyclic Graph (DAG).

🏭 Using the implicit chronological order

We’ve got a graph with implicit dependencies, but where to place the barriers between our passes? We’ve mentioned that our initial knowledge of our Render Graph is mainly about the resources and their versions. For now we’ll assume each resource can truly only be written to by a single pass, i.e. there’s only a single pass doing a write operation to a specific resource and version combination. For the usage of a versioned resource, we can identify a few different cases: it’s only read by one or more passes, it’s only written to by one pass, and it’s read by one or more passes and written by one pass.

In all three cases we need to place a barrier between the pass doing the version increment and the passes using the version incremented resource. For the first and second cases there’s really not much more to do. The third one, however, has a small caveat: if we would use the same barrier for the reads as we would for the write, the write could happen before (or at the same time, being a race condition) the read operations, invalidating the data we expected to be there for the read. Thus, we need to make sure any read operations happen before any write operations. This can probably be solved in multiple ways, but we opted in our implementation to add an “invisible” dependency link from the reading passes to the writing pass. Let’s look at a visual example:

In this example I have labelled each pass with what resources are used as input, as well as their usage and version. If I am not mistaken, the graph covers all different cases. Adding barriers to it would result in the following:

Another way of looking at this is that we’re slicing the render graph into groups, where each group contains one or more passes which can be executed in any order (or all at the same time) without barriers.

🧑‍💻 Translating logic to an algorithm

In our system there are various code paths that do generate passes, but turn out to be unused for the actual output, these are mostly alternative (debugging) versions of the main render output. We do know which output we want for this frame, but we do not immediately know which passes become unused. Removing these unused passes is what we call dead stripping. The versioned resource which must be executed we’ll call root resources.

If we were to start at the beginning/bottom of our graph, we’d have to do quite a bit of tracking in order to determine the path we have taken so far even touches a root resource. However, if we would flip our logic and start at the end/top of our graph, we can just use the root handles as our beginning and get any dead stripping done for free! After all, the passes are unused so we won’t even touch those.

Actually iterating the graph can be done in multiple ways, however we’ve opted to do it breadth first search style as this is super simple to implement. A tl;dr on breadth first search: we have some starting point(s) which we add to a queue. We pop the first element from the queue, with which we can now do whatever we want. Then each of its dependencies it added back to the queue. Whether each or only some of its dependencies is added depends on the implementation. In our case we also keep track of which passes we have already seen during the iteration, so we don’t visit a single pass twice. As the name suggests, contrary to depth first search (in which a stack is used), this algorithm processes elements closer to the root before continuing with elements that have a higher “depth”. As a fun bonus the depth value is thus monotonic (i.e. never decreases).

We also need a good definition of when a pass is deemed “ready to be executed”. In our implementation we’ve solved this by doing an additional iteration of the graph. In this iteration we count how many passes use any of the outputs the selected pass creates. During the iteration in which we place passes into groups we can check this dependency count. When this value is anything other than zero this means there are passes which rely on output of this pass which we haven’t handled yet. Since we’re iterating the graph “backwards”, this means we should skip this pass for now and will visit it again later through some other pass dependency.

The last thing on our list is placing the passes into actual groups. Here too we iterate the graph starting at the root passes in a breadth first search manner. Since we’ve already done our setup and determined the (indirect) dependency count for each pass, there are only a few minor things left to do. Firstly we need to check for each dependency of the selected pass whether its count is zero. If it is, the dependency needs to be added to the queue, if it isn’t, its dependency count is decremented. The group to which this pass belongs is then determined by the depth we’re currently at. In our implementation we add all our root passes to the first group and, during iteration, set the depth value for the dependency passes to the next depth value of the current pass.

So to put the entire algorithm together, we get the following:

1. Create a list mapping each versioned resource to a pass.
2. Create a list mapping each pass to all of its dependencies (i.e. passes that should execute before this pass).
3. Add all passes which read a versioned resource to the pass that writes to said versioned resource as artificial dependencies to the list just created.
4. Create a list of root passes from the root versioned resources.
5. Iterate the graph and find the dependency count per pass.
6. Iterate the graph and place each pass in a group when the dependency count reaches zero.

⌛ Is this it?

So now you’ve seen a general overview of the design of our Render Graph and a rough idea how its implementation is done. There are still a few things left untouched, such as async compute and automatic resource transitions, but those might be touched upon at some later point!

Additionally, if you’re looking for a more information about it, I’d also point you to Maister’s blog post, who also built a Render Graph, but has a slightly different approach to the blog post!

An update to our Render Graph was originally published in Traverse Research on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this post, we’ll go over the basic building blocks of our real-time volumetric rendering process in the Breda graphics framework (developed at Traverse Research). This post is aimed at people experienced with general graphics algorithms like ray marching, writing shaders and some experience with bit manipulations. The following design goals were made very early in the development process and have guided the implementation so far:

— Loading of OpenVDB files
 — Index space of 4096³
 — Animation playback
 — Voxels represent floating point data
 — Small memory footprint to have enough space left over for other assets
 — Needs to be ray traced in real time on an RTX 3070
 — Multiscattering and emission support (not in this blog post)

Our data structure

Our data structure is a trimmed down version of the VDB data structure. We load VDB files using our custom parser called vdb-rs, and then load our structure into a format that includes only the features that we require. As with most VDB implementations we use the 5–4–3 structure which means that our top level node has an index space of 32³ (1<<5), our first internal node has an index space of 16³ (1<<4) and our last internal node has an index space of 8³ (1<<3). These combined result in an index space of 4096³ (1<<5+4+3). From now on, the different nodes will be referred to by their size, so Node5 is the top level node and Node3 the bottom level node. Below is a schematic of the structure:

These nodes are split up into 2 parts: a header, and a bitmask. Below is a description of one of these nodes, here the size of the bitmask is dependent on the level at which this node is located in the structure:

struct Node{
  child_pointer: u32,   // Points to the start of N_CHILDREN child nodes
  tile_value: f32,      // When the entire region is homogeneous
  min_value: f32,       // Minimum value of all nodes below it
  max_value: f32,       // Maximum value of all nodes below it
  mean_value: f32,      // Mean value of all nodes below it
  // Mask where every bit determines if one child node either 
  // contains data or not
  child_mask: [u32; N_CHILDREN / 32],
}

All of these nodes have to be stored somewhere in a GPU friendly way, and thus we create a structure which we can upload to the GPU. This structure contains the following information:

struct Volume{
  constants: Vec<VolumeConstants>,
  nodes5: Vec<Node5>,
  nodes4: Vec<Node4>,
  nodes3: Vec<Node3>,
  voxel_data: VoxelData,
}

As can be seen, we have vectors for the different node types. Additionally, we have a vector of constants, this contains information about the translation, scale and aabb of our volume. The reason why this is a vector is because these constants can be unique per animation frame. The last piece of the puzzle is our voxel data, which is stored in a large buffer containing the actual floating point data that the Node3 point to. This data is stored in bricks of 8³ voxels. Since this is by far the part of our data structure that consumes the most memory, our goal was to optimize it as much as possible. This is achieved by first changing the layout from a flat array (where each brick is stored as 8³ consecutive values) to a 3d texture layout, and then by storing the values in such a way that each 4 values in the z direction (of the brick) are stored as the RGBA components of our 3d texture. This allows us to run BC7 block compression and end up with 2 bits per voxel (floating point value). Below is a slice of the resulting texture:

Here each chunk of 8² pixels is half of an 8³ chunk, two of these slices make up the entire chunk. As can be seen, there are visible edges, corners and swirls in our data, even at this incredibly small scale.

Animations

As said earlier, we support animated volumes. These animations consist of sequences of VDB files which can be exported by programs like EmberGen or Houdini, or downloaded directly from JangaFX’s website (like we did in our examples below). You might have noticed that our Node5’s are also stored in a vector, but each of those nodes already has our desired index space of 4096³. So, instead, each of our animation frames has a unique Node5. This way, when switching to the next animation frame, we only have to change the starting point of our structure traversal.

Structure sampling

Raymarching through volumetric space requires sampling the data structure a lot of times. This will likely be the hot-loop of our shader, so this has to be fast.

Sampling our structure requires checking if a child node is valid, and then follow the pointer down the tree. If the child node is *not* valid, we return the tile value, which indicates the value of all nodes beneath it. Once we are sure that the voxel exists, by checking the bit in our Node3, we sample our voxel data texture. Below is our code for doing that, note that all multiply, division and modulo operations are done using powers of 2, thus these are compiled to bit shifts and bitwise and operations.

float getVoxel(uint3 position) {
  // Assume we are in the aabb of the volume
  uint indexNode5 = computeIndexNode5(position);
  if (!getBitNode5(indexNode5, animationFrame)) {
    return tileValueNode5(animationFrame);
  }

  uint indexNode4 = computeIndexNode4(position);
  uint node5ChildPointer = getChildPointerNode5(animationFrame);
  uint node4Index = node5ChildPointer * kNode5Children + indexNode5;
  if (!getBitNode4(indexNode4, node4Index)) {
    return tileValueNode4(node4Index);
  }

  uint indexNode3 = computeIndexNode3(position);
  uint node4ChildPointer = getChildPointerNode4(node4Index);
  uint node3Index = node4ChildPointer * kNode4Children + indexNode4;
  if (!getBitNode3(indexNode3, node3Index)) {
    return tileValueNode3(node3Index);
  }

  uint node3ChildPointer = getChildPointerNode3(node3Index);

  // Our texture supports a maximum of kMaxBricksInDim^3 bricks, 
  // so wrap the indices to that region
  // node3ChildPointer to brick texture coordinates
  uint3 brickIndex;
  brickIndex.x = (node3ChildPointer % (kMaxBricksInDim));
  brickIndex.y = (node3ChildPointer / kMaxBricksInDim) % kMaxBricksInDim;
  brickIndex.z = node3ChildPointer / (kMaxBricksInDim * kMaxBricksInDim);
  brickIndex *= uint3(8, 8, 2); // multiplied by brick size, 
                                // each z texel stores 4 values

  // Offset inside brick
  uint3 texel;
  texel.x = position.x % 8;
  texel.y = position.y % 8;
  texel.z = (position.z >> 2) % 2;
  uint3 toSample = brickIndex | texel;

  // Position sample correctly in texture
  float4 chunkData = voxelData.load3D<float4>(toSample);

  // Grab correct z slice out of float4
  return chunkData[position.z & 3];
}

One function that needs further explaining is computeIndexNode. This is exactly the same as the internalOffset calculation in the OpenVDB paper. As input we get a position, then for every dimension we select only the bits that are relevant to the layer we are calculating the offset for. Then we place the selected bits into a single unsigned integer which is the the local index. Below is a code example which does this:

uint computeIndexNode5(uint3 position) {
  uint3 selected = (position & kNode5SLog) >> (kNode3Log + kNode4Log);
  selected.x <<= kNode5Log + kNode5Log;
  selected.y <<= kNode5Log;
  selected.z <<= 0;
  return selected.x | selected.y | selected.z;
}

There are a number of constants: kNode5SLog describes the log2 span of our Node5 (3+4+5=12) and the different kNodeXLog are simply the X values (so kNode5Log = 5).

Simple rendering

To get some simple rendering going with our data structure, we can employ techniques commonly used in the game industry, as seen -in example- in 2.5D clouds rendering. For an in-depth overview of those methods, the challenges faced, and the solutions employed, we refer the reader to the extensive work available from Sébastien Hillaire as well as the various Nubis-related talks from Guerrilla Games.

Participating media is composed of particles which either scatter or absorb photons. Due to the extremely high amount of particles in a volume, this is all modelled through a statistical process in which the different kinds of particles are described by absorption (μₐ) and scattering (μₛ) coefficients.

A beam of light travelling through a differential volume element can undergo the following processes:
 — Absorption
 — Out-scattering
 — In-scattering

The change of radiance travelling towards the ray direction must also account for emission, but we ignore it for this article to keep things contained.

We also focus on a single in-scattering event for the time being: the amount of light scattered back to the view as a result of the interaction between a light (in our case, the sun) and the media.

Absorption and out-scattering “remove” photons from the light ray travelling towards our eye, either because photons are absorbed or because they bounce off of the volume particles and away from the view. This process is a function of both the absorption and scattering coefficients, which -summed up- make up the extinction coefficient.
This can be represented with a single differential process, integrated along the ray direction, which gives us the transmittance along the ray as a function of the extinction coefficient. This is the Beer Lambert law.

In-scattering is a function of both the scattering coefficient and the phase function. The phase function describes the angular distribution of scattered rays for a volume, and is conceptually similar to a BRDF for surfaces.

We now have all the tools to understand the volume rendering equation

which tells us that the incoming radiance from point x towards the camera (ω) is the result of an integration process (with upper bound the surface point z) in which -at each step- the in-scattered radiance is modulated by the transmittance accumulated up to that step, and by the scattering coefficient at y. The exitant radiance at the surface z also contributes to the final output and is modulated by the total transmittance accumulated along the ray (the volume “translucency”).

We can approximate this integral as a Riemann sum, using ray marching.
Notice that transmittance is multiplicative along a ray

T(a→c) = T(a→b) · T(b→c)

which means that we can simply accumulate transmittance that way at each step of our integration.

With all that said, a simple (and naive) sampling routine would look like this:

for each pixel {

  TraceContext ctx = computePrimaryRay();

  float t0, t1;
  computeVolumeIntersectionBounds(out t0, out t1);

  // Variable needed for our final output
  float3 transmittance = 1.0;
  float3 scatteredLight = 0.0;

  // Results of our surface passes
  float3 reflectedRadiance = loadReflectedRadiance();

  // Better approach: start offset on view-aligned planes
  for each step with `stepSize` starting at t0 {

    // Compute scattering location
    float3 pos = ctx.origin + ctx.dir * (t0 + steppingDist);

    // Sample medium density at location
    float density = volume.sampleDensity(pos);

    if (density > 0.0) {
      // Calculate absorption, scattering and extinction coefficients
      VolumeCoefficients coeff = volume.getCoefficients(density);

      // Compute direct lighting
      float3 directLighting;
      {
        // Secondary volumetric trace towards the light
        float3 volumetricShadow = traceShadowRay(pos, dirToSun);

        // Compute phase function value
        float phaseFunction = volume.phaseFunctionEval(dot(dirToSun, -ctx.dir));

        directLighting = SUN_INTENSITY * phaseFunction * volumetricShadow;
      }

      float3 incomingLight = coeff.scattering * directLighting;

      float3 stepTransmittance = exp(-coeff.extinction * stepSize);

      // Improved integration for lower sample counts as presented by Sébastien Hillaire
      float3 integrationStep = (incomingLight - incomingLight * stepTransmittance) / coeff.extinction;
      scatteredLight += integrationStep * transmittance;

      // Accumulate transmittance
      transmittance *= stepTransmittance;
    }
  }

  float3 result = reflectedRadiance * transmittance + scatteredLight;
  writeOutput(result);
}

Conclusion

This blog post went over the foundation of our real time volume renderer. Below is a demo of the renderer of a standalone volume. We can easily play back animations or manually change to the desired animation frame.

Below, we see a render from our Breda renderer. Here we placed a volumetric animation inside the Bistro scene.

This blog post was written by Emilio Laiso and Rosalie de Winther at Traverse Research. If you still have any questions, feel free to reach out on Twitter or directly to Traverse through our website.

Intro to rendering of animated volumetric models in Breda was originally published in Traverse Research on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

In this blog post, I’ll take a deep dive into how we build our CDF for light sampling on the GPU at Traverse Research. I’ll give a quick rundown of what everything is, why you need it and finally how we made it run fast on the GPU.

The first few sections are an introduction to concepts that may not be clear to everyone, so feel free to skip ahead to the part where I dive into optimizing the GPU algorithm if you already have a good understanding of CDFs!

What is a CDF?

CDF stands for “cumulative distribution function”. In probability theory, it is a function that can tell you the probability that a random variable has taken on a certain value. This is done by summing the value of the probability density function (PDF) for each value that the random variable can assume.

The below diagram might make it a bit clearer if I’ve confused you up until this point: The red line is the value of the CDF for a random variable. The random variable can take on the values of A, B, C and D (X-axis), and the Y axis represents the value of the PDF for each of those values.

Personally I prefer to look at the CDF the way it is represented in code: an inclusive prefix sum array, where each element is the sum of the value of all elements that came before it (including its own value). The below diagram shows the PDF and CDF for the four values our random variable can assume in array form:

If you are interested in reading more about probability theory, then I recommend this blog post by Jacco Bikker.

Where and how are CDFs used?

With all that out of the way, you may be wondering why you would need a CDF. Well, it turns out that they are great when you want to stochastically select an element from a set proportional to its weight, and relative to the weight of all other elements in the set.

This is possible, because a useful property of CDFs is that you can invert them: rather than querying the CDF what value corresponds to a known element, you randomly generate a CDF value and find which element corresponds to it. That way you can select your samples with a priority for the important ones (the ones where the PDF gives a higher value).

A common scenario where this occurs is any algorithm that relies on monte carlo integration. If your dataset is too big to fully consider each iteration, then integrating over a few stochastic samples each cycle will eventually also converge to the correct result. By not taking random samples, but stochastically selecting based on weights you will converge to the right result faster. The stronger the relation between the importance of an element and its weight, the faster it will converge.

There are many algorithms where this applies, but for the rest of this blog I will be focusing on our use case: sampling lights in a real time ray tracer.

Light sampling algorithm

Let’s translate all that previous information to our use-case. We are running a real-time ray tracer with a scene that contains many lights. For every pixel, we want to know per light how much it contributes to the irradiance. In our actual implementation, we use monte carlo integration and we feed our selected light samples into a ReSTIR implementation, but for simplicity I will assume that we are only going to consider a single light per pixel, and that we are letting the image converge over time.

The algorithm is fairly simple, and works as follows:

1. For every pixel, pick a single light using the CDF.
2. Calculate the irradiance that the light contributes to the pixel, and divide it by the relative weight of the light.
3. Average the light contribution per pixel over many frames to get the correct result.

Step 1: Inverting the CDF
In an earlier section, I talked about inverting the CDF. This is exactly what we are going to do for step 1 of the algorithm! Let’s take the CDF from a previous example, and invert it. Pretend that A, B, C and D are the lights in our scene. Each light has a radiance, which we are going to use as its weight. This means lights that emit more energy will be stochastically selected by the CDF more often (since they are probably more important than weaker lights).

First we generate a random float between 0 and 1. Let’s say it landed on 0.75. In the image below, you can see we map this random float to the lights in the CDF and pick whichever one it landed on. In this case, that would be light C. Mapping the random float to the CDF is straightforward: multiply the random float with the sum of all light weights (which is the last element in the CDF) and then do a binary search in the array to find the element that contains the resulting value.

In this example, you would be multiplying 0.75 with 5.5, which gives you 4.125. Binary searching would yield you element C because it contains all values between 3.5 and 4.5.

Step 2: Calculating Irradiance
Cool, we have selected a light! We can now shade our pixel, but there’s still one important step: we must divide the resulting irradiance by the relative weight of our chosen light. Since we did not pick our lights uniformly, some lights will be overrepresented and others will seldom occur. Ignoring this problem would mean we are creating energy out of thin air, which is highly illegal. Fortunately for us, we can calculate the probability of having picked any one light by dividing its weight by the total weight sum of all lights.

We divide the shading result by the pick chance of our light. This will scale the light’s contribution up to compensate for all the lights that we did not pick this frame, proportional to how much the light contributes to the total light power in the scene. Light B would be scaled up less than light A, but B also gets picked more often. In the end this cancels out, and you are left with the same result that you would get from fully random light selection, but you needed less iterations to get to that result.

Step 3: Algorithm Integration & Conclusion
The result from one frame of this algorithm would be noisy, but if you average the results of a few frames it will quickly start to converge. For this scenario where we only have four lights, the algorithm does not make a large difference. But in modern path tracers, having hundreds of thousands of lights is a common scenario. That’s when this algorithm greatly reduces the complexity of shading.

CPU CDF Implementation

Now that we have all the theory out of the way, it’s time to actually build our CDF. On the CPU, a naive implementation of an inclusive prefix sum array is very simple:

After executing, the CDF would contain [1.0, 6.0, 8.5, 11.6, 12.6, 14.7].

The real downside to this implementation is that it is single-threaded, and building it takes O(N) time. If we had 16 elements we wanted to calculate the sum of, we would need to perform 15 additions in sequential order since the next addition always depends on the previous result.

We can then start using the CDF by stochastically selecting elements using some random number generation and a binary search like in the following code snippet:

Parallel prefix sum

How do we make the build process faster than O(N)? Introducing: the parallel prefix sum! An algorithm that can calculate the prefix sum of an array of values using many threads at the same time, bringing the complexity down to O(log N). This algorithm requires a large amount of threads to be available at the same time, so it’s a perfect fit for the GPU.

The algorithm works by building a tree of local summed values in multiple passes. When the tree is completed, each element contains the sum of the elements below it. The top of the tree will contain the sum of all elements.

In our example, we will be building a binary tree from an array of 16 elements, so the algorithm needs 4 (log2(16)) passes. Each arrow represents a thread adding two values together and storing the result. If you count the arrows, you can see that we still need a total of 15 additions to get the sum of our 16 values. The difference is that we can run the additions largely in parallel! During the first pass, we perform 8 additions, but if we have 8 threads performing them at the same time, the total time taken is the same as doing one addition. Given that we have enough threads, the total runtime will be log2(N) additions where N is the amount of elements we want to sum.

Now that we have summed our values in tree format, we still need to flatten it into an array where each element contains the prefix sum at the index corresponding to the original weights array.

This requires one more pass where we launch a thread for each element in our weights array (16 threads in this case). Each thread remembers its element index, and that is where it will eventually write in the CDF array. We traverse the tree top-down, and our goal is to sum all values we find that are only composed of elements that have an index smaller than or equal to the current thread’s element index.

The logic looks as follows:

• Is the sum stored at the current tree node composed only of elements with an index smaller than or equal to the current thread’s element index? Then add the sum to our total. If the value already contains the current element index, terminate. We have found our prefix sum. If not, go down the right node of the parent node and repeat the algorithm.
• If the sum stored at the current tree node was composed of elements that lie to the right of our current thread’s element index, then we do not sum this value. Instead go down the node on the left and repeat the algorithm for that node.

The image above illustrates this logic for the thread that has been tasked with finding the prefix sum for the sixth element in our array.

1. It starts at the top of the tree, and realizes that the value there (27) also contains the sum of elements that came after index number six. It goes down the node on the left.
2. It checks if the value here (15) lies entirely to the left, but sees that it also contains elements seven and eight, which come after six. It goes down the tree on the left.
3. Finally! The sum here (7) is only composed of elements one, two, three and four. It adds 7 to our total sum. This sum does not yet contain our own element (six) so we go down the parents right node.
4. The value we find here (8) also sums elements that lie to the right of our current element. We go down the node on the left.
5. We find the sum of elements five and six (4). We add it to our total sum, which adds up to 11. Because the current node’s sum includes our current element (six) we can terminate the algorithm and write the prefix sum of index six: 11.

When all threads have finished executing, we are left with our prefix sum array. Knowing all this, it should be somewhat straightforward to implement the parallel prefix sum algorithm in a compute shader. There are however a few fun optimizations we can make!

Single Pass Tree Builder

First of all, the algorithm requires multiple shader passes, one for each layer of the tree to be more precise. There needs to be an explicit synchronization between these passes because each layer reads from the previous one. Adding these execution barriers into our GPU timeline is slow, especially when we get near the top of the tree where we only need a few threads to do a few operations. Doing multiple compute passes also adds extra overhead.

Luckily, we can build the entire tree in a single compute pass by using a few HLSL synchronization tricks. This approach is based on a code snippet by Sebastian Aaltonen. I’ve made a greatly simplified version of our CDF tree building shader to showcase how it works:

We start by launching enough thread groups to build the first layer of the tree: one thread for every 2 elements in our CDF array. Each thread will take the two elements from the previous tree layer, and then write a single element away to the next layer in the tree. We then kill off half our thread groups, since we only need to do half the operations in the next layer of our binary tree. Repeat this process until we have only a single thread that writes the top of the tree away.

The main issue with doing this is that every layer of the tree depends on the values in the previous having been written, so we must ensure that this writing has finished before we start reading.

The synchronization works because we keep re-using the same thread groups. The thread group that merges the two outputs from the previous layer, is the same thread group that wrote one of those values away in the first place. The other value was written by another thread group, but that group incremented the same atomic value as our current group. Since only the last group to write survives, we know that all data is there when we read it.

Only the first thread in each thread group updates the atomic in our atomics buffer, and writes the result away to group shared memory. This way all threads within that group can read the value after synchronizing.

You may have also noticed that I marked the tree buffer as “globallycoherent”. This is required because thread groups will be reading values written by other thread groups within the same shader invocation. That means that changes aren’t guaranteed to be written away to global memory from the local caches by the time we read again. The globallycoherent keyword ensures that any changes we make will be visible to other thread groups.

To make all this a bit more intuitive, here’s an example of the thread group lifetimes for a tree that is being built for an array containing 8 weights.

• We start with four thread groups, each consisting of a single thread for simplicity.
• Group 1 sums weight ‘2’ and ‘3’ and stores the result. It increments the atomic, and finds that it is the first to do so. It terminates.
• Group 0 sums weight `0` and `1` and stores the result. When incrementing the atomic, it reads back a value of 1, which means the other thread group it’s paired with has already finished executing. It survives because it knows it is the last group to finish.
• In the next iteration group 0 now reads the value it previously wrote, and the value group 1 wrote. From another part of the tree, group 3 is summing its own values and finishes executing first, so it terminates.
• Group 0 is now the last living group. During the last iteration of the tree building, it sums the value it wrote previously and the value written by group 3.

Optimal GPU usage

We now have a single compute shader invocation that can build our entire CDF tree, but we are not utilizing the GPU in a very efficient way. Having a single thread per thread group is far from optimal, so we should take a closer look at the hardware. It turns out that most modern GPUs schedule thread groups in smaller batches (warps or waves) of around 4 to 64 threads. If we make our thread groups the same size, we will be 100% utilizing the hardware’s thread lanes!

The next optimization involves wave intrinsics. GPUs execute shader code at exactly the same time for all threads within a wave. Special operations built into HLSL use this to allow you to read variables from other threads in your wave, since they are already loaded into memory anyways. We will be using the powerful “WavePrefixSum” intrinsic, which calculates the prefix sum for a variable for all threads in our wave with a lower index than the current thread.

I’ve changed the previous code snippet a little bit to show how we would use this. Rather than one thread per group, we now have 32 (the most optimal number differs per GPU). Each thread loads a single element into memory, and the wave intrinsic calculates the sum incredibly fast. The last thread in our group then writes this sum away for the next level of the tree.

As you may have noticed, our tree is no longer base-2, but base-32. This is great because it means our tree isn’t as deep as it was before. We won’t need as much synchronization and we can process more elements in parallel. This does require some changes to the rest of our algorithm: the offsets for each tree layer will change, as well as the offsets for our atomics. Our thread groups are also no longer writing to an atomic in pairs of two, but in batches of 32. Each batch will be writing to the same atomic, and only the last one to write will survive. This makes sense: one out of every 32 thread groups survives, just like how we have 32 times fewer elements whenever we go up one level in our tree.

Similarly, the down-sweep pass that flattens the tree into an array will now also need to be updated to consider 32 children per node instead of just two.

Bonus optimization: since your wave size and tree base are powers of two, you can bit shift indices to the left and right instead of doing multiplication, division or modulo operations. This is a quite common scenario in this algorithm because you will often be calculating the offsets into the tree buffer for a given layer and thread index.

Layering multiple CDFs

That is pretty much the entire algorithm out of the way! I did however skip over one important detail of our ray tracing mesh light implementation. Up until this point I’ve been assuming that all lights are neatly laid out in a single array. Of course it can’t be that easy. In our actual implementation, any triangle of a mesh can be emissive, so when we pick a light we need to know the mesh and triangle index.

The solution here is to add more CDFs. We can build a prefix sum array for the number of emissive triangles in each mesh, and then inverse this CDF by flattening it into an array.

We can now use this flattened array as input for our light CDF tree shader. It will just involve a little bit of extra work for the threads that would normally only read the weights from the bottom level of the tree: they’ll have to load the triangle light using the indices in this buffer, and calculate that weight themselves (and store it for the bottom of the tree).

Floating point precision

While this algorithm is very fast, it does suffer from a few problems. The most important one is floating point precision. In a scene with a million lights, calculating the prefix sum means adding a lot of floats together. You end up with quite large numbers towards the end, which means lights with tiny weights may not actually change the sum at all anymore. They will thus never be considered. This problem can be mostly mitigated by sorting lights from low to high weight, but such a sorting pass would itself also take up time and may not be worth it.

Conclusion & Timings

And that is how we stochastically select our lights at Traverse Research! In scenes with ~100.000 emissive triangles the whole process of breaking down our meshes and building the CDF for individual emissive triangles takes about 0.07 milliseconds on an AMD RX 6800 and 0.054 milliseconds on an Nvidia RTX 3080.

I hope this blog post was informative, and inspired you to use these techniques to speed up your own shaders. This was my first time writing a blog post, so feel free to send me some constructive feedback!

Fast CDF generation on the GPU for light picking was originally published in Traverse Research on Medium, where people are continuing the conversation by highlighting and responding to this story.

This blog post is aimed towards people with an interest in graphics who have knowledge about BRDFs and the math involved. In this post we will briefly go over how the layered material model we implemented works.

Layered material models are an expandable solution to achieve more complex visual appearances like coatings. The core idea is to combine multiple lobes by treating them as a stack of layers where light reflects off and refracts through. We based our implementation on the Weidlich-Wilkie material model, mainly to determine how layers interact with each other using simplifications from the paper to drastically reduce the computational complexity. For conductors we use measured data to get a more realistic approximation. We used three wavelengths to approximate the entire spectrum of light and use measured indices of refraction and extinction coefficients for our simplified Fresnel derivation. Simple BRDFs often hack conductors using Schlick’s Fresnel approximation with an eye-balled color that represents the conductor as F0.

Fresnel and our conductor model

We use two derivations of the Fresnel equations together with measured data for results closer to reality; one for dielectrics and one for conductors. For both derivations we assume unpolarized light, therefore the ratio of reflection and transmission depend on the average of p- and s-polarization of the incident ray. Since the per-wavelength varying extinction coefficient has a great impact on the visual appearance of conductors we use a slightly more complex derivation than we are using for dielectrics with an extinction coefficient factor. Since we are not working with a spectral renderer we had to pick frequencies to sample so we decided to go with three wavelengths to approximate the entire spectrum, 0.63 µm (red), 0.532 µm (green) and 0.465 µm (blue). The derivation to be used is selected based on the layer’s metallic factor, when this is zero it is a dielectric, otherwise it is treated as a conductor. Using metalness textures can lead to situations in which both derivations are linearly interpolated based on the metallic factor. Blending Fresnel for conductors and dielectrics is not physically plausible but is supported since it is often encountered in PBR assets and clamping or rounding the metallic factor would lead to loss of detail. Because we are approximating the entire spectrum with only three wavelengths, we use a lookup table with indices of refraction and extinction coefficients per wavelength for the following measured materials: aluminum, brass, copper, gold, iron, lead, mercury, platinum, silver and titanium. Our data set is obtained from https://refractiveindex.info/ of which we use the metal section of the 3D shelf.

For composing layered materials, we currently support GGX reflection, GGX transmission and Lambertian diffuse lobes. We support up to 4 components and enforce that conductors and diffuse lobes can only exist as bottom layer. Scattering within layers is ignored, and outgoing rays that encounter total internal reflection are invalidated. The invalidation of total internal reflection layers can cause energy loss, but this is compensated by an energy compensation factor.

Supported lobes

The Lambertian diffuse reflection lobe can only be used as bottom layer and represents an ideal diffuse material with a user specified base color. This layer represents a cheap approximation of scattering below the surface, causing the surface to look the same independently of the view direction. This layer only has a single base color parameter specifying the reflectance of the diffuse layer. This can be represented as a RGB color value or SRGB texture.

The GGX specular reflection lobe can be used to represent dielectric coatings on top of diffuse layers, but can also be used to model conductors. This is all determined by a metallic factor, which should either be zero or one. When metal-roughness maps upload values between zero and one it becomes a blend between dielectrics and conductors which is not physically plausible. We still support conductor dielectric blends in textures since it is commonly encountered in PBR metalness textures. Conductors in our layered model are assumed to not transmit energy since all energy is absorbed before it reaches the next layer.

The GGX specular transmission lobe is used to represent transparent materials, this adds a GGX lobe below the stack of layers. For indirect illumination we currently render the result of this lobe to a different texture since it cannot be handled by our indirect specular reflection denoiser in our hybrid renderer.

The layered BRDF

To sample the BRDF as a whole we sample a direction in all lobes individually and importance sample one based on a heuristic that determines which sample will likely have a high contribution. For diffuse lobes this returns the luminance of the diffuse albedo and the specular component will return the luminance of the Fresnel component. The heuristic for the Fresnel component can in the future be improved by taking the context of the layer into account, reducing the weight of samples from which a large fraction of incoming radiance has been absorbed by the layers above.

When a sample has been taken the probability density has to be calculated. Since a sample can be valid for all lobes, the total probability density needs to be calculated for the entire BRDF. When imagining a GGX specular reflection lobe on top of a Lambertian diffuse lobe, all samples taken on the GGX lobe are also valid for the diffuse lobe.

Re-using samples for all lobes makes our sampleBrdf and brdfPdf functions slightly more complex than when dealing with a single layer at a time. When we sample a BRDF we use random numbers to sample a direction in a lobe for each layer and a random number to sample one of the sampled directions that will be traced. Each sample will then get a weight assigned based on is luminance as mentioned before. A CDF is built using the weight for each layer and the third random number is used to sample one of these layers, only the sampled layer is traced. These weights are then normalized and sampled as CDF to determine which layer will be stored and traced.

The probability density function needs to account for the importance sampling of layers when calculating the probability density in order to converge to the correct result. This requires the brdfPdf function to be aware of the probability that the layer is sampled on the lobe. Since the sample can be valid for multiple lobes we multiply the sum of probability densities for sampling the sampled direction in each individual lobe by the probability density of sampling the lobe itself to get the total probability density that the direction is sampled. We validated our implementation by comparing the converged result to a result converged with next event estimation.

Calculating the reflectance of the BRDF consists out of several steps. We first calculate the reflectance of the top layer with given incoming and outgoing directions, then the absorption factor is calculated to attenuate the transmitted fraction of the incoming radiance. After that both the incoming and outgoing directions are refracted for the evaluation of the next layer. This uses Snell’s law with the index of refraction of the layer the light is refracted through. The total reflectance of all individual lobes summed results in the total reflectance of the BRDF.

Hopefully this post gave you some insight in how we tried to achieve more realistic layered materials by briefly going over our conductor model and by describing how separate components can be combined to achieve more complex results.

Our layered material model was originally published in Traverse Research on Medium, where people are continuing the conversation by highlighting and responding to this story.