Shallow-River | Devpost

FPS over time
Latency over time
Stats summary
Memory copy comparison
Program Flow Diagram

Github: Shallow River Detailed report: Report.pdf

Submission for the Physical AI Optimization Track

Inspiration

We wanted to create and optimize a multi-cam body tracker for VR, cobots, and health applications, utilizing the full potential of the ARM features in the Jetson Orin. Most camera body trackers are single camera examples and are not edge enabled. This leaves a large gap in the industry for physical AI hardware optimization, where low latency and high throughput are required. Our goal was to beat out NVIDIA's Deep Stream example application see here by utilizing both the CPU and GPU with unified memory for a no copy pipeline approach.

What it does

Shallow River is a C++ alternative to the NVIDIA Deep stream plugin API for Gstreamer. Shallow River demonstrates how to harness an asynchronous pipeline of TensorRT models and CPU based computer vision models for a real time embedded application. This architecture is useful for industry applications such as motion capture, virtual reality player tracking, and health/exercise research. Shallow River delvers increased real time performance compared to other programs running similarly quantized models.

Shallow River takes two off the shelf USB cameras as input, calibrates the camera intrinsics with a charuco board, generates an optimized engine file, and then outputs the 3d position of a person from the perspective of both cameras. See the README in our github link for specific runtime instructions. Shallow River was designed for and runs on the Jetson Orin Nano Super at a combined 15 FPS, 7 FPS per camera. You can view the 3D outputs using our helper Python scripts. There are also options to print out performance logging to verify out claims in our report . See report. To test out our modified versions of the Deep Stream application, vist: Deep Stream Modified Repot.

We also provide a .DEB packaged release in our repository to make the installation as easy as possible, as well as an install script for necessary packages when building from source. This greatly reduces the friction for users to get started.

How we built it

We built Shallow River using a couple of helpful tools. The first was OpenCV. We used OpenCV to handle camera calibration and also to run some common computer vision algorithms required to pre and post process data for the AI models. We used peopleNET and bodypose3dNET from Nvidia's NGC registry for our models. To run the models we used TensorRT, which also created the .engine files at runtime for the Jetson from the ONNX model files.

The main goal of Shallow River’s architecture was to achieve a no-copy asynchronous pipeline to optimize the latency and throughput of a multi-cam computer vision program. Several optimizations were implemented into Shallow River to deliver the best possible real time performance. The most notable optimizations include using no-copy architecture, unified memory, a multi-stage multi-producer single consumer pipeline, and runtime model optimization using TensorRT.

We used runtime logging to compute FPS and latency, and Nvidia Nsight to analyze bottlenecks, CPU, memory, and GPU performance. See the report for detailed analysis.

Challenges we ran into

One of the main problems we ran into running our program was memory ownership. Many existing physical AI projects run on edge rely on global objects. This is not a practical approach for creating an asynchronous pipeline that used both CPU and GPU memory. We solved this problem by creating a shared managed frame pool and passing references through the pipeline. This way we could remove frame copying as well as limit total memory consumption. Another problem we had was how memory was allocated. We had to change from CUDA managed memory to CUDA pinned memory to avoid segfaulting when running the model. Pinned memory locks the address space which was necessary when it is being accessed by both the CPU and GPU asynchronously.

Another issue we had with memory was that the 3D pose model required camera intrinsics as an input. When assigning the tensors dynamically with no-copy pointer assignment it would not update the intrinsics and the 3D output stopped working. To fix this we had to have a single cudamemcpy to get the GPU to update the model inputs correctly. We also had to be careful about floating point underflow. Camera intrensic calculations have very small floating point values which are then multiplied. This prevented the use of a fp16 model, as the outputs would underlfow and produce NaN.

Accomplishments that we're proud of

Our biggest accomplishment, other than getting real 3D output from a stereo camera setup on the Jetson was actually beating the Deep Stream application in latency. Most of the Deep Stream performance gains were from the optimized model it was using not the actual program architecture. This means by using a more optimized model setup we should be able to match it in throughput as well as beat it in latency.

What we learned

We learned a lot about memory management on unified memory chips. We had to go pretty far into the weeds to get the program to run without segfaulting. This involved reading a ton of CUDA and TensorRT documentation. We also learned a lot about computer vision pipelines and each stage required to turn pixel data into meaningful output.

We also learned how powerful ARM unified memory designs are in the AI field. These architectures are the best way we've seen to run performant AI, especially in real time applications.

What's next for Shallow River

We are going to keep adding to Shallow River to create an accessible body tracker for VR, robotics, and health applications. The next step is to add the stereo calibration to create the final unified 3D output. Our goal is to create a Jetson release and a desktop release coming soon.