This is a stable fork of utils3d maintained as a dependency for microsoft/MoGe.

A pure-Python collection of useful functions for 3D computer vision and graphics research.

• NumPy / PyTorch pairs: most functions have both implementations.
• Flat & non-modular: standalone functions only, no classes, no hierarchies.
• Native types: always use native Python / NumPy / PyTorch types.
• Vectorized only: no Python loops beyond O(log N).

⚠️ This repo changes quickly. Functions may be added, removed, or modified at any time.

• If you are looking for one or two specific functions, feel free to copy them directly from the source code. This repo may serve a better purpose as a reference implementation.

• If installing utils3d as a dependency, use commit id or fork one if you need stability.

  pip install git+https://github.com/EasternJournalist/utils3d.git@<commit_id>

• If you are willing to accept the latest changes, fork this repo and join me in making 3D research easier!

• Use utils3d.{function} to call the function automatically selecting the backend based on the input type (Numpy ndarray or Pytorch tensor).
• Use utils3d.{np/pt}.{function} to specifically call the Numpy or Pytorch version.

The links below will take you to the source code of each function with detailed documentation and type hints.

The PyTorch-only pose_graph_optimization and pose_graph_optimization_gnc promote compact edge moments and weights to float64 before graph assembly. Rotation synchronization, scale synchronization and translation solves run in float64, including under autocast; GNC also keeps residuals and reweighting in float64 between iterations. This avoids discarding weak nonzero graph modes through float32 pseudoinverse rank thresholds. Results retain the input cross-covariance's dtype and device. Raw point-map reduction, index dtypes, solver objectives and robust-rejection settings are unchanged.

Run the precision regression checks from the repository root with PYTHONPATH=. python test/test_pose_graph_precision.py.

utils3d.pt.pose_graph_optimization_huber (also available through automatic dispatch) provides the harmonic-distance Huber IRLS solver. Like the other PGO implementations it is PyTorch-only; no NumPy PGO backend exists.

moments = utils3d.pt.pose_graph_edge_moments(x, y, q)
Q = moments[-1]
harmonic_ranges = torch.stack((Q / (q / range_source).sum(-1), Q / (q / range_target).sum(-1)), dim=-1)
poses, weights, info = utils3d.pt.pose_graph_optimization_huber(
    num_nodes, edges, *moments, harmonic_ranges=harmonic_ranges,
    mode="similar", threshold=0.3,
)

Use positive, guarded ranges from actual observer origins and the same point weights as the original moments. Only the outer score is normalized by s_ij * H_source * H_target; inner point moments/information remain unchanged. Opposite queries share one Huber weight, clamped below by 1e-3. Defaults are 128 outer updates, weight-change tolerance 1e-4, and ten inner rotation refinement steps. All compact calculations use FP64, preserving input dtype and device at the output boundary. info reports iterations and convergence; nonconvergence emits a RuntimeWarning. This solver does not add graph edges or repair initially disconnected graphs. Pixel weight totals and optional edge priors remain coefficients of the inner solver, not the percentage score.

No per-iteration CPU graph diagnostics or raw point decoding are performed. The ordinary PGO and existing GNC APIs retain their behavior.

pose_graph_edge_moments already returns the total point weight as its sixth tensor, so two calls can supply all statistics needed by a relative RMS score:

inner = utils3d.pt.pose_graph_edge_moments(x, y, q)
scoring = utils3d.pt.pose_graph_edge_moments(x, y, q / range_source / range_target)
poses, weights = utils3d.pt.pose_graph_optimization_gnc(
    num_nodes, edges, *inner, mode="similar", threshold=0.05,
    residual_moments=scoring, residual_scale="geometric_mean",
)

Here ranges are fixed distances to the actual camera origins (include any calibration shifts), with positive, unit-consistent near-zero guards. Both calls must use the same point correspondences and validity mask. The inner PGO always uses inner; only the outer residual uses scoring. If the weight sums are Q=sum(q) and A=sum(q/(range_source*range_target)), the squared outer score is A/Q times the scoring-moment mean squared residual, additionally divided by relative Sim(3) scale in "geometric_mean" mode. The resulting RMS is invariant to source/target exchange of the same correspondences and independent local unit changes; 0.05 means 5%. Omitting both new keywords preserves the previous metric and GNC behavior. This change does not alter the continuation schedule, pair directional weights, or add connectivity safeguards.

No point-count output is needed for this weighted metric: its denominator is Q, not the sample count. Callers may separately retain point counts for diagnostics. The cached moments remain valid while point predictions, weights, calibration shifts and range guards stay fixed.

Both backends return original point indices, so the same selection can be applied to coordinates, colors, normals, or other attributes:

indices = utils3d.voxel_grid_sampling(points, voxel_size=0.01)
indices = utils3d.hierarchical_voxel_sampling(points, num_samples=100_000)
sampled_points = points[indices]

Inputs are finite floating-point arrays/tensors of shape (N, 3); outputs are one-dimensional int64 indices, on the input device for PyTorch.

• voxel_grid_sampling keeps the first input point in each occupied cubic voxel. The grid is anchored at the coordinate-wise minimum. Every input point is within sqrt(3) * voxel_size of a selected point, up to numerical rounding.
• hierarchical_voxel_sampling sorts 21-bit-per-axis Morton codes once, chooses an octree level, and refines a spatially dispersed subset of its cells. Final cells differ by at most one level. Each is represented by the first point in Morton order, with original index breaking ties. For 0 < num_samples < N, if U finest-grid cells are occupied and T = min(num_samples, U), the output count is between max(1, T - 6) and T. Duplicate coordinates or finest-grid quantization can reduce U; outliers can limit effective resolution. A zero target gives empty indices, and a target at least N returns all indices, including coincident points.

These are grid-based coverage samplers, not FPS or Poisson-disk sampling: there is no minimum point-separation guarantee, no progressive FPS ordering, and selections need not be nested across targets. Both use float64 grid arithmetic, O(N) workspace, and sorting (O(N log N) in the comparison model), without loops over input points or output samples. PyTorch selection runs without autograd; gathering the returned indices can still propagate gradients. The PyTorch implementations require a device supporting float64, read bounds and (for the hierarchical sampler) at most five level counts on the CPU, and otherwise keep point data on the input device.

Invalid shapes, dtypes, parameters, nonfinite coordinates, and unrepresentable coordinate ranges raise errors. Single-scale voxel coordinates must fit int64; if only the packed key would overflow, lexicographic sorting preserves distinct voxels instead of allowing collisions.

Run the NumPy / PyTorch CPU and available-CUDA regression checks with PYTHONPATH=. python test/test_sampling.py.

utils3d.pt.rasterize_triangles, utils3d.pt.rasterize_lines, and utils3d.pt.rasterize_point_cloud share a small, point-sampled Triton rasterizer. It implements homogeneous view-frustum clipping, backface culling, a top-left shared-edge rule with eight subpixel bits, and perspective-correct interpolation. Unsigned 64-bit atomic depth/face keys select the visible triangle; equal depths choose the lowest face ID. It scans triangle bounding boxes, without a scene graph, binning queues, or cached graphics contexts. Large overlapping or thin triangles can be slow; this is a reference implementation, not a replacement for a full graphics engine.

Install compatible GPU builds of PyTorch and Triton separately (pip install torch triton). No nvdiffrast, OpenGL context, or custom CUDA/C++ extension is needed for the Torch path. NVIDIA execution still requires a GPU driver and the PyTorch/Triton runtime; it does not require compiling .cu sources or configuring a CUDA toolkit. Triton is loaded only when rendering is requested. The NumPy/ModernGL backend is unchanged.

import torch
import utils3d

vertices = torch.tensor([
    [-0.8, -0.8, 0.0], [0.8, -0.8, 0.0], [0.0, 0.8, 0.0],
], device="cuda")
faces = torch.tensor([[0, 1, 2]], device="cuda")
colors = torch.eye(3, device="cuda", requires_grad=True)
result = utils3d.pt.rasterize_triangles(
    (256, 256), vertices=vertices, faces=faces, attributes=colors,
    return_interpolation=True,
)
result["image"].sum().backward()  # Gradients flow to colors, not geometry.

Coordinates and camera matrices are float32 GPU tensors. Faces are shared (T, 3) int32/int64 indices; omit them for (..., T, 3, D) triangle soup. Vertex coordinates may have 2, 3, or 4 components. Vertex, camera and attribute batch dimensions broadcast. Attribute tensors support any number of channels and retain their floating-point dtype. Use attributes_domain="face" for constant per-face attributes.

Outputs follow NumPy's top-left image convention: image has shape (..., H, W, C), and mask has shape (..., H, W). Optional depth is camera-space -z/w, with inf for background; with identity transforms this is simply negative input z. Optional interpolation_id contains original face IDs (-1 for background), and interpolation_uv contains the first two perspective-correct barycentric weights (zero for background). NDC depth uses strict < 1 far-plane testing. Geometry outputs are detached; only attributes receive gradients through native PyTorch indexing and weighted sums. Validation checks finite transformed coordinates and index bounds with one GPU-to-CPU synchronization. Background attributes are zero unless supplied explicitly.

Triangle, line and point-cloud rendering accept a background dictionary. Compositing uses PyTorch operations after rasterization, without changing the JIT core:

result = utils3d.pt.rasterize_triangles(
    (256, 256), vertices=vertices, faces=faces, attributes=colors,
    background={
        "image": background_image,  # (..., H, W, C), same dtype/channels as colors
        "depth": background_depth,  # (..., H, W), float32 camera-space -z/w
    },
    return_depth=True,
)

Every entry is optional. The image uses the same top-left origin as the outputs. Background depth is linear camera-space depth, not NDC, ray distance or an alpha channel. Finite values (including signed depths for identity transforms) and +inf are accepted; NaN and -inf raise errors. A covered foreground pixel wins only when its depth is strictly smaller; ties retain the background. Without background depth, foreground coverage alone selects the winner. Depth occlusion also works with return_depth=False: depth is computed internally but not returned. Point billboards use their center's camera depth for this test.

Background dictionaries may additionally carry mask (bool), interpolation_id (int32) and interpolation_uv (float32, last dimension 2) for triangles/lines, or point_id (int32) for points. All requested output maps follow the same winner. Missing fields default to zero image/UV, inf depth, -1 ID and false mask, matching NumPy's independent background-mask semantics. Mask is not inferred from an image or finite depth, and a false mask does not disable depth occlusion; supply a mask explicitly if background coverage should be reported. Metadata is returned only when its corresponding return flag is enabled. IDs are preserved without rebasing, so use a consistent ID convention when chaining render passes.

All tensors must share the geometry's GPU; spatial dimensions must match the requested image, while leading batch dimensions broadcast with geometry, cameras and attributes. Background images retain gradients, along with foreground attributes. Depth, selection and metadata remain detached. Inputs are not mutated. Without foreground attributes, a background image still produces an image output, with zero-valued foreground pixels. This is depth-based selection, not alpha blending. Depth peeling retains its separate background-free layer interface.

Point billboards and line rectangles are expanded on the GPU and reuse the triangle coverage/depth core, rather than maintaining separate depth algorithms. Attribute lookup and interpolation remain native PyTorch operations.

point_result = utils3d.pt.rasterize_point_cloud(
    (256, 256), points=vertices, attributes=colors,
    point_sizes=12, point_shape="circle", return_point_id=True,
)
line_result = utils3d.pt.rasterize_lines(
    (256, 256), vertices=vertices, attributes=colors,
    lines=torch.tensor([[0, 1], [1, 2], [2, 0]], device="cuda"),
    line_width=2.5, return_interpolation=True,
)

• Points: all five NumPy shapes are supported: triangle, square, pentagon, hexagon and circle. Sizes can be a scalar or a broadcasting (..., N) / (..., 1) GPU tensor. point_size_in="2d" uses pixels; "3d" uses camera-plane units, matching NumPy's billboard convention. Square size is its side length; circle size is its diameter; other polygons use the circumcircle diameter. Zero sizes are invisible. The full billboard is clipped, so centers outside the image can still contribute. Circles discard outside fragments before depth testing. Optional point_id stores original IDs with background -1; depth is the point center's camera-space depth.
• Lines: supply shared (L, 2) indices or unindexed (..., L, 2, D) endpoints. attributes_domain="line" selects constant per-line attributes; the default interpolates endpoint attributes perspective-correctly. Centerlines are clipped against all six frustum planes, then expanded to flat-capped rectangles with the requested perpendicular width in pixels. Widths need not be integers. Projected zero-length lines are invisible. interpolation_id stores original line IDs, while interpolation_uv stores the two original endpoint weights, including after clipping.
• Coverage is intentionally not bit-identical to OpenGL lines: the Torch implementation uses geometric rectangles, not driver-specific aliased line/diamond-exit rules. Diagonal edges and endpoints may cover different pixels. There is no antialiasing or automatic joining of adjacent segments.

Positions, cameras and point sizes are detached; only attributes receive gradients. The same batching, dtype, background, and optional-dependency rules apply to all three renderers. Expanded primitives require O(N) geometry storage, not an image-by-primitive tensor.

utils3d.pt.rasterize_triangles_peeling returns a context-managed layer iterator. It accepts the same geometry, camera, attribute-domain and output options as triangle rendering, without a graphics context:

with utils3d.pt.rasterize_triangles_peeling(
    (256, 256), vertices=vertices, faces=faces, attributes=colors,
    return_depth=True, return_interpolation=True,
) as layers:
    for i, layer in enumerate(layers):
        image = layer["image"]  # (..., H, W, C), including the final empty layer.
        if i == 3:
            break

The first layer matches ordinary rendering. Each next layer selects the closest fragment at strictly greater float32 NDC depth per pixel. Equal-depth fragments are skipped together; face ID breaks ties within a layer. Exhausted pixels stay empty, and iteration ends after yielding one entirely empty layer across the batch. Geometry is prepared once, then rerasterized per layer, with one CPU coverage check per layer. There is no fixed layer limit; leaving the with block closes the iterator. Outputs can be retained, and attribute losses can be summed over layers before backward. Do not mutate inputs while iterating.

Migration from the nvdiffrast wrapper: the Torch RastContext, sample_texture, and texture_composite APIs have been removed, as have the ctx, antialiasing, and return_image_derivatives rendering parameters. There is no geometry gradient, screen-space derivative, mipmap or antialiasing path. Texture sampling can be composed separately with PyTorch operations. Background image/depth compositing is supported as described above. Triangle peeling is now available through the Triton layer iterator above. NumPy-only functions remain available through utils3d.np and automatic dispatch.

Run the GPU, independent CPU-reference, optional NumPy/OpenGL, attribute-gradient and lazy-import regression checks with PYTHONPATH=. python test/test_rasterization.py.

• Camera & Projection & Coordinate Transforms
• Pose Solver
• Image & Maps
• Mesh
• Point Sampling
• Rasterization
• Array Utils
• Segment Array Operations
• IO

Function	Numpy	Pytorch
utils3d.bounding_rect_from_mask Get bounding rectangle of a mask	-	utils3d.pt.bounding_rect_from_mask(mask)
utils3d.build_grid_mesh Get mesh of height * width faces arranged in a 2D grid.	utils3d.np.build_grid_mesh(height, width, shared_vertices)	-
utils3d.build_mesh_from_depth_map Get a mesh by lifting depth map to 3D, while removing depths of large depth difference.	utils3d.np.build_mesh_from_depth_map(depth, maps, intrinsics, extrinsics, atol, rtol, domain, tri)	utils3d.pt.build_mesh_from_depth_map(depth, other_maps, intrinsics, extrinsics, atol, rtol, tri)
utils3d.build_mesh_from_map Get a mesh regarding image pixel uv coordinates as vertices and image grid as faces.	utils3d.np.build_mesh_from_map(maps, mask, domain, tri)	utils3d.pt.build_mesh_from_map(maps, mask, tri)
utils3d.chessboard Get a chessboard image	utils3d.np.chessboard(size, grid_size, color_a, color_b)	utils3d.pt.chessboard(size, grid_size, color_a, color_b)
utils3d.colorize_depth_map Colorize depth map for visualization.	utils3d.np.colorize_depth_map(depth, mask, near, far, cmap)	-
utils3d.colorize_normal_map Colorize normal map for visualization. Value range is [-1, 1].	utils3d.np.colorize_normal_map(normal, mask, flip_yz, normalize)	-
utils3d.colorize_probability_map Colorize probability map for visualization.	utils3d.np.colorize_probability_map(probability, mask, cmap, alpha, beta)	-
utils3d.colorize_segmentation_map Colorize segmentation map for visualization. The same value will be assigned with the same color.	utils3d.np.colorize_segmentation_map(segmentation, mask, vdim)	-
utils3d.depth_map_aliasing Compute the map that indicates the aliasing of x depth map, identifying pixels which neither close to the maximum nor the minimum of its neighbors.	utils3d.np.depth_map_aliasing(depth, atol, rtol, kernel_size, mask)	utils3d.pt.depth_map_aliasing(depth, atol, rtol, kernel_size, mask)
utils3d.depth_map_edge Compute the edge mask from depth map. The edge is defined as the pixels whose neighbors have large difference in depth.	utils3d.np.depth_map_edge(depth, atol, rtol, ltol, kernel_size, mask)	utils3d.pt.depth_map_edge(depth, atol, rtol, ltol, kernel_size, mask)
utils3d.depth_map_to_normal_map Calculate normal map from depth map. Value range is [-1, 1]. Normal direction in OpenCV identity camera's coordinate system.	utils3d.np.depth_map_to_normal_map(depth, intrinsics, mask, edge_threshold)	utils3d.pt.depth_map_to_normal_map(depth, intrinsics, mask)
utils3d.depth_map_to_point_map Unproject depth map to 3D points.	utils3d.np.depth_map_to_point_map(depth, intrinsics, extrinsics)	utils3d.pt.depth_map_to_point_map(depth, intrinsics, extrinsics)
utils3d.flood_fill Flooding fill the holes in the image(s) according to the mask.	utils3d.np.flood_fill(image, mask, return_index)	utils3d.pt.flood_fill(image, mask, return_index)
utils3d.fractal_perlin_noise_map Generate fractal Perlin noise map.	utils3d.np.fractal_perlin_noise_map(size, base_frequency, octaves, lacunarity, gain, seed, dtype)	utils3d.pt.fractal_perlin_noise_map(size, base_frequency, octaves, lacunarity, gain, seed, dtype, device)
utils3d.masked_area_resize Resize 2D map by area sampling with mask awareness.	utils3d.np.masked_area_resize(image, mask, size)	utils3d.pt.masked_area_resize(image, mask, size)
utils3d.masked_nearest_resize Resize image(s) by nearest sampling with mask awareness. Suitable for sparse maps.	utils3d.np.masked_nearest_resize(image, mask, size, return_index)	utils3d.pt.masked_nearest_resize(image, mask, size, return_index)
utils3d.normal_map_edge Compute the edge mask from normal map.	utils3d.np.normal_map_edge(normals, tol, kernel_size, mask)	-
utils3d.perlin_noise Generate Perlin noise for the given coordinates.	utils3d.np.perlin_noise(x, seed)	utils3d.pt.perlin_noise(x, seed)
utils3d.perlin_noise_map Generate Perlin noise map.	utils3d.np.perlin_noise_map(size, frequency, seed, dtype)	utils3d.pt.perlin_noise_map(size, frequency, seed, dtype, device)
utils3d.pixel_coord_map Get image pixel coordinates map, where (0, 0) is the top-left corner of the top-left pixel, and (width, height) is the bottom-right corner of the bottom-right pixel.	utils3d.np.pixel_coord_map(size, top, left, convention, dtype)	utils3d.pt.pixel_coord_map(size, top, left, convention, dtype, device)
utils3d.point_map_to_normal_map Calculate normal map from point map. Value range is [-1, 1].	utils3d.np.point_map_to_normal_map(point, mask, edge_threshold)	utils3d.pt.point_map_to_normal_map(point, mask)
utils3d.screen_coord_map Get screen space coordinate map, where (0., 0.) is the bottom-left corner of the image, and (1., 1.) is the top-right corner of the image.	utils3d.np.screen_coord_map(size, top, left, bottom, right, dtype)	utils3d.pt.screen_coord_map(size, top, left, bottom, right, dtype, device)
utils3d.uv_map Get image UV space coordinate map, where (0., 0.) is the top-left corner of the image, and (1., 1.) is the bottom-right corner of the image.	utils3d.np.uv_map(size, top, left, bottom, right, dtype)	utils3d.pt.uv_map(size, top, left, bottom, right, dtype, device)