Generate dynamic mode documentation

[szkarpinski]

Category:

Other (e.g. Documentation, Tests, Configuration)

Description:

New content

• API reference, documenting classes such as Batch, Tensor, EvalContext and related functions.
• Data reader reference, listing the available readers
• Operator reference, with a table listing Dynamic Mode operators.

Internal changes

• autodoc_submodules.py has been extended with dynamic_autodoc and dynamic_readers_autodoc, generating autodoc stubs for dynamic mode operators.
• operations_table.py has been generalized, to be able to generate tables for dynamic ops & readers. Some functions now get additional api_name argument instead of hardcoding "fn".

Other changes

• Small formatting fixes in existing docstrings to make RST render correctly

Additional information:

Affected modules and functionalities:

• Sphinx documentation and the docs autogeneration scripts

Key points relevant for the review:

• Does the organization of the docs make sense?
• Are links clickable, do they redirect to correct pages, does page content match the title?
• fn docs still look good? (the code is shared)
• The docs are generated from whatever is in the signatures/function docstrings. There's definitely room for improvement there, but this is out of scope.

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: DALI-4446

[greptile-apps]

Greptile Overview

Greptile Summary

This PR adds comprehensive documentation for DALI Dynamic Mode, including API references, operator listings, and data reader documentation. The implementation properly extends the existing documentation generation infrastructure to support the dynamic API alongside the traditional fn API.

Key changes:

• Documentation structure: Creates new docs/dali_dynamic/ directory with index, overview, API reference, operators reference, and readers reference pages
• Code generation: Adds dynamic_autodoc() and dynamic_readers_autodoc() functions in autodoc_submodules.py to generate RST stubs for dynamic mode operators and readers
• Table generation: Generalizes operations_table.py to support multiple APIs by adding api_name parameter and to_other_module() helper
• Docstring fixes: Improves RST formatting in Python source files (_batch.py, _tensor.py, _eval_context.py) to ensure proper rendering
• Refactoring: Consolidates duplicate module creation logic in _op_builder.py to use shared _internal.get_submodule() utility

The documentation organization is logical and consistent with existing DALI docs. All links use proper Sphinx cross-references. The code properly handles the distinction between function-based operators (using single_fun_file) and class-based readers (using single_class_op_file).

Confidence Score: 5/5

• This PR is safe to merge with no issues identified
• The changes are well-structured documentation additions with proper code reuse and refactoring. All modifications follow existing patterns, RST formatting is correct, and the code properly handles both function-based and class-based documentation generation. No functional logic changes to runtime code.
• No files require special attention

Important Files Changed

File Analysis

Filename	Score	Overview
docs/autodoc_submodules.py	5/5	Adds dynamic_autodoc and dynamic_readers_autodoc functions to generate documentation for dynamic mode operators, with proper refactoring and code reuse
docs/operations_table.py	5/5	Generalizes table generation to support dynamic API with new api_name parameter and adds dynamic_readers_table function
docs/conf.py	5/5	Integrates dynamic mode documentation generation by creating output paths and calling new autodoc functions
docs/dali_dynamic/index.rst	5/5	New dynamic mode landing page with overview and toctree linking to API reference, readers, and operations
docs/dali_dynamic/api_reference.rst	5/5	New API reference documenting core classes like Batch, Tensor, EvalContext, and related functions
dali/python/nvidia/dali/experimental/dynamic/_batch.py	5/5	Fixes RST formatting in docstrings: adds cross-references, fixes code blocks, improves list formatting

Sequence Diagram

sequenceDiagram
    participant Sphinx as Sphinx Build
    participant conf.py as conf.py
    participant ops_table as operations_table.py
    participant autodoc as autodoc_submodules.py
    participant fs as File System
    
    Sphinx->>conf.py: Initialize documentation build
    conf.py->>fs: Create docs/dali_dynamic/operations/ directory
    
    conf.py->>ops_table: operations_table(dynamic_table, module="nvidia.dali.experimental.dynamic")
    ops_table->>ops_table: Detect api_name="dynamic" from module
    ops_table->>ops_table: Generate table for dynamic operators
    ops_table->>fs: Write dynamic_table
    
    conf.py->>ops_table: dynamic_readers_table(dynamic_readers_table)
    ops_table->>ops_table: Filter readers from dynamic ops
    ops_table->>ops_table: Generate readers table
    ops_table->>fs: Write dynamic_readers_table
    
    conf.py->>autodoc: dynamic_autodoc(dynamic_autodoc, paths, references)
    autodoc->>autodoc: get_modules(dynamic_modules)
    autodoc->>autodoc: Filter functions with schema attribute
    autodoc->>autodoc: write_module_file() for each module
    autodoc->>autodoc: write_function_files(single_fun_file) for operators
    autodoc->>fs: Write dynamic_autodoc and operator RST files
    
    conf.py->>autodoc: dynamic_readers_autodoc(dynamic_readers_autodoc, paths, references)
    autodoc->>autodoc: Filter modules with "readers"
    autodoc->>autodoc: Filter classes subclassing Reader
    autodoc->>autodoc: write_module_file() for each module
    autodoc->>autodoc: write_function_files(single_class_op_file) for readers
    autodoc->>fs: Write dynamic_readers_autodoc and reader RST files
    
    Sphinx->>fs: Read generated RST files
    Sphinx->>Sphinx: Render documentation

Loading

[greptile-apps]

_{4 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile
React with 👍 or 👎 to share your feedback on this new summary format}

[greptile-apps]

style: missing newline at end of file

[rostan-t]

Nitpick: This would be nice occurrences of Tensor/Batch could reference the actual classes. This was not possible with the fn API as there's no TensorList class exposed in Python but it makes sense for dynamic mode.

[klecki]

):

The if condition is pretty big, and ideas on how to make it a tad more readable? Should we just split it to more conditions?

[szkarpinski]

Split, hope it's still equivalent :P

[klecki]

I wonder if we should maybe make the intro contain the code sample, rather than the key features? Or maybe just keep all of the introduction at the top page without repeat?

[szkarpinski]

Replaced with sample, good point

[klecki]

Yet another way of accessing the schema?

Sidenote, if ndd operators have a public schema member, maybe they shouldn't.

[klecki]

Do we plan to have any other classes than readers? If yes, I'm starting to spin the crank on our ski jumping hero. ⚠️ 📯

[szkarpinski]

Random ops, I guess

[klecki]

This is almost a copy of fn_autodoc. Same for dynamic_readers_autodoc. Can we generalize.

Maybe we can take out the filtering of what to document to the higher scope or make it a function, and the rest is the same - loop over filtered members -> generate doc -> write to a file.

[szkarpinski]

Refactored. Thanks!

[greptile-apps]

_{16 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

syntax: missing newline at end of file

Suggested change

	and transition between the two modes is straightforward.
	and transition between the two modes is straightforward.

[greptile-apps]

_{16 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{15 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[szkarpinski]

!build

[dali-automaton]

CI MESSAGE: [39073360]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [39073360]: BUILD PASSED