Add more provider module categories to the provider page - #70190

Lee-W · 2026-07-21T12:54:22Z

Why

Follow up of #70122. Extra links, queues, plugins, auth managers, DB managers, and dialects were not browsable in the registry

What

Add extra links, queues, plugins, auth managers, DB managers, and dialects. Each of them now gets it own category tab. The row now sorted by module count and tail collapses into a "More" menu so the page won't be too messy

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: [Claude] following the guidelines

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

The items blocks for queues, plugins and task-decorators listed their field names directly under items, where they are not JSON Schema keywords, so the blocks were inert and any shape passed. A provider author following what those blocks appeared to document — a dict for queues, or a path key for a decorator — got no error and was quietly dropped by the consumers instead. Pinning each section to the shape the provider.yaml files and generated payloads actually use turns that into a failure at authoring time. The custom-provider howto named the same nonexistent path field. Raised in review of apache#70190.

extract_versions.py hardcoded a 4-entry FQCN_SECTIONS map, so backfilled (older) provider version pages only showed notifications, secrets backends, logging, and executors. The newer categories added for the provider page (extra links, queues, auth managers, db managers) only showed up on the latest version page, which used a separate code path already wired to all categories. extract_versions.py now derives its category list from CLASS_LEVEL_SECTIONS instead of duplicating it, and the category overrides that used to live only in extract_parameters.py move to a shared CLASS_LEVEL_CATEGORY_OVERRIDES constant so both scripts stay in sync going forward.

"Queues" was easy to confuse with the "queue" argument on BaseOperator, which is unrelated to message queue providers.

Class-level sections, plugins, dialects, and task-decorators each repeated the same split/import/getattr logic for resolving a dotted class or function path, making it easy for the failure handling in one copy to drift from the others. Also widens _get_first_docstring_line's parameter type to object, since it only ever reads __doc__ and doesn't require an actual class -- mypy caught the mismatch once the new helper's return type stopped inferring as Any at the task-decorators call site.

Dialects and Plugins showed up on a provider's latest page but silently vanished on every older version page: postgres, common.sql and microsoft.mssql for dialects, and seven providers for plugins. The version-page extractor only walked the yaml sections whose entries are bare class paths, and these two carry dict entries instead. Each extractor also remembered the dict entries' class-path field names on its own, which is what let the two fall out of step to begin with. They now read one definition, so a future dict-shaped section reaches both without a second edit.

types.json drives three more class families than the provider-page tabs: the stats page builds one card per entry with no zero-count filter, and the provider card's breakdown bar keys off the type id directly. All three stopped at decorator, so eight types rendered as colourless chips, grey share bars, and transparent breakdown slices. The new rules read the colour tokens through rgb(from var(...)) rather than copying each token's hex into an rgba() literal, the way the module tabs already do — a token change that misses a hand-copied literal is invisible until someone looks at the page.

Choosing a type from the overflow menu left no tab highlighted at all: the shared listener clears every tab and then marks the one inside the menu that had just been hidden, so nothing on screen said what the list was filtered by. The More button now carries that state. Closing the menu also dropped focus to the body whenever focus sat inside it, so keyboard users lost their place. Focus returns to the trigger only when it was inside the menu, which leaves a click elsewhere on the page alone. The menu also advertised role="menu"/"menuitem" without wiring up the arrow key navigation those roles imply. Dropping them keeps the remaining promise — aria-haspopup, aria-expanded and aria-controls — one the markup honours.

The plugins and dialects sections both list dict entries, and the two loops that walked them differed only in which key carries the integration name. Keeping that key in the shared table alongside the class-path key means a future dict-shaped section cannot drift the way the class-path field already did earlier in this branch.

C reads as connection to anyone scanning the module tabs, so it is worth keeping free for a future connection category rather than spending it on Dialects, which can take the initial of its own label once DB Managers moves to the initial of Manager.

The chip letter was derived from the type id at each render site, so the letters curated in types.json only ever reached the module tabs. That put db_manager on D beside decorator and dialect, which is the collision the curated letters exist to prevent. Reading the letter from one place removes the second derivation, and the card colour rules now cover every type instead of leaving the newer ones with a transparent chip.

The existing checks run in the subset direction, so a section added to the flat table but to neither class-level table passes them while both extractors skip it silently — the failure this branch already hit once. The FQCN suffix lookup cannot catch it either, since it only fires for keys already in a table.

The items blocks for queues, plugins and task-decorators listed their field names directly under items, where they are not JSON Schema keywords, so the blocks were inert and any shape passed. A provider author following what those blocks appeared to document — a dict for queues, or a path key for a decorator — got no error and was quietly dropped by the consumers instead. Pinning each section to the shape the provider.yaml files and generated payloads actually use turns that into a failure at authoring time. The custom-provider howto named the same nonexistent path field. Raised in review of apache#70190.

boring-cyborg Bot added area:dev-tools area:registry backport-to-v3-3-test Backport to v3-3-test labels Jul 21, 2026

github-project-automation Bot added this to Airflow Registry Jul 21, 2026

github-project-automation Bot moved this to Backlog in Airflow Registry Jul 21, 2026

Lee-W removed the backport-to-v3-3-test Backport to v3-3-test label Jul 21, 2026

Lee-W marked this pull request as ready for review July 21, 2026 14:11

Lee-W requested review from amoghrajesh, ashb, bugraoz93, choo121600, ephraimbuddy, gopidesupavan, jason810496, jedcunningham, jscheffl, kaxil, potiuk and vatsrahul1001 as code owners July 21, 2026 14:11

potiuk approved these changes Jul 21, 2026

View reviewed changes

phanikumv reviewed Jul 21, 2026

View reviewed changes

Comment thread dev/registry/extract_parameters.py Outdated

kaxil reviewed Jul 21, 2026

View reviewed changes

Comment thread dev/registry/registry_tools/types.py

kaxil reviewed Jul 21, 2026

View reviewed changes

Comment thread dev/registry/registry_tools/types.py

Lee-W force-pushed the add-more-registry-module-categories branch 3 times, most recently from 1bae222 to e0be06b Compare July 22, 2026 05:43

Lee-W requested a review from kaxil July 22, 2026 10:12

phanikumv reviewed Jul 22, 2026

View reviewed changes

Comment thread dev/registry/registry_tools/types.py

Lee-W force-pushed the add-more-registry-module-categories branch 2 times, most recently from a1ffe7e to 4ab25a3 Compare July 23, 2026 07:10

uranusjr reviewed Jul 31, 2026

View reviewed changes

Comment thread dev/registry/registry_tools/types.py Outdated

Lee-W force-pushed the add-more-registry-module-categories branch 3 times, most recently from 7ab87de to 8e5c851 Compare August 3, 2026 10:06

kaxil approved these changes Aug 3, 2026

View reviewed changes

Comment thread registry/src/_data/types.json

Comment thread dev/registry/tests/test_types.py

Lee-W mentioned this pull request Aug 4, 2026

Validate provider metadata list entries instead of ignoring them #71104

Open

1 task

Lee-W force-pushed the add-more-registry-module-categories branch 2 times, most recently from 0c5244c to 841df8d Compare August 5, 2026 06:40

Lee-W force-pushed the add-more-registry-module-categories branch 2 times, most recently from db45faf to b32ba75 Compare August 5, 2026 12:24

Lee-W and others added 12 commits August 6, 2026 15:00

Add more provider module categories to the provider page

2039c05

Relabel Message Queues module type on the provider page

8943510

"Queues" was easy to confuse with the "queue" argument on BaseOperator, which is unrelated to message queue providers.

Fix a typo in the dotted-path resolution test name

7560975

Lee-W force-pushed the add-more-registry-module-categories branch from b32ba75 to d530123 Compare August 6, 2026 07:00

Lee-W merged commit b2a195a into apache:main Aug 6, 2026
156 checks passed

Lee-W deleted the add-more-registry-module-categories branch August 6, 2026 08:22

github-project-automation Bot moved this from Backlog to Done in Airflow Registry Aug 6, 2026

Navigation Menu

Uh oh!

Add more provider module categories to the provider page - #70190