Add `semver_ord(num)` SQL function and `version.semver_ord` column #10763

Turbo87 · 2025-03-06T07:45:17Z

We currently rely on the Rust semver crate to implement our "sort by semantic versioning" functionality, that is used by web interface, but also to determine the "default version". This has the downside that we need to load the full list of version numbers for a crate from the database to the API server, sort it and then throw away the ones that we don't need.

Ideally, we would be using https://pgxn.org/dist/semver/ or https://github.com/pgcentralfoundation/pgrx to solve this, but unfortunately most PostgreSQL hosters don't allow/support custom extensions, and it would also make local development a bit more challenging.

As a workaround, this PR implements a semver_ord(num) pgSQL function that returns a JSONB array, which has the same ordering precedence as the Semantic Versioning spec (https://semver.org/#spec-item-11), with the small caveat that it only supports up to 15 prerelease parts. The maximum number of prerelease parts in our current dataset is 7, so 15 should be plenty.

Update: After discussion in this PR we changed the maximum number of prerelease specifiers to 10, but the remaining specifiers are now appended to the array as a string, to at least ensure uniqueness and a stable sort order.

The database migration in this commit also adds a new semver_ord column to the versions table, and an on-insert trigger function that automatically derives the semver_ord column from the num column value:

Once this migration has run, the existing versions can be backfilled by running the following SQL script, until all versions are processed:

with versions_to_update as (
    select id, num
    from versions
    where semver_ord = 'null'::jsonb
    limit 1000
)
update versions
    set semver_ord = semver_ord(num)
    where id in (select id from versions_to_update);

This PR does not yet implement any code to actually use this new column, since it will need to be backfilled first. Once that has happened and verified to produce correct results, we can start to migrate our codebase to move the semver sorting into the database and potentially even calculate the "default version" directly in the database too.

migrations/2025-03-06-060640_semver_ord/up.sql

eth3lbert

Thank you for this excellent PR. This will significantly simplify the work involving version sorting. 💪 💪 💪
I've left some non-blocking nitpicks, but everything is currently great! 👍

crates/crates_io_database/tests/semver_ord.rs

migrations/2025-03-06-060640_semver_ord/up.sql

eth3lbert · 2025-03-06T17:54:13Z

migrations/2025-03-06-060640_semver_ord/up.sql

+                -- In JSONB a number has higher precedence than a string but in
+                -- semver it is the other way around, so we use true/false to
+                -- work around this.


Here are some references to facilitate the review. The ordering for jsonb, as excerpted from the official doc¹, is as follows:

Object > Array > Boolean > Number > String > null Object with n pairs > object with n - 1 pairs Array with n elements > array with n - 1 elements

The SemVer's pre-release ordering, as excerpted from the spec², is as follows:

Precedence for two pre-release versions with the same major, minor, and patch version MUST be determined by comparing each dot separated identifier from left to right until a difference is found as follows: 1. Identifiers consisting of only digits are compared numerically. 2. Identifiers with letters or hyphens are compared lexically in ASCII sort order. 3. Numeric identifiers always have lower precedence than non-numeric identifiers. 4. A larger set of pre-release fields has a higher precedence than a smaller set, if all of the preceding identifiers are equal.

Footnotes

https://www.postgresql.org/docs/current/datatype-json.html#JSON-INDEXING ↩

https://semver.org/#spec-item-11 ↩

Turbo87 · 2025-03-07T09:44:36Z

@Gankra you know a lot about semver edge cases. do you want to try and break this before we actually start using the new column? :D

Gankra · 2025-03-07T14:42:01Z

ahahahaha

I think as long as it handles
https://crates.io/crates/cursed-trying-to-break-cargo/
ok there's not much else for me to throw at it?

Gankra · 2025-03-07T14:58:22Z

Also dang I know that page induces NaNs (improved from "bootloops crates.io's client") but what a chaotic sort haha.

Turbo87 · 2025-03-07T14:59:36Z

I think as long as it handles
https://crates.io/crates/cursed-trying-to-break-cargo/
ok there's not much else for me to throw at it?

it's looking good so far :)

Turbo87 · 2025-03-07T15:00:26Z

Also dang I know that page induces NaNs (improved from "bootloops crates.io's client") but what a chaotic sort haha.

are you sure you were in semver sorting mode? https://crates.io/crates/cursed-trying-to-break-cargo/versions?sort=semver looks okayish to me 😅

Gankra · 2025-03-07T19:42:48Z

Oh you're totally right, it's so rare to see date and semver not be 1:1 for crates I forgot date is the default :)

LawnGnome

My PL/pgSQL is very rusty, but this LGTM. 👍

LawnGnome · 2025-03-07T19:50:25Z

crates/crates_io_database/tests/semver_ord.rs

+    insta::assert_snapshot!(check("0.0.0").await, @r#"[0, 0, 0, {}]"#);
+    insta::assert_snapshot!(check("1.0.0-alpha.1").await, @r#"[1, 0, 0, [true, "alpha", false, 1, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, ""]]"#);
+
+    // see https://crates.io/crates/cursed-trying-to-break-cargo/1.0.0-0.HDTV-BluRay.1020p.YTSUB.L33TRip.mkv – thanks @Gankra!


I love that this is represented as-is. 😆

LawnGnome · 2025-03-07T19:53:33Z

migrations/2025-03-06-060640_semver_ord/up.sql

+        -- A JSONB object has higher precedence than an array, and versions with
+        -- prerelease specifiers should have lower precedence than those without.


I went to check that we're not relying on any undocumented behaviour, and found this gem:

The btree ordering for jsonb datums is seldom of great interest[...]

Little did they know!

yep, see #10763 (comment) :)

We currently rely on the Rust `semver` crate to implement our "sort by semantic versioning" functionality, that is used by web interface, but also to determine the "default version". This has the downside that we need to load the full list of version numbers for a crate from the database to the API server, sort it and then throw away the ones that we don't need. This commit implements a `semver_ord(num)` pgSQL function that returns a JSONB array, which has the same ordering precedence as the Semantic Versioning spec (https://semver.org/#spec-item-11), with the small caveat that it only supports up to 15 prerelease parts. The maximum number of prerelease parts in our current dataset is 7, so 15 should be plenty. The database migration in this commit also adds a new `semver_ord` column to the `versions` table, and an on-insert trigger function that automatically derives the `semver_ord` column from the `num` column value. Once this migration has run, the existing versions can be backfilled by running the following SQL script, until all versions are processed: ```sql with versions_to_update as ( select id, num from versions where semver_ord = 'null'::jsonb limit 1000 ) update versions set semver_ord = semver_ord(num) where id in (select id from versions_to_update); ```

…xt item

Turbo87 · 2025-03-10T13:01:41Z

migrations/2025-03-06-060640_semver_ord/up.sql

+-- with versions_to_update as (
+--     select id, num
+--     from versions
+--     where semver_ord = 'null'::jsonb


I just noticed that I didn't update this correctly. this line should now be where semver_ord is null.

Turbo87 added C-enhancement ✨ Category: Adding new behavior or a change to the way an existing feature works A-backend ⚙️ labels Mar 6, 2025

Turbo87 requested a review from a team March 6, 2025 07:45

Turbo87 force-pushed the semver-ord branch from 6d901dc to 4a45211 Compare March 6, 2025 07:55

eth3lbert reviewed Mar 6, 2025

View reviewed changes

migrations/2025-03-06-060640_semver_ord/up.sql Show resolved Hide resolved

Turbo87 requested a review from eth3lbert March 6, 2025 15:46

eth3lbert approved these changes Mar 6, 2025

View reviewed changes

Turbo87 added this to crates.io team meetings Mar 7, 2025

Turbo87 moved this to For next meeting in crates.io team meetings Mar 7, 2025

LawnGnome approved these changes Mar 7, 2025

View reviewed changes

Turbo87 added 5 commits March 10, 2025 11:31

database/semver_ord: Add serialization test

7012704

database/semver_ord: Add remaining prerelease identifiers as final te…

e3ebad5

…xt item

database/semver_ord: Return null if version number can't be parsed

7438bdf

database/semver_ord: Add descending order snapshot too

2f6d121

Turbo87 force-pushed the semver-ord branch from 1d8fa47 to 2f6d121 Compare March 10, 2025 10:31

Turbo87 merged commit 7a11278 into rust-lang:main Mar 10, 2025
10 checks passed

Turbo87 deleted the semver-ord branch March 10, 2025 11:23

Turbo87 commented Mar 10, 2025

View reviewed changes

Turbo87 mentioned this pull request Mar 12, 2025

database/schema: Add missing semver_ord column #10814

Merged

LawnGnome moved this from For next meeting to Backlog in crates.io team meetings Mar 12, 2025

Turbo87 mentioned this pull request Mar 13, 2025

Use semver_ord column for sorting versions #10819

Merged

		-- A JSONB object has higher precedence than an array, and versions with
		-- prerelease specifiers should have lower precedence than those without.

Add semver_ord(num) SQL function and version.semver_ord column #10763

Add semver_ord(num) SQL function and version.semver_ord column #10763

Uh oh!

Conversation

Turbo87 commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

eth3lbert left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eth3lbert Mar 6, 2025

Choose a reason for hiding this comment

Footnotes

Uh oh!

Turbo87 commented Mar 7, 2025

Uh oh!

Gankra commented Mar 7, 2025

Uh oh!

Gankra commented Mar 7, 2025

Uh oh!

Turbo87 commented Mar 7, 2025

Uh oh!

Turbo87 commented Mar 7, 2025

Uh oh!

Gankra commented Mar 7, 2025

Uh oh!

LawnGnome left a comment

Choose a reason for hiding this comment

Uh oh!

LawnGnome Mar 7, 2025

Choose a reason for hiding this comment

Uh oh!

LawnGnome Mar 7, 2025

Choose a reason for hiding this comment

Uh oh!

Turbo87 Mar 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Turbo87 Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Add `semver_ord(num)` SQL function and `version.semver_ord` column #10763

Add `semver_ord(num)` SQL function and `version.semver_ord` column #10763

Turbo87 commented Mar 6, 2025 •

edited

Loading