Media: Add UltraHDR (ISO 21496-1) gain map support#74873
Open
adamsilverstein wants to merge 17 commits intotrunkfrom
Open
Media: Add UltraHDR (ISO 21496-1) gain map support#74873adamsilverstein wants to merge 17 commits intotrunkfrom
adamsilverstein wants to merge 17 commits intotrunkfrom
Conversation
adamsilverstein
requested review from
ajitbohra,
gziolo,
nerrad,
ntwb and
ryanwelcher
as code owners
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the Unlinked AccountsThe following contributors have not linked their GitHub and WordPress.org accounts: @gregbenz. Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases. If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
adamsilverstein
added
[Type] Enhancement
github-project-automation
bot
moved this to 🔎 Needs Review
in WordPress 7.0 Editor Tasks
|
Flaky tests detected in dbd21b2. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/24368334844
|
|
Size Change: +221 kB (+2.86%) Total Size: 7.96 MB 📦 View Changed
ℹ️ View Unchanged
|
Contributor
andrewserong
left a comment
andrewserong
left a comment
There was a problem hiding this comment.
Nice, this is testing well for me! I was going to ask how heavy the upstream library is, and then I noticed that it's from your own repo, so I assume you're quite comfortable with it 😄
The main question I have is to do with the idea that if most uploaded JPEGs are not UltraHDR, how expensive is the operation to detect it? If it's very cheap, then this seems like a great addition. I've left a comment, but before I looked at the code, my assumption is that this kind of operation would flag if it succeeds / does something, but wouldn't warn if it can't find an UltraHDR data.
What do you think?
This is all still behind an experiment of course, so don't let my questions here stop you from progressing! (Also I'll be AFK early next week, so apologies if my reply is delayed until mid-week)
Comment on lines
+460
to
+464
| } catch ( error ) { | ||
| // If UltraHDR detection fails, continue with regular upload | ||
| // eslint-disable-next-line no-console | ||
| console.warn( 'UltraHDR detection failed:', error ); | ||
| } |
Contributor
There was a problem hiding this comment.
I'm getting this warning for uploading normal (non UltraHDR) jpegs from my desktop. Given that the majority of jpegs won't be ultra HDR, should we just fail silently here?
Member
Author
There was a problem hiding this comment.
yes, this is mainly for debugging purposes and I will remove before merging to trunk.
Member
Author
I am comfortable with it working based on the tests, although I also plan to ask for help doing manual testing to verify it properly handles real world images.
That is a great question and worth investigating more. I will review the detection logic to see if it can be optimized. Also, I'm still not certain about what we should output. My assumption is that UltraHDR JPEGs are decoder compatible with existing JPEGs so we would want to output UltraHDR for uploaded UltraHDR. Then again, some users may prefer to use the SDR version for their website exclusively because they want smaller file sizes and don't care about HDR. We might need a new way for developers to implement that since the mime type is still image/jpeg and out=r
Hopefully for something fun! |
|
@adamsilverstein libultrahdr includes a probe() method to validate that required components are present without decoding. I haven't tested it for performance, but it may serve as inspiration. Gain maps require an auxiliary image. You could check the header and reject if that's missing. If it exists, you could then further check for a supported gain map (ie a more robust check, but you've already quickly skipped most images that aren't relevant). There are two kinds of map encoding supported by libultra: (a) ISO standard gain maps and (b) gain maps encoded with the Android XMP spec. The ISO encoding is widely used for new images now and has been for a while in most encoders. Many now dual encode (as the auxiliary image is the same and you're just writing redundant data - binary in the aux image for ISO and standard XMP for the Android spec). The only thing you'd miss by skipping XMP would be older images captured on Android or exported with old versions of Adobe software (Apple has a proprietary method they used pre-ISO and it is not supported by libultra). I believe skipping XMP would generally be ok to do, as it is a very limited edge case (and the impact would be that transcoding falls back to SDR rather than total failure). |
|
@adamsilverstein If the source includes a supported gain map, the output should preserve it by default. SharpJS is working towards a keepGainMap() method which should do this, however, I'm not sure how much guidance libvips needs. Coordinating with SharpJS may make sense here, as both efforts are likely chasing the same goals for default transcoding (ie ability to do basic crop/resize/compress while retaining an output which shows high fidelity to the original). Questions for transcoding will inevitably come up around tuning (compression of the base image / map). Testing to make sure the final result is ok based on the compression applied to both the base and map will be an important step to validate quality vs size objectives. Existing approaches for the base image probably translate well, and this is probably mostly a question of how to compress the map. Aside from that, most of the options chosen for encoding should remain as they were in the source. The metadata should be unchanged (HDR capacity, offsets, ranges, etc). The map scaling (1:1 vs 1:2 or 1:4) and number of channels (1 for luminosity only map vs 3 for full color) should remain the same. It should not alter sub-sampling (ie keep 444 if that's how it was encoded as the map is not a color image and the assumptions behind sub-sampling are not applicable in this domain, and can cause artifacts if altered). An affordance for aggressive compression of the image (including loss of HDR) may be useful for some users / workflows, but is not required and could get complicated. There are also other ways beyond stripping the map which can compress the image without stripping HDR, but they involve quality tradeoffs that would need careful assessment. For example, you could downgrade the map from 1:1 resolution in full color to a 1:2 map with luminosity only. That will cause loss of high frequency detail in the HDR rendition, as well as color error (which may be a small or significant issue depending on the relationship of the SDR to the HDR rendition). Due to the effort, complexity, and risk of confusion here, it may be best that initial support just preserve the gain map and not use more complex options to compress the image further. That can be managed now via the encoding of the file uploaded to WordPress (upload SDR if you only want SDR, phone captures are already low resolution luminosity maps, etc). Advanced transcoding for compression might best be left for 3rd-party plugins to address. |
Member
Author
great, thanks for the tip - i will take a look at that.
👍🏼
Right, be default we will always try to use the uploaded format for the output (so uploaded UltraHDR should output UltraHDR). My pondering was more about how would enable developers to choose to extract only the SDR image for output/front end if thats what they wanted. The current output mapping won't work, so we may need a more explicit filter to choose HDR or SDR encoding when an UltraHDR is uploaded. |
Member
Author
I created an issue to add a probe mode to lib-open-ultrahdr: |
Member
Author
Thanks again for the tip @gregbenz - I added the probe feature in adamsilverstein/lib-open-ultrahdr#6 based on the libultrahdr approach |
Member
Author
Thanks again for the question @andrewserong - it turns out libultrahdr has a |
adamsilverstein
removed request for
ajitbohra,
gziolo,
nerrad,
ntwb and
ryanwelcher
adamsilverstein
added
the
[Status] In Progress
github-actions
bot
added
[Package] DOM
Introduces two new packages for UltraHDR/HDR gain map image support: - @wordpress/ultrahdr-wasm: Rust/WASM library implementing ISO 21496-1 gain map specification with JPEG parsing, XMP metadata handling, and gain map encode/decode functionality - @wordpress/ultrahdr: TypeScript bindings for the WASM library with async loading and WordPress-style API Key features: - Detect UltraHDR images via isUltraHdr() - Decode UltraHDR to extract SDR base, gain map, and metadata - Encode UltraHDR from SDR + HDR image pairs - Extract SDR base for backwards compatibility Also updates: - upload-media store with UltraHDR operation types - webpack config with WASM asset handling - Fixes import order in webpack.config.js Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replace the local @wordpress/ultrahdr-wasm dependency with the published open-ultrahdr-wasm package from npm, updating imports and WASM file paths accordingly. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Switch from the local @wordpress/ultrahdr and @wordpress/ultrahdr-wasm packages to the published open-ultrahdr npm package, which provides identical functionality. This removes ~28 local files and simplifies the dependency chain. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The Attachment interface in upload-media was missing the `meta` property that exists in the media-utils package. This caused TypeScript errors when trying to use the meta field for UltraHDR metadata storage. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Switch from isUltraHdr() to probeUltraHdr() for richer probe results including hdrCapacity. Upload UltraHDR JPEGs unmodified since they already embed a backwards-compatible SDR base that non-HDR displays can use. Remove ExtractSdr operation and the console.warn for non-UltraHDR JPEGs (per reviewer feedback).
The dependency belongs only in the upload-media workspace package, not at the monorepo root. Regenerate lockfile from trunk's base to fix CI check-local-changes failure.
When an UltraHDR JPEG is detected, replace remaining operations with Upload/ThumbnailGeneration/Finalize to skip any transcoding that would strip HDR gain map data. Also normalize meta before spreading since the Attachment type allows empty arrays.
Contributor
andrewserong
left a comment
andrewserong
left a comment
There was a problem hiding this comment.
Just catching up on this one again, I've added a few questions, but nothing really blocking.
Can you expand on why the webpack changes need to be made, I thought the wp-build process handled inlining wasm?
And is this one critical for 7.0? My thinking is that if UltraHDR support isn't already in WordPress, is this more of an enhancement rather than a bug fix at this stage of the release? (I'm mostly cautious about adding additional dependencies at this point in the cycle)
| Finalize = 'FINALIZE', | ||
| // UltraHDR operations | ||
| DetectUltraHdr = 'DETECT_ULTRAHDR', | ||
| CreateUltraHdr = 'CREATE_ULTRAHDR', |
Contributor
There was a problem hiding this comment.
Is CreateUltraHdr being used anywhere?
Comment on lines
+829
to
+833
| operations: [ | ||
| OperationType.Upload, | ||
| OperationType.ThumbnailGeneration, | ||
| OperationType.Finalize, | ||
| ], |
Contributor
There was a problem hiding this comment.
Just double-checking: this is replacing the list of operations with this new list? It stood out to me because this action is detectUltraHdr and it's invoked by prepareItem which adds the additional operations. So it seemed a little odd to me that detect would then also handle queuing up more operations, and that they match the ones that are queued by prepareItem anyway?
| // Dynamically import to avoid loading WASM unless needed | ||
| const { probeUltraHdr } = await import( 'open-ultrahdr' ); | ||
| const buffer = await item.file.arrayBuffer(); | ||
| const result = await probeUltraHdr( buffer ); |
Contributor
There was a problem hiding this comment.
The try/catch block encompasses dispatching operations as well, but I wasn't sure how much the try/catch block is intending to catch. Is it just the above three lines?
Member
Author
This probably has some stale changes in it from a previous state, let me clean it up!
Planning UltraHDR support for 7.1. |
Narrow the try/catch in detectUltraHdr to only wrap the probe logic, making it clearer what errors are being caught. Move the UltraHDR detection result check outside the try block. Add comment explaining why operations are replaced (to skip transcoding that would strip HDR gain map data). Remove unused CreateUltraHdr enum value.
Revert the webpack asset/resource rule for .wasm files and instead inline the UltraHDR WASM binary at build time using the existing esbuild wasmInlinePlugin, matching the pattern used by vips. This avoids separate file downloads and MIME type serving issues. Use the new setWasmUrl() API from open-ultrahdr 0.1.2 to pass the inlined data URL directly to the WASM initialization.
4 tasks
When an UltraHDR/ISO 21496-1 image is uploaded, sub-sizes (thumbnails, scaled versions) now retain the gain map data. During detection, the gain map and metadata are decoded and cached. Each sub-size is resized normally via vips, then re-encoded as UltraHDR by reconstructing HDR pixel data from the SDR base and gain map using the ISO 21496-1 formula. Format transcoding is skipped for UltraHDR images since it would strip the gain map. Cache entries are cleaned up when the parent is finalized.
Use .buffer to get the underlying ArrayBuffer from the Uint8Array when constructing a Blob, satisfying stricter TypeScript BlobPart type.
Wrap cached.gainMap.buffer with new Uint8Array() to convert ArrayBufferLike (which includes SharedArrayBuffer) to a valid BlobPart type accepted by the Blob constructor.
TypeScript 5.9 treats Uint8Array.buffer as ArrayBufferLike which includes SharedArrayBuffer, but Blob only accepts ArrayBuffer. Use explicit cast since decoded JPEG data is always ArrayBuffer.
Member
Author
Note: this PR currently uses open-ultrahdr due to license incompatibility with libultrahdr. However, Google has agreed to dual license the Once google/libultrahdr#386 is reviewed and merged, we can switch this PR to using that library. Since the libraries offer similar functional signatures, the PR should be relatively minor for this change. |
Resolve conflict in packages/upload-media/src/store/private-actions.ts by keeping the PR branch's UltraHDR-aware changes.
Preserve trunk's vipsBatchResizeImage() optimization for non-UltraHDR uploads while keeping the per-thumbnail fallback as the primary path for UltraHDR images. The batch routine uses thumbnailImage() which strips gain maps, so UltraHDR sub-sizes must be resized individually with EncodeUltraHdr to preserve HDR data.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
open-ultrahdrnpm package (which wrapsopen-ultrahdr-wasm) for all UltraHDR operationsCloses #74874
Similar to adamsilverstein/client-side-media-experiments#15
Changes
open-ultrahdrdependency to@wordpress/upload-mediaDetectUltraHdrandExtractSdroperations to the upload queueprobeUltraHdrfor gain map detection in uploaded imagesTest plan