You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add definitions / diagrams for subtle video frame concepts #166
Triage note: marking 'editorial', as there is broad consensus on what these terms / concepts mean, this issue merely tracks the need to document that better.
I agree, from a high-level POV, but this needs to be precisely specified, because we all know small details matter when shipping APIs on the Web.
That said, it's not hard to spec I assume, granted we use the commonly accepted definitions, an attempt below:
The coded width is the size of a horizontal pixel line, that is bigger or equal to the stride. The coded height is the size of a vertical pixel line, that is bigger or equal to the height of the picture (but is that useful ?). The coded height x the coded width allows knowing how much memory an image occupies in practice. Then this memory interpreted by the metadata found on VideoFrame: a crop rectangle is applied (trimming off what's outside the cropped region), and then scaling is applied via displayWidth and displayHeight. Scaling is defined as being the linear transform we all know.
The coded width is the size of a horizontal pixel line, that is bigger or equal to the stride.
There is no inherent relationship because codedWidth is measured in samples and stride is measured in bytes. codedWidth is also a measurement of the whole frame while stride is per-plane.
What is always true is that planeCodedWidth * planeSampleBytes <= planeStride.
There is still some discussion happening re: display size in #94. The current proposal is to treat the display size as a display aspect ratio, and then apply the usual 'scale up in exactly one dimension to match' approach that <video> uses. To be clear, I expect the attributes on VideoFrame to be the actual display dimensions, but they may be scaled compared to the VideoDecoderConfig/VideoFrameInit.
(Perhaps in the future we would add a pixelWidth/pixelHeight for cases that prefer using a pixel aspect ratio.)
We can only work on this after #165 then. Thanks for the comment outlining my mistake. I'll work on a PR to add this precisely with all the comments taken into account.
Activity
Add refcounting semantics to VideoFrame
Add WebCodecs VideoFrame as CanvasImageSource.
chcunningham commentedon May 12, 2021
Triage note: marking 'editorial', as there is broad consensus on what these terms / concepts mean, this issue merely tracks the need to document that better.
padenot commentedon May 17, 2021
I agree, from a high-level POV, but this needs to be precisely specified, because we all know small details matter when shipping APIs on the Web.
That said, it's not hard to spec I assume, granted we use the commonly accepted definitions, an attempt below:
The coded width is the size of a horizontal pixel line, that is bigger or equal to the stride. The coded height is the size of a vertical pixel line, that is bigger or equal to the height of the picture (but is that useful ?). The coded height x the coded width allows knowing how much memory an image occupies in practice. Then this memory interpreted by the metadata found on
VideoFrame: a crop rectangle is applied (trimming off what's outside the cropped region), and then scaling is applied viadisplayWidthanddisplayHeight. Scaling is defined as being the linear transform we all know.It can well be that the type of scaling needs to be specified, maybe by using https://html.spec.whatwg.org/#resizequality ?
chcunningham commentedon May 18, 2021
Thanks @padenot. I think those definitions match my expectation. @sandersdan to double check.
One small nit
I think this should be <= to the stride.
Re: display scaling, I defer to @sandersdan.
sandersdan commentedon May 18, 2021
There is no inherent relationship because
codedWidthis measured in samples andstrideis measured in bytes.codedWidthis also a measurement of the whole frame whilestrideis per-plane.What is always true is that
planeCodedWidth * planeSampleBytes <= planeStride.There is still some discussion happening re: display size in #94. The current proposal is to treat the display size as a display aspect ratio, and then apply the usual 'scale up in exactly one dimension to match' approach that
<video>uses. To be clear, I expect the attributes onVideoFrameto be the actual display dimensions, but they may be scaled compared to theVideoDecoderConfig/VideoFrameInit.(Perhaps in the future we would add a
pixelWidth/pixelHeightfor cases that prefer using a pixel aspect ratio.)padenot commentedon May 19, 2021
We can only work on this after #165 then. Thanks for the comment outlining my mistake. I'll work on a PR to add this precisely with all the comments taken into account.