Repository navigation
Fire QuotaExceeded if codec resources are exhausted/reclaimed #121
Description
The UA (and underlying platform) will often have a limit for how many codecs can be simultaneously instantiated. Particularly for hardware codecs. We could throw a QuotaExceeded exception via the error callback when this occurs.
Additionally, the UA should have the ability to reclaim codec resources if it can identify they're not being used. Maybe this could be done silently (no use -> no sideffects -> why notify). If the app picks up usage later, we could silently reallocate the resources and chug along. If the resources are no longer available, throw the same QuotaExceeded described above.
Similar consideration: UAs (at least Chromium) may suspended video decoding for <video> when tabs are put into the background (and resumed when the tab re-enters foreground). Similar techniques could be used with WebCodecs, which would again throw the QuotaExceeded error (or some other error). This is tricky though: we can't reliably know that VideoDecoder (and kin) isn't doing useful work in the background. We might consider a reserving the right to suspend backgrounded Window & DedicatedWorker VideoDecoders, but offer a different lifetime contract for SharedWorker and ServiceWoker. We haven't thought through the SharedWorker and ServiceWorker usage yet (I'll soon reduce the exposed= to just DedicateWorker in the IDL).
Activity
chcunningham commentedon Jan 8, 2021
@sandersdan
sandersdan commentedon Jan 8, 2021
I like QuotaExceeded as it's clear that you may be able to obtain resources in the future. Usually with quotas an app will want to be able to query capacity, which we don't always know, but at least in the case of background/foreground transitions there are already events apps could listen to.
Expose interfaces to DedicatedWorker, not Worker
chcunningham commentedon May 12, 2021
Triage note: marked breaking as this would be a new error. Not seriously breaking, as its an error to the callback (not thrown), but breaking nonetheless.
tguilbert-google commentedon Jun 8, 2021
We outlined two cases where we might want to fire a
QuotaExceedederror:accelerationPreference = "require"config, due to all hardware codecs being allocated.Offline, we acknowledged that the solution for recovering from both of these errors is different, and might warrant the use of different errors:
accelerationPreference = "allow"will fix the issue.What about using a
TimeoutErrorfor the reclaimed codec case? Or is it a common enough failure case for "regular" codec operations, that we'd want to leave it free for UAs to report timeouts in an active codec pipeline?tguilbert-google commentedon Jun 15, 2021
I am working on a PR. For now this is the general direction it will be going.
There are roughly 4 scenarios where we might want to throw an exception due to insufficient/reclaime resources.
Our current implementation limits the number of parallel requests internally sent to Encoders/Decoders. Outstanding requests accumulate in a higher level queue, which should be managed by different backpressure mechanisms.
Proposal: do not add any new errors/exceptions here.
hardwareAcceleration = "require", and not having enough hardware codecs availableIf a call to configure() with
hardwareAcceleration = "require"fails, the app can re-issue the same configure call withhardwareAcceleration = "allow"to solve the issue. However, raising aQuotaExceedederror specifically for this case would make it easier for websites to fingerprint the number of available hardware codecs.Note: isConfigSupported() is not a hard guarantee that a configure call with the returned config succeeds.
Proposal: Do not surface a distinct QuotaExceeded error in this case, primarily because of fingerprinting concerns. Instead, surface the same exception as regular initialization errors. Add a note in the privacy section documenting this choice.
A backgrounded website might be actively using codecs. The User Agent might want to reclaim the instantiated codecs for use by foreground websites instead.
Proposal: Add a “Resource Reclamation” portion to the spec, which outlines how user agents may be allowed to reclaim codecs. Ideally, the UA should prioritize how long a codec has been backgrounded for, how active that codec is, and what type of resources it is holding on to. Errors raised through the
error()callback can beQuotaExceeded, or a newly mintedCodecReclaimedexception.WebCodec object instances might be holding on to hardware codecs, and not be closed, but not be actively used. The UA should have the ability to reclaim those resources.
Proposal: In the “Resource Reclamation” section, document a UA’s ability to reclaim a webpage’s inactive codecs. Raise the same error type as in scenario
#3, via theerror()callback, albeit with a different message.tguilbert-google commentedon Jun 15, 2021
Question raised by @sandersdan:
Should a User Agent be allowed to reclaim an active codec on a foreground page?
chcunningham commentedon Jun 25, 2021
My leaning is 'no'. To my knowledge we don't do this in Chrome
<video>today, so I don't immediately see a use case. Open to hearing more on why / when this could be used.Some design discussion has come up in #282. I want to move that here for easier reading / visibility.
@padenot wrote:
We hear this. In addition to the creation software, we think these other use cases could be broken by the proposed background reclamation.
Still, we want to avoid the inverse problem of only being able to offer software codecs to these use cases because some trivial uses of WebCodecs (say advertising) have hold of all the hardware codecs in background tabs.
Here's strawman:
This would remedy the RTC and media playback use cases I mentioned.
It doesn't fix the creation software case. That one is harder to distinguish from the trivial uses. I think solving that is a pretty big design space (e.g. something akin to the wakelock API or the background fetch API, but for WebCodecs). I favor moving forward with the proposal above, and scoping the remedy for background creation software to a follow up extension.
@aboba @padenot WDYT?
padenot commentedon Jun 25, 2021
You're going to have creation software play music when rendering then.
chcunningham commentedon Jun 25, 2021
Ha, yes that may occur. Note that this would only save them from us reclaiming AudioDecoders. The idea with the playing audio case is that this enables the same reclamation Firefox and Chrome do today where you suspend video decoding for
<video>in the background while the audio track continues to play (user mostly doesn't notice).Does that sound reasonable?
padenot commentedon Jul 5, 2021
I'm mostly concerned about long-running background jobs here. I don't know that encoders can be safely reclaimed. I think an exception for WebRTC APIs make sense, but generally it should be possible to encode a long movie without seeing the encoding job paused halfway why the user isn't looking. The problem is that the source of this encode job might be a decoding job, in the common scenario of transcoding.
Decoders are easier: audio is rarely limited in number of instances. Video sometimes is, but it's common to release codec instances when backgrounded and this empirically works well (as implemented natively by multiple UAs).
chcunningham commentedon Jul 7, 2021
Editors call:
@padenot
chcunningham commentedon Jul 21, 2021
Editors call:
@aboba also prefers longer inactivity timeout as mentioned above. 15s could be a bit long when it comes to contended hardware resources.
@tguilbert-google let us know your thoughts on the above.
tguilbert-google commentedon Jul 23, 2021
15s of inactivity sounds good to me. The point of not reclaiming decoders if encoders are active sounds good too.
I unfortunately didn't get a chance to update this PR yet, and I will be OOO next week, but I will update it before the Aug 4th editor's call
dominic-p commentedon Apr 17, 2025
I apologize for resurrecting an ancient issue. I have just started using WebCodecs for a security camera application, and my VideoDecoders are getting reclaimed due to inactivity if the tab stays in the background for too long.
I understand from these docs that the only way to keep an active VideoDecoder in the background from being reclaimed is to have an active VideoEncoder or an active AudioEncoder on the same global object. The problem for my use case is that these are security camera feeds that don't have audio (even if they did, I would only be decoding it).
I could make a dummy AudioEncoder and run it in my worker to try to workaround this, but I thought I would bring up the issue in case we should consider additional reclamation rules to cover this use case.
padenot commentedon Apr 17, 2025
You should probably open a bug on Chromium's tracker (because this is browser engine specific), and open a new issue if you have a use-case that comes to mind to not have the decoder reclaimed.
Running some sort of image analysis algorithm in the background, and alert/log in case of event immediately comes to mind, but it's better if you state your use case in a new issue, so we can discuss it.
Thanks for the message, in any case.
dominic-p commentedon Apr 17, 2025
Thanks for the response. I really appreciate it. I opened #889 to discuss this use case.