Make run-at document-start more reliable

[Yonezpt]

Currently when using document-start with Greasemonkey my script runs when the HTML element is created, before the Head element is loaded with new elements. However, the same isn't happening with Tampermonkey, resulting in an unreliable behavior which I was compelled to try and fix by forcing the page to reload in an attempt to get the script running when it should.

With this simple example script for YouTube I get mixed results:

// ==UserScript==
// @version     1
// @name        ReadyState check
// @namespace   Test script
// @match       https://www.youtube.com/*
// @run-at      document-start
// @grant       none
// @noframes
// ==/UserScript==
(function () {
    'use strict';
    window.stop();
    console.log(document.readyState, document.body || document.querySelector('[name="html5player/html5player"]'));
}());

If I open a YouTube page for the first time it usually runs the script right at start when I need it, but if I open a video in a new tab (midle-click on link, for example) then sometimes this isn't true. On some cases the entire head is fully loaded and the body already started loading before the script could run, which can be confirmed by inspecting the page code after the script makes it stop. Also I witness often that the head already contains scripts that I need to load after my script runs, not before, which can be seen in the console.log.

The worst case is when I open a new window via window.open() which the script fails to run when it should every single time for up to 10 or 20 times in a row until it is finally able to run when it should. With the above script this isn't happening because I am not making it reload until it queues correctly, but you can see that the pages will be often complete before the script even has a chance to run.

If I pack the userscript in Chrome into a testing standalone addon, which runs-at document-start there are 0 issues like this, the script runs as soon as the page exists flawlessly, so Tampermonkey should be able too.

I can't tell if this is being caused because the Tampermonkey extension is not running as soon as it should whenever the pages open or if the userscript is not being injected as soon as it should due to some slowliness caused by Tampermonkey, but this behavior is very unreliable and is forcing me to stop considering supporting my scripts for Tampermonkey because I don't know what else I can do if the extension is not reliable enough to run the code when it should or reasonably close to it.

I have no issues with Greasemonkey nor when I convert the userscript into a Chrome extension, it's only with Tampermonkey that I am having these complications which are making the script unusuable.

Can you update the extension to be more reliable in this aspect? Preferably it should follow Greasemonkey's setting, which I quote:

The script will run before any document begins loading, thus before any scripts run or images load.

Instead of Tampermonkey's current:

The script will be injected as fast as possible.

Also I am using Tampermonkey 3.10.84 and running just the one script, no other.

[derjanb]

Can you update the extension to be more reliable in this aspect?

Unfortunately not. The (Tampermonkey) content script running at the page needs to get the scripts that should executed from the background page. This is done with some asynchronous communication that doesn't take long, but some pages are faster.

The solution for this is Chromium issue 63979 [1]. Once this is fixed the background page can actively inject the runtime env + all scripts into the matching frames before the page script runs.

[1] https://code.google.com/p/chromium/issues/detail?id=63979

[Yonezpt]

So it is a browser limitation, not extension limitation, good to know.

For the moment I managed to make the script runtime more reliable from my userscript (or at least less dependent on Tampermonkey's timing), was working on trying to find a better solution since I posted here, so for now it seems like I will be able to continue extending support in my userscript for Tampermonkey. I hope they fix that issue because this will allow me to reduce code size and overhead.

Thanks for the quick response.

@derjanb

The solution for this is Chromium issue 63979 [1]. Once this is fixed the background page can actively inject the runtime env + all scripts into the matching frames before the page script runs.

looks like the issue was fixed

[derjanb]

Unfortunately there is a new showstopper... https://bugs.chromium.org/p/chromium/issues/detail?id=598896

[tophf]

Currently Tampermonkey adds a lot of overhead in its content script as seen in DevTools Timeline profiler, and because of that document-start userscripts are postponed until the initial spike of webpage parsing is complete, which can easily take 1 second or more.

1. ~50ms: content.js is loading/compiling and requesting page.js to load - the message to ask background page for the scripts might have been sent at this stage if chrome.runtime.sendMessage was used directly in top level code.
2. ~30ms: page.js is loading/compiling
3. ~1sec: the webpage is being actively parsed by browser, no extensions run anymore
4. Tampermonkey code finally gets executed and sends its message to the background page
5. The userscripts are loaded

I don't know the architecture of TM but when I see fancy beautiful OOP wrapper, I know immediately with 100% certainty that this code can be "flattened" and become faster. Sometimes for just a few milliseconds, sometimes several/many times faster.

---

For example, Stylish-chrome also injects user-defined styles at document-start, and does it reliably in time, before the page is shown, so no FOUC occurs. It's because 1) Stylish immediately sends the message to its background page, 2) the styles are cached in memory to avoid DB access overhead, 3) so the response is sent back immediately, not in a new message that may be postponed by browser.

[derjanb]

Currently Tampermonkey adds a lot of overhead

Styles don't need to be separated at the page context and they need no runtime environment with powerful GM_* functions to work, they can simply be injected into the page.

For example, Stylish-chrome also injects user-defined styles at document-start, and does it reliably in time, before the page is shown, so no FOUC occurs.
It's because 1) Stylish immediately sends the message to its background page, 2) the styles are cached in memory to avoid DB access overhead, 3) so the response is sent back immediately, not in a new message that may be postponed by browser.

TM also caches the scripts and sends them back immediately. However, since extensions run in their own process and the communication is asynchronous there is no guarantee that the page didn't finished loading. This means Stylish also has that race condition. If I'm wrong, then please tell me how synchronous message passing works. :)

[tophf]

how synchronous message passing works.

Of course, it's not synchronous. I meant Stylish gets the data sooner, the response returns in 20-30ms:

But anyway, I was sloppy in interpreting the profiler graph. TM also sends the request in the first milliseconds (I was wrong about that), but it doesn't help much, because TM's background page takes 150ms to send the response so browser has time to finish parsing DOM:

So even though I was mistaken in diagnosing the problem, maybe you can make the background page respond faster (100ms is hell of a lot of time) as it seems crucial to the browser.

[tophf]

Maybe you can prepare the list of userscripts to be injected for a given URL in webNavigation.onBeforeNavigate listener and keep this list ready to be sent when content script asks for it. Of course, since onBeforeNavigate may be canceled, the list should autodestroy in like 1 second.

[tophf]

Also, you can try repeatedly sending (using setInterval) the list of userscripts in webNavigation.onCommitted - it'll fail the first time since no content script is running at this point but one of the subsequent attempts should succeed, supposedly, and the content script will receive the data much sooner.

[derjanb]

Maybe you can prepare the list of userscripts to be injected for a given URL in webNavigation.onBeforeNavigate

TM uses a blocking webRequest callback to do this.

Also, you can try repeatedly sending (using setInterval) the list of userscripts in webNavigation.onCommitted

I just tried this and it's working good so far. I don't know how large Stylish's user styles usually are, but user scripts and their resources might get really big. This solution would cause a lot of overhead, because all data is sent (and stringified) at least two times or more. So I'm not sure if this is desired behavior. Maybe I'll put this behind a config option.