feat(ci): Adds playwright tests for sveltekit to bazel

[jamesmcnamara]

Closes https://github.com/sourcegraph/sourcegraph/issues/60881

Attempting to run playwright tests out of bazel. This changes how the app is served in the tests, specifically playwright will intercept all network calls to the local server and serve the static assets directly or serve root index.html file if nothing is matched.

Just running pnpm test with this change, about 68% of tests pass (43 out of 67), indicating that about a 1/3 of tests likely have some network dependency.

Enabling this in bazel we get just slightly more tests failing (4 more to be exact), which means there is some difference between the two. For instance, this test fails in bazel but not through PNPM. Adding some printlns I can see that this path is getting hit in bazel but not in pnpm. Would love an extra set of eyes on this.

TODO:

• Fix tests running locally
• Fix tests failing only in bazel
• Fetch browsers properly in bazel

Test plan

Should be run in CI and go green by the end!

[jhchabran]

Nice to finally see this going green 🚀

I tried running it locally, but it trips on the browser location.
After digging a bit, it seems that:

• sg setup checks for Chromium being installed, which passes for me (I'm running the sg setup from the branch).
• sg bazel test //client/web-sveltekit:e2e_test fails because it cannot find the browser.

After closer inspection, it's because npx playwright install gets me chromium-1117 ffmpeg-1009 firefox-1449 webkit-2003 but the Bazel targets looks for chromium-1105.

Is that a "de-sync" in between the Bazel side and the local env? Wouldn't it be safer to provide the classic select(...) approach and always install chromium through Bazel?

[bahrmichael]

Yay green! I'm excited that we're at this point now :)

To give some more input for the browser version mismatch, here's what I've seen. I ran npx playwright uninstall --all beforehand. I didn't run sg setup because the test command also installed the browser.

Running with bazel test //client/web-sveltekit:e2e_test it expects chromium-1105:

Error: browserType.launch: Executable doesn't exist at /Users/michael/Library/Caches/ms-playwright/chromium-1105/chrome-mac/Chromium.app/Contents/MacOS/Chromium

I saw that it downloaded chromium-1117 via

    http_archive(
        name = "chromium-linux-x86_64",
        integrity = "sha256-T7teJtSwhf7LIpQMEp4zp3Ey3T/p4Y7dQI/7VGVHdkE=",
        url = "https://playwright.azureedge.net/builds/chromium/1117/chromium-linux.zip",
        build_file_content = CHROMIUM_BUILDFILE.format("chrome-linux/chrome"),
    )

@jhchabran when would you run sg bazel vs just bazel?

When I run pnpm test in the web-sveltekit folder, it appears to target chromium-1091. Even after I rerun pnpm i in that directory to make sure the dependencies are up to date.

/Users/michael/Library/Caches/ms-playwright/chromium-1091/chrome-mac/Chromium.app/Contents/MacOS/Chromium

According to the playwright docs, each version of playwright is tied to a certain browser version. Playwright's stance is that you shouldn't influence this and that they won't allow you to make this configurable, but there's an escape hatch via the launch options to specify a binary (we're using that in this PR).

With every release, Playwright updates the versions of the browsers it supports, so that the latest Playwright would support the latest browsers at any moment. It means that every time you update Playwright, you might need to re-run the install CLI command.

[jhchabran]

@jhchabran when would you run sg bazel vs just bazel?

@bahrmichael All the time. You get the remote cache set up for free with sg bazel. Benefits are not as clear cut as you'd expect, as it's not yet something that everyone uses, so it's not warm very often for all the tasks you'd want. GCS bucket is located in Germany so you should be pretty close to it 😊

As for the issues you mentioned, let's see when James come online, and I hope we can come up with a way to fix that, there are other solutions, but it ain't going to be pretty if we have to parse the version and dynamically download it on the fly.

[bahrmichael]

Noticed another thing. After running bazel clean --async and then running bazel test ... again, playwright complains about the browser missing from /Users/michael/Library/Caches/ms-playwright/chromium-1105/chrome-mac/Chromium.app/Contents/MacOS/Chromium. This is probably a false alert, since ls /Users/michael/Library/Caches/ms-playwright yields nothing.

[jhchabran]

(woops accidental "update branch", thought this was another PR of mine)

[jamesmcnamara]

@jhchabran @bahrmichael The reason we can't just install chromium for Mac is because Bazel cannot handle runfiles with spaces (more details in this merged PR). Noah had some suggestions in this slack thread but I haven't tried them.

As for why it's installing the wrong version of playwright it's probably because I used npx instead of just creating an npm-script within the sveltekit project so it would be tied to the version. I'll do that.

[jhchabran]

@jhchabran @bahrmichael The reason we can't just install chromium for Mac is because Bazel cannot handle runfiles with spaces (more details in this merged PR). Noah had some suggestions in this slack thread but I haven't tried them.

Ah right, it's fine then, if sg setup takes care of it we're covered, not sure it's worth going into that rabbithole.

As for why it's installing the wrong version of playwright it's probably because I used npx instead of just creating an npm-script within the sveltekit project so it would be tied to the version. I'll do that.

Ack, np!

[bahrmichael]

@jamesmcnamara I updated this PR's description to auto-close my tracking ticket when it's merged. Just FYI.