Create a dashboard for background jobs

[vdavid]

• Completes https://github.com/sourcegraph/sourcegraph/issues/44899
• 2-min demo to show the UI changes
• 5-min front-end demo for reviewers
• 10-min back-end demo for reviewers

We have 50+ goroutines running in the background in our app, and their observability is rather poor.

This PR adds an admin page that lists our known background routines and provides stats and noteworthy events around them.

It looks like this:

2-min Loom where I demo this.

Features

• As part of this PR, most of the routines we're running in worker (complete list here) now log their start, stop, and run events in Redis. This means 70% of all routines, or 87% if we don't consider resetters.
• Then there is the above UI that polls a new GraphQL endpoint every 5 seconds and fetches the latest job/routine data.
• The main use cases for admins that I expect to see from this feature:
  • Seeing what routines we have an using it as a reference (as their description is included)
  • Noticing errors early and investigating them easier
  • Noticing routines that take a long time to run, either in absolute terms or relative to their interval. (If a routine takes a similar time to run to its interval, that usually indicates danger.)
• The UI supports pausing the 5-sec polling to make it easier to investigate transient issues
• The UI also supports filtering for problematic routines to save on some scrolling from the admins
• Records in Redis are retained for 7 days. Detailed logs are kept for the last 100 recent runs.

Changes in this PR that affect other parts of the code base

• Front end: Moved formatDurationLong from client/web/src/components/ExecutionLogEntry.tsx to a new web/src/util/time.ts to reuse is functionality. Also created a unit test for it, then safely restructured it to suit my purposes.
• Front end: In ValueLegendList, formatted large numbers nicer in the tooltip. Was like 106123, now it is 106,123.
• Back end: Got rid of the dependency on workerutil in the internal/types package that caused a circular dependency with my code added. I think it's neater this way, it was quite a weird direction of dependencies.

Test plan

• Test if the page is refreshed as intended
• Test it on small screens
• Make sure tests pass
• Add automated tests as baseline QA
• Test with connection off/on
• Ask for a front-end peer review—this 5-min Loom should help do it efficiently
• Ask for a back-end peer review—this 10-min Loom should help do it efficiently

[sg-e2e-regression-test-bob]

Bundle size report 📦

Initial size	Total size	Async size	Modules
0.06% (+1.76 kb)	0.17% (+23.66 kb) 🔺	0.19% (+21.90 kb) 🔺	0.14% (+1) 🔺

Look at the Statoscope report for a full comparison between the commits 0846982 and a93117a or learn more.

Open explanation

• Initial size is the size of the initial bundle (the one that is loaded when you open the page)
• Total size is the size of the initial bundle + all the async loaded chunks
• Async size is the size of all the async loaded chunks
• Modules is the number of modules in the initial bundle

[sourcegraph-bot]

Codenotify: Notifying subscribers in CODENOTIFY files for diff a93117a...0846982.

Notify	File(s)
@efritz	cmd/worker/shared/main.go internal/goroutine/periodic.go internal/goroutine/recorder/common.go internal/goroutine/recorder/common_test.go internal/goroutine/recorder/reader.go internal/goroutine/recorder/recorder.go internal/workerutil/worker.go
@eseliger	enterprise/internal/batches/scheduler/scheduler.go
@unknwon	enterprise/cmd/frontend/internal/authz/resolvers/resolver_test.go