RFC: Implement a runner for fast background jobs

[marcelklehr]

How to use GitHub

• Please use the 👍 reaction to show that you are interested into the same feature.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Is your feature request related to a problem? Please describe.
Currently, some apps require the setup of a fast background job worker using occ background-job:worker. Some apps require more than one worker of the same job, even. Setting these up e.g. with systemd is cumbersome and repetitive. Additionally for nextcloud AIO, it would be beneficial to have a generic way to run these without hard coding jobs for all possible apps in the AIO code.

Describe the solution you'd like
We could have a new php script next to cron.php and occ or just another occ command that

• figures out which jobs to run
• runs the necessary background workers as sub processes
• keeps track of them and restarts them when they fail or exit
• Passes the stdout of the workers through to its own stdout

Background services could be set via a a php file, or a json file. A service could be either a background-job:worker "worker" or a "dedicated-cron" cron.php runner, with a set list of jobs. This would cater to both QueuedJobs and TimedJobs (the latter cannot be run by the bg-job worker).

{
  "services": {
    "TaskProcessing 1": {
      "type": "worker",
      "jobs": ["OC\\TaskProcessing\\SynchronousBackgroundJob"],
    },
   "TaskProcessing 2": {
      "type": "worker",
      "jobs": ["OC\\TaskProcessing\\SynchronousBackgroundJob"],
    },
    "WebhookListeners": {
      "type": "worker",
      "jobs": ["OCA\\WebhookListeners\\BackgroundJobs\\WebhookCall"],
    },
    "ContextChat": {
      "type": "dedicated-cron"
      "jobs": ["OCA\\ContextChat\\BackgroundJobs\\CrawlJob", "OCA\\ContextChat\\BackgroundJobs\\IndexerJob"],
    }
  }
}

Services would be started only if the respective job classes are loadable in the main php process.

Describe alternatives you've considered
We could also leave out the dedicated cron feature, which would make the whole thing less powerful but still useful.

Additional context

[julien-nc]

So this would save some effort for admins and make it more "declarative" for them to have runners. I like the idea!
Also, if one prefers to keep control of what runs, they can still have systemd services or whatever they want. This new php script would be optional to use.

An alternative to the json config file for the services would be to have that information in the job itself, like a new interface that job classes can extend, with a new method to declare how many workers this job class needs.

But your approach with the json config file is maybe simpler and has the benefit of being retro-compatible.

[kyteinsky]

Nice! It won't feel like a workaround if this gets implemented and easy to activate.
Supervisord (https://supervisord.org/) could also be used as a process management tool in absence of a more suitable orchestrator.

[oleksandr-nc]

Good day to all. The topic is complex and very important.

I don’t see an ideal way from my side, but I couldn’t find an alternative better than the initial idea proposed by Marcel.

What I Like About the Proposed Idea

• Single entry point
  Running everything via occ background-job hides job names and desired concurrency from the admin.
• Survivable config
  Config lives in a JSON or PHP file inside the data directory—survives upgrades and works in AIO.
• Health metrics
  Ability to expose health and status metrics over HTTP.

---

Drawbacks

• Re-inventing the wheel
  We re-implement what mature process managers already handle (restart policies, logging, resource limits, cgroup integration).
• Single point of failure
  The supervisor itself is one more long-running PHP process; we should do some watchdog to catch crashes/hangs.
• Long-lived PHP issues
  Potential for slow GC leaks in long-running CLI processes—needs periodic self-restart logic.

---

Alternatives Considered

1. SupervisorD
   • Already used in AIO.
   • Pros: widely available on most OSes.
   • Cons: extra service to install/(auto)configure.
2. RabbitMQ
   • Quickly discarded: adds heavy infrastructure and will be difficult to do this universally and simply for all types of installations.

Question for @szaimen:
Is it feasible to leverage the existing SupervisorD in AIO instead of rolling our own PHP supervisor?
We’d need it to start persistent workers and monitor scale-up/scale-down as config changes.

---

Configuration File

We could either:

1. Ship a JSON/YAML file under config/ (or data/), or
2. Add a PHP array in config.php for simplicity.

A standalone JSON is more future-proof if we later switch to SupervisorD, for example.

Example JSON snippet

{
  "services": {
    "ai_task_processor_1": {
      "type": "worker",
      "job": "OC\\TaskProcessing\\SynchronousBackgroundJob",
      "count": 2,
      "restartTimeout": 60
    }
  }
}

• count
  Number of identical worker processes to run
• restartTimeout
  Seconds after which each worker is restarted (maps to -t)

---

App-Provided Background Services

Allow apps to ship their own snippet at appinfo/background-services.json.
At runtime, occ background-service:run would:

1. Load all app-provided JSON snippets

2. Merge them with the admin’s config/background-services.json

   • Admin can override or disable any app-defined service

Alternative: store these definitions in info.xml instead of a separate JSON file.

---

OCC Commands

We should extend occ with sub-commands to manage services:

occ background-service:list
occ background-service:status
occ background-service:start <service_name>
occ background-service:stop  <service_name>
occ background-service:reload

---

Overall, I think that we can move forward with what Marcel proposed; it is a very important functionality.

[szaimen]

Question for @szaimen:
We’d need it to start persistent workers and monitor scale-up/scale-down as config changes.
Is it feasible to leverage the existing SupervisorD in AIO instead of rolling our own PHP supervisor?

Yes, that should be possible 👍

Important for me is only that it is easy to configure for example by running one additional cron file for fast background jobs via supervivsord which automatically picks up all fast jobs from installed apps without me having to configure this for each app in AIO manually...

[icewind1991]

After a discussion around #52055, we've started collecting some ideas for background job improvements in #52877

It's a separate ticket from this because the scope is larger (but includes the things talked about here)

[alx-tuilmenau]

Be aware that there should be an alternative, in most shared hosting environments you don't have console access or can run jobs in the background.

[sptcguy]

I currently distribute background jobs by setting up non-request taking instances and phase shifting cron calls on each. This is hacky though and requires custom monitoring to catch worker failures and avoid delays. It also doesn't scale well. Having the ability to distribute tasks properly would be an absolute game changer. Is there any room in this scope for distribution? If going with Supervisord the XML-RPC API could be used to centralize control.

In other projects, the way I've tackled task running purely with PHP was to have pool of workers where one was the primary and doled out jobs/tasks. The secondaries would form a quorum and elect a new primary if the existing one failed. When a worker completed an assigned task it would add itself back to the pool.