[SPARK-59916][CORE] Support hold and resume in REST Submission API

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR adds hold and resume actions to the REST Submission API of the Spark Standalone Master:

• POST /v1/submissions/hold/<app-id>
• POST /v1/submissions/resume/<app-id>

They do what the Master web UI's Hold and Resume buttons (SPARK-59061) do, for scripts:

• [SPARK-59061][CORE] Hold and resume an application from the Master UI #58361

Unlike kill and status, these actions take an application ID rather than a submission ID, so they also work for applications submitted in client mode. This is the first REST Submission API action keyed by an application ID. A submission ID cannot be used here: the Master does not link a driver to the application it registers, and a client-mode application has no submission ID.

sequenceDiagram
    participant R as REST client
    participant S as StandaloneRestServer
    participant M as Master
    participant C as StandaloneAppClient (driver)

    R->>S: POST /v1/submissions/hold/<app-id>
    S->>M: RequestApplicationHold (ask)
    Note over M: check the Master's and the application's<br/>spark.ui.holdEnabled, and the reported hold support
    M->>C: SetApplicationHold (ask, not awaited, unchanged)
    M-->>S: ApplicationHoldResponse(success, message)
    S-->>R: HoldApplicationResponse (JSON)
    C-->>M: ApplicationHoldUpdated (held/draining in /json/)

Loading

• RequestApplicationHold can now be asked as well as sent. When it is asked, the Master replies with a new ApplicationHoldResponse(success, message). The Master web UI keeps sending it, so the UI behaves as before. Like kill, a Master that is not ALIVE rejects the request.
• The shared Master.handleApplicationHold now returns that response. It rejects a request when either the Master or the application disables spark.ui.holdEnabled, when the application has not reported that it can be held, or when the application is unknown or finished. Otherwise it forwards the request to the driver without waiting for the driver's answer, as before.
• Like kill, the response only tells whether the request was forwarded to the driver, not its outcome. The driver drains its executors asynchronously, and the outcome shows up in the held and draining fields of the Master's /json/ endpoint.
• No new configuration. The Master-wide spark.ui.holdEnabled gates both the web UI controls and these actions.
• On the REST side, HoldRequestServlet and StandaloneHoldRequestServlet relay the request to the Master the way the kill servlets do. A new HoldApplicationResponse message carries appId, success, and message.

$ curl -XPOST http://IP:PORT/v1/submissions/hold/app-20260930120000-0000
{
  "action" : "HoldApplicationResponse",
  "appId" : "app-20260930120000-0000",
  "message" : "Requesting application app-20260930120000-0000 to hold.",
  "serverSparkVersion" : "4.4.0",
  "success" : true
}

The docs add the two actions to the REST API table and describe them in Held Applications next to the scripted web UI requests documented by SPARK-59915:

• [SPARK-59915][DOCS] Document how to hold and resume applications via scripted HTTP requests #59176

Why are the changes needed?

Since SPARK-59353, the Master web UI's hold and resume endpoints require the per-UI CSRF token, so a script has to read the token from the page before sending the request:

• [SPARK-59353][UI] Avoid pre-fetch and crawlers triggering actions in UI #58640

The REST Submission API is the Master's API for scripts and already offers kill, killall, and clear. Holding is the graceful alternative to killing: the application gives back its executors but keeps its driver and the shuffle output already written. It belongs in the same API, without the token round trip. These actions also reach client-mode applications, which the existing REST kill cannot, because kill takes a submission ID.

Does this PR introduce any user-facing change?

Yes. The Standalone Master REST Submission API gets the new hold and resume actions. Since hold and resume are new in the unreleased Apache Spark 4.4.0, there is no behavior change compared to the released versions.

Within the unreleased branches, the Master now rejects a hold request for an application that has not reported that it can be held, instead of forwarding it to the driver. The Master web UI never offers the controls for such an application, so the UI is unaffected.

Like the other REST actions, hold and resume check no ACLs. The docs recommend protecting the REST API with JWSFilter or disabling it with spark.master.rest.enabled=false.

How was this patch tested?

Pass the CIs with the newly added test cases:

• StandaloneRestSubmitSuite: SPARK-59916: hold and resume an application
  • Both actions relay the Master's reply together with the application ID.
  • A request without an application ID is rejected with 400.
• AppClientSuite: SPARK-59916: hold and resume an application through an ask to the Master
  • A real Master rejects the request, without reaching the driver, for an application that has not reported that it can be held and for an unknown application.
  • Once the application reports that it can be held, the Master accepts both requests and both reach the driver.
• MasterSuite: SPARK-59916: Reject the hold and resume requests when the Master disables holding

The existing SPARK-59061: hold and resume an application from the Master test now reports the hold support first, since the Master requires it. The JSON in the documentation example is the actual output of HoldApplicationResponse.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

[dongjoon-hyun]

cc @peter-toth

[dongjoon-hyun]

Could you review this PR when you have some time, @HyukjinKwon ?

[dongjoon-hyun]

Thank you always, @HyukjinKwon ! 😄

[dongjoon-hyun]

Merge Summary:

• merged into master 3e49951

Posted by merge_spark_pr.py

[dongjoon-hyun]

Also, landed at branch-4.x 7eba355