[SPARK-59915][DOCS] Document how to hold and resume applications via scripted HTTP requests - #59176
Closed
dongjoon-hyun wants to merge 1 commit into
Closed
dongjoon-hyun wants to merge 1 commit into
dongjoon-hyun wants to merge 1 commit into
Conversation
…scripted HTTP requests
Member
Author
|
cc @peter-toth |
Member
Author
|
Could you review this PR, @holdenk ? |
Contributor
|
Oooh this is a great addition, yes I'll take a look. |
HyukjinKwon
approved these changes
Oct 1, 2026
Member
Author
|
Thank you, @HyukjinKwon and @holdenk . |
dongjoon-hyun
added a commit
that referenced
this pull request
Oct 1, 2026
…cripted HTTP requests ### What changes were proposed in this pull request? This PR documents how to hold and resume an application through scripted HTTP requests (e.g. `curl`). The endpoints are the same ones the **Hold**/**Resume** buttons already submit to: - `docs/web-ui.md`: the driver web UI endpoints `/jobs/hold/` and `/jobs/resume/`, plus the existing `holdstatus` REST API for checking the result. - `docs/spark-standalone.md` (**Held Applications**): the Master web UI endpoints `/app/hold/` and `/app/resume/`, and the `held`/`draining` fields of `/json/` for checking the result, since the Master forwards the request to the driver asynchronously. For each endpoint, the docs describe: - the request must be a POST (on the driver UI, GET is also accepted when `spark.ui.actionsViaGetEnabled` is on); - it must carry the per-UI `csrfToken`, which is the value of the hidden `csrfToken` field on the corresponding page and stays fixed for the lifetime of that UI; - the trailing slash is required: without it, the request is only redirected and does not take effect; - a request without a valid token is rejected with 403; - when the UI requires authentication, reading the token and sending the request both need the same credentials. The examples use placeholders (`<csrf-token>`, `<app-id>`) in the same style as the existing REST API examples. ```bash curl -X POST -d "csrfToken=<csrf-token>" http://<driver-host>:4040/jobs/hold/ curl -X POST -d "id=<app-id>" -d "csrfToken=<csrf-token>" http://<master-host>:8080/app/hold/ ``` ### Why are the changes needed? Since SPARK-59353 (#58640), state-changing UI endpoints accept only POST requests that carry a per-UI CSRF token. The token check cannot be disabled. So the plain calls operators used before, such as `curl -X POST http://<driver-host>:4040/jobs/hold`, now fail with 403, and nothing in the hold/resume docs explains how to call these endpoints from a script. `docs/configuration.md` only says "Scripted clients can read the token from the jobs page". - #58640 This PR fills that gap without weakening the CSRF protection: no token-less endpoint is added, and the endpoints behave exactly as they do today. ### Does this PR introduce _any_ user-facing change? No. This is a documentation-only change. ### How was this patch tested? I manually checked the documented requests on a local Standalone cluster: one Master, one Worker with `spark.shuffle.service.enabled=true`, and a `spark-shell` application with `spark.decommission.enabled=true`. | Request | Driver UI | Master UI | |---|---|---| | POST with a valid `csrfToken` | 302, then `holdstatus` shows `held: true` | 302, then `/json/` shows `held` toggled by resume/hold | | POST without a token | 403 | 403 | | POST with a wrong token | 403 | - | | POST without the trailing slash | 301, no effect | 301, no effect | I also built the docs locally and checked that the new sections render correctly. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 Closes #59176 from dongjoon-hyun/SPARK-59915. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit ac2831c) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Member
Author
dongjoon-hyun
added a commit
that referenced
this pull request
Oct 1, 2026
### What changes were proposed in this pull request? This PR adds `hold` and `resume` actions to the REST Submission API of the Spark Standalone Master: - `POST /v1/submissions/hold/<app-id>` - `POST /v1/submissions/resume/<app-id>` They do what the Master web UI's **Hold** and **Resume** buttons (SPARK-59061) do, for scripts: - #58361 Unlike `kill` and `status`, these actions take an application ID rather than a submission ID, so they also work for applications submitted in client mode. This is the first REST Submission API action keyed by an application ID. A submission ID cannot be used here: the Master does not link a driver to the application it registers, and a client-mode application has no submission ID. ```mermaid sequenceDiagram participant R as REST client participant S as StandaloneRestServer participant M as Master participant C as StandaloneAppClient (driver) R->>S: POST /v1/submissions/hold/<app-id> S->>M: RequestApplicationHold (ask) Note over M: check the Master's and the application's<br/>spark.ui.holdEnabled, and the reported hold support M->>C: SetApplicationHold (ask, not awaited, unchanged) M-->>S: ApplicationHoldResponse(success, message) S-->>R: HoldApplicationResponse (JSON) C-->>M: ApplicationHoldUpdated (held/draining in /json/) ``` - `RequestApplicationHold` can now be asked as well as sent. When it is asked, the Master replies with a new `ApplicationHoldResponse(success, message)`. The Master web UI keeps `send`ing it, so the UI behaves as before. Like `kill`, a Master that is not `ALIVE` rejects the request. - The shared `Master.handleApplicationHold` now returns that response. It rejects a request when either the Master or the application disables `spark.ui.holdEnabled`, when the application has not reported that it can be held, or when the application is unknown or finished. Otherwise it forwards the request to the driver without waiting for the driver's answer, as before. - Like `kill`, the response only tells whether the request was forwarded to the driver, not its outcome. The driver drains its executors asynchronously, and the outcome shows up in the `held` and `draining` fields of the Master's `/json/` endpoint. - No new configuration. The Master-wide `spark.ui.holdEnabled` gates both the web UI controls and these actions. - On the REST side, `HoldRequestServlet` and `StandaloneHoldRequestServlet` relay the request to the Master the way the `kill` servlets do. A new `HoldApplicationResponse` message carries `appId`, `success`, and `message`. ```bash $ curl -XPOST http://IP:PORT/v1/submissions/hold/app-20260930120000-0000 { "action" : "HoldApplicationResponse", "appId" : "app-20260930120000-0000", "message" : "Requesting application app-20260930120000-0000 to hold.", "serverSparkVersion" : "4.4.0", "success" : true } ``` The docs add the two actions to the REST API table and describe them in **Held Applications** next to the scripted web UI requests documented by SPARK-59915: - #59176 ### Why are the changes needed? Since SPARK-59353, the Master web UI's hold and resume endpoints require the per-UI CSRF token, so a script has to read the token from the page before sending the request: - #58640 The REST Submission API is the Master's API for scripts and already offers `kill`, `killall`, and `clear`. Holding is the graceful alternative to killing: the application gives back its executors but keeps its driver and the shuffle output already written. It belongs in the same API, without the token round trip. These actions also reach client-mode applications, which the existing REST `kill` cannot, because `kill` takes a submission ID. ### Does this PR introduce _any_ user-facing change? Yes. The Standalone Master REST Submission API gets the new `hold` and `resume` actions. Since hold and resume are new in the unreleased Apache Spark 4.4.0, there is no behavior change compared to the released versions. Within the unreleased branches, the Master now rejects a hold request for an application that has not reported that it can be held, instead of forwarding it to the driver. The Master web UI never offers the controls for such an application, so the UI is unaffected. Like the other REST actions, `hold` and `resume` check no ACLs. The docs recommend protecting the REST API with `JWSFilter` or disabling it with `spark.master.rest.enabled=false`. ### How was this patch tested? Pass the CIs with the newly added test cases: - `StandaloneRestSubmitSuite`: `SPARK-59916: hold and resume an application` - Both actions relay the Master's reply together with the application ID. - A request without an application ID is rejected with 400. - `AppClientSuite`: `SPARK-59916: hold and resume an application through an ask to the Master` - A real Master rejects the request, without reaching the driver, for an application that has not reported that it can be held and for an unknown application. - Once the application reports that it can be held, the Master accepts both requests and both reach the driver. - `MasterSuite`: `SPARK-59916: Reject the hold and resume requests when the Master disables holding` The existing `SPARK-59061: hold and resume an application from the Master` test now reports the hold support first, since the Master requires it. The JSON in the documentation example is the actual output of `HoldApplicationResponse`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 Closes #59178 from dongjoon-hyun/SPARK-59916. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
dongjoon-hyun
added a commit
that referenced
this pull request
Oct 1, 2026
### What changes were proposed in this pull request? This PR adds `hold` and `resume` actions to the REST Submission API of the Spark Standalone Master: - `POST /v1/submissions/hold/<app-id>` - `POST /v1/submissions/resume/<app-id>` They do what the Master web UI's **Hold** and **Resume** buttons (SPARK-59061) do, for scripts: - #58361 Unlike `kill` and `status`, these actions take an application ID rather than a submission ID, so they also work for applications submitted in client mode. This is the first REST Submission API action keyed by an application ID. A submission ID cannot be used here: the Master does not link a driver to the application it registers, and a client-mode application has no submission ID. ```mermaid sequenceDiagram participant R as REST client participant S as StandaloneRestServer participant M as Master participant C as StandaloneAppClient (driver) R->>S: POST /v1/submissions/hold/<app-id> S->>M: RequestApplicationHold (ask) Note over M: check the Master's and the application's<br/>spark.ui.holdEnabled, and the reported hold support M->>C: SetApplicationHold (ask, not awaited, unchanged) M-->>S: ApplicationHoldResponse(success, message) S-->>R: HoldApplicationResponse (JSON) C-->>M: ApplicationHoldUpdated (held/draining in /json/) ``` - `RequestApplicationHold` can now be asked as well as sent. When it is asked, the Master replies with a new `ApplicationHoldResponse(success, message)`. The Master web UI keeps `send`ing it, so the UI behaves as before. Like `kill`, a Master that is not `ALIVE` rejects the request. - The shared `Master.handleApplicationHold` now returns that response. It rejects a request when either the Master or the application disables `spark.ui.holdEnabled`, when the application has not reported that it can be held, or when the application is unknown or finished. Otherwise it forwards the request to the driver without waiting for the driver's answer, as before. - Like `kill`, the response only tells whether the request was forwarded to the driver, not its outcome. The driver drains its executors asynchronously, and the outcome shows up in the `held` and `draining` fields of the Master's `/json/` endpoint. - No new configuration. The Master-wide `spark.ui.holdEnabled` gates both the web UI controls and these actions. - On the REST side, `HoldRequestServlet` and `StandaloneHoldRequestServlet` relay the request to the Master the way the `kill` servlets do. A new `HoldApplicationResponse` message carries `appId`, `success`, and `message`. ```bash $ curl -XPOST http://IP:PORT/v1/submissions/hold/app-20260930120000-0000 { "action" : "HoldApplicationResponse", "appId" : "app-20260930120000-0000", "message" : "Requesting application app-20260930120000-0000 to hold.", "serverSparkVersion" : "4.4.0", "success" : true } ``` The docs add the two actions to the REST API table and describe them in **Held Applications** next to the scripted web UI requests documented by SPARK-59915: - #59176 ### Why are the changes needed? Since SPARK-59353, the Master web UI's hold and resume endpoints require the per-UI CSRF token, so a script has to read the token from the page before sending the request: - #58640 The REST Submission API is the Master's API for scripts and already offers `kill`, `killall`, and `clear`. Holding is the graceful alternative to killing: the application gives back its executors but keeps its driver and the shuffle output already written. It belongs in the same API, without the token round trip. These actions also reach client-mode applications, which the existing REST `kill` cannot, because `kill` takes a submission ID. ### Does this PR introduce _any_ user-facing change? Yes. The Standalone Master REST Submission API gets the new `hold` and `resume` actions. Since hold and resume are new in the unreleased Apache Spark 4.4.0, there is no behavior change compared to the released versions. Within the unreleased branches, the Master now rejects a hold request for an application that has not reported that it can be held, instead of forwarding it to the driver. The Master web UI never offers the controls for such an application, so the UI is unaffected. Like the other REST actions, `hold` and `resume` check no ACLs. The docs recommend protecting the REST API with `JWSFilter` or disabling it with `spark.master.rest.enabled=false`. ### How was this patch tested? Pass the CIs with the newly added test cases: - `StandaloneRestSubmitSuite`: `SPARK-59916: hold and resume an application` - Both actions relay the Master's reply together with the application ID. - A request without an application ID is rejected with 400. - `AppClientSuite`: `SPARK-59916: hold and resume an application through an ask to the Master` - A real Master rejects the request, without reaching the driver, for an application that has not reported that it can be held and for an unknown application. - Once the application reports that it can be held, the Master accepts both requests and both reach the driver. - `MasterSuite`: `SPARK-59916: Reject the hold and resume requests when the Master disables holding` The existing `SPARK-59061: hold and resume an application from the Master` test now reports the hold support first, since the Master requires it. The JSON in the documentation example is the actual output of `HoldApplicationResponse`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 Closes #59178 from dongjoon-hyun/SPARK-59916. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 3e49951) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR documents how to hold and resume an application through scripted HTTP requests (e.g.
curl). The endpoints are the same ones the Hold/Resume buttons already submit to:docs/web-ui.md: the driver web UI endpoints/jobs/hold/and/jobs/resume/, plus the existingholdstatusREST API for checking the result.docs/spark-standalone.md(Held Applications): the Master web UI endpoints/app/hold/and/app/resume/, and theheld/drainingfields of/json/for checking the result, since the Master forwards the request to the driver asynchronously.For each endpoint, the docs describe:
spark.ui.actionsViaGetEnabledis on);csrfToken, which is the value of the hiddencsrfTokenfield on the corresponding page and stays fixed for the lifetime of that UI;The examples use placeholders (
<csrf-token>,<app-id>) in the same style as the existing REST API examples.Why are the changes needed?
Since SPARK-59353 (#58640), state-changing UI endpoints accept only POST requests that carry a per-UI CSRF token. The token check cannot be disabled. So the plain calls operators used before, such as
curl -X POST http://<driver-host>:4040/jobs/hold, now fail with 403, and nothing in the hold/resume docs explains how to call these endpoints from a script.docs/configuration.mdonly says "Scripted clients can read the token from the jobs page".This PR fills that gap without weakening the CSRF protection: no token-less endpoint is added, and the endpoints behave exactly as they do today.
Does this PR introduce any user-facing change?
No. This is a documentation-only change.
How was this patch tested?
I manually checked the documented requests on a local Standalone cluster: one Master, one Worker with
spark.shuffle.service.enabled=true, and aspark-shellapplication withspark.decommission.enabled=true.csrfTokenholdstatusshowsheld: true/json/showsheldtoggled by resume/holdI also built the docs locally and checked that the new sections render correctly.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5