[fix](export) remove export task executor in TransientTaskExecutor and fix concurrency issue (#42950)(#43051)(#43109)(#43250)#43305
Merged
morningman merged 4 commits intoapache:branch-3.0from Nov 6, 2024
Conversation
Contributor
Author
|
run buildall |
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
…he#43051) ### What problem does this PR solve? Related PR: apache#42950 Problem Summary: PR apache#42950 change some logic in ExportJob, by removing the `taskIdToExecutor`, which is a thread safe ConcurrentHashMap. But there is a problem that, when cancelling a export job, it will clear the `jobExecutorList` in ExportJob, and meanwhile, this `jobExecutorList` may being traversed when creating the export job, causing concurrent modification exception. This PR fix it by locking the writeLock of ExportMgr when cancelling the export job.
### What problem does this PR solve?
Problem Summary:
```
2024-11-01 19:42:52,521 WARN (mysql-nio-pool-117|9514) [StmtExecutor.execute():616] Analyze failed. stmt[250257, 59c581a512e7468f-b1cfd7d4b63fed33]
org.apache.doris.common.NereidsException: errCode = 2, detailMessage = java.util.ConcurrentModificationException
at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:780) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:601) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:564) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:554) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:340) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:243) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:208) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:236) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:413) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = java.util.ConcurrentModificationException
... 13 more
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList.forEach(ArrayList.java:1513) ~[?:?]
at org.apache.doris.load.ExportMgr.addExportJobAndRegisterTask(ExportMgr.java:120) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.commands.ExportCommand.run(ExportCommand.java:149) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:749) ~[doris-fe.jar:1.2-SNAPSHOT]
... 12 more
```
### Check List (For Committer)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [x] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [x] Previous test can cover this change.
- [ ] No colde files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
- Release note
<!-- bugfix, feat, behavior changed need a release note -->
<!-- Add one line release note for this PR. -->
None
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
…"create export job" edit log (apache#43250) Problem Summary: Fix bug like: ``` java.lang.NullPointerException: Cannot invoke "org.apache.doris.load.ExportJob.replayExportJobState(org.apache.doris.load.ExportJobState)" because "job" is null at org.apache.doris.load.ExportMgr.replayUpdateJobState(ExportMgr.java:475) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.persist.EditLog.loadJournal(EditLog.java:390) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.replayJournal(Env.java:2965) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.master.Checkpoint.doCheckpoint(Checkpoint.java:135) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.master.Checkpoint.runAfterCatalogReady(Checkpoint.java:80) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.common.util.Daemon.run(Daemon.java:119) ~[doris-fe.jar:1.2-SNAPSHOT] ``` This is because the export job is finished before adding it. The "finish" log is saved before "creating" log, so when replaying the "finish" log, the export job can not be found. This PR change the order of "writing create export job log" and `adding export task` event, to avoid this error
morningman
changed the title
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 40357 ms |
TPC-DS: Total hot run time: 193467 ms |
morningman
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cherry pick from (#42950)(#43051)(#43109)(#43250)