Test ReplicaTest.testBrokenSnapshotRecovery(File) is unstable

[wuchong]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

main (development)

Please describe the bug 🐞

https://github.com/apache/fluss/actions/runs/18854343205/job/53798365412

2025-10-27T20:09:00.9728658Z [ERROR] Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 19.71 s <<< FAILURE! - in org.apache.fluss.server.replica.ReplicaTest
2025-10-27T20:09:00.9730071Z [ERROR] org.apache.fluss.server.replica.ReplicaTest.testBrokenSnapshotRecovery(File)  Time elapsed: 2.211 s  <<< ERROR!
2025-10-27T20:09:00.9731152Z java.util.NoSuchElementException
2025-10-27T20:09:00.9732362Z 	at java.util.AbstractQueue.remove(AbstractQueue.java:117)
2025-10-27T20:09:00.9733404Z 	at org.apache.fluss.testutils.common.ManuallyTriggeredScheduledExecutorService.triggerNonPeriodicScheduledTask(ManuallyTriggeredScheduledExecutorService.java:252)
2025-10-27T20:09:00.9734551Z 	at org.apache.fluss.server.replica.ReplicaTest.testBrokenSnapshotRecovery(ReplicaTest.java:528)
2025-10-27T20:09:00.9735269Z 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2025-10-27T20:09:00.9735814Z 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2025-10-27T20:09:00.9736476Z 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2025-10-27T20:09:00.9737083Z 	at java.lang.reflect.Method.invoke(Method.java:498)
2025-10-27T20:09:00.9737615Z 	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
2025-10-27T20:09:00.9738606Z 	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
2025-10-27T20:09:00.9739471Z 	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
2025-10-27T20:09:00.9740331Z 	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
2025-10-27T20:09:00.9741123Z 	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
2025-10-27T20:09:00.9741925Z 	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
2025-10-27T20:09:00.9742924Z 	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
2025-10-27T20:09:00.9744029Z 	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
2025-10-27T20:09:00.9745161Z 	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
2025-10-27T20:09:00.9746302Z 	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
2025-10-27T20:09:00.9747325Z 	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
2025-10-27T20:09:00.9748241Z 	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
2025-10-27T20:09:00.9749137Z 	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
2025-10-27T20:09:00.9750036Z 	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
2025-10-27T20:09:00.9750984Z 	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
2025-10-27T20:09:00.9751848Z 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
2025-10-27T20:09:00.9752730Z 	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
2025-10-27T20:09:00.9753588Z 	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
2025-10-27T20:09:00.9754445Z 	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
2025-10-27T20:09:00.9755433Z 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
2025-10-27T20:09:00.9756308Z 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
2025-10-27T20:09:00.9757257Z 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
2025-10-27T20:09:00.9757995Z 	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
2025-10-27T20:09:00.9758699Z 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
2025-10-27T20:09:00.9759520Z 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
2025-10-27T20:09:00.9760351Z 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
2025-10-27T20:09:00.9761118Z 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
2025-10-27T20:09:00.9762192Z 	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
2025-10-27T20:09:00.9763750Z 	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.executeNonConcurrentTasks(ForkJoinPoolHierarchicalTestExecutorService.java:155)
2025-10-27T20:09:00.9765442Z 	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:135)
2025-10-27T20:09:00.9766625Z 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
2025-10-27T20:09:00.9767572Z 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
2025-10-27T20:09:00.9768400Z 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
2025-10-27T20:09:00.9769185Z 	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
2025-10-27T20:09:00.9769891Z 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
2025-10-27T20:09:00.9770732Z 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
2025-10-27T20:09:00.9771572Z 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
2025-10-27T20:09:00.9772420Z 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
2025-10-27T20:09:00.9773486Z 	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
2025-10-27T20:09:00.9775000Z 	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129)
2025-10-27T20:09:00.9776182Z 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
2025-10-27T20:09:00.9777085Z 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
2025-10-27T20:09:00.9777963Z 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
2025-10-27T20:09:00.9778697Z 	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
2025-10-27T20:09:00.9779390Z 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
2025-10-27T20:09:00.9780212Z 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
2025-10-27T20:09:00.9781032Z 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
2025-10-27T20:09:00.9781796Z 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
2025-10-27T20:09:00.9782854Z 	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
2025-10-27T20:09:00.9783841Z 	at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
2025-10-27T20:09:00.9784334Z 	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
2025-10-27T20:09:00.9784924Z 	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
2025-10-27T20:09:00.9785474Z 	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
2025-10-27T20:09:00.9786020Z 	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
2025-10-27T20:09:00.9786358Z 
2025-10-27T20:09:00.9849264Z [INFO] Running org.apache.fluss.server.replica.NotifyReplicaLakeTableOffsetTest

Solution

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[rionmonster]

I think I may be able to take a stab at this.

I was able to pretty consistently reproduce it via a repeated iteration within IntelliJ, so it's very likely we can add some necessary retries or other behavior into the test harness to prevent it from happening:

@Test
void testBrokenSnapshotRecovery(@TempDir File snapshotKvTabletDir) throws Exception {
    int iterations = 50;
    for (int i = 0; i < iterations; i++) {
        // Associate separate bucket for each iteration
        TableBucket tableBucket = new TableBucket(i, 1);

        /** Add remaining body of test within try-catch to capture failures */
    }
}

[rionmonster]

I've issued a proposed PR to address this issue via a retry-handler within the affected test. After the test was reproducible via the previously mentioned iterations-approach, the proposed fix repeated passed as expected.

It may be worth considering introducing this change to the other explicit calls of triggerNonPeriodicScheduledTask() with the wrapper if future flaky tests are encountered.