[Bug] Routine Load properties silently revert to defaults after FE restart #69928

Search before asking

• I had searched in the issues and found no similar issues.

Version

Apache Doris master, commit 8e10b235cf0.

FE version: doris-0.0.0-8e10b235cf0

Reproduced in a single-FE development environment using the unpatched implementation.

What's Wrong?

Routine Load jobs can silently lose their explicitly configured execution properties after an FE restart or metadata recovery.

The following properties are affected:

• max_filter_ratio
• exec_mem_limit
• send_batch_parallelism
• load_to_single_tablet

These properties are correctly applied when a Routine Load job is initially created. However, after the FE restarts and reconstructs the job from persisted metadata, their effective runtime values can revert to defaults instead of retaining the user-configured values.

For example, a job configured with:

"max_filter_ratio" = "0.0",
"exec_mem_limit" = "3333333333",
"send_batch_parallelism" = "2",
"load_to_single_tablet" = "true"

can recover with default runtime settings, including max_filter_ratio = 1.0 and exec_mem_limit = 2147483648.

Potential impact

This is concerning because FE restarts are part of normal cluster maintenance and recovery, but should not change the execution semantics of existing ingestion jobs.

• Data-quality enforcement: A strict max_filter_ratio of 0.0 can revert to 1.0, allowing a much higher proportion of filtered rows to be tolerated. This can weaken ingestion validation and potentially result in unexpected data omissions.
• Memory management: exec_mem_limit can revert to the default, changing resource constraints and potentially causing unexpected memory consumption or task failures.
• Ingestion performance: send_batch_parallelism can revert to its default, changing task execution parallelism and throughput.
• Tablet-loading behavior: load_to_single_tablet can revert to false, changing the configured distribution behavior of load tasks.

The issue is especially problematic because the affected job may otherwise recover successfully, without an explicit ALTER ROUTINE LOAD operation or an immediately visible error indicating that its runtime configuration has changed.

What You Expected?

Routine Load jobs should preserve their effective execution configuration across FE restarts and metadata recovery.

Specifically:

1. All explicitly configured properties should retain their original values after recovery.

2. Properties modified through ALTER ROUTINE LOAD should retain their latest values rather than reverting to the original CREATE statement or defaults.

3. Recovery of metadata written by older FE versions should preserve nondefault configuration values wherever the persisted properties or original CREATE statement provide sufficient information.

4. Job identity, state, Kafka partition progress, and stored offsets should remain unaffected by configuration recovery.

For the reproduction below, the expected values after restarting the FE are:

An FE restart should not silently alter user-defined ingestion behavior.

How to Reproduce?

Prerequisites: A Doris cluster with an available BE, an accessible Kafka topic, and an OLAP destination table using random distribution (required for load_to_single_tablet = true).

Step 1: Create a Routine Load job with nondefault properties.

CREATE ROUTINE LOAD rl_config_recovery_test
ON rl_recovery_events
PROPERTIES (
    "max_filter_ratio" = "0.0",
    "exec_mem_limit" = "3333333333",
    "send_batch_parallelism" = "2",
    "load_to_single_tablet" = "true",
    "format" = "json"
)
FROM KAFKA (
    "kafka_broker_list" = "<broker-host>:9092",
    "kafka_topic" = "<test-topic>"
);

Step 2: Inspect the job configuration before restarting the FE.

SHOW ROUTINE LOAD FOR rl_config_recovery_test\G
SHOW CREATE ROUTINE LOAD FOR rl_config_recovery_test\G

Verify that the effective properties reflect the configured values.

Step 3: Gracefully restart the FE, preserving its existing metadata directory.

Step 4: Inspect the recovered job configuration.

SHOW ROUTINE LOAD FOR rl_config_recovery_test\G
SHOW CREATE ROUTINE LOAD FOR rl_config_recovery_test\G

Step 5: Compare the properties before and after recovery.

The job may recover with default values instead of the explicitly configured settings.

For example:

The failure is reproducible through FE metadata recovery without requiring any user-initiated modification to the job configuration.

Anything Else?

Initial root cause analysis

The issue appears to originate in RoutineLoadJob metadata serialization and deserialization.

Certain execution properties are maintained in runtime fields that are not independently serialized into FE metadata.

Although some of the corresponding values are retained in the serialized jobProperties map, the existing gsonPostProcess() recovery path reconstructs the Routine Load description without restoring these runtime fields from their persisted configuration.

Additionally, exec_mem_limit is not originally recorded in jobProperties during job creation.

Consequently, deserialization can reconstruct a valid Routine Load job while leaving its effective runtime configuration initialized to defaults.

Backward compatibility considerations

Persisted configuration should take precedence over the original CREATE statement because properties such as max_filter_ratio may have been modified through ALTER ROUTINE LOAD.

For older metadata that does not contain a particular persisted property, the original CREATE statement can provide a fallback where the configured value is available.

Related issue

SHOW CREATE ROUTINE LOAD also omits supported properties such as send_batch_parallelism and load_to_single_tablet.

This is a separate DDL-rendering issue and should be addressed independently of the runtime metadata recovery bug.

Proposed scope

A focused fix can restore the affected runtime configuration during FE metadata recovery, persist the missing execution memory-limit property, maintain backward compatibility with existing job metadata, and introduce regression tests for serialization and recovery.

The primary concern is the silent change to configured ingestion semantics. No specific production data-loss incident is being claimed.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct