3.1.3 Release Notes

[morrySnow]

3.1.3 Release Notes

🚀 New Features

🗄️ Storage & Filesystem

• Support Azure Blob Storage (branch-3.1: [feat](storage)Support Azure Blob Storage #56861 #57219)
• Upgrade libhdfs to 3.4.2 (branch-3.1: [feat](hdfs) upgrade libhdfs to 3.4.2 #57621 #57638)
• Add TotalGetRequestTime profile metric to S3 reader (branch-3.1:[enhancement](s3)Add TotalGetRequestTime profile metric to the S3 reader. (#57519) #57636)

📚 Catalog

• Support MaxCompute catalog (project-schema-table) (branch-3.1: [feature](maxcompute)support maxcompte catalog read project-schema-table. #57012 #57286)
• Refactor HDFS dependency management (move jars to FE) (branch-3.1: [Enhancement](sql-dialect) Support retrying original sql after error #57498)

☁️ Cloud Mode

• Support balance sync warm-up (branch-3.1: [feature](cloud) Support balance sync warm up #56164 #57666)
• Support peer BE cache read in same cluster (branch-3.1: [feature](cloud) Read peer be cache when balance in same cluster #56384 #57672)

🧮 SQL Engine & Planner

• Check SQL regex block rule before plan phase (branch-3.1: [enhancemment](planner)check sql regex block rule before plan phase #57477 #57706)

🔢 Functions

• Enhance EXPLODE function with struct-type support (branch-3.1: [opt](function) Enhance the handling of a single struct-type argument in the EXPLODE function #57050 #57827)

🌐 Web & Platform

• Adapt Web service to Spring Boot 3 and Jetty 12 (branch-3.1: [feat](web)Adapted for Spring Boot 3 and Jetty 12 #57710 #57822)

---

⚙️ Optimizations

🧠 Query Execution & Planner

• Optimize variant cast when only NULL values (branch-3.1: [Improve](variant) optimize cast performance when variant only contains null values #57003 #57161)
• Optimize FROM_UNIXTIME performance (branch-3.1: [Opt](function) Optimize the performance of FROM_UNIXTIME #57423 #57573)
• Improve graceful shutdown behavior and query retry (branch-3.1: [opt](scheduler) Improve Graceful Shutdown Behavior for BE and FE, and Optimize Query Retry During BE Shutdown #56601 #57805)

💾 Storage & Compaction

• Make read slice size configurable (MergeIO) (branch-3.1: [Opt](MergeIO) make read slice size is configurable #55936 #57272 #57159)
• Add score threshold for cold data compaction (branch-3.1: [chore](cold) add a score threshold for cold data compaction #55333 #57217)
• Add configurable threshold for small memory task protection (branch-3.1: [enhancement](memory) Add configurable threshold for small memory task protection #56994)
• Improve jemalloc configuration to reduce page faults (branch-3.1: [mem](jemalloc) Improve memory reuse efficiency and reduce page faults to change jemalloc conf #57076 #57152)

☁️ Cloud Mode

• Expose cloud balance metrics (branch-3.1: [opt](cloud) Exposes cloud balance metrics #57200 #57352)
• Optimize create warm-up job logic (branch-3.1: [opt](cloud) optimize create warmup job logic #57752 #57865)
• Enhance warm-up job and peer read efficiency (branch-3.1: [fix](cloud) Fix cloud balance warm up lost idx file #57114 #57554 branch-3.1: [fix](cloud)Fix read from peer use thread pool not asyncio #57587 #57807)

🔍 Index & Search

• Add custom analyzer support (char_filter, basic & ICU tokenizer) (branch-3.1: [opt](inverted index) add custom analyzer support with char_filter, basic and icu tokenizer #56243 #57055 #57137)
• Support builtin analyzer names in custom analyzer ([opt](inverted index) support builtin analyzer names in custom analyzer field #57727)

---

🧩 Bug Fixes

💽 Storage & File I/O

• Fix segcompaction coredump when adding key column (branch-3.1: [fix](segcompaction) Fix segcompaction coredump when adding key column #57162 #57212)
• Fix Parquet RLE_DICTIONARY decode performance issue (branch-3.1:[enhancement](parquet)Optimize the performance of parquet reader when decode RLE_DICTIONARY encoding (#57208) #57614)
• Fix schema change expr cache misuse (branch-3.1: [fix](schema_change) fix incorrect cache usage of schema change expr leading to schema errors #56602 #57517)
• Replace tablet report with ForkJoinPool (branch-3.1: [fix](tablet report)Replace tablet report with ForkJoinPool #57382 #57927)

☁️ Cloud Mode

• Fix cloud pipeline task number calculation (branch-3.1: [fix](cloud) fix correct pipeline task number calculation in cloud mode #57129 #57261)
• Fix rebalance residual metrics cleanup (branch-3.1: [fix](cloud) Fix the residual metrics of cluster after drop compute group #57235 #57438)
• Skip tablet report when rebalance not inited (branch-3.1: [fix](cloud) Skip tablet report when CloudTabletRebalancer is not inited #56989 #57393)
• Fix domain user default cluster report error (branch-3.1: [fix](cloud) Fix domain user set default cluster report err #56032 #57555)
• Fix wrong private endpoint config (branch-3.1: [fix](cloud) fix wrong private endpoint when only configured port number #57429 #57675)
• Fix peer read bugs and thread handling (branch-3.1: [fix](cloud) Fix read peer file cache some bugs #57838 #57910 branch-3.1: [fix](cloud)Fix read from peer use thread pool not asyncio #57587 #57807)
• Fix filecache metrics and microbench issues (branch-3.1: [fix](filecache) fix filecache microbench bvar #56360 #57535 branch-3.1: [fix](filecache) fix BytesScannedFromCache/Local always zero #55929 #57536)

📚 Catalog

• Fix MaxCompute predicate pushdown null pointer (branch-3.1: [fix](maxcompute)fix maxcompute push down predicates null pointer. #57474 #57567)
• Fix Iceberg client.region and REST credentials issues (branch-3.1: [fix](iceberg) Support client.region and iceberg.rest.* properties invended credentials #57521 #57539)
• Fix Iceberg catalog NPE and query error (branch-3.1: [fix](catalog) fix iceberg catalog NPE issue #57696 #57796 branch-3.1: [fix](iceberg) fix querying iceberg partition table error #57571 #57790)
• Fix Paimon S3 prefix unification and configuration (branch-3.1: [fix](paimon)Support user-defined S3 config prefixes and unify to HDFS S3A protocol (#57116) #57526)
• Fix JDBC catalog compatibility for zeroDateTimeBehavior (branch-3.1: [fix](jdbc catalog) Compatible with different versions of zeroDateTimeBehavior=convertToNull writing method #57682 #57731)
• Fix Parquet schema analysis error (branch-3.1: [fix](outfile) fix analysis error when specifying parquet schema (#57232) #57500)
• Fix Parquet all row groups filtered issue (branch-3.1: [fix] (parquet-reader) Fix parquet all row groups are filtered.  #57646)
• Fix CSV reader wrong result when escape equals enclose (branch-3.1: [fix](csv reader) fix wrong result when escape same as enclose #57632 #57762)
• Prevent removing catalog from refresh queue (branch-3.1: [fix](catalog) Prevent removing catalog from refresh queue on CREATE/ALTER #57671)
• Fix max_meta_object_cache_num must > 0 config (branch-3.1: [fix](catalog) max_meta_object_cache_num config must > 0 #57742 #57793)

🧮 SQL Engine & Planner

• Fix constant folding of FROM_UNIXTIME with decimal literal (branch-3.1: [fix](nereids)fold constant rule can't fold from_unixtime when its 1st param is decimalv3 literal #57363 #57606)
• Fix MV rewrite fail with group sets and filter (branch-3.1: [fix](mtmv) Fix mv rewrite fail when mv contains group sets and filter above scan #57343 #57674)
• Fix prepare statement stage only explain SQL (branch-3.1: [fix](prepare statement)Prepare stage only explain sql to get output slots. #56224 #57504)
• Release physical plan in Profile.releaseMemory() (branch-3.1: [fix](nereids)relase physical plan in Profile.releaseMemory() #57257 #57316)
• Fix aggregation elimination error when group sets exist (branch-3.1: [fix](nereids) Fix not in aggregate's output err after eliminate by uniform when group sets exist #56942 #57885)
• Fix LargeInt overflow on max_value+1 (branch-3.1: [fix](LargeInt)Fix large int max_value + 1 not overflow bug. #56083 #57351)
• Fix decimal256 overflow when casting to float (branch-3.1: [fix](decimal256) fix unstable overflow error when casting decimal256 to float #56848 #57503)

🔌 Networking & Platform

• Fix MySQL SSL unwrap infinite loop (branch-3.1: [fix](mysql) Fix SSL unwrap infinite loop on handshake failure #57364 #57599)
• Disable TLS renegotiation in MySQL (branch-3.1: [Fix](mysql) Disable renegotiation during TLS #57631 #57748)
• Fix unaligned uint128 constructor (branch-3.1: [Bug](ip) fixed uint128 constructor in not align address #57371 #57430)
• Fix JNI local/global ref leak (branch-3.1: [Bug](Jni)should delete local ref after convert to global ref #57513 #57597)
• Make Scanner.close() thread-safe (branch-3.1: [fix](scanner) Make Scanner close() method thread-safe using atomic operations #57436 #57644)
• Fix exchange coredump due to nullptr (branch-3.1: [Bug](exchange) avoid coredump coz nullptr #57640 #57698)

---

🧱 Chores & Infrastructure

• Deprecated LakeSoul external catalog (branch-3.1: [chore](external catalog) deprecated lakesoul external catalog #56843 #57163)

---

✅ Summary

Apache Doris 3.1.3 brings major improvements in:

• Storage compatibility (Azure Blob, Hadoop 3.4.2, S3 metric support)
• Cloud performance and reliability (warm-up, rebalance, peer cache)
• SQL planner stability
• Dependency modernization and security hardening

This release greatly enhances stability, performance, and cloud-native integration across hybrid data environments.

---

✅ Full Changelog: [apache/doris@3.1.2-rc01...3.1.3-rc01](3.1.2-rc01...3.1.3-rc01)

---

Contributors

@0AyanamiRei @BiteTheDDDDt @CalvinKirs @Gabriel39 @HappenLee @JNSimba @Jibing-Li @TangSiyang2001 @Yukang-Lian @bobhan1 @cambyzju @csun5285 @dataroaring @deardeng @englefly @felixwluo @freemandealer @gavinchou @hubgeter @jacktengg @kaka11chen @liaoxin01 @linrrzqqq @liutang123 @luwei16 @morningman @morrySnow @mrhhsg @mymeiyi @qzsee @seawinde @sollhui @starocean999 @suxiaogang223 @wenzhenghu @wyxxxcat @zclllyybb @zhangstar333 @zy-kkk @zzzxl1993

[Nj-kol]

Hello @morrySnow,

Could you please show an example on how to use an Azure Storage vault? Does this doc still hold - https://doris.apache.org/docs/dev/sql-manual/sql-statements/cluster-management/storage-management/CREATE-STORAGE-VAULT#8-create-azure-storage-vault?

I just tried to run Doris 4.0.1 on Azure Kubernetes Service with Azure Vault as a storage in storage computed decoupled mode.

CREATE STORAGE VAULT IF NOT EXISTS azure_vault
PROPERTIES (
    "type" = "S3",
    "s3.endpoint" = "mystorageaccount.blob.core.windows.net",
    "s3.access_key" = "mystorageaccount",
    "s3.secret_key" = "myaccountkey", 
    "s3.root.path" = "doris",
    "s3.region" = "eu-west-1",  
    "s3.bucket" = "mystorageaccount",
    "provider" = "azure"
);

After, I added azure vault, the compute group crashes with the following error log :

F20251118 10:19:04.645164 209 s3_util.cpp:255] BE is not compiled with azure support, export BUILD_AZURE=ON before building
I20251118 10:19:05.449136 209 daemon.cpp:231] sys physical memory 3.00 GB. process memory used 1.11 GB(= 1.11 GB[vm/rss] + 0[reserved] + 0B[waiting_refresh]), limit 2.70 GB, soft limit 2.43 GB. sys available memory 2.16 GB(= 2.16 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 153.60 MB,

[plastikman]

@Nj-kol I just stood up a disagragated storage cluster using azure blob and had to make a few changes.

Your provider needs to be AZURE seems the case matters

The docs say this

CREATE STORAGE VAULT IF NOT EXISTS azure_demo_vault
PROPERTIES (
    "type" = "S3",                                       -- required
    "s3.endpoint" = "blob.core.windows.net",             -- required
    "s3.access_key" = "xxxxxx",                          -- required,  Your Azure AccountName
    "s3.secret_key" = "xxxxxx",                          -- required,  Your Azure AccountKey
    "s3.region" = "us-east-1",                           -- required
    "s3.root.path" = "azure_demo_vault_prefix",          -- required
    "s3.bucket" = "xxxxxx",                              -- required,  Your Azure StorageAccount
    "provider" = "AZURE"                                 -- required
);

My example looks like this, the major change was that you MUST put the storage account name in the endpoint

              CREATE STORAGE VAULT IF NOT EXISTS azure_vault
              PROPERTIES (
                  "type" = "S3",
                  "s3.endpoint" = "${STORAGE_ACCOUNT}.blob.core.windows.net",
                  "s3.access_key" = "${STORAGE_ACCOUNT}",
                  "s3.secret_key" = "${AZURE_STORAGE_ACCOUNT_KEY}",
                  "s3.region" = "us-east-1",
                  "s3.bucket" = "${AZURE_CONTAINER}",
                  "s3.root.path" = "doris-${PROFILE}-${REGION}",
                  "provider" = "AZURE"
              );

These are the definitions of my variables

STORAGE_ACCOUNT = dorisstoragebucket
AZURE_STORAGE_ACCOUNT_KEY = xxxxxxxxxxxxxx
AZURE_CONTAINER = blob_container
PROFILE = dev, qa, prod, etc
REGION = eastus, eastus2, etc

AZURE_STORAGE_ACCOUNT_KEY is the secret in the vault mounted with eso
PROFILE and REGION are specific to my deployment these can be what ever.

I also had to set specific configuration to make this work in fe and be, i set it in ms just to be safe. Without the below setting configured the fe and be both balked trying to connect to the wrong endpoints or the the http endpoints.

force_azure_blob_global_endpoint = true

Also based on this

F20251118 10:19:04.645164 209 s3_util.cpp:255] BE is not compiled with azure support, export BUILD_AZURE=ON before building

I would bet you are running on ARM, i was not able to make this run on ARM using azure blob, you will need to use x86.

[plastikman]

I also just ran into this (upgrade to 4.0.1), seems that the 4.0.1 branch was released after this, and the code made it into 4.0.1 but the images were not built with BUILD_AZURE=ON :(

[Nj-kol]

@Nj-kol I just stood up a disagragated storage cluster using azure blob and had to make a few changes.

Your provider needs to be AZURE seems the case matters

The docs say this

CREATE STORAGE VAULT IF NOT EXISTS azure_demo_vault
PROPERTIES (
    "type" = "S3",                                       -- required
    "s3.endpoint" = "blob.core.windows.net",             -- required
    "s3.access_key" = "xxxxxx",                          -- required,  Your Azure AccountName
    "s3.secret_key" = "xxxxxx",                          -- required,  Your Azure AccountKey
    "s3.region" = "us-east-1",                           -- required
    "s3.root.path" = "azure_demo_vault_prefix",          -- required
    "s3.bucket" = "xxxxxx",                              -- required,  Your Azure StorageAccount
    "provider" = "AZURE"                                 -- required
);

My example looks like this, the major change was that you MUST put the storage account name in the endpoint

              CREATE STORAGE VAULT IF NOT EXISTS azure_vault
              PROPERTIES (
                  "type" = "S3",
                  "s3.endpoint" = "${STORAGE_ACCOUNT}.blob.core.windows.net",
                  "s3.access_key" = "${STORAGE_ACCOUNT}",
                  "s3.secret_key" = "${AZURE_STORAGE_ACCOUNT_KEY}",
                  "s3.region" = "us-east-1",
                  "s3.bucket" = "${AZURE_CONTAINER}",
                  "s3.root.path" = "doris-${PROFILE}-${REGION}",
                  "provider" = "AZURE"
              );

These are the definitions of my variables

STORAGE_ACCOUNT = dorisstoragebucket
AZURE_STORAGE_ACCOUNT_KEY = xxxxxxxxxxxxxx
AZURE_CONTAINER = blob_container
PROFILE = dev, qa, prod, etc
REGION = eastus, eastus2, etc

AZURE_STORAGE_ACCOUNT_KEY is the secret in the vault mounted with eso PROFILE and REGION are specific to my deployment these can be what ever.

I also had to set specific configuration to make this work in fe and be, i set it in ms just to be safe. Without the below setting configured the fe and be both balked trying to connect to the wrong endpoints or the the http endpoints.

force_azure_blob_global_endpoint = true

Also based on this

F20251118 10:19:04.645164 209 s3_util.cpp:255] BE is not compiled with azure support, export BUILD_AZURE=ON before building

I would bet you are running on ARM, i was not able to make this run on ARM using azure blob, you will need to use x86.

Hi @plastikman thank you so much for your detailed reply on this. I had got this working fortuntelty from this blog - https://ai.feishu.cn/docx/ULT8dYXKtoP1MrxSeNwcEYgVn0d

I just wish the official doc had this level of clarity. For me, it has worked even without force_azure_blob_global_endpoint=true.

Also, for doris 4.x, I struggled to make it work and ultimatelty downgraded to 3.x

[plastikman]

I ended up building the be and ms for 4.0.1 with BUILD_AZURE=ON and it worked. I will hope that in 4.0.2 the BUILD_AZURE=ON will be set :)