Skip to content

Doris Roadmap 2023 #16392

@morningman

Description

@morningman

This is Apache Doris Roadmap 2023.

The plan is currently under discussion, so if you have comments or suggestions on any aspect of the plan or beyond, please feel free to leave a comment or send an email to dev@doris.apache.org.

We encourage developers to discuss anything in the dev mailing list, to subscribe to the mailing list please refer to How to subscribe.

We will gradually create issues for each direction of the plan to describe and track the progress in detail. Developers who wish to contribute are also welcome to create issues directly and associate with them (just leave a comment)

Roadmap 2022

Our Main Focus

  • Blazing fast OLAP
    • Reporting
    • Ad-hoc query
    • Customer or User facing analytics (high-concurrency)
  • Blazing fast query engine for datalake and lakehouse
    • Query acceleration for Hive
    • Query acceleration for open table format (Iceberg, Hudi, DeltaLake)
  • Semi-structured data storage and analysis
    • Log storage, retrieval, and analysis
    • Time series data storage, retrieval, and analysis
  • High-speed data processing (data engineering)
    • ETL/ELT acceleration
    • Streaming data warehouse

Release Schedule

We plan to release Apache Doris at the following pace:

V 1.2.x V 2.0.x V 2.1.x V 2.2.x
Jan. 1.2.1
Feb. 1.2.2
Mar. 1.2.3 2.0.0 alpha
Apr. 1.2.4 2.0.0 beta1
May 1.2.5 2.0.0 beta2
Jun. 1.2.6 2.0.0
Jul. 2.0.1 .
Aug. 2.0.2
Sept. 2.0.3 2.1.0 alpha
Oct. 2.0.4 2.1.0 beta
Nov. 2.1.0
Dec. 2.1.1

Features

We plan to develop or continuously optimize these features:

Hybrid Workloads

  • Query excution engine
    • Pipeline task parallelism
    • CodeGen
    • Adaptive execution enhencment
  • Spill To Disk
    • Sort Node
    • HashJoin Node
    • Aggregation Node
    • Sort Merge Join
    • Sort Aggregation
    • Optimize Spill To disk like compression, encryption, spill disk managment
    • New query management framework by using Spill To Disk
  • Workload manager for hybrid workloads
    • Resource isolation based on pipeline engine (CPU, Memory, IO)
    • Resource queue
    • Async execution
    • Query priority
    • Query scheduler

Semi-Structure Data Analysis

  • Complex Data Type
    • Array data type & functions
    • Jsonb data type & functions
    • Map data type & functions
    • Struct data type & functions
    • IPv4 & IPv6 data type & functions
    • GEO data type & functions
  • Index Enhancement
    • Ngram bloom filter index
    • Full-Text index for string/number/date
    • BKD numeric index for string/number/date
    • Full-Text & BKD index for Array
    • Full-Text & BKD index for Map
    • Full-Text & BKD index for Struct
    • BKD index for IPv4 & IPv6
    • BKD index for GEO
  • Dynamic Schema Table
    • Dynamic Schema Table syntax
    • Dynamic Schema Table write and read
    • Dynamic Schema Table index

Lakehouse & Data Integration

  • Query acceleration for datalake and lakehouse
    • Parquet, csv, orcfile
    • Iceberg
    • Hudi MOW
    • Hudi MOR
    • DeltaLake
    • Paimon(Flink Table Store)
  • Catalog & Cloud Storage integration
    • Hive Meta Store
    • AWS Glue
    • Alibaba Cloud DLF
    • Object Storage of AWS , Azure, GCP, Alibaba Cloud, Tencent Cloud, Huawei Cloud
  • Managed lake engine
    • Parquet writer
    • ORC writer
    • Doris Catalog for Iceberg
    • Managed Iceberg lake engine
  • Data Security
    • Keberos
    • KMS
    • Apache Ranger integration
    • Public Cloud (Alibaba Cloud, AWS) IAM Role
  • New Spark/Flink Load
    • Writing Doris data format file externally.
    • Refractor the framework of Spark/Flink Load to support batch load.
  • Hive/Presto/Spark function compatibility
  • Graph database federated query support

New Optimizer (Nereids)

  • Features
    • Fully feature support or replace the old query optimizer
    • DML (insert, update, merge)
    • Query cache
  • Performance
    • Optimize the time consumption of the plan stage
    • RBO Rules enhancement
    • CBO Rules enhancement, inline CTE, etc.
  • Support for hybrid workloads
    • Optimize rules for datalake engine
    • Adaptive query plan
    • Adaptive sort/agg algorithm
  • Statistics enhancement
    • Statistics derivation optimization, improve accuracy, support complex expressions
    • Richer statistics to support non-uniform distribution data
    • Optimize statistics persistence and caching mechanism
    • Auto collect statistics
    • Optimiza cost model that is more adaptable to distributed scenarios

Cost Efficiency & Performance

  • Cloud Native
    • Cold & Hot Data Separation
    • Elastic Compute Node
  • Low-latency, high-concurrency point query
  • Aggregating index & projection
  • Performance Self Tunning
  • Multi-Table Materialized View
    • Automatic Incremental refresh
    • Automatic query rewriting

Data Modeling & Storage Engine

  • Cross Cluster Replication (CCR) & Binlog
    • CCR to enable higher HA
    • Binlog to enable streaming computing
  • Unique Key Constraint
    • Merge-on-Write (MoW) Unique Key Table
    • Partial Column Update on MoW UNIQUE Key Table
  • DDL Simplification
    • Support functions in partitioning
    • Auto Bucket Number
  • Unified Data Model
  • General Delete, Update, Merge Support
  • Light Schema Change
    • Do not effect on historical data and work on newer data

Ecosystem

  • Enhance BI tools compatibility
    • Matebase
    • Superset
    • Tableau
  • Enhance doris-dbt
  • Enhance Doris-Airbyte
  • Enhance integration with cloud data integration tools

Utility & Stability

  • RBAC (Role-Based-Access-Control) enhancement
  • Support column-level authorization
  • Profiling / Tracing enhancement
  • Doris Manager enhancement
  • Multi-language UDF
  • More Fuzzy tests
  • All HTTP APIs support HTTPS and authorization
  • Full support for K8s deployment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions