Apache OpenDAL™ is an Open Data Access Layer that provides unified, programmatic access to diverse storage services through a consistent API. This document provides a high-level overview of OpenDAL's architecture, core concepts, and organizational structure within the codebase.
Scope: This page covers the overall architecture and organization of the OpenDAL project. For detailed information about specific components, see:
OpenDAL operates under the vision of "One Layer, All Storage", providing a single abstraction layer for accessing any storage service through any method in any programming language. The project is built on five core principles:
Sources: README.md1-10
The OpenDAL repository is organized as a monorepo containing five major subsystems:
Sources: core/Cargo.toml18-30 README.md13-98 Diagram 1
The repository uses Cargo workspaces to manage multiple related packages:
| Directory | Purpose | Example Packages |
|---|---|---|
core/ | Rust core implementation | opendal, opendal-core |
core/services/ | Storage backend implementations | opendal-service-s3, opendal-service-redis |
core/layers/ | Middleware layer implementations | opendal-layer-retry, opendal-layer-logging |
bindings/ | Language binding implementations | opendal-python, opendal-nodejs, opendal-java |
integrations/ | Ecosystem integration crates | object_store_opendal, dav-server-opendalfs |
core/testkit | Shared testing utilities | opendal-testkit |
Sources: core/Cargo.toml18-30
OpenDAL implements a three-tier architecture that separates concerns between the user-facing API, middleware processing, and storage backend implementations:
Sources: core/src/lib.rs1-241 core/core/src/layers/mod.rs1-37 Diagram 2
The opendal crate serves as a facade that re-exports all public APIs from opendal-core and optionally compiled services/layers. This design allows independent versioning and modular compilation:
Sources: core/src/lib.rs22-25 core/Cargo.toml61-294
Operator is the primary entry point for all storage operations. It provides a unified API regardless of the underlying storage backend:
All operations (read, write, list, delete, stat, etc.) are exposed through this interface.
Sources: Diagram 2
The Access trait defines the core operations that all storage backends must implement. Each service creates a backend struct that implements this trait:
Sources: core/core/src/services/mod.rs1-24 Diagram 2
Services follow a standardized construction pattern:
Example from Redis service:
Sources: core/services/redis/src/backend.rs44-202 core/services/redis/src/config.rs78-125
OpenDAL uses a runtime registry system for dynamic service discovery:
The registration happens automatically via #[ctor::ctor] attribute, which ensures init_default_registry() is called before main:
Sources: core/src/lib.rs30-241 core/services/redis/src/lib.rs36-38 Diagram 6
OpenDAL uses Cargo feature flags for granular compilation control. Services and layers are optional dependencies that can be enabled selectively:
This allows users to compile only the services they need, reducing binary size and compilation time.
Sources: core/Cargo.toml82-188
OpenDAL supports 50+ storage backends organized by type:
| Category | Services |
|---|---|
| Object Storage | S3, Azure Blob, GCS, OSS, COS, OBS, B2, Swift, Upyun, Vercel Blob |
| File Systems | fs, HDFS, HDFS Native, WebDAV, Alluxio, Azure DLS, Azure File, GridFS, IPFS, WebHDFS, CompFS, DBFS |
| Key-Value Stores | Redis, Memcached, etcd, RocksDB, Sled, Redb, TiKV, FoundationDB, Persy, Dashmap, Memory, Cacache, Cloudflare KV |
| Databases | PostgreSQL, MySQL, SQLite, MongoDB, SurrealDB, D1 |
| Cloud Storage | Google Drive, OneDrive, Dropbox, Aliyun Drive, Koofr, pCloud, Seafile, Yandex Disk |
| Caches | GitHub Actions Cache, Memcached, Mini Moka, Moka, Vercel Artifacts |
| Protocols | HTTP, FTP, SFTP, WebDAV |
| Version Control | Huggingface |
Each service implements the same Access trait, providing consistent behavior across all backends.
Sources: README.md136-208 core/README.md25-99 .env.example1-202
All services follow a consistent four-component pattern. Using Redis as an example:
Components:
Configurator traitBuilder traitAccess traitSources: core/services/redis/src/config.rs28-76 core/services/redis/src/backend.rs44-202 core/services/redis/src/core.rs115-213
Layers provide middleware functionality for cross-cutting concerns. They wrap the Operator to add behavior before/after operations:
Built-in Layers (always included in opendal-core):
Optional Layers (feature-gated):
Sources: core/core/src/layers/mod.rs1-37 core/README.md101-155
OpenDAL provides bindings for 14+ programming languages using two architectural approaches:
Languages with mature FFI frameworks integrate directly with Rust:
Languages without direct Rust FFI use the stable C ABI:
This approach simplifies binding development and provides a stable interface.
Sources: bindings/nodejs/package.json1-117 bindings/java/pom.xml1-309 Diagram 3
OpenDAL provides integrations with various ecosystems:
Sources: README.md99-135 Diagram 5
OpenDAL employs a comprehensive behavior test system that validates all storage backends against a common specification:
Each service must pass the same set of behavior tests, ensuring consistent functionality. Configuration is managed through .env.example templates.
Sources: core/tests/behavior/main.rs1-84 .env.example1-202
The OpenDAL project uses independent versioning for different components:
opendal): Currently v0.55.0opendal-python): Independent versionopendal-nodejs): v0.49.2opendal-java): v0.48.2This allows bindings to evolve at their own pace while maintaining stability.
Sources: core/Cargo.toml41 bindings/nodejs/package.json4 bindings/java/pom.xml35 README.md14-15
For information on using OpenDAL, see:
Sources: README.md1-239 core/README.md1-159
Refresh this wiki
This wiki was recently refreshed. Please wait 7 days to refresh again.
