Overview

Relevant source files

• .github/workflows/core-hadoop2-ci.yml
• .github/workflows/core-hadoop3-ci.yml
• .github/workflows/deps.yml
• .github/workflows/docker-images.yml
• .github/workflows/openapi-sdk-ci.yml
• .github/workflows/trino-ci.yml
• .gitignore
• CONTRIBUTING.md
• README.md
• amoro-ams/pom.xml
• amoro-common/pom.xml
• amoro-common/src/main/thrift/README.md
• amoro-format-hudi/pom.xml
• amoro-format-iceberg/pom.xml
• amoro-format-mixed/amoro-mixed-hive/pom.xml
• amoro-format-mixed/amoro-mixed-spark/amoro-mixed-spark-3-common/pom.xml
• amoro-format-mixed/amoro-mixed-spark/amoro-mixed-spark-3-common/src/test/java/org/apache/amoro/spark/test/utils/TestTableUtil.java
• amoro-format-mixed/amoro-mixed-spark/pom.xml
• amoro-format-mixed/amoro-mixed-spark/v3.3/amoro-mixed-spark-3.3/pom.xml
• amoro-format-mixed/amoro-mixed-spark/v3.4/amoro-mixed-spark-3.4/pom.xml
• amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-3.5/pom.xml
• amoro-format-mixed/amoro-mixed-trino/pom.xml
• amoro-format-paimon/pom.xml
• amoro-metrics/amoro-metrics-prometheus/pom.xml
• amoro-metrics/pom.xml
• amoro-openapi-sdk/pom.xml
• amoro-openapi-sdk/src/test/java/org/apache/amoro/openapi/OpenAPITest.java
• amoro-optimizer/amoro-optimizer-common/pom.xml
• amoro-optimizer/amoro-optimizer-flink/pom.xml
• amoro-optimizer/amoro-optimizer-spark/pom.xml
• amoro-optimizer/amoro-optimizer-standalone/pom.xml
• amoro-optimizer/amoro-optimizer-standalone/src/main/java/org/apache/amoro/optimizer/standalone/StandaloneOptimizer.java
• amoro-optimizer/amoro-optimizer-standalone/src/main/resources/log4j2.xml
• amoro-optimizer/pom.xml
• dev/dependencies.sh
• dev/deps/dependencies-hadoop-2-spark-3.3
• dev/deps/dependencies-hadoop-3-spark-3.5
• dev/reformat
• docker/README.md
• docker/amoro/Dockerfile
• docker/build.sh
• docker/optimizer-flink/Dockerfile
• docs/admin-guides/deployment.md
• pom.xml
• tools/releasing/create_binary_release.sh
• tools/releasing/create_source_release.sh

Apache Amoro is a Lakehouse management system built on open data lake formats. It provides self-optimizing tables, unified catalog services, and automated maintenance operations for data lakes stored on HDFS, S3, or other distributed filesystems.

Key features:

• Self-Optimizing Tables: Automatic compaction, sorting, and data expiration
• Multi-Format Support: Iceberg, Mixed-Iceberg, Mixed-Hive, Paimon, and Hudi tables
• Unified Catalog: Single metadata interface for Flink, Spark, and Trino engines
• Flexible Optimization: Deploy optimizers on local JVM, Flink, Spark, or Kubernetes
• High Availability: ZooKeeper-based leader election with automatic failover

For detailed information about specific subsystems, see:

• Amoro Management Service (AMS) - Central service architecture and APIs
• Optimizer Framework - Pluggable optimization execution
• Table Management - Table lifecycle and self-optimizing
• Deployment - Installation and configuration

System Architecture

The following diagram illustrates Amoro's high-level architecture with actual component and class names from the codebase:

System Architecture with Code Entities

This diagram shows Amoro's architecture mapping natural language concepts to concrete code entities. The AMS (implemented in AmoroServiceContainer) serves as the central orchestration point, exposing three service interfaces. Optimizer containers implement the AbstractOptimizerContainer interface to provide pluggable execution strategies. All metadata operations flow through MyBatis mappers to a relational database, while data operations interact directly with the Hadoop FileSystem API.

Sources: pom.xml47-58 amoro-ams/pom.xml29-32 README.md40-61 docs/admin-guides/deployment.md71-97

Core Components

Amoro Management Service (AMS)

The AMS is the central management service implemented in AmoroServiceContainer located in the amoro-ams module. It exposes three primary service endpoints configured via AmoroManagementConf:

Service	Port Config Property	Default	Protocol	Purpose
Dashboard Server	ams.http-server.bind-port	1630	REST/HTTP (Javalin)	Web UI and REST API for management operations
Table Service	ams.thrift-server.table-service.bind-port	1260	Thrift	Table metadata access for compute engines
Optimizing Service	ams.thrift-server.optimizing-service.bind-port	1261	Thrift	Task distribution and optimizer coordination

The AMS supports high availability through ZooKeeper-based leader election configured via ams.ha.enabled and ams.ha.zookeeper-address properties:

• Always-Active Services: DashboardServer and REST Catalog endpoints run on all instances
• Leader-Only Services: OptimizingService and table optimization orchestration run only on the leader node
• Failover: Leader election managed by amoro-shade-zookeeper-3 with automatic failover on leader failure

The service container lifecycle progresses through states: CREATED → STARTING → STARTED → STOPPING → STOPPED.

Sources: amoro-ams/pom.xml29-32 docs/admin-guides/deployment.md71-97 README.md54-60 docs/admin-guides/deployment.md128-139

Optimizer Framework

Optimizers execute self-optimizing operations on tables through a pluggable container abstraction. The framework is organized in the amoro-optimizer module hierarchy:

Optimizer Container Architecture

Container configuration in config.yaml:

• container-impl: Fully qualified class name (e.g., org.apache.amoro.server.manager.LocalOptimizerContainer)
• properties: Environment variables and container-specific settings (JVM args, Hadoop config, resource limits)

Each container manages:

• Optimizer instance lifecycle (start, scale, stop)
• Resource allocation and quota management via OptimizerGroup
• Heartbeat monitoring and failure detection
• Task distribution through OptimizingQueue

Sources: docs/admin-guides/deployment.md142-176 README.md57-60 pom.xml51-52 amoro-optimizer/amoro-optimizer-standalone/src/main/java/org/apache/amoro/optimizer/standalone/StandaloneOptimizer.java1-100

Storage Layer

Amoro operates on top of existing data lake storage and metadata systems:

• Data Lake Storage: HDFS, S3, OSS, or any Hadoop-compatible filesystem
• Metadata Store: MySQL, PostgreSQL, or embedded Derby database
• Catalog Types: Supports Hive Metastore, AWS Glue, Hadoop catalogs, and native Iceberg REST catalogs

Sources: README.md62-71 docs/admin-guides/deployment.md104-126

Supported Table Formats

Amoro manages tables across four distinct formats, each optimized for different use cases:

Format	Module	Use Case
Iceberg	amoro-format-iceberg	Direct management of Apache Iceberg tables with all native Iceberg capabilities
Mixed-Iceberg	amoro-format-mixed	Enhanced format for streaming updates with optimized change data capture (CDC)
Mixed-Hive	amoro-mixed-hive	Upgrade path for existing Hive tables to lakehouse format via metadata migration
Paimon	amoro-format-paimon	Integration with Apache Paimon tables for metadata display and management
Hudi	amoro-format-hudi	Integration with Apache Hudi tables

The Mixed-Iceberg format introduces a three-tier storage architecture:

• BaseStore: Immutable baseline data
• ChangeStore: Incremental changes for streaming scenarios
• LogStore: Message queue integration (Kafka/Pulsar) for millisecond-latency access

Sources: README.md62-71 pom.xml53-56 amoro-format-iceberg/pom.xml28-30 amoro-format-paimon/pom.xml29-30

Supported Compute Engines

Engine Integration Matrix

The following table shows compute engine support for different table formats:

Engine	Versions	Batch Read	Batch Write	Streaming Read	Streaming Write	Module
Flink	1.16.x - 1.18.x	✓	✓	✓	✓	amoro-mixed-flink
Spark	3.3, 3.4, 3.5	✓	✓	✗	✗	amoro-mixed-spark-{version}
Trino	406+	✓	✗	✗	✗	amoro-mixed-trino
Hive	2.x, 3.x	✓	✗	✗	✗	amoro-mixed-hive

Each Spark version has a dedicated integration module (amoro-mixed-spark-3.3, amoro-mixed-spark-3.4, amoro-mixed-spark-3.5) built on a common base (amoro-mixed-spark-3-common).

Sources: README.md79-89 pom.xml47-58 amoro-format-mixed/amoro-mixed-spark/pom.xml1-54

Maven Module Structure

Maven Module Dependency Graph

The module structure follows these design principles:

1. Shaded Dependencies: Critical libraries are relocated to org.apache.amoro.shade.* packages to prevent classpath conflicts:

   • amoro-shade-thrift (0.20.0) - Thrift RPC framework
   • amoro-shade-guava-32 (32.1.1-jre) - Google Guava utilities
   • amoro-shade-jackson-2 (2.14.2) - JSON serialization
   • amoro-shade-zookeeper-3 (3.9.1) - ZooKeeper client

2. Format Abstraction: Each format implements the TableDescriptor interface defined in amoro-common, enabling pluggable format support.

3. Multi-Version Support: Engine connectors use Maven profiles to build for multiple versions:

   • Spark: Separate artifacts for 3.3, 3.4, 3.5 with common base in amoro-mixed-spark-3-common
   • Flink: Version-specific modules for 1.15, 1.18, 1.20 with common base in amoro-mixed-flink-common
   • Runtime JARs use Maven Shade Plugin to create uber-JARs with relocated dependencies

4. Binary Distribution: The dist module uses maven-assembly-plugin to package:

   • AMS server with all dependencies in lib/
   • Optimizer binaries in plugin/optimizer/
   • Configuration templates in conf/
   • Shell scripts in bin/

Sources: pom.xml47-58 README.md99-115 amoro-common/pom.xml30-64 amoro-ams/pom.xml39-61 amoro-format-mixed/amoro-mixed-spark/pom.xml1-54 amoro-format-mixed/amoro-mixed-spark/v3.3/amoro-mixed-spark-3.3/pom.xml29-32

Key Capabilities

Self-Optimizing

Amoro continuously optimizes tables through background processes that:

• Compact Small Files: Merges small files to improve read performance
• Sort and Deduplicate: Organizes data for optimal query patterns
• Expire Data: Removes old snapshots and data based on retention policies
• Clean Orphan Files: Identifies and removes unreferenced files

The self-optimizing process is managed through the TableRuntime state machine, which tracks optimization status and schedules tasks to optimizer instances.

Sources: README.md92-93 docs/admin-guides/deployment.md142-176

Unified Catalog Service

The CatalogManager provides a unified catalog abstraction supporting:

• Hive Metastore integration
• AWS Glue
• Hadoop catalogs
• Iceberg REST Catalog protocol

Compute engines access tables through the Table Service (port 1260), which provides consistent metadata access regardless of the underlying catalog implementation.

Sources: README.md94 docs/admin-guides/deployment.md75-76

Multi-Format Management

Through its modular architecture, Amoro provides unified management capabilities across different table formats while preserving format-specific features. The TableManager coordinates operations across formats through a pluggable provider system.

Sources: README.md93 pom.xml53-56

Deployment Options

Amoro supports multiple deployment models with corresponding build artifacts:

Deployment Artifact and Build Pipeline

Configuration Files

Key configuration files and their primary settings:

File	Purpose	Key Properties
conf/config.yaml	AMS service configuration	ams.server-bind-host ams.thrift-server.table-service.bind-port ams.database.* ams.ha.*
conf/jvm.properties	JVM tuning	xms, xmx, jmx.remote.port, extra.options
containers section	Optimizer deployment	container-impl (fully qualified class name) properties (env vars and config)

The AmoroManagementConf class in amoro-ams loads configuration from config.yaml with environment variable overrides via the ConfigurationHelper.

Build Profiles

Maven build profiles control which components are included:

Profile	Effect
-Phadoop2	Use Hadoop 2.x dependencies instead of default Hadoop 3.x
-Pspark-3.3 / -Pspark-3.4 / -Pspark-3.5	Build for specific Spark version (3.5 is default)
-Psupport-all-formats	Include Paimon and Hudi format modules in distribution
-Pflink-optimizer-pre-1.15	Build Flink optimizer for versions before 1.15

Docker Images

Three Docker images are provided:

1. apache/amoro: AMS service based on eclipse-temurin:11-jdk-jammy
2. apache/amoro-flink-optimizer: Flink-based optimizer from flink:${FLINK_VERSION}-java11
3. apache/amoro-spark-optimizer: Spark-based optimizer with custom base image

Build scripts: docker/build.sh1-243 GitHub Actions workflow: .github/workflows/docker-images.yml1-267

Sources: README.md117-135 docs/admin-guides/deployment.md68-294 docker/build.sh1-243 .github/workflows/docker-images.yml1-267 docker/amoro/Dockerfile1-64 docker/optimizer-flink/Dockerfile1-34

System Requirements

Amoro requires the following runtime dependencies:

• Java: Version 11 (required), 17 supported for Trino module
• Database: MySQL 5.5+, PostgreSQL 14+, or embedded Derby (optional)
• ZooKeeper: Version 3.4.x+ (optional, for HA deployments)
• Build Tools: Maven 3.9.11+ for compilation

Sources: docs/admin-guides/deployment.md31-35 pom.xml73-77

Building from Source

The project uses Maven for building with various profiles to control which components are included:

The build produces:

• dist/target/apache-amoro-x.y.z-bin.tar.gz - Complete distribution package
• Version-specific runtime JARs in each module's target/ directory
• Docker images via docker/build.sh script

Sources: README.md117-135 CONTRIBUTING.md86-88 docker/build.sh1-243