Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Introduction to e6 Ingestion Engine

e6 Ingestion Engine is a distributed streaming ingestion engine by e6data. It connects data sources to data sinks through real-time SQL pipelines, running natively on Kubernetes.

What e6 Ingestion Engine Does

e6 Ingestion Engine continuously ingests data from sources like Kafka, databases (via CDC), HTTP endpoints, and cloud storage - transforms it with SQL - and writes it to sinks like Apache Iceberg, Delta Lake, and Kafka topics. All processing happens in real time with exactly-once delivery guarantees.

Image
Introduction Diagram

Key Capabilities

SQL-native stream processing - Write standard SQL (based on Apache DataFusion) extended with streaming constructs like tumbling windows, session windows, event-time joins, and CDC-style update streams.

Exactly-once semantics - Asynchronous barrier snapshots with two-phase commit ensure every record is processed exactly once, even across failures and restarts.

Event-time processing - All operations use event time (embedded in the data), not wall-clock time. This makes results deterministic and correct regardless of when data arrives.

Stateful operations - Windowed aggregations, stream-stream joins, and lookup joins with automatic state management and fault-tolerant checkpointing.

Kubernetes-native - Deployed via Helm with Custom Resource Definitions (CRDs). Define tables, pipelines, and connection profiles as Kubernetes resources.

Architecture

e6 Ingestion Engine has two main components:

  • Controller - The brain of the system. Compiles SQL into distributed execution plans, schedules work across nodes, and coordinates checkpoints.

  • Workers - Stateless processes that execute the dataflow. Workers can be killed and restarted at any time - all state lives in checkpoint storage (S3, GCS, or Azure Blob).

Pipelines are compiled into distributed DAGs (directed acyclic graphs) where each node runs as a parallel subtask across workers. See the Architecture deep-dive for full details.

SQL Support

e6 Ingestion Engine's SQL dialect is based on Apache DataFusion and extended for streaming. See the SQL overview for usage guidance and the full reference.

  • Windows - Tumbling, sliding (hopping), and session windows for time-bucketed aggregations

  • Joins - Windowed joins, updating joins, and lookup joins against external systems

  • Functions - Math, string, JSON, datetime, array, struct, regex, hashing, aggregate, and window functions

  • DDL - CREATE TABLE with connector configuration, virtual fields, schema inference

  • CDC - Read and write Debezium-formatted change streams for updating semantics

Continue

Continue with Choose Your Pipeline to select the source, destination, and guide for your use case.

NextChoose your pipeline

Last updated

Was this helpful?