Home
Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework that:
- Has a built-in Web UI providing a convenient way to monitor and manage Karafka-based applications.
- Supports parallel processing in multiple threads (also for a single topic partition work) and processes.
- Automatically integrates with Ruby on Rails
- Has ActiveJob backend support (including ordered jobs)
- Has a seamless Dead Letter Queue functionality built-in
- Supports in-development code reloading
- Is powered by librdkafka (the Apache Kafka C/C++ client library)
- Has an out-of the box AppSignal and StatsD/DataDog monitoring with dashboard templates.
Basics¶
- Getting Started
- Code Editors and LLM Setup
- Configuration
- Producing Messages
- Consuming Messages
- Web UI
- Testing
- Components
- FAQ
- Support
- Status
Web UI¶
- About
- Getting Started
- Configuration
- Transactions
- Features
- Tagging
- Multi App Mode
- Single Process Setup
- Development vs Production
- Data Management
- Operational Cost Breakdown
- Components
WaterDrop¶
- About
- Getting Started
- Configuration
- Usage
- Error Handling
- Monitoring and Logging
- Transactions
- Testing
- Middleware
- Labeling
- Variants
- Custom Partitioners
- Idempotence and Acknowledgements
- Connection Management
- Connection Pool
- Async Ecosystem Integration
- Instrumentation Events
Operations¶
- Development vs Production
- Deployment
- Persistent Pausing
- Signals and States
- Monitoring and Logging
- Instrumentation Events
- Error Handling and back off policy
- AWS MSK Guide
Admin¶
Advanced¶
- Upgrading
- Routing
- Active Job
- Dead Letter Queue
- Declarative Topics
- Auto reload of code changes in development
- CLI
- Integrating with Ruby on Rails and other frameworks
- Concurrency and multithreading
- Deserialization
- Offset management (checkpointing)
- Pausing, Seeking and Rate-Limiting
- Inline Insights
- WaterDrop reconfiguration
- Exit codes
- Embedding
- Swarm / Multi Process
- Multi-Cluster Setup
- Env Variables
- Assignments Tracking
- Active Record Connections Management
- Forking
- Resources Management
- Latency and Throughput
- Broker Failures and Fault Tolerance
- Topic Auto Creation
- Articles and other references
- Versions Lifecycle and EOL
- Problems and Troubleshooting
- Debugging
- Software Bill of Materials (SBOM)
Karafka Pro¶
- Build vs. Buy
- Purchase Karafka Pro
- Getting Started
- Rotating credentials
- Pro FAQ
- Pro Support
- Commercial License
- Security
- Compliance Certifications
- HIPAA, PHI, PII Support
- FIPS Support
- Enterprise
- Enterprise Workshop Session
- Enterprise License Setup
Features and Enhancements¶
- Features List
- Features Compatibility
- Transactions
- Offset Metadata Storage
- Virtual Partitions
- Parallel Segments
- Delayed Topics
- Long-Running Jobs
- Non-Blocking Jobs
- Adaptive Iterator
- Periodic Jobs
- Expiring Messages
- Routing Patterns
- Rate Limiting
- Filtering API
- Scheduling API
- Iterator API
- Cleaner API
- Granular Backoffs
- Direct Assignments
- Multiplexing
- Piping
- Recurring Tasks
- Scheduled Messages
- Messages At Rest Encryption
- Enhanced Swarm / Multi Process
- Enhanced Dead Letter Queue
- Enhanced Active Job
- Enhanced Reliability
- Enhanced Inline Insights
- Enhanced Web UI
Librdkafka¶
Kafka¶
- Setting Up Kafka
- Best Practices
- Topic Configuration
- Cluster Configuration
- New Rebalance Protocol (KIP-848)
Upgrade Notes¶
It is recommended to do one major upgrade at a time.
