Home
Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework that:
- Has a built-in Web UI providing a convenient way to monitor and manage Karafka-based applications.
- Supports parallel processing in multiple threads (also for a single topic partition work) and processes.
- Automatically integrates with Ruby on Rails
- Has ActiveJob backend support (including ordered jobs)
- Has a seamless Dead Letter Queue functionality built-in
- Supports in-development code reloading
- Is powered by librdkafka (the Apache Kafka C/C++ client library)
- Has an out-of the box AppSignal and StatsD/DataDog monitoring with dashboard templates.
Basics¶
- Why Kafka and Karafka
- Getting Started
- Code Editors and LLM Setup
- Configuration
- Producing Messages
- Consuming Messages
- Consumer Groups vs Share Groups
- Web UI
- Testing
- Components
- FAQ
- Support
- Status
- Articles and Other References
Web UI¶
- About
- Getting Started
- Configuration
- Transactions
- Features
- Tagging
- Multi App Mode
- Single Process Setup
- Development vs Production
- Data Management
- Operational Cost Breakdown
- Components
WaterDrop¶
- About
- Getting Started
- Configuration
- Usage
- Error Handling
- Monitoring and Logging
- Transactions
- Testing
- Middleware
- Labeling
- Variants
- Custom Partitioners
- Idempotence and Acknowledgements
- Connection Management
- Connection Pool
- Async Ecosystem Integration
- Instrumentation Events
- Reconfiguration
Consumer Groups¶
- Routing
- Consuming Messages
- Concurrency and Multithreading
- Deserialization
- Offset Management (Checkpointing)
- Pausing, Seeking and Rate-Limiting
- Dead Letter Queue
- Active Job
- Inline Insights
- Assignments Tracking
- Error Handling and Back Off Policy
- Persistent Pausing
Share Groups¶
Infrastructure¶
Operations¶
- Development vs Production
- Deployment
- Signals and States
- Monitoring and Logging
- Instrumentation Events
- CLI
- Exit codes
Configuration¶
- Integrating with Ruby on Rails and other frameworks
- Declarative Topics
- Env Variables
- Multi-Cluster Setup
- Active Record Connections Management
- Embedding
- Swarm / Multi Process
- Forking
- Auto reload of code changes in development
- Admin API
- ACLs API
- Configs API
- Replication API
- Recovery API
Reliability¶
Troubleshooting¶
Karafka Pro¶
- Build vs. Buy
- Purchase Karafka Pro
- Getting Started
- Rotating credentials
- Pro FAQ
- Pro Support
- Commercial License
- Security
- Compliance Certifications
- HIPAA, PHI, PII Support
- FIPS Support
- Enterprise
- Enterprise Architecture Consultation
- Enterprise Workshop Session
- Enterprise License Setup
Features and Enhancements¶
- Features List
- Features Compatibility
- Routing Patterns
- Cleaner API
- Recurring Tasks
- Scheduled Messages
- Messages At Rest Encryption
- Enhanced Swarm / Multi Process
Consumer Groups Features¶
- Transactions
- Offset Metadata Storage
- Virtual Partitions
- Parallel Segments
- Delayed Topics
- Long-Running Jobs
- Non-Blocking Jobs
- Adaptive Iterator
- Scheduling API
- Iterator API
- Granular Backoffs
- Direct Assignments
- Multiplexing
- Enhanced Dead Letter Queue
- Enhanced Active Job
- Enhanced Reliability
- Enhanced Inline Insights
- Periodic Jobs
- Expiring Messages
- Rate Limiting
- Filtering API
- Piping
Share Groups Features¶
Enhanced Web UI¶
- About
- Getting Started
- Consumers
- Commanding
- Health
- Explorer
- Policies
- Search
- Recurring Tasks
- Scheduled Messages
- Topics Insights
- Errors
- DLQ / Dead
- Branding
- Custom Styling
- Topics Management
Librdkafka¶
Kafka¶
- Kafka Setup
- Best Practices
- Topic Configuration
- Cluster Configuration
- New Rebalance Protocol (KIP-848)
Upgrade Notes¶
It is recommended to do one major upgrade at a time.
