Kafka Cheatsheet

Kafka Fail Reason 1: Lack of Expertise and Resources

• scaling and implementing Kafka solutions requires a high degree of expertise
• installation, development, operations
• underprovisioned configuration results in high risk of system failures, but overprovisioned configuration results in wasted resources
• once in production, ongoing operations and maintenance are essential for a Kafka project's success
  • effective monitoring practices is necesary to capture important metrics such as message throughput, latency, and resource utilization

Kafka Fail Reason 2: Difficulty Moving from Dev to Prod

• configuration params: buffer sizes, replication factors, retention policies, security settings
  • optimized configuration requires a balance between scalability, performance, and security
• production should be designed for HA and FT (fault tolerance)
  • replication across multiple brokers
  • disaster recovery mechanism
    • leader election, data consistency across replicas
    • 3rd party data replication tools: Apache Kafka MirrorMaker 2.0

Kafka Fail Reason 3: Unpredictable Outages and Downtime