Kafka Administrator

We are seeking an experienced Kafka Administrator with strong expertise in Apache Kafka, Confluent technologies, AWS MSK, and large-scale production environments. The ideal candidate will have a background in Site Reliability Engineering (SRE) or DevOps, with hands-on experience managing highly available Kafka clusters, performing major cluster rebuilds and hardware refreshes, and executing migrations with minimal downtime.

Key Responsibilities

• Design, deploy, configure, and maintain highly available Apache Kafka clusters across on-premises, cloud, and managed environments, including AWS MSK and Confluent Cloud.

• Administer Kafka topics, partitions, quotas, retention policies, replication settings, and consumer groups to ensure optimal performance and cost efficiency.

• Manage Kafka ecosystem components, including Kafka Connect, Schema Registry, MirrorMaker, Confluent Replicator, and REST Proxy.

• Execute Kafka upgrades, patching, hardware refreshes, cluster rebuilds, and migrations while minimizing downtime and operational risks.

• Perform Kafka performance tuning, throughput testing, and benchmarking across producers, consumers, brokers, and controllers.

• Optimize batching, compression, acknowledgments ( acks ), in-sync replicas (ISR), and cluster health.

• Develop capacity plans, partitioning strategies, infrastructure sizing models, and cluster right-sizing recommendations.

• Integrate Apache Kafka with AWS MSK and other enterprise systems.

• Implement monitoring, alerting, troubleshooting, backup and recovery, and disaster recovery procedures.

• Apply strong Linux and networking fundamentals, including TCP/IP, DNS, and load balancing, to troubleshoot infrastructure and connectivity issues.

• Develop architecture diagrams, operational standards, runbooks, and migration documentation.

• Conduct knowledge transfer sessions and ensure smooth documentation handoff at key milestones and project completion.

Required Skills and Qualifications

• 8–10 years of overall relevant IT experience.

• 4+ years of hands-on experience operating Kafka in production at scale within an SRE or DevOps environment.

• Strong experience with Apache Kafka administration and production operations.

• Hands-on experience with AWS MSK operations, configuration, and integration.

• Practical experience with Kafka Connect, Schema Registry, MirrorMaker, and Confluent Replicator.

• Strong Linux administration and networking fundamentals, including TCP/IP, DNS, and load balancing.

• Proven experience executing hardware refreshes, major cluster rebuilds, and migrations with minimal downtime.

• Experience with Kafka performance tuning, capacity planning, cluster sizing, and troubleshooting.

• Knowledge of high availability, replication, disaster recovery, monitoring, and operational best practices.

• Ability to create technical documentation, migration plans, architecture diagrams, and operational runbooks.

• Strong communication and collaboration skills.