Senior Data Platform Engineer

Data Platform Engineer

We are seeking a world-class Data Platform Engineer to take complete operational and architectural ownership of one of the largest Apache Druid deployments in the industry. Our environment operates at extreme velocity and petabyte scale, processing real-time streaming data via Kafka across ~2,000 parallel ingestion tasks. If you have pushed Druid, Zookeeper, and the Linux kernel to their absolute breaking points—and know exactly how to tune them to stay up—this role is built for you. In this role, you will bridge the gap between deep infrastructure engineering, Unix Linux systems administration, and distributed big data platforms. You will design, scale, and automate massive Middle Manager/Indexer fleets, optimize deep storage, eliminate metadata DB bottlenecks, and implement bulletproof query isolation. Utilizing Ansible and Infrastructure-as-Code methodologies, you will ensure our clusters are highly available, self-healing, and meticulously monitored. If you are a platform engineer who thrives on solving complex concurrency, JVM Garbage Collection, and high-throughput streaming challenges at scale, join us to redefine what is possible with real-time analytics.

Job Responsibilities:

• Design, deploy, administer, and optimize large-scale Apache Druid clusters supporting PB-scale datasets.
• Manage and troubleshoot Druid services.
• Perform cluster upgrades, patching, capacity planning, and platform modernization activities.
• Monitor cluster health, ingestion performance, query latency, segment distribution, and resource utilization.
• Troubleshoot ingestion failures, stuck tasks, compaction issues, retention policies, and segment management problems.
• Optimize Druid queries, partitioning strategies, indexing specifications, and compaction configurations.
• Configure and maintain Druid metadata stores using MySQL.
• Develop operational automation using Ansible and Infrastructure-as-Code methodologies.
• Create dashboards, alerts, and monitoring solutions using Grafana, Prometheus, and logging platforms.
• Work with platform teams to ensure high availability, disaster recovery, backup, and security compliance.
• Participate in production support, incident response, root cause analysis, and performance tuning initiatives.
• Collaborate with architects and business teams to understand analytical requirements and translate them into scalable solutions.