Big Data Platform Support Engineer

Big Data / Cloudera Lead Engineer

The Big Data / Cloudera Lead Engineer will lead the design, development, and support of large-scale data platforms, focusing on Cloudera, Hadoop, Kafka, and related technologies. This role involves ensuring high performance, scalability, and reliability of data pipelines and services while mentoring junior engineers and collaborating with cross-functional teams to optimize system performance and implement new solutions.

Responsibilities:

• Lead a team of software engineers in the design, development, and implementation of multiple applications.
• Perform detailed technical design, coding, code reviews, and testing of new and existing applications.
• Serve as a subject matter expert for specified systems and ensure compliance with DevSecOps practices and policies.
• Identify gaps in requirements and develop solutions to improve system performance and availability.
• Lead evaluations of risk, opportunities, and technology integration across projects.
• Support the Project Manager as the technical lead, estimating technical costs and defining system requirements.
• Participate in architectural planning, roadmap discussions, and transformation initiatives to new software architectures.
• Extend open-source libraries and systems to meet team requirements.
• Ensure adherence to code quality, deployment standards, and practices.
• Mentor junior developers and foster a culture of continuous improvement.

Experience:

• 8 years of software design, development, testing, and requirements management.
• Strong experience supporting Cloudera applications running in cloud environments.
• Hands-on experience with Cloudera Manager for monitoring and maintaining production clusters.
• Expertise in writing shell scripts for Hadoop daemon service health checks and automation tasks.
• Experience working with Hive, Impala, HBase, and SOLR, including query optimization and database management.
• Proficient in coordinating with application development, security, and platform teams for configuration and performance enhancements.

Skills Must Have:

• Cloudera administration and optimization (CDP, CDSW, Hive, Spark, NiFi).
• Cloud application support with MongoDB.
• Experience in cloud disaster recovery strategies.
• Strong Kafka administration and troubleshooting, including Streams API, integration with MQ, topic/offset management, and broker management.
• Apache NiFi administration, registry management, and integration with Kafka, HBase, and Solr.
• HBase administration, troubleshooting, and database management.
• Solr administration, managing shards, collections, logging levels, and query optimization.
• Flume integration with Kafka, NiFi, and IBM MQ.

Additional Skills:

• Perform critical data migrations from CDH to CDP.
• Design and implement big data pipelines and automated data flows using Python/R and NiFi.
• Perform incremental updates and upgrades to Cloudera environments.
• Manage job workflows using Oozie and Hue.
• Implement security policies across Hadoop clusters using Ranger.
• Configure and manage Cloudera Data Science Workbench (CDSW).
• Troubleshoot Kerberos, TLS/SSL, and model-related issues.
• Support Big Data/Hadoop databases through all development and production phases.
• Optimize database performance, troubleshoot replication, and manage connectivity/security issues.
• Create and implement automation scripts for maintenance and monitoring.
• Experience with Kafka ecosystem (Brokers, Connect, Zookeeper) in production environments.
• Support streaming technologies including Kafka, Spark, and Kudu.

Qualification And Education:

• Bachelor’s degree in Information Technology or Computer Science.
• In lieu of a degree, an additional 4 years of relevant work experience is required.