Big Data Engineer

We are seeking an experienced Kafka/Spark Data Engineer with strong expertise in real-time data processing, streaming technologies, and distributed systems. The role will focus on Kafka, Spark, Redis, Python, and cloud-based Big Data solutions.

Roles and Responsibilities

• Design, develop, and support real-time data streaming and Big Data solutions.
• Develop streaming applications using Apache Kafka, KStreams, and KTables.
• Work with Apache Spark for large-scale data processing.
• Develop applications using Python, Scala, Java, and Bash/Shell scripting.
• Work with NoSQL databases such as Redis, MongoDB, and HBase.
• Design scalable, distributed, and fault-tolerant data architectures.
• Implement data integration, security, and authentication solutions, including Kerberos.
• Work with streaming technologies such as Flink or Storm.
• Support cloud-based data solutions across AWS, Azure, or GCP.
• Apply data analysis, statistical methods, and machine learning techniques where required.
• Create and support data visualizations using Tableau.
• Troubleshoot business-critical applications and provide effective technical solutions.

Required Qualifications

• 9+ years of overall IT/industry experience, with strong Big Data and real-time processing experience.
• Bachelor's or Master's degree in Computer Science, Engineering, Science, or related field.
• Strong hands-on experience with Kafka and real-time streaming.
• Experience with Spark, Redis, and Python.
• Strong understanding of distributed systems, data partitioning, and fault-tolerant architectures.
• Experience with NoSQL databases and cloud platforms.
• Strong problem-solving, analytical, and communication skills.

Preferred Qualifications

• Experience with Flink, Storm, Scala, or Java.
• Knowledge of machine learning and predictive analytics.
• Experience with Tableau and real-time data visualization.