Senior Engineer

Software Engineer (Spark / SparkSQL / Python / Scala)

We are seeking a highly skilled Software Engineer with strong expertise in Apache Spark, SparkSQL, Python, Scala, Hive, and Parquet to build and maintain large-scale data processing solutions. The ideal candidate will have experience developing distributed data applications, optimizing big data workloads, and building reliable data pipelines that support analytics, reporting, and machine learning initiatives.

Key Responsibilities

• Design, develop, and maintain scalable data processing applications using Apache Spark and SparkSQL.
• Build high-performance ETL and data transformation pipelines using Python and Scala.
• Develop and optimize batch and near-real-time data processing workflows.
• Work with large-scale datasets stored in Parquet, Hive, and distributed storage systems.
• Write efficient SparkSQL queries and optimize execution plans for performance and scalability.
• Design data models, partitioning strategies, and storage layouts to improve processing efficiency.
• Collaborate with Data Engineers, Software Engineers, Data Scientists, and Product Teams to deliver data-driven solutions.
• Troubleshoot performance bottlenecks and optimize resource utilization across distributed computing environments.
• Implement automated testing, monitoring, and deployment processes for data applications.
• Ensure data quality, reliability, security, and governance standards are met.
• Participate in code reviews and promote engineering best practices.

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related technical field.
• 5+ years of professional software engineering or data engineering experience.
• Strong hands-on experience with: Apache Spark/SparkSQL/Python/Scala
• Experience building distributed data processing applications.
• Strong understanding of Hadoop ecosystem technologies, including: Hive/Parquet/HDFS
• Experience developing and optimizing ETL pipelines processing large-scale datasets.
• Strong SQL programming and query optimization skills.
• Familiarity with Linux environments and shell scripting.
• Experience using version control systems such as Git.
• Strong problem-solving and debugging abilities.

Preferred Qualifications

• Experience with cloud platforms such as AWS
• Experience with distributed computing and large-scale data architectures.
• Familiarity with Airflow orchestration framework
• Experience working with data lakes and Lakehouse architectures.
• Knowledge of data formats and storage technologies: Iceberg / ORC / Avro
• Experience with containerization technologies such as Docker and Kubernetes.

Mandatory Skills

• Apache Spark
• Spark SQL
• Python
• Scala
• Hive
• Parquet