Spark/Scala Engineer

Description

Responsible for developing and operating big data platform using open source or other solutions to aid critical applications, such as analytics, reporting, and AI/ML apps. This includes working to optimize performance and cost, automate operations, and identifying and resolving production errors and issues to ensure the best data platform experience.

Responsibilitie

• sDevelop and operate large-scale big data platforms using open source and other solutions
• .Support critical applications including analytics, reporting, and AI/ML apps
• .Optimize platform performance and cost efficiency
• .Automate operational tasks for big data systems
• .Identify and resolve production errors and issues to ensure platform reliability and user experienc

e
Minimum Qualification

• s:3+ years of professional software engineering experience with large-scale big data platforms, including strong programming skills in Java , Sca la, Python, or G
• o.Proven expertise in designing, building, and operating large-scale distributed data processing systems with a strong focus on Apache Spar
• k.Experience with contribution t o Open-Source projects is a plu
• s.Hands-on experience with table formats and data lake technologies such as Apache Iceberg, ensuring scalability, reliability, and optimized query performanc
• e.Skilled at coding for distributed systems and developing resilient data pipeline
• s.Strong background in incident management, including troubleshooting, root cause analysis, and performance optimization in complex production environment
• s.Proficient with Unix/Linux systems and command-line tools for debugging and operational suppor

t.
Preferred Qualificatio

• ns:Expertise in designing, building, and operating critical, large-scale distributed systems with a focus on low latency, fault-tolerance, and high availabili
• ty.Experience with multiple public cloud infrastructure, managing multi-tenant Kubernetes clusters at scale and debugging Kubernetes/Spark issu
• es.Experience with workflow and data pipeline orchestration tools (e.g., Airflow, DB
• T).Understanding of data modeling and data warehousing concep
• ts.Familiarity with the AI/ML stack, including GPUs, MLFlow, or Large Language Models (LLM
• s).A learning attitude to continuously improve the self, team, and the organizati
• on.Solid understanding of software engineering best practices, including the full development lifecycle, secure coding, and experience building reusable frameworks or librari