Senior Data Engineer

• We are seeking a highly skilled and experienced Big Data Engineer to design, develop, and optimize large-scale data processing systems. In this role, you will work closely with cross-functional teams to architect data pipelines, implement data integration solutions, and ensure the performance, scalability, and reliability of Big Data platforms.
• The ideal candidate will have deep expertise in distributed systems, cloud platforms, and modern Big Data technologies such as Hadoop, Spark etc.
• Demonstrated technical expertise in Object Oriented and database technologies/concepts which resulted in deployment of enterprise quality solutions.

Responsibilities:

• Design, develop, and maintain large-scale data processing pipelines using Big Data technologies (e.g., Hadoop, Spark, Python, Scala).
• Design, develop, test, implement, and support technical solutions within an Agile development environment, utilizing distributed microservices architecture, big-data technologies (HIVE, Presto, Spark, Hadoop, Trino), and full-stack systems.
• Partner with cross-functional teams to understand application requirements and testing scenarios, delivering scalable technical solutions that meet business objectives.
• Build and maintain high-performance API platforms on AWS infrastructure, enabling interactive-speed access to big data while adhering to AWS Well-Architected Framework principles.
• Leverage AI tools and technologies to enhance development efficiency, code quality, and solution innovation throughout the software development lifecycle.
• Drive technical innovation by staying current with emerging technology trends, including AI/ML advancements, evaluating new tools and frameworks, and actively contributing to internal and external technology communities.
• Demonstrate a growth mindset, take ownership of deliverables, maintain customer-centric focus, and exercise sound technical judgment in decision-making.

Must have skills:

• Python and/or Scala
• Spark, Big Data (we work with terabytes per day)
• AWS
• SQL
• Orchestration, deployment understanding, Kubernetes (troubleshooting, configuring for scale), can be AWS, GCP, or Azure Kubernetes. Should have maintained existing Kubernetes infrastructure
• Knowledge on GenAI for use in engineering (ex: Kiro), LLMs, etc.

Qualifications:

• 8+ years of professional software engineering experience with demonstrated leadership capabilities.
• 3+ years of hands-on development experience in Data Engineering, Python / Scala / Java, and SQL.
• 3+ years working with Big Data technologies such as Hadoop, Spark, or Presto, etc.
• 2+ years developing REST/web services and writing complex SQL queries for relational databases.
• Experience with AWS services including EMR (Hive/Presto), Glue, Athena, ECS, Lambda, and S3.
• Proficiency with DevOps practices and CI/CD pipelines using tools like Jenkins, Cloud Formation, etc.
• Working knowledge of scripting languages.
• Bachelor's degree in computer science, information technology, engineering, or related technical field.

LI-CGTS

TS-3142