Data Engineering (Palantir, Spark, PySpark, Python)- W2 Role

Job Description

Job Description

Job Title: Technical Lead – Data Engineering (Palantir, Spark, PySpark, Python)

Please dont apply if you dont have PALANTIR experience

Onsite work :One or two days in a month
W2 role
OPT with 7 years exp is fine

Job Summary

We are looking for a hands-on Technical Lead in Data Engineering to drive the design, development, and delivery of scalable data solutions in a large retail enterprise. This role requires strong expertise in Palantir Foundry, Spark/PySpark, SQL, and Python, along with the ability to lead engineering teams and partner with business stakeholders across supply chain, merchandising, and store operations.

Key Responsibilities

• Lead the design and implementation of scalable data platforms and pipelines using PySpark, Spark, and Python
• Drive adoption and best practices for Palantir Foundry (data pipelines, ontology, workflows, and operational applications) 
• Architect and optimize high-performance data processing solutions for large-scale datasets 
• Provide technical leadership and mentorship to data engineers, ensuring code quality and best practices 
• Collaborate with cross-functional teams (business, analytics, data science) to translate requirements into scalable solutions 
• Design robust data models, ETL/ELT frameworks , and data integration strategies 
• Ensure data quality, governance, security, and compliance across enterprise data platforms 
• Lead performance tuning, troubleshooting, and optimization of data pipelines 
• Drive CI/CD implementation , code reviews, and release management 
• Stay current with emerging data engineering technologies and recommend improvements 

Required Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field 
• 8+ years of experience in data engineering , with at least 2–3 years in a technical leadership role
• Strong hands-on experience with:
  • Python & PySpark
  • Apache Spark
  • Advanced SQL
• Experience with Palantir Foundry or similar modern data platforms 
• Deep understanding of data engineering principles (ETL/ELT, data modeling, distributed systems) 
• Experience designing and managing large-scale data architectures
• Strong leadership, communication, and stakeholder management skills 

Preferred Qualifications

• Experience in retail domain (supply chain, inventory, merchandising, store analytics) 
• Experience with cloud platforms (Azure, AWS, or GCP) 
• Familiarity with orchestration tools (Airflow, Azure Data Factory, etc.) 
• Experience with real-time/streaming data pipelines
• Exposure to DevSecOps practices 

Key Skills

• Technical Leadership 
• Data Engineering Architecture 
• PySpark & Spark 
• Python Programming 
• SQL Optimization 
• Palantir Foundry 
• Data Modeling & Warehousing 
• CI/CD & DevOps

Flexible work from home options available.