Lead Data Engineer

Location: NY / NJ / Pittsburgh Onsite

Employment Type: Full-Time

Experience: 8+ Years

Job Summary

We are looking for a hands-on Technology Engineering Lead to design and deliver enterprise-scale data engineering solutions, including a batch feature store supporting analytics and AI use cases.

This is a technical leadership + individual contributor role . The ideal candidate must be comfortable working directly with clients while remaining deeply hands-on with PySpark, SQL, database architecture, and ETL/ELT pipeline development .

Key Responsibilities

• Lead architecture and development of enterprise-scale data pipelines.

• Design and build scalable PySpark and Impala SQL solutions.

• Develop ETL/ELT pipelines across Hive and Oracle data platforms.

• Own data architecture, feature logic, data quality, and reconciliation.

• Drive pipeline performance tuning, monitoring, and failure recovery.

• Manage batch scheduling and dependencies using CA7 or equivalent .

• Conduct design reviews, code reviews, and hands-on troubleshooting.

• Establish Git/Bitbucket, CI/CD, and testing practices.

• Lead production releases and SDLC/change-management activities.

• Work directly with business, analytics, and technology stakeholders.

• Mentor engineers while remaining actively involved in technical delivery.

Required Qualifications

• 8+ years of Data Engineering / Data Platform experience.

• 5+ years of hands-on PySpark and enterprise ETL/ELT development .

• Advanced SQL and strong pipeline performance-tuning skills.

• Strong database/data-platform architecture experience.

• Hands-on experience with Hive and Oracle .

• Technical leadership experience across complex data-engineering programs.

• Experience with Linux, Git/Bitbucket, and CI/CD.

• Strong client-facing communication skills.

• Ability to work effectively in a fast-paced environment.

Preferred Skills

• Banking / Financial Services experience.

• CA7 or equivalent batch scheduling tools.

• Jira and Agile/Scrum experience.

• Experience with AI-assisted development tools.

Primary Skills

PySpark, SQL, ETL/ELT, Data Engineering, Data Architecture, Hive, Oracle, Impala SQL, Database Architecture, Data Pipelines, Performance Tuning, CI/CD, Git/Bitbucket, Linux, CA7