Senior Data Engineer

Job Description

Hiring: Lead Data Engineer - Databricks

\n

\n

Important

\n

\n

Visa: USC and GC only

\n

Location: Manhattan, New York (Onsite)

\n

\n

JD:

\n

\n

Role Overview

\n

\n
• Client is seeking a Lead Data Engineer with deep hands-on expertise in Databricks, SQL, Python, and
\n
• PySpark to design, develop, and optimize scalable data engineering solutions.
\n
• This role offers an opportunity to work on enterprise-scale data platforms supporting analytics, digital
\n
• products, and emerging AI use cases. The candidate will drive Databricks engineering best practices,
\n
• solve challenging technical problems, and provide technical leadership across multiple initiatives.
\n

\n

\n

Key Responsibilities

\n

\n

\n
• Design and develop scalable data pipelines on Databricks, following Medallion Architecture.
\n
• Build and optimize data ingestion, transformation, and orchestration workflows for batch and
\n
• incremental processing.
\n
• Optimize data processing workloads, SQL queries, and compute configurations for performance
\n
• and cost efficiency.
\n
• Ensure data quality, observability, lineage, and reliability across production pipelines, including
\n
• troubleshooting complex issues.
\n
• Provide hands-on technical leadership through solution design, code reviews, reusable
\n
• engineering patterns, and mentoring other engineers.
\n

\n

\n

Required Qualifications

\n

\n

\n
• 6+ years of Data Engineering experience, with substantial hands-on experience delivering
\n
• production solutions on Databricks.
\n
• Strong proficiency in SQL, Python, and PySpark, including distributed data processing and
\n
• performance optimization.
\n
• Strong understanding of Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and
\n
• workflow orchestration.
\n
• Familiarity with Git, CI/CD, and production deployment practices.
\n
• Strong troubleshooting, problem-solving, and communication skills.
\n

\n

\n

Preferred Qualifications

\n

\n

\n
• Databricks Certified Data Engineer Associate or Professional certification.
\n
• Experience working with AWS cloud services, particularly S3 and cloud-based data integrations.
\n
• Experience with advanced ingestion patterns such as Auto Loader, Structured Streaming, or
\n
• Change Data Capture (CDC).
\n
• Familiarity with Databricks Asset Bundles or Apache Airflow.
\n
• Experience with large-scale media datasets: streaming, clickstream, or advertising data.
\n

\n

\n

Interested? Apply now or DM us to explore this opportunity! You can share resumes at View email address on ziprecruiter.com