Data Management AI/Databricks Engineer (New York)

I have a full time opportunity working remotely that is looking for an Data Management AI/Databricks Engineer to join the team. Please see the job description below and feel free to reach out with any questions. Please send a word version of your resume for consideration.

This role is onsite 3 days a week in Midtown Manhattan.

It is $150-$160k, plus discretionary bonus.

As a Senior Data Engineer, you will own the design and delivery of complex data engineering solutions that power enterprise analytics, AI, and reporting capabilities. Reporting to the Lead Data Engineer, you will drive technical decisions, set engineering standards, and ensure the reliability and scalability of data platform across 40+ integrated enterprise source systems.

This role demands deep technical mastery in SQL, Python, and Azure cloud data engineering, combined with a product orientation—understanding how the data assets you build translate into decisions, reports, and AI outputs for the business. You will mentor Data Engineers, contribute to architectural direction, and serve as a technical anchor for delivery across the team's sprint cycles.

Advanced Pipeline Development & Ownership

• Architect, build, and own complex data pipelines for high-volume, high criticality workstreams across enterprise data platform.
• Lead the design and implementation of ELT/ETL frameworks using SQL, Python, Azure Data Factory, Databricks, and Azure Synapse Analytics.
• Establish pipeline reliability standards—monitoring, alerting, error handling, and recovery protocols—and ensure adherence across the team.

Data Architecture & Platform Evolution

• Drive the design of scalable data models supporting dimensional warehousing, data lake architectures on Azure.
• Contribute to architectural decisions on data storage, partitioning, compute optimization, and consumption layer design.
• Lead migrations from legacy data solutions to modern cloud-native platforms, managing risk and business continuity throughout.

AI & Analytics Enablement

• Design and deliver feature pipelines and data preparation frameworks that support machine learning model development and deployment.
• Partner with Data Scientists to translate model requirements into production-grade data assets and feature stores.
• Collaborate with Analytics Engineers to ensure data models are optimized for analytical consumption and reporting performance.

Data Quality & Governance Leadership

• Define and implement data quality frameworks—validation rules, SLAs, anomaly detection, and automated testing for pipeline outputs.
• Lead data governance initiatives including metadata management, lineage tracking, data cataloging (Microsoft Purview), and access control.
• Ensure platform compliance with HIPAA, data policies, and applicable regulatory requirements.

Mentorship & Technical Leadership

• Mentor Data Engineers—providing code reviews, technical guidance, and architectural feedback that elevates team capability.
• Contribute to engineering standards, reusable frameworks, and technical documentation.
• Participate in Agile ceremonies and model strong engineering discipline—clear DevOps hygiene, sprint commitment, and delivery accountability.

Education:

• Bachelor's degree in computer science, Data Science, Information Technology, or a related quantitative field.

Experience:

• 6+ years of progressive experience in data engineering with demonstrated ownership of complex, production-grade data platforms.
• Expert-level SQL (query optimization, indexing strategy, execution plans) and Python (PySpark, pipeline frameworks, testing).
• Deep hands-on experience with Azure data services: Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage.
• Proven experience designing dimensional data models and data lake architecture at enterprise scale.
• Experience building data pipelines that directly support machine learning feature engineering and model serving.
• Strong background in data quality engineering—automated validation, SLA enforcement, and lineage tracking.
• Experience with relational databases (SQL Server, Oracle) and migration from legacy to cloud-native platforms.

Certifications & Licenses:

The following are considered favorable:

• Microsoft Certified: Azure Data Engineer Associate
• Databricks Certified Associate Developer for Apache Spark

Knowledge:

• Advanced SQL and Python for enterprise-scale data engineering—optimization, testing, and framework design.
• Azure data platform architecture in depth—ADF, Databricks, Synapse, ADLS, and their integration patterns.
• Modern data platform paradigms—data lake, medallion architecture, data mesh concepts, and consumption layer design.
• Machine learning pipeline requirements—feature engineering, training data preparation, and model data dependencies.
• Data governance frameworks—metadata management, lineage, cataloging, access control, and regulatory compliance (HIPAA).
• Agile engineering practices—sprint delivery, DevOps hygiene, CI/CD for data pipelines, and technical documentation standards.

Skills:

• Cultural competency and the ability to communicate effectively in a culturally sensitive manner with both individuals and groups from diverse backgrounds.
• Architect and deliver complex, production-grade data pipelines that meet enterprise reliability and performance standards.
• Design scalable data models and platform structures that serve analytics, reporting, and AI consumption patterns simultaneously.
• Lead data quality engineering—automated testing, validation frameworks, SLA monitoring, and incident response.
• Mentor and elevate Data Engineers through code review, architectural feedback, and knowledge transfer.
• Translate product and analytical requirements into sound engineering designs and delivery plans.

Abilities:

• Communicate complex data concepts clearly to both technical and non-technical stakeholders.
• Work independently and manage competing priorities in a lean, fast-paced team environment.
• Embrace accountability and take ownership of deliverables end-to-end.
• Incorporate feedback constructively and seek continuous improvement.

Lori Sklarski

Senior Technical Recruiter

PRI Technology

P: View phone number on apply.placed-app.com

C: View phone number on apply.placed-app.com

F: View phone number on apply.placed-app.com

View email address on apply.placed-app.com