Lead Data Engineer (DataOps) | Azure, Databricks, CI/CD

Job Title: DataOps Team Lead

Location: Short Hills, NJ / Hartford CT / Charlotte, NC (Hybrid working - 3 days a week)

Perm/FTE Role

US citizens and Green Card Holders and those authorized to work in the US are encouraged to apply. We are unable to sponsor  H1b candidates at this time

Job Details:

• Seeking an energetic and collaborative DataOps Team Lead responsible for establishing, scaling, and operating modern data pipelines and platforms that enable reliable, secure, and high-quality analytics across the enterprise.
• This role bridges data engineering, analytics, platform engineering, and governance to ensure data products are delivered efficiently, consistently, and at scale.
• This is a key leadership position in the next phase of the organization’s data and analytics evolution, with responsibility for shaping how data engineering work is delivered, operated, and governed across teams.
• The ideal candidate brings strong hands-on experience with cloud data platforms, CI/CD for data, operational excellence, leading teams, setting operational standards, and influencing how data engineering is practiced across the organization.

Responsibilities

• Define and evolve the DataOps operating model, including how data pipelines and platforms are designed, deployed, supported and scaled across teams.
• Lead the design and implementation of DataOps frameworks, standards, and best practices across the organization.
• Partner with Data Architecture, Analytics, AI/ML, and Platform teams to operationalize data products end-to-end.
• Establish operational expectations for data products/data contracts, including reliability targets, change management, and consistency of consumption for analytics and AI use cases.
• Drive adoption of automation, orchestration, and observability capabilities for data pipelines.
• Implement and manage CI/CD for data, including automated testing, deployments, and rollback strategies.
• Ensure high availability, performance, and reliability of data platforms and analytics workloads.
• Establish monitoring, alerting, and incident management for data systems, including root-cause analysis and post-incident improvements.
• Embed governance requirements into data operations, including data quality controls, lineage visibility, and access enforcement within pipelines.
• Monitor and optimize cloud consumption and platform costs for data workloads.
• Drive performance tuning and capacity planning for data platforms.
• Own, report and provide transparency into platform health, costs, and operational metrics.
• Lead, mentor, and grow high-performing DataOps and data engineering team members.
• Serve as a thought leader on modern data engineering and operational practices.

Qualifications Required

• Proven history of leading others to complete complex deliverables.
• Demonstrated ability to provide technical leadership through design reviews,
• architectural decisions, and operational trade-offs.
• Excellent communication skills
• 8+ years of experience in data engineering, analytics engineering, or platform engineering.
• 3+ years in a lead or senior engineering role managing data platforms or operations.
• Strong experience with cloud data platforms (Azure, AWS, or GCP).
• Experience with Databricks, or modern Lakehouse architectures.

Hands-on expertise with:

• Data pipelines and orchestration (e.g., Airflow, ADF, Dagster, dbt)
• CI/CD tools (Azure DevOps, GitHub Actions, GitLab, etc.)
• SQL and Python/PySpark
• Solid understanding of data modeling, data quality, and data lifecycle management.

Preferred

• College degree in Computer Science, Management Information systems or related
• Familiarity with streaming platforms (Event Hubs, Kafka, Kinesis).
• Experience supporting machine learning or AI workloads in production.
• Knowledge of FinOps practices for cloud cost management.
• Background in regulated or enterprise environments (financial services, insurance, healthcare).
• SAFe agile work experience or similar agile process

A reasonable, good faith estimate of the minimum and maximum base salary for this position is $140k to $160K . This position will also include a discretionary bonus or 10% that generally will be depending on a variety of factors.  Employee benefits  will also be available, and details are available like Pension/401K/ Paid Vacation/ Life, Medical & Dental insurance etc.

* The pay range listed above reflects the expected starting salary /Pay rate for this role. This range may be adjusted based on market conditions, location, and other relevant factors. The Company will determine the final starting salary/Pay rate in consultation with the selected candidate(s), in full compliance with applicable laws