Senior Data Engineer

Core Technical & Functional Requirements 10+ years of experience in data engineering, data platform development, or related technical fields. 3–5 years of hands‑on experience building solutions with Databricks and Apache Spark . Strong command of SQL , including complex transformations, optimization, and analytical querying. Proven experience designing and maintaining ETL/ELT pipelines , data models, and performance‑tuned workflows. Proficiency in Python or Scala for data engineering and automation tasks. Ability to troubleshoot complex data issues and communicate clearly with both technical and non‑technical stakeholders. Role Overview We are looking for a highly skilled Databricks Data Engineer who can design, build, and optimize data pipelines and analytics workflows in a cloud environment. This role requires strong engineering fundamentals, hands‑on development experience, and the ability to operate in a fast‑moving environment. Experience with insurance data or Power BI is helpful but not required. Key Responsibilities Data Pipeline & Workflow Development Build and maintain scalable ETL/ELT pipelines using Spark , PySpark , and Databricks . Develop efficient SQL transformations to support analytics, reporting, and downstream applications. Partner with product owners, architects, and analysts to design data models, Delta Lake structures, and end‑to‑end workflows. Implement orchestration, scheduling, and monitoring for data jobs and pipelines. Ensure data quality, reliability, and performance across all stages of the pipeline. Develop and maintain CI/CD processes for Databricks notebooks, jobs, and workflows. Work within cloud‑based data platforms (Azure preferred; AWS/GCP acceptable). Work closely with business teams and data analysts to translate requirements into scalable engineering solutions. Support production environments, troubleshoot issues, and ensure high availability of data pipelines. Contribute to best practices around coding standards, documentation, and operational excellence. Required Skills & Experience 10+ years in data engineering or related technical roles. 3–5 years of hands‑on Databricks and Spark development. Strong SQL expertise and analytical problem‑solving skills. Experience with ETL/ELT design, data modeling, and performance tuning. Proficiency in Python or Scala for data engineering. Strong communication skills and the ability to work cross‑functionally. Nice to Have Familiarity with insurance industry data structures or workflows. Exposure to Power BI or similar BI tools. #J-18808-Ljbffr New York Technology Partners