Senior Data Engineer

Role Description

We are looking for a Senior Data Engineer with strong hands-on experience building scalable data pipelines, optimizing performance, and enabling AI use cases. This role will design and implement large-scale data systems on Databricks and SQL Server , while also adopting AI tools and automation frameworks that accelerate development velocity and operational excellence.

You will help modernize our data ecosystem, contribute to writing efficient pipelines, and leverage AI technologies to improve productivity, reduce manual work, and enhance pipeline reliability.

Qualifications

• 5–7+ years of hands-on data engineering experience.
• Advanced proficiency in Python, SQL, PySpark, and ETL/ELT frameworks.
• Strong experience with Databricks and distributed data processing.
• Demonstrated ability to optimize SQL/Spark workloads for performance and cost.
• Experience using AI development tools such as: GitHub Copilot, Databricks Assistant, SQL/IDE AI copilots, AI-based data quality or monitoring tools.
• Solid understanding of data modeling, governance, and CI/CD for data pipelines.
• Excellent communication skills and ability to work in a cross-functional environment.
• Experience working in Agile environments with tools like Jira and Confluence.

Requirements

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, and SQL.
• Optimize data workflows and SQL/Spark queries for performance, cost efficiency, and reliability.
• Contribute to the design of our enterprise Data Lake and curated data assets.
• Build robust ingestion frameworks, transformation logic, and reusable data engineering components.
• Support data migrations and modernization efforts from legacy systems to cloud-based platforms.
• Use AI coding copilots (Databricks Assistant, GitHub Copilot, SQL copilots) to accelerate development, code review, test creation, and documentation.
• Automate repetitive tasks (schema evolution, config generation, logging templates, test data creation) with generative AI tools.
• Leverage AI to auto-detect pipeline anomalies, data drift, quality issues, and performance bottlenecks.
• Incorporate AI-assisted query optimization tools to improve Spark & SQL performance.
• Ability to learn and utilize LLM-powered tools to generate, refactor, and explain complex SQL, PySpark logic, and pipeline configurations.
• Work cross-functionally with Product, data consumers to deliver high-quality data solutions.
• Participate in Agile ceremonies & contribute to sprint planning & technical design discussions.
• Mentor junior data engineers, promote best practices in AI-assisted data engineering.
• Ensure strong documentation, standards adoption, & continuous improvement across the team.

Benefits

• Medical, Vision and Dental benefits for you and your family, including Flexible Spending Accounts (FSA) and Health Savings Accounts (HSAs).
• Generous paid time off policy including paid holidays, sick time and paid days off for your birthday, Free concert tickets.
• 401(k) program with company match, Stock Program.
• New parent programs & support including caregiver leave and childcare cash, infertility support.
• Tuition reimbursement, student loan repayment internal growth and development programs & trainings.
• Volunteer time off, crowdfunding network.