Data Engineer

div:has([data-free-thinking-preview-answer=true])+:is(.text-message,.relative:has(> .text-message))]:-mt-2 grow">

Job Summary
We are seeking an experienced Lead Data Engineer with approximately 70% data engineering and 30% analytical responsibilities. The role requires strong proficiency in modern cloud-native data platforms, large-scale data processing, and analytical and data science workflows. The ideal candidate will have deep Databricks and PySpark expertise, finance or payroll domain experience, and the analytical fluency to partner closely with data scientists, economists, and business stakeholders on macroeconomic and financial data research. This role will span production pipeline development, platform optimization, exploratory data analysis, and scalable data platform development for financial analytics and research.


Key Responsibilities
• Design, develop, and maintain scalable data pipelines for the ingestion, transformation, and distribution of payroll, macroeconomic, and financial datasets.
• Build and support Databricks-based platforms that enable financial research and analytical workloads.
• Implement ETL/ELT frameworks using PySpark and Delta Lake for structured and unstructured data from internal and external sources.
• Develop data models and data marts optimized for analytical, reporting, and machine learning use cases.
• Ensure data quality, consistency, lineage, governance, and observability across data assets.
• Optimize performance for large-scale datasets, including billions of records, multi-terabyte environments, and time-series data.
• Translate business requirements into scalable and maintainable data solutions.
• Perform exploratory data analysis, including profiling datasets and identifying distributions, outliers, missing patterns, and data drift.
• Translate data scientist logic into efficient and scalable PySpark implementations, including cross-sectional metrics and time-windowed aggregations.
• Build validation dashboards and exploratory notebooks to verify pipeline outputs and data quality.
• Support feature engineering by implementing complex aggregation and transformation logic at scale.
• Independently validate and sanity-check analytical outputs and identify results requiring further investigation.
• Perform ad hoc analytical work using pandas and NumPy alongside PySpark to support research initiatives.
• Contribute to architecture design discussions and evaluate tradeoffs related to catalog design, lakehouse architecture, medallion architecture, and data mesh concepts.
• Implement CI/CD pipelines using Databricks Asset Bundles, Bitbucket Pipelines, and Jenkins.
• Manage Unity Catalog governance, access patterns, and schema design.
• Maintain security, compliance, and data governance standards across data platforms.
• Leverage AI-assisted coding tools such as GitHub Copilot, Amazon Q, Kiro, or equivalent platforms to accelerate development.
• Review AI-generated code for correctness, performance, scalability, and maintainability.
• Integrate AI-assisted development workflows into engineering and analytical activities.


Required Qualifications
• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, Statistics, Economics, Finance, or a related field.
• 5+ years of experience in Data Engineering or Data Platform development.
• Finance or payroll domain experience, including familiarity with payroll data structures, pay-period logic, compensation and deduction relationships, or financial-services data environments.
• Experience handling large-scale datasets, including billions of records, multi-terabyte environments, and time-series data.
• Strong proficiency with Databricks, including Unity Catalog governance, access patterns, and catalog/schema design.
• Strong understanding of Delta Lake internals, including optimization, clustering, Change Data Feed, and versioning.
• Experience with Databricks Workflows, including orchestration, dependencies, and monitoring.
• Experience with Databricks Asset Bundles or equivalent deployment frameworks.
• Experience participating in architecture-level design decisions and evaluating technical tradeoffs.
• Strong proficiency in Python, SQL, PySpark, data modeling, and ETL/ELT development.
• Analytical fluency, including exploratory data analysis, basic statistical concepts, distributions, correlations, time-series patterns, and feature engineering.
• Proficiency with pandas and NumPy for ad hoc analytical work alongside production PySpark.
• Experience implementing CI/CD using tools such as Bitbucket Pipelines, Jenkins, and automated deployment frameworks.
• Experience with AI-assisted development tools such as GitHub Copilot, Amazon Q, Kiro, or equivalent platforms.
• Experience implementing data quality and validation frameworks.
• Strong analytical and problem-solving skills with statistical literacy.
• Ability to build reliable, scalable, and maintainable data pipelines with consideration for long-term ownership and support.
• Strong understanding of data governance, data quality, and metadata management.
• Excellent communication and documentation skills.
• Ability to work effectively in a fast-paced, data-driven environment.
• Ability to work across both engineering and analytical responsibilities, including production ETL development, data profiling, and dataset investigation.
• Ability to independently conduct exploratory analysis from an initial dataset through findings and recommendations.
• Ability to identify questionable data, investigate anomalies, and propose data-driven improvements.


Preferred Qualifications
• Experience with macroeconomic, capital markets, or financial-services data.
• Exposure to data architecture design, including lakehouse patterns, data mesh concepts, and medallion architecture.
• Experience with streaming and event-driven pipelines using Kafka or Structured Streaming.
• Census or geographic data processing experience, including TIGER and FIPS codes.
• Infrastructure-as-code experience with Terraform, CDK, or similar technologies.
• Experience migrating legacy data platforms to modern technology stacks, including Glue, EMR, or HDInsight to Databricks.
• Experience supporting machine learning and AI-driven analytics solutions.
• Data visualization experience using Power BI, Tableau, Databricks Dashboards, or similar platforms.
• Experience with Scala.
• Experience with SQL Server, PostgreSQL, Delta Tables, or NoSQL databases.
• Experience with Python visualization libraries, including Matplotlib and Plotly.
• Experience with Git and related DevOps and automation practices.
• Experience with payroll data structures and processing.
• Experience with macroeconomic analysis and forecasting.
• Experience with financial markets and alternative data.
• Experience with large-scale time-series data engineering.
• Experience with census and geographic data processing.
• Ability to use AI tools effectively to accelerate engineering and analytical workflows.
• Strong ownership of complex data initiatives and ability to collaborate with technical and business stakeholders.


Certifications
• Databricks certifications at the Associate or Professional level.