Data Engineer

Section

Details

Data Engineer

Full Time

Quincy,MA

Onsite

Full Time

$120-$135k/yr.

• We are seeking an experienced Azure Databricks Data Engineer with 8+ years of hands-on experience in data engineering, ETL/ELT, distributed data processing, and large-scale data pipeline development.
• The ideal candidate will have strong expertise in Databricks, Apache Spark, Scala, PySpark, SQL, Azure, and data lakehouse architectures, with proven experience designing and optimizing batch and streaming data solutions.
• The role involves developing scalable data ingestion, transformation, and processing pipelines using Azure Databricks, implementing Delta Lake and Medallion architectures, optimizing Spark workloads, and supporting high-volume streaming environments.
• The candidate will also work closely with DevOps teams to troubleshoot pipelines, Azure services, AKS environments, and production Databricks workloads.

• Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Databricks, Spark, Scala, PySpark, and SQL.
• Develop complex data ingestion, transformation, aggregation, and processing workflows for large-volume structured and unstructured datasets.
• Build and optimize Databricks notebooks, Spark jobs, DataFrames, Datasets, and SQL-based solutions.
• Implement Delta Lake, Unity Catalog, Auto Loader, Lakehouse, and Medallion architecture patterns.
• Develop real-time and near-real-time data pipelines using Spark Structured Streaming.
• Design streaming solutions capable of processing high-volume data with appropriate performance, scalability, fault tolerance, and data-quality controls.
• Configure and optimize Databricks clusters, including sizing, autoscaling, compute configuration, and workload optimization.
• Develop and manage Databricks job orchestration, scheduling, dependencies, retries, alerts, and production workflows.
• Perform Spark, Scala/PySpark, and SQL performance tuning, including optimization of transformations, joins, partitioning, caching, file formats, and query execution.
• Design and implement enterprise data lake and data warehouse/lakehouse solutions using Azure and Databricks.
• Develop data pipelines using Databricks Serverless capabilities where appropriate.
• Monitor, debug, troubleshoot, and resolve issues in live/production Databricks jobs and data pipelines.

• Must have atleast 8+ years of extensive Data engineering experience in Databricks, ETL/ELT using data pipelines, SQL/Procesurr experience
• Must have 8 years of hands on development proficiency in implementing Databricks solutions using scala and spark with data frames and notebooks/sql
• Must have 8+ years of hands on development and performance tuning/ enhancement experience in scala/pyspark and sql
• Must have strong experience with Databricks, including developing and optimizing spark jobs, data transformations, and data processing workflows
• Must have experience and a good understanding on the below topics
• Databricks delta lake storage
• Unity catalog
• Autoloader
• Lakehouse architecture
• Medallion architecture
• Databricks serverless options
• Dataset and dataframe
• Databricks clusters configuration and sizing
• Databricks job orchestration
• Spark structured streaming
• Must have experience with complex and large volume data streaming and data transformations using Azure Databricks
• Must have extensive experience in datawarehouse / data lakehouse implementation
• Must have experience with monitoring, debugging, and resolving issues in kive Databricks jobs
• Must have strong hands on expertise in troubleshooting devops pipelines, azure , and AKS services
• The candidate shd be able to provide data ingestion, streaming and transformations solutions and do development work
• Experience with apache big data implementation with programming proficiency like pyspark, R or Java prior to Databricks is advantageous
• Good to have experience with containerization technologies such as docker and Kubernetes