Senior Data Engineer

The Senior DataOps/Cloud Data Engineer is responsible for designing, developing, automating, and optimizing complex data pipelines and unified data platforms primarily using Databricks, Azure Data Factory, and cloud technologies. This role supports large-scale data migration, integration, and management across diverse environments, focusing on delivering scalable, secure, and efficient data solutions within an enterprise setting. The position requires strong expertise in ETL/ELT processes, advanced SQL, big data frameworks, and cloud data services, with a commitment to onsite work and collaboration within government-related projects. Responsibilities:

• Design, build, automate, and optimize complex data ETL/ELT pipelines using Databricks and Delta Live Tables (DLT).
• Develop and maintain unified data platforms leveraging Databricks and Unity Catalog or equivalent technologies.
• Create and manage Azure Data Factory pipelines for data ingestion, transformation, and movement.
• Perform data migration and integration across platforms such as Oracle, Azure Data Lake, Databricks, and Microsoft Fabric.
• Develop and implement data models (relational and dimensional) and optimize SQL queries for performance.
• Manage cloud data services including storage, data lakes, repositories, and virtual machines.
• Implement DataOps best practices, CI/CD pipelines, and automation for continuous integration and deployment.
• Monitor and tune DataOps performance and troubleshoot data pipeline issues.
• Collaborate with cross-functional teams to meet project deadlines and deliver high-quality data solutions.

• 8+ years of experience in data engineering with expertise in Databricks, Delta Live Tables, and Unity Catalog.
• Advanced SQL skills including PL/SQL and TSQL for data extraction, transformation, and optimization.
• Proficiency in Azure Data Factory pipeline creation and integration with CI/CD tools such as Azure DevOps Git.
• Experience with big data processing frameworks, specifically PySpark.
• Strong background in relational databases (Oracle, MySQL, SQL Server) and data modeling techniques.
• Expertise in data migration and integration across cloud and on-premise platforms.
• Knowledge of DataOps principles and Agile project methodologies.
• Ability to work onsite full-time during standard business hours.

• Experience designing and developing data lakes and lakehouses using medallion architecture.
• Hands-on experience with cloud platforms such as Azure, Google Cloud, and AWS.
• Proficiency in scripting languages including Python, Unix shell, and Scala.
• Advanced skills in Oracle database administration and Azure SQL Server management.
• Experience with ETL troubleshooting, performance tuning, failover, and recovery processes.
• Familiarity with digital product data analysis, data provisioning, and data security best practices.
• Strong interpersonal, communication, and presentation skills.

• Mandatory Skills and Experience - 70 Points
• Desirable Skills and Experience - 25 Points
• Education - 3 Points
• Public Sector Experience - 2 Points

• Fully automated and optimized ETL/ELT data pipelines using Databricks and Azure Data Factory.
• Unified data platform implementations with Databricks and Unity Catalog.
• Successful migration and integration of data across Oracle, Azure Data Lake, Databricks, and Microsoft Fabric.
• Documentation of data models, pipeline architecture, and operational procedures.
• Performance reports and tuning recommendations for DataOps processes.
• CI/CD pipelines for continuous deployment of data workflows.

• Expertise in Databricks, Delta Live Tables, and Unity Catalog development.
• Advanced SQL proficiency (PL/SQL, TSQL) and relational/dimensional data modeling experience.
• Proficient in Azure Data Factory pipeline creation and CI/CD integration.
• Strong experience with big data frameworks such as PySpark.
• Ability to work onsite five days a week during standard business hours.

Preferred