Sr Data Engineer

Job Title: Sr Data Engineer

Location: Remote

Job type: Contract

Relevant Experience: 10+ Years

Technical / Functional Skills

Overall Experience:

• 8 12 years of overall data engineering experience
• At least 3 years of substantial hands-on Databricks development

Required Skills:

• Azure Databricks Strong hands-on experience building production-grade batch pipelines and lakehouse solutions.
• PySpark / Spark SQL Advanced transformations, joins, window functions, reusable frameworks, debugging, and performance tuning.
• Delta Lake Table design, schema evolution, merge/upsert patterns, optimization, data reliability, and history management.
• Medallion Architecture Practical delivery experience across source-aligned, conformed, and consumption-ready layers.
• Data Modeling Fact/dimension design, conformed dimensions, slowly changing dimensions, aggregates, and semantic-consumption considerations.
• Data Quality & Testing Profiling, rule implementation, reconciliation, unit/integration testing, controls, and defect analysis.
• Azure Ecosystem Working knowledge of Azure Data Lake [remaining text is cut off in the screenshot] .
• DevOps Git-based development, code reviews, CI/CD, environment promotion, release controls, and automated deployment.
• Security/Governance Experience implementing least-privilege access and working with catalog, lineage, classification, and row-level security requirements.
• SQL Advanced SQL development and query optimization for large analytical datasets.

Preferred Skills

• Hands-on exposure to Databricks Lakeflow Spark Declarative Pipelines or Delta Live Tables (DLT) and the ability to adopt evolving Databricks-native pipeline standards.
• Experience with Unity Catalog ; familiarity with enterprise data governance and data-quality tooling such as Informatica .
• Experience with HR data from Workday, Dayforce, PeopleSoft, recruiting, workforce management, or talent systems .
• Understanding of Power BI consumption patterns, star schemas, refresh needs, aggregates, and performance requirements.
• Experience creating accelerators, reusable engineering templates, metadata-driven pipelines, or automated test frameworks .
• Databricks Data Engineer certification or equivalent cloud/data engineering credential.
• Experience in a blended onshore/offshore Agile delivery model .

Key Technology Stack

Azure | Databricks | PySpark | Spark SQL | Delta Lake | Medallion Architecture | SQL | Azure Data Lake | DevOps | CI/CD | Unity Catalog | Informatica | Power BI | Workday | Dayforce | PeopleSoft

Technical Responsibilities

Mandatory

• Optimize Spark workloads, Delta tables, joins, partitions, file sizes, and cluster usage for performance and cost.
• Apply enterprise security and governance standards including Unity Catalog, permissions, sensitive data handling, and row level access patterns.
• Coordinate with ingestion, data governance, modeling, DevOps, and Power BI teams to define data contracts and acceptance criteria.
• Contribute to CI/CD, source control, automated testing, deployment, monitoring, and production readiness.
• Participate in requirement workshops and explain technical choices to business, HR, analytics, and engineering stakeholders.
• Support knowledge transfer and mentor engineers/analysts.

Generic Managerial Skills:

• Strong oral and written communication skills.
• Ability to collaborate in teams and work independently.

Roles and Responsibilities:

• Design and develop scalable batch pipelines in Azure Databricks using PySpark and Spark SQL.
• Build and maintain bronze, silver, and gold data layers including certified fact/dimension products.
• Translate business metric definitions, rules, filters, granularity, history, and refresh needs into technical designs.
• Create reusable HR domain transformations (retention, hiring, transfers, promotions, reviews, hours, scheduling).
• Implement data quality checks, reconciliation, exception handling, auditability, lineage, and documentation.