ETL Tester

Job Type- Contractual//Fulltime Role Context: We are migrating our SQL Server data platform (stored procedures, SSIS/SQL jobs, reporting tables) to Databricks on Azure, orchestrated by Azure Data Factory. You will own the testing strategy for this migration: proving data parity between legacy SQL and Databricks, defining the quality gates each release must pass, and validating ADF pipelines end to end. Key responsibilities: Define the migration test strategy and plan covering schema, data, transformation logic, performance and regression testing. Build automated source-to-target reconciliation: row counts, column-level checksums/hashes, aggregates, and full row-level diffs for critical tables. Convert legacy SQL business logic (stored procedures, views) into test cases and validate the equivalent PySpark/Spark SQL output. Define and enforce quality gates across the lifecycle: code merge, Bronze/Silver/Gold layer promotion, UAT sign-off and production cutover. Validate ADF pipelines: parameters, triggers, dependencies, copy activity mappings, failure/retry paths, incremental loads and watermark logic. Test Delta Lake behaviour: MERGE/upsert, SCD Type 1/2, schema evolution, deduplication and late-arriving data. Integrate tests into CI/CD (Azure DevOps or GitHub Actions) with Databricks Asset Bundles or equivalent. Report defects, reconciliation results and go/no-go readiness to project leadership. Partner with the Data Quality Engineer to embed DQX checks into pipelines. Must-have skills : 5+ years in data/ETL testing, including at least one data platform migration. Strong SQL (T-SQL and Spark SQL): joins, window functions, EXCEPT/MINUS-based diffs, aggregates. Hands-on Databricks: notebooks, Jobs/Workflows, Delta Lake, Unity Catalog basics. Hands-on Azure Data Factory: pipelines, linked services, datasets, triggers, monitoring and debugging runs. PySpark for building reusable test and reconciliation frameworks. Experience defining quality gates, entry/exit criteria and test metrics. Test automation with pytest or similar, integrated into CI/CD. Nice-to-have skills : Databricks Labs DQX, Great Expectations, Soda or Lakeflow (DLT) expectations. Performance testing of Spark jobs (partitioning, file sizes, cluster sizing). Knowledge of SSIS, SSRS or Power BI report validation. Responsibilities: Reconciliation framework running automatically for all migrated tables. Quality gates documented, agreed and enforced in the release pipeline. Zero critical data defects found in production after each migration wave. #J-18808-Ljbffr BuzzClan