Databricks Quality Lead Tester

Job Description

Job Title : Databricks Quality Lead Tester

Job Location: New York, United States

Job Duration : 6 Months

Client Interview : Yes

Skill Cluster : DA-Data Testing.

Primary Skill : SQL, PySpark, Agile

Experience : 10 years

Job Description :

Job Requirements Key Responsibilities

Data Testing & Validation

• Design and implement test plans, test cases, and test automation for data pipelines (ELT/ETL).
• Validate payer data, including claims, membership, provider, and clinical datasets.
• Conduct FHIR-based API testing for CMS Interoperability compliance.
• Validate HEDIS measure calculations and quality reporting data.

CI/CD Integration

• Develop Azure DevOps test planswith QA gates for Databricks Jobs/DLT and ADF deployments.
• Technical DeliveryCollaborate with product owners, Azure Data Factory (ADF), Databricks engineering teams to ensure end-to-end data quality.
• Build and maintain data quality automation frameworks leveraging PySpark, SQL, and Python.Work with data governance and business teams to resolve data quality issues.

Performance & Scalability

Design automated load and stress tests for large-scale data pipelines.

Compliance & Best Practices

• Ensure data QA practices adhere to HIPAA, CMS, and payer industry standards.
• Document QA processes, test results, and provide

Skills:

• Hands-on experience with Azure Data Factory (pipelines, monitoring, CI/CD).
• Proficiency in Python, PySpark, SQL for test automation.
• Knowledge of data governance, PII compliance, and data quality frameworks.
• Strong communication and leadership abilities for cross-functional teamwork.