Data Tester

We are seeking an experienced Data Tester with strong expertise in Databricks, PySpark, and Big Data ecosystems. This role focuses on validating data pipelines, ETL workflows, and analytical models to ensure data integrity, accuracy, and performance across distributed systems. The ideal candidate will have hands-on experience in cloud environments and automation frameworks, with a deep understanding of data lake testing and SQL-based validation. Key Responsibilities Validate end-to-end data pipelines developed in Databricks and PySpark. Develop and execute test plans, test cases, and automated scripts for ETL and data quality validation. Perform data validation, reconciliation, and regression testing using SQL, Python, and PySpark DataFrame APIs. Verify data transformations, aggregations, and schema consistency across raw, curated, and presentation layers. Test Delta Lake tables for schema evolution, partitioning, versioning, and performance. Collaborate with data engineers, analysts, and DevOps teams to ensure high-quality data delivery. Analyze Databricks job logs, Spark execution plans, and cluster metrics to troubleshoot issues. Participate in Agile/Scrum ceremonies and contribute to sprint planning and defect triage. Maintain documentation for test scenarios, execution reports, and data lineage verification. Required Qualifications 8+ years of experience in data testing or QA in enterprise data environments. 5+ years of experience testing ETL/Big Data pipelines and validating data transformations. 4+ years of hands-on experience with Databricks, including notebook execution and job scheduling. 4+ years of experience in PySpark (DataFrame APIs, UDFs, joins, transformations). 5+ years of strong proficiency in SQL for complex data validation. 3+ years of experience with Delta Lake or data lake testing. 3+ years of experience in Python scripting for automation. 3+ years of experience with cloud platforms (Azure, AWS, or GCP). 2+ years of experience in test automation using tools like pytest or custom Python utilities. 4+ years of experience in data warehousing, data modeling, and data quality frameworks. 4+ years of experience with Agile/SAFe methodologies. 6+ years of analytical and debugging experience for data pipeline issues. Preferred Qualifications Experience with CI/CD tools for Databricks or data testing (e.g., GitHub Actions, Jenkins, Azure DevOps). Exposure to BI validation tools (e.g., Power BI, Tableau, Looker). Knowledge of REST APIs for metadata or integration testing. Familiarity with big data tools such as Hive, Spark SQL, Snowflake, and Airflow. Certifications Microsoft Azure Data Engineer Associate or AWS Big Data Specialty (preferred). #J-18808-Ljbffr