Databricks / Spark Test Engineer

Results-driven Senior QA Engineer with 6 to 8+ years of experience specializing in ETL testing, Big Data validation, Apache Spark testing, and Hadoop ecosystem-based quality engineering. Proven expertise in validating large-scale data pipelines across EMR, Databricks, and distributed data processing environments. Strong background in automation testing (UI, API), database testing, and data integrity validation within enterprise applications. Adept at working in Agile environments, collaborating with cross-functional teams, and ensuring high-quality data-driven systems in production.







Required Skills -

• 6+ years of software testing experience in quality engineering, quality assurance or a similar role.
• Bamboo, and Github experience required
• Cucumber-TestNG framework using JAVA, Selenium, RestAssured, Maven for UI and API Automation Testing experience preferred.
• Nunit Framework using C#, Selenium, RestAssured, Nugut for UI and API testing experience preferred.
• Excellent analytical skills with ability to troubleshoot problems and find root causes.
• It would be great to see candidates with exposure to areas such as Apache Spark testing, EMR/Hadoop-based environments, data validation, and other related Big Data testing activities.
• Good hands on experience with quality engineering and strategies.
• Good experience with test automation, database testing, API testing and Java application testing.
• Experience with use of AI technologies within testing is preferred.

Core Skills

• Big Data Testing: Apache Spark, Hadoop, EMR, Databricks
• ETL Testing & Data Validation: Data pipeline validation, transformation testing, reconciliation
• Automation: Selenium, Cucumber, TestNG, RestAssured, NUnit
• API Testing: REST APIs, Postman, RestAssured
• Database Testing: SQL Server, Oracle, Data validation queries
• Programming: Java, C#, SQL
• CI/CD: Bamboo, GitHub
• Frameworks: BDD (Cucumber), TDD
• Tools: Maven, Jenkins, JIRA

Job Requirements -

• 6+ years of software testing experience in quality engineering, quality assurance or a similar role mainly in ETL, Data and Hadoop testing.
• Bamboo, and Github experience required.
• Good experience with test automation, API testing and Java application testing.
• Experience with use of AI technologies within testing is preferred.
• Cucumber-TestNG framework using JAVA, Selenium, RestAssured, Maven for UI and API Automation Testing experience preferred.
• Nunit Framework using C#, Selenium, RestAssured, Nugut for UI and API testing experience preferred.
• Excellent analytical skills with ability to troubleshoot problems and find root causes.
• HealthCare experience pertaining to Medical Claims processing is preferred.
• Ability to think "outside of the box".
• Flexible in thought and creative in approach to testing.
• Perform end-to-end ETL testing validating large-scale data pipelines across distributed systems.
• Design and execute Apache Spark testing strategies to validate transformations, aggregations, and streaming pipelines.
• Conduct data validation and reconciliation testing between source systems, staging layers, and target data warehouses.
• Work with the Hadoop ecosystem (HDFS, Hive, Spark) for validating batch and real-time data processing.
• Validate data pipelines deployed on AWS EMR and Databricks environments ensuring data accuracy and completeness.
• Develop SQL-based validation frameworks to verify data integrity, consistency, and transformation correctness.
• 4 Year Degree in Computer Science or Management Information Systems or equivalent
• Perform complex SQL validation for large datasets.
• Validate data migration and transformation logic across multiple systems.
• Ensured data quality, completeness, and accuracy through validation rules and checks.

Desired Skills & Experience -
Experience with Vibe programming.