Senior Software Engineer in Test

Senior Software & AI Test Engineer

We are hiring a hands-on Senior Software & AI Test Engineer to design and operationalize a scalable, automation-first quality framework across our software and AI-driven systems. This role owns test strategy, infrastructure, and execution, ensuring high-confidence releases across APIs, cloud services, data pipelines, and AI/ML components. The mandate is twofold: (1) build robust, modern testing systems and (2) embed a pragmatic culture of quality that keeps pace with rapid product development.

Key Responsibilities

• Test Architecture & Infrastructure · Design and implement a unified test framework across: · Backend services and APIs · Cloud platforms and distributed systems · Data pipelines and data quality layers · AI application and evaluation systems · Define test environments, mocking/simulation strategies, and synthetic data generation. · Build and maintain CI/CD pipelines. · Integrate testing deeply into CI/CD pipelines with clear gating signals.
• Automated Testing & Tooling · Build and maintain automated test suites: · Unit, integration, system, regression, and performance testing · Develop test orchestration, reporting dashboards, and failure triage workflows. · Ensure tests are deterministic, reproducible, and fast enough for developer iteration.
• Data & Pipeline Validation · Establish validation strategies for data pipelines: · Schema validation, anomaly detection, and data integrity checks · Build automated tests for ETL workflows and downstream system dependencies. · Ensure reproducibility between offline experimentation and production behavior.
• Debugging & Root Cause Analysis · Lead investigation of complex failures across services, data, and AI layers. · Establish structured approaches to failure classification and regression prevention.
• AI/ML Testing & Evaluation · Build continuous evaluation pipelines tied to model releases. · Define acceptance criteria and release gates for AI features (beyond traditional QA). · Develop benchmarking tools for comparing models across datasets and scenarios.
• Quality Culture & Process · Introduce scalable quality practices: · Shift-left testing and testability in design · Definition of done includes validation and observability · Partner with engineering and product to define: · Measurable quality metrics (e.g., defect escape rate, test signal quality) · Release criteria aligned with risk · Balance thoroughness with speed—avoid over-engineering test systems.

Required Qualifications

· 6–10+ years in software testing, SDET, or quality engineering. · Strong programming skills (Python required; experience with backend systems preferred). · Proven track record building test frameworks and automation from scratch. · Deep understanding of: · API testing, distributed systems, and cloud architectures · CI/CD systems (e.g., GitHub Actions, Jenkins) · Test methodologies (boundary testing, fuzzing, fault injection) · Hands-on experience validating AI/ML systems, including: · Model evaluation metrics and tradeoffs · Dataset validation and management · Experiment tracking tools (e.g., MLflow, Weights & Biases, or equivalent) · Experience with LLM or computer vision evaluation is a plus.

Preferred Qualifications

· Experience testing data-intensive systems or analytics platforms. · Familiarity with data engineering tools and workflows. · Experience with performance, scalability, and reliability testing. · Exposure to observability tooling (logs, metrics, tracing) for test validation. · Experience working in fast-paced product environments with evolving requirements. · Design and implement evaluation frameworks for AI/ML systems: · Model performance (accuracy, precision/recall, etc.) · Robustness, edge cases, and failure modes · Data quality, drift detection, and dataset versioning

What We're Looking For

· Builder mindset: Creates frameworks and tools, not just test cases. · Systems thinker: Understands interactions across APIs, data, and AI layers. · Pragmatic operator: Applies the right level of rigor for the stage of product maturity. · Quality driver: Elevates engineering standards without becoming a bottleneck. · Hands-on depth: Writes code, debugs systems, and owns outcomes.

Success Metrics (First 6–12 Months)

· Production-grade automated test framework integrated into CI/CD. · Clear, adopted AI evaluation framework used across AI applications releases. · Reduction in escaped defects and regression incidents. · Measurable improvement in data pipeline reliability and validation coverage. · Increased developer confidence in releases without slowing iteration speed.

Why Join Covalent

At Covalent, you'll work alongside world-class scientists and engineers in a dynamic, collaborative environment. We empower our team members to take ownership of their work, innovate constantly, and engage directly with customers, shaping the future of technology.