Senior Software Test Engineer

We are hiring a hands‑on Senior Software & AI Test Engineer to design and operationalize a scalable, automation‑first quality framework across our software and AI‑driven systems. This role owns test strategy, infrastructure, and execution, ensuring high‑confidence releases across APIs, cloud services, data pipelines, and AI/ML components. The mandate is twofold: (1) build robust, modern testing systems and (2) embed a pragmatic culture of quality that keeps pace with rapid product development. Key Responsibilities 1) Test Architecture & Infrastructure Design and implement a unified test framework across backend services and APIs, cloud platforms and distributed systems, data pipelines and data quality layers, and AI application and evaluation systems. Define test environments, mocking/simulation strategies, and synthetic data generation. Build and maintain CI/CD pipelines. Integrate testing deeply into CI/CD pipelines with clear gating signals. 2) Automated Testing & Tooling Build and maintain automated test suites: unit, integration, system, regression, and performance testing. Develop test orchestration, reporting dashboards, and failure triage workflows. Ensure tests are deterministic, reproducible, and fast enough for developer iteration. 3) Data & Pipeline Validation Establish validation strategies for data pipelines: schema validation, anomaly detection, and data integrity checks. Build automated tests for ETL workflows and downstream system dependencies. Ensure reproducibility between offline experimentation and production behavior. 4) Debugging & Root Cause Analysis Lead investigation of complex failures across services, data, and AI layers. Establish structured approaches to failure classification and regression prevention. 5) AI/ML Testing & Evaluation Build continuous evaluation pipelines tied to model releases. Define acceptance criteria and release gates for AI features. Develop benchmarking tools for comparing models across datasets and scenarios. 6) Quality Culture & Process Introduce scalable quality practices: shift‑left testing and testability in design. Definition of done includes validation and observability. Partner with engineering and product to define measurable quality metrics (defect escape rate, test signal quality) and release criteria aligned with risk. Balance thoroughness with speed—avoid over‑engineering test systems. Required Qualifications 6–10+ years in software testing, SDET, or quality engineering. Strong programming skills (Python required; experience with backend systems preferred). Proven track record building test frameworks and automation from scratch. Deep understanding of API testing, distributed systems, cloud architectures, CI/CD systems (e.g., GitHub Actions, Jenkins), test methodologies (boundary testing, fuzzing, fault injection). Hands‑on experience validating AI/ML systems, including model evaluation metrics, dataset validation, experiment tracking tools (e.g., MLflow, Weights & Biases). Experience with LLM or computer vision evaluation is a plus. Preferred Qualifications Experience testing data‑intensive systems or analytics platforms. Familiarity with data engineering tools and workflows. Experience with performance, scalability, and reliability testing. Exposure to observability tooling (logs, metrics, tracing) for test validation. Experience working in fast‑paced product environments with evolving requirements. Design and implement evaluation frameworks for AI/ML systems: model performance, robustness, edge cases, failure modes, data quality, drift detection, dataset versioning. What We’re Looking For Builder mindset: Creates frameworks and tools, not just test cases. Systems thinker: Understands interactions across APIs, data, and AI layers. Pragmatic operator: Applies the right level of rigor for the stage of product maturity. Quality driver: Elevates engineering standards without becoming a bottleneck. Hands‑on depth: Writes code, debugs systems, and owns outcomes. Success Metrics (First 6–12 Months) Production‑grade automated test framework integrated into CI/CD. Clear, adopted AI evaluation framework used across AI application releases. Reduction in escaped defects and regression incidents. Measurable improvement in data pipeline reliability and validation coverage. Increased developer confidence in releases without slowing iteration speed. Why Join Covalent At Covalent, you’ll work alongside world‑class scientists and engineers in a dynamic, collaborative environment. We empower our team members to take ownership of their work, innovate constantly, and engage directly with customers, shaping the future of technology. Compensation 125,000 – 175,000 USD per year (Covalent Sunnyvale) #J-18808-Ljbffr