Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Staff Software Engineer - Testing & Evals Frameworks

$174k - $229k

IDMe

Job Description

Job Description

Company Overview

IDMe is the next-generation digital identity wallet that simplifies how individuals securely prove their identity online. Consumers can verify their identity with IDMe once and seamlessly login across websites without having to create a new login and verify their identity again. Over 175 million users experience streamlined login and identity verification with IDMe at 22 federal agencies, 50 state government agencies, and more than 100 healthcare organizations. Over 600 consumer brands use IDMe to verify communities and user segments to honor service and build more authentic relationships. IDMe's technology meets the federal standards for consumer authentication set by the Commerce Department and is approved as a NIST 800-63-3 IAL2 / AAL2 credential service provider by the Kantara Initiative. IDMe is committed to "No Identity Left Behind" to enable all people to have a secure digital identity. To learn more, visit 

IDMe is a full-time, in-office culture. Unless a specific job description explicitly states otherwise, all roles are on-site five days per week at one of our offices in McLean, VA; Mountain View, CA; New York City, NY; or Tampa, FL. Certain roles — such as field-based sales or other remote-by-design positions — may have different work arrangements as noted in their individual postings.

At IDMe, we embrace the thoughtful use of AI tools in our daily work and there are even occasions where we leverage AI in our hiring process. However, during the interview process, we want to understand your individual skills and experiences. Therefore, we have guidelines on how AI can be appropriately used during your application and interviews which can be found here.

Position Overview

This Staff Engineer role sits at the intersection of engineering, testing, AI evals, and developer experience. You will own the architecture and strategy for the evaluation frameworks, automation tools, and testing infrastructure that give IDMe engineering teams the confidence to ship quickly and safely across functional, end-to-end, LLM/eval, and security-focused testing practices, as well as the CI/CD quality gates the org depends on.

It requires deep engineering rigor, the judgment to set testing strategy for complex, asynchronous, event-driven systems, and the ability to build tooling and standards that other teams adopt widely. You will raise the quality bar across the engineering org, mentor senior and mid-level engineers, champion AI-assisted testing practices, and partner directly with engineering, product, and security teams to embed quality into every stage of development for a platform that verifies the identity of millions of users.

Key Responsibilities
  • Own Test Architecture & Strategy: Set the architecture and long-term strategy for automation frameworks and testing infrastructure (Playwright, Python, Jest, API automation) across IDMe's services, and ensure comprehensive, reliable coverage org-wide.
  • Lead Testing Strategy for Distributed Systems: Define end-to-end and integration testing strategy for complex, asynchronous, event-driven, and eventually-consistent systems, including workflow orchestration platforms such as Temporal.
  • Drive Testing-in-Production as an Org Standard: Build and champion canary deployments, progressive/phased rollouts, synthetic monitoring, and automated rollback triggers to validate real-world behavior safely before full release.
  • Own Security-Focused Test Automation: Set the standard for testing authentication/authorization flows (OAuth 2.0, OIDC), RBAC/ReBAC, secrets management, and tenant isolation.
  • CI/CD Quality Gates: Integrate automated testing into CI/CD pipelines (e.g., GitHub Actions) org-wide to give every team fast, reliable feedback.
  • Drive AI & Evals Culture: Build robust evaluation pipelines for AI-assisted features and LLM agents, while applying AI tools to accelerate test authoring, test plan generation, and code review across the org.
  • Cross-Team Collaboration: Partner with Security, Platform, and Product teams as a testing strategist, advising on testable code structure, coverage gaps, and quality checks throughout the development lifecycle.
  • Mentorship: Guide senior and mid-level engineers on test architecture, testing strategy, and quality practices.
  • Debug & Root Cause: Debug and root-cause issues across frontend, backend, APIs, and distributed services, and drive fixes to closure.
  • Reliability & Eval Metrics: Define and track reliability, quality, and eval accuracy metrics (e.g., time to detect, time to remediate, output quality scores) for the systems you evaluate and test.
Minimum Qualifications
  • 10+ years of experience (or equivalent: 10+ years with a Master's degree, or a PhD with 5+ years) as a software engineer in modern web applications and distributed systems.
  • Proficiency in Python/TypeScript/JavaScript/Java.
  • Hands-on experience with Playwright, Jest, API automation, or evaluation frameworks for software/AI systems.
  • Experience integrating automated tests into CI/CD pipelines (e.g., GitHub Actions, ArgoCD).
  • Proven ability to lead cross-team technical initiatives and influence engineering standards.
  • Track record of mentoring senior engineers and raising the quality bar across teams.
  • Strong written and verbal communication across engineering, product, and leadership.
Preferred Qualifications
  • API & Systems Testing: REST APIs, microservices, distributed and event-driven architectures; workflow orchestration (e.g., Temporal).
  • Identity & Access Management: OAuth 2.0, OIDC, SAML 2.0, SCIM 2.0; enterprise IdPs (Okta, Entra ID, Auth0, Ping); RBAC/ReBAC (Zanzibar, SpiceDB) and multi-tenant authorization.
  • Security & Compliance: AuthN/authZ testing, secrets management, tenant isolation, least-privilege access; IDV/KYC and PII/compliance validation.
  • Testing Strategy & Ops: Canary/phased rollouts, automated rollback, contract testing (Pact), performance/load testing (Locust, k6, Gatling).
  • Engineering & Platform: Java/Spring Boot or Ruby on Rails; GCP, Docker, Kubernetes; observability and chaos engineering.
  • AI-Augmented Testing: Using/building agentic tools for test generation, code review, and test planning.
  • AI Agent Evals: Experience building eval frameworks for LLM agents (e.g., correctness graders, LLM-as-judge, human-in-the-loop evals, benchmark dataset curation); familiarity with agentic frameworks (Claude API/Anthropic SDK, BrainTrust, LangChain, LangGraph, CrewAI, or similar); production monitoring for AI systems (behavioral drift detection, output sampling, shadow scoring); red-teaming or adversarial testing experience for AI models or agents; experience with RAG pipelines, including data chunking and pre-retrieval data quality gates.
  • Background in identity verification, fraud detection, or regulated industries.
  • Experience with observability tooling (Datadog, OpenTelemetry).

The annual base salary listed does not include a company bonus, incentive for sales roles, equity and benefits which will be determined based on experience, skills, education, relevant training, geographic location and role. 

IDMe offers comprehensive medical, dental, vision, health savings account, flexible spending accounts (medical, limited purpose, dependent care, commuter benefit accounts), basic and voluntary life and AD&D insurance, 401(k) with company match, parental leave, ability to participate in unlimited paid time off subject to the terms and conditions of the PTO policy, including 8 company wide holidays, short and long-term disability insurance, accident and critical illness insurance, referral bonus policy, employee assistance program, pet insurance, travel assistant program, wellbeing and childcare discounts, benefit advocates, and a learning and development benefit.

Final offers may vary from the amount listed based on qualifications, professional experiences, skills, education, relevant training, geographic location, and other job related factors.

U.S. Pay Range

$174,000—$229,000 USD

Mountain View, CA Pay Range

$194,000—$253,000 USD

IDMe maintains a work environment free from discrimination, where employees are treated with dignity and respect. All IDMe employees share in the responsibility for fulfilling our commitment to equal employment opportunity. IDMe does not discriminate against any employee or applicant on the basis of age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable laws, regulations and ordinances. IDMe adheres to these principles in all aspects of employment, including recruitment, hiring, training, compensation, promotion, benefits, social and recreational programs, and discipline. In addition, IDMe's policy is to provide reasonable accommodation to qualified employees who have protected disabilities to the extent required by applicable laws, regulations and ordinances where a particular employee works. Upon request we will provide you with more information about such accommodations.

Please review our Privacy Policy, including our CCPA policy, at id.me/privacy. If you provide IDMe with any personally identifiable information you confirm that you have read and agree to be bound by the terms and conditions set out in our Privacy Policy.

IDMe participates in E-Verify.

Vacancy posted 21 days ago
Similar jobs that could be interesting for youBased on the Staff Software Engineer - Testing & Evals Frameworks in Mountain View, CA vacancy
  •  ...patients worldwide. We’re a team of engineers, clinicians, and innovators...  ...of Position Our Servo Frameworks and Simulator team is responsible for the real-time software infrastructure that supports critical...  ..., System Analysis, System Test, User Interface Software Engineering... 
    Suggested
    Full time
    Local area
    Worldwide
    Flexible hours

    Intuitive Surgical

    Sunnyvale, CA
    5 days ago
  • $70 per hour

     ...and tens of billions in simulation across 15+ U.S. states. Software Engineering builds the brains of Waymo's fully autonomous driving...  ...Systems Intelligence and Machine Learning (SIML) Subteam: ML Frameworks and Efficiency Description: Scale the world's most trusted... 
    Suggested
    Hourly pay
    Full time
    Internship
    Summer internship
    Relocation package

    Waymo

    Mountain View, CA
    11 hours ago
  • $143k - $286k

     ...Segment: Home OfficePlatform Engineering Execution: Delivers...  ...infrastructure, tooling, and ML frameworks (for example, CI/CD, SDKs, model...  ...agility across systems. Software Development & Lifecycle Ownership...  ...lifecycle: coding, testing, CI/CD deployment, observability... 
    Suggested
    Full time
    Temporary work
    Part time
    Local area

    Walmart

    Sunnyvale, CA
    11 hours ago
  • $189k - $280k

     ...intuitive design, intelligent software, and next-generation safety...  ...driving. In this hands-on Staff Software/ML Engineer role, you will partner with...  ...open-loop behavioral tests, closed-loop simulation, and...  ...with PyTorch or comparable frameworks and ML training or evaluation... 
    Suggested
    Full time
    Local area
    Remote work
    Work from home
    Relocation
    Relocation package
    Flexible hours

    General Motors

    Sunnyvale, CA
    1 day ago
  • $262k - $364k

     ...hardware design analysis, and systems engineering.Architect and build generative AI...  ..., system schematics, software contracts, and historical test execution telemetry.Transform how...  ...experience using machine learning frameworks and data processing tools.Knowledge... 
    Suggested
    Worldwide

    Google

    Sunnyvale, CA
    1 day ago
  • $140.6k - $171.8k

    Fortinet is looking for a Staff Software Dev QA Engineer to assist FortiADC TeamThe software development...  ...for building high effective automation test process to complete most of test work...  ...maintain scalable network simulation frameworks for performance and scalability... 
    Full time
    Worldwide

    Fortinet

    Sunnyvale, CA
    1 day ago
  • $153.2k - $234.1k

    Job DescriptionThe Role We’re AV Frameworks within AV Platform Core at General Motors....  ...reliable, observable, and fast. As a Senior Software Engineer, you will design and deliver the core...  ...Apply production rigor: code reviews, testing strategies (unit, integration, soak),... 
    Permanent employment
    Full time
    Local area
    Work from home
    Relocation package
    Flexible hours

    General Motors

    Sunnyvale, CA
    4 days ago
  • $265k - $331.3k

     ...Our Pose Team is a diverse group of algorithm developers / software engineers focused on delivering mapping, localization, and calibration...  ...Define, review, and refactor the architecture, algorithms, and frameworks for solving pose, calibration, and SLAM problems. The... 
    Full time
    Contract work
    Temporary work
    Part time
    Local area
    Shift work

    Rivian

    Palo Alto, CA
    2 days ago
  •  ...Synopsys is the leader in engineering solutions from silicon to systems...  ...Technical ownership at the staff level means end-to-end...  ...that sustaining production software demands more than feature delivery...  ...establishing robust testing frameworks, comprehensive documentation... 
    Immediate start
    Remote work

    Synopsys, Inc.

    Sunnyvale, CA
    11 hours ago
  • $206.5k - $258.1k

     ...generations. Role Summary Rivian's Autonomy org needs a Staff Software Engineer, Compute & Storage to own the distributed compute and...  .... ~3+ years with a distributed processing or training framework (Ray, Spark, Flink, or Dask). ~3+ years hands-on with Go,... 
    Full time
    Contract work
    Temporary work
    Part time
    Local area
    Shift work

    Rivian

    Palo Alto, CA
    4 days ago
  • $228k - $285k

     ...protect it for future generations. Role Summary As a Staff Software Engineer, ML training and inference infrastructure, you will be a...  ...Deep knowledge of PyTorch Knowledge of model training framework (e.g. PyTorch Lightning, ray, etc.) In-depth knowledge of... 
    Full time
    Contract work
    Local area

    Rivian

    Palo Alto, CA
    3 days ago
  • $206.5k - $258.1k

     ...Infrastructure team to give hundreds of ML engineers a training platform they can trust at fleet scale. We are seeking a Staff Software Engineer to help design and build the...  ...where every GPU-hour goes, and the training framework layer that lets ML engineers write model... 
    Full time
    Contract work
    Temporary work
    Part time
    Local area
    Shift work

    Rivian

    Palo Alto, CA
    11 hours ago
  • $206k - $258k

     ...generations. Role Summary As a Software Engineer specializing in safety-critical self-...  ...support development, debugging, and testing Assist in software-design and code...  ..., OpenVX, or relevant acceleration frameworks in perception applications ~ Experience... 
    Full time
    Contract work
    Local area

    Rivian

    Palo Alto, CA
    1 day ago
  •  ...generation of embedded platform software to deliver state of the art...  ..., to achieve that goal our testing processes, tools, and infrastructure...  ...We are looking for energetic engineers to help us deliver high...  ...Senior Software Developer - Test Framework at GM, you will help to lead... 
    Full time
    Work at office
    Local area
    Work from home
    Relocation package
    Flexible hours
    3 days per week

    General Motors

    Mountain View, CA
    13 days ago
  • $189k - $303k

     ...for all. We’re searching for a Staff Security Software Engineer. This is a software architecture/development...  ...and development of new security frameworks and technologies for our autonomous...  ...high quality production code, unit tests, integration tests Able to debug... 
    Work at office
    Local area
    3 days per week

    Aurora Innovation

    Mountain View, CA
    2 days ago
  • $228k - $285k

     ...Summary We are seeking a Staff Geospatial Data Engineer to lead the architectural...  ...and own high-performance software related to route processing...  ...optimization of spatial frameworks and routing engines to support...  ..., scalable, thoroughly tested, and performant. Qualifications... 
    Full time
    Contract work
    Temporary work
    Part time
    Local area
    Shift work

    Rivian

    Palo Alto, CA
    3 days ago
  • $110k - $220k

     ...OfficeRole summary: Walmart is seeking a Staff Software Engineer to lead the development and delivery...  ...infrastructure, tooling, and AI/ML frameworks. This role involves driving the full...  ...lifecycle activities such as coding, testing, deployment, monitoring, and maintenance... 
    Full time
    Temporary work
    Part time

    Walmart

    Sunnyvale, CA
    11 hours ago
  •  ...leader in catastrophe-exposed property insurance, is seeking a Staff Software Engineer to help design, build, and evolve the software platforms...  ...continuously improving.What you’d be doing:Design, develop, test, and maintain scalable software applications, services, APIs... 
    Live in
    Remote work

    SageSure

    Mountain View, CA
    2 days ago
  • $202.5k - $274k

     ...possible.Job OverviewCome join the Intuit Mid Market team as a Staff Software Engineer. In this role you will get to build AI native platforms...  ...is an added advantageExperience of data Tools and spark frameworks (i.e. Python, PyTorch, Numpy, Pandas etc).Knowledge of Applied... 
    Work experience placement
    Worldwide

    Intuit

    Mountain View, CA
    2 hours ago
  •  ...fast decode on the Cerebras Wafer-Scale Engine.We are hiring a Software Engineer to productionize and...  ...with a high-performance model-serving framework such as vLLM, SGLang, TensorRT-LLM, Triton...  ..., or performance-regression test systems.Experience collaborating directly... 

    Cerebras Systems

    Sunnyvale, CA
    1 day ago
  • $110k - $220k

     ...OfficeRole summary:Join Walmart as a Staff Software Engineer to lead the design, development, and...  ...including infrastructure, tooling, and AI/ML frameworks. This role drives the full software...  ...lifecycle activities such as coding, testing, deployment, monitoring, and... 
    Full time
    Temporary work
    Part time

    Walmart

    Sunnyvale, CA
    11 hours ago
  • $218.3k - $327.5k

     ...RoleThe Site Reliability Engineering (SRE) team at Rubrik...  ...the intersection of software development and systems...  ...structural resiliency. As a Staff Site Reliability...  ...controllers, and automation frameworks in Go or Python to...  ...data structures, and test-driven software design... 
    Shift work

    Rubrik

    Palo Alto, CA
    11 hours ago
  • $143k - $286k

     ...WalmartBusiness Segment: Home OfficeRole summary:The Staff Software Engineer is responsible for leading the design, development, testing, and deployment of scalable, cloud-native...  ..., tooling, and machine learning frameworks aligned with roadmap milestones.Apply cloud... 
    Full time
    Temporary work
    Part time

    Walmart

    Sunnyvale, CA
    4 days ago
  •  ...quality. We own large scale test scenario libraries, continuous...  ...cases in a unified analytics framework. By joining this team, you will...  ...evaluate autonomous driving software performance across the...  ...efforts with autonomy, systems engineering, simulation, and data teams to... 
    Full time
    Local area
    Work from home

    General Motors

    Sunnyvale, CA
    3 days ago
  •  ...decode on the Cerebras Wafer-Scale Engine.We are hiring a Software Engineer to build and evolve the ML...  ...production inference traffic.Develop testing and qualification infrastructure. Create...  ...software across service, framework, runtime, and infrastructure boundaries... 

    Cerebras Systems

    Sunnyvale, CA
    11 hours ago
  • $198k - $326k

     ...of the team.The Capacity & Efficiency Engineering team drives LinkedIn’s end-to-end infrastructure...  ...sustainably.We are seeking a Sr. Staff Software Engineer to join LinkedIn's Capacity &...  .../PubSub systems, Flink, Airflow, ETL framework and distributed storage (HDFS, S3/EBS,... 
    For contractors
    Work at office
    Immediate start
    Flexible hours

    Linkedin

    Mountain View, CA
    4 days ago
  • $143k - $286k

     ...About the Role:Walmart is seeking a Staff Software Engineer to lead the development and delivery...  ...including infrastructure, tooling, and AI/ML frameworks. This role involves driving the full...  ...lifecycle activities such as coding, testing, deployment, monitoring, and... 
    Full time
    Temporary work
    Part time
    Remote work

    Walmart

    Sunnyvale, CA
    1 day ago
  • $213.48k - $319.72k

     ...destinations safely and efficiently. We continuously refine our routing engine to calculate more efficient routes, deliver highly accurate...  ...(BigQuery, Snowflake, etc).You know your way around RPC frameworks (gRPC, Thrift) and pubsub systems (Kafka, Cloud PubSub, Celery,... 
    Immediate start
    Flexible hours

    Nuro

    Mountain View, CA
    2 days ago
  • $150k - $183k

    We are seeking a Staff Frontend Engineer to join our FortiWeb team, responsible for developing and maintaining the web-based management interface...  ...LeadershipLead the migration strategy from legacy frameworks to modern Angular 21+ and React 18+Design and implement micro... 
    Full time
    Worldwide

    Fortinet

    Sunnyvale, CA
    11 hours ago
  • $130k - $220k

    Santa Clara, CASoftware Engineering - Motion Planning /Full-time /HybridPlusAI...  ...AI-based virtual driver software for factory-built autonomous...  ..., performance analysis, testing, and runtime monitoring....  ...similar inference acceleration frameworks.Strong software engineering... 
    Full time

    Plus.ai

    Santa Clara, CA
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Staff Software Engineer - Testing & Evals Frameworks. Be the first to apply!