Senior SDET - LLM Evaluation & Automation for AI Pipelines
Apple
Apple seeks a Senior SDET to lead automated model evaluation across our generative AI pipelines, standing up LLM-as-a-judge and integrating it into CI/CD workflows.
You will build end-to-end eval coverage, curate rubrics, run scalable eval jobs, and help surface reliable quality signals before human review or deployment.
This hands-on, senior individual-contributor role requires strong Python, testing, and collaboration with modeling, framework, and infra teams.
#J-18808-LjbffrVacancy posted 4 hours ago
Similar jobs that could be interesting for youBased on the Senior SDET - LLM Evaluation & Automation for AI Pipelines in San Diego, CA vacancy
- ...builds the tooling and automation that keeps Apple... ...We are looking for a Senior SDET to lead the design and... ...implementation of automated model evaluation: standing up LLM-as-a-judge in existing and new pipelines, and building the... ...powering Generative AI features. Package...PipelineSenior
Apple
San Diego, CA4 hours ago $208k - $300k
...vehicles at Zoox. In this role, you will develop metrics and evaluation pipelines to assess simulator fidelity and guide improvements to our GenAI... ...experiences, and skills.We may use artificial intelligence (AI) tools to support parts of the hiring process, such as...PipelineSeniorFull timeTemporary workRelocation package$49.4 - $57.21 per hour
...Development Engineer In Test II (SDET II)Brain Corp is a San... ...California, USA-based AI company creating... ...and efficient automated solutions in both commercial... ...testingContribute to the evaluation and adoption of modern... ...continuous delivery (CI/CD) pipelines by integrating...PipelineWork experience placementWork at officeLocal areaImmediate startFlexible hoursSoftbank Investment Advisers
San Diego, CA3 days ago$110k - $170k
...Time Employee /On-siteShield AI is a venture-backed defense-tech... ...:Shield AI's Software Test & Automation Team develops the... ...Loop (VIL) environments. As a Senior Software Test & Automation Engineer... ...issues. Experience with CI/CD pipelines and continuous validation practices...PipelineSeniorFull timeTemporary workPart timeWorldwide- ...expertise with software, automation, and data systems to... ...scalable. The Role As a Senior Software Engineer, you’... ...Leverage AI tools to increase development... ...expertise into software Evaluate and adopt the latest AI... ...GCP), APIs, and data pipelines ~ Ability to reason about...PipelineSeniorFlexible hours
Cedar
San Diego, CA4 hours ago $142.2k - $213.4k
...establish responsible AI practices, model... ...executionProvide senior technical... ...learning, including LLM-enabled solutions)... ...APIs/services, data pipelines, and production deployment... ..., code reviews, automated testing, CI/CD,... ...fundamentals (model evaluation, reproducibility,...PipelineSeniorFull timeRemote workShift workClearanceJobs
San Diego, CA4 days ago$163.2k - $220.8k
...growth. Wilson Sonsini is looking for a Senior AI Security Engineer to join the Security... ...engineering function: designing secure AI pipelines, evaluating MCP server integrations and agentic... ...designing secure-by-default patterns for LLM integration, agentic workflows,...PipelineSeniorFull timeWork experience placementRemote workWorldwideShift work$89k - $135.75k
...Description Summary The Senior Data & AI Analyst, Finance Systems... ...workflows for data quality, process automation, anomaly detection, and... ...process monitoring AI/LLM & Intelligent Agent Development... ...document processing pipelines using LLMs for invoice extraction...PipelineSeniorPermanent employmentFull timeContract workWork experience placementWork at officeAeroVironment
San Diego, CA2 days ago$112.7k - $169.06k
...better world, so we can all enjoy living in it.Senior Gas Turbine Performance Engineer: AI Accelerated EngineeringWhy this roleIf you love... ...and maintaining the ecosystem (models, workflows, automation, and data pipelines) that turns thermodynamics, test, and field data...PipelineSeniorFull timePart timeWork at officeWorldwideRelocationFlexible hours$131.29k - $190.11k
...Intelligent Edge. ADI combines analog, digital, AI, and software technologies into... ...world, and help drive advancements in automation and robotics, mobility, healthcare,... .... Learn more at and on LinkedIn and X.Senior Design Evaluation EngineerAbout the RoleJoin our industry...SeniorPermanent employmentFull timeWork at officeDay shift$184.03k - $266.49k
...ADI combines analog, digital, AI, and software technologies... ...and help drive advancements in automation and robotics, mobility, healthcare... ...reviews Guiding layout, evaluation, and test engineering teams... ..., with architectures such as pipeline ADC, SAR ADC, or current-...PipelineSeniorPermanent employmentFull timeWork at officeRemote workWork from homeDay shift- ...AppFolio in San Diego is seeking a Senior Machine Learning Engineer to design... ...generation of voice and conversational AI agents within Realm-X. This role... ...production voice and chat agent pipelines, working at the intersection of LLM agent frameworks, real-time voice technology...PipelineSenior
AppFolio
San Diego, CA4 hours ago $168k - $211k
...Get AI-powered advice on this job and more exclusive features... ...collaboration. As a Senior AI Data Scientist, you will... ...MLOps, model optimization, LLM prompting/guiding/evaluating, and an ability to go from... ...Implement and optimize data pipelines and inference models to meet...PipelineSeniorFull timeTemporary workWork at officeRemote workWork from homeHome officeFlexible hoursFaro Health
San Diego, CA4 hours ago$140k - $180k
...dedicated to enabling seniors to successfully... ...a Senior Applied AI Engineer to join... ...architectures and automated workflows—that amplify... ...existing AI and LLM tools,... ...generation (RAG) pipelines, and automated analytical... ...instructions, and evaluation frameworks that ensure...PipelineSeniorTemporary workWork at officeImmediate startRemote workMonday to FridayFlexible hoursWest Health
San Diego, CA9 days ago- ...missing data layer for production AI agents — and this role leads that effort. As Senior Director of Agentic Database... ...Semantic Context Layer, Agent Memory, LLM Cache, Agent Tracer, and Elastic/... ...-text search, JSONB, MVCC, CDC pipelines, and serverless branching.Design...PipelineSeniorTemporary work
$140k - $180k
...liver, and our clinical pipeline includes disease... ...PositionArrowhead is seeking a Senior Manager, ERP Finance... ...365 F&O capabilities, AI advancements, and... ...solution integration, evaluation, and ongoing partnership... ...and SK Global Treasury Automation SuiteStrong...PipelineSenior$141k - $206.8k
...the world.The Biologics Automation and High-throughput Technologies... ...lab digitalization, and AI-driven orchestration... ...applications and data pipelines that connect... ...current with developments in LLM tooling and agent frameworks and help evaluate what is practically applicable...PipelineFlexible hours- ...Valid8 Financial, Inc. is seeking a Senior Software Engineer to lead XPath 1.0 / XSLT development and data integration initiatives.... ...compatibility, and branding accuracy in document outputs, with a focus on modernizing legacy pipelines and delivering robust data #J-18808-LjbffrPipelineSenior
Valid8 Financial, Inc.
San Diego, CA4 hours ago $202k - $215k
...facing tests Building inference-adjacent evaluation machinery: calibration checks, confidence... ...system you can discuss Comfort using AI-assisted development tools while still... ...Rust- or C++-based inference or numerical pipelines, including custom FFI to CUDA, cuDNN, TensorRT...PipelineSeniorFull timeContract workLocal areaSan Diego Stealth Startup
San Diego, CA18 days ago- ...Infrastructure. This is a senior technical role for a high-impact, complex automation ecosystem supporting U.... ..., an experienced AI/automation architect, and... ...machine learning pipelines across varied data domains... ...engineering judgment to evaluate tradeoffs among deployment...PipelineFull timeFor contractors
General Atomics
San Diego, CA4 hours ago - ...leading research accelerator for frontier AI labs and a trusted partner for global... ...with high-quality data, advanced training pipelines, plus top AI researchers who specialize in... ...Overview What Does a Typical Day Look Like? Evaluate and refine AI-generated code to ensure...PipelineSeniorFull timeFor contractorsRemote workFlexible hours
Turing
San Diego, CA3 days ago - ...right time to stop exceptions. OUR STACK Cloud: AWS, Vercel Automation: Github Pipelines, AWS CodeDeploy. Technologies: NestJs (Typescript), NextJs... ...Management tool: JIRA, githu 5+ years of experience as a Senior QA Engineer / Manager with both Manual and Automation...PipelineSenior
GrabJobs
San Diego, CA3 days ago $109k - $174.8k
...Innovative Medicine is recruiting for a Senior Scientist, AI Applications, R&D Oncology to join our... ...research across the drug development pipeline, including novel target discovery,... ...extended discovery workflowsPrototype and evaluate innovative approaches for uncertainty...PipelineSeniorFull timeTemporary workLocal areaRemote work$160k - $200k
...SAIC is seeking a Senior Principal Cloud Architect... ...Provides detailed analysis, evaluation and recommendations... ...-as-a-service; automation; hypervisors; containers... ...analytics workloads, and AI/ML development... ...enable scalable data pipelines, advanced analytics, and...PipelineSeniorWork experience placementInterim roleLocal areaRemote workSaic
San Diego, CA4 hours ago$158k - $216k
Job DescriptionWe are seeking a Senior Client Service Manager (CSM)... ...relationship health, opportunity pipeline, and competitive insights.... ...Sales Leader on key pursuits; evaluate opportunities and develop win... ...digital tools, data systems, and AI-enabled platforms to support...PipelineSeniorFull timeTemporary work$183.5k - $232.5k
...and smart cameras, plus the AI and fleet services layered on... ...a small, elite team of senior technical operators to own this... ...example rather than by mandate.Evaluate emerging technologies,... ...data-intensive systems.CI/CD pipelines, automated deployments, and infrastructure...PipelineSeniorFull timeWork at officeLocal areaImmediate startRemote work$163.2k - $244.8k
...Machine Learning EngineerFounded in 2015, Shield AI is a venture-backed defense-tech company with the mission... ...and production, our engineers build the data pipelines, supervised fine-tuning (SFT) workflows, evaluation frameworks, and deployment infrastructure needed to...PipelineSeniorTemporary workPart timeWork experience placementWorldwideShield AI
San Diego, CA2 days ago- ...applied intelligence that brings AI into the real world. Our... ...infrastructure the platform runs on: the pipelines that turn large-scale... ..., formatting, and the evaluation sets that keep training honest... ...latency, and anomalies, with automated regression checks that flag trouble...PipelineSenior
Xora Innovation
San Diego, CA16 days ago - ...About Titan Titan builds AI software for banks:... ...function. There is no evaluation framework, no regression... ...evaluation framework for LLM and agentic AI outputs... ...write and maintain the automated test suite: end-to-end,... ...APIs, document ingestion pipelines, AI inference workflows...PipelineLive inRemote work
GrabJobs
San Diego, CA23 hours ago $191k - $258.5k
...OverviewWe’re looking for a Principal, Data & AI Systems Lead to join Intuit’s Marketing... ...capabilities. You will build AI‑powered automation workflows, predictive analytics... ...customer experience.Build automated reporting pipelines that quantify the impact of Marketing Futures...PipelineWorldwide
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior SDET - LLM Evaluation & Automation for AI Pipelines. Be the first to apply!
Related searches
- senior lead project manager San Diego, CA
- senior robotics software engineer San Diego, CA
- senior devops engineer remote San Diego, CA
- senior sas administrator San Diego, CA
- senior IT manager San Diego, CA
- senior contracts analyst San Diego, CA
- sr project manager San Diego, CA
- senior windows systems engineer San Diego, CA
- consultant senior consultant San Diego, CA
- senior manager data science San Diego, CA