Software Engineer — Ai Evaluation & Automation : 26-02760
$65 - $73 per hourAkraya, Inc.
Primary Skills: Python (advanced), Git (advanced), AI coding tools (advanced), API Integration (intermediate), Docker (Proficient)
Contract Type: W2
Duration: 6 months (Possible extension)
Location: San Diego, CA ( PST)
Pay Range: $65.00- $73.00 per hour
#LP
Job Summary:
This role focuses on building and scaling the systems used to measure the performance of AI-powered software development tools. You will engineer evaluation frameworks and automate benchmarking processes to ensure reproducible and reliable results. The core of this position is ensuring the accuracy of measurements, not solely the automation of tasks.
Key Responsibilities:
- Develop and integrate evaluation systems for AI development tools.
- Automate benchmark runs with versioned, reproducible environments.
- Calibrate evaluation metrics against human judgment for accuracy.
- Support performance benchmarking across quality and efficiency.
- Analyze and improve the reliability of evaluation workflows.
- Experience with AI coding tools and harnesses.
- Proven evaluation and benchmarking experience.
- Strong proficiency in Git and containerization with Docker.
- Python, Java or JavaScript
Experience in software development, particularly with AI-assisted coding tools and their evaluation, is highly beneficial.
ABOUT AKRAYA
Akraya is an award-winning IT staffing firm consistently recognized for our commitment to excellence and a thriving work environment . Most recently, we were recognized Stevie Employer of the Year 2025, SIA Best Staffing Firm to work for 2025, Inc 5000 Best Workspaces in US (2025 & 2024) and Glassdoor's Best Places to Work (2023 & 2022)!
Industry Leaders in Tech Staffing
As Talent solutions provider for Fortune 100 Organizations, Akraya's industry recognitions solidify our leadership position in the IT staffing space. We don't just connect you with great jobs, we connect you with a workplace that inspires!
Join Akraya Today!
Let us lead you to your dream career and experience the Akraya difference. Browse our open positions and join our team!
Vacancy posted 14 hours ago
Similar jobs that could be interesting for youBased on the Software Engineer — Ai Evaluation & Automation : 26-02760 in San Diego, CA vacancy
$141k - $206.8k
...the world. The Biologics Automation and High-throughput Technologies... ...an Advisor Lab Automation Software Engineer to help build the... ...platforms, lab digitalization, and AI-driven orchestration work together... ...agent frameworks and help evaluate what is practically...SuggestedFull timeFlexible hours$201k - $315k
Foster City, CA / San Diego, CASoftware - Software Systems /Full-time /HybridSimulation is... ...this role, you will develop metrics and evaluation pipelines to assess simulator fidelity and... ...skills.We may use artificial intelligence (AI) tools to support parts of the hiring...SuggestedFull timeTemporary workRelocation packageZoox
San Diego, CA22 hours ago- ...and support production AI services and workflows... ...including LLM-powered automation, retrieval systems,... ...and technical teams to evaluate AI quality and safety,... ...in computer science, engineering, or a related technical... ...years of professional software engineering experience...Suggested
engineeringjobs.net, Inc.
San Diego, CA1 day ago $30 per hour
...geared toward providing multi-software platform exposure that... ...with immediate exposure to AI-driven automation.For this position, you will... ...for reliable results, and evaluate AI output for accuracy, edge... ...digital contentPractice sound engineering discipline: version control...SuggestedRotational programPermanent employmentFull timeInternshipWork at officeLocal areaImmediate startAxos Bank
San Diego, CA3 days ago$68.1k - $95.4k
...design practice. Our architects, engineers, interior designers,... ...develop computational design, automation, and design technology innovation... ...exploration, option studies, layout evaluation, and decision-making through... ...usability.Experience using AI-assisted development tools to...SuggestedFull timeTemporary workPart timeCasual workLocal areaFlexible hoursStantec
San Diego, CA2 days ago$150.1k - $225.1k
...entertainment excellence and creativity.Software Engineer II - Platform Data Reliability &... ...Engineer II focused on developing reliable, automated infrastructure platforms that support data... ...infrastructure APIs.Experience using AI/ML or generative AI to improve infrastructure...WorldwideSony Interactive Entertainment America
San Diego, CA4 days ago$100k - $200k
...scientists.The Opportunity The Software CybersecurityEngineer... ...tools. They will also use AI-powered tools and automation to speed up security reviews... ...security reviews with engineering teams, and translating vulnerability... ...tools and copilots, and evaluate them for safe, policy-...WorldwideShift workAbbott
San Diego, CA4 days ago- ...driven analytic insights, systems engineering, strategies and plans that... ...and simulation tools to evaluate warfighting effectiveness and... ...be provided.SPA is seeking a Software Engineer in San Diego, CA to... ...experienceExperience with Agentic AI coding tools to facilitate...Flexible hours
Systems Planning and Analysis
San Diego, CA22 hours ago $280.6k - $420.85k
...SummarySenior Director, Enterprise AI Platform EngineeringThe... ...Director, Enterprise AI Platform Engineering will define and lead Insulet’... ..., AI agents, intelligent automation, and advanced AI-driven services... ..., integration adapters, evaluation harnesses, playbooks, and communities...Full timeWork at officeInsulet Corporation
San Diego, CA4 days ago$141k - $228.8k
...better for people around the world. We are seeking a Lab Automation Software Engineering Advisor to join our biotherapeutics discovery automation team... ..., you'll be at the forefront of designing and building AI-integrated, closed-loop autonomous discovery ecosystem whereby...Full timeFlexible hours$255k - $300k
...Client Services / Solutions ) - Sales Engineering /Full Time /HybridAHEAD builds... ...together advances in cloud infrastructure, automation and analytics, and software delivery, we help enterprises... ...benefits for additional details. Use of AI:We may use artificial intelligence (...Full timeWork at officeAHEAD
San Diego, CA4 days ago$117.5k - $179.25k
...Manager, Software Engineering The Manager, Software Engineering, leads a software QA, automation, and DevSecOps team, growing the people, processes... ..., training, performance evaluation and career development.... ...risks and issues Embraces AI-assisted development and test...Permanent employmentContract workWork experience placementWork at officeImmediate start$75 - $85 per hour
...Skills: React (Advanced), Python (Advanced), JavaScript (Advanced), AI Tools (Advanced), Cyber Security (Intermediate), Contract... ...functionalities into existing platforms. The position requires a proactive engineer to support ongoing development and maintenance efforts within a...Hourly payContract workAkraya, Inc.
San Diego, CA14 hours ago$177.3k - $265.9k
...level up your career? Join PlayStation as a Senior Software Engineer focused on Platform Data Reliability and Automation and help drive innovation at the forefront of... ...driving automation-first practices, and applying AI/ML where applicable, you’ll enhance system uptime...Permanent employmentWorldwideSony Interactive Entertainment America
San Diego, CA22 hours ago$144k - $193k
...CA / San Diego, CASoftware - Software Platforms and Product /Full-time... .... We build the tooling and automation that improves the release... ...driving a release end to end and engineering the systems that streamline... ...use artificial intelligence (AI) tools to support parts of the...Full timeTemporary workRelocation packageZoox
San Diego, CA4 days ago- ...delivers real business value with AI.What You’ll DoWe are seeking a Staff Software Engineer to lead the design, development,... ...across large-scale MPP systems.Evaluate and drive adoption of emerging... ...Python for systems programming, automation, and diagnostics.Experience with...Flexible hours
Teradata
San Diego, CA3 days ago $185k - $251k
...Diego, CASoftware - Autonomy Software /Full-time /HybridThe Collision... ...Artificial Intelligence (AI) autonomy stack, and has a close... ...data, software and systems engineers defining and building metrics... ...closely with CAS engineers to evaluate system performance.Collaborate...Full timeTemporary workRelocation packageZoox
San Diego, CA2 days ago- ...Cloud engineers to build the infrastructure and automation scenarios frontier AI labs use to test model judgment. AfterQuery builds the expert evaluation data that frontier AI labs use to measure and improve their models. We're hiring cloud engineers as contractors...Weekly payContract workFor contractorsRemote work10 hours per week
- ...Manufacturing company, is looking for a Software Engineer for their San Diego, CA location. Responsibilities... ...: Design and develop AI agents and autonomous systems capable of... ...learning models using PyTorch Evaluate and deploy appropriate LLMs based on specific...
$177.3k - $265.9k
...Senior Software Engineer - AI & Agentic Architecture Why Sony Interactive Entertainment? Sony... ..., validation/retry loops) and build evaluation frameworks to measure and improve output... ...development into CI/CD pipelines for automated code review, test generation, and PR...$129.5k - $194.3k
...Technologies, Inc. Job Area Engineering Group, Engineering Group ADAS R&D Software General Summary At Qualcomm... ...the-art artificial intelligence (AI) capabilities, and an innovative... ...module and works across teams to evaluate the impact of results and validates...Work experience placementWork from home$120k - $155k
...Software Engineer II (Backend) $120,000-$155,000 Base Salary, Position eligible for additional... ...process by combining intelligent automation, thoughtful product design, and data-driven... ...underwriting and real-time risk evaluation to intuitive customer workflows and seamless...Full time3 days per weekFairSquare
San Diego, CA12 days ago$130k - $190k
...across North America. Its software is deeply embedded in customers... ...technical standards, and AI-augmented engineering practices from the inside.... ...better prompts, context, evaluation approaches, and agentic... ...subtle quality issues that automated checks may miss. Own...Kismet Search
San Diego, CA14 hours ago- ...Lilly seeks a Lab Automation Software Engineering Advisor at the San Diego Biotechnology Center to architect AI-integrated, closed-loop discovery systems for biotherapeutics. You will design software to orchestrate automated workflows, build data pipelines, and deploy...
$95.9k - $159.9k
..., R&D, Clinical, and Process Engineering to support the investigation,... ...documentation, and resolution of software- and device-related... ...architecture to effectively evaluate product performance, investigate... ...unsolicited resumes/applications.AI in Hiring Notice: We may use...WorldwideDexcom
San Diego, CA4 days ago- ...The RoleQuidelOrtho is seeking a Staff Software Test Engineer, Embedded Systems, to support our R&D Software... ...diagnostic platforms and developing automated test suites. This includes leading the... ..., GitHub ActionsExperience using AI-assisted software testing tools to improve...Full timeSummer workNight shift
QuidelOrtho Corporation
San Diego, CA1 day ago $80 - $90 per hour
...We are seeking a Staff Software Engineer to lead the design, development,... ...foundation of our database and AI solutions. You will... ...large-scale MPP systems. Evaluate and drive adoption of emerging... ...Python for systems programming, automation, and diagnostics. Experience...Remote workThe Carrera Agency
San Diego, CA14 hours ago- MissionAs a Software Architect, you will contribute to the architecture... ...solutions while advancing AI-empowered software... ...collaborative product team, comprised of engineers, product leaders, designers,... ...repeatable practices such as evaluation harnesses, code-review agents...Temporary workLocal area
Slalom
San Diego, CA3 days ago $140k - $180k
...traded HealthTech company using AI, advanced imaging, data science, and cloud software to transform how complex spine... ...clinical data, imaging, planning, automation, and manufacturing to improve... ...for a Senior Backend Software Engineer to build and evolve the backend...Kismet Search
San Diego, CA14 hours ago- Senior Applied AI EngineerRole SummaryWe are hiring... ...a Senior Applied AI Engineer to help us find and... ...applications, and workflow automations on the platforms we... ..., testing, CI/CD, and evaluation of AI output quality.... ...Five to eight years in software, solutions, data, or...Work at officeLocal areaImmediate start
Hologic
San Diego, CA22 hours ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Software Engineer — Ai Evaluation & Automation : 26-02760. Be the first to apply!
Related searches
- software engineer full time San Diego, CA
- graduate software developer no experience San Diego, CA
- software engineer healthcare San Diego, CA
- network software engineer San Diego, CA
- software engineer internship San Diego, CA
- senior software engineer San Diego, CA
- software system engineer San Diego, CA
- software developer San Diego, CA
- ngo software engineer San Diego, CA
- startup software engineer San Diego, CA