Software Developer for AI Trace Evaluation

$70 - $90 per hour

SaidGig

Role Overview
Review AI-assisted software development traces that support the training and evaluation of advanced AI models. Assess complete coding sessions created with AI-enabled developer tools, including code correctness, workflow quality, reasoning, and adherence to sound engineering practices. Deliver clear written feedback using established evaluation criteria.
Key Responsibilities
Evaluate the quality and correctness of end-to-end AI-assisted coding sessions.
Judge coding workflows, technical reasoning, and implementation decisions for accuracy and best-practice alignment.
Provide concise, rubric-based written feedback on reviewed traces.
Qualifications
At least 3 years of professional software development experience.
Hands-on experience with AI-assisted coding tools and agentic or specification-driven workflows, such as Cursor, GitHub Copilot, Claude Code, or similar tools.
Strong code-reading and debugging abilities across full-stack or backend systems.
Ability to assess multi-step coding trajectories for correctness and engineering best practices.
Preferred Qualifications
Experience with Kiro or Amazon CodeCatalyst.
Previous experience evaluating or grading AI-generated code.
Contributions to developer tooling.
Work Terms
Remote role available to candidates located in the United States.
Hourly engagement.
Compensation
$70 to $90 per hour.

Apply

Vacancy posted 4 days ago

Similar jobs that could be interesting for youBased on the Software Developer for AI Trace Evaluation in Remote vacancy

Senior Software Engineer AI Evaluation & Benchmarks (Python)
$80 - $100 per hour
...verification for this role. What You'll Be Doing Design and build the coding benchmarks and evaluation pipelines used to test frontier AI models on real software engineering work: Design coding benchmarks that evaluate frontier models on real-world programming...
Suggested
Remote job
Full time
Contract work
For contractors
G2i
Miami, FL
1 day ago
Staff Software Engineer, Simulator Evaluation
$238k - $302k
...physical dynamics, and state-of-the-art Generative AI to create a training ground for the Waymo Driver. The Simulator Evaluation team faces the ultimate data challenge: How... ...is "real"? We are looking for a Staff Software Engineer to act as the Technical Architect for...
Suggested
Full time
Remote work
Shift work
Waymo
San Francisco, CA
20 hours ago
AI Engineer, Evaluation
$150k - $250k
...About Distyl AI Distyl is an applied AI technology company... ..., we build AI systems using Evaluation-Driven Development —an approach... ...alongside the AI system itself Develop and maintain evaluation... ...What We Require ~2+ years of software engineering experience ~...
Suggested
Full time
Work at office
Flexible hours
3 days per week
Distyl Ai
Remote
1 day ago
Senior Cloud Engineer
$122k - $204k
...TechOps and work closely with developers, product engineers, and other... ...and RCAs, while continuously evaluating systems against industry best... ...capability using generative AI tools (e.g., Gemini,... ...meaningful metrics, logs, and traces to reduce alert noise and drive...
Suggested
Flexible hours
Fandom
San Francisco, CA
4 days ago
Application Security Engineer IV - AI Harness
...monitoring, and improving an AI-enabled AppSec harness used to evaluate application and... ...evidence is repeatable, developer workflows remain aligned... ...Science, Cybersecurity, Software Engineering, Information... ...understanding of logs, metrics, traces, health checks, alert thresholds...
Suggested
Temporary work
Work at office
Home office
Flexible hours
3 days per week
Edward Jones
Tempe, AZ
3 days ago
Software Engineer (Python/C++) - Code Evaluation | Remote
...Role Responsibilities Review and refine AI-generated prompts, responses, and code Validate algorithms and software concepts for technical accuracy Provide structured... ...language Support benchmarking efforts to evaluate and compare model capabilities...
Part time
Remote work
Crossing Hurdles
United States
1 day ago
AI Evaluation Engineer
$180k
...: Guiding GenAI professionals towards their dream careers AI Evaluation Engineer $180,000 Remote (US-based) Are you passionate... ...eval into every engineer’s workflow. Requirements Strong software engineering background (TypeScript a plus) Deep experience...
Full time
Remote work
DeepRec.ai
Denver, CO
4 days ago
Staff Software Engineer, Autonomy Evaluation
...Our partner is looking for a Staff Software Engineer, Autonomy Evaluation based in the United States. This... ...workflows and release decisions. Develop and apply new statistical, machine learning... ...How Jobgether works: We use an AI-powered matching process to ensure...
Full time
Remote work
Work from home
jobgether
United States
1 day ago
Sr Lead Software Engineer - Cloud / ML / GenAI
...products. As a Senior Lead Software Engineer at JPMorgan Chase... ...Learning and Generative AI Engineer in Public Cloud Engineering... ...through training, evaluation, deployment, and ongoing optimization... ...engineering techniques, tracing, evaluations, guardrails, and...
Full time
JPMorgan Chase & Co.
Remote
1 day ago
Senior software engineer, AI platform
...Watershed is building the AI suite for companies to measure... .... We're looking for software engineers to help build the... ...powers Watershed's products Develop the observability and tracing layer for agent decisions,... ...it possible to debug, evaluate, and improve agent behavior...
Full time
Work at office
Remote work
Watershed
San Francisco, CA
1 day ago
Remote Senior Python Engineer - LLM Evaluation (US-based)
...leading research accelerator for frontier AI labs and a trusted partner for global... ...plus top AI researchers who specialize in software engineering, logical reasoning, STEM, multilinguality... ...What Does a Typical Day Look Like? Evaluate and refine AI-generated code across...
Remote job
For contractors
Flexible hours
Turing
Chicago, IL
2 days ago
Senior Software Engineer, AI Platform
$226k - $306k
...Rakuten Viber, Mixpanel’s AI-first digital... ...that is essential for developing AI features with confidence... ...experienced and driven Senior Software Engineer to join our... ...persistent memory. Evaluation & Reliability: Build... ..., and distributed tracing across all deployed AI...
Full time
Contract work
Remote work
Mixpanel
Remote
20 hours ago
Principle Software Engineer, AI Observability & Evals Platform
$230k - $270k
...the real world, helping developers move from prototypes to production-ready AI agents that teams can... ...platform for building, evaluating, deploying, and operating... ...of AI systems. From tracing and annotation to run rules... ...a Principal/Lead level Software Engineer to join the...
Full time
Work at office
Flexible hours
LangChain
Remote
1 day ago
Senior Software Engineer, AI Platforms and Products
$140k - $160k
...Department Overview The New AI Products and Platforms... ...Times. We are hiring a Senior Software Engineer to help build the backend... ...and workflow management, evaluation and tracing, and monitoring for cost,... ...experiences. Design and develop retrieval-augmented generation...
Full time
Work at office
Local area
Flexible hours
2 days per week
The New York Times Company
Remote
1 day ago
Senior Software Engineer, App Platform
...the future of physical AI. Founded in 2017 and now... ...the role Design, develop, and maintain secure and... ...testing strategies, and evaluate, integrate, and maintain... ..., and distributed tracing. Monitor platform performance... ...of experience in a software engineer role with a...
Full time
For contractors
For subcontractor
Casual work
Work at office
Remote work
Day shift
Applied Intuition
Remote
20 hours ago
Senior Software Engineer, Data Platform (Canada)
...World. TRM Labs provides AI-powered intelligence... ...enable investigators to trace illicit activity, build... ...services. As a Senior Software Engineer on the Data Platform... ...of blockchains. Develop complex ETL pipelines... ...leverage You will be evaluated on applied AI fluency...
Remote job
Full time
Worldwide
Trm Labs
United States
20 hours ago
Principal Engineer Tech Lead, Embodied AI & Off-Board Performance Evaluation
$200k - $275k
...the forefront of the physical AI revolution. Motional isn’t... ...is the crucial bridge between software development and real-world deployment... ...understanding, intelligence evaluation, and root cause analysis.... ...Vehicle Performance Metrics developer. You will lead the...
Full time
Work at office
Local area
Remote work
Motional
Massachusetts
27 days ago
Product Designer / AI-Native Engineer for AI Evaluation
$70 - $80 per hour
...Role Overview Design evaluation tasks and senior-level scenarios that test AI systems used for rapid prototyping, product... ..., natural-language-driven software development and rapid prototyping... ...research, and usability testing. Develop scenarios that use tools such as...
Hourly pay
Remote work
SaidGig
United States
29 days ago
Senior Applied AI Engineer
$160k - $190k
...looking for a Senior Applied AI Engineer to help build and ship... ...rapidly prototype solutions, evaluate approaches, and take the best... ...: ~7+ years of professional software engineering experience, including... ...with LLM observability and tracing (LangSmith, Langfuse, Arize, or...
Remote job
Full time
Work at office
Visa sponsorship
Work visa
Flexible hours
Tango
Remote
1 day ago
AI/ML Research Engineer, LLM Training & Evaluation
...of technical experts in LLM training, post-training, and evaluation systems. As an AI/ML Research Engineer, LLM Training & Evaluation , you will... ...and ensure consistency across model comparisons Software / Data Engineering Proficiency in Python and strong...
Full time
Innodata
Remote
a month ago
Applied AI Engineer
...About Sully.ai Sully.ai is building the AI operating system for healthcare.... ...job. What you’ll do Build and scale evaluation systems Design and scale automated evaluation... ...where clinician feedback and production traces drive targeted, evidence-grounded...
Full time
Work at office
Remote work
Night shift
Sully.ai
Remote
1 day ago
Applied AI Engineer, Inference
$188k - $275k
...CoreWeave is The Essential Cloud for AI™. Built for pioneers by... ...tuning serving configurations, evaluating runtime features, and... ...changes against representative traces and benchmarks. Design and... ...and provides the opportunity to develop innovative solutions to complex...
Permanent employment
Full time
Temporary work
Casual work
Work at office
Flexible hours
Core Weave
Remote
1 day ago
Staff Software Engineer (backend)
$230k - $250k
...workflow platform applies AI, automation, and... ...still a small but growing software engineering organization... ...issues, and use logs, traces, metrics, and tests to... ...We invest heavily in developer experience and care deeply... ...tests. We’re constantly evaluating and refining our use of...
Full time
Live in
Work at office
Remote work
Tines
United States
1 day ago
Staff Frontend Software Engineer, AI Product: Observe by Snowflake
...this new era, we seek AI-native thinkers across... ...and store logs, metrics, traces, and events on an open,... ...hiring a Staff Frontend Software Engineer on the AI... ...time integrations, and developer tooling come together to... ...to the CLI tooling and evaluation pipelines, defining the...
Full time
Immediate start
Snowflake
Remote
1 day ago
Applied AI Engineer
$180k - $240k
...About Sauna by Wordware Most AI products hand you a coworker... ...time shipping production software. ~ You've built and deployed... ...using plugins, auth flows) ~ Evaluation: success metrics, regression... ...Familiarity with LLM ops, tracing, observability, and failure handling...
Full time
Live in
Work at office
Relocation
Visa sponsorship
Flexible hours
Sauna
Remote
1 day ago
Senior Applied AI Engineer
...this new era, we seek AI-native thinkers across... ...measurable quality metrics, evaluation frameworks, and golden... ..., and actively develop their skills and careers... ...the loop from production traces and user feedback back... ...years of professional software engineering experience....
Full time
Snowflake
Remote
1 day ago
Applied AI Engineer, Kernel Performance
$2,000 per month
...intelligence. We co-design chips, racks, software, and manufacturing to deliver... ...breakthroughs will come from AI systems that can understand... ...efficiency. Turn profiler traces, simulation, hardware counters... ...roadmap. Continuously evaluate new model releases and deploy...
Full time
Work at office
Relocation package
Etched
Remote
1 day ago
Applied AI Engineer, GTM Growth Engineering
...GTM Growth Engineering builds AI-native products that help OpenAI... ..., agent behavior, feedback, evaluation, experimentation, and appropriate... ...Our work brings together software engineering, applied AI, product... ...agent behavior using production traces, user feedback, evaluation,...
Full time
OpenAI
San Francisco, CA
1 day ago
Senior Software Engineer, Agent Platform (AI for the Planet)
$146.88k - $220.32k
...the Allen Institute for AI working on AI for the... ...: EarthRanger is a software solution from AI2 that... ...the docs they read, the traces they use when something breaks, and the evaluation tooling that tells them... ...is any good. We want a developer outside our team to get...
Full time
Contract work
Work at office
Local area
Flexible hours
Weekend work
The Allen Institute For Artificial Intelligence
Remote
20 hours ago
Staff Software Engineer, Platform & Solutions
$170k - $205k
...seeking an experienced Staff Software Engineer who can... ...security patterns, improving developer experience, and helping... ...GitOps, observability, AI, and developer tooling.... ...implementation. Evaluating vendors, frameworks,... ...including metrics, logs, traces, alerting, dashboards,...
Remote job
Full time
Temporary work
Work at office
Local area
Wpromote
Remote
20 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Software Developer for AI Trace Evaluation. Be the first to apply!