AI Evaluation Engineer: Coding Task Architect
United States Digital Space LLC
United States Digital Space LLC is building a dataset to evaluate AI coding agents by creating realistic developer environments with codebases, infrastructure, tickets, docs, and a believable development history. You will design tasks from intermediate states and craft what a solution must achieve to be considered solved. The role emphasizes writing tests that accept all valid approaches while rejecting incorrect ones, and iterating on tasks based on QA feedback to keep evaluations fair and #J-18808-Ljbffr United States Digital Space LLC
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the AI Evaluation Engineer: Coding Task Architect in New York, NY vacancy
$60 - $90 per hour
...technical talent with leading AI research labs.... ...Position: Software Engineering Expert Type: Contract... ...limits of today's best AI coding agents. Build... ...limitations. Review tasks created by peers and provide... ...AI training , model evaluation, or benchmark/task authoring...SuggestedContract workSummer workRemote work- ...Description Senior Software Engineer Job Type: Contractor (... ...to support an AI training project by creating... ...learning environments that evaluate AI models on complex software engineering tasks using Model Context... ...tasks accurately measure coding ability, problem-solving...SuggestedRemote jobFor contractors
- ...consulting opportunity for experienced software engineers with strong Python skills. You will design realistic, multi-step coding tasks and build verifiable Python reference... ...environment setup and tests. The role evaluates AI coding assistants and requires rigorous documentation...SuggestedRemote jobFull time
$125k - $150k
...our team. SUMMARY The AI Engineer builds agent-based systems... ..., reporting, and operational tasks. Develop agent orchestration... ...tasks. Agentic Coding Build CodeAI agents that... ...Improve agent reliability through evaluation pipelines, observability metrics...SuggestedFull timeWork at officeLocal area- Mindrift is offering a project-based opportunity to craft and evaluate AI coding tasks. You will build a realistic developer environment—a virtual company with a codebase, infrastructure, tickets, docs, and conversations—to test AI agents. You design tasks from intermediate...Suggested
$175k - $215k
...AI Engineer At BBH, partnership is more than a form of ownership—it... ...financial services data schemas Evaluate and improve AI output quality... ...cleanly Participate in code reviews and maintain high... ...source LLMs for domain-specific tasks Open-source AI...Local areaBrown Brothers Harriman
New York, NY4 days ago$150k - $275k
...Health is building the AI-powered financial... ...machine learning engineer with 5+ years of... ...engineer, you'll architect and build the intelligent... ...Create evaluation frameworks and feedback... ...Collaborate on code reviews and technical... ...craft, not just a task, and you're...Full timePrivate practiceWork at office3 days per weekJoyful Health
New York, NY5 days ago- ...Staff/Senior AI Engineer, AI for CodeAt JetBrains, we build developer... ...millions of engineers. The AI for Code team works on the next... ...plan and execute multi-step tasks, collaborate with developers,... ...context engineering, tool use, evaluation, feedback loops, and product...Work at officeLocal areaRemote workWork from homeRelocation packageFlexible hours
JetBrains
New York, NY1 day ago $140k - $240k
...autonomous organization of AI agents that run the... ...firms and building the engine they run on to become the... ...lawyers in third-party evaluations. We're targeting mid-... ...agents that perform legal tasks with expert-level... ...production environments; strong coding skills Demonstrated...Work at officeRelocation packageFlexible hoursAugust Law
New York, NY20 hours ago$220k - $300k
...move fast and ship daily. AI-assisted engineering isn't a talking point here;... ...our Manager Agent platform — task generation, review... ...store's own data Build the evaluation harness for both offline and... ...and experience using agentic coding tools for large-scale parallel...Afternoon shiftSimplex
New York, NY12 days ago- ...New York, NY, where you will ensure our AI-powered features are high quality and reliable... ...Your responsibilities include designing evaluation frameworks, building automated tests, and... ...in quality assurance and evaluation tasks related to AI, along with exceptional communication...
- Description AI Solutions Engineer to help lead innovation around GitHub Copilot... ...with MCP servers or AI coding assistants and can bring real... ...and MCP server integrations. Evaluate AI‑assisted code generation... ...‑of‑concepts for automating tasks such as version upgrades, dependency...Full timeRemote workFlexible hours
$142.32k - $213.48k
...Senior GenAI Platform Engineer to spearhead the design... ...-edge Generative AI platforms. This role is... ...lines of business by architecting and delivering high-impact... ...projects through in-depth evaluation of complex business... ...and enforce robust coding standards, testing methodologies...Full time$85 per hour
...and technical talent with leading AI research labs. Headquartered in... ...Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience) Type: Contract... ...coding agents to complete and evaluate complex infrastructure engineering tasks. Review model-generated implementations...Contract workSummer workRemote workMercor
New York, NY3 days ago- ...product management, design, and engineering together with the autonomy to... ...Are You are an exceptional AI-native software engineer who... ...build and how you build. You use coding agents and other AI... ...constraints, establish tests and evaluations, and review generated output...Permanent employmentFull timeWork experience placementLive inWork at officeLocal area
- Mercor partners with a leading AI research lab to support a Frontier Code Agents project, focusing on realistic data engineering workflows and model evaluation. Contributors help evaluate and improve... ...data systems. Compensation is task-based and distributed upon acceptance...
$190k
...Digital Platforms, AI at Scale,... ...ll DoLead AI Tech Architects - AI Platforms at... ...relationships across engineering, data, product, and... ...operational complexity.Evaluate emerging Ai frameworks... ..., including task decomposition, memory... ...AI-assisted coding and development tools...Work at officeLocal area$125k - $135k
...are seeking an Intermediate AI Developer to build intelligent... ...will work closely with prompt engineers and crossfunctional teams to... ...solutions Experience using AI coding agents such as AMP and Claude... ...Exposure to model evaluation observability and monitoring...Full timeLocal area$204k - $255k
...Collibra’s Unstructured AI TeamWork at the forefront of context engineering - shaping how AI... ...production-grade backend code (Python, FastAPI) — you... ...workflows.Knowledge of model evaluation best practices.... ...solve for data at scale, architecting the high-performance pipelines...Work experience placementWork at officeFlexible hours2 days per week$204k - $255k
...Intelligence teamCollibra is the context engine that powers trustworthy enterprise AI, giving LLMs and agents the... ...world.Building, instrumenting, and evaluating AI/ML models and treat a quiet, no... ...architecture, such as sandboxed code execution, tool permissioning, safely...Work experience placementWork at officeFlexible hours2 days per week$175k
...SummaryAssured Guaranty is seeking a Senior AI Software Engineer to build production-grade AI... ...development tools (e.g., Copilot, Claude Code) to accelerate deliveryCandidate QualificationsBachelor... ...applications (prompting, integration, evaluation)Exposure to agent-based systems or...$400 per month
Obsidian is seeking contributors for a Frontier Code Agents project, focused on evaluating AI coding models in fraud and risk engineering. Candidates will use AI coding tools to handle complex tasks and provide technical assessments. The role requires 2+ years of experience...$200k
...in data, analytics & AI consulting, dedicated... ...data scientists, data engineers, and digital experts are... ...orchestration layers, evaluation frameworks, guardrails... ...cases.Platform Stack: Architecting and deploying solutions... ..., RAG, and agentic coding — and in cloud-native...$155k - $215k
...seeking a hands-on Developer Engagement Engineer to join the Firmwide developer tools group... ...engineering, developer experience, and AI coding tool adoption. You'll help ensure... ..., Codex, Cursor, Claude Code)Ability to evaluate where AI tools add value — and where they...Temporary workWork at office$175k - $250k
...Rates business is seeking an exceptional AI Engineer to drive the development and... ...and operational efficiencyDevelop agentic coding systems that can autonomously generate,... ...current with emerging AI technologies and evaluate their applicability to Global Rates business...Full time$207k - $340k
...York, NY.Responsibilities: As a Principal AI Engineer you will be setting the technical... ...best practices for experimentation, model evaluation, and responsible AI within the asset generation... ...or related technical field involving coding, or equivalent technical experience....For contractorsWork experience placementWork at officeFlexible hours$96k - $181k
...RoleKeyBank is seeking a Lead AI Engineer to design, build, and... ...on engineer responsible for architecting, developing, and delivering... ...their time designing, building, coding, integrating, and optimizing... ...experiences.Develop, test, evaluate, and optimize prompts, workflows...Full timeWork at officeRemote workFlexible hours$163.2k - $220.8k
...Sonsini is looking for a Senior AI Security Engineer to join the Security... ...security — a rare opportunity to architect how one of the world’s... ...designing secure AI pipelines, evaluating MCP server integrations and... ...firm — including secure coding practices for AI-adjacent code...Full timeWork experience placementRemote workWorldwideShift work$198.72k - $260.82k
...'re Oscar. We're hiring a Staff Applied AI Engineer to join our AI team.Oscar is the first health... ...the cutting edge of the AI ecosystem, evaluating and integrating emerging technologies to... ..., rapid prototyping, and hands-on coding to deliver real-world impact.Experience...Full timeWork at officeFlexible hours$210k - $260k
...collapse it into a single AI-native platform that... ...a ticket, while engineering teams retain the governance... ..., RAG design, evaluation frameworks, and the interaction... ...deeply technical - architect and implement core AI... ...Active daily user of AI coding assistants - this is a...Full timeLocal areaHome office
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Evaluation Engineer: Coding Task Architect. Be the first to apply!
Related searches