Senior Software Engineer | AI Evaluation

Role Description

We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks. Essentially, we're building an olympiad for AI models: the better the tasks, the more precisely we see where models are weak, and the faster they improve.

What you'll actually do:

• Create tasks and the criteria to evaluate them. Each task consists of:
• An isolated environment - an emulation of a developer's workstation: a Linux machine with dev tools (terminal, CLI), MCP servers (repository, task tracker, messenger, documentation, etc.), and a real web application codebase;
• A task - a prompt, just like in real work (a ticket link, a messenger conversation, anything). The agent figures it out on its own - reads the ticket, checks discussions, searches docs, explores the code. All of that context is also part of the task and needs to be created and thought through;
• Tests and evaluation criteria - how to verify the agent solved the task correctly.
• The work breaks down into two phases:
• Building the virtual company - following a high-level plan, you create a company: codebase, infrastructure, context (conversations, documentation, tickets). The result is a realistic environment with development history.
• Assembling and calibrating tasks - from intermediate states of the company, task drafts are assembled (based on a taxonomy). You turn a draft into a full task: write the prompt, define evaluation criteria, ensure the task is solvable and the evaluation is fair.
• Then an AI agent receives the task and tries to solve it the way a developer would - and you evaluate the result.

What the work looks like in practice:

• Iterating with an AI agent on tests - they should catch real problems, not miss bad solutions, not break on good ones;
• Reviewing code written by agents;
• Analyzing why an agent failed or succeeded;
• Designing edge cases and adversarial scenarios.

A significant part of the work is done together with AI - it's very hard to create tasks that are hard for frontier models without using frontier models.

Why this is hard:

• Frontier models are already good at coding.
• Creating a task that genuinely challenges the best models is non-trivial - you need to understand where models fail and what reveals the difference between a good and a bad solution.
• Tasks have many valid solutions.
• Tests must be neither too strict (breaking valid solutions) nor too lenient (passing bad ones).

Core tech stack:

• Backend: Python, FastAPI
• Frontend: JavaScript/TypeScript, React
• Infrastructure: Docker, Postgres, Kafka, Redis

You don't need to be an expert in every item, but you should be comfortable reading and reasoning about code across the stack - that's what lets you design realistic tasks, write meaningful tests, and evaluate results.

Growth:

• You start with task and dataset work, and if it goes well, we expect you to move into a project team - with a broader scope and deeper involvement.

Qualifications

• Strong software engineering experience: 8+ years in any modern programming language, with a willingness and ability to work primarily in Python in our projects;
• Hands-on experience using coding agents such as Claude Code and Codex as part of practical engineering workflows, while maintaining ownership of architecture, correctness, code quality, and delivery;
• Problem-solving and clarity of thinking;
• Ability to communicate technical decisions and reasoning;
• Practical understanding of Docker and development workflows;
• Overall readiness to contribute in a technical, execution-focused role.

Benefits

• Freelance collaboration, with a path into a project team if things go well;
• Flexible, fully remote schedule;
• Work at the leading edge of AI development, alongside a dedicated and dynamic team of experts;
• Projects with customers that are AI industry leaders and well-known household names;
• Friendly community.

Important Notice

Scam Alert Regarding Fake Job Postings: It has come to our attention that an individual or group is fraudulently impersonating Toloka to post fake jobs and solicit personal information from applicants. Please be aware:

• Our recruiting team will only contact you from an official "toloka.ai" email address. We will NEVER use Gmail, Yahoo, Tolokainc, toloka.inc, or other personal or seemingly business email accounts.
• We will never ask for your bank account details, credit card number, or any fees as part of the application or interview process.
• All legitimate job openings are posted on our official careers page:

If you see a suspicious job posting or have been contacted by someone you suspect is a scammer, please do not provide any personal information. Instead, report the incident to us directly at View email address on remotive.com and report the profile/post to LinkedIn. We are taking this matter very seriously and are working with the appropriate parties to resolve it. Thank you for your vigilance!