Senior Software Engineer — Task Ideation

About the Project

We build hard, verifiable command-line tasks used to evaluate frontier AI agents.

AI agents are good at the textbook. They still fail where experience matters: a default nobody questions, a rule nobody wrote down, a fix that looks right and isn't. Much of the work starts from real episodes from your own experience, such as a production incident, a performance problem, or a result that looked right but was wrong. These are captured through an AI-guided interview and turned into tasks.

Some engineers also pair with domain specialists (scientists, operations professionals, hardware engineers) and turn their expertise into working tasks.

Who we're looking for

• Core Task Engineer: senior generalist (backend, data or SRE), 5+ years at a real company, has handled production incidents and writes tests. Production Python plus Go, Rust, TypeScript or C++; Docker, pytest, advanced Linux. No specific domain needed.
• Systems & Storage Specialist: 6+ years working inside engines like Postgres, RocksDB, Kafka, Spark or the Linux kernel, not just using them. Production C, C++, Rust, Go or Scala; expert Linux (concurrency, I/O, ptrace).
• ML Systems & Research Specialist: ML infrastructure (GPUs, distributed training, serving), not notebook data science. MSc/PhD or 3–5 years in ML infra; PyTorch or JAX, CUDA or Triton.
• Security Researcher: AppSec, cryptography, forensics or firmware; 3+ years of CTF or 4+ years in industry. Python and C, Ghidra, QEMU. Published CTF challenges or credited CVEs are a strong signal.
• Red-Team / QA Reviewer: attacker mindset (CTF player, pentester, bug hunter); 4+ years in SWE, SRE or security, or 3+ years of CTF. Reviews tasks in every category and tries to pass the tests without solving the problem.
• Benchmark Infra Engineer: DevOps, SRE or platform, 5+ years; Docker/BuildKit, CI pipelines, cloud GPUs. Builds the environments tasks run on; does not write tasks.
• Calibration Lead: 7+ years; has led code review, QA or evaluation of technical work. Sets the quality bar and writes the guidelines. Excellent written English. Very few positions.

Responsibilities

Depending on your profile:

• Share real episodes from your work through AI-guided interviews, with concrete detail (tools, versions, files, error messages, numbers)
• Design and build tasks: Docker environment, instruction, reference solution and tests
• Pair with domain specialists to turn their expertise into working tasks
• Review tasks and try to break them: pass the tests without solving the problem

Project Details

Remote, task-based work in Vetto Arena. Compensation is paid per approved task.

This role is not about writing more code. It is about the hard problems you have solved, and turning them into tasks a program can check.