AI Evaluation Engineer

To support innovative AI evaluation projects, the part-time AI Evaluation Engineer will create realistic developer environments, design challenging tasks for AI agents, and write tests to verify their solutions, all while working remotely. Key responsibilities Build realistic developer environments with context and infrastructure for evaluating AI coding agents Design tasks and define evaluation criteria to ensure AI agents can effectively solve them Write and iterate on tests based on QA feedback to ensure robust evaluation of agent solutions Required qualifications 5+ years of experience in software development Proficiency in Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis Experience in writing functional and integration tests English proficiency at B2 level or higher