Software Engineer - AI Code Evaluation
Temporary
Gramian Consulting Group
About Gramian
Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.
About the Role
We are looking for Software Engineers to help train and evaluate next-generation AI models on real-world software engineering tasks. You will assess AI-generated code for correctness, quality, and design, create challenging programming problems with verified reference solutions, and evaluate model reasoning across architecture, debugging, and system design. This role is ideal for technically strong professionals who can identify engineering flaws and provide clear, actionable feedback to improve AI performance.
SENIORITY: Mid-Senior
LANGUAGE: Strong written English
Key Responsibilities
- Evaluate AI-generated code for functional correctness, code quality, maintainability, and sound software design.
- Create challenging coding problems, verified reference solutions, and comprehensive test cases.
- Review AI reasoning on software architecture, debugging, algorithms, and system design.
- Identify logical errors, inefficient implementations, edge cases, and architectural weaknesses.
- Provide detailed technical feedback and explanations to help improve AI model performance.
- Apply consistent evaluation criteria to ensure accurate and reliable assessments.
Requirements
At least 3 years of overall professional experience in Data Science or Software Engineering.
At least 1 year of professional experience at a leading technology, finance, or research company.
A bachelor’s degree in Data Science, Software Engineering, Computer Science, or a related field from a university in the United States, Canada, Australia, New Zealand, or the United Kingdom.
Strong programming and software engineering fundamentals.
Ability to evaluate code correctness, identify bugs, and assess software design decisions.
Strong analytical thinking and the ability to solve complex technical problems.
Excellent written English and the ability to communicate technical feedback clearly.
Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the Software Engineer - AI Code Evaluation in Remote vacancy
- ELITEBRAIN is seeking a Junior Software Engineer for an AI code evaluation program. This fully remote, contract role involves evaluating AI-generated code, creating challenging programming problems, and supplying detailed feedback to improve AI model performance. Ideal...SuggestedRemote jobContract work
$150 per hour
Junior Software Engineer - AI Code Evaluation (Remote) Contract Our client is a leading organization in the artificial intelligence industry, collaborating with top technology companies and AI research teams to advance the development of next-generation AI systems. The...SuggestedRemote jobHourly payContract workFor contractors$312k - $416k
...least one year of professional software engineering experience at a leading... ...data structures, debugging, and code optimization. A solid grasp... ...Preferred: familiarity with AI-generated code or AI-assisted... ...country. Responsibilities: Evaluate AI-generated code for...SuggestedHourly payFull timeContract workFor contractorsRemote workELITEBRAIN
New York, NY1 day ago- ...Role Overview Evaluate and benchmark the coding abilities of frontier AI models by reviewing AI-generated solutions, validating them against real-world software engineering tasks, and shaping high-quality evaluation datasets and benchmarks. This role suits experienced...SuggestedContract workFor contractorsRemote work
SaidGig
United Statesmore than 2 months ago - RemoteJobsOne is seeking a Junior Software Engineer for a fully remote, worldwide contractor role. This position is... ...Australia, Ireland, and New Zealand. As part of an AI lab partnership, you will evaluate AI-generated code, create coding problems, and help improve model...SuggestedRemote jobFor contractorsWorldwide
$100 - $150 per hour
...Role Overview Help improve AI coding agents by reviewing their work across real-world software repositories. You will assess whether... ...feedback will help create the evaluation signals used to strengthen... ...closely with researchers and engineers to convert qualitative technical...Hourly payFor contractorsImmediate startRemote workSaidGig
United States14 days ago$100 - $150 per hour
...Evaluate and improve AI coding models by reviewing generated code. Freelance · Remote · North America... ...training data, evaluations, and engineering talent. Engagement Details Compensation... ...looking for experienced, hands-on software engineers to help evaluate and...Hourly payPart timeFor contractorsFreelanceImmediate startRemote workTuring
San Francisco, CA1 day ago$65 per hour
...programming tasks for accuracy and difficulty Verify the logic and test cases within provided evaluation harnesses Assess whether coding environments effectively measure software engineering skills Walk us through your thought process while analyzing complex code structures...Contract workTemporary workFor contractorsRemote workTerac
San Francisco, CA3 days ago$135.2k
...Researching We're running a paid study on the coding environments and programming tasks used to... ...intelligence agents. Creating robust evaluation harnesses ensures that AI models are tested against realistic software engineering scenarios. This work directly feeds into improving...Hourly payContract workRemote work$50 per hour
...Role Overview Evaluate, benchmark, and help improve the coding capabilities of advanced AI models by assessing AI-generated solutions against real software engineering problems. You will review code for correctness and quality, reproduce and debug issues, design evaluation...Contract workFor contractorsRemote workSaidGig
United States11 days ago- ...Evaluate and improve AI-generated code for accuracy and quality. About Turing Turing is one of the world’s fastest-growing AI companies, accelerating... ...Role Overview We are looking for experienced Software Engineers to help evaluate, benchmark, and improve the coding...Contract workPart timeFor contractorsFreelanceRemote work
Turing
San Francisco, CA15 days ago $144.7k - $261.3k
...DescriptionAbout the TeamThe Evaluation Foundations team, part of Embodied AI's Scaling Foundations,... ...vehicle development. We engineer high-performance tools... ...in design and code reviews and work with Data... ...experienceExperience shipping production software and iterating quickly in...Full timeLocal areaRemote workWork from homeRelocation packageFlexible hoursGeneral Motors
Austin, TX1 day ago$144.7k - $221.4k
...About the Organization The Evaluation team builds and evolves... ...clear feedback for engineering and leadership, and help... ...autonomous driving software performance at interfaces... ...thoughtful system design, code reviews, testing,... ...Experience leveraging AI-assisted development and...Full timeWork at officeLocal areaWork from homeRelocationRelocation packageFlexible hoursGeneral Motors
Sunnyvale, CA3 days ago$119.8k - $234.7k
...Less than 25%Profession: Software EngineeringDiscipline:... ...Studio Code is where millions of developers... ...for a Senior Software Engineer to help build its next... ...to delegating work to AI. This is hands-on product... ...issues, and use telemetry, evaluations, and user feedback to guide...Ongoing contractLocal areaRemote work3 days per weekMicrosoft
Redmond, WA4 days ago- ...Senior Software Engineer — AI Coding Evaluator is a remote engineering review track for evaluating production code, debugging traces, and developer-facing AI outputs against real-world correctness standards. Reviewers reproduce failures, write the unit test the model should...Remote jobHourly payFor contractors10 hours per week
$388k
...a part of what’s next.AI and ML power innovation... ...for a hands-on senior engineer to build the frameworks... ...platform, model performance, evaluation, and vendor integration... ...need:Experience in software, AI/ML, or platform engineering... ...single use case.Strong coding skills in Python and at...Hourly payFull timeImmediate startFlexible hoursNetflix
Los Gatos, CA1 day ago$75 - $90 per hour
...5 - 10 years Location: United States | Canada + 21 more Help a top AI lab evaluate and improve large language models through security-focused coding tasks. Bring your software-engineering judgment and hands-on security experience to work involving vulnerabilities...Contract workRemote workBraintrust
San Francisco, CA15 days ago- A leading tech company is seeking an experienced software engineer to evaluate and refine AI-generated code, ensuring it is efficient and reliable. The role involves collaboration with cross-functional teams and developing agents for code verification. Candidates must...Remote jobFor contractors10 hours per weekFlexible hours
- To support a research initiative on AI evaluation tasks, remote software engineers will review and assess programming challenges and evaluation harnesses,... ...assess the quality of realistic programming tasks Evaluate coding environments and technical harnesses for AI testing...Contract workRemote work
- Jobgether SRL is seeking an experienced software- engineering professional for a fully remote, part-time independent contractor role. You will curate datasets, evaluate AI-generated code, and develop evaluation frameworks across multiple languages and lifecycles. The role...Remote jobPart timeFor contractors
- ...Role Overview Design and evaluate high-quality datasets and evaluations... ...large language models for code. You will work with... ...across multiple languages, assess AI-generated implementations for... ...model capabilities across the software engineering lifecycle. Key Responsibilities...For contractorsRemote work10 hours per weekFlexible hours
SaidGig
Canadamore than 2 months ago $1,000 per unit
Role Description We're looking for experienced software engineers to build AI coding tasks from a codebase they know well. US$1,000 per task, and you... ...coding tool. ~Experience writing tests, rubrics or evaluation criteria, ideally for AI models. ~You can explain a...Contract workTemporary workRemote workAskable
United States1 day ago- ...development of high-quality datasets and evaluation pipelines that improve and benchmark large language models for code generation and software engineering tasks. You will curate and author reference code, evaluate and refine AI-generated solutions across multiple programming...Full timeFor contractorsRemote work10 hours per weekFlexible hours
SaidGig
United Statesmore than 2 months ago $70 - $150 per hour
...Role Overview Help shape how frontier AI systems are trained and evaluated by applying production software engineering judgment to real-world code, designs, requirements, and failure modes. This is an ongoing talent pool for AI training and evaluation projects, rather...Hourly payImmediate startRemote workSaidGig
United Statesa month ago- ...Analyze GitHub issues and evaluate LLM performance on real-world code. Job Title: Senior Software Engineer – LLM Evaluation & Repository Validation About the projects... ...Turing is one of the world’s fastest-growing AI companies accelerating the advancement and deployment...Part timeFor contractorsFreelanceRemote work
Turing
San Francisco, CA3 days ago - To support the development of cutting-edge AI-driven systems, the full-time Software Engineer, Coding Agent will build and modify coding agent harnesses, analyze... ...agent harnesses at scale Familiarity with agent evaluation frameworks and inference performance metrics Hands...Full timeRemote work
$139.2k - $235.2k
...ship better, more secure software faster. The same principles... ...team works: we embrace AI as a core productivity... ...role As a Senior Backend Engineer on GitLab's Security Factory: Code Security team, you help developers... ...tooling that develops, evaluates, and ships those...Full timeLive inRemote workFlexible hours$50 - $100 per hour
...Apply advanced software engineering and problem-solving expertise to code generation and model evaluation work that helps improve how next-generation AI systems learn, reason, and perform. This remote contract role centers on real-world coding challenges, codebase improvement...Hourly payContract workFor contractorsRemote workSaidGig
United Statesa month ago- ...Role Title: Research Engineer - Code Generation & Model Evaluation Role Type: Contractor Location: Remote micro... ...to help train next-generation AI systems. Your work will shape how models... ...or participation in collaborative software projects. Strong analytical...Temporary workFor contractorsRemote work
micro1
Remote29 days ago $60 per hour
...Senior Lua Developer (Roblox) – AI Code Evaluation An enterprise client is seeking experienced Lua Developers to support AI model training... ...AI models by leveraging the expertise of experienced software developers. Selected freelancers will help evaluate AI-generated...Weekly payContract workFreelanceRemote work10 hours per weekLifted, an Upwork Company™
United States4 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Software Engineer - AI Code Evaluation. Be the first to apply!
Related searches
- junior software developer Remote
- scientific software engineer Remote
- intel software engineer Remote
- vice president software engineering Remote
- director software engineering Remote
- software engineer Remote
- software sales engineer Remote
- software engineer mainframe Remote
- software engineer student Remote
- senior robotics software engineer Remote