AI Evaluation Engineer
Virtual Vocations Inc
To support the advancement of AI systems for public safety agencies, the full-time remote AI Evaluation Engineer will design and maintain automated evaluation pipelines, develop prompt engineering strategies, and analyze model performance to ensure reliability and accuracy in real-world applications. Key Responsibilities Design, build, and maintain automated AI evaluation pipelines for production LLM applications Develop prompt engineering strategies and iterate on prompts using quantitative evaluation methods Analyze production AI behavior with Python and SQL to identify opportunities for improvement Required Qualifications Must have US citizenship and pass FBI fingerprint and background check in multiple states 3+ years of experience in software engineering, machine learning, data science, or a related technical field Experience designing evaluation metrics and interpreting AI model performance Strong Python development and SQL skills with experience analyzing large datasets Experience building or supporting production LLM or Generative AI applications
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the AI Evaluation Engineer in United States vacancy
- ...Position Summary As an AI Evaluation Engineer at Judi Health, you will build the testing frameworks, metrics, and tooling used to assess the safety, reliability, and accuracy of AI models and autonomous agents in production. This role bridges the gap between model development...SuggestedLocal areaFlexible hours
Capital Rx
Denver, CO1 day ago $150k - $250k
Role Description As a Senior AI Engineer focused on Agentic Evaluation and Verification and Validation (V&V), you will join the AI and Data Science team within Slingshot’s Research and Development organization. You will contribute to advancing how intelligent systems are...SuggestedFull timeRemote workSlingshot Aerospace
Remote3 days ago$150k - $250k
About Distyl Distyl is an applied AI technology company partnering with the world’s most... ...For At Distyl, we build AI systems using Evaluation‑Driven Development—a process where... ...AI behavior in production. AI Evaluation Engineers design and implement the evaluation systems...SuggestedWork at office3 days per week$350k
Mirendil is a tech-first company located in San Francisco, California, seeking a Research Engineer to develop evaluation infrastructure for AI models. You'll design frameworks that measure model capabilities, implement automated pipelines, and create workflows for inspecting...Suggested$80 per hour
A tech consultancy seeks an AI Specialist to engage with project-based opportunities focused on evaluating and improving AI systems. Responsibilities include reviewing coding tasks, writing functional tests, and analyzing AI failures. Candidates should have a degree in...SuggestedContract workFlexible hours$150k - $250k
Distyl is seeking an AI Evaluation Engineer to design evaluation systems for AI-driven solutions. Located in San Francisco, this position requires strong Python skills and a commitment to utilizing AI tools to enhance evaluation processes. The role offers a competitive...Work at office3 days per week$50 per hour
...Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is... ...is NOT: Not data labeling. Not prompt engineering. Not writing code from scratch - the agent...Permanent employmentTemporary workPart timeMindrift
Remote7 hours ago$100 per hour
...Description We’re looking for a highly experienced software engineer (SR+) to help evaluate the quality of interactions with modern coding agents such... ...model thinks like a great engineer. You will assess how AI coding agents behave in real-world scenarios — focusing on...Full timeImmediate startG2i Inc.
Remote5 days ago$40 per hour
A leader in AI training for cybersecurity is seeking experienced cybersecurity professionals to evaluate AI-generated content and solve technical problems. This role offers full-time or part-time remote work with the flexibility to choose projects and work hours. Candidates...Hourly payFull timePart timeRemote workDataAnnotation
Madison, WI2 days ago$40 per hour
A cybersecurity firm is seeking experienced professionals to evaluate AI-generated security content and solve technical problems. This role is flexible, allowing you to choose projects and work on your own schedule. Candidates should have over 2 years of hands-on cybersecurity...Hourly payRemote workFlexible hoursDataAnnotation
Honolulu, HI3 days ago- A cybersecurity consultancy is seeking experienced individuals to enhance AI capabilities by evaluating cybersecurity content and solving security-related challenges. You will play a pivotal role in validating AI outputs and providing critical feedback to advance cybersecurity...Remote workFlexible hours
DataAnnotation
Phoenix, AZ2 days ago $40 per hour
...cybersecurity professionals to join our team to help train AI models. In this role, you will evaluate AI-generated security content, solve technical... ...penetration testing, red teaming, incident response, detection engineering, DFIR, malware analysis, threat intelligence, or...Hourly payFull timePart timeRemote workDataAnnotation
Hartford, CT2 days ago$40 per hour
A cybersecurity company is seeking experienced cybersecurity professionals to join their team. You will evaluate AI-generated security content, solve technical problems, and provide critical feedback to enhance AI systems. This role is remote, flexible, and offers hourly...Hourly payRemote workFlexible hoursDataAnnotation
Oregon, WI2 days ago$40 per hour
A cybersecurity solutions company is seeking experienced professionals to evaluate AI-generated security content and solve technical security problems. Candidates should have over 2 years of hands-on experience in cybersecurity and coding skills, with strong writing and...Hourly payRemote workFlexible hoursDataAnnotation
Saint Paul, MN2 days ago$40 per hour
A leading tech firm is seeking experienced cybersecurity professionals to evaluate AI-generated content and solve technical problems. In this remote role, you will work to enhance AI systems, requiring 2+ years of hands-on experience in cybersecurity and some coding knowledge...Hourly payRemote workFlexible hoursDataAnnotation
Iowa, LA2 days ago- AI Evaluation Engineer Location: Remote (Hybrid opportunity if live in Denver, CO) Type: Full-Time Clearance: Must have US citizenship and pass FBI fingerprint and background check in multiple states About GovWorx GovWorx is helping public safety rise to today's greatest...Full timeLive inRemote work
$40 per hour
A cybersecurity startup is seeking experienced cybersecurity professionals to evaluate AI-generated content and solve technical problems. This role offers the flexibility to work remotely while contributing to innovative security AI models. Candidates should have 2+ years...Hourly payRemote workDataAnnotation
Bismarck, ND2 days ago- A leading cybersecurity firm is seeking experienced professionals to evaluate AI-generated security content and solve complex cybersecurity problems. This role is fully remote, allowing candidates to choose projects and work on their own schedule. Ideal candidates should...Remote work
DataAnnotation
Lincoln, NE2 days ago $40 per hour
A cybersecurity company is seeking experienced professionals to evaluate AI-generated security content and contribute to building reliable AI tools. This remote role offers flexibility to choose projects and work hours, with pay starting at $40+ per hour. Ideal candidates...Hourly payRemote workDataAnnotation
Helena, MT3 days ago$50 - $75 per hour
A leading AI technology company is seeking proficient programmers to advance AI development remotely. Responsibilities include designing solutions for coding challenges and evaluating AI-generated code. Candidates should have strong English fluency, preferred experience...Remote jobHourly payFlexible hours- Prolific, located in Raleigh, North Carolina, is seeking an AI & Machine Learning Engineer specialized in training and evaluating large language models. The ideal candidate will have a strong background in deep learning and familiarization with contemporary frameworks....Remote jobFlexible hours
$200k - $250k
Fieldguide, a remote-first company based in San Francisco, is seeking a Senior AI Engineer to lead the evaluation infrastructure for AI agents. This role focuses on enhancing the reliability and performance of AI systems to support top accounting and consulting firms....Remote jobFlexible hours$40 per hour
DataAnnotation is seeking an experienced Chemical Engineer to evaluate and improve AI chatbot models. Key responsibilities include measuring the progress of these models and ensuring accuracy in chemistry-related queries. Ideal candidates possess strong backgrounds in general...Remote jobHourly payFor contractorsFlexible hours$40 per hour
DataAnnotation is seeking an experienced Chemical Engineer to evaluate AI models by providing complex chemistry questions and assessing their outputs. You'll have the flexibility to choose projects and work from home, starting at $40+ per hour. The ideal candidate has a...Remote jobHourly payWork from home$50 - $75 per hour
...technology company is looking for proficient programmers to develop AI systems. Enjoy fully remote work from anywhere in the US with... ...designing coding problems for AI, writing clear code, and evaluating AI-generated outputs. Ideal candidates should have strong programming...Remote jobHourly payFlexible hours$50 - $75 per hour
A leading AI development company in the United States is seeking proficient programmers to contribute to cutting-edge AI systems. This... ...for training AI systems, writing high-quality code, and evaluating AI-generated outputs. The position requires fluency in English and...Remote job- Luma AI is seeking a Research Engineer to design and scale the infrastructure powering model evaluation efforts. You will build pipelines, metrics, and automated systems that connect model output, evaluation, and improvement across research, engineering, and product teams...
$40 per hour
DataAnnotation is seeking a Chemical Engineer to help train AI models. In this role, you will evaluate the logic of AI chatbots and assess their performance in response to complex chemistry questions. The ideal candidate should possess solid knowledge of chemistry concepts...Remote jobHourly payFor contractors$80 per hour
Prolific is looking for an AI & Machine Learning Engineer to evaluate AI architectures, audit code, and refine frameworks. Ideal candidates will hold a BS, MS, or PhD in relevant fields and have experience in ML model development. Responsibilities include providing critical...Remote jobHourly payFlexible hours$40 per hour
DataAnnotation is seeking a Chemical Engineer to assist in training AI models related to chemistry. Responsibilities include evaluating AI outputs and ensuring the quality of the models. Ideal candidates should be fluent in English, detail-oriented, and possess a solid...Remote jobHourly payFor contractors
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Evaluation Engineer. Be the first to apply!
Related searches