AI Product Design Evaluator & Criteria Architect
Mercor
Mercor is partnering with a leading AI research organization to engage experienced UI/UX and product designers for a project that evaluates how well AI systems perform real-world digital product design work. You will define what excellent work looks like by designing task-specific grading criteria and scoring samples with rigorous, written justifications, guiding evaluation rather than delivering designs. You will contribute to an objective assessment framework and collaborate with senior #J-18808-Ljbffr Mercor
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the AI Product Design Evaluator & Criteria Architect in New York, NY vacancy
- Mercor is partnering with a leading AI research organization to engage experienced UI/UX and product designers for a project focused on evaluating how well AI systems perform real-world... ...like: designing task-specific grading criteria and scoring completed work samples...Suggested
- Mercor is partnering with a leading AI research organization to engage experienced brand designers for a project focused on evaluating how well AI systems perform real-world brand... ...excellent work looks like by creating grading criteria and scoring samples with well-reasoned...Suggested
$120k - $140k
...Nexstar is seeking a senior AI Architect to lead our efforts to design, develop and deploy AI and... ...built on the architecture Evaluate and select LLM frameworks... ...consistency, establishing the criteria that determine when an agent is production-ready Define user-facing...SuggestedFull timeTemporary workLocal areaNexstar Broadcasting
New York, NY3 days ago- ...high-fidelity simulated work environments used to evaluate and improve AI agents on real marketing tasks. You will design creator and influencer tasks, specify the... ...requires, and write the reference answer and the criteria that separate a strong response from a plausible...Suggested
- ...seeking a hands-on field marketing professional to design and pressure-test AI training environments for evaluating marketing workflows. You will craft briefs,... ...required artifacts, and write reference answers and criteria, then review agent attempts against your standards...Suggested
- ...building high-fidelity simulated marketing analytics environments to evaluate and improve AI agents on real marketing work. You will design task briefs, specify the artifacts, and establish the judgment criteria a strong practitioner would reach. Your role spans diagnosing...
- ...high-fidelity simulated work environments to test AI agents in real marketing workflows. You will help design and pressure-test PR and communications... ...documents and tool states, and write or review reference answers and evaluation criteria. #J-18808-Ljbffr Paradigm
- ...building high-fidelity simulated work environments to evaluate AI agents in real marketing work. You will design and pressure-test Field Marketing environments, including briefs, artifacts, and the decision criteria that separate strong responses from plausible ones. The...
- ...New York builds high‑fidelity simulated work environments to evaluate AI agents on real marketing tasks. You will design creator and influencer scenarios, including briefs, artifacts, and the judging criteria that separate strong responses from plausible but incorrect ones...
- Mercor is partnering with a leading AI research organization to design tasks and grade real-world marketing work. You will define what excellent work looks like, write criteria, and provide rigorous written justifications for scores. Ideal candidates bring 5+ years in...
- Warner Bros. Discovery, CNN Digital Product Design Team, seeks a Senior Design Technologist to advance the Vossi Design System across design, engineering, and AI workflows. Translate design intent into scalable UI, drive AI-powered design tooling, and ensure accessible,...
- Rise Data Labs seeks a Media & Information AI Training Specialist to craft realistic... ...in journalism, editing, and video production. You will draft and source reference materials... ...video production experience and comfort evaluating professional-grade work to benchmark AI...
- Mercor is partnering with a leading AI research organization to engage... ...engineers for a project focused on evaluating how well AI systems perform real-world... ...what excellent work looks like by designing task-specific grading criteria and scoring completed work samples...
$150k
...consulting along with technology and design, corporate and digital ventures—and business... ..., at scale.What You'll DoSenior AI Tech Architects at BCG Platinion are:Collaborative.... ...modules in IT implementation projects.Evaluating products, SW packages, and solutions,...Work at officeLocal area$154.38k - $193.13k
Job DescriptionAI Architect, AI & AutomationAbout the Role The applicant should have deep, hands-on experience architecting, designing, and implementing enterprise-grade AI, Machine Learning, and Generative AI solutions, and experience leading technical teams delivering...Full timeTemporary workWork experience placement- ...are seeking a highly skilled AI/ML Engineer with extensive knowledge... ...environment.Responsibilities Design, develop, and deploy AI/ML... ...for model training, evaluation, and optimization.Mentor junior... ...in deploying AI solutions in production environments.Familiarity with...Full time
$187.6k - $220.7k
...is looking for an Advisory Architect, Data/AI due diligence to join our Technology... ..., and progress on all product programs.Be a thought leader... ...in data modeling and data design and must have hands on experience... ...salaries are determined by evaluating a variety of factors...H1bLocal areaImmediate startFlexible hours$122k - $240.5k
Position Summary Google AI Architect/AI and EngineeringJoin our AI... ...a Service provides complete design, implementation, and... ...deliver complex programs with a product engineering approach. Our flexible... ...and cost.Design, fine-tune, evaluate, and govern LLM solutions with...Local areaVisa sponsorshipFlexible hours$140k - $190k
...applications and delivery, leveraging AI, automation, and... ...MetLife is seeking a Principal Architect to lead the design and evolution of AI-powered... ..., including the ability to evaluate and compare platformsProven... ...on voluntary insurance products including auto and home, pet...Full timeTemporary workWork at officeLocal areaWorldwideRelocation package3 days per week- Intapp’s Applied AI Architects (AAA) operate at the intersection of product, engineering, and enterprise delivery—you will lead end-to-end deployments of frontier... ...discovery and technical scoping through system design, build, and production rolloutEmbed with customer engineering...Full timeLocal areaFlexible hours
$222.5k - $261.9k
...Monroe is seeking a Senior Architect to lead the technical... ...proprietary software assets, AI capabilities, and... ..., you will partner with Product Management, Engineering, Experience Design, Data & AI, and Consulting... ...platforms and frameworks. Evaluate emerging technologies...Local areaImmediate startFlexible hours- ...KPMG is currently seeking an AI Context Architect to join our Technology... ....Responsibilities:Lead the design of context architecture for... ...engineers, content owners, and product managers to translate... ...rigorous experimentation and evaluation framework to measure retrieval...Local area
$215k - $275k
...technology and software products with clients who are... ...transformation journeys. We blend design, product engineering,... ..., automation, and AI-native delivery to... ...enterprise scale. As an AI/ML Architect, you’ll design and... ...orchestration, agentic workflows, evaluation, deployment, monitoring...Temporary workLocal area$190k - $199k
...Wellbeing: Competitive benefits designed to support the wellbeing of... ...You Will Get To Do:Own the AI vision for the contact centre... ...and regulatory requirements. Evaluate opportunities based on value,... ...technologies and platforms in production environments. Ability to translate...Ongoing contractFull timeWork at office- ...is one-of-a-kind. You can design a career you’ll love from top... ...EisnerAmper is seeking an AI Platform Architect, Strategy & Enablement who... ...will build model lifecycle, evaluation, and multi-vendor orchestration... ...architecture, or platform/product leadership including...Full timeLocal areaRemote workWork visa
$330k
...focused, deeply technical Distinguished Architect to join our Product Solutions Architecture (PSA) team. In... ...multiplier for the world's leading AI labs and AI-native companies. You will... ...Lead deep-dive architecture reviews and design engagements with customer teams and their...Work at officeShift work$187.6k - $220.7k
...ready to make an impact?Agentic AI Architect-Google Cloud West Monroe is... ...into secure, scalable, and production-ready AI solutions. The role... ..., and measurable outcomes. Design scalable AI solutions using... ...exercising sound judgment in evaluating outputs. Apply AI...Local areaImmediate startFlexible hours$258.8k - $304.2k
...make an impact?Applied AI Engineer/Architect West Monroe is... ...Architect to lead the design and delivery of enterprise... ...orchestration, evaluation frameworks, observability... ...integrations, and secure production deployment patterns.... ...metrics, and release criteria to ensure reliable...Local areaImmediate startFlexible hours- ...A tech-focused research lab is seeking Property Manager Experts to design and evaluate property management challenges for training AI models. This remote role allows you to leverage your expertise in real estate operations while making a significant impact on AI solutions...Remote work
AfterQuery Experts
New York, NY2 days ago - ...One of New York's most ambitious and holistic design studios is seeking an Architect to join their small, Manhattan-based studio. This firm work in... ...residential, small scale cultural, boutique workplace, product design and landscape architecture - making them a hub of...
Rhythm Careers
New York, NY22 hours ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Product Design Evaluator & Criteria Architect. Be the first to apply!
Related searches
- building designer New York, NY
- entry level architectural designer New York, NY
- junior project architect New York, NY
- architectural project designer New York, NY
- remote architectural designer New York, NY
- senior project architect New York, NY
- architect architectural designer New York, NY
- junior architectural designer New York, NY
- project architect New York, NY
- director product design New York, NY