AI Product Design Evaluator & Criteria Architect
Mercor
Mercor is partnering with a leading AI research organization to engage experienced UI/UX and product designers for a project that evaluates how well AI systems perform real-world digital product design work. You will define what excellent work looks like by designing task-specific grading criteria and scoring samples with rigorous, written justifications, guiding evaluation rather than delivering designs. You will contribute to an objective assessment framework and collaborate with senior #J-18808-Ljbffr Mercor
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the AI Product Design Evaluator & Criteria Architect in New York, NY vacancy
- Mercor is partnering with a leading AI research organization to engage experienced UI/UX and product designers for a project focused on evaluating how well AI systems perform real-world... ...like: designing task-specific grading criteria and scoring completed work samples...Suggested
- ...New York builds high‑fidelity simulated work environments to evaluate AI agents on real marketing tasks. You will design creator and influencer scenarios, including briefs, artifacts, and the judging criteria that separate strong responses from plausible but incorrect ones...Suggested
- ...building high-fidelity simulated work environments to evaluate AI agents in real marketing work. You will design and pressure-test Field Marketing environments, including briefs, artifacts, and the decision criteria that separate strong responses from plausible ones. The...Suggested
$225k - $275k
...leverages embedded AI agents and embedded financial products to reinvent the way... ...Principal Applied AI Architect to set how data and... ...as delivery criteria rather than afterthoughts... ..., guardrails, and evaluation fit on top of that... ...review that keeps new designs on that spine. You...SuggestedWork at officeRemote workFlexible hours$189.89k - $230.1k
...Commercial Orchestration Architect to serve as BMS’s... ...standards, evaluation framework, and technical... ...partner, AI model, or proprietary... ...interaction patterns. Design the orchestration... ...acceptance criteria and the definition... ...test evidence, and production-readiness assessments...SuggestedHourly payFull timeSummer workLive inLocal areaRemote workFlexible hoursShift work- Rise Data Labs seeks a Media & Information AI Training Specialist to craft realistic... ...in journalism, editing, and video production. You will draft and source reference materials... ...video production experience and comfort evaluating professional-grade work to benchmark AI...
- Mercor is partnering with a leading AI research organization to engage... ...engineers for a project focused on evaluating how well AI systems perform real-world... ...what excellent work looks like by designing task-specific grading criteria and scoring completed work samples...
- ...One of New York's most ambitious and holistic design studios is seeking an Architect to join their small, Manhattan-based studio. This firm work in... ...residential, small scale cultural, boutique workplace, product design and landscape architecture - making them a hub of...
Rhythm Careers
New York, NY3 days ago - ...A tech-focused research lab is seeking Property Manager Experts to design and evaluate property management challenges for training AI models. This remote role allows you to leverage your expertise in real estate operations while making a significant impact on AI solutions...Remote work
AfterQuery Experts
New York, NY5 days ago - ...Own the system design for agentic capabilities, including orchestration,... ...across engineering teams Drive AI-assisted and agentic development... ...customers Report to the Chief Architect and partner with engineering leadership, Product, and Data Science Requirements...
Jobtailor
New York, NY4 days ago $110k - $120k
...recommendations, and help design and implement technology transformations... ...engineering, data, AI, cybersecurity, product and technology... ...and Google Cloud Platform Evaluate scalability, resilience, performance... ...Collaborate with client architects, engineers, product leaders...Full timeSummer workInternshipWork at officeLocal area$190k - $199k
...provider of mission-critical, AI-powered technology and... ...Wellbeing: Competitive benefits designed to support the wellbeing of our... ...and regulatory requirements. Evaluate opportunities based on value,... ...technologies and platforms in production environments. Ability to...Ongoing contractFull timeWorldwide$270k - $330k
...established cybersecurity SaaS company, focused on shaping AI strategy and designing large-scale AI solutions across the organization.... ..., guiding cross-functional teams toward scalable, production-grade systems. Evaluate and define AI governance, safety, and compliance...Full timeWork at office- Overview: AI Test Automation Architect The AI Architect will lead and own the strategic vision and execution... ...frameworks and pipelines across product teams. AI Innovation & Enablement:... ...Architecture & Framework Development: Design and evolve scalable AI-powered automation...
- Mercor is building high-fidelity simulated work environments to evaluate and improve AI agents on real marketing work. You will help design the briefs, artifacts, the judgment calls, and the errors a weaker practitioner would miss. Tasks span diagnosing, prioritizing, planning...
- ...high-potential controllership professional for a transformative AI financial software company in New York. You will influence month... ...reconciliations, SOX/ICFR, and audit-ready evidence while collaborating with product, engineering, and senior finance leaders. The role requires 5-10...
- LinkedIn is hiring a Sr. Staff Analytics Architect to define the target-state architecture for People Analytics data products, governance, and AI-enabled capabilities. You will lead across analytics engineering, governance, and reporting platforms to establish standards...
- Mercor is seeking senior litigation professionals to build evaluation tasks for AI systems operating in civil litigation and dispute resolution contexts. The workflows are calibrated to case complexity, evidentiary stakes, and procedural scope of major commercial litigation...
$80 - $90 per hour
AI Architect -26-02164 Hybrid/Onsite in NYC 2yrs Duration TempW2orC... ...systems from prototype to production at real scale — a system that... ...contributor on someone else's design Experience at a company... ...scale‑up) Experience building evaluation systems that proved the...Hourly payTemporary workRemote workFlexible hours$136k - $160k
...systems. This role will focus on ensuring speedy and reliable access to information as the company grows. The ideal candidate will design and maintain the knowledge architecture while collaborating with various teams. A competitive salary range of $136,000 to $160,000...- ...seeking a results-driven sales professional to own a designated territory in the United States. This remote, full-time... ...involves selling premium living walls and related interior products to design directors, architects, and contractors. You will nurture relationships, lead...Remote jobFull timeFor contractors
- searchneedsLOVE seeks brilliant minds who have moved past the old “Keyword-Chasing” mindset and are ready to act as true Revenue Architects. You can join freelance, part-time, or full-time from anywhere in the world, with 100% remote engagement. You’ll lead Revenue Architectures...Remote jobFull timePart timeFreelanceImmediate start
$120k - $130k
Digitalzone in New York is seeking an AI Architect to bridge engineering, AI, and business strategy. This role involves collaborating across departments to design AI solutions that improve efficiency. Candidates should hold an engineering degree from IIT/IIM with 2+ years...- ...discovery with Healthcare Client IT and contact center stakeholders in New York. You will assess Five9 configuration, call routing/IVR design, and integrations across CRM, WFM, and analytics. You will identify gaps and opportunities, validate findings against Five9...
- Senior Agentic AI Engineer Are you looking to join... ...of applied research, product innovation, and... ...modeling architecture while designing platform-level tooling... ...workflows, quality evaluators, constraint validation... ...product-oriented solutions. Architect large-scale graph,...Work at officeRemote workFlexible hours
$225k - $304.2k
...an impact? Applied AI Engineer/Architect West Monroe is seeking... ...to lead the design and delivery of enterprise... ...orchestration, evaluation frameworks, observability... ...integrations, and secure production deployment patterns.... ...metrics, and release criteria to ensure reliable...Full timeLocal areaImmediate startFlexible hours- Overview The AI Solutions Architect defines the technical direction for AI, ML, and data‑driven capabilities... ...priorities into AI‑enabled solution designs and reusable reference architectures.... ..., inference layers, and governance. Evaluate emerging AI and cloud capabilities and...
- ...a seasoned Financial Services Industry Architect to translate banking and risk data into... ...measurable outcomes. You will drive end-to-end designs for core banking use cases such as fraud... ...360, and you will contribute to product roadmaps and field enablement. #J-18808...
- ...PLM, MBSE, digital twins, and AI-powered engineering tools. Establish... ...and research organizations to evaluate emerging AI and graph... ...Enterprise Architecture, Legal, Product, and Technology teams to align... ...following areas is required: Designing enterprise-scale knowledge graphs...
$80 per hour
...experienced curriculum coordinators to train an AI language model on professional document,... ...tasks. The role focuses on creating, evaluating, and refining AI-generated curricula... ...administration and strong standards-aligned instructional design. #J-18808-Ljbffr EthosRemote jobImmediate startFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Product Design Evaluator & Criteria Architect. Be the first to apply!
Related searches
- building designer New York, NY
- project architect New York, NY
- architectural project designer New York, NY
- senior project architect New York, NY
- entry level architectural designer New York, NY
- junior architectural designer New York, NY
- remote architectural designer New York, NY
- junior project architect New York, NY
- architect architectural designer New York, NY
- product design manager New York, NY