Staff AI Agent Architect: Evaluations & Environments
Jobleads-US
Asymmetric Security is seeking a Member of Technical Staff focused on Agents, Evaluations, and Environments. You will design advanced LLM agents, build evaluation suites with deterministic graders, and develop scalable, privacy-preserving environments to measure and improve defensive AI models.
You will work closely with incident responders to capture expert judgement, shape research directions, and push the boundaries of forensic reasoning in AI-enabled security.
#J-18808-Ljbffr Jobleads-USVacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Staff AI Agent Architect: Evaluations & Environments in New York, NY vacancy
- ...Member of Technical Staff: Agents, Evaluations, and Environments Asymmetric is the first AI-native Digital Forensics and Incident Response company. Our platform has helped incident responders handle hundreds of cyber attacks. Building on that experience, we create...Suggested
- ...Must have skills/Matrix AWS Bedrock / Bedrock AgentCore Agentic-AI Architecture & Multi-Agent Orchestration Tool-Use / Function Calling & Typed Contracts Guardrails, Safety Design & Policy-as-Code Workload Identity, OAuth/OIDC & Delegated-Authority Models...Suggested
Intelliswift - An LTTS Company
New York, NY4 days ago - Obsidian in New York builds high‑fidelity simulated work environments to evaluate AI agents on real marketing tasks. You will design creator and influencer scenarios, including briefs, artifacts, and the judging criteria that separate strong responses from plausible but...Suggested
- Mercor is building high-fidelity simulated work environments to evaluate AI agents in real marketing work. You will design and pressure-test Field Marketing environments, including briefs, artifacts, and the decision criteria that separate strong responses from plausible...Suggested
- micro1 is seeking a Member of Technical Staff, Law Research, to lead the development of evaluation frameworks for AI-enabled legal reasoning. You will design benchmarks and collaborate with researchers to shape enterprise AI workflows, focused on robust evaluation across...SuggestedRemote job
- Rise Data Labs seeks a Media & Information AI Training Specialist to craft realistic media tasks and rubrics for AI training in journalism... ..., editing, writing, or video production experience and comfort evaluating professional-grade work to benchmark AI performance. #J-18808-...
- Kraken seeks an AI Agents Solutions Architect to design AI-driven HR solutions that automate and optimize People operations. The role focuses on building... ...impact by improvements in HR efficiency and allowing staff to focus on strategic advisory work. #J-18808-Ljbffr...Remote job
- Progressive Leasing is seeking a Forward Deployed Agent Builder to collaborate with business and product teams to design, build, and deploy AI agents that solve real-world workflows. This role sits at the intersection of agent engineering, product thinking, and stakeholder...Remote job
- A forward-thinking technology company is looking for a detail-oriented individual to design structured evaluation scenarios for AI agents. This entry-level part-time role allows you to contribute remotely, creating test cases for LLM-based agents. The ideal candidate will...Remote jobPart time
- ...About the Role As a Software Engineer on the Agent Quality team at SpaceXAI, you'll build the measurement, evaluation, and feedback-loop infrastructure that makes the... ...on Designing and building best-in-class AI evaluation system: curated datasets, offline...Full timeWork at office
Anysphere, Inc
New York, NY5 days ago - Mercor is partnering with a leading AI research organization to engage experienced sales engineers for a project focused on evaluating how well AI systems perform real-world technical sales work. You will define what excellent work looks like by designing task-specific...
- Alice is seeking a seasoned researcher-leader to helm AI safety benchmarks and evaluations. You will own the taxonomy, harness, and release cadence, coordinating researchers and freelancers while reporting to the CTO office. You will influence the public research agenda...FreelanceWork at office
- ...senior software leader to own the technical vision for major capability areas—agent-initiated settlement, edge AI-driven on-device financial agents, and portable verifiable trust. You will architect solutions that endure industry convergence and guide multi-team...
- ...architecture for cloud-native platforms and CI/CD pipelines across multiple teams. You will design scalable infrastructure, integrate agentic AI workflows, and set organization-wide standards for IaC, security, and observability. You will mentor engineers, lead design...
- ...We are building AI to simulate the world through merging art... ...Senior Product Manager to build agent products. You’ll own new... ...features. ~ You thrive in an environment with rapid pace and a high accountability... ...with prompt engineering and evaluation frameworks. ~ Excellent...Work at office
$200k - $250k
...Applied AI Engineer, Agents & Evaluation Compensation: $200k–$250k base + equity Location: New York City — fully onsite, 5 days per week... ...that agents rely on. Build for a high-stakes environment where false findings matter and reliability is a core product...Full timeFor contractorsPeople In AI
New York, NY4 days ago$80 per hour
...collective human intelligence to ethically shape the future of AI. What We Do The Mindrift platform connects specialists... ...for someone who can design realistic and structured evaluation scenarios for LLM-based agents. You'll create test cases that simulate human-performed...Part timeFreelanceRemote workFlexible hours- Mercor is partnering with a leading AI research organization to engage experienced UI/UX and product designers for a project that evaluates how well AI systems perform real-world digital product design work. You will define what excellent work looks like by designing task...
$148k - $243k
...People Data Foundations architects and stewards... ...trusted, accessible, and AI-ready.As a Sr. Staff Analytics Architect,... ...reviews, evaluate technology options,... ...workloads in production environments.Experience with AI and... ..., knowledge graphs, agent-based systems, and AI...For contractorsWork at officeFlexible hoursLinkedin
New York, NY2 days ago- ...engineering problems across our memory and agent architecture. You will help build user-... ...shipping production software, passion for AI and human psychology, and experience with... ...a collaborative, in-person, fast-paced environment in NYC. #J-18808-Ljbffr Jobleads-US
- ...Fleet AI, Inc. is an applied AI company building human-AI collaboration at scale. Our platforms enable fast, verifiable training and evaluation of agents across environments, backed by top investors. The team owns the agent runtime, environment deployment, and the production...
- ...Job Title : Enterprise Genai LLM Solution Architect/ AI Architect Location specific: Required... ..., RAG pipelines, prompt engineering, AI agents) into core enterprise systems. Establish... ..., security, and maintainability. Evaluate and select GenAI platforms, frameworks,...
Mindlance
New York, NY2 days ago $65 per hour
...Job Title: Enterprise Gen AI LLM Solution Architect Type: Contract Assignment Duration: 6 months... ...RAG pipelines, prompt engineering, AI agents) into core enterprise systems. • Establish... ..., security, and maintainability. • Evaluate and select GenAI platforms, frameworks...Contract workWork experience placementWork at officeBernard Nickels & Associates
New York, NY4 days ago- ...Applied AI Architect (Data, Systems & Governance) | Brillio Architect the Foundations of... ...operate safely within highly regulated environments. About Brillio Brillio is one of... ...knowledge layers, and AI data pipelines Evaluate and select AI models based on use case,...Full time
Brillio
New York, NY3 days ago - ...AI Architect – GenAI / ML Role Overview Photon is looking for... ...retrieval-augmented generation, agents, and AI orchestration... ...routing, model versioning, and evaluation . Design architectures for... ...large-scale enterprise environments. Deep hands-on programming...Full time
Photon
New York, NY1 day ago - ...the Role: We are looking for a senior AI Engineering & Architecture Leader to design... ..., AI architecture, model fine-tuning, evaluation, performance optimization, and... ...Experience in a startup or product-company environment is preferred/required for the role. ~...Full time
Galent
New York, NY1 day ago $180k - $300k
...United States Permanent AI Platform Architect - New YorkOverviewOur... ...routing, AI and API gateways, agent frameworks, MCP integration... ..., tool calling, AI evaluation frameworks, observability,... ...highly regulated enterprise environments.Experience designing and supporting...Permanent employmentWork at officeApollo Solutions
New York, NY1 day ago$230k
...& Digital Platforms, AI at Scale, Agile, Cybersecurity... ...DoPrincipal AI Tech Architects at BCG Platinion are:... ...the unknown.Change agents. They know how to make... ...agent orchestration.Evaluate emerging AI frameworks... ...multi-cloud and hybrid environments.Integration of agent workflows...Work at officeLocal areaThe Boston Consulting Group
New York, NY1 day ago$222.5k - $261.9k
...Monroe is seeking a Senior Architect to lead the technical... ...software assets, AI capabilities, and reusable... ...AI, intelligent agents, and automation capabilities... ...and frameworks. Evaluate emerging technologies... ...products in enterprise environments. Hands-on experience with...Local areaImmediate startFlexible hoursWest Monroe Partners
New York, NY2 days ago$122k - $240.5k
Position Summary Google AI Architect/AI and EngineeringJoin our AI & Engineering team... ...security, and cost.Design, fine-tune, evaluate, and govern LLM solutions with Gemini on... ...and scaling solutions for production environments.5+ years hands-on with Google Cloud, including...Local areaVisa sponsorshipFlexible hoursDeloitte
New York, NY3 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Staff AI Agent Architect: Evaluations & Environments. Be the first to apply!
Related searches
- import export agent New York, NY
- freight broker agent New York, NY
- operations agent New York, NY
- freight agent no experience New York, NY
- cruise agent New York, NY
- commissioning agent New York, NY
- transfer agent New York, NY
- recovery agent New York, NY
- tsa agent New York, NY
- state farm agent work from home New York, NY