LLM Evaluation Engineer — Benchmarking & Diagnostics
House of Scribes
House of Scribes is seeking an experienced Python/LLM evaluation engineer to implement and stress-test a 20-dimension narrative assessment framework. The role focuses on building a reproducible Python-based evaluation harness and delivering transparent results within a fixed budget. You will prepare 16 global benchmark works, evaluate eight additional controlled works, and perform blind, repeatability and diagnostic tests, while tracking costs and token usage to provide a concise technical #J-18808-Ljbffr House of Scribes
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the LLM Evaluation Engineer — Benchmarking & Diagnostics in Seattle, WA vacancy
- ...proprietary AI-assisted story evaluation system for film,... ...need an experienced Python/LLM evaluation engineer to implement and stress-test... ...protocol and initial global benchmark manifest are already prepared... ...blind, repeatability and diagnostic sensitivity tests. Track...Suggested
- ...team and advance the state of the art in LLM inference systems and optimization. Our... ...minimal manual tuning. We embrace AI-native engineering, using AI not only as the workload we... ...more efficient inference. Profile and benchmark end-to-end workloads to identify bottlenecks...Suggested
$100k - $150k
...Associates Recruitment Consultant | Structural Engineering Colorado, Utah & Washington State... ...Engineer with strong experience in diagnostics, restoration, and forensic investigations... ...structural assessments and diagnostic evaluations of existing buildings. Supporting proposal...SuggestedFull timeFor contractorsImmediate start$70 - $84 per hour
...Location: Tukwila, WA Salary: $70.00 USD Hourly - $84.00 USD Hourly Description: Our client is currently seeking a Test & Evaluation Engineer 4 Skill Code:689-Flight/Mission/System Test Work Location: Seattle, WA Shifts: 1st shift - requires (Active Secret...SuggestedHourly payShift workDay shift- Snowflake is building the next generation of AI-native inference systems. We seek experienced researchers and systems developers to advance LLM inference, optimization, and platform-wide performance across GPUs, CPUs, and distributed runtimes. You will collaborate with model...Suggested
- A leading engineering and architecture firm is seeking a Structural Engineer to join their team in Seattle, Washington. The ideal candidate will have strong experience in structural diagnostics, restoration, and forensic investigations. Responsibilities include performing...Full time
- ...Avalanche Energy is seeking a Senior Mechanical Engineer with deep expertise in vacuum systems to... ...vacuum engineering practices and help evaluate, refine, and improve our current... ...systems supporting fusion experiments, diagnostics, and laboratory infrastructure. Work...Permanent employmentFull timeWork at office
Avalanche Energy
Tukwila, WA2 days ago $140k - $188k
...products to meet the most demanding government standards and timelines. We have an immediate need for an Operation Test and Evaluation Engineer SME to provide onsite and hybrid support out of Northern Virginia and the DC area. Responsibilities The SME Operational Test...Temporary workWork at officeImmediate startRemote workFlexible hours- ...understanding of laboratory workflows and quality control. You will have the opportunity to contribute to the development of cutting-edge diagnostic technologies while collaborating with a talented team. If you're passionate about advancing healthcare through science, this...
$125k - $185k
...Lead Product Manager, Agentic Quality Engineering Hub Fractal is a strategic AI partner... ...capabilities , including how they are evaluated, onboarded, documented, supported, governed... ...prompt engineering, MCP/tool integrations, LLM-based automation, human-in-the-loop...Hourly payFull timeContract workLocal areaFractal
Seattle, WA21 hours ago- Big Wave Digital is seeking engineers to build and deploy LLM agents and multi-agent systems that operate on constrained hardware at the tactical edge. You will interface AI with sensors, effectors and autonomous platforms for defense customers in high-stakes environments...
$33 - $40 per hour
...SUMMARY Under the supervision of Chief Engineer, directs the activities of engineering... ...assigned deadline. All safety training course evaluation testing (both digital and analog) must... ...system Possess strong electrical diagnostic skills including but not limited to explaining...Hourly payFull timeFor contractorsSeasonal workWork at officeLocal areaUrban Renaissance Group
Seattle, WA2 days ago$150k - $260k
...for our first dedicated Hardware Test Automation Engineer to build the automation backbone for the program.... ...dashboard with defined safety thresholds and target benchmarks; automatically flag regressions. Develop diagnostic tooling and support automated bug triage for...Full timeContract workFlexible hoursNight shiftWeekend work- ...Hiring: Forward Deployed AI Automation Engineer Important Visa: USC and GC only... ...practices. Practical experience developing LLM-powered applications or agentic workflows... ...tool integration. Understanding of evaluation frameworks, AI safety, security, monitoring...Full time
Wise Skulls
Bellevue, WA21 hours ago - This posting is for a contract assignment and is not a full-time employment offer with Boeing. Candidates selected for roles will be employed as contract workers through a Boeing approved 3rd party for the duration of the specified project. Develops and documents standardized...Full timeContract work
- ...Key Skills: Prompt Engineering, LLMs (GPT/Claude), Prompt Evaluation & Optimization, LangChain, RAG & AI Agents, Python, API Integration, LLM Security & Guardrails Having 8+ years of experience... ...frameworks (A/B testing, benchmarking, rubric-based scoring) to measure...
Jobrapido
Seattle, WA2 days ago $80 - $85 per hour
...Description: Our client is currently seeking an Engineer 2, AI Agentic Solutions Location:... ...agent user experiences, establish evaluation benchmarks, optimize token economics, and guide... ...guardrails, and telemetry for production LLM systems. Preferred...Hourly payContract workRemote workMonday to Friday- ...recommendations. 3. Proficient in using mainstream camera evaluation tools such as Imatest, capable of completing relevant test data... ...testing standards. 5. Conduct competitor research and deep benchmarking analysis for pre-research/innovative topics, outputting...Full time
Brinte Inc.
Seattle, WA3 days ago $135.2k - $306.4k
...Oracle hardware platform development engineering is seeking a highly driven GPU/CPU Platform... ...AI platforms. You will participate in evaluating AI platform architectures and assist... ...test and qualification, define software diagnostics features and utilize third party as well...Temporary workWork experience placementRemote workFlexible hours- ...The Supplier Quality Engineer (SQE) is responsible for evaluating, monitoring and improving the supplier base performance for SafeWorks. This is achieved... ...improvement and Lean events across all commodities. Benchmark manufacturing processes and drive implementation of...Work at office
Safeworks
Seattle, WA2 days ago $26 - $35 per hour
Job Description The Field Service Engineer is responsible for delivering exceptional technical... ...and optical tuning and setup. Analyze, evaluate and diagnose equipment; troubleshooting... ...in PC building, repairing and diagnostic testing. Two plus years' experience in...Hourly payWork at officeFlexible hours- ...Experience: 8+ years of professional software engineering experience About the Role We are... ...reliability, token usage, cost control, evaluation, and graceful failure. Apply... ...shipping AI-enabled applications using LLM APIs, Azure OpenAI, RAG, or agent orchestration...Full timeWork at office2 days per week3 days per week
InnoveIQ
Seattle, WA21 hours ago - ...Job Title: AI Automation Engineer Location: Seattle, WA - Onsite Duration: Long Term Job Description:... ...scalability, latency SLAs, and platform stability. Build LLM evaluation pipelines covering relevance, groundedness, and hallucination...
ConnectedX Inc
Seattle, WA2 days ago $60 - $70 per hour
...candidate We are seeking a hands-on Test Engineering Manager with a demonstrated commitment... ...and fast, trustworthy feedback. Evaluate and adopt testing tools and libraries pragmatically... .... Expertise adopting and leveraging LLM-based testing approaches, including...Full timeTemporary workImmediate startLabKey
Seattle, WA1 day ago- ...The Staff FPGA Engineer will lead the architecture, development, integration, and support... ...acquisition, instrument control, communications, diagnostics, and automated quality control. The... ..., and development best practices. Evaluate new devices, tools, and interface technologies...Work at office
Cytek Biosciences
Bellevue, WA1 day ago $133.5k - $194k
...re hiring a Senior Marketing Engineer - Web & AI to transform Hiya'... ...segments Integrate AI agents and LLM APIs to power dynamic content... ...Core Web Vitals, performance benchmarks, UX clarity, messaging... ...Hiring Manager (HM) Screen: We evaluate role fit, expectations, and execution...Work from homeWorldwideFlexible hours- DocuSign is seeking a Demo Engineer to join our high-performance GTM team. You will own end-to-end asset readiness for new and existing product... ...and fallback paths. You will build compelling prototypes using LLM agents and reference flows that make complex capabilities...Remote job
- ...deployment Maintain strong engineering standards while supporting marketing... ...Integrate AI agents and LLM APIs to power: Dynamic... ...Core Web Vitals and performance benchmarks Champion UX, messaging clarity... ...Hiring Manager (HM) Screen: We evaluate role fit, expectations, and...
$150k - $205k
...level Environmental, Civil, or Chemical Engineer to support our existing industrial... ..., and BMP selection Leading technical evaluation of data and developing recommendations... ...contaminants) to meet stormwater permit benchmarks and wastewater pretreatment limits Design...Work at officeFlexible hours$42.75 per hour
...excites you every day. @2026 TikTok Research Engineer Intern, Agentic Systems & AI... ...feedback. You will develop frontier methods, evaluate them rigorously, and bring them into production... ...FSE. Experience building or deploying LLM agents, AI developer tools, or agent platforms...Hourly paySummer workInternshipLocal area
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to LLM Evaluation Engineer — Benchmarking & Diagnostics. Be the first to apply!
