LLM Evaluation Researcher/Benchmarking Engineer

Budget: $2000 fixed

I'm developing a proprietary AI-assisted story evaluation system for film, television and literature.

I need an experienced Python/LLM evaluation engineer to implement and stress-test an existing 20-dimension narrative assessment framework.

The framework, evaluation protocol and initial global benchmark manifest are already prepared.

Responsibilities:

- Build a reproducible Python-based LLM evaluation harness.
- Prepare and test 16 global benchmark works.
- Evaluate eight additional controlled works and their supplied test variations.
- Run blind, repeatability and diagnostic sensitivity tests.
- Track ratings, source evidence, model consistency, token usage and API costs.
- Deliver working source code, JSON/CSV results and a concise technical report.

Requirements:

- Strong Python and LLM API experience.
- Experience with LLM evaluation, benchmarking or AI quality assurance.
- Structured outputs, automated testing, source-document parsing and reliable logging.
- Ability to work independently within a fixed budget.

This is not a website development, chatbot, creative writing or model-training project.

The objective is to test an existing diagnostic system rigorously, identify weaknesses and produce reproducible results.

Please begin your proposal with THALBERG TEST and briefly describe a comparable LLM evaluation project you've completed.