LLM Evaluation Specialist

Location: Hybrid San Francisco, CA 94107 (Open to Remote) Duration: 11/16/2026 - 12/18/2026 Schedule: 15 hours per week Team: Incubator - VizWhiz Job Details LLM Evaluation Specialist Location: Hybrid San Francisco, CA 94107 (Open to Remote) Duration: 11/16/2026 - 12/18/2026 Schedule: 15 hours per week Team: Incubator - VizWhiz About the Job Work with a small team building an AI-native data storytelling product that helps business professionals turn data into clear, honest narratives. Run recurring evaluation passes on AI-generated charts, headlines, captions, and chat responses against a defined rubric. Judge accuracy, chart-type fit, clarity, and tone; flag misleading framing, specious correlations, and awkward phrasing. Log findings in evaluation tooling and deliver a concise report to the team after each cycle. Suggest rubric improvements based on observed patterns. Part-time role: approximately 15 hours per week over a multi-week contract. Duties Perform recurring evaluation passes on AI-generated charts, headlines, captions, and chat responses using the team’s rubric. Assess whether claims are supported by the underlying data and verify chart-type fit. Identify and flag misleading framing, specious correlations, and unclear or awkward phrasing not caught by automated evaluations. Record findings in evaluation tooling and produce concise post-cycle reports for the team. Recommend concrete rubric changes informed by recurring error patterns. Required Qualifications Strong data analysis and data visualization judgment. Sharp editorial instincts and high attention to detail. Comfortable working with spreadsheets and evaluation tools (no coding required). Clear, concise written reporting skills. Bachelor’s degree in a related field such as data analytics, statistics, journalism, information science, or business, or equivalent practical experience. Preferred qualifications Prior experience evaluating LLM output. Compensation Pay Rate: $59.86/hour #J-18808-Ljbffr