Member of Technical Staff - Research

Directly builds and evaluates LLM benchmarks; mentions a \"vibe-coding\" benchmark in media, so closely related to vibe-coding evaluation and measurement. About the Role Design and build next-generation benchmarks to evaluate real-world capabilities of large language models (LLMs). Lead research and engineering efforts to create rigorous, publishable evaluations and work with model labs and enterprises to shape how foundation models are measured. Job Description Role Member of Technical Staff - Research responsible for designing and implementing novel, high-impact benchmarks that assess challenging real-world capabilities of large language models (LLMs). The role focuses on rigorous research, benchmark validity, and collaboration with foundation model labs and enterprise partners. Key Responsibilities Design and develop novel benchmarks that evaluate real-world LLM capabilities Conduct research to ensure benchmarks are valid, reliable, and meaningful Analyze model performance across benchmarks and articulate results Collaborate with foundation model labs, enterprises, and infrastructure teams to implement benchmarks at scale Publish research findings and contribute to the evaluation research community Stay current with developments in LLM capabilities and evaluation methodologies Requirements Advanced research experience (Master’s or PhD in Computer Science, NLP, Machine Learning, or related field; exceptional undergraduates may be considered) Publication track record in reputable venues (e.g., NeurIPS, ICML, ACL, EMNLP) Strong understanding of experimental design, statistical analysis, and evaluation frameworks Proficiency in Python for research and experimentation Strong communication skills for both technical and non-technical audiences Experience collaborating in research teams and integrating feedback Demonstrated track record of impactful research work / portfolio Nice to Haves Prior experience in LLM evaluation or benchmarking research Familiarity with foundation model architectures and capabilities Experience working with industry partners or in applied research settings Backgrounds in human-computer interaction, psychology, or domain-specific evaluation Experience at early-stage startups or research labs Contributions to open-source evaluation tools or datasets Python Django React (frontend) AWS (infrastructure) AWS CDK (Infrastructure as Code) Location In-person team based in San Francisco. Relocation and transportation support offered. Compensation Compensation range: $140,000 - $185,000 (USD) What We Offer Highly competitive salary and meaningful ownership Relocation and transportation support Health and dental insurance coverage Lunch and dinner provided; free snacks, coffee, and drinks Unlimited PTO $1,500 housing stipend (within one-mile radius) Opportunities to collaborate with leading AI labs and publish research findings Python Django React AWS CDK Skills Research Benchmark Design Experimental Design Statistical Analysis Data Analysis Communication Collaboration Publication / Academic Writing Project Ownership Rapid Learning Problem Solving Experience Level Senior USD 140,000 - 185,000/year Employment Type Full-time Relocation and transportation support Lunch and dinner provided Free snacks/coffee/drinks $1,500 housing stipend (within one-mile radius) Highly competitive salary Opportunity to publish and present work #J-18808-Ljbffr Vibehackers