Machine Learning Engineer, Core Evaluations

Role Description

We are seeking an experienced Machine Learning Engineer (MLE) to focus on audio model evaluation, specifically for speech generation and recognition models. This role involves:

• Designing model evaluation pipelines for models in development and production.
• Designing user studies for subjective model evaluations.
• Converting requirements into measurable metrics.
• Designing and developing automated evaluation dashboards to see model performances and compare results.
• Training new models to capture new and different evaluation metrics.
• Communicating with the model team to help design better models based on the evaluation results.
• Communicating with the data team to help decide the type of data necessary to improve model performance.
• Communicating with the product manager to ensure product requirements are correctly measured.
• Helping grow the evaluation team as the founding member.
• Leading the evaluation team in the future.

Qualifications

• Strong experience and intuition for designing metrics that capture model performance.
• Strong experience with designing user studies on Mechanical Turk or similar platforms.
• Strong experience with model training and fine-tuning for model evaluation.
• Strong statistical knowledge and experience to statistically compare evaluation results and make decisions.
• Very strong engineering and programming skills.
• Experience with training ASR, TTS models.
• Experience in ML teams working on large-scale machine learning problems (>3B models with >1m hours of data).