ML Evaluation Engineer

Overview

We are seeking an ML Evaluation Engineer/Data Scientist to infuse statistical integrity and measurement precision into the assessment of AI and LLM systems. This position will primarily concentrate on guaranteeing that evaluation outcomes are dependable, reproducible, and scientifically sound.

The ideal candidate will collaborate closely with ML Evaluation Engineers and Applied Scientists to address pivotal inquiries such as: 'Is this performance enhancement genuine?', 'Are our evaluators in agreement?', and 'Can we have confidence in these results?'

Main Responsibilities:

• Design and implement statistical analyses for evaluating AI and LLM initiatives.
• Assess and enhance inter-annotator agreement among human evaluators.
• Create sampling techniques and evaluation datasets to ensure outcomes are representative.
• Conduct significance testing and statistical validation of changes in model performance.
• Examine golden sets and benchmark datasets to evaluate model quality and consistency.
• Develop dashboards and reporting systems to track evaluation results and trends.
• Identify sources of evaluation noise, bias, and inconsistency.
• Collaborate with ML Evaluation Engineers to validate automated judges against human evaluators.
• Offer data-driven insights regarding evaluation quality and model readiness.

Essential Qualifications:

• Bachelor's degree in Statistics, Data Science, Computer Science, Mathematics, Machine Learning, or a related discipline.
• At least 2+ years of professional experience in Data Science, Evaluation Science, Applied Statistics, or related fields.
• Solid foundation in statistical analysis, hypothesis testing, experimental design, and measurement methodologies.
• Proficiency in working with structured and unstructured data using Python, SQL, and data analysis tools.
• Ability to convey statistical findings effectively to both technical and non-technical audiences.
• Experience in creating analytical dashboards and reporting frameworks.

Preferred Qualifications:

• Master's degree or PhD in Statistics, Data Science, Computer Science, Quantitative Social Sciences, or a related field.
• Hands-on experience supporting LLM, AI, NLP, or ML evaluation projects.
• Experience in measuring agreement between human raters and automated evaluation systems.
• Knowledge of annotation quality frameworks, golden-set development, sampling theory, and evaluation benchmarking.
• Familiarity with LLM-as-Judge frameworks and AI evaluation platforms.

We regret to inform you that only candidates selected for an interview will be notified.

 EA registration number: YAP JIA YI, R25157934

Allegis Group Singapore Pte Ltd, Company Reg No. 200909448N, EA Licence No. 10C4544