AS
Job Description
Staff AI Quality Engineer —
About The Role
As a Staff AI Quality Engineer -you will lead the technical test strategy and architectural validation for Lenovo's cross-device agentic AI ecosystem. You will serve as the technical authority on evaluating non-deterministic systems, designing automated quality platforms that operate across personal computing form factors (PCs, tablets, and smartphones). You will drive non-functional quality standards—ensuring that privacy, security, accessibility, internationalization, and device thermals/battery efficiency are safeguarded—while acting as a crucial release partner alongside Product Management and PMO leads.
What Will You Do
About The Role
As a Staff AI Quality Engineer -you will lead the technical test strategy and architectural validation for Lenovo's cross-device agentic AI ecosystem. You will serve as the technical authority on evaluating non-deterministic systems, designing automated quality platforms that operate across personal computing form factors (PCs, tablets, and smartphones). You will drive non-functional quality standards—ensuring that privacy, security, accessibility, internationalization, and device thermals/battery efficiency are safeguarded—while acting as a crucial release partner alongside Product Management and PMO leads.
What Will You Do
- Test Automation Architecture: Architect scalable, distributed, cross-platform test automation infrastructure capable of executing synchronized, multi-device end-to-end tests across Windows and Android devices without reliance on UI flakiness.
- Agentic AI Quality Frameworks: Design automated evaluation methodologies to validate multi-step agent reasoning, tool-calling graphs, autonomous error recovery, and transactional rollback mechanisms; establish statistical release gates for non-deterministic model behaviors.
- Hardware & Runtime Resource Profiling: Formulate automated performance and endurance soak suites to benchmark on-device SLM execution against cloud fallback paths, monitoring thermal throttling, memory leaks, and battery draw across hardware device tiers.
- Non-Functional Quality Standards: Lead automated verification across non-functional pillars: red-teaming indirect prompt injection vulnerabilities, securing inter-process data barriers, standardizing automated accessibility (screen-reader verbosity, dynamic token streaming), and validating semantic fidelity across international locales.
- CI/CD Quality Pipelines & Gating: Drive the technical implementation of shift-left gating in continuous integration pipelines, establishing automated failure analytics, regression detection, and flakiness isolation frameworks.
- Cross-Functional Peer Leadership: Serve as a core technical gatekeeper alongside Product Management and PMO; establish release acceptance criteria, lead data-informed risk assessments during schedule compressions, and influence platform design for enhanced testability.
- Bachelor's or Master's degree in Computer Science, Software Engineering, or related technical field.
- 8+ years of progressive software quality engineering, test architecture, and automation experience covering multi-device hardware ecosystems (Windows and Android).
- 3+ years of experience leading quality strategy and automated evaluation frameworks for LLMs, agentic AI workflows, or conversational platforms.
- Advanced programming and architecture skills in Python, C++, C#, or Java, with a track record of building scalable test harnesses and mock environments for OS IPC, APIs, and background daemons.
- Proven experience architecting automated non-functional test frameworks: deep profiling of device performance (NPU/GPU/CPU, battery mA/h, memory), security red-teaming/prompt-injection defense, automated accessibility (WCAG), and i18n semantic evaluation.
- Deep technical expertise in CI/CD pipeline architecture, test orchestration, and statistical quality gating.
- Hands-on expertise implementing and tuning non-deterministic evaluation platforms (e.g., RAGAS, DeepEval, LangSmith, TruLens) in production pipelines.
- Direct experience testing local NPU runtimes and quantized models (INT4/INT8 vs. FP16) on client silicon (Qualcomm, Intel, AMD).
- Background designing automated testing for multimodal inputs (synchronized voice injection, screen-capture parsing, synthetic system events).
- Demonstrated ability to resolve complex release-gating disputes with Product Management and PMO using objective risk metrics and clear mitigation trade-offs.