Gen AI Quality Engineer

The essential skill sets required for this position are outlined below:

Core Requirements

• 5 to 8 years of experience in software quality engineering, test engineering, or a comparable hands-on role that involves complex applications.

• Extensive experience in testing web user interfaces, backend services, and REST APIs.

• Proficiency with API tools and automation frameworks such as Postman, REST Assured, pytest, Playwright, Cypress, Selenium, or similar alternatives.

• Practical knowledge of Java, Python, JavaScript, or TypeScript adequate for building, reviewing, and troubleshooting test automation.

• Excellent test analysis capabilities, including reviewing requirements, assessing risks, conducting boundary analysis, performing negative testing, and ensuring traceability.

• Experience in validating distributed systems and integrations, including asynchronous processing, queues, batch jobs, and downstream dependencies.

• Ability to examine logs, traces, network calls, payloads, and database records to pinpoint the actual failure point.

• Familiarity with Git, pull requests, CI/CD pipelines, test reporting, and defect management tools.

• Knowledge of security and privacy testing principles, including access control, sensitive data handling, input validation, and auditability.

• Strong communication skills with stakeholders and the ability to challenge subpar designs, unclear expected outcomes, and hasty release decisions.

• Capacity to thrive in a dynamic environment where requirements and GenAI behavior are continually evolving.

Solution Analysis and Design Expectations

• Experience in analyzing complex applications across user journeys, business processes, APIs, data flows, and enterprise integrations.

• Ability to lead requirements discussions and translate vague business needs into well-defined functional requirements, acceptance criteria, and solution behaviors.

• Proven experience creating practical analysis artifacts such as process flows, sequence diagrams, context diagrams, interface specifications, decision tables, and user stories.

• Capacity to evaluate solution alternatives and articulate trade-offs concerning quality, cost, performance, security, operational support, and delivery complexity.

• Understanding of application architecture concepts, including synchronous and asynchronous integrations, event flows, authentication, authorization, failure handling, and system boundaries.

• Ability to differentiate between problems that require GenAI solutions and those better suited for deterministic rules, search, workflow automation, or traditional application logic.

• Confidence in collaborating with product, architecture, engineering, security, data, and business stakeholders during the discovery and solution design phases.

GenAI and Agentic Testing Expectations

• Practical knowledge of LLM-based applications, retrieval-augmented generation, prompts, context windows, embeddings, and tool-using agents.

• Ability to test probabilistic systems using evaluation datasets, repeat runs, quality thresholds, and evidence-based acceptance criteria rather than relying on fragile exact text matching.

• Experience in validating grounded answers, citations, retrieval quality, hallucination risk, prompt injection resistance, and the safe handling of restricted or unsupported requests.

• Proficiency in testing agent plans and execution paths, tool calls, memory and state management, human approval checkpoints, fallback behavior, and termination conditions.

• Awareness of evaluation and observability tools such as Langfuse, LangSmith, OpenTelemetry, Elastic, Splunk, or equivalent.

Nice to Have

• Background in testing applications within banking, finance, or other regulated enterprise environments.

• Experience with contract testing, service virtualization, synthetic monitoring, or performance testing tools.

• Familiarity with Kubernetes, OpenShift, AWS hosted services, or containerized deployments.

• Experience in accessibility testing and knowledge of WCAG-based checks.

• Experience developing streamlined quality dashboards that display release risk, defect escape patterns, flaky tests, and cycle time.

• Exposure to red teaming or adversarial testing of GenAI applications.

Key Domain/Technical Skills:

(Indicate up to 3 areas)

• UI and API Quality Engineering

• Agentic and GenAI Application Testing

• Risk-Based Test Automation and Observability