Sr GenAI Quality & Solution Analyst (Banking, 1-year renewable contract)

Dear Applicant,

If you or someone you know is interested, please send the CV directly to [HIDDEN TEXT] (most preferred, as I may overlook some CVs due to the high volume).

Please note that visa sponsorship is not available at this time.

Key Responsibilities

. Design and execute end-to-end test strategies covering user interfaces, backend APIs, integrations, data flows, and agentic AI behaviour.

. Validate GenAI applications for task completion, response accuracy, relevance, grounding, consistency, citation quality, and safe failure handling.

. Test agentic workflows, including tool selection and execution, multi-turn context, state transitions, retries, timeouts, approval checkpoints, handoffs, and recovery from failures.

. Perform functional, integration, regression, exploratory, negative, resilience, and basic performance testing across application layers.

. Develop and maintain API tests covering authentication, authorisation, validation, error handling, idempotency, rate limits, and downstream dependencies.

. Automate high-value test scenarios to improve testing efficiency, reduce manual effort, and minimise production risks.

. Analyse logs, traces, request and response payloads, network calls, and database records to identify defects and determine root causes.

. Establish reusable test scenarios, test data, traceability, and evidence to support enterprise release governance and quality reporting.

. Provide evidence-based release recommendations, highlighting known limitations, residual risks, and areas requiring further monitoring.

. Collaborate with product owners, business users, architects, engineers, security teams, and GenAI specialists to define business needs, user journeys, and expected outcomes.

. Translate business requirements into functional requirements, end-to-end solution flows, interface behaviours, business rules, acceptance criteria, and non-functional requirements.

. Analyse solution options and recommend appropriate approaches involving GenAI, retrieval-augmented generation (RAG), workflow orchestration, deterministic application logic, and human approval.

. Map interactions across user interfaces, APIs, language models, prompts, retrieval components, enterprise data sources, agent tools, and downstream systems.

. Identify design gaps, integration assumptions, ownership issues, security concerns, failure scenarios, and operational risks before development begins.

. Facilitate solution design reviews and document findings through process flows, sequence diagrams, interface specifications, decision tables, and solution documentation.

. Define measurable success criteria covering business outcomes, user experience, functional correctness, response quality, latency, reliability, and safe failure.

. Maintain traceability from business requirements through solution design, implementation, test scenarios, and release evidence.

Key Requirements

. 5-8 years of hands-on experience in software quality engineering, test engineering, or a similar role involving complex applications.

. Strong experience testing web applications, backend services, REST APIs, and enterprise integrations.

. Hands-on experience with API testing and automation tools such as Postman, REST Assured, pytest, Playwright, Cypress, Selenium, or equivalent.

. Working knowledge of Java, Python, JavaScript, or TypeScript to develop, review, and troubleshoot test automation.

. Strong test analysis skills, including requirements review, risk assessment, boundary analysis, negative testing, and traceability.

. Experience testing distributed systems and integrations, including asynchronous processing, queues, batch jobs, and downstream dependencies.

. Ability to investigate application issues using logs, traces, network calls, payloads, and database records.

. Familiarity with Git, pull requests, CI/CD pipelines, test reporting, and defect management tools.

. Understanding of security and privacy testing fundamentals, including access control, sensitive data handling, input validation, and auditability.

. Experience analysing complex applications across business processes, user journeys, APIs, data flows, and system integrations.

. Ability to facilitate requirements discussions and translate ambiguous busines needs into clear functional requirements, acceptance criteria, and solution behaviours.

. Experience producing analysis artefacts such as process flows, sequence diagrams, context diagrams, interface specifications, decision tables, and user stories.

. Understanding of application architecture concepts, including synchronous and asynchronous integrations, event flows, authentication, authorisation, system boundaries, and failure handling.

. Ability to assess solution alternatives and explain trade-offs involving quality, cost, performance, security, operational support, and delivery complexity.

. Practical understanding of LLM-based applications, RAG, prompts, context windows, embeddings, and tool-using agents.

. Ability to evaluate probabilistic AI systems using evaluation datasets, repeated test runs, quality thresholds, and evidence-based acceptance criteria rather than relying solely on exact text matching.

. Experience validating grounded responses, citations, retrieval quality, hallucination risks, prompt injection resistance, and safe handling of restricted or unsupported requests.

. Understanding of agent planning and execution, tool calls, memory and state management, human approval checkpoints, fallback behaviour, and terminationconditions.

. Familiarity with observability and evaluation tools such as Langfuse, LangSmith, OpenTelemetry, Elastic, Splunk, or equivalent.

. Strong analytical, communication, and stakeholder management skills, with the confidence to challenge unclear requirements, weak designs, and premature release decisions.

. Ability to work effectively in a fast-paced environment where requirements and GenAI behaviour continue to evolve.

Nice to Have

. Experience in banking, financial services, or other regulated enterprise environments.

. Exposure to contract testing, service virtualisation, synthetic monitoring, or performance testing.

. Familiarity with Kubernetes, OpenShift, AWS-hosted services, or containerised deployments.

. Accessibility testing experience and familiarity with WCAG-based checks.

. Experience developing quality dashboards covering release risks, defect trends, flaky tests, and testing cycle time.

. Exposure to red teaming or adversarial testing of GenAI applications.