My client is hiring a Test & AI Evaluation Lead to own how they validate its AI-driven, mission-critical systems - from multi-agent orchestration and LLM outputs through to cloud infrastructure and real-time user-facing applications.
Why This Role?
You enjoy working close to the technology, influencing how systems are built - not just tested and tackling the realities of validating AI-driven software; this role gives you genuine ownership and impact.
Key Responsibilities
Test Strategy & Leadership
* Define and own the end-to-end test strategy across AI, backend, frontend, and infrastructure layers.
* Establish testing standards appropriate for agentic AI systems, including non-deterministic behaviour and probabilistic outputs.
* Ensure testing aligns with mission-critical, safety-conscious, and security-first delivery expectations.
* Act as the primary quality authority across projects, advising engineering and product leadership on risk and readiness.
AI & Data-Focused Testing
* Design approaches for testing multi-agent workflows, including orchestration logic, memory/state handling, and tool integrations.
* Define validation strategies for LLM outputs, including groundedness, hallucination detection, task success rates, and regression testing.
* Work with AI Engineers to embed evaluation metrics and pass/fail thresholds into pipelines.
* Validate data ingestion, transformation, and inference pipelines across structured and unstructured data sources.
Automation & Tooling
* Drive a test-automation-first mindset, integrating tests into CI/CD pipelines (GitHub Actions, Argo CD).
* Oversee automated testing across API and service layers, UI (E2E and accessibility), and infrastructure and deployment workflows.
* Select, implement, and evolve testing tools and frameworks appropriate to modern cloud-native and AI systems.
Non-Functional Testing
* Own performance, scalability, reliability, and resilience testing for distributed systems.
* Coordinate security testing activities in line with secure-by-design principles (e.g. IAM, secrets handling, data boundaries).
* Validate backup, disaster recovery, and failover scenarios alongside DevOps and Backend teams.
Delivery & Collaboration
* Embed with delivery teams to ensure testing is planned early and executed continuously.
* Work closely with Product and Engineering to define clear acceptance criteria and definition of done.
* Provide clear, decision-ready quality reporting to technical and non-technical stakeholders.
* Support customer-facing demonstrations, trials, and operational readiness assessments.
Required Skills & Experience
* Proven experience as a Test Manager, Senior Test Lead, or equivalent on complex software systems.
* Strong track record of taking applications into production in regulated environments.
* Strong background in automated testing across APIs, services, and UIs, integrated into CI/CD pipelines.
* Experience testing distributed, cloud-native systems (AWS, GCP, or Kubernetes), including performance, reliability, and resilience.
* Awareness of compliance frameworks (e.g. ISO 27001, NIST, OWASP
* ISTQB Advanced / Test Manager certification or equivalent practical experience
* SC Clearance or eligibility to obtain UK SC Clearance.
Preferred Experience
* Experience in UK defence, public sector, or security environments.
* Experience testing AI/ML/LLM-based systems, including non-deterministic outputs.
* Exposure to agent-based or workflow-driven architectures.
Soft Skills
* A pragmatic, delivery-focused mindset - able to balance speed with rigour.
* Comfortable operating in fast-moving, ambiguous, R&D-heavy environments.
* Confidence challenging assumptions and raising quality risks early.
* Strong written and verbal communication, especially around complex technical risk.
Rewards
* Salary: negotiable based on experience and attributes.
* Rapid career progression with meaningful ownership of quality across all products.
* Ability to shape the direction of a fast-moving, successful early-stage business.
* Highly flexible working hours and hybrid working.
#J-18808-Ljbffr