Build and scale the "Safety & Trust" engine for our agentic AI ecosystem. You will be the technical lead responsible for ensuring our AWS Bedrock-based agents meet the highest standards of FCA Operational Resilience, DORA, and EU AI Act (Art. 15) compliance.
Core Responsibilities
Automated Red Teaming: Implement adversarial testing (Garak, Pyrit, AgentDojo) directly into CI/CD pipelines with automated release gating.
Centralised Eval Platform: Operate a firm-wide service to measure success rates, uncertainty, hallucination, and bias across all non-deterministic systems.
Secure Architecture: Map OWASP LLM Top 10 and agentic threats to technical controls; manage AWS Bedrock Guardrails and Knowledge Bases.
AI Supply Chain: Own the AI-BOM, ensuring supply chain integrity, signed artifacts, and drift monitoring.
Regulatory Evidence: Produce the technical documentation and robust testing evidence required for EU AI Act Article 15.
Technical Requirements
AWS Bedrock Expert: Hands-on experience with Bedrock Agents, Knowledge Bases, and model lifecycle management.
AI/ML Depth: Strong grasp of FMs, RAG, tool-use, and the failure modes of agentic workflows.
Security & Compliance: Deep knowledge of NIST AI RMF, OWASP LLM Top 10, and UK/EU financial regulations (FCA/DORA).
Testing Automation: Proven ability to build measurement frameworks for drift, memorization, and adversarial robustness.
Qualifications
Significant experience in UK Financial Services.
Expertise in automated adversarial testing and evaluation at scale.
Ability to bridge the gap between complex AI engineering and rigid regulatory requirements.
Randstad Technologies is acting as an Employment Business in relation to this vacancy