Data Engineering & Agentic AI Architect
UST are currently recruiting for a permanent Data Architect with experience working with Agentic AI to work with one of our clients on AI transformation project.
Location-London
Hybrid- 3 days per week in the office
About this role
The successful candidate will shape the technical direction of large, complex Data & Digital Transformation programmes and delivering engineering Agentic AI systems at scale, to unlock Transformation pace, quality and impact - leaving behind something that stands the test of time?
You will be undertaking a monumental Transformation to unlock organisational pace and Data agility, moving to a scalable Cloud Data architecture centred around AWS Glue (Apache Spark) & Apache Airflow, with a key focus on re-engineering numerous complex Data pipelines to the new framework by leveraging Agentic AI, all whilst embedding great Data Management to build deep trust in Data.
We're looking for a Staff-level Data & Agentic AI engineer to join the programme - driving the agentic AI based conversion framework & Data pipeline modernisation, partnering across a Senior team of hands on Data Engineers, and driving the programme to completion with the quality and rigour that the Data engineering organisation can own and easily maintain into the future.
What you’ll do
* Shape the technical direction for the Data pipeline modernisation and re-engineering programme:
* Define architecture, standards, engineering approach, and testing framework
* Ensure coherence across all workstreams from discovery through to validated delivery
* Architect and build the Claude Code agentic workflow:
* Parse existing data movement pipelines and logic
* Generate equivalent PySpark transformations
* Produce Airflow DAGs
* Operate, iterate, and scale across the full data pipeline estate
* Ensure rigorous creation and management of the PySpark data processing layer within AWS Glue:
* Support data pipeline re-engineering
* Ensure performance, standardisation, and cost efficiency
* Design, write, and refine prompt architecture and context management logic:
* Govern Claude Code’s output
* Ensure consistent conformance to the target framework across complex conversions
* Build automated test harnesses:
* Validate functional equivalence between existing and new pipelines
* Ensure business logic is preserved throughout modernisation
* Develop structural validation and static analysis tooling:
* Evaluate AI-generated output for correctness
* Minimise hallucination risks
* Ensure zero technical debt in modernised pipelines
* Lead and coordinate engineering team efforts:
* Align technical direction
* Resolve architectural ambiguity
* Maintain delivery momentum in a fast-paced programme
* Build and maintain stakeholder relationships:
* Engage senior leadership
* Clearly translate progress, AI output quality, and technical risks
* Enable automated, high-quality documentation:
* Cover both platform and data pipelines
* Define SLOs and data producer/consumer agreements
* Leverage AI to improve documentation and delivery speed
* Design for maintainability and transferability:
* Ensure frameworks, workflows, and pipelines are fully ownable post-programme
* Treat handover quality as a first-class engineering deliverable
What we’re looking for
* Proven experience setting technical direction and architectural standards on complex, multi-workstream programmes
* Hands-on capability to build solutions as well as design them
* Deep expertise in Python and PySpark:
* Strong track record delivering large-scale distributed data systems
* Experience with Claude Code or similar agentic AI platforms:
* Prompt engineering
* Context management
* Workflow design for scalable automated code generation in regulated environments
* Strong hands-on experience with AWS ecosystem:
* AWS Glue, S3, and related data platforms (e.g. Snowflake)
* Strong experience with Apache Airflow and DAG-based orchestration
* Experience designing configuration-driven frameworks:
* Using YAML or JSON
* Experience building automated testing for data pipelines:
* Functional equivalence testing
* Regression testing
* Proven technical leadership:
* Ability to influence without authority
* Align senior engineers across data and AI domains
* Experience working in large, regulated enterprise environments:
* Navigate governance
* Build trust while maintaining pace
* Strong business acumen:
* Ability to communicate technical concepts, risks, and delivery status to senior stakeholders
* Experience designing for maintainability and handover:
* Build systems and documentation for long-term ownership by others