AI Engineer - Databricks and Spark essential
London based.
Start in May until 31st Dec.
Certain Advantage are recruiting for an AI Engineer with hands on Databricks and Spark expertise. Industry experience in commodity or financial trading is strongly preferred.
This is a fantastic opportunity to partner with traders and trading analysts to rapidly build AI‑powered analytics over market pricing and fundamentals data, using Databricks and Spark to deliver value at speed.
Someone with familiarity in market microstructure, supply‑demand fundamentals, risk management with confident collaboration and engagement skills with trading teams would be ideal.
Responsibilities to include
Design and ship AI‑driven analytics for front‑office use (seasonality, correlation, regression, forecasting, scenario modelling).
Build reusable and scalable data pipelines in Databricks (PySpark/Spark, Delta/Unity Catalog), optimizing cost, reliability, and performance.
Run statistical/econometric analyses on large datasets (e.g., market & fundamental time series data).
Collaborate directly with traders/analysts—translate ambiguous questions into shippable solutions; communicate insights clearly.
Implement LLM/agentic workflows: prompt engineering, LangGraph orchestration, MCP integrations, tool calling, retrieval, and guardrails.
Productionize solutions with testing, observability, versioning, and documentation.What we’re looking for
Hands‑on Databricks + Spark expertise (PySpark, SQL, Delta, Unity Catalog).
Proven data engineering skills (ingestion, modelling, orchestration, performance tuning).
Strong statistics/economics/data science fundamentals for market time‑series.
Experience building LLM solutions (prompting, retrieval, agent flows; LangGraph, MCP) and integrating with trading data/services.
Experience with CI/CD, Terraform, MLflow/feature stores, vector DBs, and governance (PII handling, data lineage).
Excellent stakeholder skills; able to work on‑desk with traders/analysts and deliver fast.
Does this sound like your next career move? Apply today!
Working with Certain Advantage
We go the extra mile to find the best people for the job. If you’re hunting for a role where you can make an impact and grow your career, we’ll work with you to find it.
We work with businesses across the UK to find the best people in Finance, Marketing, IT and Engineering.
If this job isn’t for you, head to (url removed) and register for job alerts and career guidance tips