You will lead the data-engineering thread across a fast-growing, multi-project enterprise programme. Most of the active workstreams are Databricks-led — this is the engineer who unlocks them. You will design, build and own production-grade Azure + Databricks pipelines that feed AI-powered insights, natural-language querying and downstream CRM and operational activation — and you will do it client-facing, in UK hours, alongside a small, senior team.
What you'll own
* Architecture leadership. Define and document target architectures across Azure + Databricks for new and in-flight projects; review designs from other engineers.
* End-to-end pipeline delivery. Build production ingestion → bronze/silver/gold (medallion) → curated data products using ADF, PySpark, Delta Lake and Databricks Workflows / DLT.
* AI-grounded data products. Partner with the GenAI engineer to expose Gold data to RAG agents, Azure AI Search and Azure OpenAI — including chunking, embeddings, citations and evals.
* CI/CD + IaC discipline. Use Databricks Asset Bundles, Terraform/Bicep and Azure DevOps Pipelines to make every change reviewable, testable and roll-back-able.
* Governance. Apply Unity Catalog, lineage, Purview labels and least-privilege access on day one — not as an afterthought.
* Client interface. Lead design reviews, status calls and stakeholder workshops in UK hours; turn ambiguous client asks into scoped, estimable work.
* Mentorship. Coach mid-level engineers, set patterns, run code reviews and quality bars across the data team.
Must-have experience and skills
Technical (deep)
* 8+ years in data engineering, with 4+ years on Azure data services in production.
* Expert in Azure Databricks — Delta Lake, PySpark, SQL, Workflows / Jobs, cluster tuning, MLflow basics.
* Strong with Azure Data Factory (or Synapse Pipelines), ADLS Gen2, Azure SQL, Key Vault.
* Medallion / lakehouse architecture experience — bronze / silver / gold modelling with real production trade-offs.
* Hands-on with at least one IaC tool (Terraform or Bicep) and CI/CD via Azure DevOps or GitHub Actions; Databricks Asset Bundles a plus.
* Working knowledge of Azure AI Search and Azure OpenAI — including RAG patterns, embeddings, chunking strategies and grounded prompting.
* Comfortable with Unity Catalog, role-based access and data governance practices (Purview, sensitivity labels).
* Strong Python + SQL; clean Git practices; test-driven mindset.
Delivery & engagement
* Demonstrated experience leading multi-stream programmes — running design reviews, estimation, scope conversations.
* Proven client-facing track record: comfortable presenting to senior stakeholders, defending a design, navigating ambiguity.
* Excellent written English; structured, concise communication.