Lead Databricks Engineer - Hybrid - Permanent
Overview
We are seeking an experienced Lead Azure Databricks Engineer to design, build and optimise our enterprise data platforms. The role requires deep technical expertise, strong delivery capability and experience working in highly regulated environments within the Lloyd's of London market.
You will play a key role in shaping our cloud data platform, enabling advanced analytics and ensuring secure, scalable and reliable solutions across the organisation. This is a fully hands-on engineering role.
Key responsibilities
1. Design and build enterprise data platforms using Microsoft Azure services including Azure Data Factory, Azure Data Lake Storage, Azure Key Vault, Azure Functions and Azure Databricks
2. Develop scalable pipelines using Delta Lake, PySpark and Unity Catalog
3. Work with underwriting, actuarial, delegated authority, bordereaux, exposure management, reinsurance, finance and risk teams to deliver solutions aligned to Lloyd's market requirements and regulations including Solvency II
4. Translate business requirements into scalable cloud data solutions with architects, SMEs and product owners
5. Lead the resolution of complex technical challenges and drive delivery
6. Promote best practices in data engineering, cloud architecture, CI/CD and data life cycle management
<...