Job Description
Data Engineer
Azure | Databricks | PySpark | Hybrid Cloud | Fabric
Location: London (Hybrid)
Salary: £85,000 + Bonus + Benefits
Type: Permanent
A Data Engineer is required for a fast-evolving (re)insurance business at the heart of the Lloyd's market, currently undergoing a major data transformation. With a strong foundation in the industry and a clear vision for the future, they're investing heavily in their data capabilities to build a best-in-class platform that supports smarter, faster decision-making across the business.
As part of this journey, they're looking for a Data Engineer to join their growing team. This is a hands-on role focused on building scalable data pipelines and enhancing a modern Lakehouse architecture using Databricks, PySpark, and Azure. The environment is currently hybrid cloud and on-prem, with a strategic move towards Microsoft Fabric-so experience across both is highly valued.
What you'll be doing:
* Building and maintaining robust data pipelines using Databricks, PySpark, and Azure Data Factory.
* Enhance and maintain a Lakehouse architecture using Medallion principles
* Working across both cloud and on-prem environments, supporting the transition to Microsoft Fabric.
* Collaborating with stakeholders across Underwriting, Actuarial, and Finance to deliver high-impact data solutions.
* Support DevOps practices and CI/CD workflows in Azure
Technical Requirements:
* Strong hands-on experience with:
o Databricks (including pipeline development and orchestration)
o PySpark and Python for data transformation and processing
o Azure Data Factory for data integration
* Solid understanding of:
o Medallion architecture and Lakehouse design principles
o Hybrid cloud/on-prem environments
o Microsoft Azure ecosystem
* Exposure to or interest in:
o Microsoft Fabric and its evolving role in enterprise data platforms
o Azure DevOps for CI/CD and deployment
o T-SQL and dimensional modelling (Kimball methodology)
o Experience in Financial Services or Lloyd's market is a plus
Apply now or get in touch to find out more - alexh@pioneer-search.com