Principal Data Engineer, Outside IR35, Fully Remote (UK ONLY) – Databricks & Streaming Expertise
A leading consultancy is partnering with a major transport operator on a high-impact rail infrastructure programme. We’re seeking a Principal Data Engineer with deep expertise in Databricks, Spark Streaming, and Azure data services, particularly around streaming data sources and real-time telemetry.
Project Overview
This role is part of a strategic initiative to modernise data infrastructure across the rail network. You’ll architect and optimise streaming data solutions that deliver operational insights and enable real-time decision-making.
Key Skills Required
* Extensive experience with Databricks, especially in streaming data pipelines
* Strong hands-on experience with Spark Streaming
* Solid background in Azure data services, including:
* Azure Synapse
* Azure Data Factory (ADF)
* Databricks
* Proven ability to work with streaming data sources and real-time telemetry
* Ideally some experience with Scala
* Ability to work with large-scale, fast-moving datasets in production environments
* Consultancy background with experience delivering complex data solutions for enterprise clients
Principal-Level Responsibilities
As a Principal Data Engineer, you will:
* Define and lead data architecture for real-time streaming solutions
* Shape and implement data strategy aligned with business objectives
* Influence and engage senior stakeholders, providing thought leadership on data-driven decision-making
* Mentor and guide engineering teams, ensuring best practices and technical excellence
* Drive innovation in data engineering frameworks and processes
Why Apply?
* Work on a high-visibility transport project with real-world impact
* Lead cutting-edge data engineering practices at scale
* Collaborate with a forward-thinking consultancy and a major transport group
* Opportunity to shape the future of rail data systems
Apply now online for more details.