Senior Data Engineer - Hybrid in London - Outside IR35
Overview
Our client is expanding their core data engineering function and hiring two Senior Data Engineers to strengthen a modern, scalable data platform. Both will be Databricks specialists, with one role focused on deep technical delivery and the other combining hands-on engineering with team leadership and upskilling responsibilities.
Key Responsibilities
* Build and maintain Databricks Delta Live Tables (DLT) pipelines across Bronze ? Silver ? Gold layers, ensuring quality, scalability, and reliability.
* Develop and optimise Spark (PySpark) jobs for large-scale distributed processing.
* Design and implement streaming data pipelines with Kafka/MSK, applying best practices for late event handling and throughput.
* Use Terraform and CI/CD pipelines (GitHub Actions or similar) to manage infrastructure and automate deployments.
* (For leadership-focused role) Mentor and upskill engineers, define coding standards, and embed engineering excellence across the team.
What's Expected
* Proven experience delivering end-to-end data pipelines in Databricks and Spark environments.
* Strong understanding of data modelling, schema evolution, and data contract management.
* Hands-on experience with Kafka, streaming architectures, and real-time processing principles.
* Proficiency with Docker, Terraform, and cloud platforms (AWS, GCP, or Azure) for scalable data infrastructure.
* Demonstrated ability to lead, coach, and elevate team capability, fostering best practices and knowledge sharing.
JBRP1_UKTJ