Lead Data Engineer (Databricks & Cloud Migration)
£650 per day (Inside IR35)
Initial 6-Month Contract
London – Hybrid Working (3 Days per Week Onsite)
We are partnering with a leading international bank on a major Databricks-led cloud transformation programme. We are looking for a Lead Data Engineer with deep expertise in Databricks, modern data architectures and large-scale cloud migrations to help deliver a next-generation data platform. You'll play a pivotal role in designing and building Data Marts, developing high-performance data pipelines, and migrating critical data assets from legacy Oracle environments into a scalable Databricks Lakehouse architecture that underpins finance, regulatory reporting and advanced analytics across the organisation.
Key Responsibilities
Databricks & Modern Data Platform Engineering
* Lead the design and development of scalable Data Marts using Databricks.
* Build and optimise cloud-native data pipelines using SQL and Python.
* Drive engineering best practices across CI/CD, version control, orchestration and automated deployment.
* Ensure platform performance, scalability, governance and operational excellence.
* Champion modern data engineering standards and architectural best practices.
Oracle to Cloud Migration
* Lead the migration of data assets from legacy Oracle environments to a modern cloud-based platform.
* Define migration strategies, source-to-target mappings and data model redesign activities.
* Oversee the retirement of legacy ETL processes and implementation of cloud-native alternatives.
* Work closely with architects, DBAs and business stakeholders to ensure data quality, integrity and continuity throughout the migration programme.
Leadership & Stakeholder Engagement
* Provide technical leadership, mentoring and guidance to Data Engineers.
* Collaborate with Architecture, DevOps, Analytics, Finance and Risk teams.
* Communicate complex technical concepts effectively to both technical and non-technical stakeholders.
* Conduct design reviews and ensure delivery aligns with engineering standards and enterprise architecture.
Operational Excellence
* Support production releases and change management activities.
* Drive improvements in automation, monitoring, observability and platform reliability.
* Assist with incident resolution and critical release support when required.
* Ensure documentation, audit controls and governance requirements are maintained.
Required Experience
* Proven experience delivering enterprise-scale Data Engineering solutions using Databricks or similar cloud data platforms.
* Strong experience designing and developing modern data architectures and Data Marts.
* Demonstrable experience migrating Oracle or other legacy data platforms into cloud environments.
* Deep understanding of data modelling, ELT/ETL design patterns and data integration principles.
* Advanced SQL development skills.
* Strong Python development experience.
* Experience with CI/CD pipelines, Git-based workflows and DevOps practices.
* Experience leading engineering teams and providing technical mentorship.
* Strong stakeholder management and communication skills.
Desirable Experience
* Banking or financial services experience.
* Experience within highly regulated environments.
* Knowledge of Master Data Management (MDM), metadata management and data governance frameworks.
* Experience with orchestration tools such as Azure Data Factory or Airflow.
* Azure and/or Databricks certifications.
What's on Offer
* £650 per day Inside IR35.
* Initial 6-month contract with strong potential for extension.
* Opportunity to work on a high-profile Databricks and cloud transformation programme.
* Hybrid working model with 3 days per week onsite in London.
* Collaborative environment working alongside senior technology and business stakeholders.