Lead Data Engineer
Contract
Outside IR35
6 Months Initially
2/3 days a week on-site - Birmingham
ASAP Start
My client are seeking an experienced Lead Data Engineer to lead the migration of customer data from a third-party platform onto the new Databricks environment. This is a hands-on leadership role requiring deep Databricks expertise combined with architectural ownership and delivery accountability.
Key Responsibilities
Lead the migration of customer data from an external third-party provider to the new Databricks platform.
Define and implement the end-to-end data architecture strategy within Databricks.
Design scalable, secure, and high-performance data pipelines.
Establish governance, data modelling standards, and best practice frameworks.
Act as technical authority and SME for Databricks across the programme.
Provide architectural oversight while remaining hands-on with engineering delivery.
Required Experience
Deep hands-on expertise in Databricks (architecture, optimisation, deployment).
Deep expertise in Azure data services (Databricks, Data Factory, ADLS) and strong understanding of distributed data processing.
Hands-on experience with CI/CD tooling (Azure DevOps/GitHub) and modern DevOps practices for data engineering.
Strong grasp of data architecture, ETL/ELT design, and performance optimisation.
Desirable Experience
Proven experience leading enterprise data platform implementations.
Strong experience migrating customer data between platforms.
Experience operating in senior/lead/senior/lead/principle-level engineering roles.
Strong understanding of data governance, security, and compliance frameworks.
Comfortable leading small, high-performing teams.
Experience working within retail, eCommerce, or customer-focused environments (preferred).