Databricks Data Engineer – London
Location: London (hybrid, 3 office days per week). Employment Type: Permanent, full time.
About the Role
We’re focused on making data our new superpower as one of only four strategic priorities for The AA Group. This role builds, runs and evolves production‑grade Databricks Lakehouse solutions across structured streaming pipelines, Unity Catalog governance and Spark data engineering at scale.
Responsibilities
* Design, build and operate production‑grade Databricks Lakehouse solutions, including structured streaming pipelines using Python and PySpark.
* Own and evolve Unity Catalog‑based governance to ensure secure, discoverable and well‑managed data assets.
* Develop and maintain event‑driven data pipelines, coordinating closely with backend engineering teams.
* Implement and support CI/CD pipelines in Azure DevOps to enable reliable, automated Databricks deployments.
* Create high‑quality, analytics‑ready datasets that deliver actionable business insight at scale.
* Proactively improve performance, reliability, automation and observability across the Databricks data platform.
Essential Experience
* Significant, hands‑on experience with Azure Databricks in production, enterprise‑scale environments.
* Proven expertise building Spark/Databricks pipelines using Python and PySpark.
* Strong experience with structured streaming and event‑driven architectures.
* Practical experience implementing and operating Unity Catalog.
* Solid understanding of Lakehouse design principles, including dimensional and analytical modelling.
* Experience building and maintaining CI/CD pipelines, ideally using Azure DevOps.
* Confidence working with large‑scale data, performance tuning and troubleshooting complex pipelines.
* Strong debugging and problem‑solving skills, including diagnosing and resolving issues in Spark/Databricks pipelines.
What you have
* Hands‑on experience configuring and managing Databricks environments, including cluster setup, dependency management and package installation.
* Designed and built Databricks pipelines end‑to‑end.
* Made architectural decisions within Databricks environments.
* Worked with Spark internals, optimisation techniques and cluster configuration.
* Operated Databricks solutions in live, business‑critical contexts.
Interview Process
Candidates will be assessed on hands‑on technical ability in Azure Databricks from the first stage of the interview process. Assessments include discussion of real‑world implementations, debugging approaches and Databricks environment configuration – clusters, dependencies and packages.
Benefits
* Annual leave: 25 days plus bank holidays, holiday buying scheme.
* Work‑sav e pension scheme with up to 7% employer contribution.
* Free AA breakdown membership from Day 1 plus 50% discount for family and friends.
* Discounts on AA products including car and home insurance.
* Employee discount scheme providing car salary sacrifice, healthcare, shopping and holidays discounts.
* Company‑funded life assurance.
* Diverse learning and development opportunities.
* Dedicated Employee Assistance Programme and a 24/7 remote GP service.
* Annual Bonus.
Equal Opportunity
We’re an equal opportunities employer and welcome applications from everyone. The AA values diversity and the difference this brings to our culture and our customers. We actively seek people from diverse backgrounds to join us and become part of an inclusive company where you can be yourself, be empowered to be your best and feel like you truly belong. We have five communities to bring together people with shared characteristics and backgrounds and drive positive change.
#J-18808-Ljbffr