We are seeking an experienced Senior Data Engineer with strong Databricks expertise to design, build, and maintain a scalable cloud-based data platform for a private equity firm. This is a 6-month contract based in London with hybrid working.
Key Responsibilities
* Design and build robust data pipelines using Azure Data Factory and Databricks
* Develop ETL/ELT processes to transform raw data into analytics-ready datasets
* Architect and maintain data lake solutions using Azure Data Lake Storage
* Implement Databricks solutions using Delta Lake and medallion architecture (bronze, silver, gold)
* Optimise performance, data quality, and cost efficiency across pipelines and clusters
* Set up monitoring, alerting, and ensure high availability of data services
* Collaborate with data teams and stakeholders to deliver business-focused solutions
Essential Skills & Experience
1. Strong hands-on experience with Databricks (essential)
2. Proven experience with Azure services
3. Proficiency in Python and SQL
4. Experience with CI/CD, Git, and deployment pipelines
5. Solid understanding of data modelling and data architecture
6. Knowledge of data governance and security best practices
7. Experience with Terraform or Infrastructure-as-Code
8. Azure DevOps
9. Relevant Azure or Databricks certifications