The West of England Combined Authority are seeking a highly skilled Data Engineer to support the optimisation, documentation, and enhancement of our existing data pipelines within the Databricks Lakehouse environment. This role will involve hands-on development, reverse-engineering of current solutions, and close collaboration with internal teams to ensure efficient BAU operations and future scalability.
Location: West of England Combined Authority
Set-up: Hybrid – 2-3 days on-site
Rate: £249/day via umbrella inside IR35
Duration: Initial 3-month, subject to extension
Hours: Full-time position
Responsibilities include:
1. Develop, optimise, and debug Databricks notebooks, Delta Lake workflows, and Genie-bot automations
2. Design, maintain, and enhance ETL pipelines, ensuring reliability and performance
3. Validate, transform, and orchestrate data using Python and SQL within the Lakehouse environment
4. Manage data ingestion and scheduling processes via Azure Data Factory
5. Reverse-engineer, document, and communicate existing data flows and pipeline structures for handover to internal teams
Requirements:
6. Strong hands-on experience with Databricks, including notebooks, Delta Lake, and Genie bots
7. Proven background in designing and maintaining ETL pipelines, including debugging and optimisation
8. Advanced Python skills for transformation, automation, and pipeline development
9. Solid SQL proficiency for querying, validation, and performance tuning
10. Experience using GitHub for version control, collaboration, and code management
At Venn Group, we endeavour to respond to all applications However due to the volume of applications we receive, if you have not heard from us within 48 hours please assume you have been unsuccessful on this occasion