Senior Data Engineer
We are looking for a Senior Data Engineer with the ability to propose improvements with a degree of autonomy, as well as the skills to lead data pipeline development and drive data modelling initiatives in close partnership with business areas. The successful candidate will be responsible for orchestrating the entire data engineering journey — from the ingestion of data generated by Python-based Azure Functions through to the delivery of solutions built on Azure Data Factory and Databricks.
Key Responsibilities
* Develop and manage data pipelines that consume CSV files generated by Azure Functions, ensuring reliability and maintainability across the data flow.
* Structure data modelling aligned with business needs, guaranteeing quality and performance throughout the data processing lifecycle.
* Orchestrate processes within Azure Data Factory, ensuring scalability and operational efficiency across all data workflows.
* Develop and optimise data workflows in Databricks to support advanced analytics and data transformation requirements.
* Collaborate directly with business teams to translate functional requirements into robust technical solutions.
* Ensure solid data governance practices, including data quality, versioning, and end-to-end data lineage and traceability.
* Document solutions and produce clear manuals and guidelines for both technical and non-technical stakeholders.
Mandatory Technical Requirements
* Proficiency in Python, with the ability to read and interpret scripts running on Azure Functions. Strong knowledge of Azure Data Factory, with hands‑on experience in pipeline orchestration and automation.
* Proven experience with Databricks, including familiarity with the Medallion Architecture (Bronze, Silver and Gold layers) for distributed data processing.
* Solid understanding of SQL and data modelling techniques, covering both relational and non‑relational approaches.
* Experience with data ingestion, transformation and storage using Azure Blob Storage and/or Azure Data Lake.
* Understanding of data governance and security best practices within Azure cloud environments.
* Familiarity with code versioning using Git and deployment automation practices.
* Familiarity with DataOps practices applied to data engineering workflows.
* Experience working within Agile methodologies (Scrum and/or Kanban).
Behavioural Competencies
* Excellent communication skills, with the ability to engage effectively with business teams and translate their needs into technical solutions.
* Capacity to work independently, with autonomy to prioritise tasks and make sound technical decisions.
* Collaborative mindset, with the ability to lead both technical and business‑facing discussions alongside the direct manager.
Contract & Location
6‑month contract.
On‑site work 2‑3 times per week.
Candidates should preferably reside in or near the Warwick area (Manchester, Coventry, Birmingham, Banbury, Solihull, Redditch, Oxford, Northampton) and have fluent English.
J-18808-Ljbffr