Data Engineering Lead
Certain Advantage are hiring for a Data Engineering Lead based in London. This role is on an initial contract to the end of the year and is required to be onsite 3 days a week.
The successful candidate will lead the data acquisition function by defining ingestion strategy and building Azure-based data pipelines. You will translate end‑user requirements into scalable engineering solutions, act as the domain’s primary Data Manager for data governance, lineage, quality, and master data management, and lead a team of data engineers in delivering reliable, compliant, and high‑value data solutions to Trading, Risk, Finance, and Operations.
Responsibilities
* Own ingestion strategy and define scalable batch/streaming data patterns
* Design and deliver Azure-based data pipelines and platform solutions
* Standardize connectors, schemas, controls, CI/CD, IaC, and automated testing
* Lead onboarding of enterprise and market data sources with SLAs and observability
* Partner with Product Owners, Architects, and business stakeholders on data design and governance
* Act as domain Data Manager for lineage, metadata, quality, access, and master data ownership
* Ensure compliance with data lifecycle, audit, records management, and governance standards
* Drive engineering best practices, operational excellence, and continuous improvement
* Manage capacity planning, skills development, and vendor contributions
Qualifications
* Expertise in Azure data services (ADLS, ADF, Databricks, Synapse, Event Hub/Kafka), Python, SQL, Spark/PySpark
* Experience with data modeling, metadata and lineage tooling; performance & cost optimization
* Proven leadership in engineering teams; Agile/DevOps ways of working; CI/CD; test automation
* Exposure to energy trading systems (ETRM/Endur), market data (ICE, Platts, Bloomberg), SAP
* Degree in Computer Science/Engineering or equivalent experience; Azure Data Engineer or Databricks certifications a plus
#J-18808-Ljbffr