Requirements
Must have:
- Proven experience as a Lead or Senior Data Engineer in enterprise-scale environments - Hands-on expertise with Hadoop, Spark, and Splunk - Advanced Python development skills - Experience designing and optimising high-performance data pipelines - Strong understanding of CI/CD, source control, and automated testing - Analytical, problem-solving, and leadership skills - Experience working in regulated, enterprise environments (e.g., banking, fintech, government) - Agile delivery experience (Scrum/Kanban) - Ability to mentor and uplift team performance - Strong communication and stakeholder engagement skills - Collaborative, delivery-focused mindset with high accountability - Knowledge of control, compliance, and regulatory requirements - Up-to-date awareness of modern tools, cybersecurity, and data privacy regulations - Champions innovation, advanced technologies, and best practices
Responsibilities:
- Lead the design, development, and optimisation of enterprise data engineering platforms - Build and maintain robust ETL/ELT pipelines integrating large, complex datasets - Work with structured, semi-structured, and unstructured data across SQL and NoSQL technologies - Develop solutions using Hadoop, Spark, and Splunk in large-scale environments - Write maintainable Python code, applying object-oriented and functional programming principles - Implement and maintain CI/CD pipelines, automated testing, and version control - Collaborate with BI, Analytics, and downstream teams to support reporting and insights - Pair program and mentor other engineers to promote knowledge sharing and code quality - Define and maintain technical test plans including unit and integration tests - Promote SRE principles to ensure service resilience, sustainability, and recoverability
Company:
We are seeking an experienced Lead Data Engineer to design, develop, and optimise enterprise-scale data platforms for large, regulated organisations, ideally in banking, financial services, or other regulated sectors. This role requires hands-on technical expertise, leadership, and a consulting mindset to deliver scalable, resilient data solutions while promoting best practices and operational excellence. The position is a hybrid role based in Sheffield, offering a contract for 6 months with the possibility of extension at a rate of £400 per day inside IR35.