We are excited to offer a fantastic permanent Data Lake Engineer opportunity offering Hybrid working with travel up to once a week across one of Amey's offices in Birmingham, London and Liverpool. HOURS OF WORK - Monday - Friday 37.5 Hours Join our vibrant, inclusive community in Group IT, lead and scale our Azure-based data platform. You will own the end-to-end design and optimisation of high-performance, cloud-native data pipelines - ensuring secure, cost-efficient delivery of high-quality datasets across our medallion architecture.
Responsibilities
* Lead the design and implementation of distributed data processing pipelines (batch and streaming) using PySpark/Spark and Python on Azure Databricks.
* Architect and evolve our medallion data platform, ensuring data quality, observability and reliability across all layers (Bronze, Silver, Gold).
* Define and enforce strong data quality controls, validation rules and monitoring strategies to maintain trusted, production‑grade datasets.
* Make technical decisions on cluster strategy and cost optimisation (e.g., interactive vs. job clusters, autoscaling, serverless, pools, spot instances).
* Build and operationalise data workflows using Databricks Workflows, Azure Data Factory, or Fabric, ensuring efficient orchestration, scheduling and alerting.
* Drive API‑based data integrations, ensuring secure, scalable and low‑latency ingestion of data from third‑party and internal systems.
* Champion DevOps standards across the team (CI/CD, TDD, Infrastructure‑as‑Code, Git‑flow, Azure DevOps/GitHub).
* Embed and enforce security and governance best‑practices, including data masking, encryption and RBAC throughout the ETL lifecycle.
* Mentor junior engineers and contribute to technical standards and architecture roadmaps.
* Engage with data scientists, analysts and business stakeholders to deliver high‑value, production‑ready data products.
Qualifications
* Ability to drive technical direction and influence architectural decisions.
* Strong stakeholder management and communication skills.
* Mentoring mindset, collaborative and proactive.
* Operates with a product mindset and ownership mentality.
* Thrives in a fast‑moving environment with ambiguity and growth.
* Extensive experience working with large‑scale data engineering solutions in public cloud (Azure preferred).
* Deep hands‑on experience with Databricks, ADLS, Delta Lake and distributed compute paradigms.
* Prior ownership of mission‑critical production pipelines, including data quality monitoring, automated alerting and remediation.
* Strong track record of cost and performance optimisation at scale, including cluster tuning and workload refactoring.
* Experience with leading orchestration tools (e.g. Databricks Workflows, ADF, Airflow).
* Proven experience integrating APIs and external data services into data platforms.
* Proficiency in SQL, Python, PySpark and software engineering best practices.
* Experience in DevOps, CI/CD, infrastructure automation and data security.
* Exposure to MLOps, ML model deployment or LLM‑based pipelines is a plus.
Benefits
* Remuneration - Enjoy a competitive annual salary with the potential for yearly reviews to ensure you’re rewarded for your contributions.
* Career Growth - Shine in your career with advancement opportunities.
* Training Opportunities - Unlock your potential with comprehensive training, including fully funded leadership programs tailored to your personal growth.
* Holidays - Enjoy at least 24 days of holiday plus bank holidays, and the opportunity to buy further days.
* Pension - Generous pension scheme, with extra contributions from Amey.
* Flexible benefits - Customise your benefits with options such as insurance benefits, Cycle2Work scheme and access to discounted gym membership.
* Exclusive Discounts - Access our online portal filled with discounts from leading retailers, healthcare services, and more, helping you save on the things that matter.
* Give Back to community - Two Social Impact Days each year, for volunteering and fundraising opportunities.
* Family friendly policies for new parents or if you provide care for a dependant.
* Membership of our Affinity Networks who connect, support and inspire diverse communities within Amey.
#J-18808-Ljbffr