Overview
Senior Data Engineer — On-site — Full time
Methods Business and Digital Technology Limited is a UK-based IT Services Consultancy with a strong focus on transforming public sector operations and data solutions for central government departments and agencies.
Methods was acquired by the Alten Group in early 2022.
Responsibilities
We are seeking a seasoned Senior Data Engineer to design, build, and maintain sophisticated data infrastructure systems that operate across both on-premises and Azure cloud environments. The role involves deploying and managing scalable data operations that support advanced analytics and data-driven decision-making, contributing to organisational growth and innovation.
* Develop and Manage Data Pipelines: design, construct, and maintain efficient and reliable data pipelines using Python/Go/Azure Data Factory, supporting streaming and batch processing across structured, semi-structured, and unstructured data in on-premises and Azure environments.
* Hybrid Cloud and Data Storage Solutions: implement and manage data storage solutions leveraging on-premises infrastructure and Azure, ensuring seamless data integration and accessibility across platforms.
* Containerisation and Orchestration: utilise Docker for containerisation and Kubernetes for orchestration to enable scalable deployment across cloud and on-premises environments.
* Workflow Automation: use tools such as Azure Data Factory to automate data flows and manage complex workflows within hybrid environments.
* Event Streaming Experience: utilise event-driven technologies such as Kafka and NATS to handle real-time data streams.
* Security and Compliance: manage security setups and access controls, including tools like Keycloak to protect data integrity and comply with legal standards.
* Database Development: design and develop PostgreSQL databases with high performance and availability across deployment scenarios.
Essential Skills and Experience
* Strong Python Skills: expertise in Python for scripting and automating data processes across varied environments.
* Experience with ETL/ELT: demonstrable experience in developing and optimizing ETL or ELT workflows in hybrid (on-premises and Azure) environments.
* Expertise in Hybrid Cloud Data Architecture: knowledge of integrating on-premises infrastructure with Azure cloud services.
* Containerisation and Orchestration Expertise: solid experience with Docker, GitHub and Kubernetes in managing applications across on-premises and cloud platforms.
* Proficiency in Workflow Automation Tools: practical experience with Azure Data Factory in relevant environments.
* Experience in Event Streaming: proven ability in managing and deploying event streaming platforms like Kafka and NATS.
* Data Security Knowledge: experience with implementing security practices and tools, including Keycloak, across multiple platforms.
* Search and Database Development Skills: strong background in Elasticsearch and PostgreSQL across on-premises and cloud infrastructures.
Your Impact
You will empower business leaders to make informed decisions by delivering timely, accurate, and actionable data insights from a robust, hybrid infrastructure. You will champion adoption of modern data architectures, mentor team members, and advance engineering practices to cultivate a data-driven culture within the organisation.
Desirable Skills and Experience
* Azure and Other Relevant Certifications: cloud and on-premises certifications are beneficial.
* Data Engineering Experience: minimum of 5 years in data engineering with significant exposure to on-premises and cloud infrastructure.
* DevOps Experience: some DevOps engineering experience would be preferable.
#J-18808-Ljbffr