I'm partnered with a PLC organisation based in North Yorkshire who are seeking a Data Engineer who will be responsible for delivering, maintaining and supporting their cloud data platform. This will consist of implementing data pipelines (batch, micro-batch and streaming), a data lakehouse and supporting data consumption workloads including data visualisation and future artificial intelligence initiatives.
The Data Engineer will be a key member of the Group Data Team working closely with technical/non-technical colleagues to deliver data solutions to drive value and help achieve strategic outcomes. You will work collaboratively with stakeholders to aid in decision making and be a trusted, respected & knowledgeable point of contact between the Group Data Analytics team, Infrastructure and Development teams.
Key Responsibilities:
* Develop low-level design documentation based on architecture artefacts.
* Collaborate with cloud infrastructure and development teams for optimised delivery.
* Work within a Software Development Lifecycle (SDLC) to maintain quality and control.
* Partner with data analysts and system owners to define and document data transformation rules and data extraction processes, including interface contracts for each ingested data source.
* Support data modelling efforts, including capturing and maintaining transformation rules.
* Implement ETL/ELT batch and micro-batch pipelines using AWS Glue; integrate data quality assessment via AWS Glue Data Quality Services and AWS Databrew.
* Build streaming pipelines using AWS Kinesis Data Streams, Firehose, and Kafka (MSK).
* Maintain metadata and enforce quality standards using AWS Glue Data Catalog.
* Define and administer the data platform using AWS LakeFormation.
* Use Snowflake to develop modelled data sources and products for end users.
What We're Looking For:
* Experience delivering ETL/ELT pipelines with tools like AWS Glue.
* Proficiency in SQL and Python.
* Strong knowledge of cloud-based data architectures, including data lakes, lakehouses, and warehouses.
* Familiarity with AWS services such as S3, DynamoDB, Aurora, RDS, Glue, Athena, and EMR.
* Understanding of data lakehouse storage formats (e.g., Parquet, Delta, Iceberg).
* Experience with Snowflake and modern data platform practices.
* Knowledge of data modelling methods (Inmon and Kimball).
* Strong communication skills and the ability to adapt style to suit different audiences.
* Ability to produce high-quality documentation (e.g., low-level designs, runbooks).
Desirable:
* Experience with streaming technologies (Kinesis, Firehose, Kafka, Flink).
* Working knowledge of infrastructure as code (CloudFormation, Terraform).
* AWS Certified Data Engineer.
* Degree in a relevant field.
If this role is of interest please apply direct with an updated CV for consideration.
Job Type: Permanent
Pay: £50,000.00-£55,000.00 per year
Benefits:
* Additional leave
* Company pension
* Flexitime
* On-site parking
* Private medical insurance
* Work from home
Work Location: Hybrid remote in Harrogate HG1 3JZ