Data Engineer – SC Cleared - AWS
Our client is establishing a new Data Team and requires a Data Engineer to design and implement the foundational data infrastructure. Current data is dispersed across multiple systems with inconsistent structures and limited automation. You will play a key role in creating scalable, secure data pipelines and models that enable reliable ingestion, transformation, and delivery across the organisation.
Due to the nature of the work, active SC clearance is required.
Key Responsibilities
Design, build, and maintain scalable data pipelines across AWS and Azure environments to ingest, transform, and store data from multiple sources.
Develop and manage data models, schemas, and metadata to support analytics, reporting, and operational requirements.
Work closely with Data Analysts, Data Scientists, and Business Analysts to ensure data is accessible, high-quality, and fit for purpose.
Implement data quality and validation routines, including monitoring, alerting, and automated checks.
Optimise data workflows for performance, cost-efficiency, and maintainability, using platforms such as Azure Data Factory, AWS Data Pipeline, Glue, Lambda, Databricks, and Apache Spark.
Support the integration of transformed data into visualisation and analytical platforms, including Power BI, ServiceNow, and Amazon QuickSight.
Ensure compliance with data governance, security, and privacy standards across all pipelines and processes.
Produce clear documentation of data architecture, pipelines, and operational processes to support knowledge transfer and maintainability.
Contribute to the development of a modern data platform capable of both batch and real-time processing