* Senior Data Engineer
* Location - London
* Hybrid - 3 days in the office
* SC Clearance - ESSENTIAL
* Rate 550-600 per day
* Experience - 8+ years
We are seeking a highly skilled and experienced SC Cleared Senior Data Engineer to join our client and play a key role in developing and maintaining their Azure Databricks platform for economic data. This platform underpins critical Monetary Analysis, Forecasting, and Modelling activities, making this position essential to their mission.
Key Responsibilities
Data Pipeline Development & Optimisation
* Design, develop, and maintain robust, scalable data pipelines for ingesting, transforming, and loading data from diverse sources (APIs, databases, financial data providers) into Azure Databricks.
* Optimise pipelines for performance, efficiency, and cost-effectiveness.
* Implement data quality checks and validation rules within pipelines.
Data Transformation & Processing
1. Implement complex transformations using Spark (PySpark or Scala) and related technologies.
2. Develop logic for data cleaning, enrichment, and aggregation.
3. Ensure data consistency and accuracy throughout the lif...