We are currently recruiting a Data Engineer for one of our clients. The role is outside IR35 and is paying £400-500 per day, it will initially be for 6 months. It is also fully remote.
Key Responsibilities
• Design, develop, and maintain batch and streaming data pipelines using Databricks (Apache Spark)
• Build and optimize ETL/ELT workflows for large-scale structured and unstructured data
• Implement Delta Lake architectures (Bronze/Silver/Gold layers)
• Integrate data from multiple sources (databases, APIs, event streams, files)
• Optimize Spark jobs for performance, scalability, and cost
• Manage data quality, validation, and monitoring
• Collaborate with analytics and ML teams to support reporting and model development
• Implement CI/CD, version control, and automated testing for data pipelines
Required Qualifications
• 3+ years of experience as a Data Engineer
• Strong experience with Databricks and Apache Spark
• Proficiency in Python (required); SQL (advanced)
• Hands-on experience with AWS or Azure cloud services:
o AWS: S3, EMR, Glue, Redshift, Lambda, IAM
o Azure: ADLS Gen2, Azure Databricks, Synapse, Data Factory, Key Vault
• Experience with Delta Lake, Parquet, and data modeling