Job Description
Job Title: SC Cleared Python Data Engineer - Azure Cloud
Contract: 12 months
Rate: Up to £410/day (Inside IR35)
Location: UK Based - Remote
Start: January 2026
Clearance: Active SC Clearance required
Role Overview
We are seeking an SC Cleared Python Data Engineer with strong hands-on experience in PySpark, Delta Lake, and Azure cloud services. The role focuses on designing and delivering scalable, well-tested data pipelines, with particular importance placed on the ability to understand, explain, and design PySpark architectures and demonstrate deep, production-grade Python expertise.
You will work in a containerised, cloud-native environment, delivering maintainable, configurable, and test-driven solutions as part of a multi-disciplinary engineering team.
Key Responsibilities
* Design, develop, and maintain data ingestion and transformation pipelines using Python and PySpark.
* Clearly articulate PySpark architecture, execution models, and performance considerations to both technical and non-technical stakeholders.
* Implement unit and BDD testing (Behave or similar), including effective mocking and dependency management.
* Design and optimise Delta Lake tables to support ACID transactions, schema evolution, and incremental processing.
* Build and manage Docker-based environments for development, testing, and deployment.
* Develop configuration-driven, reusable codebases suitable for multiple environments.
* Integrate Azure services including Azure Functions, Key Vault, and Blob/Data Lake Storage.
* Optimise Spark jobs for performance, scalability, and reliability in production.
* Collaborate with Cloud, DevOps, and Data teams to support CI/CD pipelines and environment consistency.
* Produce clear technical documentation and follow cloud security and data governance best practices.
Required Skills & Experience
* Strong Python expertise, with a demonstrable depth of experience in designing modular, testable, production-quality code.
* Proven experience explaining and designing PySpark architectures, including distributed processing and performance tuning.
* Hands-on experience with Behave or similar BDD frameworks, including mocking and patching techniques.
* Solid understanding of Delta Lake concepts, transactional guarantees, and optimisation strategies.
* Experience using Docker across development and deployment workflows.
* Practical experience with Azure services (Functions, Key Vault, Blob Storage, ADLS Gen2).
* Experience building configuration-driven applications.
* Strong problem-solving skills and ability to work independently in agile environments.
Desirable Experience
* Databricks or Synapse with Delta Lake.
* CI/CD pipelines (Azure DevOps or similar) and infrastructure-as-code.
* Knowledge of Azure data security and governance best practices.
* Experience working in distributed or multi-team environments.