Job Description
Azure Databricks Architect
6-month contractRemote, with 2 days on-site in London per monthActive SC clearance requiredInside IR35
Overview
We are seeking an experienced Azure Databricks Architect to lead and assure the design and ongoing development of a cloud-based data and analytics platform within a public sector setting. This position centres on platform-level architecture, data design patterns, governance frameworks, and non-functional requirements. You will collaborate closely with delivery teams and stakeholders to ensure the platform meets organisational and government standards for security, scalability, and compliance.
Key Responsibilities
* Define and oversee the end-to-end architecture of Azure Databricks-driven data platforms.
* Create Lakehouse designs incorporating Azure Databricks, ADLS Gen2, and associated Azure services.
* Produce and maintain High-Level Designs (HLDs) and Low-Level Designs (LLDs).
* Develop and enforce best-practice patterns for data ingestion, transformation, and consumption.
* Shape data security models, including RBAC and data separation strategies.
* Ensure platforms adhere to public sector governance, compliance, and security expectations.
* Address performance, resilience, scalability, and cost-efficiency within architectural decisions.
* Participate in architectural assurance processes and structured design reviews.
* Define integration approaches with Azure Data Factory, event-driven and batch ingestion services, and downstream analytics/reporting platforms.
* Provide architectural guidance for CI/CD pipelines and Infrastructure-as-Code.
* Lead optimisation efforts to support sustainable platform operations.
* Act as a design authority for Databricks decisions across multiple delivery teams.
* Support discovery and early-stage design for emerging data and analytics initiatives.
* Identify technical risks, dependencies, and constraints, contributing to mitigation plans.
Essential Skills & Experience
* Strong track record designing Azure Databricks-based architectures in enterprise environments.
* Deep understanding of Lakehouse principles, Delta Lake, and governance capabilities.
* Architectural expertise with Spark-based platforms.
* Experience designing secure, governed platforms within regulated or public sector contexts.
* Proven ability to produce high-quality HLDs, LLDs, and architectural decision artefacts.
* Excellent knowledge of Azure data services and integration technologies.
Desirable Skills & Experience
* Experience delivering data platforms within UK government or wider public sector organisations.
* Familiarity with UK government cloud and security frameworks.
* Background in architectural assurance or design review bodies.
* Understanding of enterprise analytics and reporting ecosystems.
* Relevant Azure or Databricks certifications.
Engagement Model
This role is primarily advisory and architectural rather than hands-on. You will operate across multiple teams and workstreams, supporting both delivery and assurance activities throughout the lifecycle of the data platform.
JBRP1_UKTJ