Azure Databricks Architect
6-month contract
Remote, with 2 days on-site in London per month
Active SC clearance required
Inside IR35
Overview
We are seeking an experienced Azure Databricks Architect to lead and assure the design and ongoing development of a cloud-based data and analytics platform within a public sector setting. This position centres on platform-level architecture, data design patterns, governance frameworks, and non-functional requirements. You will collaborate closely with delivery teams and stakeholders to ensure the platform meets organisational and government standards for security, scalability, and compliance.
Key Responsibilities
Define and oversee the end-to-end architecture of Azure Databricks-driven data platforms.
Create Lakehouse designs incorporating Azure Databricks, ADLS Gen2, and associated Azure services.
Produce and maintain High-Level Designs (HLDs) and Low-Level Designs (LLDs) .
Develop and enforce best-practice patterns for data ingestion, transformation, and consumption.
Shape data security models, including RBAC and data separation strategies.
Ensure platforms adhere to public sector governance, compliance, and security expectations.
Address performance, resilience, scalability, and cost-efficiency within architectural decisions.
Participate in architectural assurance processes and structured design reviews.
Define integration approaches with Azure Data Factory, event-driven and batch ingestion services, and downstream analytics/reporting platforms.
Provide architectural guidance for CI/CD pipelines and Infrastructure-as-Code.
Lead optimisation efforts to support sustainable platform operations.
Act as a design authority for Databricks decisions across multiple delivery teams.
Support discovery and early-stage design for emerging data and analytics initiatives.
Identify technical risks, dependencies, and constraints, contributing to mitigation plans.
Essential Skills & Experience
Strong track record designing Azure Databricks-based architectures in enterprise environments.
Deep understanding of Lakehouse principles, Delta Lake, and governance capabilities.
Architectural expertise with Spark-based platforms.
Experience designing secure, governed platforms within regulated or public sector contexts.
Proven ability to produce high-quality HLDs, LLDs, and architectural decision artefacts.
Excellent knowledge of Azure data services and integration technologies.
Desirable Skills & Experience
Experience delivering data platforms within UK government or wider public sector organisations.
Familiarity with UK government cloud and security frameworks.
Background in architectural assurance or design review bodies.
Understanding of enterprise analytics and reporting ecosystems.
Relevant Azure or Databricks certifications.
Engagement Model
This role is primarily advisory and architectural rather than hands-on. You will operate across multiple teams and workstreams, supporting both delivery and assurance activities throughout the lifecycle of the data platform.
TPBN1_UKTJ