Overview
Platform Engineer- 400PD-Remote
We are seeking a highly skilled Cloud Platform Engineer with hands-on experience managing and scaling Databricks from a platform perspective, along with deep knowledge of Azure services, Terraform, and CI/CD pipelines using GitHub Actions. You will be part of a team responsible for designing, implementing, and maintaining scalable, secure, and cost-effective cloud infrastructure to support our data and analytics platform.
This role requires a strong background in cloud infrastructure automation, platform governance, and enterprise-grade security and compliance in Azure.
Responsibilities
* Design, build, and manage infrastructure using Terraform to enable repeatable and scalable cloud environments in Azure.
* Implement and maintain CI/CD pipelines using GitHub Actions to automate deployment and testing processes.
* Manage and optimize Azure Private Link configurations to ensure secure connectivity between services.
* Define and enforce Azure Policies and RBAC for governance and compliance.
* Implement and monitor Azure Security Center, defender policies, and other security controls across cloud resources.
* Monitor and manage cloud costs using Azure Cost Management, ensuring resources are optimized and within budget.
* Architect and support Databricks workspaces, focusing on Unity Catalog, data access control, workspace deployment, and platform reliability.
* Collaborate with data engineering and analytics teams to ensure Databricks is performant, secure, and well-integrated with enterprise systems.
* Create and maintain platform documentation, templates, and reusable modules to support self-service provisioning.
* Participate in incident response and troubleshooting for cloud platform issues, particularly those related to Databricks and underlying infrastructure.
Required Skills & Experience
* Proven experience with Terraform in a production environment.
* Deep understanding of GitHub Actions and CI/CD best practices.
* Strong experience working with Azure Private Link, VNET integration, and network security.
* Hands-on experience with Azure Policy, RBAC, and resource governance.
* Knowledge of Azure Security practices, tools, and compliance standards (e.g., SOC 2, ISO, etc.).
* Familiarity with Azure Cost Management, tagging strategies, and budget controls.
* Extensive experience managing Databricks as a platform, including Unity Catalog, identity federation, cluster policies, workspace management, and automation.
* Strong scripting skills (e.g., Python, PowerShell, Bash) and comfort working with APIs and automation tools.
* Excellent problem-solving and communication skills; ability to work cross-functionally with security, data, and DevOps teams.
Nice to Have
* Experience with multi-cloud or hybrid cloud architectures.
* Knowledge of Infrastructure as Code testing frameworks (e.g., Terratest, Checkov).
* Familiarity with Azure DevOps, ARM templates, or Bicep.
* Experience supporting data science and analytics teams in a regulated environment.
To apply for this role please submit your CV or contact Dillon Blackburn on (phone number removed) or at (url removed).
We are the go-to recruiter for Data & AI roles in the UK offering more opportunities across the country than any other recruitment agency. We are the proud sponsor and supporter of SQLBits, Power Platform World Tour, and the London Fabric User Group. We are the global leaders in Data & AI recruitment.
#J-18808-Ljbffr