Job Role
We are seeking a Databricks DevOps Engineer to design and operate secure, scalable and reliable Databricks-based data platforms on AWS. This role focuses on infrastructure automation, CI/CD implementation, platform governance and observability. You will help build the engineering foundations that enable teams to deliver reliable data products.
Key Responsibilities
- Design and implement ELT/ETL pipelines using PySpark and Databricks
- Build scalable batch and streaming data pipelines using Spark and Kafka
- Develop optimised SQL and Python pipelines for data transformation and integration
- Integrate external data sources using REST APIs and data ingestion frameworks
- Optimise Spark jobs and cluster performance for reliability and cost efficiency
- Implement data quality checks, validation and monitoring
- Apply CI/CD practices and version control for pipeline deployment
- Work closely with solution architects, DevOps engineers and business analysts to deliver data products
- Produce technical documentation, architecture diagrams and operational runbooks
Key Requirements
- Eligible and willing to pass relevant background checks
- 3+ years experience building production data pipelines
- Hands‑on experience with Databricks, Spark and PySpark
- Experience with Delta Lake, Unity Catalog and Databricks Workflows
- Experience with data pipeline design, testing and deployment
- Familiarity with test‑driven development, CI/CD practices and Git-based development
- Strong collaboration and communication skills
- Working knowledge of Hadoop
- REST APIs and Kafka experience
Nice to Have
- Cloud experience with AWS, Azure or GCP
- Certifications in Databricks, AWS or data engineering
- Infrastructure as Code (Terraform)
- Orchestration: Airflow or AWS Glue
- Streaming technologies including Structured Streaming
- Observability tooling for data pipelines
- Experience working with regulated industries such as banking or financial services
