Role Overview
This role is for a Databricks Engineer on a "contract" basis, offering a pay rate of "$XX/hour". The position requires expertise in Databricks, Apache Spark, and cloud platforms (preferably Azure), with a focus on building scalable data pipelines. Location: Luton, England, United Kingdom.
Databricks Engineer with strong experience in building scalable data pipelines and processing large datasets using Apache Spark. Skilled in designing efficient ETL workflows, optimising performance, and working within cloud-based data platforms. Experienced in delivering reliable, high-quality data solutions that support analytics and business decision-making.
Responsibilities
* Develop and maintain scalable data pipelines using Databricks and Apache Spark
* Design and implement ETL/ELT processes for structured and unstructured data
* Work with Delta Lake to ensure data reliability, versioning, and performance
* Optimise Spark jobs for speed, cost efficiency, and resource utilization
* Ingest data from multiple sources including APIs, databases, and streaming platforms
* Collaborate with data analysts, data scientists, and stakeholders to deliver data solutions
* Implement data validation, monitoring, and logging frameworks
* Manage and orchestrate workflows using tools like Databricks Workflows or Azure Data Factory
* Ensure data governance, security, and compliance best practices
Technical Skills
* Databricks (Core platform expertise)
* Apache Spark (PySpark / SQL / Scala)
* Programming: Python, SQL
* Cloud Platforms: Azure (preferred), AWS, or GCP
* Data Storage: Delta Lake, ADLS / S3 / GCS
* Orchestration: Azure Data Factory, Airflow, or Databricks Workflows
Additional Skills
* Experience with data modeling (star schema, dimensional modeling)
* Understanding of batch and streaming data processing
* Familiarity with CI/CD pipelines and version control (Git)
* Knowledge of performance tuning and query optimization
* Exposure to BI tools like Power BI or Tableau
Qualifications
* Bachelor’s degree in Computer Science, Engineering, or related field
* Relevant experience in data engineering or big data technologies
* Databricks or cloud certifications (preferred but not mandatory)
Key Strengths
* Strong problem-solving and analytical thinking
* Ability to work in fast-paced, client-focused environments
* Clear communication and collaboration skills
* Focus on building scalable and maintainable data solutions
#J-18808-Ljbffr