Senior Data Engineer (Databricks, Spark, Python)
Remote (UK based)
3 - 6 months rolling
£550 - £650 per day
Outside IR35
A Senior Data Engineer with broad knowledge of Databricks is needed to join a leading consultancy renowned for its innovative approach and expertise in delivering state-of-the-art data solutions.
As a proud Databricks Partner, our client is at the forefront of driving transformation and excellence in data strategies for businesses across various sectors. This contract role is an exceptional opportunity for a Senior data engineering professional looking to make a substantial impact while enjoying the flexibility of a fully remote, outside IR35 contract.
As the Senior Data Engineer, you will play a pivotal role in designing and implementing cutting-edge data solutions utilising Databricks technologies.
Key Responsibilities:
* Architect and execute efficient data solutions leveraging Azure Data Factory, Synapse, Databricks, and Azure DevOps.
* Develop, maintain, and optimise data models and ETL pipelines for premier data processing efficiency.
* Design and implement processes to ingest data from various sources into Databricks.
* Use Databricks to integrate, consolidate, and cleanse data to ensure it is accurate and ready for analysis.
* Develop, test, and maintain robust, scalable data pipelines using Databricks.
* Utilise Databricks' support for languages like Python, Scala, SQL, and R to process and transform data.
* Design and implement data models that effectively represent business processes and requirements.
* Optimise data storage in Databricks for efficient querying and analysis, leveraging Delta Lake for ACID transactions and versioned data management.
* Translate complex business requirements into technical specifications in collaboration with various teams.
* Implement and maintain data governance policies and practices, ensuring data quality and consistency.
* Manage data access and security within Databricks, aligning with organizational policies and compliance requirements.
* Engage with clients to support architectural designs and solve technical queries.
* Remain abreast of the latest developments and Databricks technologies, ensuring best practices in data governance and security compliance.
Essential Skills and Qualifications:
* A minimum 5 years' experience in data engineering
* Minimum 2-3 projects delivered with hands-on experience in Databricks
* Expertise in data modeling, ETL development, and data warehousing.
* Familiarity with Databricks medallion architecture
* Experience working with Delta Live Tables
* Familiarity with Unity Catalog
* Strong programming experience using Python
* Experience with distributed computing with Spark
* Streaming experience using technologies such as Kafka
* Strong problem-solving abilities, capable of adapting quickly in a dynamic environment.
* Excellent communication skills, with experience in mentoring and collaborative teamwork.
* Prior experience in client-facing roles is highly advantageous.
* Proficiency in Agile and DevOps methodologies.
* Databricks Data Engineering Professional certification is highly desirable
Contract Highlights:
* Competitive day rate of £550 - £650.
* Fully remote work, offering unparalleled flexibility.
* A 6-month contract outside IR35, providing freedom and independence in your professional engagements.
* An opportunity to engage with innovative projects and leadership roles within a high-growth, value-driven consultancy.
Application Process:
To apply for this opportunity, please click apply or send your CV to CV@primus-connect.com
Primus Connect champions diversity and inclusion, welcoming applications from all qualified individuals. Join us for an enriching contract experience that promises professional growth and a collaborative working environment.