Job Description
Are you a seasoned Data Engineer looking to make a significant impact in the renewable energy sector? Look no further! We're seeking a Senior Data Engineer to spearhead our migration to Databricks, one of the most exciting projects in the UK energy market.
Job Title: Senior Data Engineer (Migration to Databricks)
Duration: Initially 3 months
Location: Predominantly remote with occasional onsite travel to nearest site
Contract Type: Outside of IR35
Immediate Start
Job Overview:
As a Senior Data Engineer, you'll play a pivotal role in our data transformation journey. You'll implement scalable data pipelines within Databricks, ensuring adherence to best practices and data governance principles.
Your responsibilities will include:
* Building and maintaining efficient data pipelines within Databricks, evolving them as requirements evolve.
* Developing and maintaining an enterprise data model within Databricks.
* Integrating new data sources through batch and streaming processes while meeting SLAs.
* Documenting data pipelines and systems, prioritizing security and monitoring.
* Ensuring data quality and reliability through effective processes.
Requirements:
To thrive in this role, you'll need:
* Strong Python skills with at least 5 years of experience, including advanced concepts.
* Proficiency in SQL, database design, and data architecture.
* Experience with Databricks and/or Spark.
* Familiarity with data governance, cataloguing, and quality principles.
* Expertise in data extraction, joining, and aggregation tasks, especially with big and real-time data.
* Ability to perform data cleansing and transformation operations.
* Understanding of data storage concepts and logical data structures.
* Experience writing production-quality code for data pipelines.
* Previous data migration experience is advantageous.
* Openness to new technologies and ways of working.
* Self-motivation and the ability to troubleshoot and devise solutions independently.
If you are looking for a fast moving role, using your expertise in Data Modelling, Python and Databricks, please apply for your CV to reach me directly.