Insight Global is seeking a Azure Data Engineers to join a prestigious energy client based in London. The successful candidates will be responsible for designing, implementing, and managing live data streaming pipelines for the client’s Energy Trading team. This role involves designing data pipelines for Crude and Products trading data from on-premise DB2 databases to an Azure Cloud environment.
This is an Inside IR35 contract role and the individual must sit out of the UK.
Key Responsibilities:
* Pipeline Management: Design, implement, and manage live data streaming pipelines using Azure Databricks to ensure seamless data flow and real-time processing.
* Process Evaluation: Assess and optimize on-premise to cloud data exchange processes for accuracy, efficiency, and scalability.
* Code Review and Debugging: Conduct thorough code reviews and debugging sessions, providing guidance and mentorship to junior data engineers to ensure high-quality code and best practices.
* Problem Solving: Develop innovative solutions to address computing and cost challenges, leveraging advanced technologies and methodologies.
Must Haves:
* Data Engineering experience
* Deep expertise with Azure Databricks (Delta Live Tables, Data Streaming, Unity Catalogue etc.)
* Prove experience designing high volume, live data streaming solutions using Azure DLT (Delta Live Tables)
* Expert with Apache Spark and PySpark (ability to review quality of code and debug issues)
* Background in Data warehousing (SAP Hana, BI/BW, Oracle etc.)
* Proficient with SQL
* Proven ability to problem solve and think outside the box
Plusses:
* Background in Trading domain, specifically energy commodities trading