Job Overview
We are seeking a skilled and motivated Data Engineer to join our dynamic team. In this role, you will be responsible for designing, building, and maintaining robust data pipelines and architectures that support our data-driven decision-making processes. You will work closely with data scientists, analysts, and other stakeholders to ensure that our data infrastructure is efficient, scalable, and aligned with business needs.
Responsibilities
* Design and implement robust data pipelines using Azure Data Factory (ADF) and Databricks.
* Develop scalable ETL/ELT processes to ingest, transform, and load data from various sources.
* Write clean, efficient, and reusable Python code for data processing and automation.
* Optimize and manage SQL queries for data extraction, transformation, and reporting.
* Collaborate with business stakeholders to understand data requirements and deliver solutions aligned with business goals.
* Ensure data quality, integrity, and security across all data platforms.
* Monitor and troubleshoot data pipelines to ensure high availability and performance.
* Implement logging, alerting, and error-handling mechanisms for proactive issue resolution
* Apply domain knowledge of trading and financial markets to enhance data models and analytics capabilities.
Requirements
* Proficiency in programming languages such as Java, Python, and SQL.
* 5+ years of experience in data engineering or related roles.
* Proven expertise in Azure Data Factory and Azure Databricks.
* Strong proficiency in Python for data engineering tasks.
* Solid understanding of SQL and relational databases.
* Exposure to real-time data processing frameworks (e.g., Spark Streaming, Kafka).
* Experience in maintaining and supporting production-grade data pipelines.
* Experience with version control systems (e.g., Git) and CI/CD pipelines.
* Excellent problem-solving skills and attention to detail.
* Strong communication and collaboration abilities
* Excellent problem-solving skills coupled with strong communication abilities. If you are passionate about leveraging technology to drive business insights through effective data management, we encourage you to apply for this exciting opportunity as a Data Engineer.
Job Types: Full-time, Permanent, Graduate
Pay: £50,000.00-£60,000.00 per year
Benefits:
* Employee mentoring programme
* Flexitime
* Health & wellbeing programme
* UK visa sponsorship
* Work from home
Work Location: Hybrid remote in Peterborough PE1 5PG