Position: Data Engineer
Overview of the Role
An established organisation is currently undergoing a data transformation and is seeking a skilled Data Engineer to join its growing data team. This role plays a key part in building and deploying modern data solutions based on Azure Databricks, enabling faster and more informed business decisions.
You'll work hands-on with Azure Databricks, Azure Data Factory, Delta Lake, and Power BI to design scalable data pipelines, implement efficient data models, and ensure high-quality data delivery. This is a great opportunity to shape the future of data within the organisation while working with advanced cloud technologies.
Key Responsibilities and Deliverables
* Design, develop, and optimise end-to-end data pipelines (batch & streaming) using Azure Databricks, Spark, and Delta Lake.
* Implement Medallion Architecture to structure raw, enriched, and curated data layers efficiently.
* Build scalable ETL/ELT processes with Azure Data Factory and PySpark.
* Support data governance initiatives using tools like Azure Purview and Unity Catalog for metadata management, lineage, and access control.
* Ensure consistency, accuracy, and reliability across data pipelines.
* Collaborate with analysts to validate and refine datasets for reporting.
* Apply DevOps and CI/CD best practices (Git, Azure DevOps) for automated testing and deployment.
* Optimise Spark jobs, Delta Lake tables, and SQL queries for performance and cost-effectiveness.
* Troubleshoot and proactively resolve data pipeline issues.
* Partner with data architects, analysts, and business teams to deliver end-to-end data solutions.
* Stay current with emerging data technologies (e.g., Kafka/Event Hubs for streaming, Knowledge Graphs).
* Promote best practices in data engineering across the team.
Skills & Experience
* Hands-on experience with Azure Databricks, Delta Lake, Data Factory, and Synapse.
* Strong understanding of Lakehouse architecture and medallion design patterns.
* Proficient in Python, PySpark, and SQL, with advanced query optimisation skills.
* Proven experience building scalable ETL pipelines and managing data transformations.
* Familiarity with data quality frameworks and monitoring tools.
* Experience working with Git, CI/CD pipelines, and in Agile environments.
* Ability to write clean, maintainable, and well-documented code.
* Exposure to Power BI or similar data visualisation tools.
* Knowledge of IoT data pipelines is a plus.
#J-18808-Ljbffr