Job Description
Key Responsibilities
My client are expanding their data and technology capabilities and seeking a skilled Data Engineer to join this growing team. The successful candidate will help ingest, cleanse, and model data into unified, analytics-ready datasets supporting Business Intelligence, machine learning, and Data Science initiatives.
Core Duties
* Design and develop cohesive data repositories to support data-driven decision-making.
* Ingest data from a variety of sources, including on-premise SQL databases, REST APIs, and Apache Kafka streams.
* Apply data cleansing rules to ensure data meets quality and consistency standards.
* Model data appropriately for usage scenarios, including designing single-source-of-truth Kimball-style datasets (dimensions and facts).
* Follow DevOps best practices for software development: write clean, testable code with proper linting, unit/integration testing, CI/CD pipelines, and peer reviews.
* Monitor and support data pipelines to ensure timely and accurate data delivery, including real-time incident resolution as needed.
* Actively contribute to team processes such as backlog grooming, sprint planning, demonstrations, and retrospectives.
* Translate business requirements into technical specifications, estimate complexity, and deliver within sprint cycles.
* Conduct ad hoc analyses of structured and unstructured data to guide solution design.
* Maintain comprehensive documentation in the data catalog, including ownership, stewardship, data dictionaries, glossaries, lineage, and data sensitivity.
* Take ownership of assigned work items, collaborating with data owners and stewards to ensure high-quality, compliant deliverables (e.g., GDPR compliance, PII handling, data retention).
* Document solution designs in internal wikis.
* Provide support and maintenance across all data platforms, ensuring their smooth and reliable operation.
Knowledge and Experience
Must Have:
* Minimum 3 years of experience as a Data Engineer or in a related data-focused role
* Strong hands-on experience with Azure data services, including:
* Azure Data Factory V2
* Azure Data Lake Storage V2
* Azure Databricks
* Azure Function Apps & Logic Apps
* Azure Stream Analytics
* Azure Resource Manager tools: Terraform, Azure Portal, Azure CLI, and Azure PowerShell
* Proficient in PySpark, Delta Lake, Unity Catalog, and Python
* Ability to write unit and integration tests using unittest, pytest, etc.
* Solid understanding of software engineering principles, including SOLID design, dependency injection, code structuring, and testing
* Experience with version control systems and CI/CD pipelines
* In-depth knowledge of Kimball data modeling techniques, such as star and snowflake schemas
* Strong SQL skills and ability to write performant queries
* Proficient in data analysis and interpretation
* Excellent verbal and written communication skills
* Proven track record of delivering high-quality work under pressure, with strong attention to detail and time management
* A genuine passion for leveraging technology to create business value
Nice to Have:
* Experience with Azure DevOps (including Git and multi-stage YAML pipelines)
* Additional programming languages such as C# or PowerShell
* Infrastructure as Code (IaC) experience, e.g., Terraform, ARM templates, Bicep
* Familiarity with test-driven development (TDD) methodologies
* Experience with streaming technologies, e.g., Azure Stream Analytics or Spark Structured Streaming
* Power BI engineering or integration experience
* Certified Scrum Developer (CSD) or similar Agile certification
* Experience with data governance tools, such as Microsoft Purview
* Exposure to machine learning and artificial intelligence use cases
Personal Attributes
* Self-motivated and able to work independently without close supervision
* Organised, methodical, and structured in approach
* Comfortable navigating fast-paced, dynamic environments with evolving priorities
* Passionate about both technology and process improvement, with a focus on delivering value
* Clear and credible communicator, capable of engaging effectively with stakeholders at all levels
* Strong attention to detail with a commitment to high quality and accuracy
* Positive and collaborative team player who uplifts those around them
* Fast learner with the ability to adapt to new technologies and industry trends
* Excellent time management and organisational skills