Responsibilities
* Design, develop, and maintain metadata-driven data pipelines using Azure Data Factory and Databricks.
* Build and implement end-to-end metadata frameworks, ensuring scalability and reusability.
* Optimize data workflows leveraging SparkSQL and Pandas for large-scale data processing.
* Collaborate with cross‑functional teams to integrate data solutions into enterprise architecture.
* Implement CI/CD pipelines for automated deployment and testing of data solutions.
* Ensure data quality, governance, and compliance with organizational standards.
* Provide technical leadership and take complete ownership of assigned projects.
Technical Skills Required
* Azure Data Factory (ADF): Expertise in building and orchestrating data pipelines.
* Databricks: Hands‑on experience with notebooks, clusters, and job scheduling.
* Pandas: Advanced data manipulation and transformation skills.
* SparkSQL: Strong knowledge of distributed data processing and query optimization.
* CI/CD: Experience with Azure DevOps, Git, or similar for automated deployments.
* Metadata-driven architecture: Proven experience designing and implementing metadata frameworks.
* Programming: Proficiency in Python and/or Scala for data engineering tasks.
Seniority level
* Associate
Employment type
* Contract
Job function
* Information Technology
* IT Services and IT Consulting
Referrals increase your chances of interviewing at Response Informatics by 2x.
Location: Salford, England, United Kingdom.
#J-18808-Ljbffr