Responsibilities
* Design, build, and optimize data pipelines and transformation workflows on Databricks, leveraging Python and Spark.
* Collaborate with Data Architects and Business Analysts to develop robust data models and clearly document data flows and ETL logic.
* Implement and execute data quality checks and validation modules using Python.
* Maintain transparency and accountability by tracking work and progress in Jira.
* Ensure datasets and pipelines are accurately registered in relevant catalogues and consoles, meeting governance and privacy standards.
* Proven experience developing data pipelines and solutions on Databricks.
* Strong proficiency in Python, including libraries for data transformation (e.g., pandas).
* Solid understanding of ETL concepts, data modelling, and pipeline design.
* Experience with Spark and cloud data platforms.
* Ability to document data flows and transformation logic to a high standard.
* Familiarity with project management tools such as Jira.
* Collaborative mindset and strong communication skills.
Qualifications
* Proven experience developing data pipelines and solutions on Databricks.
* Strong proficiency in Python, including libraries for data transformation (e.g., pandas).
* Solid understanding of ETL concepts, data modelling, and pipeline design.
* Experience with Spark and cloud data platforms.
* Ability to document data flows and transformation logic to a high standard.
* Familiarity with project management tools such as Jira.
* Collaborative mindset and strong communication skills.
Preferred qualifications
* Experience in financial services or large enterprise data environments.
* Knowledge of data governance, privacy, and compliance requirements.
* Exposure to business analysis and requirements gathering.
#J-18808-Ljbffr