The Role:
1. Designing, building, and optimizing data pipelines, data transformations, data storage functions, for data consumption using Azure Synapse, Azure Data Factory, and Azure Fabric.
2. Writing and fine-tuning PySpark notebooks to handle massive data workloads efficiently.
3. Troubleshooting and enhancing ETL/ELT workflows in Azure Synapse.
4. Managing and organizing Data Lakes to ensure seamless data access and performance.
5. Integrating AI/LLM models into data pipelines to drive innovation and insights.
6. Collaborating with Data Scientists, AI Engineers, and Analysts to create powerful data-driven solutions.
7. Ensuring data security, governance, and compliance within our Azure ecosystem.
8. Staying ahead of the curve with emerging cloud, AI, and big data technologies.