Lead Data Engineer:
We need some strong Data engineer profiles… they need good experience with
Pyspark, Python, SQL, ADF and preferably Databricks experience
Job description:
* Building new data pipelines and optimizing data flows using the Azure cloud stack.
* Building data products from scratch.
* Support Business Analysts and Data Architects with discovery and best practices.
* Self-directed and comfortable supporting the data needs of multiple teams, systems and products.
What’s required ?
* Advanced Azure knowledge and experience working and migrating data products from on-prem to Azure.
* Experience building and optimizing ‘big data' data pipelines, architectures and data sets using Py-spark.
* Experience building real-time data pipelines using Event-hub, storage queues and Azure stream analysis.
* Strong analytic skills related to working with unstructured datasets.
* Build processes supporting data transformation, data structures, metadata, dependency and workload management.
* Cloud Big Data Analytics in Azure Synapse Analytics, Azure Analysis Services
* Data Ingestion and Storage including Azure Data Factory, Azure Databricks, Azure Data Lake, Kafka and Spark Streaming, Azure EventHub/IoT Hub, and Azure Stream Analytics
Experience with Object-oriented/ object function scripting languages: Python preferred