We are looking for an experienced Senior Data Engineer to join one of our client's team.
Role and Responsibilities:
* Design, develop, and maintain scalable data pipelines using Python, PySpark and Databricks.
* Implement data orchestration workflows using Airflow or similar tools.
* Manage and optimize data storage solutions on Azure Cloud.
* Develop and maintain containerized microservices using Docker.
* Collaborate with cross-functional teams to ensure data quality and integrity.
* Support the setup of blockchains.
* Build tools to acquire data from blockchains and perform analysis on the data using Databricks.
* Support production infrastructure, including monitoring and automation using tools like Datadog.
* Participate in agile development practices, contributing to planning exercises and working in cross-functional groups.
Skills and Qualifications:
* Bachelor's or Master's degree in Computer Science, Data Science, or a related field.
* Advanced Python knowledge for data processing and scripting.
* Proven experience with Spark and SQL for data engineering and analysis.
* Experience with data orchestration tools like Airflow.
* Hands-on experience with one or more cloud services (Azure, AWS, GCP).
* Proficiency in RDBMS/NoSQL data stores and appropriate use cases.
* Experience with Data as Code; version control, small and regular commits, unit tests, CI/CD, packaging, familiarity with containerization tools such as Docker (must have) and Kubernetes (plus).
* Solid understanding of the software development life cycle.
* Understanding of blockchain technologies and data structures.
* Knowledge of cryptography and its application in blockchain is a plus.
* Experience with blockchain indexing is a plus.
* Experience with Databricks for data ingestion and transformation is a plus.
* Familiarity with Delta Lake and data warehousing concepts is a plus.
* Strong communication, interpersonal and presentation skills.
If interested, please get in touch via contact details provided or click “Apply” to forward an up-to-date copy of your CV.
Cavendish (Recruitment) Professionals Ltd are proud to be an equal opportunity employer and we believe that inclusivity begins with the candidate experience.All qualified applicants will receive consideration for employment regardless of, gender, race, age, sexual orientation, religion, or belief.
#J-18808-Ljbffr