Senior Data Engineer
Bristol
6-month contract
Paying up to £80p/h (Outside IR35)
*** Please note that due to the nature of the work you must hold or be eligible to obtain DV level security clearance ***
Responsibilities:
* Design, develop, and maintain secure and scalable data pipelines using the Elastic Stack (Elasticsearch, Logstash, Kibana) and Apache NiFi.
* Implement data ingestion, transformation, and integration processes, ensuring data quality and security.
* Collaborate with data architects and security teams to ensure compliance with security policies and data governance standards.
* Manage and monitor large-scale data flows in real-time, ensuring system performance, reliability, and data integrity.
* Develop robust data models to support analytics and reporting within secure environments.
* Perform troubleshooting, debugging, and performance tuning of data pipelines and the Elastic Stack.
* Build dashboards and visualizations in Kibana to enable data-driven decision-making.
* Ensure high availability and disaster recovery for data systems, implementing appropriate backup and replication strategies.
* Document data architecture, workflows, and security protocols to ensure smooth operational handover and audit readiness.
Experience required:
* Experience working as a Data Engineer in secure or regulated environments.
* Expertise in the Elastic Stack (Elasticsearch, Logstash, Kibana) for data ingestion, transformation, indexing, and visualization.
* Strong experience with Apache NiFi for building and managing complex data flows and integration processes.
* Familiarity with containerization and orchestration tools such as Docker and Kubernetes.
* Experience with monitoring and alerting tools such as Prometheus, Grafana, or ELK for data infrastructure
* Knowledge of security practices for handling sensitive data, including encryption, anonymization, and access control.
* Familiarity with data governance, data quality management, and compliance standards in secure environments.
* Experience in managing large-scale, real-time data pipelines and ensuring their performance and reliability.