Big Data Engineer (Large-Scale Text Data) Worcestershire (Hybrid – 1 day per week) £80,000 - £85,000 Bonus Excellent Pension SC Clearance Required (DV eligible) The Opportunity We are supporting a specialist technology consultancy delivering advanced data and AI solutions within highly secure government environments. This is a genuinely unique engineering role, focused on solving complex challenges at scale by working with extremely large, text-heavy datasets to support real-world, high-impact outcomes. If you enjoy tackling complex data problems in distributed systems rather than standard reporting or dashboarding, this role will be a strong fit. The Role Work with multi-terabyte to large-scale datasets (100TB), primarily text-based Build and optimise data ingestion and processing pipelines Improve batch processing performance and throughput across large datasets Solve challenges related to scalability, reliability, and data flow Contribute to the development of distributed data platforms Collaborate within a highly technical, research-driven environment What We're Looking For Experience working with large-scale datasets (multi-terabyte minimum) Strong background in data engineering within distributed systems Proven ability to build and optimise data pipelines at scale Solid Python skills with exposure to tools such as Spark, PySpark, and Hadoop Strong understanding of data ingestion and batch processing methodologies Experience with text or unstructured data is advantageous but not essential A hands-on, problem-solving mindset with a focus on engineering quality Environment & Culture Calm, focused, and highly technical – not a high-pressure commercial setting Strong emphasis on autonomy, trust, and long-term thinking Work alongside intelligent, down-to-earth engineers solving complex problems Clear opportunities for progression and technical growth Package £80,000 - £85,000 base salary Approximately 10% bonus Approximately 10% pension contribution Hybrid working – one day per week on-site Strong benefits package and long-term career development Security Requirements Due to the nature of the work, candidates must hold active SC clearance and be willing to undergo DV clearance.