Overview
We are seeking a Senior Data Engineer with extensive infrastructure implementation experience, especially with Snowflake and connected applications to lead technical delivery of our Snowflake migration project. To help us migrate from an in-house developed data warehouse to Snowflake as our core data infrastructure, supporting multiple client and 1P integrations as well as our own reporting visualisation solution, Seer.
You will have current advanced Snowflake qualifications, a track record of successful implementations and migrations and experience with MongoDB, BigQuery, AWS and GCP, ETL and data warehouse technologies. Experience working with advertising and marketing platform data would be a considerable advantage.
In this role, you will leverage your knowledge of Snowflake and its application in a marketing environment to interpret, advise on and plan infrastructure implementation requirements. Provide guidance and recommendations, and develop a comprehensive, documented implementation plan that you will then work with the team to deliver.
You will work with our Senior Director - Infrastructure and Delivery, the senior Engineer who developed much of the existing infrastructure and our team of data and software engineers, to drive the migration and ensure the team is left with a well-documented, maintainable solution that they can support and manage on an ongoing basis.
This is a focused project role. This role is delivery-focused and time-bound. Working alongside our Senior Director - Infrastructure and Delivery. The long-term platform strategy, operating model, and product roadmap will remain with the Senior Director of Infrastructure & Delivery.
Key Responsibilities
1. Architecture & Migration (MongoDB/AWS/BigQuery to Snowflake)
o Design & Strategy: Design and implement a Snowflake environment in line with an agreed migration strategy. Plan how to map nested JSON documents into Snowflake (using VARIANT vs. normalized tables) to balance query performance with schema flexibility, documenting trade-offs and recommendations for internal review.
o Solution presentation and collaboration during the planning stages, regularly present architectural decisions and trade-offs to key stakeholders, allowing for input and shared decision-making on key points.
o Infrastructure as Code: specific setup of Snowflake warehouses, roles, and security policies based on defined security and governance standards using Terraform or similar tools.
o Data Migration: Execute the migration of historical marketing data (TB scale) from AWS/Mongo to Snowflake, ensuring high data parity to run platforms concurrently, with validation agreed with stakeholders..
2. Application Integration (The "Seer" Connection)
o Backend Engineering: Build the "connective tissue" between the Seer frontend and Snowflake. This may involve:
o Adapt and extend the existing Seer Data API (FastAPI) to support Snowflake as a backend, alongside or in place of MongoDB where required (with support from the APIs developer). This includes updating query patterns, performance optimisation, and ensuring backward compatibility during migration.
o Optimizing Snowflake query performance (Clustering, Materialized Views) to ensure fast (cached) response times for interactive user dashboards.
o Security Bridge: In collaboration with the platform owner, implement a security model that maps Seer’s existing user management system to Snowflake’s Row-Level Security (RLS) or Dynamic Data Masking policies, ensuring clients only see their own data.
o Pipeline Development: Review and migrate ELT pipelines (using dbt, Airflow, or where needed Fivetran) to ingest data from 3rd-party marketing APIs (LinkedIn, Google Ads, CRM) into the new Snowflake backend.
o Assess and implement Snowflake Openflow or direct data sharing options where available, documenting impacts on existing Fivran and custom connector usage.
o Best Practices: Contribute to and implement agreed development standards for CI/CD, version control (Git).
o Orchestration will initially remain within the existing Seer backend unless otherwise agreed. Evaluation of alternative orchestrators (e.g. Dagster) may be documented but is not a primary delivery objective.
3. Integration consultancy, guidance and establishing best practice:
o Design, develop, and maintain scalable data pipelines and ETL processes to ensure timely and accurate data availability.
o Implement data quality checks, validation, and monitoring frameworks, with handover to the internal team.
o Develop and optimise data storage to support analytics and reporting needs.
o Drive automation of the most common recurring data tasks and reporting processes to improve team productivity and minimise manual intervention.
o Implement security and lifecycle controls in line with existing compliance and governance frameworks
o Support the team and train them on key implementation specifics points to ensure knowledge transfer, allowing them to support Snowflake and related infrastructure with confidence.
4. Project plan overview (subject to confirmation)
o Months 1-2: Discovery & Foundation. Auditing data, setting up Snowflake environments, building connection structures and workflow, including Snowflake native ETL utilisation (vs custom, Fivetran etc). Establishing naming conventions.
o Months 3-4: Migration & Modelling. Moving historical data, writing dbt models to clean and optimise data.
o Month 5: BI Integration. Connecting front-end tools Seer (and dashboards Tableau/PowerBI), testing data accuracy with stakeholders / validating with frontend users and optimising.
o Month 6: Handoff & Documentation. Training internal teams, documenting the code, monitoring and addressing any performance issues.
Qualifications
* Snowflake qualifications in relevant “Advanced” courses
* Extensive experience in a data engineering or analytics engineering role, with exposure to marketing analytics strongly preferred.
* Bachelor’s or Master’s degree in Computer Science, Data Engineering, Statistics, or
* Strong programming skills, particularly in Python, SQL, and familiarity with other languages like Java or Scala.
* Proficiency in ETL tools and frameworks (e.g., Apache Airflow, dbt) and data integration techniques.
* Experience with data visualisation tools (e.g., Tableau, Power BI) and an understanding of how to enable analysts through clean and usable data.
* Familiarity with cloud-based platforms (e.g., AWS, Google Cloud, Azure) and data warehouse technologies (e.g., Snowflake, Redshift, BigQuery).
* Solid understanding of database design principles and query optimisation techniques.
* Knowledge of version control systems (e.g., Git) and experience with CI/CD pipelines.
* Excellent problem-solving capabilities, with a focus on building scalable, reliable solutions.
* Strong communication skills to effectively collaborate with technical and non-technical stakeholders.
Budget and application
This project will have a fixed budget and timeline, agreed with the consultant. We would welcome a cost estimate based on the details and assumptions set out in the document. For the applicants that we shortlist, there will be an opportunity to discuss the plan in more detail and revise the plan, time-scales and budget.
#J-18808-Ljbffr