Job Title: Senior Technical Lead Location: Norwich, Norfolk (3 Days a week) Job Type: Contract (Inside IR35) Duration: 6 Months The Role We are looking for a Senior Technical Lead who combines hands-on engineering excellence with strong leadership and stakeholder management. You will own the end-to-end technical delivery of data platforms and pipelines built in AWS-with a focus on AWS Glue, Managed Workflows for Apache Airflow (MWAA), and Python-and collaborate closely with Directors, Senior Architects, and Program Leadership to deliver business outcomes at scale. This is a player-coach role: you will design, build, review, and optimize complex data workflows while mentoring engineers and driving engineering best practices. Ideal for: Someone who has delivered multiple production programs in a modern AWS data engineering landscape, can communicate trade-offs clearly to senior stakeholders, and can lead teams through ambiguity to predictable, high-quality outcomes. Your Responsibilities: Lead the design and implementation of scalable, secure, and cost-efficient ETL/ELT pipelines using AWS Glue, Python (PySpark), and MWAA (Airflow). Define solution architectures, data models, orchestration patterns, and CI/CD for data workflows. Own the technical roadmap, decomposition, and delivery plan-including sizing, sprint planning, and risk mitigation. Drive performance optimization (e.g., partitioning strategies, Glue job tuning, job bookmarks, dynamic frames vs DataFrames, retry/backoff strategies in Airflow). Ensure robust observability (logging, metrics, tracing) and data quality (unit tests, Great Expectations/Deequ-style checks, validations). Act as the technical point of contact for Senior Architects, and Program Managers; translate business needs into technical designs and delivery milestones. Present architecture decisions, trade-offs, and TCO to senior stakeholders with clarity, data, and rationale. Manage vendor/partner coordination where relevant. Establish coding standards, code review practices, branching strategies, and secure-by-design principles. Implement DevSecOps for data: infrastructure-as-code (IaC), secrets management, environment promotion, and automated testing. Ensure compliance with data governance, security, and regulatory requirements (e.g., PII/PCI, encryption, auditability, lineage). Mentor and upskill engineers; foster a culture of learning, ownership, and continuous improvement. Your Profile Essential skills/knowledge/experience: 10 years of total experience in software/data engineering, with 5 years leading delivery of production solutions in an AWS data engineering environment. Advanced hands-on expertise with: Python (including PySpark & data engineering patterns) AWS Glue (Jobs, Crawlers, Glue Studio, Glue Catalog, PySpark, Job bookmarks) MWAA (Apache Airflow) (DAG design, scheduling, sensors, retries, XComs, task isolation, best practices) Strong across broader AWS services: S3, Lambda, Step Functions, IAM, CloudWatch, KMS, Secrets Manager, Athena, EMR (nice to have), Redshift (nice to have) Proven experience delivering multiple end-to-end programs (architecture build test deploy operate) with measurable outcomes (SLAs, cost targets, performance). Excellent stakeholder communication and executive presence; able to engage Directors, Senior Architects, and Program Leadership. Solid grounding in data modeling, data governance, security/compliance, and cost optimization on AWS. Experience with CI/CD (e.g., CodePipeline/GitHub Actions/Bitbucket Pipelines), IaC (CloudFormation/Terraform), and containerization (Docker). Architectural thinking: designs for scale, reliability, cost, and evolvability. Delivery excellence: breaks down complex work, sets milestones, manages risks, and delivers on time. Communication & influence: distills complexity for senior stakeholders; backs decisions with data. Hands-on leadership: sets the technical bar through reviews, pairing, and exemplars. Ownership & clarity: aligns teams on problem statements, success criteria, and measurable outcomes. Languages: Python (PySpark), SQL AWS: Glue, MWAA (Airflow), S3, IAM, KMS, CloudWatch, Lambda, Step Functions, Athena, Redshift (nice), EMR (nice) DevOps: Git, CI/CD (CodePipeline/GitHub Actions), Terraform/CloudFormation, Docker Data Quality/Observability: Great Expectations/Deequ (nice), OpenLineage (nice) Desirable skills/knowledge/experience: Domain experience in BFSI (risk, pricing, regulatory reporting, underwriting, fraud, payments, or actuarial data). Experience with event-driven and near-real-time pipelines (Kafka/Kinesis, streaming ETL). Knowledge of data quality frameworks (Great Expectations, Deequ) and data lineage/catalog (Atlas, Alation, Collibra). Exposure to Databricks or EMR for advanced Spark workloads. Certifications: AWS Solutions Architect / Data Analytics / DevOps Engineer. Prior experience leading multi-team programs with offshore/nearshore models.