The ONS operates a flexible hybrid working model across the UK, with colleagues linked to one of our contractual locations working between office and remote throughout the week. The locations for this role are Newport, Titchfield (Fareham), Manchester, Edinburgh, London.
All colleagues are required to work from their contractually allocated site for at least 40% of their working time. Due to current capacity constraints there is currently an exception for colleagues based at the Darlington Economic Campus, and the Manchester office with office attendance being 20%, this is expected to move to 40% attendance in 2025-26 and 2027-28 for Darlington.
The induction process for the role will be conducted in person.
About the job
Job summary
The Office for National Statistics (ONS) is the UK's largest producer of official statistics, covering a range of key economic, social and demographic topics. These include measuring changes in the value of the UK economy, estimating the size, geographic distribution, and characteristics of the population, and providing indicators of price inflation, employment, earnings, crime, and migration.
We are looking for talented Data Engineers to join teams in the Public Sector Division. In this role, you will focus on building robust, scalable data solutions and Python-based software that our production teams can integrate into their operations. You'll be responsible for managing our Python codebase, creating intuitive user interfaces, and connecting to cloud databases, helping to ensure our systems remain efficient and safe.
Job description
You'll have a passion for combining data engineering principles with software development to create data products that enhance our ability to process, store, and retrieve information across various platforms. You will work closely with a multidisciplinary team of developers, engineers, and production staff, ensuring the smooth and efficient development and deployment of solutions.
Responsibilities
1. Design, develop, and maintain ETL pipelines and data processing workflows to manage and manipulate datasets, ensuring data quality, consistency, and accuracy.
2. Create intuitive and user-friendly interfaces for internal production teams, making sure that all software products are highly usable and meet the users' requirements.
3. Manage and enhance the Python codebase used for both data processing and user interface functionality, ensuring clean, maintainable, and well-documented code.
4. Build and maintain scalable, secure, and efficient cloud-based solutions, including designing database schemas and managing data flows.
5. Implement software development best practices, including version control (Git), continuous integration, testing, and deployment (CI/ CD).
6. Troubleshoot and optimise existing data pipelines and software, ensuring performance improvements and resilience in production environments.
7. Collaborate with data scientists, analysts, and other teams to ensure efficient data integration, storage, and processing, and the transfer of appropriate data engineering knowledge.
8. Provide line management and support to junior team members, guiding their growth and development.
Person specification
Essential Skills Criteria:
9. Ability to determine customer requirements and translate these into delivery design for data processing.
10. Ability to design, write, and operate ETL pipelines, in the context of distributed processing, applying coding, data, and documentation standards, in the language required by the business area.
11. Experience of developing in one of the data platforms ONS use; Cloudera or GCP. With experience in Python and knowledge of SQL and/ or R would be an advantage.
12. Experience of writing clean, efficient and well documented code for data processing tasks to a specification.
13. Experience of using and applying code version control across all work using Git with GitHub or GitLab.
14. Ability to lead the development of ETL pipelines, including effective delegation of work.
15. Good understanding of the principles of data processing in a distributed system, and ability to use this understanding to ensure and design robust coding in a distributed environment, including the creation of flow charts to illustrate coding flows and unit tests and integration testing.
Behaviours
We'll assess you against these behaviours during the selection process:
16. Leadership
17. Changing and Improving
18. Managing a Quality Service
19. Working Together
Technical skills
We'll assess you against these technical skills during the selection process:
20. Programming and Build
21. Data Integration Design