Senior Data Engineer (Pandas, Polars, PySpark)
Category: Project Management
Main location: United Kingdom, London
Position ID: J0825-1007
Employment Type: Full Time
Position Description:
Our Senior Data Engineers work closely with clients on impactful projects. CGI has been recognized in the Sunday Times Best Places to Work List 2025 and named one of the 'World's Best Employers' by Forbes. We offer a competitive salary, excellent pension, private healthcare, and a share scheme (3.5% + 3.5% matching).
We value inclusivity and diversity, supporting careers in tech and the Armed Forces, holding a Gold Award for our support of the Armed Forces Covenant. Join us to be part of an open, friendly community and develop your career with our support.
Your future duties and responsibilities:
1. Develop ETL pipelines using Python to scrape data from various sources (APIs, SFTP, websites, emails, PDFs).
2. Deploy, orchestrate, and maintain pipelines on Airflow.
3. Use Python web-scraping libraries (requests, BeautifulSoup, Selenium).
4. Handle structured, semi-structured, and unstructured data formats.
5. Perform data transformations using libraries like pandas and polars.
6. Write unit tests using pytest framework.
7. Collaborate with platform and framework teams to maintain scraping platforms.
8. Query datasets using SQL (Databricks, PostgreSQL, MSSQL).
9. Promote data processing best practices within the team.
10. Conduct code reviews to standardize code quality.
11. Maintain automated CI/CD pipelines (Azure DevOps, Jenkins).
12. Upskill and onboard team members across locations, including offshore teams.
Required qualifications:
* Proficiency in Python with Pandas and/or Polars.
* Experience with web scraping, including Selenium.
* Knowledge of SQL.
* Familiarity with Azure DevOps.
* Experience with Airflow.
Hybrid working: 2 days in London office.
Skills:
* English
* Pandas
* Python
* Azure Data Factory
What you can expect from us:
Join CGI to turn insights into action. We foster ownership, teamwork, respect, and belonging. Be an owner from day one, shaping our strategy and success. Develop innovative solutions, build relationships, and access global capabilities. Grow your career in a company built to last, supported by caring leaders. Be part of one of the world's largest IT and business consulting firms.
#J-18808-Ljbffr