Data Engineer Skills and Technologies
Data Engineering:
1. Strong understanding of data concepts - data types, data structures, schemas (both JSON and Spark), schema management etc.
2. Strong understanding of complex JSON manipulation.
3. Experience working with Data Pipelines using a custom Python/PySpark frameworks.
4. Strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each, particularly managing/handling Reference Data.
5. Strong understanding of Data Security principles.
6. Experience managing data owners and access controls.
7. Knowledge of row and column level security, GDPR, and handling sensitive datasets.
8. Strong problem-solving and analytical skills, particularly able to demonstrate these intuitively.
Required skills:
1. Languages / Frameworks:
* JSON
* YAML
* Python (as a programming language, not just able to write basic scripts; Pydantic experience would be a bonus)
* SQL
* PySpark
* Delta Lake
* Bash (both CLI usage and scripting)
* Git
* Markdown
* Scala (bonus, not compulsory)
* Azure SQL Server as a HIVE Metastore (bonus)
1. Technologies:
* Azure Databricks
* Apache Spark
* Delta Tables
* Data processing with Python
* PowerBI (Integration / Data Ingestion)
* JIRA
Due to the nature and urgency of this post, candidates holding or who have held high-level security clearance in the past are most welcome to apply. Please note successful applicants will be required to be security cleared prior to appointment, which can take up to a minimum of 10 weeks.
LA International is a HMG approved ICT Recruitment and Project Solutions Consultancy, operating globally from the largest single site in the UK as an IT Consultancy or as an Employment Business & Agency depending upon the precise nature of the work, for security cleared jobs or non-clearance vacancies.
LA International welcomes applications from all sections of the community and from people with diverse experience and backgrounds.
Award Winning:
LA International, winner of the Recruiter Awards for Excellence, Best IT Recruitment Company, Best Public Sector Recruitment Company and overall Gold Award winner, has now secured the most prestigious business award that any business can receive, The Queens Award for Enterprise: International Trade, for the second consecutive period.
#J-18808-Ljbffr