Data Engineer
Glasgow, UK
Contract Inside iR35
Day Rate: GBP/Day
Hybrid: 2-3 Days Onsite in a week
Job Description:
Must-Have Primary Skill - Data and Intelligence-Data Management-ETL Data Integration
This role requires the resource to be onsite 3 days a week in Glasgow.
Key Responsibilities:
� Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack.
� Provide thought leadership by recommending the most appropriate technologies and solutions for a given use case, covering the entire spectrum from the application layer to infrastructure.
� Demonstrate proficiency in coding skills, utilizing languages � Python & SQL, to efficiently move solutions into production while prioritizing performance, security, scalability, and robust data integrations.
� Collaborate seamlessly across diverse technical stacks, including Cloudera, Databricks, Snowflake, Azure, AWS, etc.
� Develop various components in Python of our unified data pipeline framework.
� Contribute towards the establishment of best practices for the optimal and efficient usage of data across various on-prem and cloud platforms.
� Assist with the testing and deployment of our data pipeline framework utilizing standard testing frameworks and CI/CD tooling.
� Monitor the performance of queries and data loads and perform tuning as necessary.
� Provide assistance and guidance during the QA & UAT phases to quickly confirm the validity of potential issues and to determine the root cause and best resolution of verified issues.
� Develop SQL-based data validation, reporting, analysis, etc. as required
� Generate comprehensive solution documentation, including sequence diagrams, class hierarchies, logical system views, etc.
� Adhere to Agile practices throughout the solution development process.
� Design, build, and deploy databases and data stores to support organizational requirements.
Skills / Qualifications:
� 4+ years of experience supporting Software Engineering, Data Engineering, or Data Analytics projects.
� 4+ years of experience in data development and solutions in highly complex data environments with large data volumes.
� 3+ years of SQL / PLSQL experience with the ability to write ad-hoc and complex queries to perform data analysis.
� 3+ years of experience developing data pipelines and data warehousing solutions using Python and libraries such as Pandas, NumPy, PySpark, etc.
� 2+ years of experience developing solutions in a hybrid data environment (on-Prem and Cloud)
� Hands on experience with developing data pipelines for structured, semi-structured, and unstructured data and experience integrating with their supporting stores (e.g. RDBMS, NoSQL DBs, Document DBs, Log Files, etc.)
� Experience with performance tuning SQL queries, Spark job, and stored procedures.
� An understanding of E-R data models (conceptual, logical, and physical).
� Strong analytical skills, including a thorough understanding of how to interpret customer business requirements and translate them into technical designs and solutions.
� Strong communication skills both verbal and written. Capable of collaborating effectively across a variety of IT and Business groups, across regions, roles and able to interact effectively with all levels.
� Self-starter. Proven ability to manage multiple, concurrent projects with minimal supervision. Can manage a complex ever changing priority list and resolve conflicts to competing priorities.
� Strong problem-solving skills. Ability to identify where focus is needed and bring clarity to business objectives, requirements, and priorities.
Preferred Qualifications
� Experience in financial services