We are seeking a highly skilled Python Data Engineer with hands-on experience in Behave-based unit testing, PySpark development, Delta Lake optimization, and Azure cloud services. This role involves designing, developing, and deploying scalable data processing solutions in a containerized environment, with an emphasis on maintainable, configurable, and test-driven code delivery.
Key Responsibilities:
* Develop and maintain data ingestion, transformation, and validation pipelines using Python and PySpark.
* Implement unit and behavior-driven testing using Behave, ensuring robust mocking and patching of dependencies.
* Design and maintain Delta Lake tables for optimized query performance, ACID compliance, and incremental data loads.
* Build and manage containerized environments using Docker for consistent development, testing, and deployment.
* Develop configurable, parameter-driven codebases to support modular and reusable data solutions.
* Integrate Azure services, including Azure Functions for serverless transformation logic, Azure Key Vault for secure credential management, and Azure Blob Storage for data lake operations.
* Collaborate closely with cloud architects, data scientists, and DevOps teams to ensure seamless CI/CD workflows, version control, and environment consistency.
* Troubleshoot and optimize Spark jobs for performance and scalability in production environments.
* Maintain technical documentation and adhere to best practices in cloud security and data governance.
Required Skills and Experience:
* Strong proficiency in Python programming with emphasis on modular and test-driven design.
* Demonstrated experience in writing unit tests and BDD scenarios using Behave or similar frameworks.
* In-depth understanding of mocking, patching, and dependency injection in Python testing.
* Proficiency in PySpark with hands-on experience in distributed data processing and performance tuning.
* Solid understanding of Delta Lake concepts, transactional guarantees, and schema evolution.
* Experience with Docker for development, testing, and deployment workflows.
* Familiarity with Azure components such as Azure Functions, Key Vault, Blob Storage, and Data Lake Storage Gen2.
* Ability to implement configuration-driven applications for flexible deployment across environments.
* Experience with CI/CD pipelines (Azure DevOps or similar) and infrastructure-as-code tools is a plus.
* Strong problem-solving skills and ability to work independently in fast-paced, agile environments.
Preferred Qualifications:
* Experience developing in Databricks or Synapse with Delta Lake integration.
* Knowledge of best practices in data security and governance within Azure ecosystems.
* Strong communication skills and experience collaborating with distributed teams.