Spark Architect / SME
Contract Role- 6 months to begin with & its extendable
Location: Sheffield, UK (min 3 days onsite)
* Context: Legacy ETL code for example DataStage is being refactored into PySpark using Prophecy low-code no-code and available converters.
* Converted code is causing failures/performance issues.
Skills:
* Spark Architecture – component understanding around Spark Data Integration (PySpark, scripting, variable setting etc.), Spark SQL, Spark Explain plans.
* Spark SME – Be able to analyse Spark code failures through Spark Plans and make correcting recommendations.
* Spark SME – Be able to review PySpark and Spark SQL jobs and make performance improvement recommendations.
* Spark – SME Be able to understand Data Frames / Resilient Distributed Data Sets and understand any memory related problems and make corrective recommendations.
* Monitoring – Be able to monitor Spark jobs using wider tools such as Grafana to see whether there are Cluster level failures.
* Cloudera (CDP) – Knowledge of understanding how Cloudera Spark is set up and how the run time libraries are used by PySpark code.
* Prophecy – High level understanding of Low-Code No-Code prophecy set up and its use to generate PySpark code.