Splunk Developer – Technical Lead (ITSI)
Location: Edinburgh (Hybrid – 2 days onsite per week)
Contract Role
We are seeking a highly skilled Splunk Cloud & ITSI Engineer with strong software engineering and coding capability to design, build, and maintain enterprise‑grade observability, analytics, and service health platforms.
This is a hands‑on engineering role, requiring strong depth in Splunk SPL development, automation, data engineering, and scripting/programming to build scalable monitoring solutions across a complex enterprise environment.
The role combines ITSI service modelling, advanced SPL development, automation engineering, and Splunk Cloud platform administration, with a strong emphasis on engineering‑quality solutions rather than configuration‑only work.
Key Responsibilities
* Design and implement ITSI service models including service trees, dependencies, KPIs, and health scoring frameworks.
* Develop advanced KPI logic using complex SPL, scripted inputs, and custom calculations.
* Build adaptive thresholds, SLO‑based indicators, and golden signal‑based monitoring.
* Engineer correlation logic to detect service degradation and performance anomalies.
2. Advanced SPL Development & Engineering
* Write complex, production‑grade SPL queries for dashboards, alerting, correlation searches, and analytics.
* Optimise SPL performance through query refactoring, acceleration, and search‑time tuning.
* Develop reusable SPL macros, modular search components, and reusable knowledge objects.
* Debug and enhance large‑scale distributed search workloads.
3. Automation, Scripting & Integration Engineering
* Develop automation scripts (Python or similar) to support data ingestion, enrichment, and ITSI workflows.
* Build integrations between Splunk and external systems using APIs, webhooks, and automation frameworks.
* Engineer data pipelines and transformation logic for observability datasets.
* Support event‑driven automation and remediation workflows.
* Support Splunk Cloud architecture including ingestion pipelines, HEC, forwarders, and deployment servers.
* Manage CIM alignment, data normalization, and structured onboarding of complex data sources.
* Implement RBAC models, index strategies, and data lifecycle management.
* Optimise platform performance, search concurrency, and workload management.
5. Observability & Analytics Engineering
* Build real‑time dashboards using SPL, data models, and accelerated datasets.
* Develop predictive monitoring using MLTK and anomaly detection techniques.
* Integrate logs, metrics, and events into unified observability views.
* Implement alerting frameworks with intelligent suppression, routing, and enrichment.
* Configure NEAP policies to reduce noise and improve signal quality.
* Build Glass Tables, Service Analyzer views, and executive dashboards.
* Design service degradation detection models and incident correlation logic.
* Integrate ITSI outputs with ITSM and CMDB systems.
Required Skills & Experience
* 4–8+ years of hands‑on experience in Splunk Enterprise / Splunk Cloud engineering environments
* Strong software engineering mindset with hands‑on coding/scripting ability (Python or equivalent preferred)
* Expert‑level proficiency in SPL (including complex, multi‑stage queries and optimisation)
* Strong experience designing and building ITSI service models and KPIs
* Experience with automation, APIs, and scripting for integration and data processing
* Deep understanding of observability principles (logs, metrics, traces, golden signals)
* Experience with Splunk Cloud architecture including ingestion, indexing, RBAC, and performance tuning
* Strong troubleshooting skills across search performance, ingestion pipelines, and distributed systems
* Experience with MLTK, anomaly detection, or predictive analytics in observability contexts
Preferred Skills
* Python or similar scripting language for automation and integrations
* Experience with CI/CD pipelines for observability or monitoring platforms
* Splunk certifications (Admin, ITSI Admin, Architect preferred)
* Experience working in large‑scale enterprise or regulated environments
J-18808-Ljbffr