SmartSearch’s distinctive Anti‑Money Laundering verification software protects our clients by offering the most advanced and comprehensive features available from an AML provider. SmartSearch has grown rapidly by fostering an incredibly collaborative and supportive culture. As we continue our ambitious growth plans, we will strive to remain a truly exciting, rewarding, and unique place to work.
HOW WILL YOU MAKE A DIFFERENCE?
We’re looking for an AI Quality Analysist to help validate and improve Generative AI products and features. This role focuses on testing LLM-based systems, prompt workflows, agent behaviours, RAG pipelines, and AI‑driven user experiences. You’ll work closely with product, engineering, and AI teams to identify quality risks, design robust test strategies, and ensure our GenAI solutions are reliable, safe, and production ready.
Varied Day To Day Responsibilities
* Design and execute test plans for Generative AI applications, including chat, summarization, extraction, classification, and agent workflows
* Validate LLM outputs for quality, relevance, accuracy, consistency, tone, and policy compliance
* Test prompt behaviour across normal, edge, and adversarial scenarios
* Evaluate RAG systems for retrieval quality, grounding, hallucination risk, and citation accuracy
* Create test cases for multi‑step AI workflows, tool use, memory, and orchestration logic
* Partner with engineers and product managers to define acceptance criteria for AI features
* Identify and document defects, regressions, safety issues, and model failure patterns
* Build and maintain test datasets, evaluation rubrics, and benchmark suites
* Support automated testing for AI features, APIs, and end‑to‑end workflows
* Track model and prompt changes to assess quality impact over time
* Contribute to release readiness by reporting on AI quality metrics, risks, and recommendations
WHAT ARE WE LOOKING FOR IN A CANDIDATE?
* 3+ years in QA, software testing, or test engineering
* Hands‑on experience testing AI/ML or Generative AI applications
* Strong understanding of LLM behaviour, including hallucinations, prompt sensitivity, grounding, and non‑deterministic outputs
* Experience writing detailed test cases, bug reports, and test summaries
* Familiarity with API testing tools and test automation frameworks
* Ability to evaluate model outputs using structured rubrics and human judgment
* Strong analytical skills and attention to detail
* Comfortable working cross‑functionally in a fast‑moving product environment
Preferred Qualifications
* Experience testing chatbots, copilots, agents, or RAG‑based applications
* Familiarity with prompt engineering and prompt/version evaluation
* Experience with Python, SQL, or scripting for test data and automation
* Knowledge of AI safety, red teaming, bias testing, and policy evaluation
* Experience with model evaluation frameworks, telemetry, or experimentation
* Exposure to cloud AI platforms and LLM providers
WHAT IS LIFE LIKE AT SMARTSEARCH?
* We are a multi‑award winning Tech company with an aspirational mentality
* Some of our most recent recognitions include: named in the renowned RegTech100 list for 2024, listed in the Top 100 Fasted Growing Tech Companies by Northern Tech Awards 2024 as well as being named Technology Provider of the Year by Corporate Finance Awards 2024
* We have been Great Place To Work Certified since 2022
* There are excellent progression opportunities due to our growth and you will have personal development goals, regular feedback and support
* We are a diverse and inclusive team committed to promoting Diversity & Inclusion and Social Responsibility. Through our DE&I group, charitable initiatives and support for local schools, we actively foster a positive Impact on our community
Company Benefits
Our comprehensive benefit package includes:
* 25 days holiday rising to 30 with each year of service
* Private Medical Insurance covering dental and optical
* Company pension scheme
* Life Assurance – 4x your annual salary
* 1 day paid volunteering per year
* Enhanced maternity / paternity offerings
* Employee Assistance Programme
* Cycle to work scheme
* Access to a gym
#J-18808-Ljbffr