WE ARE
SoftServe isn’t just about technology - it’s about open-minded people who love technology and have everything they need to pursue their passion for creating innovation. SoftServe is a global digital services and consulting company founded in 1993, working on more than 2,000 projects with clients across the USA, Europe, APAC, and LATAM regions.
With over 30 years of experience in digital software development and consulting, SoftServe helps organisations solve complex business challenges, create industry-disrupting technology, and accelerate growth and innovation while optimising operational efficiency.
Our R&D has over 130 experts in Machine Learning, Agentic AI, and MLOps, covering the entire AI lifecycle - from research and prototyping to deployment and optimisation. We partner with government and enterprise clients to design, deploy, and maintain AI systems that are robust, ethical, and high performing.
We are now inviting an AI Evaluation & Quality Assurance Lead to join our UK-based team on a Fixed-Term Contract or contractor position, helping to define and execute the testing and evaluation strategy for large-scale AI-driven services - including systems handling sensitive and high-impact data in complex, multi-stakeholder environments.
IF YOU ARE
* An AI Quality Assurance and Evaluation specialist with over 4 years of experience validating and testing complex systems, including AI, ML, conversational, or agentic technologies.
* Skilled in designing and executing comprehensive AI testing strategies, defining evaluation frameworks, metrics, and quality gates for non-deterministic and probabilistic systems
* Experienced in AI model evaluation, including accuracy, robustness, fairness, bias detection, explainability, reproducibility, and model drift analysis
* Capable of designing and leading evaluation frameworks for LLMs, RAG pipelines, multi-agent workflows, or conversational systems - measuring performance, reasoning quality, latency, safety, and cost efficiency
* Proficient in Python and familiar with AI testing and evaluation tools such as Langfuse, Helicone, TruLens, or other LLM observability platforms
* Skilled in testing Conversational AI and voice-based systems, ensuring intent accuracy, response quality, and multi-turn dialogue reliability
* Able to build and manage evaluation datasets, create realistic test scenarios, and analyse performance under varying model and system conditions
* Comfortable working with teams responsible for automated testing and evaluation pipelines, ensuring effective integration into CI/CD and MLOps processes
* Adept at analysing and visualising model performance data, producing clear and actionable insights for both technical and non-technical stakeholders
* Experienced in leading QA teams and embedding a culture of continuous improvement, data-driven decision-making, and evaluation excellence across development lifecycles
* Confident collaborating with multidisciplinary teams - AI Engineers, Data Scientists, Architects, and Product Managers - to ensure AI quality and reliability at every stage
AND YOU WANT TO
* Define and lead the AI quality and evaluation strategy for mission-critical AI systems and digital services
* Develop and maintain evaluation frameworks, test suites, and dashboards for models, agents, APIs, and conversational systems
* Establish clear quality standards and evaluation criteria for AI deployment, retraining, and monitoring
* Work closely with ML, DevOps, and Product teams to embed testing earlier in the lifecycle (“shift-left” quality assurance)
* Lead initiatives around bias assessment, model drift detection, hallucination prevention, and reasoning quality assurance
* Collaborate with technical teams to automate and operationalise evaluation and monitoring, ensuring consistent and reproducible results
* Translate evaluation findings into practical recommendations for performance improvement, cost efficiency, and responsible AI governance
* Communicate evaluation outcomes to senior stakeholders and governance bodies, supporting transparency and accountability in AI delivery
* Continuously research and adopt emerging AI testing methodologies and tools, helping SoftServe maintain leadership in AI quality assurance
TOGETHER WE WILL
* Deliver trusted, high-quality AI systems that are transparent, reliable, and safe in production environments
* Build evaluation-driven development practices, setting a new benchmark for AI engineering quality across industries
* Collaborate with world-class engineers, data scientists, and QA experts in SoftServe’s AI & Data Science Centre of Excellence
* Support high-impact projects that improve citizen experience and organisational efficiency through responsible AI innovation
* Access SoftServe University and Udemy Business, with more than 11,000 learning opportunities to advance your technical and leadership skills
* Grow your career through SoftServe’s People Excellence Programme, supported by structured development and mentoring
* Enjoy a flexible, inclusive, and supportive work environment, focused on wellbeing and professional growth
* Play a pivotal role in ensuring that SoftServe’s AI solutions are accurate, ethical, and trusted across the public and enterprise sectors
All qualified applicants will receive consideration for employment without regard to race, color, religion, age, sex, national origin, disability, sexual orientation, gender identity/expression, or protected veteran status. SoftServe is an Equal Opportunity Employer.