Hg Capital: Productionising a GenAI chatbot
Hg Capital is a leading investor in European and transatlantic software and services businesses. It identifies companies with great potential and works with them to accelerate progress and support transformation, building valued products or services that improve the way they work.
Targeting productivity gains from GenAI
In exploring AI’s potential to support Hg’s portfolio companies, the Data team had identified a customer service use case. Agents drew on a range of information, documentation and knowledge to close support tickets. Applying Generative AI (GenAI) to this use case could increase productivity, reduce costs and improve customer service. A GenAI chatbot could assist agents by rapidly retrieving summaries from relevant documentation and supplementing them with less structured information from Jira tickets and conversation histories.
Hg built a proof-of-concept chatbot called Retina to test this hypothesis. It was built on Azure services and used OpenAI’s GPT 4.0 as its Large Language Model (LLM). The prototype employed Retrieval-Augmented Generation (RAG) techniques to optimise the output of the LLM. This allowed it to retrieve data from the structured and unstructured sources mentioned above before generating a response.
Very few companies are managing to move from piloting mode to production mode when it comes to GenAI. With Scott Logic’s proven experience of successfully productionising AI prototypes, Hg engaged us to support the conversion of the proof of concept into a secure and scalable production-ready system. In ten weeks we did just that, while also adding value by producing a roadmap for the chatbot’s future development, complete with estimates.
Feedback from users indicated that over 80% of the chatbot’s answers were useful in closing tickets... When projected across the whole cohort, this would correspond to ~£1m+ in annual productivity benefits, without impacting customer satisfaction.
Preparing the chatbot for a successful pilot
We started a two-week discovery phase by exploring the problem space and conducting an in-depth analysis of the prototype. Using our insights, we shaped a prioritised backlog of improvements that were required to deploy the chatbot at scale into the portfolio, and make it production-ready. Once we had implemented the improvements, we were ready to pilot the chatbot with companies in Hg’s portfolio.
An eight-week pilot took the form of A/B testing. Half the customer service agents in each company had access to the Retina chatbot and half performed their duties as normal. The pilot ran for a few weeks and yielded dramatic results.
Feedback from users indicated that over 80% of the chatbot’s answers were useful in closing tickets. An analysis of the closure rates showed a 12% increase in productivity in the pilot group over the control. When projected across the whole cohort, this would correspond to ~£1m+ in annual productivity benefits, without impacting customer satisfaction.
Harnessing AI to business needs is not straightforward. While many companies are running pilot projects, few are demonstrating a measurable impact. With our support, Hg was able to prepare and run a pilot in a matter of weeks which proved irrefutably the business value of its production-ready chatbot.
Charting a roadmap for future development
Hg prioritised speed to market with its Retina chatbot and we supported our client in achieving that aim, focusing on what was needed to prepare the bot for frontline use. However, we also wanted to add value to Hg by providing insights to inform the chatbot’s future development in a rapidly evolving space. We designed and estimated a product roadmap, working in close partnership with the team at Hg. This roadmap set out improvements that could be made over time to enhance the bot’s performance, scalability and upgradability, allowing Hg to keep Retina at the cutting edge.
In support of this roadmap, we also provided an analysis of an off-the-shelf, open source solution. This analysis included an examination of the geographies in which the open source tool would host Hg’s data; this could have regulatory implications for companies in its portfolio. The analysis also looked at the costs of advanced features – for example, filtering the chatbot’s responses using custom access policies to ensure the wrong people aren’t presented with confidential information. By highlighting risks and the difference in operating costs, we helped Hg validate that a product strategy focused on bespoke software development was the right approach.
Thanks to our support, the Retina chatbot went from prototype to production and roll-out across Hg’s portfolio of companies in just a few weeks. Equipped with our recommendations and fully estimated roadmap, Hg is well-positioned to keep pace with GenAI’s rapid evolution and maintain its portfolio companies’ competitive advantage.