Upgrading to Natural Language Understanding: Challenges, Metrics, and Results
Published on : 27-10-2017
Alfa Bank Ukraine is the part of Alfa Group Consortium, one of the largest privately owned investment groups involved in commercial and investment banking, asset management, insurance, retail trade and water utilities The group includes 5 banks: Alfa-Bank Ukraine, Alfa-Bank Russia, banks in the Netherlands, in Belarus and in Kazakhstan with representative offices in Cyprus, the United States and the United Kingdom.
Alfa Bank’s call center is available in four languages, has over 240 agents working in a 24/7 service environment and handles well over one million service calls a year. In examining their customer care service, Alfa Bank identified a number of old technologies that stood between them and a happy customer including a unwieldy DTMF Menu Tree, use of Structured / Directed dialogues and many different agent to agent transfers.
agent costs approached
5x the cost of IVR
Of the problems identified, the DTMF Menu Tree was found to be the biggest deterrent to customers using the call center due to incorrect call routings and lengthy average handling times due to many agent to agent transfers. Research showed that agent costs approached 5x the cost of IVR and that a clear demand for self service was increasing.
The Challenges of Implementing NLU
The challenges involved in implementing NLU were broken down into four major sectors:
Alfa customers speak Ukrainian, Russian, and a mixed language as well. It is common to start a number in one language and finish it in another. A mixed language ASR Engine was essential.
The sound quality reaching the call center can be poor and most ASR engines failed to recognize properly with Word Error Rates as high as 50%. An adapted acoustic model needed to be developed.
Customer Learning Curve
It can be difficult to introduce customers to new systems. A special VUI (Voice User Interface) to help customers get accustomed to the new system was a necessity.
Complex IT Environment
The contact center IT environment was based around a variety of different vendors and interfaces. The NLU solution needed to be able to integrate and upgrade seamlessly.
Establishing KPIs Measuring Success
Omilia’s integration with Alfa Bank’s contact center went from initiation to complete go live in less than a year and three main technical KPIs were established:
1. Word Error Rate (WER)
The percentage of words that the ASR engine incorrectly transcribes.
2. Concept Identification Success rate (CISR)
The extent to which the system successfully extracted the correct meaning out of each individual caller utterance.
3. Task Completion Rate (TCR)
A task is considered completed when the system processes the customer request in accordance with the established business logic.
With Omilia’s Deep Neural Networks powering Automatic Speech Recognition we achieved state-of-the-art results:
And the above results are just the beginning! Omilia’s conversation IVR solution resulted in better customer care quality including a 43% increase in call capacity and a 15% increase in self service usage.
The new solution also significantly reduced agent overheads including a 50% reduction in average IVR handling time, a 15% reduction in internal transfers and a 21% reduction in agent operating costs.