Part 1: Plan and Research
The world of Conversational AI is changing the way we interact with technology. This blog post series is a Comprehensive Guide to understanding and managing Conversational AI applications, from inception to ongoing maintenance. Whether you’re a seasoned practitioner or just starting out, this guide covers key considerations like testing with real users, collecting data for measuring performance, and formulating responses that balance brevity with informativeness and completeness.
Start with the Objective
Perhaps the most fundamental consideration for any conversational initiative is to understand the key business goals of the application. Defining what success looks like at the outset is, of course, essential to achieving it. How to do this? Here is what we recommend:
- Interview stakeholders to understand goals and driving forces
- Understand the goal for a conversational assistant compared to other channels, such as a web and mobile app
- Get clarity on executive buy-in and what constitutes success
A key outcome of these activities is a set of well-defined metrics which we can then use to measure the progress of our initiative once underway. These metrics may include:
- Containment – the percentage of how many customer inquiries can be handled without human intervention
- Task success – the percentage of customers that received a satisfactory/successful response to their inquiry
- Customer satisfaction – this can be as simple as Net Promoter Score (NPS), or based on more granular survey data, as well as on analytics tools such as Sentiment Analyzers.
- Accuracy rate – how well-understood users are by speech recognition and natural language systems.
For each of these metrics targets should be set at the outset. Though they may be imprecise, it provides a baseline for initial assessment, which can be revised and expanded upon as the system evolves.
Use Case Selection
The process of identifying initial use cases is especially important. Ideal use cases are:
- Low-complexity – they are relatively easy to implement with modern platforms
- High-value – they address a use-case that is common and useful to users
- Self-contained – ideally, the initial use case solves a problem in its entirety, as opposed to being a step in a long process that involves, say, talking to an agent
- Variable Demand – use cases where demand varies from day to day and even hour-to-hour are a great fit for Conversational AI. These are situations that are hard to address with manual processes, leading to either over-staffing when demand is low, and under-staffing and long wait times when demand is high
There are many, many potential applications for Conversational AI, and we believe that organizations will have the opportunity to grow into them over time. But to start, it is critical to select ones that will lead to successful implementations. This means avoiding some of the pitfalls and anti-patterns of Conversational AI.
Conversational AI Anti-Patterns
Some use cases that do NOT work well, or are at least prone to failure for teams that are relatively new to Conversational AI are:
- Applications with extremely large numbers of intents, such as FAQs
- Applications with multi-step dialogs, for example, a bot that takes the place of what would normally be a web form
- Applications with extremely large numbers of entities, such as product catalogs that might be offered by grocery stores, online delivery services, or music players. These catalogs might contain 1000s of entries and require significant pruning, cleaning, and testing.
- Open-ended, “just talk to me” applications – though some argue this is the true spirit of Conversational AI, in practice, it is important to solve more narrow, tailored problems, and ensure users’ expectations for what an application can do are set properly from the outset
There are of course success stories for all of the above, but they should not be a starting point, and even expert practitioners need to proceed with caution for these use cases. The last one especially is simply a “no-go” zone – though it may be the holy grail of Conversational AI, it is simply not enabled by current technology.
Building A Roadmap
Once initial use cases are selected, it’s important to look at the big picture on which ones to start with, how they can be expanded, and when additional use cases will be implemented.
One very AI-centric way to look at it is that the entirety of an organization’s interactions can be thought of as intents (things the user would like to do or information they would like to get), some of which are handled automatically, and some of which are handled manually.
The goal is not, though, to automate all the intents on day one, or even day 1000. Instead, think of the process of adopting Conversational AI as an intent-by-intent expansion, in which over more time the balance of intents that are handled automatically versus manually is steadily chipped away at.
Our roadmap then is the map of all these intents, and how we want to apply self-service to them over time.
This roadmap is laid out in GitHub, but you can use any tool that is convenient and familiar. The key thing is to:
- Identify the core behaviors that will be supported
- Identify timelines for adding support
- Assigning resources for near-term items on the roadmap, and ensuring that the proper team members and tools are in place/available
For our conversational assistant, we start with handling fairly simple queries, such as “how are you” and “hi”. In the second phase, we get to slightly more complex queries, such as those related to the weather and status, which may require a lookup to an external source of information.
In our third phase, we look at more complex requests, such as “I would like food” or “bring me a drink” – both will require a multi-step process to capture the complete set of information needed to fulfill these requests.
Though this starts off as a fairly simplistic assistant, we see how it will become more complex over time. Our immediate goal is to make it handle basic scenarios and to build from there.
Setting Objectives and Metrics
There are many metrics to look at for Conversational AI. Depending on the stage of the life cycle, we have suggested specific detailed, tactical ones.
At a high-level, though, common key goals are:
- Increasing customer self-service
- In contact center scenarios, reducing wait times and average handle times
- For embedded and general-purpose assistants, increasing usage and return rates
- Ensuring high levels of customer satisfaction
- Ensuring high-level task accuracy – how often was the right result delivered for the customer’s request/query
And, of course, the overall ROI of the investment is critical to look at, as it is with any application of technology.
For all these objectives, having a baseline idea of current performance is essential. Though the first three objectives are commonly tracked and often readily available, the last two are less common and data quality is harder to ensure. That does not make them less worthwhile though – it is critical to understand how customers perceive the quality of the application.
Similarly, understanding in objective terms, whether or not their requests were fulfilled is essential to improving customer satisfaction over time. Though it is not the only aspect of how customers will perceive your application, it is fundamental to evaluating how well a Conversational AI is working. It also complements and summarizes other critical indicators for performance such as speech recognition accuracy, and natural language accuracy, which we have found in practice to be essential to success.
Follow Along
In order to clearly illustrate how to build a conversational experience, we will build a bot from scratch as part of this.
The bot we are building is available on GitHub, a free resource for hosting and collaborating on software projects. We will also employ a variety of third-party tools to assist in building it. Though the tools we use are meant to be ones that we find useful and helpful, keep in mind they are not meant to be endorsements of specific vendors.
For this example project, we will build out a conversational companion that will respond to commands and questions. This example is meant to be whimsical and fun – follow along as we construct a four-legged, digital friend.
Research
The research covers all aspects of understanding the potential users of the system, their preferred methods of interaction as well as the landscape of similar/competitive offerings. Key considerations when designing conversational experiences:
- Identify and document the core audience and personas for the application
- Identify key channels of communication – telephone, webpage, WhatsApp, etc.
- Identify key environments in which the application will be used – in the car, on the street, at home, etc.
Understand Needs And Expectations
The focus here is on understanding user needs and expectations for the conversational application. There are several ways to approach this:
- Review user interactions on similar channels, e.g. for a new chatbot, examine speech-enabled IVR to determine use cases and user request phrasing
- Review user interactions in other channels, such as mobile applications, to determine the most popular use-cases
- Learn user likes and dislikes from competitive and comparative research, e.g. competitive apps or internal assistants.
The information gleaned from this process becomes grist for identifying the initial use cases for implementation.
Define The Audience
This is about mapping out who the key users are for the system. We analyze their psychographic and demographic information, such as below:
Note – the target audience is distinct from a persona – which is the “idealized” form of the user.
Identify Key Channels
Over time, the channels of a conversational application will expand. But it is critical to identify the initial areas of focus.
These may be Amazon Alexa, IVR, SMS, WhatsApp, etc. Each channel has its own unique idiosyncrasies for how people interact with it, as well as potentially distinct audiences, constraints, and challenges.
In selecting an initial channel(s), we recommend picking ones that will fit the use-case well as well as are most commonly used amongst the target audience.
For our assistant, as it is meant to be a general-purpose companion and assistant, we will start with two very common channels: the telephone and WebChat. The choice of using IVR for this particular assistant may not seem obvious, but we want to build something that is available to anyone. Additionally, for the purpose of this guide, we want to cover some of the most commonly used channels.
Identify Key Environments
Along with the channels that our bot will support, identifying the environments in which it will be used is critical, especially for assistants that rely on voice. When we are interacting with users via audio, some key factors are:
- How far the user is from the device
- How noisy the environment is around them
- Are there other people that are present for the conversation? Are they participating?
This information is then leveraged as part of our design. We cover that in our next section.
Ready to revolutionize your conversational AI systems? Sign up for a free trial at Bespoken today! Our innovative tools and expert guidance will help you create conversational experiences that your users will love. With our powerful model and monitoring capabilities, you can fine-tune your systems to deliver exceptional performance and results. Don’t miss out on this opportunity to take your conversational AI to the next level – sign up for a free trial at Bespoken now!