We are making Alexa skill testing easy. In this video, watch how Bespoken will monitor your Alexa skill script continuously – before and after publication – to ensure it’s working properly. Follow along in this video and then try Bespoken testing on your Alexa skills for free.
Sign up for a FREE 30-day-trial or log in.
Want some help getting started? The Bespoken team has been writing and testing hundreds of interactions with Alexa and can recommend a few helpful tips and tricks to get you started with automatically testing and monitoring your Alexa skills using the Bespoken Validation.
To start, it will help to understand how the test script sequence works and how the comparisons between inputs and outputs are configured.
Write and Execute Alexa Skill Test Scripts
First, let’s take a look at how to set up your tests. When you select your Alexa skill in the Bespoken Dashboard and click on the “Test” button, you will see 3 columns with the option to add multiple rows using the plus sign at the bottom. In the first column, you will type your inputs to Alexa – just like you were talking to the device out loud. Make sure to specify the short-hand property on the slot next to it. In the second column, type what you expect Alexa to say back to you, and in the third column, the results from Alexa Voice Service (AVS) will generate and show up there.
The first row or line of your Bespoken testing script should always include an Alexa skill launch phrase, such as “launch”, “ask”, “tell”, “load”, “begin”, or “open” followed by your skill invocation name. Our Bespoken Dashboard is set up to work as one sequence, so you only have to launch your skill once. Check out the example sequence below with three lines.
After you open your skill in the first line, continue to add rows – each row is an interaction with Alexa and your skill, and part of a conversation. Try out different utterances and sequences, with the goal of building a valuable conversation that is an effective, simple test for your skill.
Understanding Your Results
By default, Bespoken performs a partial match. A partial match means the expected response needs to just be a part of the actual response. For example, the expected response “welcome” is enough to qualify as OK, even though the full response is “welcome to Quick List…” The goal is to write tests that ensure your skill is working properly, without over-specifying them and causing false alarms.
Tips for Writing Alexa Skill Test Scripts Using Bespoken Testing
Moving on to the details of the script, here are 3 tips to help you write your own Alexa skill test scripts and optimize them for automated Bespoken testing and monitoring.
1. Simplicity: Like most good things, simplicity yields the best results. Use clear and natural language; we recommend keeping inputs and expected results under 20 words. Remember, the session will remain intact within the script. And FYI, it is not necessary to include spelling or pronunciation in the expected results.
2. Wildcards: We rely on speech recognition to take the audio results from Alexa and turn them into text. This is an imperfect process, and sometimes it will misrecognize words. When this happens, we recommend you use the wildcard (*) for these words, so that your test does not mistakenly flag the result as a failure.
In this example, I have found through multiple tests, the word butter is sometimes recognized as better or buyer. So I have used an * in parts of the script – “add milk and butter and sugar…” becomes “milk and * and sugar…”
3. Notifications
When you enable monitoring on your skill – as seen in the video – Bespoken will ping your service every 30 minutes to ensure it’s working correctly.
What makes a monitoring service valuable are effective notifications. When Bespoken detects an outage or error in your script, you will get an email with the details of the error. In addition, you will also receive a weekly validation report that will include a full summary of your results and a week-over-week view, so you can track improvements over time.
So there you have it. Use these 3 tips to help you use Bespoken testing tools as a part of your end-to-end testing program. And to check out how to cover all 4 phases of testing voice apps (manual, unit testing, end-to-end, and continuous), read this Amazon Alexa blog post written by our very own John Kelvie, Building Engaging Alexa Skills: Why Testing and Automation Matter.
Questions About Bespoken Testing Tools?
Our suite of Bespoken Tools continues to grow. Browse our other voice development tools here and contact us with questions or comments.