[Video] Continuous Alexa Skill Testing with Bespoken Validation

We are making Alexa skill testing easy. In this video, watch how Bespoken will monitor your Alexa skill script continuously – before and after publication – to ensure it’s working properly. Follow along in this 90-second video and then try Bespoken testing on your Alexa skills for free.

Create a free Bespoken account or log in.

Want some help getting started? The Bespoken team has been writing and testing hundreds of interactions with Alexa and can recommend a few helpful tips and tricks to get you started with automatically testing and monitoring your Alexa skills using the Bespoken Validation.

To start, it will help to understand how the test script sequence works and how the comparisons between inputs and outputs are configured.

Write and Execute Alexa Skill Test Scripts

First, let’s take a look at how Bespoken Validation is setup. When you select your Alexa skill in the Bespoken Dashboard and click on the Validation tab, you will see 3 columns with the option to add multiple rows using the plus sign at the bottom. In the first column, you will type your inputs to Alexa – just like you were talking to the device out loud. In the second column, type what you expect Alexa to say back to you, and in the third column, the results from Alexa Voice Service (AVS) will generate and show up there.

The first row or line of your Bespoken testing script should always include an Alexa skill launch phrase, such as “launch”, “ask”, “tell”, “load”, “begin”, or “open” followed by your skill invocation name. Bespoken Validation is set up to work as one sequence, so you only have to launch your skill once. Check out the example sequence below with three lines.

After you open your skill in the first line, continue to add rows – each row is an interaction with Alexa and your skill, and part of a conversation. Try out different utterances and sequences, with the goal of building a valuable conversation that is an effective, simple test for your skill.

Bespoken Testing Validation Example Screenshot

Understanding Your Results

By default, Bespoken Validation performs a partial match. A partial match means the expected response needs to just be a part of the actual response. For example, the expected response “hello” is enough to qualify as OK, even though the full response is “hello, this is your frizz forecast…” The goal is to write tests that ensure your skill is working properly, without over-specifying them and causing false alarms.

Tips for Writing Alexa Skill Test Scripts Using Bespoken Testing

Moving on to the details of the script, here are 3 tips to help you write your own Alexa skill test scripts and optimize them for automated Bespoken testing and monitoring.

1. Simplicity

Bespoken Testing Simplicity

Like most good things, simplicity yields the best results. Use clear and natural language; we recommend keeping inputs and expected results under 20 words. Remember, the session will remain intact within the script. And FYI, it is not necessary to include spelling or pronunciation in the expected results.

2. Wildcards

Bespoken Testing Wildcard Example Image

We rely on speech recognition to take the audio results from Alexa and turn them into text. This is an imperfect process, and sometimes it will misrecognize words. When this happens, we recommend you use the wildcard (*) for these words, so that your test does not mistakenly flag the result as a failure.

In this example, I have found through multiple tests, the word frizz is sometimes recognized as first or fruits. So I have used an * in parts of the script – “your frizz forecast today is…” becomes “Your * forecast today is…”

3. Notifications

When you enable monitoring on your skill – as seen in the video – Bespoken will ping your service every 30 minutes to ensure it’s working correctly.

What makes a monitoring service valuable are effective notifications. When Bespoken detects an outage or error in your script, you will get an email with the details of the error. In addition, you will also receive a weekly validation report that will include a full summary of your results and a week-over-week view, so you can track improvements over time.

So there you have it. Use these 3 tips to help you use Bespoken testing tools as a part of your end-to-end testing program. And to check out how to cover all 4 phases of testing voice apps (manual, unit testing, end-to-end, and continuous), read this Amazon Alexa blog post written by our very own John Kelvie, Building Engaging Alexa Skills: Why Testing and Automation Matter.

Questions About Bespoken Testing Tools?

Bespoken Validation is the newest feature to join the Bespoken Suite. Browse our other voice development tools here and contact us with questions or comments.

Leave a Reply

Your email address will not be published. Required fields are marked *