Automated Testing for Alexa Skills – Validation

Automated Testing Alexa Skills Blog Image

Note: Please see updated blog post.

We are very pleased to announce our Validation feature is available in Beta on the Bespoken Dashboard. You now have the power to run automated testing for your Alexa Skills – and you can do it all without talking!

With Validation, you can create a sequence of tests for Alexa by just typing. These tests are then automatically run against the actual Alexa Voice Service (AVS) and your skill. It’s magical, it’s extremely useful for ensuring voice app performance, and it goes great with our new bst speak command as well as our Virtual Device SDK. They all rely on the same underlying technology to make testing Alexa easier than ever.

First, here is a quick demo of Validation, accessed via our Bespoken Dashboard:

Pretty cool, right?

So what exactly is happening? Let’s look at our script:

Each line represents an interaction with Alexa – what will be said, and what we expect Alexa to reply. The first input, “open guess the price”, opens the skill we want to interact with. The second part of the line, after the colon, is what we expect in reply: “guess”.

When the test is run, the expected will be compared to what Alexa actually says, but only for partial matches. That means for the expected phrase “guess”, replies of “guess who i am”, “guess who”, and “guess” would all be accepted.

The second line represents the next step in the conversation – the session is maintained between calls, so after the skill is opened and replies, “one” is said next.

When we click “Run”, we see the actual results coming back from Alexa:


Since we expected a result of “guess” for the first interaction, and it came back with “welcome to guess the price how many persons are playing today”, that is success (remember, it’s just a partial match).

The whole sequence is run through, and the final result is shown in the summary.

So what is actually happening under the covers?

How Validation Works

Validation makes use of our Virtual Device SDK. The SDK does the following:

  • Turns the input “open guess the price” into audio using text-to-speech
  • Sends the audio to the Alexa Voice Service (AVS)
  • Receives audio in reply from AVS (as well as metadata like card info)
  • Turns the audio back into text for comparison

This allows you to test with Alexa without using your voice or an actual Alexa device.

How Do I Get It?

First, you need to signup for a Bespoken Dashboard account – go ahead and do that here. No hurry (though it takes only two minutes) – we’ll be here when you get back 🙂

Next, create a new source, and select the Validation tab:

Once you are there, we need to setup a virtual device with Alexa. This is done by clicking on the links at the top of the page:

First enter your Vendor ID (which can be found at the link below the field). Once that is entered, setup your Virtual Device. This will take you through the AVS authorization flow. We need access to your Alexa account, as well as information about your skills (this is to authenticate that you are in fact the owner of the skill and allowed to access it).

Once authorized, you will be able to run your own tests, as well as utilize the Virtual Device SDK programmatically in your projects. Using this, end-to-end tests can be setup that are automatically run on an interval, or anytime code changes for your skill. Next week, we’ll go into more depth on how to use the SDK in Continuous Integration (CI) processes.

For now, we encourage you to try it out and let us know what you think! And feel free to ping us Gitter or in the Alexa slack channel (jpkbst or ivan) if you have any questions at all. Happy testing!

Leave a Reply

Your email address will not be published. Required fields are marked *