Unit Testing Alexa Skills

We have been doing a great deal of work with the new AudioPlayer interface.

As part of this work, we recognized the need to be able to automatically unit test Alexa skills that we develop, both during development as well as part of Continuous Integration.

This has always been a need – but with the release of the AudioPlayer, with its additional complexity, it has become essential.

We have previously released our bst speak and bst intend commands.

We found these to be very useful for doing basic debugging and manual-testing.

But now for unit tests and CI, we have expanded on this, building a full-fledged Alexa emulator. It is meant to be interacted with programmatically – to create automated unit tests and integrate neatly into a CI process.

Why We Think This Is So Useful

Up until now, ensuring skill quality has been done via a combination of several, not-so-reliable techniques:

• Testing with a real Alexa device (i.e., an Echo – assuming you are lucky enough to live in a country where it is available for sale :-))
• Testing with the Alexa Service Simulator
• Testing with EchoSim.io (which does not support the AudioPlayer interface)
• Testing with hand-rolled JSON requests

Putting all of these together, one can gain a degree of confidence in a skill.

However, with each change to the code, these processes need to be repeated. Even if the QA steps are documented and executed perfectly, it is still easy to miss something.

And for many skills, these steps are likely not documented at all. This creates serious regression testing challenges.

As developers, we luckily came up with an answer to this some time ago – automated unit tests. It just has not been readily available for the Alexa until now.

The Bespoken Tools API (*)

The API can be found here.

In particular, the main class used for unit testing is BSTAlexa.

Though it is written in JavaScript, it can be used to test skills written in other languages. This is because the emulator communicates with a skill over HTTP.

So tests can be written in JavaScript to test skills written in Python, Java or whatever other language is preferred. Example of this will be coming soon.

How It Works

Here is a quick sample of what a unit test looks like written using our emulator:
it('Launches and then plays first',
function(done) {
// Launch the skill via sending it a LaunchRequest
alexa.launched(function (error,payload){
// Check that the introduction is played as outputSpeech
assert.equal(payload.response.outputSpeech.ssml,
<speak> <audio src="https://s3.amazonaws.com/bespoken/streaming/bespokenspodcast-INTRODUCTION.mp3" />' +
'You can say play, scan titles, or about the podcast </speak>');


// Emulate the user saying 'Play'
alexa.spoken('Play',function(error,payload){
// Ensure the correct directive and audioItem is returned
assert.equal(payload.response.directives[0].type,'AudioPlayer.Play';
assert.equal(payload.response.directives[0].audioItem.stream.token,'0');
assert.equal(payload.response.directives[0].audioItem.stream.url,
'https://traffic.libsyn.com/bespoken/TIP103.mp3?dest-id=432208';done();
});
});
});


We have created a detailed tutorial here (*). It walks you through getting setup with the emulator and provides more detailed information on how to use it.

Additionally, we have created a reasonably complex sample project that demonstrates the AudioPlayer interface and the emulator working together.

We call it the Streamer. It takes a Podcast RSSFeed and delivers it to Alexa. It’s a neat use of the AudioPlayer interface and we encourage you to check it out.

The unit tests using the BSTAlexa emulator can be found here.

Additionally, we have in turn hooked it into a CI framework. Codeship, in this case, because of its excellent native support for Docker (we use Docker because our skill relies on ffmpeg – a non-trivial dependency to setup in a production/runtime environment).

However, ffmpeg aside, any CI framework can be used with it.

What Is Next?

We would love to get feedback on our API. Though it covers many aspects of the Alexa service, there are additional parts we plan to add, including:

• Validation of JSON payloads returned by skills
• Emulating Alexa error requests (System.ExceptionEncountered and SessionEndedRequest)
• Support for the PlaybackController interface
• Adding convenience methods that make the emulator as easy to work with as possible (suggestions appreciated!)

The Standard Stuff

Talk to us on Gitter, and stay updated through GitHub.

(*) Please note – our Virtual Alexa project has replaced the BSTAlexa classes. You can read more on Virtual Alexa here.