Quality Assurance

Voice Bot – Skill Testing

Gone are the days when Interactive Bots like Alexa/Google were only seen as doing basic entertainment or information chores for the end user. These Bots are now being used to expand the horizon of the AI usage in our daily lives. You can order your food, get your latest quotes of the stocks invested in, invoke your house appliances or even book an appointment while running on your treadmill.

Voice based search and Interactive Bots are hot trends in the market. More and more organizations are thus making all attempts to tap this opportunity and finding ways to reach out to the end users. Lets have a brief understanding of the Alexa ‘Skills’ functionality which transforms this personal assistant’s ability to interact with other devices and build any possible feature one can think of. Amazon’s ‘Alexa Skills Kit’ can be leveraged to quickly built the suitable skills based on the Skill models provided by Alexa. Developers can choose from the pre-build skill models like Smart home skills, Video skills, briefing skills, music skills or even utilize the custom interaction model for any action that can be imagined.

The Alexa skill directory over the years has gained to so much traction that it finds use cases in almost all the possible domains one can think of. For analogy perspective, it can be considered as an application store, where skills related to business, finance, connected cars, lifestyle, health, education, news, shopping, sports and lot of other categories can be acquired quickly.

On the device connectivity front, Alexa Skill module enables it to connect to other smart devices and take complete control of the other devices and become the most trusted one stop bot for the user. Alexa can make the smart security systems, TVs, Washing machines, refrigerators, thermostats or other equipment’s laced with smart devices capabilities to work based on the instructions fed by the users.

Developing a skill using the Alexa Skill Kit does not require extensive skill set and time. Developer just needs to understand the building blocks from the skill kit. Skill development process contains build, test, distribution and certification. Developer Kit also provides the ability to capture analytics for the skill developed. This analytics details help in retraining the model for better results.

There are two methods to test a skill. One is through the Developer console and other is through the Beta testing.

Testing through Developer Console

Alexa Simulator can be used in the developer console. The simulator maintains the skill session with the skill just as a device would, so the complete workflow can be tested. If your skill supports multiple languages, select the language you want to test from the drop-down list. Manual JSON or voice tone options can also be used. Voice tone option primarily use plain text or SSML (Speech Synthesis Markup Language) to feed into Alexa. Alexa simulator comes with a set of limitations and may not be the best method for testing the skill.

Alexa Simulator can be used in the developer console. The simulator maintains the skill session with the skill just as a device would, so the complete workflow can be tested. If your skill supports multiple languages, select the language you want to test from the drop-down list. Manual JSON or voice tone options can also be used. Voice tone option primarily use plain text or SSML (Speech Synthesis Markup Language) to feed into Alexa. Alexa simulator comes with a set of limitations and may not be the best method for testing the skill.

Beta Testing for Alexa Skills

The skill beta testing tool can be used to test Alexa skill in beta before releasing it to production.For security purpose, Alexa device used by the beta tester must be associated with the email address in the tester’s invitation.  The skill development agency need to ensure configuration of the beta tester email.

TADigital QA team was recently roped in to do a beta testing for a skill being hosted by one of its distinguished customers.

Following are the major steps involved in Beta testing.

Alexa Settings Configuration (Echodot Device with PC):

1.       Install Amazon Alexa App in your PC.

2.       Open any Browser & Login to the https://alexa.amazon.com in your PC.

3.       Click on “CREATE A NEW AMAZON ACCOUNT”.

4.       Create a New Account with Name, Email Address (Skill registered with the ID) and password.

5.       Sign In with your account created above by accessing:  https://alexa.amazon.com in a New  Private Window.

6.       Click on “Settings ” in the left side of the page & select “Set up a new device” under Devices”.

7.       Select the “Echo Dot” under Setup.

8.       After selecting the “Echo Dot” the page gets redirected to the new “Begin Echo Dot Setup”.

9.       A New Setup page is displayed with the “Wait for the Orange Light” with some Instructions in the Echo Dot .

10.   Plug the “Echo Dot” device in to Switch Board.

11.   Wait for 1 to 3 minutes. In the “Echo Dot” device the Color will change from Blue to Orange.

12.   The “Echo Dot” device is successfully enabled with the https://alexa.amazon.com

Note: Both “Echo Dot” device & the PC should be connect with same WIFI only.

Registration of Beta skill under test with Alexa: 

1.       Open the URL https://alexa.amazon.com with a Browser in a new private window & Login with your Credentials.

2.       Click on “Skills” tab on the left side of the page.

3.       A new “All Skills” page is displayed with different Categories & with the Search button.

4.       Search for the skill to be tested on “Your Skills” tab on the Header Section of the Page.

5.       Enable the skill under test. A New Window is opened with “Skill Permissions”.

Click on “ Save Permissions” button. We are good to go with testing the skill.

Alexa skills are developed keeping few standard set of principle for the successful implementation. The tests are designed around these requirements:

  1. Invocation: Invocation name is used for invoking skill (to initiate Alexa App). It  has to be precise and meaningful, and something that user can remember, as user needs to call skill’s name to open it. Ideally skill name should be of two words and matching with complete invocation sentence for eg. Alexa, Open Batman game. Here “Batman game” is invocation name. There can be few different words that can be used in place of “Open” like Launch, Load, Begin etc.
  1. Intent:Intents are the function that our skill will perform. So users can call different Intents to perform different functions. For eg. if any skill does Turn on/off light. So it can have one Intent for Turn on/off light and one Intent for Adding new user.
  1. Utterances:Utterances are synonyms of sentence which we have for calling different Intents. While building Alexa Skill there is option to write utterance manually, these utterances are sentence which user can tell to go to particular intent, or to open particular stage inside our skill. We should write as many utterances as possible. An example of utterance could be

What time the trains leave for Central Park in the morning?

Morning trains for Central Park?

Trains for Central Park?

Please let me know the train time for Central Park

Is there a train for Central Park in the morning?

Although Alexa automatically understands similar or matching sentences, but it is good to provide many utterances. Tests need to be built in using variant of the utterances

Following are some of the suggested validations around which Alexa test design and execution model can be built :

Responses to the invocations

Alexa should be able to respond to the invocations in the desired fashion. Eg: Tell me about ‘Skill’, launch ‘Skill’ should lead to the desired response from Alexa. The responses should be meaningful and engage customer well. In case program is not able to comprehend the invocations, it should gather feedback from the user.

Validation of the intent and Utterances

Tests need to be built around validating all the intents defined are leading to desired results. Need to ensure a good user experience. Multiple combinations of utterances need to be validated for a sample intent. Tests can be build keeping varied users in mind. There are likely different ways of asking the same question. Program should be able to handle exceptional cases.

Validation of Motivation/Compliments/Tips for interactive skills like games/quizzes

Need to validate that motivation or compliments dictionary configured gets invoked based on the conditions setup in the skill. For successful or correct steps followed by the users during playing a game, motivation dictionary could be invoked with responses like ‘Superb’, ‘Amazing’ , ‘Perfect’, ‘Well done’ etc

Similarly for incorrect answers, the motivation dictionary could be invoked as : ‘May be next time, the correct answer is ‘xyz’, Good try, but the answer was ‘abc’

In the spirit of great user experience and involvement, while playing games, Alexa can invoke Tips dictionary. Responses like ‘ Practice daily for improvement’ , ‘The difficulty of this game will increase only if you answer enough questions correctly’

 

Some of the typical issues encountered during testing could be:

  1. Rules/Responses configured for invocation/intent/utterances are not in line with the desired result
  2. Alexa not able to comprehend question
  3. Alexa responses are vague
  4. Alexa not letting the narration complete

 

Analytics can help categorize the issues and help the developer in rebuilding the model. Alexa also has a built in functionality of taking feedback from customer in case user feels that Alexa skill was not able to help him/her with the desired response.

Quality assurance is an integral part of the skill development and can play a pivotal role in the success of any launched skills. Brand value and business objectives are at stake hence an effective quality strategy is must for ensuring success.

About The Author