Testing Furhat skills

Introduction

Testing a Furhat skill is more complex than one could think, which has to do with the fact that it's a multi-modal system and that real-time input and output it quite hard to simulate. That said, we try to make it easy for you to test the skills that you build, both when running locally on your developer machines with a Virtual Furhat and when running on a robot.

We recommend you to use the dashboard, explained below, to manually test and monitor your skills. For deployed runs, we will soon launch a log-viewer that allows you to track runs of your skills remotely.

The dashboard

There is an interactive dashboard available as a page on your web-interface (by default localhost:8080 on your SDK, or on the robot's IP if you run on a robot - see robot). This dashboard contains a few important components that will help you test and debug your skills;

  1. Camera feed (only on robots for now) allowing you to see what the robot sees.
  2. Situation model: a 2d representation (you can toggle between top or side) of the interaction space around the robot. Identified users will show up here, and move around as the camera detects them moving. The system will also highlight if users are speaking, and if so which user is attributed the speech.
  3. Interaction log: a chat-style message log of utterances spoken by the robot, by the users and important events such as users entering or leaving the interaction space. You will also be able to see what user was attributed each speech action here.
  4. A log-viewer, showing all your logger.info() messages.

Logging

See logging

Testing NLU

Running through the skill testing various spoken utterances can be tedious, so it makes sense to create unit-tests for your NLU models. Since the NLU is context-dependent (i.e various intents are active depending on which state you are in), the best way is usually to use the getIntentClassifier() method on a state instance to test specific utterances.

The below script shows how you can run the intent-classifier of a state Active and classify input from command-line. You might want to create your own unit-tests in a similar fashion where you assert the truth of classification of specific utterances.

while(true) {
    val utterance = readLine()
    val results = Active().getIntentClassifier(lang = Language.ENGLISH_US).classify(utterance!!)
    if (results.isEmpty()) {
        println("No match")
    }
    else {
        results.forEach {
            println("Matched ${it.intents} with ${it.conf} confidence")
        }
    }
}