Tutorial: Your first skill

Introduction

In this tutorial, we will walk through the basics of starting a skill and inspecting the default skill template that new skills are based on.

This tutorial assumes that you have followed the basic instructions for:

  • Creating a skill (you can name it GettingStarted)
  • Imported the skill into IntelliJ IDEA.
  • Made sure you have a working microphone
  • Made sure you can run it from IntelliJ, either with the virtual or physical robot.

Inspecting the skill package

The main files to care about for your skill are in the src folder (or more specifically src/main/kotlin/furhatos/app/gettingstarted). Navigate there and you will notice a few files has been created for you.

The main.kt file

The main.kt file contains the following:

package furhatos.app.gettingstarted

import furhatos.app.gettingstarted.flow.*
import furhatos.skills.Skill
import furhatos.flow.kotlin.*

class GettingstartedSkill : Skill() {
    override fun start() {
        Flow().run(Init)
    }
}

fun main(args: Array<String>) {
    Skill.main(args)
}

Let's inspect the contents (we're skipping the package definition and imports):

class GettingstartedSkill : Skill() {
    override fun start() {
        Flow().run(Init)
    }
}

This is where we define the main class representing the skill.

The start() method is, just as it sounds, what starts your skill. Looking inside it, Flow().run(Init) creates a new Flow instance. This instance runs an initial state Init, which will set the first parameters and go to an Idle state. You can see the flow (also referred to as Dialog Flow) as the control mechanism of the skill. It is a state-chart implementation that basically consists of a set of states and transitions between them. For more information, see the Flow docs.

Next up, we have a main method:

fun main(args: Array<String>) {
    Skill.main(args)
}

This is what you use to start the skill from your IDE during development and testing. Behind the scenes it:

  1. Connects to the Furhat SDK server, assuming it runs on localhost.
  2. Once connected, initializes an instance of the main class (provided in the skill.properties file in the root of the skill. In this example you will see that the main class is furhatos.app.gettingstarted.GettingstartedSkill in a new thread.
  3. Sets up logging for the skill.
  4. Starts Furhat's autobehaviors (blinks, face microexpressions etc).
  5. Runs the skill's start() method.

The flow files

In the flow folder (and the sub-folder main), you will find a few boilerplate states that help you get going. As you build more complex skills, you will create more complex behaviors to handle these situations, but this boilerplate will work well for many basic interactions.

Let's walk through them:

Init state

Starting off, we review the init.kt and the state named Init (the one we set as the initial state above). It is mainly a transition state where we initialize parameters and settings before starting the interactions.

val Init: State = state {
    init {
        /** Set our default interaction parameters */
        users.setSimpleEngagementPolicy(DISTANCE_TO_ENGAGE, MAX_NUMBER_OF_USERS)
    }
    onEntry {
        /** start interaction */
        when {
            furhat.isVirtual() -> goto(Greeting) // Convenient to bypass the need for user when running Virtual Furhat
            users.hasAny() -> {
                furhat.attend(users.random)
                goto(Greeting)
            }
            else -> goto(Idle)
        }
    }
}

In this state, there are two triggers defined. The onEntry trigger is run everytime the state is transitioned to. The init trigger is only triggered once (and not if we return to the state). Note that the init state is triggered before the onEntry state. For more info, see the Flow triggers.

In the init trigger, we set up a SimpleEngagementPolicy, with the parameters defined in the /settings/interactionParams.kt file. This allows us to fine-tune how close a person needs to get to the robot in order to activate a state's onEntry trigger, and how many users we allow at the same time. If you want to specify the voice and the character of the robot, this would be a good place to do that.

In the onEntry state, we then check the following:

  1. If we are running with the virtual robot, go directly to the interaction (Greeting)
  2. If we are running with a physical robot, check if there are any users detected. If so, attend a random user and start the interaction. Otherwise,
    go to the Idle state, to await users.

Virtual users

If you want to see what it is like to actually handle users entering and leaving, even if you are running with the virtual robot, you can remove the first line in the when-clause. You can then test the interaction with virtual users:

  1. Go to the web interface (using the default port, it will be hosted on http://localhost:8080 if your SDK server is running).
  2. Go to the Dashboard using the left menu.
  3. As you can see, there are two zones around Furhat (shown as grey circles in the situation view). When the user enters the inner zone, it is deemed to enter the interaction, and Furhat will start interacting with the user. To simulate this in the SDK, you can double-click on the situation view somewhere inside Furhat's inner interaction space to add a virtual user. This should trigger the interaction to start.
  4. You can also drag the user around to move it. As you can see, Furhat will continue to attend the user. When the user leaves the outer zone, the user will be considered to have left the interaction and Furhat will stop interacting with the user. If you double click on a user, it will disappear. You can also insert more users by double-clicking again somewhere else.

The Idle state

The first passive logic bit can be found in the idle.kt file and the state named Idle. The purpose of this state, as the name implies, is to handle the resting behavior of the robot - i.e when you first start up your skill but before the main interaction begins AND at any point later on when the interaction has ended and Furhat waits for another interaction to start.

val Idle: State = state {
    onEntry {
        furhat.attendNobody()
    }

    onUserEnter {
        furhat.attend(it)
        goto(Greeting)
    }
}

In this state we have two triggers:

  • When entering the state (onEntry), Furhat's attention is reset.
  • onUserEnter is triggered when a user appears in Furhat's engagement zone. This will make Furhat attend the user (through the Kotlin keyword it) and then transitions to the Greeting state.

The Parent state

Next up, we review the parent state in parent.kt. The purpose of this state is to handle basic user management during an interaction, allowing the robot to behave in a socially appropriate manner as users come and go from the interaction space. Since these behaviors might be needed at any point during an interaction, this state would be used as a Parent state, which would be inherited by other states throughout the skill, as explained below.

val Parent: State = state {
    onUserEnter(instant = true) {
        when { // "it" is the user that entered
            furhat.isAttendingUser -> furhat.glance(it) // Glance at new users entering
            !furhat.isAttendingUser -> furhat.attend(it) // Attend user if not attending anyone
        }
    }

    onUserLeave(instant = true) {
        when {
            !users.hasAny() -> { // last user left
                furhat.attendNobody()
                goto(Idle)
            }
            furhat.isAttending(it) -> furhat.attend(users.other) // current user left
            !furhat.isAttending(it) -> furhat.glance(it.head.location) // other user left, just glance
        }
    }
}

Reviewing the state definition above, the state contains two handlers that both have an instant = true flag. This flag means that the trigger is expected to run instantaneously, with no blocking calls such as furhat.say() or call(). The instant flag is needed since this state is only used as a parent state and we don't want to stop the execution of the child state when either of these triggers are activated - which would be the normal behavior if instant was false (default). For more info, see Triggers when calling states

Looking at the first trigger, onUserEnter is executed if a user enters Furhat's interaction space. Here, we check for two conditions:

  • If Furhat is already attending someone (i.e., the user who entered is a secondary user), Furhat should just glance at the secondary user.
  • If Furhat is not attending anyone, Furhat should attend the user who entered.

Note how we can refer to the user that entered by the it variable.

The second handler onUserLeave is executed if an engaged user leaves (here the it variable refers to the user that left):

  • If there are no more users engaged, Furhat should reset its attention and go to the Idle state.
  • If Furhat was attending to the user that left, Furhat should attend to some other engaged user.
  • If Furhat not attending to the user that left, Furhat should just glance at the user that left.

The Greeting state

The last state provided by the template is a simple interaction state, that we find in greeting.kt. As can be seen, the Parent state is provided as a parameter to the state definition. This is how you define that the Greeting state inherits the Parent state. This means that the two handlers described above, onUserEnter and onUserLeave will be active in the this state. For more info on the state machine, see the Dialog flow docs and specifically State inheritance.

val Greeting : State = state(Parent) {
    onEntry {
        furhat.ask("Should I say Hello World?")
    }

    onResponse<Yes> {
        furhat.say("Hello World! .")
    }

    onResponse<No> {
        furhat.say("Ok.")
    }
}

Here we have three triggers. When entering the state (onEntry), Furhat will ask a question ("Should I say Hello World?"). This is the same as doing a furhat.say()

The two onResponse handlers are defined, each triggered by a specific Intent - Yes and No respectively. These two intents are system-defined intents. As you start building your skill, you will define your own intents. For more info on intents and other parts of the NLU (Natural language understanding) of the Furhat SDK, see NLU docs.

When a furhat.listen() or a furhat.ask() is executed, Furhat will start listening. If speech is detected, the recognized speech will be classified as being one of the intents in the provided onResponse handlers, in either the current state or in any of the inherited states. For more information, see Listening docs

When running the skill, you might notice that if you answer something else than "yes" or "no" to Furhat's question he will reply that he doesn't understand, and he will repeat the question. This is a default behavior that you can override if you want. See Changing default responses.

Other files

There are a few other files created aswell, with the following uses:

  • build.gradle (in skill root): the build-file for the skill. Any dependencies you would like to add to the skill can be added here in a typical gradle fashion. This is also where you list what version of the SDK library furhat-commons that you want to use. When we release a new version of the SDK, this is where you will update the file.
  • skill.properties (in skill root): a file with a few settings for your skill