Your first skill

Introduction

In this tutorial, we will walk through the basics of starting a skill and inspecting the default skill template that new skills are based on.

This tutorial assumes that you meet the following criterias (If you don't, you can follow our Getting started guide):

  • have a running SDK
  • have a configured microphone.
  • have IntelliJ IDEA (or an equivilant IDE) installed and running.

Creating the skill

Create a skill by going to your SDK directory in your favorite terminal and then write the following command specifying the location where you want the skill to be placed.

On Windows:

cd <path-to-your-SDK-directory>
gradlew createSkill --name=GettingStarted --folder=../skills

On Mac/Linux:

cd <path-to-your-SDK-directory>
./gradlew createSkill --name=GettingStarted --folder=../skills

Importing the skill into your IDE

Open up IntelliJ Idea and import the newly created skill by going to File->New->Project from Existing Sources if you want to create a brand new IntelliJ project, or File->New->Module from Existing Sources if you want it as a part of an existing project.

Important: Furhat skills use Gradle as a build tool so for your IDE to know that it should fetch the skill's dependencies using gradle you need to import the skill as a Gradle project by selecting the build.gradle file in the root of your new skill in the import dialog.

The main files to care about for your skill are in the src folder (or more specifically src/main/kotlin/furhatos/app/gettingstarted). Navigate there and you will notice a few files has been created for you.

Running the skill

To test-run the skill, follow this process:

  1. Make sure you have the Furhat SDK server running.

  2. In IntelliJ IDEA, navigate to the main.kt file of the skill and press the green play button next to the main() method to start the skill.

    Important note: If your skill crashes due to "not able to read skill.properties" then ensure that the working directory of your skill is set to the root folder of the skill. Go to Run -> Edit configurations to do this. To fix this for all future skills, go to Edit configurations -> Defaults -> Kotlin and change Working Directory to $MODULE_DIR$ - See Running a skill for more info.

  3. Go to the web interface (using the default port, it will be hosted on http://localhost:8080 if your SDK server is running).

  4. Go to the Dashboard using the left menu.
  5. As you can see, there are two zones around Furhat (shown as grey circles in the situation view). When the user enters the inner zone, it is deemed to enter the interaction, and Furhat will start interacting with the user. To simulate this in the SDK, you can double-click on the situation view somewhere inside Furhat's inner interaction space to add a virtual user. This should trigger the interaction and allow you to test it.

You can also drag the user around to move it. As you can see, Furhat will continue to attend the user. When the user leaves the outer zone, the user will be considered to have left the interaction and Furhat will stop interacting with the user. If you double click on a user, it will disappear. You can also insert more users by double-clicking again somewhere else.

Inspecting the skill package

The main.kt file

The main.kt file contains the following:

package furhatos.app.gettingstarted

import furhatos.app.gettingstarted.flow.*
import furhatos.skills.Skill
import furhatos.flow.kotlin.*

class GettingstartedSkill : Skill() {
    override fun start() {
        Flow().run(Idle)
    }
}

fun main(args: Array<String>) {
    Skill.main(args)
}

Let's inspect the contents (we're skipping the package definition and imports):

class GettingstartedSkill : Skill() {
    override fun start() {
        Flow().run(Idle)
    }
}

This is where we define the main class representing the skill.

The start() method is, just as it sounds, what starts your skill. Looking inside it, Flow().run(Idle) creates a new Flow instance and in it, runs an initial state Idle. You can see the flow (also referred to as Dialog Flow) as the control mechanism of the skill. It is a state-chart implementation that basically consists of a set of states and transitions between them. For more information, see the Flow docs.

Next up, we have a main method:

fun main(args: Array<String>) {
    Skill.main(args)
}

This is what you use to start the skill from your IDE during development and testing. Behind the scenes it:

  1. Connects to the Furhat SDK server, assuming it runs on localhost.
  2. Once connected, initializes an instance of the main class (provided in the skill.properties file in the root of the skill. In this example you will see that the main class is furhatos.app.gettingstarted.GettingstartedSkill) in a new thread.
  3. Sets up logging for the skill.
  4. Starts Furhat's autobehaviors (blinks, face microexpressions etc).
  5. Runs the skill's start() method.

The flow files

In the flow folder, you will find a few boilerplate states that help you get going. As you build more complex skills, you will create more complex behaviors to handle these situations, but this boilerplate will work well for many basic interactions.

Let's walk through them:

The Idle state

Starting off, we review the general.kt and the state named Idle (the one we set as the initial state above). The purpose of this state, as the name implies, is to handle the resting behavior of the robot - i.e when you first start up your skill but before the main interaction begins AND at any point later on when the interaction has ended and Furhat waits for another interaction to start.

The state basically acts on users entering and if so transitions to another active state.

val Idle: State = state {

    init {
        furhat.setVoice(Language.ENGLISH_US, Gender.MALE)
        if (users.count > 0) {
            furhat.attend(users.random)
            goto(Start)
        }
    }

    onEntry {
        furhat.attendNobody()
    }

    onUserEnter {
        furhat.attend(it)
        goto(Start)
    }
}

Reviewing the above code, this state has an init method (an empty constructor - a Kotlin standard) that is run the first time the state is entered, only. Here, we set an initial voice and then we check if we have any users. If so, we attend one of them and goto the first active state Start.

init {
  furhat.setVoice(Language.ENGLISH_US, Gender.MALE)
  if (users.count > 0) {
      furhat.attend(users.random)
      goto(Start)
  }
}

After that, we have an onEntry handler (also called trigger) that simply resets Furhat's attention. OnEntry is run everytime the state is transitioned to (but after the init method the first time the state is entered). So, if we had any user upon skill start, the onEntry would never execute since the init method transitions to the Start state. For more info, see the Flow triggers.

onEntry {
  furhat.attendNobody()
}

Finally, we have an onUserEnter trigger that attends the user (through the Kotlin keyword it) and then goes to mentioned Start state.

onUserEnter {
    furhat.attend(it)
    goto(Start)
}

The interaction state

Next up, we review the second state in general.kt named Interaction. The purpose of this state is to handle basic user management during an interaction, allowing the robot to behave in a socially appropriate manner as users come and go from the interaction space. Since these behaviors might be needed at any point during an interaction, this state would be used as a Parent state, which would be inherited by other states throughout the skill, as explained below.

val Interaction: State = state {

    onUserLeave(instant = true) {
        if (users.count > 0) {
            if (it == users.current) {
                furhat.attend(users.other)
                goto(Start)
            } else {
                furhat.glance(it)
            }
        } else {
            goto(Idle)
        }
    }

    onUserEnter(instant = true) {
        furhat.glance(it)
    }

}

Reviewing the state definition above, the state contains two handlers that both have an instant = true flag (This is an advanced feature that first-timers can skip for now). This flag means that the trigger is expected to run instantaneously, with no blocking calls such as furhat.say() or call(). The instant flag is needed since the Interaction state is only used as a parent state and we don't want to stop the execution of the child state when either of these triggers are activated - which would be the normal behavior if instant was false (default). For more info, see Triggers when calling states

Looking at the first handler, onUserLeave is executed if a user leaves Furhat's interaction space. If we have any user left, we check if it was the user that we attended that left - at which will attend another user and restart the interaction by going to Start again. If the user that left was not attended, we simply glance (temporary gaze) at this user for a second while continuing the interaction where we were. Finally, if we don't have any user left we go back to Idle state.

onUserLeave(instant = true) {
    if (users.count > 0) {
        if (it == users.current) {
            furhat.attend(users.other)
            goto(Start)
        } else {
            furhat.glance(it)
        }
    } else {
        goto(Idle)
    }
}

The second handler onUserEnter should hopefully make sense by now. Since the use of this Interaction state is as a parent to other states, we are assuming that we are attending a user already. Thus, we only glance briefly at the newcomer here.

onUserEnter(instant = true) {
    furhat.glance(it)
}

The Start state

The third state provided by the template is a simple interactive state, that we find in interaction.kt. We can note here we pass the Interaction state as a parameter to the state definition. This is how you define that the Start state inherits the Interaction state. This means that the two handlers described above, onUserEnter and onUserLeave will be active in the this state. For more info on the state machine, see the Dialog flow docs and specifically State inheritance.

val Start : State = state(Interaction) {

    onEntry {
        furhat.ask("Hi there. Do you like robots?")
    }

    onResponse<Yes>{
        furhat.say("I like humans.")
    }

    onResponse<No>{
        furhat.say("That's sad.")
    }
}

Starting off, in onEntry furhat will ask a question. This is the same as doing a furhat.say() followed by a furhat.listen().

onEntry {
    furhat.ask("Hi there. Do you like robots?")
}

Then, two onResponse handlers are defined, each triggered by a specific Intent - Yes and No respectively. These two intents are system-defined intents. As you start building your skill, you will define your own intents. For more info on intents and other parts of the NLU (Natural language understanding) of the Furhat SDK, see NLU docs.

onResponse<Yes>{
    furhat.say("I like humans.")
}

onResponse<No>{
    furhat.say("That's sad.")
}

When a furhat.listen(), or as in this case a furhat.ask() after specified sentance is said, are executed, Furhat will start listening. If Furhat hears speech, he will try to classify the utterance as any of the intents used by onResponse handlers in the present state, parent states and caller states. For more info, see Listening docs

When running the skill, you might notice that if you answer something else than "yes" or "no" to Furhat's question he will reply that he doesn't understand, and he will repeat the question. This is a default behavior that you can override if you want. See Changing default responses.

Other files

There are a few other files created aswell, with the following uses:

  • nlu/nlu.kt: an empty file where we suggest you start putting your intents
  • users.kt: an empty file where we suggest you start putting any user data fields your application needs
  • build.gradle (in skill root): the build-file for the skill. Any dependencies you would like to add to the skill can be added here in a typical gradle fashion. This is also where you list what version of the SDK library furhat-commons that you want to use. When we release a new version of the SDK, this is where you will update the file.
  • skill.properties (in skill root): a file with a few settings for your skill