Actions

The actions are the main building blocks that makes Furhat speak, listen, gesticulate or change attention.

Speech

Speech is arguably the most important action for social robot interactions. To have Furhat say something, use:

furhat.say("Some text")

Behind the scenes, say is calling a say-state that will send an event to the synthesizer to synthesize the text, and then return when the synthesizer is done. This means that your application will continue receiving events while speaking.

Speech synthesis actions are added to a speech synthesis queue. If the queue is empty (i.e., the system is silent), it is synthesized directly, otherwise it will be played when the currently queued utterances are completed. You can control this behavior as described below.

Controlling speech

Two ways of controlling the start of the speech synthesis exists through the following flags:

  • abort: Abort the current speech queue (empty it), and play the new utterance directly. (Default is false)
  • ifsilent: Will only synthesize the utterance if the queue is currently empty. (Default is false)
furhat.say("Hello there", abort = true) // Aborts the current speech and immediately starts speaking this utterance
furhat.say("My name is Furhat", ifsilent = true) // Speaks only if system is currently not speaking

Async speech

The synthesis action is by default blocking, which means that the synthesis is completed before the next action is taking place. If you want to continue with the next action immediately, you can pass the async parameter:

furhat.say("Some other text", async = true) // whatever comes after here will be executed immediately

Non-verbal

Non-verbal sounds (a.k.a Voice gestures) is a key aspect in making your interaction feel human. Voice gestures are voice-specific and examples are "ehh", "ahh" "uhm", coughs, laughs, yawns etc. For more information, see Voice gestures.

Typically, voice gestures are used as normal strings and transformed by the speech synthesizer. Hence, it's important to make sure that you are using a voice that has the specific voice gesture. If not, the below gesture will simply read out "gesture tut tut".

An example of a voice gesture with the Cereproc William voice is:

furhat.say("GESTURE_TUT_TUT")

Listening

To have Furhat listen, you use the listen command:

furhat.listen() // Listen with a default timeout of 8 seconds

furhat.listen(timeout = 4000) // Listen with a timeout of 4 seconds

Behind the scenes, this calls a listen-state where Furhat will start listening through the selected microphones and return once he receives some speech OR the timeout occurs. If speech is caught, the Furhat system will use Natural language processing to parse the meaning of the utterance - for more info see the [NLU documentation].

You catch the speech and no-response events with the onResponse respective onNoResponse handlers - more info in the flow documentation. If no timeout is listed, the default listening time is 8 seconds.

Asking

Asking (a.k.a prompting) is a short hand for combining a say and a listen. It will trigger the same events as the listen method.

furhat.ask("What is your name?") // Ask with a default timeout of 8 seconds

furhat.ask("How old are you? Now you have to answer fast", timeout = 4000) // Ask with a timeout of 4 seconds

Attention

Attention is an important aspect of a robot interaction, especially if it is multi-party. When an agent (user or system) is attending to (looking at) a another agent, this typically means that the attended agent is supposed to respond. Thus, we need to be able to both control the attention of the system agent (the Furhat robot) in an appropriate way, and check the attention of the users.

Controlling the Robot's attention

Attending users

When you attend users, two things happen: 1. The robot will turn to the user 2. The UserManager will change the current user to the new user

Note: if you attend a location, currentUser will not change

Use the furhat.attend() method to have the robot attend users. To access specific users, please see User management.

furhat.attend(user) // Attend a specific user
furhat.attend(userId) // Attend a user based on userId
furhat.attendNobody() // Reset attention

To attend users based on responses or events, use the following examples

onResponse {
  furhat.attend(it.userId) // Attend the user of an onResponse
}

onUserEnter {
  furhat.attend(it) // Attend the user of an event
}

Attending locations

You can also choose to attend a location as follows:

// attend a location at x = 1, y = 0, z = 1 in other words 1m right and 1m in front of the robot
val location = Location(1,1,0)
furhat.attend(location)

Checking the attention

To check what user the robot is currently attending you can use the flow property users.current. More info, see User management.

Glancing

In multi party interactions, sometimes you might want to acknowledge a user that you are currently not attending without changing your current user. Glance is a good way to do this.

furhat.glance(users.other, 2) // glance on a random user that isn't the current user for 2 seconds

The use-case might be that you are talking to a person and another user is entering the interaction, in which case you might want to glanze at the new user for a few seconds before reverting attention to the currentUser. Example below:

onUserEnter {
  furhat.glance(it, 1) // quickly glance on the new user before reverting attention.
}

Gestures

Gestures are a key element to making an interaction lively and is done by the gesture() method. Gestures are by default synchronous/blocking but you can choose to have them asynchronous/non-blocking with the async = true parameter. Furhat comes with a library of basic gestures (In the Gestures class). Examples:

furhat.gesture(Gestures.Smile) // Do a smile

furhat.gesture(Gestures.ExpressAnger, async = false) // Express anger but continue execution immediately

You can also define your own gestures easily directly in Kotlin, preferably in an own gestures-file.

Random

You can add random behaviour to the flow by using the random method

random (
    { say("Would you like to play a game with me?") },
    { say("Are you up for some game?") }
)