Gestures

Performing gestures

Gestures are a key element to making an interaction lively and is done by the gesture() method. Gestures are by default synchronous/blocking but you can choose to have them asynchronous/non-blocking with the async = true parameter. Furhat comes with a library of basic gestures (In the Gestures class). Examples:

furhat.gesture(Gestures.Smile) // Do a smile

furhat.gesture(Gestures.ExpressAnger, async = false) // Express anger but continue execution immediately

Defining gestures

You can define your own gestures in kotlin like this:

val MySmile = defineGesture("MySmile") {
    frame(0.32, 0.72) {
        SMILE_CLOSED to 0.5
    }
    frame(0.2, 0.72){
        BROW_UP_LEFT to 1.0
        BROW_UP_RIGHT to 1.0
    }
    frame(0.16, 0.72){
        BLINK_LEFT to 0.1
        BLINK_RIGHT to 0.1
    }
    reset(1.04)
}

As can be seen, the gesture contains a list of key frames. These are points in time (defined in seconds from the start of the gesture), where certain parameter values should be reached. Each frame contains a list of parameters and the value they should reach.

As can be seen, it is also possible to define several points in time for a frame. This is typically used to sustain a certain parameter setting. In the example above, the SMILE_CLOSED will reach the value 0.5 after 0.32 seconds, and then stay at this value until 0.72 seconds after the gesture started.

To define the end of the gesture, you typically end the list of frames with a reset(), specified with the number of seconds after the gestures starts (in this example, the gesture would be 1.04 seconds long). The reset() command restores all parameters affected by the gesture to their default values. If you do not provide a reset() command, the parameters will stay at the last value until another gesture will affect them (i.e., the gesture will be sustained indefinitely).

The gesture can then be used like this in the flow:

furhat.gesture(MySmile)

Persistent gestures and priority

Normally a gesture ends after the last frame. You can however make it persist using the flag persist=true on the last frame. This is for example the case for the built-in gesture CloseEyes:

val CloseEyes = defineGesture("CloseEyes") {
    frame(0.4, persist = true) {
        BLINK_RIGHT to 1.0
        BLINK_LEFT to 1.0
    }
}

However, when a new gesture is called that involves the same parameters, these parameters will be overridden. So, if for example a Blink gesture is performed after the CloseEyes gestures, Furhat will blink and then keep the eyes open. To specify which gesture should have precedence, you can call it with a priority parameter:

furhat.gesture(CloseEyes, priority=10)

The default priority for all gestures is 0, so this would mean that Furhat would not blink anymore after performing a CloseEyes gesture with a higher priority. To make Furhat open the eyes again, you could call the gesture OpenEyes with a priority that is higher or equals to the one used to close the eyes:

furhat.gesture(OpenEyes, priority = 10)

Since the OpenEyes gesture does not end with a persistent frame, the parameters BLINK_LEFT and BLINK_RIGHT are now free, which means that Furhat will start blinking again:

val OpenEyes = defineGesture("OpenEyes") {
    frame(0.4) {
        BLINK_RIGHT to 0.0
        BLINK_LEFT to 0.0
    }
}

Reactive gestures

Per default, Furhat reacts to certain events with gestures:

  • When "prominence" is detected in the synthesized speech (i.e., a part which is stressed), Furhat raises the eyebrows (Gestures.BrowRaise).
  • When the user starts to speak (while the system is listening), Furhat smiles (Gestures.Smile).

You can change or turn off these behaviours like this:

// Note that you have to make the following import for these parameters to be accessible:
import furhatos.autobehavior.*

// ...

// Randomly perform one of the provided gestures at prominence:
furhat.prominenceGesture = listOf(Gestures.BrowFrown, Gestures.Thoughtful)

// Make no gesture at speech start (empty list):
furhat.userSpeechStartGesture = listOf()

Microexpressions

Microexpressions are small facial expressions that run continuously to make Furhat appear more "alive".

By default, Furhat has a set of pre-defined microexpressions, but you can configure these from the skill. If you want to switch on or off the basic parameters, you can just do like this:

furhat.setDefaultMicroexpression(blinking = true, facialMovements= true, eyeMovements = false)

The basic parameters are:

  • blinking: Furhat blinks
  • facialMovements: Small movements in the face around the mouth and eyes
  • eyeMovements: The eyes gaze continuously shifts a little bit, similar to small saccades.

If you want more detailed control, you can define your own microexpressions:

furhat.setMicroexpression(
    defineMicroexpression {
        // Fluctuade facial movements. First parameter is frequency, second amplitude, third adjustment.
        fluctuate(0.025, 0.06, 0.12, BasicParams.BROW_UP_LEFT, BasicParams.BROW_UP_RIGHT);
        fluctuate(0.025, 0.2, 0.5, BasicParams.SMILE_CLOSED);
        fluctuate(0.025, 0.2, 0.5, BasicParams.EXPR_SAD);
        // Adjust eye gaze randomly (between -3 and 3 degrees) with a random interval of 200-400 ms.
        repeat(200..400) {
            adjust(-3.0..3.0, BasicParams.GAZE_PAN)
            adjust(-3.0..3.0, BasicParams.GAZE_TILT)
        }
        // Blinking with a random interval of 2-8 seconds
        repeat(2000..8000, Gestures.Blink)
    })