Flow and States

The flow (short for dialog flow) in FurhatOS is a state chart implementation with additional functionality such as inheritance, global variables and dynamic states (states defined as functions); this makes it powerful for reusing code and building complex interactions.

Note: This page focuses on states, event-handlers and transitions between states. For all actions you can do inside of states, see Speech, Gestures, User management and attention, response handling and NLU.

Defining states

The State is the fundamental building block of the flow. The flow is always in one particular state and makes transitions between them. The state defines triggers, which in turn contain actions to be performed (including state transitions).

States are defined like this (in Kotlin):

// Normal state defined as immutable variable (val)
val MyState = state {
    onEntry {
        furhat.say("hello world")
    }
    // Other handlers
}  

// State with parameters defined as function (fun)
fun MyDynamicState(text: String) = state {
  onEntry {
    furhat.say("hello $text") // Kotlin string interpolation
  }
}

Important note: in the current Kotlin version, sometimes you get a recursive error when you have loops of transitions. In this case you might have to explicitly define one of the states as a State like this: val MyState : State = state { ... }

Triggers

The flow can define different types of triggers, described below:

onEntry

Trigger that is executed everytime the flow transitions to the state. If onReentry is defined, onEntry will not be executed on reentry().

onReentry

Trigger that is executed only on reentry(), and if so, prevents onEntry from being called.

onExit

Trigger that is executed when a transition is made to another state (see below).

onEvent

Trigger catching events. To only catch a certain event type, you do like this:

onEvent<MySpecialEvent> {
  // Do something when this event is caught.
  // You can access event variables through the implicit "it" variable.
  // Note that "it" is automatically typed to MySpecialEvent, so you IDE will know available event parameters.
  println(it.myParameter)
}

or with the name of the event:

onEvent("MySpecialEvent") {
  // Do something. In this case your IDE will not know the type of the "it" variable.
}

init

A special handler that is executed only once for each state (type). It is especially useful in initial states to set variables, authenticate to APIs, etc.

val MyState = state {
    init {
        users.setMaxUsers(3)
        furhat.voice = myVoice
        furhat.setTexture("female")
    }

  }

When the state is called for the first time, after init completes executing, onEntry will be triggered right-after.

onTime

This trigger creates a timer:

val MyState = state {

    // Creates a timer that is repeatedly triggered with a random interval between 1000 and 3000 milliseconds.
    onTime(repeat=1000..3000) {
        furhat.gesture(Gestures.Smile)
    }

    // Creates a timer that is triggered 10000 milliseconds after the state was entered.  
    onTime(delay=10000) {
        furhat.say("Now 10 seconds have passed")
    }

}

The parameters repeat and delay can be combined, in which case the repetition is started after the initial delay. The timers are automatically cancelled when the state is exited.

onUserEnter, onUserLeave

Triggered when a user enters or leaves the interaction. See User management and Attention.

onUserGesture, onUserGestureEnd

Triggered when a user shows a facial gesture, and stops showing that facial gesture. See User Gesture Detection.

onResponse

Triggered when the user says something. See Listening docs.

onPartialResponse

Triggered when the user says something that is classified as two different intents. See Listening docs.

onNetworkFailed

Triggered when during speech synthesis the network connection drops, for failing ASR see Listening docs. Can be used in a flow like this:

onNetworkFailed {
  //Triggers when speech synthesis fails and network connection has been dropped.
}

Trigger conditions and event propagation

Triggers are checked in the order they are defined. When a trigger is executed, no more triggers will be checked for the event. You can add a condition to the trigger, so that it is only executed when this condition holds, otherwise the event is checked against other triggers.

onEvent<MySpecialEvent>(cond = { it.count > 2 }) {
  // Do something
}
onEvent<MySpecialEvent> {
  // This trigger is only executed if the condition above evaluates to false.
}

Another possibility is to catch an event and then explicitly propagate it, if the trigger is not supposed to handle the event:

onEvent<MySpecialEvent> {
    if (it.count > 2) {
        // Do something
    } else {
        propagate()
    }
}
onEvent<MySpecialEvent> {
  // This trigger is only executed if the condition above evaluates to false.
}

State inheritance

One key feature to building scalable applications is to use inheritance to group common functionality. Following this principle, states in Furhat application development can inherit/exted other states.

Any state can be extended by another state by using defining your state as state(parent = MyParentState) { /* ... */ }. Any non-caught event will be automatically propagated to parent states. By default, Furhat applications comes with a dialog state that all states inherit. Any uncaught response events will be picked up here.

Full example, also showing propagation of a response event:

val MyOtherState = state(parent = MyParentState) {
    onEntry {
        furhat.ask("What happens now then?")
    }

    onResponse<MyIntent> {
        println("Caught my first intent")
    }
}

val MyParentState = state {
  onResponse<MyOtherIntent> {
    println("Caught my other intent")
  }

  onResponse {
    println("Caught response not matching any of my intents")
    // Do something else
    propagate() // Manually propagate to default Dialog state
  }
}

By default onEntry, onReentry and onExit are not inherited. You can include them by setting the inherit parameter. 

val Parent: State = state() {
    onEntry(inherit = true) {
        furhat.say("Say this when the child state does not specify anything on entry")
    }
}

Note 1: If * denotes a priority flag set to true, the inheritance order would be grandparent* > parent* > child* > child > parent > grandparent.

Note 2: This inheritance will not execute all onEntry, onReentry or onExit at once. To have a persistent inheritance that would execute everytime, you would need to add priority and propagation to the parent state as well:

val Parent: State = state() {
    onEntry(inherit = true, priority = true) {
        println("This line will be printed every time a child state is entered, before the child onEntry")
        propagate()
    }
}

val Child: State = state(Parent) {
    onEntry {
        //Child action executed after the parent
    }
}

Partial states and extending states

In addition to having states inheriting other states, you can reuse triggers/handlers by using including partial states in your state. The difference between including a state and inheriting the same is marginal for most use-cases - but in general, the biggest difference is that you can never transition to a partial state, as opposed to a state. As a guide, it usually feels intuitive to use inheritance that maps to a dialog as it would be visualized in a dialog tree, whereas you would use include(partialState { ... }) to include blocks of functionality that you want to use in several states. An example would be wizard-button (onButton()) groups.

Example use as followed:

val wizardButtons = partialState {
    onButton("My button") {
        // Do something
    }
}

val myExternalOnEntry = partialState {
    onEntry {
        furhat.say("this is taking precedence")
    }
}

val MyState = state {
   include(myExternalOnEntry)

   onEntry {
       // This will not be said
   }

   include(wizardButtons)
}

State transitions (goto)

You can transition between states with the goto action. When a goto is issued, all remaining actions in the trigger are ignored.

val MyState = state {
  onEntry {
    println("Entering MyState")
    goto(MyOtherState)
  }
  onExit {
    println("Leaving MyState")
  }
}

val MyOtherState = state {
  onEntry {
    println("Entering MyOtherState")
  }
}

Running MyState, the output will be:

Entering MyState
Leaving MyState
Entering MyOtherState

State transitions (Call)

Another type of transition is to call a state. The difference from goto is that the called state can return (using terminate()), in which case the calling state will resume the execution of remaining actions in the trigger.

Example:

val MyState = state {
  onEntry {
    println("Entering MyState")
    call(MyCalledState)
    goto(MyOtherState)
  }
  onExit {
    println("Leaving MyState")
  }
}

val MyCalledState = state {
  onEntry {
    println("Entering MyCalledState")
    terminate()
  }
  onExit {
    println("Leaving MyCalledState")
  }
}

val MyOtherState = state {
  onEntry {
    println("Entering MyOtherState")
  }
}

Final output will be:

Entering MyState
Entering MyCalledState
Leaving MyCalledState
Leaving MyState
Entering MyOtherState

Many of the built-in actions that Furhat can do are in fact making a call to another state:

val MyState = state {
  onEntry {
    furhat.say("Hi there")
    furhat.ask("What is your name?")
  }
}

In this example, furhat.say() issues a call to a state. When Furhat is done saying "Hi there", that state returns, and furhat.ask issues a call to another state.

Triggers when calling states

When the flow is a called state, events will be checked against:

  1. The called state's triggers
  2. The called state's parent's (and grand-parent states) triggers
  3. The calling state's triggers
  4. The calling state's parent's (and grand-parent states) triggers
  5. ... (and so on, if the calling state has a calling state)

Important note: If a trigger in a calling state is triggered, the called state will automatically terminate before the trigger is executed. If you want to avoid this, the trigger must be defined with the parameter (instant=true), which says that the trigger is guaranteed to run instantaneously. Such a trigger is not allowed to make any calls.

Here is an example:

val Interaction = state {
    onUserEnter(instant=true) {
        furhat.glance(it)
    }
}

val Interaction = state(parent=Interaction) {
    onEntry {
        furhat.ask("How may I help you?")
    }
}

A key insight here is that when Furhat is asking a user How my I help you?, the flow is in a called state. Now, if a second user enters the interaction, Furhat glances at that user. Since the trigger onUserEnter is marked as (instant=true), Furhat does not abort the question he is asking, but simply continues. This works since furhat.glance() is called asynchronously per default, and therefore does not call another state.

Two important implications of this behavior are:

  1. If you declare a trigger as instant but include in it any call (including furhat.say(), furhat.ask(), furhat.listen() or any other method call with async = false - for example furhat.gesture(Gestures.SMILE, async = false)) these calls will be ignored and a warning issued if the trigger is executed.
  2. If you declare a trigger as instant but include in it other blocking methods, for example API-calls, the flow will be rendered unresponsive until the blocking methods are finished if the trigger is executed.

Returning values from called state

It is also possible for a called state to return a value:

val MyState = state {
  onEntry {
    val result = call(PlusState(5, 3)) as Int
  }
}

fun PlusState(a : Int, b : Int) = state {
  onEntry {
    terminate(a + b)
  }
}

Calling anonymous states

You should never call a method that is expected to take longer time in the main thread of the flow, since this will make the flow unresponsive. If you have to do that, for example to make an API call that is expected to take time, it should be done in a called state. There is a convenient way of doing this in an anonymous state:

val MyState = state {
  onEntry {
    val result = call {
        myAPI.getResult()
    }
  }
  onTime(delay=1000) {
       // 1000 ms passed, we aborted the API call since we did not provide the instant = true flag.
  }
}

Important note: The onTime handler will abort the call if it hasn't returned since it does not include an instant = true flag (See Triggers when calling states)

Pausing the flow execution (delay)

You can easily pause the flow execution with the delay method:

val MyState = state {
  onEntry {
    furhat.say("One")
    delay(1000)
    furhat.say("Two")
  }
}

Behind the scenes, the delay() method calls a state that implements the pause, which means that the flow is still responsive while pausing i.e., the pause can be aborted if an event is triggered and will be so if the trigger is not instant (instant = true, See Triggers when calling states).

Raising and sending events

You can easily define your own events:

class MyCustomEvent(val text : String) : Event()

You can also create events without any parameters inline with send("MyEventName") and raise("MyOtherEvent").

val MyState = state {
  onEntry {
       send("MyEventName")
     raise("MyOtherEvent")
  }

  onEvent("MyEventName") {
       println("Got it (and the rest of the system got it as-well)")
  }

  onEvent("MyOtherEvent") {
       println("Got it (and the rest of the system won't)")
  }
}

val MyCalledState = state {
  onEntry {
    raise(MyCustomEvent("Hello"))
    println("That was cute")
  }
}

Note: if you want to catch an event (that you defined as a class) in an external system, for example a skill GUI you have to use the full class name (for example furhatos.app.MySkillName.MyEventName) unless you explicitly set the event name with class MyCustomEvent(val text : String) : Event("MyCustomEventName").

You can either send or raise such events. Send means that the event is sent to the rest of the system. Raise means that it is only being sent within the flow. This is useful, for example to let a called state communicate with the calling state:

val MyState = state {
  onEntry {
       call(MyCalledState)
  }

  onEvent<MyCustomEvent>(instant = true) {
       println("The called state said " + it.text)
  }
}
val MyCalledState = state {
  onEntry {
    raise(MyCustomEvent("Hello"))
    println("That was cute")
  }
}

which would print

  The called state said hello
  That was cute

Raising responses

Similar to how you can raise events, you can also raise responses. This is documented under the NLU docs since this is mainly a tool for capturing several intents.

From a flow-perspective though, this allows you to pass responses between states.


val myState = state {
  onEntry {
      furhat.listen()
  }

  onResponse<MyIntent> {
      goto(myOtherState(it))
  }
}

fun myOtherState(response : Response<*>) = state {
    onEntry {
        raise(response)
    }

    onResponse<MyIntent> {
        furhat.say("I'm handling it here")
    }
}

Adding variations to flow (random)

You can add random to flows by using the random() method and enclosing your method with curly brackets (needed since otherwise the method would run instantly instead of on call)

random (
    { furhat.say("Would you like to play a game with me?") },
    { furhat.say("Are you up for some game?") },
    { furhat.gesture(Gestures.SMILE) }
)

Adding custom methods to flow

Once you start building your skill, you might find yourself wanting to reuse some method across states. You can define any method in Kotlin in the global scope but if you want to access the furhat and users objects to for example do a say with furhat.say() or check the current user with users.current you have to instead create extention methods for the underlying classes Furhat or FlowControlRunner:

fun myReusableMethod() {
    // This can be accessed from any state...
    println("I can do some things here")
}

fun Furhat.myFurhatMethod() {
  /*
    ... and so can this, but here you can do furhat actions as well. Note, you don't need to use the furhat object but instead use the methods directly since the scope of the method is in furhat already.
  */
  say("I can say things")
}

fun FlowControlRunner.myOtherMethod() {
    /*
      ... and so can this, but here you can do everything you could do in the flow such as user querys and flow actions like goto as well.
    */
    val user = users.random
    furhat.attend(user)
    furhat.say("I see you ${user.name}")
    goto(GreetUser(user))
}

val myState : State = state {
    onEntry {
        myReusableMethod()
        myOtherMethod()
    }
}

Parallel flows

You can also run actions in a separate thread, using the parallel command:

val MyState = state {
    onEntry {
        parallel {
            // The parallel flow will transition to state A,
            // but the main flow will still be in MyState
            goto(A)
        }
    }
    onTime(repeat=1500) {
        println("Still in MyState")
    }
}
val A = state {
    onEntry {
        println("A")
        delay(1000)
        goto(B)
    }
}
val B = state {
    onEntry {
        println("B")
        delay(1000)
        goto(A)
    }
}

The parallel flow will print "A" and "B" in an interval of 1000 ms. The main flow will print "Still in MyState" every 1500 ms.

The parallel flow will terminate if MyState is exited. To avoid this, you can provide abortOnExit=false to the parallel command.