User management

Users are detected by the camera and are assigned a unique id which is kept constant as long as the user is in view. If the camera looses the user for more than a brief period, the user is likely to be assigned a new id, and will therefore be treated as a new user. For now, Furhat doesn't have a memory of past users.

In the skill, a UserManager keeps track of users and raises events related to the status of the users. The User manager can be accessed in the flow through the userManager property.

Engaged users

Just because Furhat can see a user, it doesn't mean that the user is interested in talking to Furhat. Therefore, the system keeps track of which users are engaged (i.e., engaged in the interaction). To decide which users are considered to be engaged, the system uses different engagement policies.

If nothing else is specified in the skill, the system uses a SimpleEngagementPolicy. This policy says that when users enter an inner interaction space (1 meter from the robot per default), they will be considered to be engaged. To add robustness, a larger interaction space (0.5 meters outside the inner space per default) is used to trigger users leaving the interaction (becoming disengaged). In the default setting, the number of engaged users is also limited to 2.

To change the parameters of the SimpleEngagementPolicy, you do it easily in your initial state:

val Idle : State = state {

    init {
        // Set the inner space to 2.0 meters (outer distance becomes 2.5) and maximum users to 3
        users.setSimpleEngagementPolicy(2.0, 3)
    }

}

Users entering and leaving

When a user becomes engaged (i.e., enters the interaction space), you will get a SenseUserEnter event, and when a user becomes disengaged, a SenseUserLeave event is sent. These can be caught in the flow like this:

val Idle = state {
  onEntry {
      furhat.attendNobody()
  }

  onUserEnter {
      furhat.attend(it)
      goto(Attending)
  }
}

val Attending = state {
  onEntry {
      furhat.say("Hello there")
  }

  onUserLeave {
      furhat.say("Goodbye")
      goto(Idle)
  }
}

As can be seen, the user entering can be accessed through the it variable.

Accessing users

The UserManager keeps track of the current user. This is typically the user Furhat is currently attending. When Furhat detectes speech, the default assumption is that that it is the current user who is speaking. The current user can be accessed in the flow through the users.current property in the flow. The specific getters you can use are:

  • users.current: Gets the current user. This is usually the user Furhat is attending, unless you have instructed Furhat to attend a location or Furhat is currently glancing at something.
  • users.random: Gets a random user from the set of engaged users, including the current user.
  • users.other: Gets a random user from the set of engaged users, which is not the current users.

Each of these properties return a User object.

User data

The User object contains a property called "data" (of type UserData), where the skill can store information about the user. To make this convenient, it is recommended to create an extention property in kotlin (for example in a separate file called "user.kt"):

class GameData(
        var score : Int = 0,
        var played : Boolean = false) : Record()

val User.gameData : GameData
    get() = data.getOrPut(GameData::class.qualifiedName, GameData())

This creates a field in the data property of the user object, which stores the skill specific information.

This way, you can easily store and access information with strong typing:

users.current.gameData.score++

if (users.current.gameData.played) {
    // Do something
}

Attention

Attention is an important aspect of a robot interaction, especially if it is multi-party. When an agent (user or system) is attending to (looking at) another agent, this typically means that the attended agent is supposed to respond. Thus, we need to be able to both control the attention of the system agent (the Furhat robot) in an appropriate way, and check the attention of the users.

Attending users

When you attend users, two things happen: 1. The robot will turn to the user 2. The UserManager will change the current user to the new user

Note: if you attend a location, the current user will not change

Use the furhat.attend() method to have the robot attend users. To access specific users, please see User management.

furhat.attend(user) // Attend a specific user
furhat.attend(userId) // Attend a user based on userId
furhat.attendNobody() // Reset attention

To attend users based on responses or events, use the following examples

onResponse {
  furhat.attend(it.userId) // Attend the user of an onResponse
}

onUserEnter {
  furhat.attend(it) // Attend the user of an event
}

Attending locations

You can also choose to attend a location as follows:


// attend a location at x = 1.0, y = 0.0, z = 1.0 in other words 1m right and 1m in front of the robot

val location = Location(1.0, 0.0, 1.0)
furhat.attend(location)

Glancing

In multi party interactions, sometimes you might want to acknowledge a user that you are currently not attending without changing your current user. Glance is a good way to do this.

furhat.glance(users.other) // glance on a random user that isn't the current user

furhat.glance(users.other, 2000) // glance for 2 seconds (2000 milliseconds)

The use-case might be that you are talking to a person and another user is entering the interaction, in which case you might want to glanze at the new user for a few seconds before reverting attention to the current user. It's important to remember to set the instant = true option, otherwise the trigger will interrupt any called state. For more info, see Triggers when calling states.

Example below:

onUserEnter(instant = true) {
  furhat.glance(it) // glance on the new user before reverting attention.
}

Checking Furhat's attention

The current user can be accessed with users.current.

To check if Furhat is currently attending a certain user, you can use furhat.isAttending(user). Note that the current user might not always be attended, for example if Furhat is glancing somewhere or is attending a location.

Multiparty interaction

TBA

Identification

TBA

Changing the EngagementPolicy

TBA