In this tutorial, you will learn the basics of how to give Furhat eyes by adding visual sensors and creating a situation model to be able to attend users in his interaction space.

Note: this tutorial assumes that you:

This tutorial will benefit from having a Microsoft Kinect sensor (V1 or V2) but does not depend on it.

We will start using the same template as in the previous tutorial, Presentation.

The situation and interaction space

Start by running the skill and go to the Dashboard in the left menu. Click "Show situation" and you will see the situation view:


In the GUI you see the situation from a top-down view, with the system agent in the center. There is a large grey circle around the system agent, which indicates the agent's interaction space. Only users that are within this space will be considered interaction partners.

You can now simulate that a user is entering the interaction in front of the agent by adding a virtual user.

If you have a Kinect, you can instead move in front of the sensor and your user should show on the situation view.

SDK users: the extended situation simulator

If you are running the SDK you have an extended situation simulator that you can start from the terminal window opened on server start. Press "Open Situation manager" to open the situation simulator.

  1. Double click on either situation panel to create a user.
  2. Move the user in-and out of the grey interaction space to simulate sense.user.enter and sense.user.leave events.


Making Furhat react to users in his interaction space

Let's say we want Furhat greet a user once he enters his interaction space, and subsequently say goodbye when a users leaves.

We will do that by modifying our state to look like this:

<state id="Start">
      <if cond="system.hasUsers()">
        <agent:say>Hi again!</agent:say>

  <onevent name="sense.user.enter">
      <agent:attend target="event:user"/>
      <agent:gesture name="'smile'"/>
      <agent:say>Hi there!</agent:say>

  <onevent name="sense.user.leave">
      <agent:gesture name="'express_sad'"/>

As you see, we have now added two more event listeners. As well as the <onentry>, we now have two <onevent> with different event names.

On entry, if there already are users in Furhat's interaction space he will attend a random user and say "hi again". If there are no users he will attend no one, returning to his default position.

On the sense.user.enter event, sent whenever a user enters Furhat's interaction space, Furhat will attend that user (identified though the user parameter of the event using event:user), smile, and greet the user.

On the sense.user.leave event, sent when a user leaves the space, Furhat will express sadness, say goodbye and then reenter the flow. Once he reenters he will yet again check if there are any remaining users and if so say "hi again" to them.

Try out the flow by toggling virtual users in the Dashboard view!