Graphical interfaces with Furhat skills

Graphical interfaces can be a great support for a robot interaction by listing complex information, allowing text input or showing a game interface. While you can connect any kind of application with graphical interfaces to a Furhat skill, we have made it really easy for a skill to create and host a Web GUI. With this we mean a webpage/webapp hosted locally by the skill, with an html entry, our JavaScript library and your custom JavaScript logic.

You can create three kinds of web GUIs for a Furhat skill;

  1. HostedGUI with our default template.
  2. HostedGUI with your own custom webapp.
  3. RemoteGUI where you host your webapp separately.

You can use any number of GUIs with a skill.

HostedGUI with our default templates

Note There is a known issue where, if you run your skill remotely (i.e. from your developer machine to a robot), the hosted GUI button on the robot's web interface will not give the right URL to the hosted GUI. It will think the GUI resides on <robot-ip>:port but in reality it resides at <developer-machine-ip>:port. This will be fixed in a future release. For now, you can just manually go onto <developer-machine-ip>:port in your browser to access your gui while testing on a robot.

Furhat SDK comes with pre-defined GUIs that satisfy very basic needs of displaying text on screen (Currently only this 'BASIC' template is bundled with the SDK).

To create predefined GUI in the skill flow, insert the following line:

  // GUI hosted on a random open port
  val default_gui = HostedGUI("My Default GUI")

  // GUI hosted on a specific port
  val default_gui_on_custom_port = HostedGUI("My Default GUI", port = 51234)

Before writing to the HostedGUI (here instantiated as default_gui), you need to wait for it to be connected. 

onEvent<SenseSkillGUIConnected> {
  default_gui.write("banana") // Adds an initial line of text
  default_gui.append("split") // Appends a line to already existing text
  furhat.say("Do you see banana split on the GUI?")
}

To clear the screen, you can use the following syntax:

  default_gui.clear()

The GUI will be accessible from the web-interface's dashboard page when the skill is running.

HostedGUI with your custom webpage/webapp

To create a custom GUI you need to supply a relative path to a folder containing the HTML, JS and CSS files.

To add a hosted custom GUI to your skill, use:

// GUI hosted on a random open port
val MyCustomGUI = HostedGUI("MyCustomGUI", "relative/path/to/GUI")

// GUI hosted on a specific port
val MyCustomGUIonStaticPort = HostedGUI("MyCustomGUI", "relative/path/to/GUI", port = 51234)

The GUI will be accessible from the web-interface's dashboard page when the skill is running.

For creation of a custom GUI, see below.

RemoteGUI with your custom webpage/webapp

The third option is for you to use an externally hosted GUI with your skill. External in this sense can mean it's hosted in the cloud or on some internal server that the Furhat skill has access to but it could also be your own webserver that you host in the skill (for example a Spark server or KTOR).

To add a remotely hosted GUI to your skill, use:

val MyRemoteGUI = RemoteGUI("MyRemoteGUI", hostname = "example.com", port = 1234)

If you host the GUI on your skill using some web server of your choice, you can omit the hostname which will make the hostname default to the Furhat server's public IP:

val MyRemoteGUI = RemoteGUI("MyRemoteGUI", port = 1234)

For creation of a custom GUI, see below.

Creating and communicating with a custom GUI

Creation

Now that you have defined your GUI in the flow, you need to create the GUI in itself.

To create the JS application, as suggested earlier, you can use a transpiler like babel to write the application with the furhat-gui library and use a bundler like webpack or gulp to create the bundled JS file.

We recommend either:

  • Building your own frontend, and using the furhat-gui npm package for communication. Install with npm install furhat-gui.
  • Using our tutorial skill FurGUI as a base, within the Kotlin skill framework.

Communication

The communication between the custom GUI and the skill is two-way. You can send an event from the GUI to the Skill and the vice versa.

Sending event from GUI to flow

In the boilerplate, you'll find that in the index.js file in the src/ folder, the furhat-gui library in imported and initialized by calling the Furhat() methods by passing a callback function as follows:

  import Furhat from 'furhat-gui'

  Furhat((furhat) => {
    // Methods to be executed on successful connection to the skill
  })

In the callback function you are passing to the Furhat method you can access the furhat object which gives you the ability to send events to the skill and subscribe to events from the skill. You can send events from the skill to Furhat using the following syntax inside the callback method:

  import Furhat from 'furhat-gui'

  Furhat((furhat) => {
    const INSTANCE = this;

    // This will be sent on successful connect
    furhat.send({
      event_name: 'MyEvent'
      param1: 'paramValue',
    })

    // This can be called outside, for example on a button click
    INSTANCE.sendClickEvent = (data) => {
      furhat.send({
        event_name: "ClickButton",
        param1: data.param1,
        param2: data.param2
      })
    }
  })

  // This will call the above function. Note, it will only be defined on successful connect.
  this.sendClickEvent({
    param1: "foo",
    param2: "bar"
  })

In the Furhat skill flow you can access events inside any state using the following syntax:

  onEvent("MyEvent") {
    println(it.get("param1")) // "paramValue" or "null" if param1 is not set
  }

  onEvent("ClickButton") {
    println(it.get("param2") ?: "baz") // "bar" or "baz" depending on if param2 is set
  }

Note: The above null-handling are key features of Kotlin. Read more in the Kotlin docs on null safety

Sending event from flow to GUI

As mentioned before, we can also send an event from the flow to the GUI.

If you require no parameters, you can do it inline:

  send("MyCustomEvent")

If you want to pass some data, prefer a typed experience or plan to pass the same event multiple times, you can define your own event by extending the furhatos.event.Event class:

package "furhatos.app.MySkillName"

import furhatos.event.Event

// Declare event, it needs to extend Event. The event name will be the full class name - "furhatos.app.MySkillName.MyCustomEvent"
class MyCustomEvent(
      var param1 : String? = null
) : Event()

// Declare event with a custom name - "MyOtherCustomEvent"
class MyOtherCustomEvent(
      var param1 : String? = null
) : Event("MyOtherCustomEvent")

// Send the event (inside a state)
send(MyCustomEvent("Some string"))

To receive this event (along with any speech event) in the GUI use the following syntax in the main callback:

  import Furhat from 'furhat-gui'

  Furhat((furhat) => {
    furhat.subscribe('furhatos.app.MySkillName.MyCustomEvent', (event) => {
      console.log(event.param1)
    })
    furhat.subscribe('furhatos.event.senses.SenseSpeech', (event) => {
      console.log("User said: " + event.text)
    })
    // .. add for any other event you want to receive
  })

Where furhatos.app.MySkillName.MyCustomEvent is the full class name of your custom event defined above and furhatos.event.senses.SenseSpeech is the full class name of the system-level user speech events.

Access the GUI

To access your GUIs, press the links shown on the top-bar of the web interface's Dashboard when running the skill.

Other types of GUIs

We will add support for other types of GUIs at a later point. For now, feel free to reverse-engineer the JavaScript library above to see how web-sockets can be used to communicate with Furhat's realtime API from for example iOS, Android or Unity. Contact us if you have questions on this.