An interactive FruitSeller skill

Introduction

In this tutorial, you will learn how to build a basic interactive skill for Furhat - a fruit seller. If you want to jump ahead to a working full code-example, please see the full example on Github.

This tutorial assumes that you:

The goal of the skill

The idea of this example skill is to build a fruit selling skill, where Furhat has a fruit stand and offers passing-by humans the option of buying any fruit in stock. Not all humans passing-by might be interested in buying fruits. Fulfillment will not be considered in this tutorials, i.e. no fruits will come to harm.

Creating and importing the skill into your IDE

Follow the same principles as in the previous tutorial to create a new skill (called "FruitSeller") and import it into IntelliJ IDEA.

Note: Note that you can, if you prefer, instead download and import the complete example skill from Github. Instructions on importing are available in the repo's readme, or in the above link to the first tutorial.

Some requirements for the skills

To set the bar for our skill, we settle on the following requirements on the skill based on the overall goal above:

  • A user should be able to enter the interaction space to start the interaction.
  • The interaction should start with Furhat attending to and greeting the user, and asking if the user desire some fruits.
  • The user should be able to ask what products or services are available.
  • The user should be able to answer yes, and be queried a follow-up question of what variation of the product or service (as well as quantity if it makes sense) he/she wants, or directly answer what variation and/or quantity he/she wants.
  • Furhat should confirm the order.
  • The interaction should end with Furhat thanking the user and then waiting for another user.

Interaction triggers

As walked through in the previous tutorial, the skill template contains boilerplate handlers for interaction triggers based on users entering and leaving the robot's interaction space. This will be sufficient also for our fruit seller. Note, if you want to expand the skill to, for example, be able to handle multiple customers at the same time you're going to have to modify these.

Thus, we can focus on the active part of the interaction, starting with the Start state.

Greeting the user

We will start by deleting the default handlers in the Start state any replacing it with a greeting. We will add some variation already here since we know that repetitive robots are the worst. Finally, we go to an order-taking state.

val Start = state(Interaction) {
  onEntry {
    random(
      {   furhat.say("Hi there") },
      {   furhat.say("Oh, hello there") }
    )

    goto(TakingOrder)
  }
}

Taking an order

Next, we want to ask the user if he/she might be interested in our delicious, fresh assortment of fruits. We do a simple furhat.ask() and then add handlers for a Yes and a No.

val TakingOrder = state {
  onEntry {
    random(
      { furhat.ask("How about some fruits?") },
      { furhat.ask("Do you want some fruits?") }
    )
  }

  onResponse<Yes> {
    random(
      { furhat.ask("What kind of fruit do you want?") },
      { furhat.ask("What type of fruit?") }
    )
  }

  onResponse<No> {
    furhat.say("Okay, that's a shame. Have a splendid day!")
    goto(Idle)
  }
}

If the user answers yes, we ask a follow-up question (that we are currently not handling) of what fruits they want, if we get a no we simply thank them and go back to Idle.

An intuitive thing for the user to reply to the initial question "How about some fruits?" is to reply with for example "yes, I'd like an apple". This would then also be a natural response to the followup-question "What kind of fruit do you want". Let's add a handler for our first custom intent BuyFruit:

Note: Kotlin's neat String interpolation is used here to include our variable in the spoken string

onResponse<BuyFruit> {
  furhat.say("${it.intent.fruit}, what a lovely choice!")
}

Now, we have to teach our skill what a BuyFruit intent looks like. We open the nlu.kt file and add a new (Enumeration) Entity called Fruit and a new Intent called BuyFruit:

// Our Fruit entity.
class Fruit : EnumEntity(stemming = true, speechRecPhrases = true) {
  override fun getEnum(lang: Language): List<String> {
    return listOf("banana", "orange", "apple", "cherimoya")
  }
}

// Our BuyFruit intent
class BuyFruit(val fruit : Fruit? = null) : Intent() {
  override fun getExamples(lang: Language): List<String> {
      return listOf("@fruit", "I want a banana", "I would like an apple", "I want to buy a @fruit")
  }
}

The Fruit entity is now an enumeration of all strings that correspond to fruits. For a full application, you probably want to add a longer list than these 4 examples - even fruits that you don't have in stock since you might want to explain this to a visitor. For more info, see Entities in the NLU docs.

The BuyFruit intent is now an intent that contains examples of different ways a user might try to buy a fruit. Machine learning is used behind the scene to determine if what the user says matches these examples and generally speaking, the more examples you add the better performance you will see. This is one of the most time-consuming parts of building a Furhat skill since extensive testing with users is needed to build a rich enough model to catch all variance different users will throw at your skill. For more info, see Intents in the NLU docs.

A few things to note:

  • In our Fruit entity, we are using parameters stemming = true and speechRecPhrases = true. Stemming makes the natural language processor "stem" the words to allow various forms of the word to be seen as the same. For example, "banana" and "bananas" will both match "banana". speechRecPhrases is a way to tip the recognizer that certain words are likely to appear. For a exotic word like "cherimoya", this will increase the likelyhood of a match.

Note: As your skill and entity list of examples grow, you might not want to pass on all phrases to the recognizer due to performance reasons and the fact that most recognizers have a limit. But for now, you don't have to worry about this.

  • Our intent has a variable of our entity type Fruit called fruit. This variable is then used in the required getExamples() method that you use to teach the system different ways users might say something intending them to buy a fruit. You can also, however, use any of the Entity values instead of this variable. Now, if you go back to the onResponse handler, you will see how this variable can be accessed in the flow directly (and typed!):
onResponse<BuyFruit> {
  furhat.say("${it.intent.fruit}, what a lovely choice!")
}
  • As you see, the getExamples() method of Intents has a language parameter. This can be used at a later point when you might want to support several user input languages with the same intent. For now, you don't have to worry about it.

Once you have added the new onResponse handler to your TakingOrder state, we recommend you to test-run the skill to make sure it works as intended.

Telling users what fruits are available

As per our requirements, we want to be able to answer users asking what fruits we have in stock. To do this, we define the following intent RequestOptions.

class RequestOptions: Intent() {
  override fun getExamples(lang: Language): List<String> {
    return listOf("What options do you have?",
            "What fruits do you have?",
            "What are the alternatives?",
            "What do you have?")
  }
}

To answer this, we'll create an instance of our Fruit() entity and then use the optionsToText() method to concatinate the options to a String. We'll reply with this listing and then ask if the user want's some fruits.

onResponse<RequestOptions> {
  furhat.say("We have ${Fruit().optionsToText()}")
  furhat.ask("Do you want some?")
}

Allowing users to buy several types as well as specified quantities of fruits.

Next up, we might want to allow users to buy several fruits at once as well as specifying how much of each fruit they want. For this, we add two new entities; FruitList and QuantifiedFruit:

class FruitList : ListEntity<QuantifiedFruit>()

class QuantifiedFruit(
  val count : Number? = Number(1),
  val fruit : Fruit? = null) : ComplexEnumEntity() {

  override fun getEnum(lang: Language): List<String> {
      return listOf("@count @fruit", "@fruit")
  }

  override fun toText(): String {
    return generate("$count $fruit")
  }
}

FruitList should be pretty straight-forward, it's an entity of type ListEntity, which in turn is of type QuantifiedFruit.

QuantifiedFruit is of type ComplexEnumEntity, meaning it has more functionality that EnumEntity lacks. This is for example the ability to contain other entities, in this case a Number called count and a Fruit called fruit (the latter you should remember from our our BuyFruit intent). An important note here is that "@fruit" is one of the examples - i.e QuantifiedFruit also supports utterances without a quantity, like "apple". Since our application will assume all Quantified fruits have a quantity however, we set the default value of count to 1 since if you say "apple" you likely mean "one apple". Since we want to be able to confirm our orders repeating by "one banana" even if the user says only "banana", we implement the toText() method as above.

We then update our BuyFruit intent to take in a FruitList instead of a Fruit:

class BuyFruit(var fruits : FruitList? = null) : Intent() {
  override fun getExamples(lang: Language): List<String> {
      return listOf("@fruits", "I want @fruits", "I would like @fruits", "I want to buy @fruits")
  }
}

Finally, we have to adjust our response-handler in the flow:

onResponse<BuyFruit> {
  furhat.say("${it.intent.fruits}, what a lovely choice!")
}

Storing a user's order on the user object

To keep track of what user ordered what fruit, it makes sense to save this information somewhere. To save it on the user-model, we open users.kt and add the following:

class FruitData (
  var fruits : FruitList = FruitList()
)

val User.order : FruitData
  get() = data.getOrPut(FruitData::class.qualifiedName, FruitData())

From the top, we define a Kotlin data class FruitData with a variable fruits of type FruitList instantized as an empty FruitList. Then, we add an extention variable order to the User model of type FruitData, with a custom getter function that goes beyond the scope of this tutorial. Please see the Kotlin documentation for more information.

This then allows us to save ordered fruits to users as follows:

onResponse<BuyFruit> {
  val fruits = it.intent.fruits
  furhat.say("${fruits.text}, what a lovely choice!")
  fruits.list.forEach {
    users.current.order.fruits.list.add(it)
  }
}

Allowing follow-up orders

Now that we can store fruits to users, the next challenge is to allow users to "top up" an order, for example adding an apple to an existing order of three bananas.

To prepare for this, we will first move the order handling from the TakingOrder state to an OrderReceived state. The reason for this is that we need to have a different handling of fruit orders and yes/no intents.

Our TakingOrder, after modification, then looks like this:

val TakingOrder = state {

  /*
  ...
  */

  onResponse<BuyFruit> {
    val fruits = it.intent.fruits
    if (fruits != null) {
        goto(OrderReceived(fruits))
    }
    else {
      propagate()
    }
  }
}

The only difference being that we go to the OrderReceived state and pass along the fruits list we received in the response. This is a functional way of programming, you could also use a global variable to keep track of the FruitList between states.

Next up, the new state to handle orders received:

fun OrderReceived(fruits: FruitList) : State = state {
  onEntry {
    furhat.say("${fruits.text}, what a lovely choice!")
    fruits.list.forEach {
      users.current.order.fruits.list.add(it)
    }
    furhat.ask("Anything else?")
  }

  onReentry {
    furhat.ask("Did you want something else?")
  }

  onResponse<BuyFruit> {
    val fruits = it.intent.fruits
    if (fruits != null) {
        goto(OrderReceived(fruits))
    }
    else {
        propagate()
    }
  }

  onResponse<RequestOptions> {
    furhat.say("We have ${Fruit().getEnum(Language.ENGLISH_US).joinToString(", ")}")
    furhat.ask("Do you want some?")
  }

  onResponse<Yes> {
    random(
      { furhat.ask("What kind of fruit do you want?") },
      { furhat.ask("What type of fruit?") }
    )
  }

  onResponse<No> {
    furhat.say("Okay, here is your order of ${users.current.order.fruits}. Have a great day!")
    goto(Idle)
  }
}

Here, we define the state as a function (fun) instead of a variable (val) and adding a parameter fruitList to be received.

Note for the curious, it's possible in Kotlin to define default values for parameters to be able to call states defined as function without any parameter.

Then, we ask "Anything else?" after which we list a bunch of handlers, many of them identical to handlers in our TakingOrder state. We see that the only one different is the handler for the No-intent, as we here want a different behavior - to confirm the existing order instead of saying goodbye. In addition, there is a onReentry handler. The purpose of this reentry-handler is to not repeat the order confirmation if a reentry is done (this will be done for example if the user says something that the system cannot pick up through the implicit dialog state handling default answers - see docs here).

What is appropriate in cases like this, is to create a parent state that contains common handlers shared among state. We create the following Options state with the common handlers:

val Options = state(Interaction) {
  onResponse<BuyFruit> {
    val fruits = it.intent.fruits
    if (fruits != null) {
        goto(OrderReceived(fruits))
    }
    else {
        propagate()
    }
  }

  onResponse<RequestOptions> {
    furhat.say("We have ${Fruit().getEnum(Language.ENGLISH_US).joinToString(", ")}")
    furhat.ask("Do you want some?")
  }

  onResponse<Yes> {
    random(
      { furhat.ask("What kind of fruit do you want?") },
      { furhat.ask("What type of fruit?") }
    )
  }
}

This state is then inherited by our TakingOrder and OrderReceived states (note the named and unnamed ways of defining inheritance) which can be slimmed down to the following:

val TakingOrder = state(parent = Options) {
  onEntry {
    random(
        { furhat.ask("How about some fruits?") },
        { furhat.ask("Do you want some fruits?") }
    )
  }

  onResponse<No> {
    furhat.say("Okay, that's a shame. Have a splendid day!")
    goto(Idle)
  }
}

fun OrderReceived(fruitList: FruitList) : State = state(Options) {
  onEntry {
    furhat.say("${fruitList.text}, what a lovely choice!")
    fruitList.list.forEach {
      users.current.order.fruits.list.add(it)
    }
    furhat.ask("Anything else?")
  }

  onReentry {
      furhat.ask("Did you want something else?")
  }

  onResponse<No> {
      furhat.say("Okay, here is your order of ${users.current.order.fruits}. Have a great day!")
      goto(Idle)
  }
}

This concludes this first interactive skill. For a working full code-example, please see FruitSeller on Github.