A form-filling Pizza ordering skill
Introduction
In this tutorial, you will learn how to use form-filling to create a flexible order-handling skill, in this example we will be impersonating a Pizza seller. For a full code-example, please see Tutorial skills on Github.
This tutorial assumes that you:
- have completed the first tutorial and the interaction tutorial.
- have a working and running SDK
Form-filling and how to use it for order-taking
In this tutorial, we want to walk you through the design-pattern called form-filling, an efficient tool to capture a big variance of user input. This pattern can be used in a multitude of interactions where you want to allow the user to answer broadly and be able to capture additional information by filling in open "slots". To visualize this, we use our Pizza seller as an example. A few example utterances that we want to be able to capture are:
"I want a pizza with tomato and ham to my home today at 6pm"
"I would like a pizza to my office at 3 pm"
"I want a pizza"
"I want to order a pizza with bacon and ham"
With these examples as basis, we identify the slots of Entities that we need to fill in order to complete an order. This is an important principle of form-filling worth repeating; we need to have a data point from the user on each of the slots in our form. To start, we mark the Entities in our examples that we want to save in order to complete the order:
"I want a pizza with tomato (Topping) and ham (Topping) to my home (Place) today (Date) at 6pm (Time)"
"I would like a pizza to my office (Place) at 3 pm (Time)"
"I want a pizza"
"I want to order a pizza with bacon (Topping) and ham (Topping)"
Our first example shows a "perfect" order since it captures all slots we are interested in. It's very unusual however that a user does this, it's more likely that they miss out a few slots or in some cases all of them - as our other three examples show.
How form-filling works is that we first identify our intent - in this case ordering a pizza, after which we fill all data slots we can. We then move through the missing/empty slots one at a time until we have a complete order. A form-filling dialogue for our Pizza order skill might look like this:
U: I want to order a pizza with bacon and ham
F: Ok, a pizza with bacon and ham
F: Where do you want it delivered?
U: To my home
F: Ok, to home
F: When do you want it delivered?
U: At 6pm
F: Ok, at 6pm
F: You want to order a pizza with bacon and ham to your home at 6pm.
F: Is this correct?
U: Yes
...
As explained above, Furhat will here ask follow-up questions until he knows an answer for each of the slots (or more technically, for all entities in the intent).
Implementing the order-taking intent
We'll start by defining our OrderPizza intent as follows:
class OrderPizza : Intent(), TextGenerator {
var count : Number = Number(1)
var topping : ListOfTopping? = null
var deliverTo : Place? = null
var deliveryTime : Time? = null
var deliveryDate : Date? = null
override fun getExamples(lang: Language): List<String> {
return listOf(
"I would like a pizza to my office at 3 pm",
"I want a pizza tomorrow",
"I want to order a pizza with bacon and ham")
}
override fun toText(lang : Language) : String {
return generate(lang, "${if (count.value?:1>1) "${count.value} pizzas" else "a pizza"} [with $topping] [delivered $deliverTo] [$deliveryDate] [$deliveryTime]")
}
override fun toString(): String {
return toText()
}
}
Here all of the variables are defined that are necessary in order to fill out the form with the default value null so that we later on can perform null-checks to see if the variables have been assigned a value, effectively making sure that all the datapoints that are necessary for the order possess a value.
Note how we are also letting our intent implement the TextGenerator interface (see Docs)and implement a toText() method. Together with the Kotlin built-in toString() method, this allows us to get a pretty text representation of our intent based on filled in slots. We use this to repeat back the order to the user.
Here we are using some predefined entitities that are built-in to the furhat SDK and some that are not. In this example the entities Number, Time, and Date are built-in. It is important to define your own entities when developing your skills if you need to access variables from user input, as well as to improve the natural language understanding to make the dialogue as fluent as possible and avoid potential errors. See Docs for more details.
Catching the Pizza order intent and saving it to the user
To catch this intent, repeat back the order to the user, and finally saving it to the user object, we do like this:
onResponse<OrderPizza> {
furhat.say("Okay, you want ${it.intent}")
users.current.order.adjoin(it.intent)
goto(CheckOrder)
}
The method adjoin() appends all caught entities to the respective parameters in the order. Now, you might wonder where we define the order of the current user (users.current.order) as well as how the method adjoin() works. We associate the user to an order through an extention variable by using a delegate (an advanced Kotlin concept - interested developers can read more in the kotlin docs):
val User.order by NullSafeUserDataDelegate { OrderPizza() }
Adjoin is a method on the Record() class - a JSON like data class that most classes in the Furhat SDK inherits. The method joins two data objects, overwriting any existing field if a new value exists. For example, if you have the existing order:
OrderPizza {
topping: ["ham", "bacon"]
deliverTo: "home"
deliveryDate: "today"
deliveryTime: "5 pm"
}
and you adjoin it with
OrderPizza {
topping: ["tomato", "bacon"]
deliverTo: "office"
deliveryDate: "tomorrow"
}
the result will be
OrderPizza {
topping: ["tomato", "bacon"]
deliverTo: "office"
deliveryDate: "tomorrow"
deliveryTime: "5 pm"
}
In other words, all fields will be overwritten except deliveryTime which will remain since the changed order did not modify said variable.
Defining the states for form-filling
State checking the current slots of the order
The next crucial part of a form-filling design is a state checking all slots. We define it like this (with `when {}`` instead of cascading if/else as Kotlin style guide suggests):
val CheckOrder = state {
onEntry {
val order = users.current.order
when {
order.deliverTo == null -> goto(RequestDelivery)
order.deliveryTime == null -> goto(RequestTime)
order.topping == null -> goto(RequestTopping)
else -> goto(ConfirmOrder)
}
}
}
This should be fairly straightforward at this point, we simply check if we have any un-filled slots and if so go to slot-filling states where we request this data point since they need to be filled in order for an order to occur.
Our first slot-filling state
Let's define our first slot-filling state, RequestDelivery as follows:
val RequestDelivery : State = state(parent = General) {
onEntry() {
furhat.ask("Where do you want it delivered?")
}
onResponse<RequestOptions> {
raise(TellDeliveryOptions())
}
onResponse<TellPlace> {
furhat.say("Okay, ${it.intent.deliverTo}")
users.current.order.deliverTo = it.intent.deliverTo
goto(CheckOrder)
}
}
We note right away that we're inheriting a General state that we haven't defined yet. The reason is that we want to abstract some handlers that we want to use in more than one place, such as our OrderPizza intent handler.
In addition, we note that we catch the (built-in) RequestOptions intent, matching questions like "what are the options?", and then raise TellDeliveryOptions event. This is our first example handling of a context-dependent intent. When in this context, after asking where the user wants the pizza delivered, the user utterance "what are the options?" likely refers to delivery options (as opposed to for example pizza topping options). When in a broader context, this utterance likely has a different meaning. This will be cleared when we look at the parent state General below.
Thirdly, when we catch the intent TellPlace using the entity Place:
class TellPlace : Intent() {
var deliverTo : Place? = null
override fun getExamples(lang: Language): List<String> {
return listOf("home", "to my home")
}
}
class Place : EnumEntity() {
override fun getEnum(lang: Language): List<String> {
return listOf("home", "office")
}
override fun toText(lang: Language): String {
return generate(lang, "to your $value");
}
}
The place is finally saved on the user's order object after which the skill transitions back to the CheckOrder form-filling state.
Parent state for common answers
The above mentioned parent state used to abstract common answers is defined as:
val General: State = state(Interaction) {
onResponse<RequestDeliveryOptions> {
raise(TellDeliveryOptions())
}
onEvent<TellDeliveryOptions> {
furhat.say("We can deliver to your home and to your office")
reentry()
}
onResponse<RequestOpeningHours> {
raise(TellOpeningHours())
}
onEvent<TellOpeningHours> {
furhat.say("We are open between 7 am and 8 pm")
reentry()
}
onResponse<RequestToppingOptions> {
raise(TellToppingOptions())
}
onEvent<TellToppingOptions> {
furhat.say("We have " + Topping().optionsToText())
reentry()
}
onResponse<OrderPizza> {
furhat.say("Okay, you want ${it.intent}")
users.current.order.adjoin(it.intent)
goto(CheckOrder)
}
}
Here we see that for have a pair of onResponse and onEvent handlers for each type of query. Starting from the top, we have:
- Another onResponse handler raising the same TellDeliveryOptions event we did in our RequestDelivery state, this time with a more specific intent RequestDeliveryOptions defined like:
class RequestDeliveryOptions : Intent() {
override fun getExamples(lang: Language): List<String> {
return listOf("where can you deliver")
}
}
-
Next, we see that we have an
onEvent<TellDeliveryOptions>handler that catch the above mentioned event. To be extra clear, this is the handler that catches both the previously raised events (both here in General and in the RequestDelivery state), answers the query and then reenters. -
Next, we find a similar pattern with two handlers for opening hours.
-
After that, we have a similar pattern with two handlers for topping options. This time, we took the chance to introduce a new method available for EnumEntity, namely
optionsToText()which will return the options comma-separated with an "and" before the last entry - i.e for example "ham, bacon and tomato" for a Topping entity. -
Finally, we note that we have our previously defined
onResponse<OrderPizza>handler. We want to put this here since user's might answer a specific question about for example delivery options with additional slots, for example "I want the pizza delivered to my home at 3pm" in which case we want to catch the time information in addition to the delivery address.
Additional slot-filling states
=======================================================================
Request delivery time from user:
val RequestTime : State = state(parent = OrderHandling) {
onEntry() {
furhat.ask("At what time do you want it delivered?")
}
onResponse<RequestOptions> {
raise(TellOpeningHours())
}
onResponse<TellTime> {
furhat.say("Okay, ${it.intent.time}")
users.current.order.deliveryTime = it.intent.time
goto(CheckOrder)
}
}
Request toppings from user:
val RequestTopping : State = state(parent = OrderHandling) {
onEntry() {
furhat.ask("Any extra topping?")
}
onResponse<RequestOptions> {
raise(TellToppingOptions())
}
onResponse<Yes> {
furhat.ask("What kind of topping do you want?")
}
onResponse<No> {
furhat.say("Okay, no extra topping")
users.current.order.topping = ListOfTopping()
goto(CheckOrder)
}
onResponse<AddTopping> {
furhat.say("Okay, ${it.intent.topping}")
users.current.order.topping = it.intent.topping
goto(CheckOrder)
}
}
The OrderHandling state
val OrderHandling: State = state(parent = Questions) {
/* Handler that re-uses our pizza intent but has a more intelligent
response handling depending on what new information we get*/
onResponse<OrderPizzaIntent> {
val order = users.current.order
// Message to be constructed based on what data points we get from the user
var message = "Okay"
// Adding topping(s) if we get any new
if (it.intent.topping != null) message += ", adding ${it.intent.topping}"
// Adding or changing delivery option and time
if (it.intent.deliverTo != null || it.intent.deliveryTime != null) {
/* We are constructing a specific message depending on if we
get a delivery place and/or time and if this slot already had a value
*/
when {
// We get both a delivery place and time
it.intent.deliverTo != null && it.intent.deliveryTime != null -> {
message += ", delivering ${it.intent.deliverTo}
${it.intent.deliveryTime} "
// Add an "instead" if we are overwriting any of the slots
if (order.deliverTo != null || order.deliveryTime != null){
message += "instead "
}
}
// We get only a delivery place
it.intent.deliverTo != null -> {
message += ", delivering ${it.intent.deliverTo} "
// Add an "instead" if we are overwriting the slot
if (order.deliverTo != null) message += "instead "
}
// We get only a delivery time
it.intent.deliveryTime != null -> {
message += ", delivering ${it.intent.deliveryTime} "
// Add an "instead" if we are overwriting the slot
if (order.deliveryTime != null) message += "instead "
}
}
}
// Deliver our message
furhat.say(message)
// Finally we join the existing order with the new one
order.adjoin(it.intent)
reentry()
}
/* Specific handler for removing toppings since this is to complex to include in our
OrderPizzaIntent (due to the ambiguity of adding vs removing toppings)*/
onResponse<RemoveToppingIntent> {
users.current.order.topping?.removeFromList(it.intent?.topping!!)
furhat.say("Okay, we remove ${it.intent?.topping} from your pizza")
reentry()
}
}
Thus, we can focus on the active part of the interaction, starting with the Start state.
Starting the interaction with the Start state.
Upon entry by user into the interaction space of the robot the user needs to know that the robot is available for dialogue. Thus, the robot should invite the user into the conversation by starting it themselves. To do this it is recommended to use a Start state as follows:
val Start = state(parent = Questions) {
onEntry {
furhat.ask("Welcome to Pizza house. How may I help you?")
}
onResponse<OrderPizzaIntent> {
users.current.order.adjoin(it.intent)
furhat.say("Ok, you want a pizza ${it.intent}")
goto(CheckOrder)
}
}
One thing of note is that the Start state inherits from the parent state Questions which will be explained below.
Here, the user is welcomed by the robot and any slots filled by the users initial response (see State checking the current slots of the order) are checked with onResponse
Utilizing inheritance to create dynamic dialogues.
Above in the top line of code it can be seen that Start has the parent state Questions. This allows for the dialogue to be more dynamic instead of static and linear, meaning that even though the current state does not have any handlers for some intents, it can inherit them from a different state.
The Questions state appears as follows:
val Questions: State = state(Interaction) {
onResponse<RequestDeliveryOptionsIntent> {
furhat.say("We can deliver to your home and to your office")
reentry()
}
onResponse<RequestOpeningHoursIntent> {
furhat.say("We are open between 7 am and 8 pm")
reentry()
}
onResponse<RequestToppingOptionsIntent> {
furhat.say("We have " + Topping().optionsToText())
reentry()
}
}
This is an example of how inheritance should be utilized, in a way so that the robot can respond to questions asked by the user at any time in the dialogue as well as to avoid unnecessary code repetition.
Ending dialogue with the ConfirmOrder state.
Same as with why the Start state it is necessary to have an ending to the dialogue that ties everything together so that the user does not attempt to continue a dialogue with the robot.
val ConfirmOrder : State = state(parent = OrderHandling) {
onEntry {
furhat.ask("Does that sound good?")
}
onResponse<Yes> {
goto(EndOrder)
}
onResponse<No> {
goto(ChangeOrder)
}
}
This state is used to tie together the conversational flow and permit the user to make any adjustments to their order, essentially modifying the form-filling properties before confirming that everything is correct. So here the user will be asked if their order is correct and then the order will either be completed or the user will have an opportunity to make any changes until they are satisfied using the ChangeOrder state.
The ChangeOrder state appears as follows:
val ChangeOrder = state(parent = OrderHandling) {
onEntry {
furhat.ask("Anything that you like to change?")
}
onReentry {
furhat.ask("I currently have a pizza ${users.current.order}. Anything that you like to change?")
}
onResponse<Yes> {
reentry()
}
onResponse<No> {
goto(EndOrder)
}
}
As can be seen this state inherits the OrderHandling state which is used to set any variables in the order in the same way as has been shown before.
This concludes the tutorial. For a working code-example, please see Github.