Natural language understanding

FurhatOS provides a Natural language understanding (NLU) component that is well integrated with the flow. The NLU has two important concepts:

  • Intent: Each user utterance can be classified according to its Intent. Think of intent as a method in programming. For example, the intent of an utterance could be a Greeting ("hello there"), a RequestRepeat ("could you repeat that") or a BuyFruit ("I want to buy an apple"). Each intent can be expressed using many different combinations of words.
  • Entity: Utterances can also contain semantic entities, that is, parts of the utterance that represent concepts such as City, Color, Time or Date. Think of entities as parameters to the method (which corresponds to the intent). Thus, the BuyFruit intent may have an entity Fruit that specifies the fruit to be bought.

Intents

Intents are defined by extending the Intent class and providing examples. These examples do not have to match exactly to what the user says. Instead, the system use machine learning to choose the intent that matches best, from a set of possible intents.

You define intents this way in Kotlin:

class Greeting : Intent() {
    override fun getExamples(lang: Language): List<String> {
        return listOf("hello there", "nice to meet you")
    }
}

Defining intent examples in a separate file

As can be seen, the examples can be provided by overriding the getExamples() method. But it is also possible to put them in a separate text file (separated by newline). Give the file the name Greeting.en.exm ("en" for English) and put it in the resource folder in the same package as the intent class. In that case, the getExamples() method should not be implemented:

class Greeting : Intent()

The Greeting.en.exm could then for example include:

Hello
Hi
Howdy
Hi there

Intents with Entities

Note: Scroll down to see the docs about Entities

To specify entities in an intent, you most easily define them in the default constructor:

class BuyFruit(val fruit : Fruit? = null) : Intent() {
    override fun getExamples(lang: Language): List<String> {
        return listOf("banana", "I want a banana", "I would like a banana")
    }
}

Important note: Entities need to be nullable and with null as a default value. This is a requirement for the NLU intent classification to work properly.

Note that the examples do not have to contain every variant of the fruit, and you do not have to point out the parameter in the example ("banana"), this is done automatically. However, you can use the name of the entity instead if you want (Using the format "I want a @fruit").

If your intent has several entities of the same type, you have to specify the field in the examples, since they cannot be automatically assigned:

class OrderTrip(
  val departure : City? = null,     
  val destination : City? = null,
  val date : Date? = null
) : Intent() {
    override fun getExamples(lang: Language): List<String> {
        return listOf(
                "i want to go from @departure to @destination on the first of august",
                "i would like to travel to @destination")
    }
}

Using intents in the flow

You can use the intents directly in the flow in the onResponse event handler:

val AskFruit = state {
    onEntry {
        furhat.ask("What fruit do you want?")
    }

    onResponse<Greeting> {
        furhat.say("Hello there")
        reentry()
    }

    onResponse<BuyFruit> {
        furhat.say(it.intent.fruit + ", what a lovely choice")
            reentry()
    }

}

As you can see, the entity of the intent can be accessed through the "it" variable.

It is possible to have onResponse handlers with intents on different levels in the state hierarchy. The system will collect all intents from all ancestors to the current state, to choose from.

Entities as Intents

If you want an intent to just cover a single entity, and the utterance corresponds to that entity (e.g. "banana" as a ReplyFruit intent, to answer the question "which fruit do you want?"), you can use the entity directly in the flow as you would an intent. However, be aware that the entities must be included fully in the utterance to match. If your entity has the defintion "lord darth vader" and you try to match it as an intent, utterances like "I like lord darth vader very much" may match but "I am lord vader" will not. Solve this by providing several definitions for each entity entry.

val AskFruit = state {
    onEntry {
        furhat.ask("What fruit do you want?")
    }

    onResponse<Fruit> {
        furhat.say(it.intent + ", what a lovely choice")
        reentry()
    }

}

When entities are used as intents like this, the it.intent field will hold the entity (Fruit in this case).

Built-in Intents

FurhatOS comes with common intents already pre-defined, in the furhatos.nlu.common package:

  • Yes
  • No
  • Greeting
  • Goodbye
  • RequestRepeat

Note: currently these intents exist only for US English, but this will be extended within short.

Dynamic Intents

The methods described above are very useful when a set of intents can be pre-defined in Kotlin. Defining intents as classes has the advantage that Kotlin understands the types of the entities, and thereby provides code completion for them in the flow.

However, sometimes it is not possible to define all intents as separate classes, but you would rather want to define them as instances of a common class. This could for example be the case if you want to read a set of intents from an external resource, and generate them on-the-fly.

There are two classes that support this behavior: SimpleIntent, in case you do not need the intent to have any entities, and DynamicIntent, if you need it to support entities.

You can easily define a SimpleIntent like this:

val Compliment = SimpleIntent("I love you", "you look fantastic" )

It can then be caught in the flow like this:

onResponse(Compliment) {
    furhat.say("That was very nice of you!")
}

Notice the difference from the previous examples, where the intent class is specified inside angle brackets (<>). Here, we instead use the actual intent object as an argument.

You can also provide a list of SimpleIntents to catch:

val NiceThingsToSay = listOf(Compliment,Praise,Gratitude)

onResponse(NiceThingsToSay) {
    furhat.say("That was very nice of you!")
}

You can also define a simple intent on-the-fly in the flow (with examples in brackets):

onResponse("what is your favorite film", "what do you like to watch") {
    furhat.say("I really like Star Wars")
}

It is also possible to extend the SimpleIntent:

class QAIntent(
        val question : String,
        val answer : String
) : SimpleIntent(listOf(question))

val Questions =
    listOf(
        QAIntent("how old are you", "I am five years old"),
        QAIntent("what is your name", "my name is Furhat"),
        QAIntent("what is your favorite food", "I love meatballs"))

As you can see below, it is now possible to access the actual intent that was matched through the it.intent variable. Kotlin knows the type of the intent (since Questions is a list of intents of type QAIntent), and you can therefore directly access the "answer" field:

onResponse(Questions) {
    furhat.say(it.intent.answer)
}

If you need to support entities in you dynamic intents, you should use the DynamicIntent class:

val OfferFruit = DynamicIntent(
    listOf("do you want a banana", "would you like an apple"),
    mapOf("fruit" to Fruit::class.java),
    Language.ENGLISH_US)

As can be seen, the constructor needs to be provided with example utterances, a map with the entities, and the language of the intent. The DynamicIntent can be used in the flow in exactly the same way as SimpleIntent, but the "it.intent" variable now has two properties: "dynamicIntent", which points to the dynamic intent object, and "entities" which points to a Record of entities found when matching the intent.

Entities

An entity (or Semantic entity) is defined as a Java class that extends the Entity class. As we will see, there are already a number of common entities implemented. For example, the entity Date corresponds to "tomorrow" or "the 3rd of July". There are also a number of abstract entity classes that can be extended, in order to make it convenient to implement them using different algorithms.

Entities have three important properties:

  1. Entities are dependent on each other. For example, Time entity ("three o'clock") is dependent on the Number entity ("three").

  2. Entities can identify themselves in a text. This means that the semantic entity encapsulates the method for identifying itself in a text in a certain language. Different entities can use different methods for the natural language processing (NLP), such as a context-free grammar, or machine learning. Since entities are identified in different processing steps, but dependent on each other, different NLP techniques can be mixed in the same interpretation chain.

  3. Entities are rich Java objects. This means that they can encapsulate powerful methods and fields. For example, the Date entity has the method asLocalDate(), which returns a LocalDate object (introduced in Java 8), which has very powerful methods for date arithmetics. The City entity contains information about the country, population, latitude and longitude.

Built-in entities

The Furhat system comes with common entities already pre-defined, in the furhatos.nlu.common package:

  • Time: Time expressions
  • Date: Date expressions
  • Number: Numbers in cardinal form (e.g. "three hundred" or "300")
  • Ordinal: Numbers in ordinal form (e.g., "third" or "3rd")
  • City: All cities in the world with a population greater than 15000.
  • Color: Colors
  • PersonName: Names of persons

Note: currently these entities are defined for US English only, but this will be extended within short.

EnumEntity

FurhatOS provides a set of base classes for easily defining different types of entities, using different NLU algorithms.

The EnumEntity is based on a simple enumeration. Such entities can easily be defined in Kotlin like this:

class Fruit : EnumEntity() {
    override fun getEnum(lang: Language): List<String> {
        return listOf("banana", "orange", "apple", "pineapple", "pear")
    }
}

Using EnumEntities

In this basic example, the language is ignored, and a simple list is returned. Note that the items are not case sensitive.

When the EnumEntity has been identified, the value of the entity (e.g. "banana") is stored in the "value" property of the EnumEntity. The text used to identify the entity is stored in the "text" property. In the simple case described above, these will be the same thing: Each alternative only has one surface form, and the value of the entity is the same as this form.

However, sometimes you want different surface forms to mean the same thing (i.e. synonyms), and the value to have a special computational meaning that is different from the surface form. For example, you typically want the value to be the same across different languages. In that case, you can specify the enum like this, where the value is followed by a colon and a list of synonyms, separated by commas (note that the value is not treated as a synonym and will not be used for matching):

coca_cola:cola,coca cola,coke
pepsi:pepsi,pepsi cola
sprite

Per default, stemming is applied to the words, which means that it assumes that for example "apple" and "apples" are the same words. If you do not want this, you can specify this in the EnumEntity constructor (EnumEntity(stemming=false)).

Note: Stemming is currently only available in English, Swedish, and German. For languages that currently don't support stemming we recommend using synonyms, as described above.

Defining EnumEntities in separate files

Enumerations can also be defined in a separate file. The system assumes the files to be given the name of the entity, plus the language, and the .enu extension. For the Fruit entity, this would be for example Fruit.en.enu. The file should be placed in the resource folder of same package folder as the entity class.

Example content of Fruit.en.enu would be:

banana
orange:orange,lemon
apple
pineapple
pear

If such an enumeration file is used, the class should not implement the getEnum() method:

class Fruit : EnumEntity()

ComplexEnumEntity

If you want your enum entity to refer to other entities, you need to use the ComplexEnumEntity. For example, you could define an entity that matches things like "five apples", which uses both the Fruit entity defined above and the built-in entity Number:

class FruitCount(
    val count : Number? = Number(1),
    val fruit : Fruit? = null) : ComplexEnumEntity() {

    override fun getEnum(lang: Language): List<String> {
        return listOf("@count @fruit", "@fruit")
    }
}

In the enum, you can use a mix of words and references to entities, which starts with the @-symbol. The referred entities are defined as variables in the class and will be instantiated when extracting the entity. In this example, we also allow just "@fruit" (e.g. "banana"), in which case the "count" field will be assigned the default value Number(1).

ListEntity

If you want to represent a list of entities as one entity (such as "banana, orange and apple"), you can define it as a ListEntity. This is very useful for intents that allow such enumarations:

class ListOfFruit : ListEntity<Fruit>()

class BuyFruit(val fruits : ListOfFruit? = null) : Intent() {
    override fun getExamples(lang: Language): List<String> {
        return listOf("banana", "I want banana apple and orange", "I would like a banana")
    }
}

If you instead of Fruit use the FruitCount entity defined above, you could match phrases like "one banana, two apples and three oranges".

GrammarEntity

If you need an entity to identify more complex syntactic structures, you can specify them using a grammar (technically a context-free grammar), using the GrammarEntity.

class Burger(
  val type : String? = null,
  val size : String? = null
) : GrammarEntity() {

    override fun getGrammar(lang : Language) : Grammar {
        return when (lang.main) {
            "en" -> BurgerGrammarEn
            else -> throw InterpreterException("Language $lang not supported for ${javaClass.name}")
        }
    }
}

val BurgerGrammarEn =
    grammar {
        rule(public = true) {
            group {
                optional { word("a", "an", "the") }
                ruleref("size")
                ruleref("type") {
                    out = Burger(size=ref["size"] as String,
                            type=ref["type"] as String)
                }
            }
        }
        rule("size") {
            "small" out "small"
            "medium" out "medium"
            "large" out "large"
            "big" out "large"
        }
        rule("type") {
            "burger" out "regular"
            "hamburger" out "regular"
            "cheeseburger" out "cheese"
            "fishburger" out "fish"
        }
    }
  • grammar: Defines a grammar consisting of a set of rules
  • rule: Defines a rule in the grammar. The rules marked as "public" are the ones that will be used to identify the entity. The other rules are sub-rules (given names), used to identify parts of the entity. The rule contains a list of tokens, of which one has to match.
  • ruleref: Matches another rule, specified by the name.
  • group: Defines a sequence of tokens that has to match.
  • choice: Defines a list of tokens, of which one has to match.
  • optional: Specifies that the contained token is optional.

Final matching of words can be defined in different ways:

// Matches the word "burger" (and returns the string "burger")
word("burger")
// Matches the word "burger" and returns the string "regular"
word("burger") {out = "regular"}
// Matches "burger" or "hamburger", and returns the string "regular"
word("burger", "hamburger") {out = "regular"}
// Matches the word "burger" and returns the string "regular"
"burger" out "regular"
// Note that you cannot just use the word "burger" without the "out" (a Kotlin restriction)

Generating text

Sometimes you need to generate a text back from an intent or an entity (referred to as Natural Language Generation, or NLG), for example if you want to confirm something that the user said.

The base class Entity implements an interface called TextGenerator, which provides a method toText(lang: Language), which generates a text string for the entity in a specific language. There is also a toText() method that calls toText(lang: Language) with the current default language. The Entity class also implements toString() by calling toText(). This way, you can directly use the entity as part of a string in a response, as we have seen before:

onResponse<BuyFruit> {
    furhat.say("${it.intent.fruit}, what a lovely choice")
    reentry()
}

The base class Entity has a property "text", which stores the text string used to identify the entity the user's utterance. If it is filled in (i.e., if the entity was extracted from the user's utterance), this is used per default to generate the text for the entity. If not, different entities have different ways of generating text. You can also override the toText(lang: Language) method in your entity if you want to generate text with some specific method.

Intents do not implement the TextGenerator interface, so they cannot be used directly in this way (toString() doesn't return anything readable). However, you can implement the TextGenerator interface in your intent:

class BurgerOrder(    
    val main : MainCourse? = null,
    val drink : Drink? = null,
    val side : SideOrder? = null,
    val payment : Payment? = null) : Intent(), TextGenerator {

    override fun getExamples(lang: Language): List<String> {
        return listOf(
                "a @drink please",
                "I want a @main with @drink",
                "I would like a @main",
                "I want a @main with @side")
    }

    override fun toText(lang : Language) : String {
        return generate(
                "($main with $side and $drink | $main with $side | $main with $drink | $main)[, paying with $payment]");
    }

    override fun toString(): String {
        return toText()
    }
}

Note that you should also implement toString() (and call the toText() method) in order to be able to use the intent directly as part of a string.

In the toText(lang: Language) method, you can take advantage of the generate() method provided in the TextGenerator interface. It parses the provided string and applies some useful logic:

  • It will not generate strings that contain the "null" word. So, if a variable returns "null", the generation will normally fail.
  • You can provide several possible strings to generate, separated by a pipe ("|"). It will then choose the first string that does not fail.
  • You can group parts of the string with parentheses
  • If you group a part of the string with brackets, the generation will not fail if the brackets contain the "null" word, instead the brackets will just generate an empty string.

In the example above, this means that it will first try to generate "$main with $side and $drink". If any of these variables returns null, it will try to generate "$main with $side", and so on. After this, it will append the string ", paying with $payment" if payment is not null, otherwise it will not append anything.