Natural language understanding

FurhatOS provides a Natural language understanding (NLU) component that is well integrated with the flow. The NLU has two important concepts:

  • Intent: Each user utterance can be classified according to its Intent. Think of intent as a method in programming. For example, the intent of an utterance could be a Greeting ("hello there"), a RequestRepeat ("could you repeat that") or a BuyFruit ("I want to buy an apple"). Each intent can be expressed using many different combinations of words.
  • Entity: Utterances can also contain semantic entities, that is, parts of the utterance that represent concepts such as City, Color, Time or Date. Think of entities as parameters to the method (which corresponds to the intent). Thus, the BuyFruit intent may have an entity Fruit that specifies the fruit to be bought.

Intents

Intents are defined by extending the Intent class and providing examples. These examples do not have to match exactly to what the user says. Instead, the system use machine learning to choose the intent that matches best, from a set of possible intents.

You define intents this way in Kotlin:

class Greeting : Intent() {
    override fun getExamples(lang: Language): List<String> {
        return listOf("hello there", "nice to meet you")
    }
}

Defining intent examples in a separate file

As can be seen, the examples can be provided by overriding the getExamples() method. But it is also possible to put them in a separate text file (separated by newline). Give the file the name Greeting.en.exm ("en" for English) and put it in the resource folder in the same package as the intent class. In that case, the getExamples() method should not be implemented:

class Greeting : Intent()

The Greeting.en.exm could then for example include:

Hello
Hi
Howdy
Hi there

Intents with Entities

Note: Scroll down to see the docs about Entities

To specify entities in an intent, you most easily define them in the default constructor:

class BuyFruit(val fruit : Fruit? = null) : Intent() {
    override fun getExamples(lang: Language): List<String> {
        return listOf("banana", "I want a banana", "I would like a banana")
    }
}

Important note: Entities need to be nullable and with null as a default value. This is a requirement for the NLU intent classification to work properly.

Note that the examples do not have to contain every variant of the fruit, and you do not have to point out the parameter in the example ("banana"), this is done automatically. However, you can use the name of the entity instead if you want (Using the format "I want a @fruit").

If your intent has several entities of the same type, you have to specify the field in the examples, since they cannot be automatically assigned:

class OrderTrip(
  val departure : City? = null,     
  val destination : City? = null,
  val date : Date? = null
) : Intent() {
    override fun getExamples(lang: Language): List<String> {
        return listOf(
                "i want to go from @departure to @destination on the first of august",
                "i would like to travel to @destination")
    }
}

Dynamic Intents

The methods described above are very useful when a set of intents can be pre-defined in Kotlin. Defining intents as classes has the advantage that Kotlin understands the types of the entities, and thereby provides code completion for them in the flow.

However, sometimes it is not possible to define all intents as separate classes, but you would rather want to define them as instances of a common class. This could for example be the case if you want to read a set of intents from an external resource, and generate them on-the-fly.

There are two classes that support this behavior: SimpleIntent, in case you do not need the intent to have any entities, and DynamicIntent, if you need it to support entities.

Built-in Intents

FurhatOS comes with common intents already pre-defined, in the furhatos.nlu.common package:

  • Yes
  • No
  • Greeting
  • Goodbye
  • RequestRepeat

Note: currently these intents exist only for US English, but this will be extended within short.

SimpleIntent - dynamic intents without entities

You can easily define a SimpleIntent like this:

val Compliment = SimpleIntent("I love you", "you look fantastic" )

It can then be caught in the flow like this:

onResponse(Compliment) {
    furhat.say("That was very nice of you!")
}

Notice the difference from the previous examples, where the intent class is specified inside angle brackets (<>). Here, we instead use the actual intent object as an argument.

You can also provide a list of SimpleIntents to catch:

val NiceThingsToSay = listOf(Compliment,Praise,Gratitude)

onResponse(NiceThingsToSay) {
    furhat.say("That was very nice of you!")
}

You can also define a simple intent on-the-fly in the flow (with examples in brackets):

onResponse("what is your favorite film", "what do you like to watch") {
    furhat.say("I really like Star Wars")
}

It is also possible to extend the SimpleIntent:

class QAIntent(
        val question : String,
        val answer : String
) : SimpleIntent(listOf(question))

val Questions =
    listOf(
        QAIntent("how old are you", "I am five years old"),
        QAIntent("what is your name", "my name is Furhat"),
        QAIntent("what is your favorite food", "I love meatballs"))

As you can see below, it is now possible to access the actual intent that was matched through the it.intent variable. Kotlin knows the type of the intent (since Questions is a list of intents of type QAIntent), and you can therefore directly access the "answer" field:

onResponse(Questions) {
    furhat.say(it.intent.answer)
}

DynamicIntent - dynamic intents with entities

If you need to support entities in you dynamic intents, you should use the DynamicIntent class:

val OfferFruit = DynamicIntent(
    listOf("do you want a banana", "would you like an apple"),
    mapOf("fruit" to Fruit::class.java),
    Language.ENGLISH_US)

As can be seen, the constructor needs to be provided with example utterances, a map with the entities, and the language of the intent. The DynamicIntent can be used in the flow in exactly the same way as SimpleIntent, but the "it.intent" variable now has two properties: "dynamicIntent", which points to the dynamic intent object, and "entities" which points to a Record of entities found when matching the intent.

Entities

An entity (or Semantic entity) is defined as a Java class that extends the Entity class. As we will see, there are already a number of common entities implemented. For example, the entity Date corresponds to "tomorrow" or "the 3rd of July". There are also a number of abstract entity classes that can be extended, in order to make it convenient to implement them using different algorithms.

Entities have three important properties:

  1. Entities are dependent on each other. For example, Time entity ("three o'clock") is dependent on the Number entity ("three").

  2. Entities can identify themselves in a text. This means that the semantic entity encapsulates the method for identifying itself in a text in a certain language. Different entities can use different methods for the natural language processing (NLP), such as a context-free grammar, or machine learning. Since entities are identified in different processing steps, but dependent on each other, different NLP techniques can be mixed in the same interpretation chain.

  3. Entities are rich Java objects. This means that they can encapsulate powerful methods and fields. For example, the Date entity has the method asLocalDate(), which returns a LocalDate object (introduced in Java 8), which has very powerful methods for date arithmetics. The City entity contains information about the country, population, latitude and longitude.

EnumEntity

FurhatOS provides a set of base classes for easily defining different types of entities, using different NLU algorithms.

The EnumEntity is based on a simple enumeration. Such entities can easily be defined in Kotlin like this:

class Fruit : EnumEntity() {
    override fun getEnum(lang: Language): List<String> {
        return listOf("banana", "orange", "apple", "pineapple", "pear")
    }
}

Using EnumEntities

In this basic example, the language is ignored, and a simple list is returned. Note that the items are not case sensitive.

When the EnumEntity has been identified, the value of the entity (e.g. "banana") is stored in the "value" property of the EnumEntity. The text used to identify the entity is stored in the "text" property. In the simple case described above, these will be the same thing: Each alternative only has one surface form, and the value of the entity is the same as this form.

However, sometimes you want different surface forms to mean the same thing (i.e. synonyms), and the value to have a special computational meaning that is different from the surface form. For example, you typically want the value to be the same across different languages. In that case, you can specify the enum like this, where the value is followed by a colon and a list of synonyms, separated by commas (note that the value is not treated as a synonym and will not be used for matching):

coca_cola:cola,coca cola,coke
pepsi:pepsi,pepsi cola
sprite

Per default, stemming is applied to the words, which means that it assumes that for example "apple" and "apples" are the same words. If you do not want this, you can specify this in the EnumEntity constructor (EnumEntity(stemming=false)).

Note: Stemming is currently only available in English, Swedish, and German. For languages that currently don't support stemming we recommend using synonyms, as described above.

Defining EnumEntities in separate files

Enumerations can also be defined in a separate file. The system assumes the files to be given the name of the entity, plus the language, and the .enu extension. For the Fruit entity, this would be for example Fruit.en.enu. The file should be placed in the resource folder of same package folder as the entity class.

Example content of Fruit.en.enu would be:

banana
orange:orange,lemon
apple
pineapple
pear

If such an enumeration file is used, the class should not implement the getEnum() method:

class Fruit : EnumEntity()

ComplexEnumEntity

If you want your enum entity to refer to other entities, you need to use the ComplexEnumEntity. For example, you could define an entity that matches things like "five apples", which uses both the Fruit entity defined above and the built-in entity Number:

class FruitCount(
    val count : Number? = Number(1),
    val fruit : Fruit? = null) : ComplexEnumEntity() {

    override fun getEnum(lang: Language): List<String> {
        return listOf("@count @fruit", "@fruit")
    }
}

In the enum, you can use a mix of words and references to entities, which starts with the @-symbol. The referred entities are defined as variables in the class and will be instantiated when extracting the entity. In this example, we also allow just "@fruit" (e.g. "banana"), in which case the "count" field will be assigned the default value Number(1).

Wildcard entities

A feature of ComplexEnumEntity is that it supports wildcards, i.e., it can match arbitrary strings. To do this, you simply use the String class for a field. The following example would catch all strings like "remind me to water the flowers", where the field "who" would be bound to "me", and "what" would be bound to "water the flowers". Note that the matching of wildcard elements is greedy, so it will match as many words as possible.

class FruitCount(
    val who : String? = null,
    val what : String? = null) : ComplexEnumEntity() {

    override fun getEnum(lang: Language): List<String> {
        return listOf("remind @who to @what")
    }
}

Of course, it is also possible to mix wildcard elements with entities (e.g., use the built-in entity PersonName for "who").

ListEntity

If you want to represent a list of entities as one entity (such as "banana, orange and apple"), you can define it as a ListEntity. This is very useful for intents that allow such enumarations:

class ListOfFruit : ListEntity<Fruit>()

class BuyFruit(val fruits : ListOfFruit? = null) : Intent() {
    override fun getExamples(lang: Language): List<String> {
        return listOf("banana", "I want banana apple and orange", "I would like a banana")
    }
}

If you instead of Fruit use the FruitCount entity defined above, you could match phrases like "one banana, two apples and three oranges".

GrammarEntity

If you need an entity to identify more complex syntactic structures, you can specify them using a grammar (technically a context-free grammar), using the GrammarEntity.

class Burger(
  val type : String? = null,
  val size : String? = null
) : GrammarEntity() {

    override fun getGrammar(lang : Language) : Grammar {
        return when (lang.main) {
            "en" -> BurgerGrammarEn
            else -> throw InterpreterException("Language $lang not supported for ${javaClass.name}")
        }
    }
}

val BurgerGrammarEn =
    grammar {
        rule(public = true) {
            group {
                - {"a" / "an" / "the")
                ruleref("size")
                ruleref("type")
                tag { Burger(size=ref["size"] as String,
                        type=ref["type"] as String) }
            }
        }
        rule("size") {
            +"small" tag {"small"}
            +"medium" tag {"medium"}
            +"large" tag {"large"}
            +"big" tag {"large"}
        }
        rule("type") {
            +"burger" tag {"regular"}
            +"hamburger" tag {"regular"}
            +"cheeseburger" tag {"cheese"}
            +"fishburger" tag {"fish"}
        }
    }
  • grammar: Defines a grammar consisting of a set of rules
  • rule: Defines a rule in the grammar. The rules marked as "public" are the ones that will be used to identify the entity. The other rules are sub-rules (given names), used to identify parts of the entity. The rule contains a list of tokens, of which one has to match.
  • ruleref: Matches another rule, specified by the name.
  • entity: Matches an entity of type T
  • group: Defines a sequence of tokens that has to match.
  • choice: Defines a list of tokens or groups, of which one has to match.

Matching of words can be defined in different ways:

// Matches the word "burger" (and returns the string "burger")
+"burger"
// Optionally matches the word "the"
-"the"
// Matches the word "burger" and returns the string "regular"
+"burger" tag {"regular"}
// Matches "burger" or "hamburger", and returns the string "regular"
+("burger" / "hamburger") tag {"regular"}
// Optionally matches "a", "an", or "the"
-("a" / "an" / "the")
// NB: Remember to use the "+" or "-". If you ommit these, Kotlin will not complain, but the token will not be added to the grammar.

WikiData entities

WikiData entities are a special type of entity that dynamically fetches information from WikiData.org. They allow you to build rich chit-chat skills without building your own extensive language/knowledge graph.

If you for example look at the built-in MusicArtist WikiData entity, it has fields such as albums and genre which are dynamically fetched when you access these entities, allowing you to ask relevant follow-up questions and similar. This is especially useful when you are using our Snippets building blocks for a chit-chat type interaction.

We are currently not recommending to build your own WikiData entities, but you can use the built-in ones at your liking. See below.

Built-in entities

We ship some commonly used entities as part of the Furhat system, currently only supporting US English.

Enum entities

The Furhat system comes with common Enum Entities already pre-defined, in the furhatos.nlu.common package:

  • Time: Time expressions
  • Date: Date expressions
  • Number: Numbers in cardinal form (e.g. "three hundred" or "300")
  • Ordinal: Numbers in ordinal form (e.g., "third" or "3rd")
  • Color: Colors
  • PersonName: Names of persons

Note: currently these entities are defined for US English only.

WikiData entities

The system also contains the following experimental entities based on WikiData. These should support multiple languages. The first time they are used on a computer, it will take some time to fetch them from the Internet (they will cache after the first fetch):

  • City: All larger cities in the world
  • Country: All larger countries in the world
  • MusicArtist: Music artists (both persons and bands, e.g. "Madonna")
  • MusicGenre: Music genres (e.g. "Reggae")
  • Sport: Sports (e.g. "soccer")
  • Film: Film titles (e.g. "The Godfather")

Use can also explore in the IDE what kind of properties these entities provide.

Using NLU in flow

The NLU entities are deeply coupled with the flow, allowing a typed experience when defining your response handlers. For

Using Intents

You can use the intents directly in the flow in the onResponse event handler with onResponse<Intent> { ... }, example as follows:

val AskFruit = state {
    onEntry {
        furhat.ask("What fruit do you want?")
    }

   onResponse<Greeting> {
        furhat.say("Hello there")
        reentry()
    }

    onResponse<BuyFruit> {
        furhat.say(it.intent.fruit + ", what a lovely choice")
            reentry()
    }

}

As you can see, the entity of the intent can be accessed through the "it" variable.

It is possible to have onResponse handlers with intents on different levels in the state hierarchy. The system will collect all intents from all ancestors to the current state, to choose from.

Using Entities as Intents

If you want an intent to just cover a single entity, and the utterance corresponds to that entity (e.g. "banana" as a ReplyFruit intent, to answer the question "which fruit do you want?"), you can use the entity directly in the flow as you would an intent. However, be aware that the entities must be included fully in the utterance to match. If your entity has the defintion "lord darth vader" and you try to match it as an intent, utterances like "I like lord darth vader very much" may match but "I am lord vader" will not. Solve this by providing several definitions for each entity entry.

val AskFruit = state {
    onEntry {
        furhat.ask("What fruit do you want?")
    }

    onResponse<Fruit> {
        furhat.say(it.intent + ", what a lovely choice")
        reentry()
    }

}

When entities are used as intents like this, the it.intent field will hold the entity (Fruit in this case).

Lists of intents

Sometimes, you might have several intents that you want to handle the same way. For example, in some contexts you might want a "maybe" to be handled the same way as a "no" (because consent is important!) but in others not. There are several ways of accomplishing this, lists of events is the first.

If you have the handlers in the same state, you can use listOf(FirstIntent(), SecondIntent()) together with onResponse. Example:

  onResponse(listOf(No(), Maybe())) {
    furhat.say("Okay, no problem!")
  }

You can get the entities from the intent above with it.intent.getEntities(), returning a map of variable name and the Entity instance.

If you want, you can also use a function that returns a list of responses in a similar fashion:

// As a lambda
onResponse({ listOf(Yes(), No()) }) {
    // Handle
}

// As named function
fun getIntentList() : List<IntentCandidate> {
    return listOf(Yes(), No())
}

onResponse(getIntentList()) {
    // Handle
}


Raising a response with a new Intent

You can also raise a response with a new response, where you create a new intent. This allows you to use an already defined response handler, perhaps in a parent state. It doesn't matter where in order the second response handler is defined.

onResponse<No> {
    furhat.say("Yes or no? I'm confused")
}

onResponse<Yes> {
    raise(it, No())
}

While this gives you more flexibility in terms of what you can do with the response, when you manually raise a response with a new intent you have to manually construct the second response and intent. This means that you also have to construct/attach any entities that your new intent might need.

Generating text

Sometimes you need to generate a text back from an intent or an entity (referred to as Natural Language Generation, or NLG), for example if you want to confirm something that the user said.

The base class Entity implements an interface called TextGenerator, which provides a method toText(lang: Language), which generates a text string for the entity in a specific language. There is also a toText() method that calls toText(lang: Language) with the current default language. The Entity class also implements toString() by calling toText(). This way, you can directly use the entity as part of a string in a response, as we have seen before:

onResponse<BuyFruit> {
    furhat.say("${it.intent.fruit}, what a lovely choice")
    reentry()
}

The base class Entity has a property "text", which stores the text string used to identify the entity the user's utterance. If it is filled in (i.e., if the entity was extracted from the user's utterance), this is used per default to generate the text for the entity. If not, different entities have different ways of generating text. You can also override the toText(lang: Language) method in your entity if you want to generate text with some specific method.

Intents do not implement the TextGenerator interface, so they cannot be used directly in this way (toString() doesn't return anything readable). However, you can implement the TextGenerator interface in your intent:

class BurgerOrder(    
    val main : MainCourse? = null,
    val drink : Drink? = null,
    val side : SideOrder? = null,
    val payment : Payment? = null) : Intent(), TextGenerator {

    override fun getExamples(lang: Language): List<String> {
        return listOf(
                "a @drink please",
                "I want a @main with @drink",
                "I would like a @main",
                "I want a @main with @side")
    }

    override fun toText(lang : Language) : String {
        return generate(
                "($main with $side and $drink | $main with $side | $main with $drink | $main)[, paying with $payment]");
    }

    override fun toString(): String {
        return toText()
    }
}

Note that you should also implement toString() (and call the toText() method) in order to be able to use the intent directly as part of a string.

In the toText(lang: Language) method, you can take advantage of the generate() method provided in the TextGenerator interface. It parses the provided string and applies some useful logic:

  • It will not generate strings that contain the "null" word. So, if a variable returns "null", the generation will normally fail.
  • You can provide several possible strings to generate, separated by a pipe ("|"). It will then choose the first string that does not fail.
  • You can group parts of the string with parentheses
  • If you group a part of the string with brackets, the generation will not fail if the brackets contain the "null" word, instead the brackets will just generate an empty string.

In the example above, this means that it will first try to generate "$main with $side and $drink". If any of these variables returns null, it will try to generate "$main with $side", and so on. After this, it will append the string ", paying with $payment" if payment is not null, otherwise it will not append anything.

Pre-loading and reloading intents and entities

Intents and entities are normally loaded/initialized the first time they are used, on state entry. If you have some dynamic property in these intents, this means that there might be a short delay if this initialization takes time - for example if you do an API call do fetch examples or if you simply have a very long list of examples.

Pre-loading of intents and entities

To cope with the above mentioned cases, you might want to preload/pre-initialize your intents. A good time to do this may be on skill startup or at some other time that makes sense for your use-case.

To do this, you use YourIntent().preload(language), for example in your first state's init method as shown below:

// Entity with enums fetched from an API
class MyEntity : EnumEntity() {
    override fun getEnum(lang: Language): List<String> {
        return mySlowAPI.fetchEntityEnums(this.javaClass.simpleName, lang)
    }
}

// Intent with examples fetched from an API
class MyIntent(entity: MyEntity) : Intent() {
    override fun getExamples(lang: Language): List<String> {
       return mySlowAPI.fetchIntentExamples(this.javaClass.simpleName, lang)
   }
}

// Initial state that preloads the above intent and entity for English US
val myFirstState : State = state {
    init {
      MyIntent().preload(Language.ENGLISH_US)
    }

    onEntry {
      goto(InteractiveState)
    }
}

If you want to manually pre-load/initialize entities without them being part of intents as above, you can use Interpreter.preload(MyEntity.class, language) (we will update this method shortly to a more friendly API similar to that of intents).

Reload intents and entities

If your intents and entities needs to be refreshed at some point, for example if you dynamically update the examples of an intent, or the enums of an entity, you can use the methods forget() and forgetAll() available for Intents and Entities and then either let the system automatically reload them the next time they are used, or explicitly do it at some time of your choosing.

// Some state where we for some reason want to forget our intents. They will then be re-initialized the next time they are used, or when they are initialized manually

val refreshedContent : State = state {
    onEntry {
      // You can forgot individual intents and entities ...
      MyIntent.forget()
      MyEntity.forget()

      // If you want, you can reload them manually as shown above
      MyIntent.preload(Language.ENGLISH_US)

      // ... or forget all intents and entities
      Intent().forgetAll()
      Entity().forgetAll()
    }
}

Note that you explicitly have to forget entities even if they are loaded/initialized through an intent. The reason is that you might use the entities elsewhere and you may not want to forget them automatically.