Furhat Remote API DEPRECATED

DEPRECATED

This application is no longer maintained. We recommend that you use the Realtime API instead.

Furhat Remote API is a way to connect and give commands to your Furhat robot from a program running on an external computer on the same network. A number of programming languages are supported, including Python, C#, JavaScript, Rust etc. (50+ languages, for full list see Swagger documentation)

We've created a wrapper specifically for Python to make it as easy as possible to start using it in a python application.

An intro, overview and setup instructions to the Remote API and the python wrapper is available in this video.

Requests for more commands to include in the Remote API can be sent to: tech@furhatrobotics.com

Setup

Run the server on the Robot

The Remote API is included in Standard and Premium packages of the Furhat Robot. It can be started from the web interface of the robot, or in the SDK Launcher. When the API is running, the robot will start a Swagger Kotlin server listening on port 54321.

Testing the remote API using Postman

If you want, you can test the Remote API with the tool Postman:

Download Postman,
Download and open the Furhat Remote API yaml specification,
Test the different requests (look for API documentation below).

Create a client for your specific programming language

In this step you will generate the code in your preferred programming language that will enable your program to communicate with Furhat.

If you want to to use the Remote API from Python, we recommend to use our pre-built Python PyPi library, which wraps the Remote API and makes it easier to install and use.

Download Swagger or use their online editor,
Paste the content from Furhat Client yaml file ,
Click 'Generate Client' and select your language,
Incorporate the client to your system and send requests to control the robot.

API Documentation

API Endpoints

All URIs are relative to http://IP of the robot:54321

Endpoint	HTTP request method	Description
/furhat/attend	POST	Attend a user/location
/furhat/face	POST	Change the character and mask, or texture and model (deprecated)
/furhat/visibility	POST	Fade in or out the face (FaceCore only)
/furhat/gesture	POST	Perform a gesture
/furhat/led	POST	Change the colour of the LED strip
/furhat/listen	GET	Make the robot listen, and get speech results
/furhat/listen/stop	POST	Make the robot stop listening
/furhat/say	POST	Make the robot speak
/furhat/say/stop	POST	Make the robot stop talking
/furhat/voice	POST	Set the voice of the robot
/furhat/gestures	GET	Get all gestures
/furhat	GET	Test connection
/furhat/users	GET	Get current users
/furhat/voices	GET	Get all the voices on the robot

Authentication

None of the endpoints require authentication. Keep in mind that while this makes it easy to use, anyone on a larger network could theoretically listen in. It is recommended to stop the Remote API skill (or turn off the robot) when you are not actively working with it.

Models

io.swagger.server.models.BasicParam
io.swagger.server.models.Frame
io.swagger.server.models.Gesture
io.swagger.server.models.GestureDefinition
io.swagger.server.models.Location
io.swagger.server.models.Rotation
io.swagger.server.models.Status
io.swagger.server.models.User
io.swagger.server.models.Voice

/furhat

GET

Summary:

Test connection

Description:

Used to verify if the server is running, return "hello world" upon success

Responses

Code	Description
200	Status update

/furhat/gestures

GET

Summary:

Get all gestures

Description:

Returns a JSON array with all gestures on the system (names + duration).

Responses

Code	Description	Schema
200	A list of possible gestures	[ Gesture ]

/furhat/voice

POST

Summary:

Set the voice of the robot

Description:

Sets the voice of the robot using the name of the voice, can be requested by doing a GET request on this endpoint.

Parameters

Name	Located in	Description	Required	Schema
name	query	The name of the voice	Yes	string

Responses

Code	Description	Schema
200	Successful operation	Status

/furhat/voices

GET

Summary:

Get all the voices on the robot

Description:

Returns a JSON array with voice names + languages.

Responses

Code	Description	Schema
200	Success	[ Voice ]

/furhat/users

GET

Summary:

Get current users

Description:

Get all current users (max: 2). Returns a JSON array containg Users (Rotation, Location, id).

Responses

Code	Description	Schema
200	successful operation	[ User ]

/furhat/say

POST

Summary:

Make the robot speak

Description:

Makes the robot speak by either using text, or a URL (linking to a.wav file). If generatelipsync=true, it uses a .pho file hosted on the same url, or generates phonemes by itself.

Note : Lipsync does not work for local audio.

Parameters

Name	Located in	Description	Required	Schema
text	query	A string depicting a utterance the robot should say.	No	string
url	query	A url link to a audio file (.wav)	No	string
blocking	query	Whether to block execution before completion	No	boolean
lipsync	query	If a URL is provided, indicate if lipsync files should be generated/looked for.	No	boolean
abort	query	Stops the current speech of the robot.	No	boolean

Responses

Code	Description	Schema
200	successful operation	Status

/furhat/attend

POST

Summary:

Attend a user/location

Description:

Provides 3 modes of attention.

Attend user based on enum (CLOSEST, OTHER or RANDOM)
Attend user based on it's id (can be retrieved by using /furhat/users)
Attend location based on coordinates (x,y,z)

Parameters

Name	Located in	Description	Required	Schema
user	query	Make furhat attend a user. Example 'CLOSEST'	No	[ ]
userid	query	Make furhat attend specified user	No	string
location	query	Make furhat attend location, usage: x,y,z. Example -20.0,-5.0,23.0	No	string

Responses

Code	Description	Schema
200	Successful operation	Status

/furhat/face

POST

Summary:

Changes the appearance of the robot, using either character and mask, or texture and model.

Description:

Changes both the character and mask (FaceCore face engine), or texture and model (deprecated OpenSceneGraph face engine), based on the mask/character or model/texture name. Case sensitive. Names can be retrieved from the web interface.

Parameters

Name	Located in	Description	Required	Schema
mask	query	Change the mask of the robot	No	string
character	query	Change the character of the robot	No	string
model	query	Change the model of the robot	No	string
texture	query	Change the texture of the robot	No	string

Responses

Code	Description	Schema
200	Successful operation	Status

/furhat/visibility

POST

Summary:

Fade in/out the face

Description:

Triggers an animation which fades the face out to black, or in again, with a set duration in the range of [0,10000] ms. Invalid input values will be coerced in, and missing durations will be set to the default value of 2000 ms. This command is only applicable to the FaceCore face engine.

Parameters

Name	Located in	Description	Required	Schema
visible	query	Whether the face should be made visible or not	Yes	boolean
duration	query	Duration of the fade animation in milliseconds	No	integer

Responses

Code	Description	Schema
200	Successful operation	Status

/furhat/gesture

POST

Summary:

Perform a gesture

Description:

Performs a gesture based on

Gesture name (retrieve by GET request to /furhat/gestures)
Gesture definition, see example

Parameters

Name	Located in	Description	Required	Schema
name	query	The gesture to do	No	string
blocking	query	Whether to block execution before completion	No	boolean
body	body	Definition of the gesture	No	GestureDefinition

Responses

Code	Description	Schema
200	Successful operation	Status
400	Parameters are wrong	Status

/furhat/listen

GET

Summary:

Make the robot listen, and get speech results

Description:

Blocking call to get user speech input, language defaults to english_US. Language parameter can be used to provide a different language. Return values can be found in the Status object as message and can be:

User speech
SILENCE
INTERRUPTED
FAILED

Parameters

Name	Located in	Description	Required	Schema
language	query	The language to listen for, defaults to en-US	No	string

Responses

Code	Description	Schema
200	Successful operation	Status

/furhat/listen/stop

POST

Summary:

Make the robot stop listening

Description:

Aborts the listen

Responses

Code	Description	Schema
200	Successful operation	Status

/furhat/led

POST

Summary:

Change the colour of the LED strip

Description:

Changes the LED strip of the robot, colours can be between 0-255 (above 255 is changed to 255). Any parameter not provided defaults to 0.

Parameters

Name	Located in	Description	Required	Schema
red	query	The amount of red	No	integer
green	query	The amount of green	No	integer
blue	query	The amount of blue	No	integer

Responses

Code	Description	Schema
200	Successful operation	Status

/furhat/say/stop

POST

Summary:

Make the robot stop talking

Description:

Stops the current speech.

Responses

Code	Description	Schema
200	Successful operation	Status

Models

Status

Name	Type	Description	Required
success	boolean		No
message	string		No

Gesture

Name	Type	Description	Required
name	string		No
duration	integer (double)		No

User

Name	Type	Required
id	string	No
rotation	Rotation	No
location	Location	No

Location

Name	Type	Description	Required
x	double	Robot's left	No
y	double	Robot's forward	No
z	double	Robot's up	No

Voice

Name	Type	Description	Required
name	string		No
language	string		No

Rotation

Name	Type	Required
x	double	No
y	double	No
z	double	No

GestureDefinition

The class name needs to be furhatos.gestures.Gesture otherwise it won't be parsed as a gesture. Examples can be found below.

Name	Type	Required
name	string	No
frames	[ Frame ]	No
class	string	No

GestureDefinition examples

All BasicParams are listed here:

kotlin

    //All parameters have values between 0.0 and 1.0 (Except for the ones at the bottom).
    EXPR_ANGER
    EXPR_DISGUST
    EXPR_FEAR
    EXPR_SAD
    SMILE_CLOSED
    SMILE_OPEN
    SURPRISE
    BLINK_LEFT
    BLINK_RIGHT
    BROW_DOWN_LEFT
    BROW_DOWN_RIGHT
    BROW_IN_LEFT
    BROW_IN_RIGHT
    BROW_UP_LEFT
    BROW_UP_RIGHT

    EYE_SQUINT_LEFT
    EYE_SQUINT_RIGHT

    LOOK_DOWN
    LOOK_LEFT
    LOOK_RIGHT
    LOOK_UP

    PHONE_AAH
    PHONE_B_M_P
    PHONE_BIGAAH
    PHONE_CH_J_SH
    PHONE_D_S_T
    PHONE_EE
    PHONE_EH
    PHONE_F_V
    PHONE_I
    PHONE_K
    PHONE_N
    PHONE_OH
    PHONE_OOH_Q
    PHONE_R
    PHONE_TH
    PHONE_W

    LOOK_DOWN_LEFT
    LOOK_DOWN_RIGHT
    LOOK_LEFT_LEFT
    LOOK_LEFT_RIGHT
    LOOK_RIGHT_LEFT
    LOOK_RIGHT_RIGHT
    LOOK_UP_LEFT
    LOOK_UP_RIGHT

    //The following parameters have values in the range -50.0 to 50.0
    NECK_TILT
    NECK_PAN
    NECK_ROLL
    GAZE_PAN
    GAZE_TILT

Note that you can also use any of the FaceCore-compatible ARKitParams or CharParams, or any pre-recorded gestures, assuming the FaceCore face engine is used.

A couple of gesture examples:

Built-in BigSmile

json

{
  "name":"BigSmile",
  "frames":[
    {
      "time":[0.32,0.64],
      "persist":false, <- Optional
      "params":{
        "BROW_UP_LEFT":1,
        "BROW_UP_RIGHT":1,
        "SMILE_OPEN":0.4,
        "SMILE_CLOSED":0.7
        }
    },
    {
      "time":[0.96],
      "persist":false, <- Optional
      "params":{
        "reset":true
        }
    }],
  "class":"furhatos.gestures.Gesture"
}

Custom gesture

json

{
  "frames": [
    {
      "time": [
        0.17, 1.0, 6.0
      ],
      "params": {
        "NECK_ROLL": 25.0,
        "NECK_PAN": -12.0,
        "NECK_TILT": -25.0
      }
    },
    {
    	"time": [
    		7.0
    	],
    	"params": {
    		"reset": true
    	}
    }
  ],
  "name": "Cool Thing",
  "class": "furhatos.gestures.Gesture"
}

Frame

A list of times can be provided, at those times the params will be executed.

Name	Type	Description	Required
time	[ double ]		No
params	BasicParam		No

BasicParam

All supported parameters can be found here BasicParam.

Name	Type	Description	Required
BasicParam	object

Python Remote API

Description

To simplify the use of the Furhat Remote API from Python, there is a package on PyPi called furhat-remote-api.

Installation

You can install the package using pip:

pip install furhat-remote-api

(you may need to run pip with root permission: sudo pip install furhat-remote-api)

Usage

This shows how the different methods in the Remote API can be invoked from Python.

python

from furhat_remote_api import FurhatRemoteAPI

# Create an instance of the FurhatRemoteAPI class, providing the address of the robot or the SDK running the virtual robot
furhat = FurhatRemoteAPI("localhost")

# Get the voices on the robot
voices = furhat.get_voices()

# Set the voice of the robot
furhat.set_voice(name='Matthew')

# Say "Hi there!"
furhat.say(text="Hi there!")

# Play an audio file (with lipsync automatically added)
furhat.say(url="https://www2.cs.uic.edu/~i101/SoundFiles/gettysburg10.wav", lipsync=True)

# Listen to user speech and return ASR result
result = furhat.listen()

# Perform a named gesture
furhat.gesture(name="BrowRaise")

# Perform a custom gesture
furhat.gesture(body={
    "frames": [
        {
            "time": [
                0.33
            ],
            "params": {
                "BLINK_LEFT": 1.0
            }
        },
        {
            "time": [
                0.67
            ],
            "params": {
                "reset": True
            }
        }
    ],
    "class": "furhatos.gestures.Gesture"
    })

# Get the users detected by the robot
users = furhat.get_users()

# Attend the user closest to the robot
furhat.attend(user="CLOSEST")

# Attend a user with a specific id
furhat.attend(userid="virtual-user-1")

# Attend a specific location (x,y,z)
furhat.attend(location="0.0,0.2,1.0")

# Set the LED lights
furhat.set_led(red=200, green=50, blue=50)

Furhat Remote API DEPRECATED ​

Setup ​

Run the server on the Robot ​

Testing the remote API using Postman ​

Create a client for your specific programming language ​

API Documentation ​

API Endpoints ​

Authentication ​

Models ​

/furhat ​

GET ​

Summary: ​

Description: ​

Responses ​

/furhat/gestures ​

GET ​

Summary: ​

Description: ​

Responses ​

/furhat/voice ​

POST ​

Summary: ​

Description: ​

Parameters ​

Responses ​

/furhat/voices ​

GET ​

Summary: ​

Description: ​

Responses ​

/furhat/users ​

GET ​

Summary: ​

Description: ​

Responses ​

/furhat/say ​

POST ​

Summary: ​

Description: ​

Parameters ​

Responses ​

/furhat/attend ​

POST ​

Summary: ​

Description: ​

Parameters ​

Responses ​

/furhat/face ​

POST ​

Summary: ​

Description: ​

Parameters ​

Responses ​

/furhat/visibility ​

POST ​

Summary: ​

Description: ​

Parameters ​

Responses ​

/furhat/gesture ​

POST ​

Summary: ​

Description: ​

Parameters ​

Responses ​

/furhat/listen ​

GET ​

Summary: ​

Description: ​

Parameters ​

Responses ​

/furhat/listen/stop ​

POST ​

Summary: ​

Description: ​

Responses ​

/furhat/led ​

POST ​

Summary: ​

Description: ​

Furhat Remote API DEPRECATED

Setup

Run the server on the Robot

Testing the remote API using Postman

Create a client for your specific programming language

API Documentation

API Endpoints

Authentication

Models

/furhat

GET

Summary:

Description:

Responses

/furhat/gestures

GET

Summary:

Description:

Responses

/furhat/voice

POST

Summary:

Description:

Parameters

Responses

/furhat/voices

GET

Summary:

Description:

Responses

/furhat/users

GET

Summary:

Description:

Responses

/furhat/say

POST

Summary:

Description:

Parameters

Responses

/furhat/attend

POST

Summary:

Description:

Parameters

Responses

/furhat/face

POST

Summary:

Description:

Parameters

Responses

/furhat/visibility

POST

Summary:

Description:

Parameters

Responses

/furhat/gesture

POST

Summary:

Description:

Parameters

Responses

/furhat/listen

GET

Summary:

Description:

Parameters

Responses

/furhat/listen/stop

POST

Summary:

Description:

Responses

/furhat/led

POST

Summary:

Description: