Tutorial: External object recognition


In this tutorial, we will walk through the steps needed to take advantage of the camera feed feature available for Furhat robots designated for research. See Camera Feed section for more info.

After following this tutorial you will be able to connect an external computer to the camera feed of your Furhat robot, run a basic object recognition externally and send the recognised object info back to the skill running on the robot.

This tutorial assumes that you have followed the getting started guide and creating your first skill tutorial. You also need a research designated Furhat robot connected to your local network and a computer connected to the same network that will be used for the external camera feed processing.

Download the code

On your workstation, clone the camerafeed-demo repository from GitHub:

git clone https://github.com/FurhatRobotics/camerafeed-demo.git

In this repository you will find:

  • A folder ObjectTalk which contains the source code for the skill that will run on the robot and communicate with the external calculation machine.
  • A folder object-detection-server that contains the code needed for the external object recognition.
  • A README.md file with short instructions.
  • A requirements.txt file listing the dependencies.

Prepare the workstation for external object recognition

NOTE: The following steps assume the workstation is running Ubuntu 18.04 LTS or newer

  • Make sure you have Python3 and wget installed sudo apt install python3 wget
  • Install pip for Python3 using python3 -m pip install --user --upgrade pip to get the latest version of pip3.
  • Navigate to the git repository folder and install the required dependencies pip3 install -r requirements.txt for Python3.
  • Navigate into the object-detection-server folder and download the Yolov3 weights. Note: this will take quite some time (236 MB) ! wget https://pjreddie.com/media/files/yolov3.weights
  • You now need to alter the launch.json file to include IP addresses of your Furhat robot and workstation. You will find the Furhat IP address on the face menu. Workstation local network IP can be found by hostname -I for example.

Prepare and build the skill

  • Open main.kt file in ObjectTalk/src/main/kotlin/furhatos/app/objectidentifier and change the objserv value to point to your workstation IP address.
  • Check that the build.gradle file in the skill project root is pointing to the furhat-commons version that matches your robot software wersion.
  • Build the skill by first navigating in to the skill folder cd ObjectTalk and then running the skill build job ./gradlew shadowJar
  • The .skill file can be found in the build/libs folder.
  • Import the .skill file using the robot web interface that can be accessed in your workstation browser by navigating to the robot IP address.

Test out the object recognition

  • First start the skill on the robot.
  • Then on the workstation navigate to the object-detection-server folder and run the server object-detection-server

A new window will open on your workstation displaying the camera feed, the bounding boxes of identified objects and their names.

Whenever you show an object to Furhat that it recognizes, it will speak the name of the object(s).