Furhat Robotics

Appearance

Display Images via Furhat AI Creator Actions ​

This tutorial shows how to let a Furhat character display selected images by using the Action feature in AI Creator. We’ll set up a simple Flask server that serves images and updates them in real time using WebSockets.

What You’ll Learn ​

  • How to use Actions in Creator to control an external server
  • How to configure an OpenAPI schema for calling the image selection endpoint
  • How to host a live-updating image viewer in Python
  • How to define the character in Creator to reference available images

Step 1: Prepare the Python Image Server ​

Create a new folder with this Python script and an images/ subfolder containing your image files. You can add any images you like. For this tutorial we will use images of animals.

bash
images/
  - human.jpg
  - horse.jpg
  - pig.jpg
  - cow.jpg
app.py

Install the necessary packages:

python
pip install flask flask-socketio

Paste this content into app.py:

python
from flask import Flask, request, send_from_directory, render_template_string, jsonify
from flask_socketio import SocketIO
import os

app = Flask(__name__)
app.config['SECRET_KEY'] = 'secret!'
socketio = SocketIO(app, cors_allowed_origins='*')

BASE_DIR = os.path.dirname(__file__)
IMAGE_FOLDER = os.path.join(BASE_DIR, 'images')
os.makedirs(IMAGE_FOLDER, exist_ok=True)
current_image = None

HTML_TEMPLATE = '''<!doctype html>
<html><head><title>Image Viewer</title>
<script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/4.5.4/socket.io.min.js"></script>
<style>
  body { font-family: sans-serif; text-align: center; background: #f0f0f0; }
  #displayed { width: 600px; height: 400px; object-fit: contain; border: 2px solid #ccc; }
</style>
<script>
const socket = io();
socket.on('image_update', data => {
  const img = document.getElementById('displayed');
  img.src = '/display/' + data.selected;
  img.style.display = data.selected ? 'block' : 'none';
});
</script>
</head><body><h1>Selected Image</h1><img id="displayed" style="display:none;" /></body></html>
'''

@app.route('/')
def index(): return render_template_string(HTML_TEMPLATE)

@app.route('/display/<filename>')
def serve_image(filename): return send_from_directory(IMAGE_FOLDER, filename)

@app.route('/display/', methods=['GET'])
def select_image():
    global current_image
    filename = request.args.get('image')
    path = os.path.join(IMAGE_FOLDER, filename)
    if not os.path.isfile(path):
        return jsonify({'error': 'Image not found'}), 404
    current_image = filename
    socketio.emit('image_update', {'selected': current_image})
    return jsonify({'status': 'success', 'selected': filename})

if __name__ == '__main__':
    socketio.run(app, host='0.0.0.0', port=8000)

Start the server:

bash
python app.py

Open http://localhost:8000 to verify that it works.

Step 2: Define the OpenAPI Action in Creator ​

Go to Integrations > +New Action, and paste this definition:

json
{
  "servers": [
    {
      "url": "http://127.0.0.1:8000"
    }
  ],
  "paths": {
    "/display/": {
      "get": {
        "description": "Selects which image to display",
        "operationId": "SelectImage",
        "parameters": [
          {
            "name": "image",
            "in": "query",
            "description": "Filename of the image to select",
            "required": true,
            "schema": {
              "type": "string"
            }
          }
        ]
      }
    }
  }
}

This tells Furhat how to call the image selection API.

Step 3: Character Description ​

In the Character Description field, define what images Furhat can show:

Anna is a loving and caring assistant. She helps me with everyday tasks. She can display images. Here are the images she can display:

human.jpg - a picture of humans
horse.jpg - a picture of a horse
pig.jpg - a picture of a pig
cow.jpg - a picture of a cow

Example Prompt ​

You can now say:

“Anna, can you show me a picture of a cow?”

Furhat will call the Action, which updates the live image viewer.

Final Notes ​

This example is intentionally simple — it's meant to show how you can connect Furhat to a Python backend using Actions.

While we’re just displaying pictures of animals here, the real takeaway is that you can host any program that takes input and performs some action. Some practical ideas:

  • Show different areas of a venue where the robot is located.
  • Display product information or availability.
  • Trigger machinery, launch videos, or control smart devices.
  • Connect to any service, database, or visualization you can build in Python.

You're free to build any backend you like. This example is just the start.