Interest maintaining system and server

Patent metadata

Field	Value
Publication number	US-10997400-B2
Application number	US-201916368187-A
Country	US
Kind code	B2
Filing date	Mar 28, 2019
Priority date	Sep 29, 2016
Publication date	May 4, 2021
Grant date	May 4, 2021

Abstract

A childcare support service offering system includes a terminal and a server. The terminal includes a camera and performs operation control and processing of transmitting terminal usage situation data including image data acquired by being picked up by the camera to the server. The server performs image analysis on the received image data, recognizes the user that watches the terminal, judges a gaze of the recognized user, and judges a degree of interest for the terminal based on the gaze. When the judged degree of interest becomes equal to or less than a predetermined threshold value, the server transmits a changing instruction for causing an operation in the terminal to be changed to another operation to the terminal.

First claim

What is claimed is:

1. An interest maintaining system, comprising:
a terminal; and
a server,
the interest maintaining system being configured to maintain an interest of a user for the terminal, wherein:
the terminal comprises:
a first communication unit;
a camera;
an operation control unit configured to control a first operation in the terminal in accordance with data or an instruction from the server received via the first communication unit; and
a transmission processing unit configured to perform processing of transmitting terminal usage situation data comprising image data acquired by being picked up by the camera to the server by the first communication unit; and

the server comprises:
a second communication unit
a face recognition unit configured to perform image analysis on the image data received via the second communication unit and recognize a face of the user that watches the terminal;
an eyeball position determination unit configured to determine positions of two eyeballs from an image of the face of the user recognized in the face recognition unit;
a gaze direction judgement unit configured to judge a gaze direction of the user based on the image of the face recognized in the face recognition unit and information on the positions of the two eyeballs determined in the eyeball position determination unit;
a blink judgement unit configured to judge a blink of an eye of the user recognized in the face recognition unit;
an interest degree judgement unit configured to judge a degree of interest for the terminal based on the determined positions of the two eyeballs, the judged gaze direction, and the judged blink of the eye; and
an operation changing instruction transmission unit configured to transmit a changing instruction for causing the first operation in the terminal to be changed to a second operation to the terminal by the second communication unit when the degree of interest judged in the interest degree judgement unit is equal to or less than a predetermined threshold value.

2. The interest maintaining system according to claim 1 , wherein the degree of interest is defined by a length or a rate of a time period in which the gaze is toward the terminal for every predetermined time period.

3. The interest maintaining system according to claim 1 , wherein the server comprises:
an event judgement unit configured to judge whether a predetermined event occurs based on the image data; and
a notification unit configured to transmit a predetermined notification when it is judged by the event judgement unit that the predetermined event occurs.

4. The interest maintaining system according to claim 3 , wherein:
the terminal comprises a microphone;
the usage situation data comprises voice data acquired by the microphone; and
the event judgement unit judges whether the predetermined event occurs based on the image data and the voice data received via the second communication unit.

5. The interest maintaining system according to claim 3 , wherein the predetermined notification is a notification for a predetermined apparatus for causing the predetermined apparatus to perform processing of changing the first operation or the second operation in the terminal to a third operation.

6. The interest maintaining system according to claim 1 , wherein:
the terminal comprises a display unit configured to display a video;
the first operation is an operation of displaying a first video content on the display unit; and
the second operation is an operation of displaying a second video content on the display unit.

7. The interest maintaining system according to claim 6 , wherein the first video content and the second video content are transmitted from the server.

8. The interest maintaining system according to claim 1 , wherein:
the terminal is a robot; and
the first operation and the second operation are a motion or a voice output of the robot.

9. The interest maintaining system according to claim 1 , wherein:
the terminal is a vacuum cleaner configured to self-travel on a floor; and
the first operation and the second operation are a motion or a voice output of the vacuum cleaner.

10. A server for an interest maintaining system,
the interest maintaining system being configured to maintain an interest of a user for a terminal,
the server comprising:
a communication unit;
a face recognition unit configured to perform image analysis on an image data from the terminal received via the communication unit and recognize a face of the user that watches the terminal;
an eyeball position determination unit configured to determine positions of two eyeballs from an image of the face of the user recognized in the face recognition unit;
a gaze direction judgement unit configured to judge a gaze direction of the user based on the image of the face recognized in the face recognition unit and information on the positions of the two eyeballs determined in the eyeball position determination unit;
a blink judgment unit configured to judge a blink of an eye of the user recognized in the face recognition unit;
an interest degree judgement unit configured to judge a degree of interest for the terminal based on the determined positions of the two eyeballs, the Judged gaze direction, and the judged blink of the eye; and
an operation changing instruction transmission unit configured to transmit a changing instruction for causing a first operation in the terminal to be changed to a second operation to the terminal by the communication unit when the degree of interest judged in the interest degree judgement unit is equal to or less than a predetermined threshold value.

11. The server according to claim 10 , wherein the degree of interest is defined by a length or a rate of a time period in which the gaze is toward the terminal for every predetermined time period.

12. The server according to claim 10 , wherein the server comprises:
an event judgement unit configured to judge whether a predetermined event occurs based on the image data; and
a notification unit configured to transmit a predetermined notification when it is judged by the event judgement unit that the predetermined event occurs.

13. The server according to claim 12 , wherein the event judgement unit judges whether the predetermined event occurs based on the image data and voice data received via the communication unit.

14. The server according to claim 12 , wherein the predetermined notification is a notification for a predetermined apparatus for causing the predetermined apparatus to perform processing of changing the first operation or the second operation in the terminal to a third operation.

15. An interest maintaining system comprising:
a terminal, and
a server;
the interest maintaining system being configured to maintain an interest of a user for the terminal, wherein
the terminal comprises:
a first communication unit;
a camera;
a microphone;
an operation control unit configured to control a first operation in the terminal in accordance with data or an instruction from the server received via the first communication unit; and
a transmission processing unit configured to perform processing of transmitting image data acquired by being picked up by the camera and voice data acquired by the microphone to the server by the first communication unit;

and the server comprises:
a second communication unit;
a face recognition unit configured to perform image analysis on the image data received via the second communication unit and recognize a face of the user that watches the terminal;
an eyeball position determination un

Assignees

• Toshiba Kk
• Toshiba Digital Solutions Corp

Inventors

• Iwasaki Motokazu
• FUKATSU Genki
• Shimada Kazuhiro
• NAKAMURA Keisaku
• Kikuchi Satoko
• Watabiki Masaru
• ASAUMI Yuuka
• MIYAWAKI Kanako
• KASHITA Masakazu
• IUCHI Shinsuke

Classifications

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06V40/20 Primary

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06V40/193

  Preprocessing; Feature extraction · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title