Situation-aware robot

Patent metadata

Field	Value
Publication number	US-11040441-B2
Application number	US-201816137153-A
Country	US
Kind code	B2
Filing date	Sep 20, 2018
Priority date	Sep 20, 2018
Publication date	Jun 22, 2021
Grant date	Jun 22, 2021

Abstract

A robot in a location interacts with a user. The robot includes a camera, an image recognition processor, a microphone and a loudspeaker, a voice assistant, and a wireless transceiver. The robot moves around and creates a model of the location, and recognizes changes. It recognizes objects of interest, beings, and situations. The robot monitors the user and recognizes body language and gesture commands, as well as voice commands. The robot communicates with the user, the TV, and other devices. It may move around to monitor for regular and non-regular situations. It anticipates user commands based on a situation. It determines if a situation is desired, and mitigates the situation if undesired. It can seek immediate help for the user in an emergency. It can capture, record, categorize and document events as they happen. It can categorize and document objects in the location.

First claim

We claim:

1. A robot capable of interacting with a user in a location, the robot comprising:
(a) a camera configured to capture local images;
(b) an image recognition processor coupled to the camera;
(c) a microphone configured to capture local sounds and a loudspeaker;
(d) a voice assistant coupled to the microphone and the loudspeaker; and
(e) a wireless transceiver capable of performing two-way communication, wherein the wireless transceiver at least one of communicates over a radio-frequency (RF) electromagnetic field and communicates optically;
wherein the robot is configured to:
(i) autonomously move around in the location;
(ii) create a model of the location and recognize changes in the location;
(iii) recognize objects of interest and situations based on results from the image recognition processor, wherein:
a. a situation is a regular situation, and wherein the robot is configured to anticipate one or more commands associated with the regular situation;
b. an anticipated command includes switching a TV to user-preferred content;
c. the robot executes and/or controls a method with the following steps:
(1) waiting by an entrance of the location and determining if the user enters:
(2) upon determining that the user enters, turning on the TV and forwarding a user profile to the TV; and
(3) on the TV, switching on content based on user preferences stored in the user profile;

(iv) recognize beings based on at least one of image recognition results from the image recognition processor and voice recognition, wherein the beings include at least one of the user, a family member, a friend, an acquaintance, a visitor, a coworker, another human, a pet, and another animal;
(v) recognize the beings based on results from the voice assistant;
(vi) monitor the user;
(vii) monitor camera images and recognize user body language and gesture commands using the image recognition processor;
(viii) monitor sounds and recognize voice commands using the voice assistant;
(ix) communicate with the user via at least one of the voice assistant and the wireless transceiver;
(x) communicate with and control a television (TV) via the wireless transceiver;
(xi) using one of a TV display and a handheld device display as a primary display to communicate with the user; and
(xii) communicate with, receive information from, and control other devices via the wireless transceiver, wherein the other devices include at least one of a television, a camera, a home alarm system, a home appliance, consumer electronics, a telephone, lighting, a heater, and an air conditioner.

2. The robot of claim 1 , wherein moving around the location further comprises monitoring the location for situations.

3. The robot of claim 2 , wherein a situation is one of user-defined, and automatically defined based on artificial intelligence learning techniques.

4. The robot of claim 1 , wherein a situation is a non-regular situation, and wherein the robot is configured to validate with the user if the situation is desired.

5. The robot of claim 4 , wherein the robot is configured to mitigate the situation if it is not desired.

6. The robot of claim 5 , wherein a situation is an emergency, and the robot is configured to seek immediate help for the user.

7. The robot of claim 1 , wherein a situation is a non-regular situation, and wherein the robot is configured to capture and record events as they happen.

8. The robot of claim 7 , wherein the situation includes one of a birthday, a party, a child's first steps, a first day of school, a holiday, a change over time, a visit, and a burglary.

9. The robot of claim 7 , wherein the robot categorizes and documents the situation.

10. The robot of claim 9 , wherein the robot is configured to periodically present key events to the user.

11. The robot of claim 10 , wherein the robot presents the key events using a built-in projector.

12. The robot of claim 1 , wherein the information includes at least one of a remote image, audio, a text, sensor information, and metadata.

13. The robot of claim 1 , wherein moving around the location further comprises categorizing and documenting objects.

14. A method for a robot to interact with a TV, comprising the following steps:
receiving one or more data streams, wherein the data streams include at least one of video from a camera, audio from a microphone, and data from another sensor configured to provide data to the robot, and wherein a source of the one or more data streams is included in one of the robot, the TV, and an external device;
recording at least one of the one or more data streams;
analyzing at least one of the one or more data streams to recognize at least one of an object of interest, a being, and a situation, and wherein analyzing a video stream includes using an image recognition processor;
selecting one of a recognized object of interest, a recognized being, and a recognized situation, and determining a status of the selected recognized item;
displaying the status on the TV;
inviting a user to command an action based upon the status;
upon receiving a user command, determining if the status must be changed; and
upon determining that the status must be changed, changing the status.

15. The method of claim 14 , wherein:
the object of interest is one of commonly found in a user household and particular to the user;
the being is one of the user, a family member, a friend, an acquaintance, a visitor, a coworker, another human, a pet, and another animal; and
changing the status is performed directly by the robot, or indirectly by the TV.

16. The method of claim 14 , further comprising using a processor in the TV as an additional resource for analyzing the at least one of the one or more data streams.

Assignees

• Sony Corp
• Sony Group Corp

Inventors

• Young David
• Miller Lindsay
• Wingo Lobrenzo
• Demerchant Marvin

Classifications

• B25J11/0005 Primary

  Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• H04L12/282

  based on user interaction within the home (receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204) · CPC title

• B25J11/008

  Manipulators for service tasks · CPC title

• B25J9/1697

  Vision controlled systems · CPC title