Eye contact sensing and control for robotic characters

Patent metadata

Field	Value
Publication number	US-2021122055-A1
Application number	US-201916662151-A
Country	US
Kind code	A1
Filing date	Oct 24, 2019
Priority date	Oct 24, 2019
Publication date	Apr 29, 2021
Grant date	—

Abstract

A system for sensing and controlling eye contact for a robot. The system includes a robotic figure with a movable eye. The system includes a light source positioned in the robotic figure to output light through a light outlet of the eye. A light sensor is included that senses light striking surfaces in a physical space in which the robotic figure is positioned including the output light from the light source. The system includes an image processor processing output of the light sensor to identify a location of a target formed by the output light striking surfaces in the physical space and to identify a location of a face of a human. Further, the system includes a robot controller generating eye movement control signals based on the location of the target and the location of the face to position the eye to provide eye contact with the human observer.

First claim

We claim:

1 . A system for sensing and controlling eye contact with a robot, comprising:
a robotic figure with a movable eye with a front light outlet; a light source positioned in the robotic figure to output light through the front light outlet of the movable eye; a light sensor sensing light striking surfaces in a physical space in which the robotic figure is positioned including the output light from the light source; an image processor processing output of the light sensor to identify a location of a target formed by the output light striking one or more surfaces in the physical space and to identify a location of a face of a human in the physical space; and a robot controller generating eye movement control signals for the robotic figure based on the location of the target and the location of the face to position the movable eye to provide eye contact with the human.

2 . The system of claim 1 , wherein the light sensor is positioned external to the robotic figure.

3 . The system of claim 2 , wherein the light sensor comprises a first camera detecting an image of the physical space using the output light and a second camera detecting an image of the physical space using visible light and wherein the image from the first camera is used by the image processor to identify the target and the image from the second camera is used by the image processor to identify the face.

4 . The system of claim 3 , further including a beamsplitter positioned between the first and second cameras to receive light reflected from the physical space including a portion of the output light and to transmit a first portion of the received light to the first camera and to reflect a second portion of the receive light to the second camera.

5 . The system of claim 1 , wherein the light source is an infrared (IR) light source and the light sensor comprises an IR camera and wherein the IR light source is mounted within an inner space of the movable eye to move with the movable eye or is mounted external to the movable eye and directs light onto a mirror positioned in the inner space.

6 . The system of claim 1 , wherein the light source is a laser and the light sensor comprises a camera configured to detect light from the laser and wherein the laser is mounted within an inner space of the movable eye to move with the movable eye or is mounted external to the movable eye and directs light onto a mirror positioned in the inner space.

7 . The system of claim 1 , wherein the eye movement control signals cause an eye movement driver of the robot to position the movable eye to direct the front light outlet of the movable eye toward the location of the face, whereby the target at least partially overlaps the face.

8 . The system of claim 7 , wherein the eye movement control signals operate the eye movement driver to position the movable eye to direct the front light outlet of the movable eye such that the target at least periodically coincides with a location of an eye on the face.

9 . The system of claim 8 , wherein the eye movement control signals further include signals causing the eye movement driver to, during the eye contact with the human, move the movable eye to provide at least one of saccades, smooth pursuit movements, vergence movements, and vestibulo-ocular movements during movements of a head of the robot containing the movable eye.

10 . The system of claim 9 , wherein the signals causing the eye movement driver to, during the eye contact with the human, move the movable eye to provide at least one of saccades, smooth pursuit movements, vergence movements, and vestibulo-ocular movements during movements of a head of the robot containing the movable eye are generated based on values processed by an attention engine, the values being defaults values or defined by human input to match a character or a personality type.

11 . The system of claim 1 , wherein the light source and the light sensor are mounted within the eye, wherein the surfaces in a physical space include portions of two eyes of the human reflecting light from the light source back to the light sensor, and wherein the face is identified by the image processor by identifying presence of the two eyes.

12 . A system for sensing and controlling eye contact with a robot, comprising:
a robot with a head including first and second eyes positionable with eye movement drivers; a light source positioned in the first eye of the robot and mounted for moving with the first eye and for outputting light from the first eye along a center axis of the first eye; a first camera capturing an image of a space in field of view of the robot based on the light from the first eye; a second camera capturing an image of the space based on visible light from a source external to the robot, wherein a human observer is positioned in the space; an image processor processing the images from the first and second cameras to locate a targeted area in the space illuminated by the light from the first eye and a face of the human observer; and a controller running an eye movement module generating control signals for the eye movement drivers to cause the targeted area illuminated by the light from the first eye to move at least partially onto the face of the human observer.

13 . The system of claim 12 , wherein the control signals are generated to cause the targeted area to be moved onto a portion of the face of the human observer determined by the image processor to contain eyes of the human observer.

14 . The system of claim 13 , wherein the control signals are generated to move the first and second eyes of the robot to move the targeted area away from and back to the portion of the face of the human observer over an operating time period.

15 . The system of claim 12 , wherein the light source is an infrared (IR) light source and the first camera comprises an IR camera or wherein the light source is a laser and the light sensor comprises a camera configured to detect light from the laser.

16 . The system of claim 12 , wherein the control signals include signals causing the eye movement drivers to, during eye contact with the human observer, move the first and second eyes to provide at least one of saccades, smooth pursuit movements, vergence movements, and vestibulo-ocular movements during movements of the head of the robot containing the first and second eyes.

17 . An eye contact method for controlling a robot, comprising:
operating a light source in a robotic figure to output light from a movable eye; sensing illumination of a target area by the output light in a physical scene; detecting a location of an object in the physical scene; determining the robot is to attend to the object; and moving the movable eye to move the target area to the location of the object.

18 . The method of claim 17 , wherein the object is a human face and wherein the location of the object includes an area of the human face including a human eye to provide eye contact.

19 . The method of claim 18 , wherein the determining the robot is to attend to the object comprises:
receiving sensor data including a plurality of identified objects in the physical scene including the object; assigning a base value to each of the identified objects; and selecting the object based on a comparison of the assigned base values.

20 . The method of claim 19 , further comprising determining properties of a subset of the identified objects, wherein the properties are each associated with an increased value,

Assignees

• Disney Entpr Inc

Inventors

• Christensen David Loyal
• Wieland Alexis P
• Smoot Lanny S
• Ayala Alfredo M
• Smithwick Quinn Y
• Mcintosh Kyna P
• Kennedy James R
• Pan Matthew K
• Niemeyer Günter D
• Campos Zamora Daniel De Jesus
• Choi Sungjoon

Classifications

• B25J11/0015 Primary

  Face robots, animated artificial faces for imitating human expressions · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06V10/147

  Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19) · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06V40/19

  Sensors therefor · CPC title