Electronic device with automatic eye gaze tracking and camera adjustment

Patent metadata

Field	Value
Publication number	US-2023276116-A1
Application number	US-202217682388-A
Country	US
Kind code	A1
Filing date	Feb 28, 2022
Priority date	Feb 28, 2022
Publication date	Aug 31, 2023
Grant date	—

Abstract

An electronic device, a method and a computer program product for camera adjustments based on eye gaze tracking. The method includes capturing, via a front facing camera, an image stream containing a face of a user and determining, via a processor, an eye gaze direction based on images retrieved from the image stream. The eye gaze direction corresponds to a location where the user is looking. In response to determining that the user is looking away from the front surface of the electronic device and towards a direction within a field of view of a rear facing camera, the method further includes identifying image characteristics of the location, generating camera settings of the rear facing camera based on the eye gaze direction/characteristics of the location and adjusting the rear facing camera using the generated camera settings such that the rear facing camera focuses on the location.

First claim

What is claimed is:

1 . An electronic device comprising:
a front surface and a rear surface; at least one front facing camera and at least one rear facing camera. a memory having stored thereon a camera selection and control module (CSCM) for controlling the at least one front facing camera and the at least one rear facing camera; and at least one processor communicatively coupled to each of the at least one front facing camera, each of the at least one rear facing camera, and to the memory, the at least one processor executing program code of the CSCM, which enables the electronic device to:
capture, via the at least one front facing camera, a first image stream containing a face of a first user;
determine a first eye gaze direction of the first user based on a first image retrieved from the first image stream, the first eye gaze direction corresponding to a first location where the first user is looking; and
in response to determining that the first user is looking away from the front surface of the electronic device and towards a direction within a field of view (FOV) of the at least one rear facing camera:
identify image characteristics of the first location;
generate at least one camera setting of a selected rear facing camera at least partially based on the first eye gaze direction and identified characteristics of the first location;
adjust the selected rear facing camera using the generated at least one camera setting such that the selected rear facing camera focuses on the first location; and
activate the selected rear facing camera as an active camera.

2 . The electronic device of claim 1 , wherein the at least one processor:
tracks the eye gaze direction by monitoring and periodically analyzing the first image corresponding to the first image stream to identify a current eye gaze direction; determines if the current eye gaze direction is towards a current location that is different from the first location; in response to determining that the current location is different from the first location, determines, based on the current eye gaze direction, if the first user is looking away from the front surface of the electronic device; and in response to determining that the first user is looking away from the front surface:
identifies image characteristics of the current location;
generates at least one new camera setting of the active camera based on the current eye gaze direction and identified characteristics of the current location; and
adjusts the active camera using the generated at least one new camera setting such that the at least one rear facing camera focuses on the current location.

3 . The electronic device of claim 1 , wherein to generate the at least one camera setting of the at least one rear facing camera based on a current eye gaze direction, the at least one processor is further enabled to:
process the current eye gaze direction through an artificial intelligence engine, which generates the at least one camera setting for the selected rear facing camera based at least partially on the current eye gaze direction; and adjust at least a directional setting of the selected rear facing camera to focus the FOV of the active camera on a current location determined from the current eye gaze detection.

4 . The electronic device of claim 1 , wherein to generate the at least one camera setting of the selected rear facing camera based on a current eye gaze direction, the at least one processor is further enabled to:
retrieve, from the memory, first camera parameters and first camera settings for the selected rear facing camera; and generate the at least one second camera setting of the selected rear facing camera at least partially based on the first eye gaze direction, the identified image characteristics of the first location, the first camera parameters, and the first camera settings.

5 . The electronic device of claim 1 , wherein the generated at least one second camera setting comprises at least one of:
a zoom level; a focus distance; a directional setting; a digital crop level; a focal distance of the focus module; a directional audio zoom; a modified electronic image stabilization; and an exposure compensation.

6 . The electronic device of claim 1 , wherein the identified image characteristics of the first location comprises at least one of:
light levels at the first location; face detection at the first location; scene detection at the first location; contrast detection autofocus at the first location; phase detection autofocus at the first location; object of interest detection using computer vision at the first location; object of interest detection using machine learning at the first location; and object of interest detection using artificial intelligence at the first location.

7 . The electronic device of claim 1 , further comprising:
at least one input device communicatively coupled to the at least one processor; and the at least one processor:
receives first input from the at least one input device;
determines a first context based on the first input received, the first context associated with at least one camera setting using the selected rear facing camera as the active camera to capture images;
generates at least one camera setting for the active camera based at least partially on the first context; and
adjusts the active camera using the generated at least one camera setting.

8 . The electronic device of claim 7 , wherein:
the at least one input device comprises an audio input device and the first input is audio input captured by the audio input device; and the at least one processor processes the audio input via a natural language processor/context evaluation (NLP/CE) engine to determine the first context.

9 . A method comprising:
capturing, via at least one front facing camera of an electronic device, a first image stream containing a face of a first user; determining, via at least one processor, a first eye gaze direction of the first user based on a first image retrieved from the first image stream, the first eye gaze direction corresponding to a first location where the first user is looking; and in response to determining that the first user is looking away from a front surface of the electronic device and towards a direction within a field of view (FOV) of at least one rear facing camera of the electronic device: identifying image characteristics of the first location; generating at least one camera setting of a selected rear facing camera at least partially based on the first eye gaze direction and identified characteristics of the first location; adjusting the selected rear facing camera using the generated at least one camera setting such that the selected rear facing camera focuses on the first location; and activating the selected rear facing camera as an active camera.

10 . The method of claim 9 , further comprising:
tracking the eye gaze direction by monitoring and periodically analyzing the first image corresponding to the first image stream to identify a current eye gaze direction; determining if the current eye gaze direction is towards a current location that is different from the first location; in response to determining that the current location is different from the first location, determining, based on the current eye gaze direction, if the first user is looking away from the front surface of the electronic device; and in response to determining that the first user is looking away from the front surface:
identifying image characteristics of the current location;
generating at least one new camera setting of the active camera based on the current eye ga

Assignees

• Motorola Mobility Llc

Inventors

• Agrawal Amit Kumar
• Desai Rahul Bharat

Classifications

• H04N23/45

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• G06F3/167 Primary

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04N23/611 Primary

  where the recognised objects include parts of the human body · CPC title

• H04N5/23219 Primary

  Electricity · mapped topic