Gaze-driven augmented reality

Patent metadata

Field	Value
Publication number	US-11250263-B2
Application number	US-201815924956-A
Country	US
Kind code	B2
Filing date	Mar 19, 2018
Priority date	Oct 11, 2013
Publication date	Feb 15, 2022
Grant date	Feb 15, 2022

Abstract

Augmented reality (AR) systems, methods, and instrumentalities are disclosed. A user's gaze point may be estimated and may be used to search for and present information, e.g., information relating to areas on which the user is focusing. The user's gaze point may be used to facilitate or enable modes of interactivity and/or user interfaces that may be controlled by the direction of view of the user. Biometric techniques may be used to estimate an emotional state of the user. This estimated emotional state may be used to be the information that is presented to the user.

First claim

What is claimed is:

1. A method for providing augmented reality (AR) to a user, comprising:
estimating a gaze point of the user to define a first region of interest (ROI);
adaptively adjusting the size of the first ROI based upon one or more of a gaze focused for a time on a portion of the first ROI exceeding a threshold or a number of objects of interest in the first ROI exceeding a threshold, wherein the adjusted ROI contains one or more physical objects currently in proximity to the user;
extracting images from the adjusted ROI and creating a set of descriptors which describe the one or more physical objects without input from the user;
estimating an emotional state of the user;
determining a subset of descriptors by comparing the set of descriptors to the estimated emotional state of the user and giving greater weight to descriptors associated with an emotional state closest to the estimated emotional state of the user, wherein representative emotional states are mapped to descriptors of objects, such that objects consistent with the estimated emotional state of the user are prioritized;
without further input from the user, searching for information regarding the one or more physical objects in the adjusted ROI using the subset of descriptors;
retrieving the searched information prioritized by the subset of descriptors; and
displaying the retrieved information.

2. The method of claim 1 , further comprising estimating a second emotional state of the user upon viewing the retrieved information prioritized by the subset of descriptors regarding the one or more physical objects.

3. The method of claim 2 , further comprising revising the subset of descriptors if the second emotional state of the user is a negative emotional state.

4. The method of claim 1 , wherein the set of descriptors specify features of the one or more physical objects.

5. The method of claim 1 , wherein the set of descriptors identify the one or more physical objects.

6. The method of claim 1 , wherein the subset of descriptors is included in a retrieval request to a server.

7. The method of claim 1 , wherein the subset of descriptors is used for a local search.

8. The method of claim 1 , wherein the emotional state of the user is determined by estimating a point for the emotional state on a valence/arousal chart.

9. The method of claim 1 , further comprising using one or more of voice recognition of an emotion of the user, facial recognition of an emotion of the user, and/or electroencephalographic (EEG) data of the user, to determine the emotional state of the user.

10. The method of claim 1 , wherein the information regarding one or more physical objects in the adjusted ROI received from a server is less than the total information available for the one or more physical objects in the adjusted ROI.

11. A wireless transmit/receive unit (WTRU) for providing augmented reality (AR) to a user, comprising:
a power source;
a display/touchpad;
a global positioning system (GPS) chipset;
a camera;
a memory; and
a processor to execute instructions from the memory, the processor configured to:
estimate a gaze point of the user to define a region of interest (ROI), the ROI being determined by a gaze of the user being focused for a time exceeding a threshold, wherein the ROI contains one or more physical objects currently in proximity to the user;
extract images from the ROI and create a set of descriptors which describe the one or more physical objects;
estimate an emotional state of the user, wherein the emotional state of the user is that the user is tired, alert, or hungry;
determine a subset of descriptors by comparing the set of descriptors to the estimated emotional state of the user and giving greater weight to descriptors associated with an emotional state closest to the estimated emotional state of the user, wherein representative emotional states are mapped to descriptors of objects, such that objects consistent with the estimated emotional state of the user are prioritized;
without further input from the user, search for information regarding the one or more physical objects in the ROI using the subset of descriptors;
retrieve the searched information prioritized by the subset of descriptors; and
display the retrieved information.

12. The WTRU of claim 11 , wherein the processor configured to estimate a second emotional state of the user upon viewing the retrieved information prioritized by the subset of descriptors regarding the one or more physical objects, and to revise the subset of descriptors if the second emotional state of the user is a negative emotional state.

13. The WTRU of claim 11 , wherein the set of descriptors specify features of the one or more physical objects.

14. The WTRU of claim 11 , wherein the set of descriptors identify the one or more physical objects.

15. The WTRU of claim 11 , wherein the subset of descriptors is included in a retrieval request to a server.

16. The WTRU of claim 11 , wherein the subset of descriptors is used for a local search.

17. The WTRU of claim 11 , wherein the emotional state of the user is determined by estimating a point for the emotional state on a valence/arousal chart.

18. The WTRU of claim 11 , wherein the processor configured to use one or more of voice recognition of an emotion of the user, facial recognition of an emotion of the user, and/or electroencephalographic (EEG) data of the user, to determine the emotional state of the user.

19. A method for providing augmented reality (AR) to a user, comprising:
estimating a gaze point of the user to define a region of interest (ROI), the ROI being determined by a gaze of the user being focused for a time exceeding a threshold, wherein the ROI contains one or more physical objects currently in proximity to the user;
extracting images from the ROI and creating a set of descriptors which describe the one or more physical objects;
estimating an emotional state of the user;
determining a subset of descriptors by comparing the set of descriptors to the estimated emotional state of the user and giving greater weight to descriptors associated with an emotional state closest to the estimated emotional state of the user, wherein representative emotional states are mapped to descriptors of objects, such that objects consistent with the estimated emotional state of the user are prioritized;
without further input from the user, searching for information regarding the one or more physical objects in the ROI using the subset of descriptors and retrieving the searched information;
displaying the retrieved information prioritized by the subset of descriptors regarding the one or more physical objects; and
thereafter, estimating a second emotional state of the user upon viewing the retrieved information prioritized by the subset of descriptors regarding the one or more physical objects.

20. A wireless transmit/receive unit (WTRU) having a memory and a processor to execute instructions from the memory, which, when executed, cause the WTRU to carry out the method of claim 19 .

21. A method for providing augmented reality (AR) to a user with a WTRU having a power source, a display/touchpad, a global positioning system (GPS) chipset, a camera, a memory, and a processor to execute instructions from the memory, the method comprising:
estimating a gaze point of the user to define a region of interest (ROI), the ROI being determined by a gaze of the user being focused for a time exceeding a threshold, wherein the ROI

Assignees

• Interdigital Patent Holdings Inc

Inventors

• Asbun Eduardo
• Reznik Yuriy
• Zeira Ariela
• Sternberg Gregory S
• Neff Ralph

Classifications

• G06V40/174

  Facial expression recognition · CPC title

• G06V40/19

  Sensors therefor · CPC title

• G06V20/20 Primary

  in augmented reality scenes · CPC title

• G06F3/015

  Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection · CPC title

• A61B5/369

  Electroencephalography [EEG] · CPC title