Enabling augmented reality using eye gaze tracking

Patent metadata

Field	Value
Publication number	US-9996150-B2
Application number	US-201313838467-A
Country	US
Kind code	B2
Filing date	Mar 15, 2013
Priority date	Dec 19, 2012
Publication date	Jun 12, 2018
Grant date	Jun 12, 2018

Abstract

Methods and apparatus relating to enabling augmented reality applications using eye gaze tracking are disclosed. An exemplary method according to the disclosure includes displaying an image to a user of a scene viewable by the user, receiving information indicative of an eye gaze of the user, determining an area of interest within the image based on the eye gaze information, determining an image segment based on the area of interest, initiating an object recognition process on the image segment, and displaying results of the object recognition process.

First claim

The invention claimed is:

1. A method for enabling object recognition using eye gaze tracking, comprising:
displaying, on a display screen, an image of a scene viewable by a user;
receiving information indicative of an eye gaze of the user;
determining an area of interest within the image based on the eye gaze information;
displaying, on the display screen, a tracking region representing the area of interest;
determining an image segment based on the area of interest, the image segment being associated with an initial position of an object in the scene, wherein the image segment is determined when the tracking region is within a threshold proximity of the object;
initiating an object recognition process on the image segment, wherein the object recognition process is limited to an area defined by the image segment;
displaying an object tag on the display screen if a result of the object recognition process indicates that augmented reality information is available, wherein the object tag remains at a fixed position relative to the initial position of the object associated with the object tag and relative to the display screen for a plurality of images of the scene, and wherein the object tag automatically remains at the fixed position in the plurality of images as the object associated with the object tag is displayed at different positions in the plurality of images;
displaying augmented reality information on the display screen if the gaze of the user is directed on or about the object tag at the fixed position; and
hiding the object tag if the gaze of the user is not directed on or about the object tag.

2. The method of claim 1 , wherein the initiating comprises performing the object recognition process.

3. The method of claim 1 , wherein the initiating comprises providing the image segment to a remote server and receiving the object recognition result from the remote server.

4. The method of claim 3 , comprising:
receiving pose information from the remote server; and
displaying the augmented reality information based on the pose information.

5. The method of claim 1 , wherein the image is captured with a first camera coupled to a mobile device, and wherein the eye gaze information is determined based on an image captured with a second camera coupled to the mobile device.

6. The method of claim 1 wherein the object tag is displayed on the display screen for a preset time period.

7. The method of claim 1 , wherein object tags associated with different objects are displayed at different fixed positions in the plurality of images as the objects associated with the object tags are displayed at different positions in the plurality of images.

8. The method of claim 1 , wherein the initial position of the object is based on a position of the object when the object tag is displayed.

9. The method of claim 1 , wherein the position of the object tag remains in the fixed position relative to the initial position of the object and relative to the display screen for a pre-determined period of time.

10. The method of claim 1 , wherein the position of the object tag remains in the fixed position relative to the initial position of the object and relative to the display screen as long as the gaze of the user is directed on or about the object tag.

11. The method of claim 1 , wherein the augmented reality information is displayed on the display screen if the gaze of the user is directed on or about the object tag at the fixed position for a threshold amount of time.

12. An apparatus for enabling object recognition using eye gaze tracking, comprising:
a memory;
at least one processor coupled to the memory and configured to:
cause an image of a scene viewable by a user to be displayed on a display screen to the user;
receive information indicative of an eye gaze of the user;
determine an area of interest within the image based on the eye gaze information;
cause a tracking region representing the area of interest to be displayed on the display screen;
determine an image segment based on the area of interest, the image segment being associated with an initial position of an object in the scene, wherein the image segment is determined when the tracking region is within a threshold proximity of the object;
initiate an object recognition process on the image segment, wherein the object recognition process is limited to an area defined by the image segment; and
cause an object tag to be displayed on the display screen if a result of the object recognition process indicates that augmented reality information is available, wherein the object tag remains at a fixed position relative to the initial position of the object associated with the object tag and relative to the display screen for a plurality of images of the scene, and wherein the object tag automatically remains at the fixed position in the plurality of images as the object associated with the object tag is displayed at different positions in the plurality of images;
cause augmented reality information to be displayed on the display screen if the gaze of the user is directed on or about the object tag at the fixed position; and
cause the object tag to be hidden if the gaze of the user is not directed on or about the object tag.

13. The apparatus of claim 12 , wherein the at least one processor is configured to perform the object recognition process.

14. The apparatus of claim 12 , wherein the at least one processor is configured to provide the image segment to a remote server and receive the object recognition result from the remote server.

15. The apparatus of claim 14 , wherein the at least one processor is configured to:
receive pose information from the remote server; and
cause the augmented reality information to be displayed based on the pose information.

16. The apparatus of claim 12 , wherein the at least one processor is configured to capture the image with a first camera coupled to a mobile device, and determine the eye gaze information based on an image captured with a second camera coupled to the mobile device.

17. An apparatus for enabling object recognition using eye gaze tracking, comprising:
means for displaying, on a display screen, an image of a scene viewable by a user;
means for receiving information indicative of an eye gaze of the user;
means for determining an area of interest within the image based on the eye gaze information;
means for displaying, on the display screen, a tracking region representing the area of interest;
means for determining an image segment based on the area of interest, the image segment being associated with an initial position of an object in the scene, wherein the image segment is determined when the tracking region is within a threshold proximity of the object;
means for initiating an object recognition process on the image segment, wherein the object recognition process is limited to an area defined by the image segment;
means for displaying an object tag on the display screen if a result of the object recognition process indicates that augmented reality information is available, wherein the object tag remains at a fixed position relative to the initial position of the object associated with the object tag and relative to the display screen for a plurality of images of the scene, and wherein the object tag automatically remains at the fixed position in the plurality of images as the object associated with the object tag is displayed at different positions in the plurality of images;
means for displaying augmented reality information on the display screen i

Assignees

• Qualcomm Inc

Inventors

• Swaminathan Ashwin
• Ramachandran Mahesh

Classifications

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06F3/0481

  based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance · CPC title

• G09G2340/12

  Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/147

  using display panels · CPC title