Object selection based on eye tracking in wearable device

Patent metadata

Field	Value
Publication number	US-12517575-B2
Application number	US-202218065436-A
Country	US
Kind code	B2
Filing date	Dec 13, 2022
Priority date	Apr 8, 2021
Publication date	Jan 6, 2026
Grant date	Jan 6, 2026

Abstract

A wearable computing device includes a frame, a camera mounted on the frame so as to capture images of an environment outside of the wearable computing device, a display device mounted on the frame so as to display the images captured by the camera, and at least one eye gaze tracking device mounted on the frame so as to track a gaze directed at the images displayed by the display device. In response to the detection of a fixation of the gaze on the display of images, the system may identify a pixel area corresponding to a fixation point of the fixation gaze on the display of images. The system may identify an object in the ambient environment corresponding to the identified pixel area, and set the identified object as a selected object for user interaction.

First claim

What is claimed is:

1 . A computer-implemented method, comprising:
capturing, by an image sensor of a wearable computing device, a first image of a portion of an ambient environment within a field of view of the image sensor, the first image including a representation of a first object in the ambient environment; outputting, by a display device of the wearable computing device, the first image in a display area of the display device; tracking, by at least one gaze tracking device of the wearable computing device, a gaze of a user of the wearable computing device; detecting a fixation of the gaze in response to detecting movement of the gaze having at least one of a magnitude or a frequency that meets a threshold; and identifying an object of focus of the fixation of the gaze, including:
in response to detecting the fixation of the gaze within the display area corresponding to the first image, identifying the first image as the object of focus; and
in response to detecting the fixation of the gaze outside of the display area, identifying a second object in the ambient environment, outside of the display area, as the object of focus, and outputting a second image including a representation of the second object in the ambient environment identified as the object of focus.

2 . The computer-implemented method of claim 1 , wherein outputting the second image includes terminating display of the first image.

3 . The computer-implemented method of claim 1 , further comprising:
capturing, by the image sensor of the wearable computing device, a plurality of images of the ambient environment, the plurality of images including the first image and the second image; outputting, by the display device of the wearable computing device, the plurality of images as the plurality of images are captured; and maintaining display of an image, of the plurality of images, in response to detecting fixation of the gaze of the user directed at the image.

4 . The computer-implemented method of claim 3 , further comprising:
detecting the fixation of the gaze within the image; and identifying a pixel area within the image corresponding to the fixation of the gaze.

5 . The computer-implemented method of claim 4 , wherein tracking the gaze includes:
identifying a fixation point of the fixation of the gaze at an intersection of the gaze with the image; and identifying the pixel area at the intersection of the fixation of the gaze with the image.

6 . The computer-implemented method of claim 4 , further comprising:
analyzing an area surrounding the pixel area; identifying features within the area surrounding the pixel area based on the analyzing; recognizing at least one of the features; and matching the pixel area to a corresponding object in the ambient environment based on the at least one of the features.

7 . The computer-implemented method of claim 6 , wherein recognizing the at least one of the features includes at least one of:
recognizing the at least one of the features based on a search of an external database of previously stored images; or recognizing the at least one of the features based on known features of the ambient environment.

8 . The computer-implemented method of claim 3 , wherein capturing the plurality of images includes capturing portions of the ambient environment within a field of view of the image sensor as at least one of a position or an orientation of the wearable computing device changes in the ambient environment.

9 . The computer-implemented method of claim 1 , wherein the wearable computing device is a head-mounted wearable computing device including a frame, with the image sensor, the display device, and the at least one gaze tracking device mounted on the frame, the at least one gaze tracking device tracking an eye gaze of a user of the head-mounted wearable computing device, and the display device outputting a plurality of images such that the plurality of images and the ambient environment are visible through a lens portion of the head-mounted wearable computing device.

10 . A computer-implemented method, comprising:
capturing, by an image sensor of a wearable computing device, a plurality of images of an ambient environment; outputting, in a display area of a display device of the wearable computing device, the plurality of images as the plurality of images are captured; tracking, by at least one gaze tracking device of the wearable computing device, a gaze of a user of the wearable computing device; detecting a fixation of the gaze in response to detecting movement of the gaze having at least one of a magnitude or a frequency that meets a threshold; and identifying an object of focus of the fixation of the gaze, including:
in response to detecting the fixation of the gaze within the display area of an image of the plurality of images, identifying a representation of an object within the image as the object of focus; and
in response to detection of the fixation of the gaze outside of the display area, identifying an object in the ambient environment as the object of focus and outputting an image including the object in the ambient environment identified as the object of focus.

11 . The computer-implemented method of claim 10 , further comprising:
maintaining display of the image including the object in the ambient environment; detecting fixation of the gaze within the image; and identifying a pixel area within the image corresponding to the fixation of the gaze.

12 . The computer-implemented method of claim 11 , wherein tracking the gaze includes:
identifying a fixation point of the fixation of the gaze at an intersection of the fixation of the gaze with the image; and identifying the pixel area at the intersection of the fixation of the gaze with the image.

13 . The computer-implemented method of claim 11 , further comprising:
analyzing an area surrounding the pixel area; identifying features within the area surrounding the pixel area based on the analyzing; recognizing at least one of the features; and matching the pixel area to a corresponding object in the ambient environment based on the at least one of the features.

14 . The computer-implemented method of claim 13 , wherein recognizing the at least one of the features includes at least one of:
recognizing the at least one of the features based on a search of an external database of previously stored images; or recognizing the at least one of the features based on known features of the ambient environment.

15 . The computer-implemented method of claim 10 , wherein capturing the plurality of images includes capturing portions of the ambient environment within a field of view of the image sensor as at least one of a position or an orientation of the wearable computing device changes in the ambient environment.

16 . The computer-implemented method of claim 10 , wherein the wearable computing device is a head-mounted wearable computing device including a frame, with the image sensor, the display device, and the at least one gaze tracking device mounted on the frame, the at least one gaze tracking device tracking an eye gaze of a user of the head-mounted wearable computing device, and the display device outputting the plurality of images such that the plurality of images and the ambient environment are visible through a lens portion of the head-mounted wearable computing device.

17 . A computer-implemented method, comprising:
capturing an image of a portion of an ambient environment within a field of view of an image sensor of a comp

Assignees

• Google Llc

Inventors

• Spencer Jason Todd
• Raphael Seth
• Bouaziz Sofien

Classifications

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title

• G06V10/255

  Detecting or recognising potential candidate objects based on visual cues, e.g. shapes · CPC title

• G06V40/193

  Preprocessing; Feature extraction · CPC title

• G06F2203/04806

  Zoom, i.e. interaction techniques or interactors for controlling the zooming operation · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title