Methods and systems of extended reality environment interaction based on eye motions

Patent metadata

Field	Value
Publication number	US-12373029-B2
Application number	US-202418671129-A
Country	US
Kind code	B2
Filing date	May 22, 2024
Priority date	Oct 20, 2020
Publication date	Jul 29, 2025
Grant date	Jul 29, 2025

Abstract

Systems and methods are described for extended reality environment interaction. An extended reality environment including an object is generated for display, and a table of eyelid motion identifiers and corresponding actions performable on the object in the extended reality environment is stored in memory. An eyelid motion is detected by using a sensor, and the detected eyelid motion is matched to one of the stored eyelid motion identifiers. An updated version of the extended reality environment, based on the action that corresponds to the matched eyelid motion, is generated for display, in response to matching the detected eyelid motion to one of the stored eyelid motion identifiers.

First claim

What is claimed is:

1. A method for extended reality (XR) environment interaction, comprising:
providing for an XR environment including an object, wherein the XR environment is provided via an XR device associated with a user;
receiving input from one or more sensors;
identifying, based at least in part on the received input, the object in a field of view;
detecting, based at least in part on the received input, a number of distinct eyelid levels over a predetermined period of time;
detecting, based at least in part on the number of distinct eyelid levels detected over the predetermined period of time, an amount of eyelid motion within the predetermined period of time, wherein the eyelid motion is directed at the object in the field of view;
comparing the detected amount of eyelid motion to a threshold amount of eyelid motion, wherein the threshold amount of eyelid motion comprises a threshold number of distinct eyelid levels;
based at least in part on the comparing, determining that the detected amount of eyelid motion exceeds the threshold amount of eyelid motion; and
based at least in part on the determining, causing performance of an action to modify the object in the field of view.

2. The method of claim 1 , wherein:
detecting the amount of eyelid motion within the predetermined period of time comprises detecting a number of blinks of the user within the predetermined period of time; and
the threshold amount of eyelid motion comprises a threshold number of detected blinks.

3. The method of claim 1 , further comprising:
determining the predetermined period of time based at least in part on historical data of the user.

4. The method of claim 1 , wherein causing performance of the action to modify the object in the field of view comprises regenerating for display the object with a modified level of detail.

5. The method of claim 1 , wherein causing performance of the action to modify the object in the field of view comprises altering an appearance of the object.

6. The method of claim 1 , further comprising:
computing respective virtual distances of a plurality of objects with respect to the user of the XR device, wherein the plurality of objects comprises the object; and
wherein identifying the object in the field of view comprises determining the object is at a closest virtual distance to the user of the respective virtual distances.

7. The method of claim 1 , further comprising:
storing, in a profile of the user, a table of a plurality of eyelid motion identifiers and corresponding actions performable on the object in the XR environment; and
based at least in part on determining at least a portion of the eyelid motion matches an eyelid motion identifier of the plurality of eyelid motion identifiers, causing performance of the action, wherein the action corresponds to an action corresponding to the matching eyelid motion identifier.

8. The method of claim 1 , further comprising:
based at least in part on determining at least a portion of the eyelid motion an eyelid motion type, causing performance of the action to modify the object in the field of view based at least in part on the eyelid motion type.

9. The method of claim 1 , further comprising:
regenerating for display the object with a modified level of detail different from an initial level of detail;
based on matching the eyelid motion to a particular eyelid motion identifier of a plurality of eyelid motion identifiers stored in a table, causing performance of the action indicated in the table as corresponding to the particular eyelid motion identifier;
detecting that a gaze has shifted from a first portion of the XR environment to a second portion of the XR environment, the object being excluded from the first portion of the XR environment and included in the second portion of the XR environment;
in response to detecting the gaze shift, generating for display within the XR environment an indicator of the gaze shift;
detecting a voice command while the indicator is in a vicinity of the object and executing an action corresponding to the voice command;
providing for display, within the XR environment, a plurality of opacity-based indicators in the vicinity of the first portion of the XR environment;
varying an opacity of at least one of the plurality of opacity-based indicators;
in response to determining that the object is in the field of view for at least the predetermined period of time, generating for display in the XR environment one or more items related to the object.

10. The method of claim 1 , wherein the received input comprises one or more images captured by a camera.

11. A system for extended reality (XR) environment interaction, comprising:
a computer memory;
a control circuitry; and
provide for display the XR environment including an object, wherein the XR environment is provided via an XR device associated with a user;
receive input from one or more sensors;
identify, based at least in part on the received input, the object in a field of view;
detect, based at least in part on the received input, a number of distinct eyelid levels over a predetermined period of time;
detect, based at least in part on the number of distinct eyelid levels detected over the predetermined period of time, an amount of eyelid motion within the predetermined period of time, wherein the eyelid motion is directed at the object in the field of view;
compare the detected amount of eyelid motion to a threshold amount of eyelid motion, wherein the threshold amount of eyelid motion is stored in the computer memory, and wherein the threshold amount of eyelid motion comprises a threshold number of distinct eyelid levels;
based at least in part on the comparing, determine that the detected amount of eyelid motion exceeds the threshold amount of eyelid motion; and
based at least in part on the determining, cause performance action to modify the object in the field of view.

12. The system of claim 11 , wherein:
the control circuitry is configured to detect the amount of eyelid motion within the predetermined period of time by detecting a number of blinks of the user within the predetermined period of time; and
the threshold amount of eyelid motion comprises a threshold number of detected blinks.

13. The system of claim 11 , wherein the control circuitry is further configured to determine the predetermined period of time based at least in part on historical data of the user.

14. The system of claim 11 , wherein the control circuitry is further configured to cause performance of the action to modify the object in the field of view by regenerating for display the object with a modified level of detail.

15. The system of claim 11 , wherein the control circuitry is configured to cause performance of the action to modify the object in the field of view comprises altering an appearance of the object.

16. The system of claim 11 , wherein the control circuitry is further configured to:
compute respective virtual distances of a plurality of objects with respect to the user of the XR device, wherein the plurality of objects comprises the object; and
identify the object in the field of view by determining the object is at a closest virtual distance to the user of the respective virtual distances.

17. The system of claim 11 , wherein the control circuitry is further configured to:
store, in a profile of the user, a table of a plurality of eyelid motion identifiers and corresponding actions performable on the object in the XR environment; and
based at least in part on determining at least a p

Assignees

• Adeia Guides Inc

Inventors

• Aher Ankur Anil
• Sen Susanto
• Balaji R
• Pulikunta Sai Durga Venkat Reddy
• Robert Jose Jeffry Copps
• T V Arun Kumar
• Saito Sakura

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06T19/003

  Navigation within 3D models or images · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title