Widget interaction for extended reality (XR) applications

Patent metadata

Field	Value
Publication number	US-12591313-B2
Application number	US-202418746306-A
Country	US
Kind code	B2
Filing date	Jun 18, 2024
Priority date	Jun 27, 2023
Publication date	Mar 31, 2026
Grant date	Mar 31, 2026

Abstract

A method includes rendering a virtual widget at a first position in an extended reality (XR) space using a head-mounted display. The method also includes detecting a hand gesture of a user at a second position in the XR space using at least one optical sensor. The second position is between the first position and the optical sensor. The method further includes determining a duration of the hand gesture and a distance between the first and second positions. The method also includes performing a first or second type of action based on the duration and/or the distance. The first type of action corresponds to a system-level control of the virtual widget, and the second type of action corresponds to a content-level control of the virtual widget. In addition, the method includes rendering a visual and/or audio feedback based on at least one of the duration or the distance.

First claim

What is claimed is:

1 . A method comprising:
rendering a virtual widget at a first position in an extended reality (XR) space using a head-mounted display (HMD); detecting a hand gesture of a user at a second position in the XR space using at least one optical sensor of the HMD, wherein the second position is between the first position and the at least one optical sensor; determining a duration of the hand gesture and a distance between the first position and the second position; performing a first type of action or a second type of action based on at least one of the duration or the distance, wherein the first type of action corresponds to a system-level control of the virtual widget and the second type of action corresponds to a content-level control of the virtual widget, and wherein a content hit target is marked to disable widget system level hits when the second type of action is performed; and rendering at least one of a visual feedback or an audio feedback based on at least one of the duration or the distance.

2 . The method of claim 1 , wherein the first type of action comprises at least one of: invoking a widget menu, moving the virtual widget, placing the virtual widget at a location, or invoking resizing handles.

3 . The method of claim 1 , wherein the second type of action comprises at least one of: selecting a button on the virtual widget, navigating or scrolling a content on the virtual widget, or invoking widget expansion for additional interactions with the virtual widget.

4 . The method of claim 1 , wherein rendering the visual feedback comprises rendering a raycast between the first position and the second position based on the distance.

5 . The method of claim 1 , wherein the hand gesture includes at least one of: a pinch gesture, a poke gesture, a pinch-and-drag gesture, a pinch-and-hold gesture, or a hover gesture.

6 . The method of claim 1 , further comprising:
detecting a second hand gesture of the user in the XR space, the second hand gesture associated with a physical object; and performing a third type of action or rendering at least one of the visual feedback or the audio feedback based on the second hand gesture.

7 . The method of claim 1 , further comprising:
detecting a voice command from the user; and performing the first type of action or the second type of action based on the voice command.

8 . An electronic device comprising:
a head-mounted display (HMD) configured to render a virtual widget at a first position in an extended reality (XR) space; and at least one processing device configured to:
receive a hand gesture of a user at a second position in the XR space detected using at least one optical sensor of the HMD, wherein the second position is between the first position and the at least one optical sensor;
determine a duration of the hand gesture and a distance between the first position and the second position;
perform a first type of action or a second type of action based on at least one of the duration or the distance, wherein the first type of action corresponds to a system-level control of the virtual widget and the second type of action corresponds to a content-level control of the virtual widget, and wherein a content hit target is marked to disable widget system level hits when the second type of action is performed; and
render at least one of a visual feedback or an audio feedback based on at least one of the duration or the distance.

9 . The electronic device of claim 8 , wherein:
the first type of action comprises at least one of: invoking a widget menu, moving the virtual widget, placing the virtual widget at a location, or invoking resizing handles; and the second type of action comprises at least one of: selecting a button on the virtual widget, navigating or scrolling a content on the virtual widget, or invoking widget expansion for additional interactions with the virtual widget.

10 . The electronic device of claim 8 , wherein, to render the visual feedback, the at least one processing device is configured to render a raycast between the first position and the second position based on the distance.

11 . The electronic device of claim 8 , wherein the hand gesture includes at least one of: a pinch gesture, a poke gesture, a pinch-and-drag gesture, a pinch-and-hold gesture, or a hover gesture.

12 . The electronic device of claim 8 , wherein the at least one processing device is further configured to:
receive a second hand gesture of the user detected in the XR space, the second hand gesture associated with a physical object; and perform a third type of action or render at least one of the visual feedback or the audio feedback based on the second hand gesture.

13 . The electronic device of claim 8 , wherein the at least one processing device is further configured to:
receive a voice command from the user; and perform the first type of action or the second type of action based on the voice command.

14 . A method comprising:
receiving a first video captured using an outward facing camera associated with a head-mounted display (HMD) worn by a user, the first video capturing a user environment; receiving a second video captured using an eye tracking camera associated with the HMD; identifying a real world object on which eyes of the user are focused based on the second video, wherein the real world object is associated with a set of control options; calibrating the real world object and user contact with the real world object to map and interpret one or more movements of the real world object and the user contact to control one or more widgets based on one or more calibration thresholds; determining a gaze of the user based on the second video; registering a widget last gazed on by the user; determining a type of interaction the user makes with the real world object during the user contact based on the first video, the type of interaction comprising one of a contact interaction, a rotation interaction, or a shift interaction; selecting a control option among the set of control options based on the determined type of interaction; selecting a virtual widget based on the gaze of the user and the registered widget; and controlling the virtual widget in an extended reality (XR) space based on the selected control option and the one or more calibration thresholds.

15 . The method of claim 14 , wherein the contact interaction is associated with a Boolean control of the virtual widget, the Boolean control having two states.

16 . The method of claim 14 , wherein:
the rotation interaction is associated with a continuous control of the virtual widget between a minimum value and a maximum value; and a change in angle caused by the rotation interaction is mapped to a value of the virtual widget between the minimum value and the maximum value.

17 . The method of claim 14 , wherein:
the shift interaction is associated with a continuous control of the virtual widget between a minimum value and a maximum value; and a change in position caused by the shift interaction is mapped to a value of the virtual widget between the minimum value and the maximum value.

18 . The method of claim 17 , wherein:
the change in position represents a change along a first axis and a second axis; the change along the first axis is used to control a first feature of the virtual widget; and the change along the second axis is used to control a second feature of the virtual widget.

19 . The method of claim 14 , further comprising:
dete

Assignees

• Samsung Electronics Co Ltd

Inventors

• Evangelista Edgar Charles
• Horii Hiroshi
• Waghulde Ajinkya
• Jiang Yujie
• Kim Sooryuh
• Cho Minkyung
• Shin Boosun
• Kim Sungman
• Lee Chaekyung

Classifications

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G06F3/0485

  Scrolling or panning · CPC title