Wrist rotation manipulation of virtual objects

Patent metadata

Field	Value
Publication number	US-12518490-B2
Application number	US-202217823814-A
Country	US
Kind code	B2
Filing date	Aug 31, 2022
Priority date	Aug 31, 2022
Publication date	Jan 6, 2026
Grant date	Jan 6, 2026

Abstract

An Augmented Reality (AR) system is provided. The AR system uses a combination of gesture and DMVO methodologies to provide for the user's selection and modification of virtual object of an AR experience. The user indicates that they want to interact with a virtual object of the AR experience by moving their hand to overlap the virtual object. While keeping their hand in an overlapping position, the user rotates their wrist and the virtual object is rotated as well. To end the interaction, the user moves their hand such that their hand is no longer overlapping the virtual object.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
displaying a virtual object in an Augmented Reality (AR) experience provided to a user by an AR system, the virtual object including first 3D coordinate data of a location of the virtual object in a real-world scene; determining a user wants to interact with the virtual object by performing Direct Manipulation of Virtual Object (DMVO) operations comprising:
determining, using one or more cameras of the AR system, second 3D coordinate data of one or more nodes of a skeletal model of a hand of a user; and
determining an overlap between the hand of the user and the virtual object from a perspective of the user using the first 3D coordinate data and the second 3D coordinate data;
in response to determining the overlap, rotating the virtual object by performing gesture recognition operations comprising:
determining a current rotation value of a wrist joint of the hand of the user based on recognizing a gesture of rotation using the one more cameras, the gesture of rotation not involving DMVO operations on the virtual object; and
generating an updated rotation value for the virtual object based on the current rotation value of the wrist joint, a default rotation value of the virtual object, and a difference between the current rotation value of the wrist joint and a default rotation value for the wrist joint; and
redisplaying the virtual object based on the updated rotation value.

2 . The computer-implemented method of claim 1 , further comprising:
determining that the hand of the user no longer overlaps the virtual object; and based on determining that the hand of the user no longer overlaps the virtual object, ending interaction of the user with the virtual object.

3 . The computer-implemented method of claim 1 , wherein operations of determining an overlap between the hand of the user and the virtual object further comprise:
capturing, using the one or more cameras, hand-tracking video frame data of the hand of the user; generating hand-tracking data based on the hand-tracking video frame data; and determining the overlap between the hand of the user and the virtual object using the hand-tracking data.

4 . The computer-implemented method of claim 1 , wherein operations of determining an overlap between the hand of the user and the virtual object further comprise:
capturing, using the one or more cameras, hand-tracking video frame data of the hand of the user; and determining the overlap between the hand of the user and the virtual object based on the hand-tracking video frame data using one or more computer vision methodologies.

5 . The computer-implemented method of claim 1 , wherein operations of determining the current rotation value of the wrist joint of the hand of the user further comprise:
capturing, using the one or more cameras, hand-tracking video frame data of the hand of the user; and determining the current rotation value of the wrist joint of the hand of the user based on the hand-tracking video frame data and a wrist rotation model generated using machine learning methodologies.

6 . The computer-implemented method of claim 1 , wherein the AR system comprises a head-worn apparatus.

7 . A computing apparatus comprising:
one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the computing apparatus to perform operations comprising: displaying a virtual object in an AR experience provided to a user by an AR system, the virtual object including first 3D coordinate data of a location of the virtual object in a real-world scene; determining a user wants to interact with the virtual object by performing Direct Manipulation of Virtual Object (DMVO) operations comprising:
determining, using one or more cameras of the AR system, second 3D coordinate data of one or more nodes of a skeletal model of a hand of a user; and
determining an overlap between the hand of the user and the virtual object from a perspective of the user using the first 3D coordinate data and the second 3D coordinate data;
in response to determining the overlap, rotating the virtual object by performing gesture recognition operations comprising:
determining a current rotation value of a wrist joint of the hand of the user based on recognizing a gesture of rotation using the one more cameras, the gesture of rotation not involving DMVO operations on the virtual object; and
generating an updated rotation value for the virtual object based on the current rotation value of the wrist joint, a default rotation value of the virtual object, and a difference between the current rotation value of the wrist joint and a default rotation value for the wrist joint; and
redisplaying the virtual object based on the updated rotation value.

8 . The computing apparatus of claim 7 , wherein the operations further comprise:
determining that the hand of the user no longer overlaps the virtual object; and based on determining that the hand of the user no longer overlaps the virtual object, ending interaction of the user with the virtual object.

9 . The computing apparatus of claim 7 , wherein operations of determining an overlap between the hand of the user and the virtual object further comprise:
capturing, using the one or more cameras, hand-tracking video frame data of the hand of the user; generating hand-tracking data based on the hand-tracking video frame data; and determining the overlap between the hand of the user and the virtual object using the hand-tracking data.

10 . The computing apparatus of claim 7 , wherein operations of determining an overlap between the hand of the user and the virtual object further comprise:
capturing, using the one or more cameras, hand-tracking video frame data of the hand of the user; and determining the overlap between the hand of the user and the virtual object based on the hand-tracking video frame data using one or more computer vision methodologies.

11 . The computing apparatus of claim 7 , wherein operations of determining the current rotation value of the wrist joint of the hand of the user further comprise:
capturing, using the one or more cameras, hand-tracking video frame data of the hand of the user; and determining the current rotation value of the wrist joint of the hand of the user based on the hand-tracking video frame data and a wrist rotation model generated using machine learning methodologies.

12 . The computing apparatus of claim 7 , wherein the AR system comprises a head-worn apparatus.

13 . A non-transitory computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform operations comprising:
displaying a virtual object in an AR experience provided to a user by an AR system, the virtual object including first 3D coordinate data of a location of the virtual object in a real-world scene; determining a user wants to interact with the virtual object by performing Direct Manipulation of Virtual Object (DMVO) operations comprising:
determining, using one or more cameras of the AR system, second 3D coordinate data of one or more nodes of a skeletal model of a hand of a user; and
determining an overlap between the hand of the user and the virtual object from a perspective of the user using the first 3D coordinate data and the second 3D coordinate data;
in response to determining the overlap, rotating the virtual object by performing gesture recognition operations comprising:
determining a current rotation value of a wrist joint of the hand of the user based on recognizing a gesture of rota

Assignees

• Snap Inc

Inventors

• Mahalingam Anoosh Kruba Chandar
• Pounds Jennica
• Rybin Andrei
• Santerre Pierre-Yves

Classifications

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title