Projection casting in virtual environments

Patent metadata

Field	Value
Publication number	US-11176745-B2
Application number	US-201916578221-A
Country	US
Kind code	B2
Filing date	Sep 20, 2019
Priority date	Sep 20, 2019
Publication date	Nov 16, 2021
Grant date	Nov 16, 2021

Abstract

The present technology relates to artificial reality systems. Such systems provide projections a user can create to specify object interactions. For example, when a user wishes to interact with an object outside her immediate reach, she can use a projection to select, move, or otherwise interact with the distant object. The present technology also includes object selection techniques for identifying and disambiguating between objects, allowing a user to select objects both near and distant from the user. Yet further aspects of the present technology include techniques for interpreting various bimanual (two-handed) gestures for interacting with objects. The present technology further includes a model for differentiating between global and local modes for, e.g., providing different input modalities or interpretations of user gestures.

First claim

We claim:

1. A method for interacting with objects in an artificial reality environment, the method comprising:
tracking a portion of a hand of a user and setting a control point based on the tracked portion of the hand of the user;
tracking a first point on a body of the user, that is not on the user's hand or forearm, and setting an origin point based on the tracked first point on the body of the user;
identifying an origin transition event by:
(1) determining the origin point should move to a tracked second point on the body of the user by identifying a direction associated with a tracked user gaze;
(2) determining the origin point should move to the tracked second point on the body of the user by identifying a measurement of an angle, in relation to a specified plane, of the line that passes through the control point and the origin point; or
(3) determining the origin point should move to the tracked second point on the body of the user by identifying a location of target objects in relation to the origin point;

in response to the origin transition event, moving the origin point from the first point on the body of the user to the second point on the body of the user;
determining a projection orientation centered on a line that passes through the control point and the origin point;
casting a ray projection from the control point, outward from the user, along the projection orientation; and
performing an action in relation to one or more objects based on one or more locations of the ray projection.

2. The method of claim 1 , wherein the moving the origin point comprises moving the origin point, at a set speed, in relation to a path connecting the first point on the body of the user to the second point on the body of the user.

3. The method of claim 2 wherein,
the first point on the body of the user is an eye or a shoulder of the user;
the second point on the body of the user is a hip of the user; and
the path connects A) the eye of the user or the shoulder of the user to B) the hip of the user by a straight line.

4. The method of claim 1 , wherein the identifying the origin transition event includes the identifying the direction associated with the tracked user gaze and wherein the determination that the origin point should move comprises:
determining a gaze angle associated with the direction of the tracked user gaze;
determining that the gaze angle is above a threshold; and
in response, selecting a high interaction mode in which the second point on the body of the user is below the first point on the body of the user.

5. The method of claim 1 wherein,
the identifying the origin transition event includes the measurement of the angle, in relation to a specified plane, of the line that passes through the control point and the origin point; and
the determining that the origin point should move comprises—
determining that the angle is below a threshold; and
in response, selecting a low interaction mode in which the second point on the body of the user is above the first point on the body of the user.

6. The method of claim 1 , wherein the portion of the hand of the user tracked in relation to the control point is fingertips of the user.

7. The method of claim 1 , wherein the portion of the hand of the user tracked in relation to the control point is a wrist of the user.

8. The method of claim 1 , wherein setting the control point based on the tracked portion of the hand of the user comprises selecting the control point at a location at a specified offset from the portion of the hand of the user.

9. The method of claim 1 , wherein the first point on the body of the user tracked in relation to the origin point is a dominant eye of the user.

10. The method of claim 1 , wherein the first point on the body of the user tracked in relation to the origin point is a shoulder of the user.

11. The method of claim 1 , wherein the first point on the body of the user tracked in relation to the origin point is a hip of the user.

12. The method of claim 1 , wherein performing the action in relation to the one or more objects includes causing the one or more objects to be selected.

13. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform operations for interacting with objects in an artificial reality environment, the operations comprising:
setting a control point based on a location of a portion of a hand of a user;
setting an origin point based on a first point on a body of the user, that is not on the user's hand or forearm;
identifying an origin transition event by:
(1) determining the origin point should move to a tracked second point on the body of the user by identifying a direction associated with a tracked user gaze;
(2) determining the origin point should move to the tracked second point on the body of the user by identifying a measurement of an angle, in relation to a specified plane, of the line that passes through the control point and the origin point; or
(3) determining the origin point should move to the tracked second point on the body of the user by identifying a location of target objects in relation to the origin point;

in response to the origin transition event, moving the origin point from the first point on the body of the user to the second point on the body of the user;
determining a projection orientation centered on a line that passes through the control point and the origin point; and
performing an action in relation to one or more objects based on one or more locations of the projection.

14. The computer-readable storage medium of claim 13 wherein,
the determining that the origin point should move comprises determining that the origin point should move to the second point on the body of the user along a path connecting the first point on the body of the user and the second point on the body of user.

15. The computer-readable storage medium of claim 13 wherein,
the identifying the origin transition event includes the identifying the direction associated with the tracked user gaze; and
the determining that the origin point should move comprises—
measuring a gaze angle associated with the direction of the tracked user gaze; and
based on the determined gaze angle, selecting an interaction mode in which the second point on the body of the user is above the first point on the body of the user.

16. The computer-readable storage medium of claim 13 wherein,
the portion of the hand is one of fingertips of the user; a palm of the user; a wrist of the user, or a first of the user; and
the first point on a body is one of a dominant eye of the user, a shoulder of the user, or a hip of the user.

17. A computing system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising—
setting a control point based on a location of a portion of a hand of a user;
setting an origin point based on a first point on a body of the user, that is not on the user's hand or forearm;
identifying an origin transition event by:
(1) determining the origin point should move to a tracked second point on the body of the user by identifying a direction associated with a tracked user gaze;
(2) determining the origin point should move to the tracked second point on the body of the user by identifying a measurement of an angle, in relation to a specified plane, of the line that passes through the control point and the origin point; or
(3)

Assignees

• Facebook Tech Llc

Inventors

• Ravasz Jonathan
• Pinchon Etienne
• Varga Adam Tibor
• Stevens Jasper
• Ellis Robert
• Jones Jonah
• Krivoruchko Evgenii

Classifications

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title