Automatic projection type selection in an artificial reality environment

Patent metadata

Field	Value
Publication number	US-11947111-B2
Application number	US-202217929512-A
Country	US
Kind code	B2
Filing date	Sep 2, 2022
Priority date	Sep 20, 2019
Publication date	Apr 2, 2024
Grant date	Apr 2, 2024

Abstract

The present technology relates to artificial reality systems. Such systems provide projections a user can create to specify object interactions. For example, when a user wishes to interact with an object outside her immediate reach, she can use a projection to select, move, or otherwise interact with the distant object. The present technology also includes object selection techniques for identifying and disambiguating between objects, allowing a user to select objects both near and distant from the user. Yet further aspects of the present technology include techniques for interpreting various bimanual (two-handed) gestures for interacting with objects. The present technology further includes a model for differentiating between global and local modes for, e.g., providing different input modalities or interpretations of user gestures.

First claim

We claim:

1. A method for interacting in an artificial reality environment, the method comprising:
determining a projection type for a projection to select multiple objects, prior to the selection of those multiple objects, by:
determining a control point and casting direction based on one or more tracked positions of one or more body parts of a user;
identifying an interaction plane of a gaze of the user, wherein the interaction plane is a viewing plane, on which the gaze of the user is focused prior to the selecting of the multiple objects, that is A) distant from the user in a 3D environment and B) intersects with a line formed by the gaze of the user, wherein the interaction plane is identified based on an interaction, at a location between the multiple objects, by the user via the gaze without selecting the multiple objects;
computing a distance between the user and the interaction plane prior to the selecting of the multiple objects; and
selecting the projection type based on the computed distance between the user and the interaction plane;

generating a projection, of the selected projection type, that begins at the control point and that extends along the casting direction; and
identifying the multiple objects based on determining which objects at least partially intersect with at least part of the projection, either simultaneously or at different times.

2. The method of claim 1 , wherein the projection type is equivalent to a cylinder type, specifying that the projection includes a cylinder that extends, beginning at the control point, outward from the user and is centered on the casting direction.

3. The method of claim 1 , wherein the projection type is equivalent to a line-and-sphere type, specifying that the projection includes at least part of a sphere that is at an end of a line that extends, beginning at the control point, outward from the user.

4. The method of claim 1 ,
wherein the projection type is equivalent to a cone type, specifying that the projection includes a cone; and
wherein:
a tip of the cone is at the control point;
the cone extends, from the tip of the cone, outward from the user; and
the cone is centered, from the tip of the cone to a center of a circular base of the cone, on the casting direction.

5. The method of claim 1 ,
wherein at least part of the projection is a cone; and
wherein:
a width of a diameter of the a of the cone is based on the computed distance.

6. The method of claim 1 ,
wherein the projection type is equivalent to a ray type, specifying that the projection includes a ray; and
wherein:
at least one point along the ray corresponds to a hook; and
each particular one of the multiple objects is identified based on the hook intersecting with that particular object when a selection gesture from the user is identified.

7. The method of claim 1 , wherein:
the casting direction is along a line connecting one of the users eyes to the control point that is based on the one or more tracked positions of the one or more body parts of the user;
the projection:
extends away from the user, and
falls along the line that intersects the control point and the one of the eyes;

the method further comprises:
continuously monitoring, according to movements of the control point and the one of the users eyes, a path of the projection;
causing a representation of at least part of the path to appear in the artificial reality environment; and
recording intersections of the projection with objects; and

the multiple objects are identified based on the recorded intersections with the multiple objects.

8. The method of claim 1 , wherein:
the projection type is equivalent to a distorted cone type;
the casting direction is along a line connecting one of the users eyes to the control point;
the method further comprises:
continuously monitoring, according to movements of the control point and the one of the users eyes, a path of a ray that extends away from the user, begins at the control point, and falls along the line that intersects the control point and the users dominant eye; and
identifying a distorted cone formed with a point of the distorted cone at the one of the users eyes and sides of the distorted cone extending through the path of the ray; and

the projection becomes the distorted cone and the multiple objects are identified based on their intersection with at least a part of the distorted cone.

9. The method of claim 1 , wherein:
the projection type is equivalent to a pyramid type;
the casting direction is away from the user beginning at an eye of the user;
the control point is a first control point that is based on a position of a part of a first hand of the user and a second control point is based on a position of a part of a second hand of the user;
a width of at least part of the projection is a length of a diagonal of a rectangle with opposite corners corresponding to the first control point and the second control point;
generating the projection comprises forming a pyramid with a tip of the pyramid at the one of the users eyes and each side of the pyramid intersecting with a different one of the lines that forms the rectangle; and
the multiple objects are identified based on their intersection with at least a part of the pyramid.

10. The method of claim 1 , wherein the selecting the projection type further based on an identified surface type of a target object.

11. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for interacting in an artificial reality environment, the process comprising:
determining a projection type for a projection to select multiple objects, prior to the selection of those multiple objects, by:
determining a control point and casting direction based on one or more tracked positions of one or more body parts of a user;
identifying an interaction plane of a gaze of the user, wherein the interaction plane is a viewing plane, on which the gaze of the user is focused prior to the selecting of the multiple objects, that is A) distant from the user in a 3D environment and B) intersects with a line formed by the gaze of the user, wherein the interaction plane is identified based on an interaction, at a location between the multiple objects, by the user via the gaze without selecting the multiple objects;
computing a distance between the user and the interaction plane prior to the selecting of the multiple objects; and
selecting the projection type based on the computed distance between the user and the interaction plane;

generating a projection, of the selected projection type, that begins at the control point and that extends along the casting direction; and
identifying the multiple objects based on determining which objects at least partially intersect with at least part of the projection, either simultaneously or at different times.

12. The non-transitory computer-readable storage medium of claim 11 , wherein the projection type is equivalent to a cylinder type, specifying that the projection includes a cylinder that extends, beginning at the control point, outward from the user and is centered on the casting direction.

13. The non-transitory computer-readable storage medium of claim 11 , wherein the projection type is equivalent to a line-and-sphere type, specifying that the projection includes at least part of a sphere that is at an end of a line that extends, beginning at the control point, outward from the user.

14. The non-transitory computer-readable storage medium of claim 11 ,
wherein the pr

Assignees

• Meta Platforms Tech Llc

Inventors

• Ravasz Jonathan
• Pinchon Etienne
• Varga Adam Tibor
• Stevens Jasper
• Ellis Robert
• Jones Jonah
• Krivoruchko Evgenii

Classifications

• G02B27/017 Primary

  Head mounted · CPC title

• G02B27/01

  Head-up displays · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06V40/107

  Static hand or arm · CPC title