Automatic projection type selection in an artificial reality environment

Patent metadata

Field	Value
Publication number	US-11468644-B2
Application number	US-202017016217-A
Country	US
Kind code	B2
Filing date	Sep 9, 2020
Priority date	Sep 20, 2019
Publication date	Oct 11, 2022
Grant date	Oct 11, 2022

Abstract

The present technology relates to artificial reality systems. Such systems provide projections a user can create to specify object interactions. For example, when a user wishes to interact with an object outside her immediate reach, she can use a projection to select, move, or otherwise interact with the distant object. The present technology also includes object selection techniques for identifying and disambiguating between objects, allowing a user to select objects both near and distant from the user. Yet further aspects of the present technology include techniques for interpreting various bimanual (two-handed) gestures for interacting with objects. The present technology further includes a model for differentiating between global and local modes for, e.g., providing different input modalities or interpretations of user gestures.

First claim

We claim:

1. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for selecting multiple objects in an artificial reality environment, the process comprising:
determining a control point and casting direction based on one or more tracked positions of one or more body parts of a user;
identifying a target object with a corresponding surface type;
selecting a type of the projection based on mappings between surface types and projection types;
generating a projection, of the selected type, that begins at the control point and that extends along the casting direction;
identifying multiple objects that at least partially intersect with at least part of the projection, either simultaneously or at different times; and
selecting the multiple objects.

2. The non-transitory computer-readable storage medium of claim 1 , wherein the type of the projection is equivalent to a cylinder type, specifying that the projection includes a cylinder that extends, beginning at the control point, outward from the user and is centered on the casting direction.

3. The non-transitory computer-readable storage medium of claim 1 ,
wherein the type of the projection is equivalent to a cone type, specifying that the projection includes a cone; and
wherein:
the tip of the cone is at the control point;
the cone extends, from the cone tip, outward from the user; and
the cone is centered, from the cone tip to the center of the circular base of the cone, on the casting direction.

4. The non-transitory computer-readable storage medium of claim 1 ,
wherein the type of the projection is equivalent to a ray type, specifying that the projection includes a ray; and
wherein:
at least one point along the ray corresponds to a hook; and
each particular one of the multiple objects is identified based on the hook intersecting with that particular object when a selection gesture from the user is identified.

5. The non-transitory computer-readable storage medium of claim 1 , wherein:
the type of the projection is equivalent to a distorted cone type;
the casting direction is along a line connecting one of the user's eyes to the control point;
the process further comprises:
continuously monitoring, according to movements of the control point and the one of the user's eyes, a path of a ray that extends away from the user, begins at the control point, and falls along the line that intersects the control point and the user's dominant eye; and
identifying a distorted cone formed with a point of the distorted cone at the one of the user's eyes and the sides of the distorted cone extending through the path of the ray; and

the projection becomes the distorted cone and the multiple objects are identified based on their intersection with at least a part of the distorted cone.

6. The non-transitory computer-readable storage medium of claim 1 , wherein the type of the projection is equivalent to a ray type, specifying that the projection includes a line extending from the control point.

7. A method for selecting multiple objects in an artificial reality environment, the method comprising:
determining a control point and casting direction based on one or more tracked positions of one or more body parts of a user;
identifying a target object with a corresponding surface type;
selecting a type of the projection based on mappings between surface types and projection types;
generating a projection, of the selected type, that begins at the control point and that extends along the casting direction;
identifying multiple objects that at least partially intersect with at least part of the projection, either simultaneously or at different times; and
selecting the multiple objects.

8. The method of claim 7 , wherein the type of the projection is equivalent to a cylinder type, specifying that the projection includes a cylinder that extends, beginning at the control point, outward from the user and is centered on the casting direction.

9. The method of claim 7 ,
wherein the type of the projection is equivalent to a cone type, specifying that the projection includes a cone; and
wherein:
the tip of the cone is at the control point;
the cone extends, from the cone tip, outward from the user; and
the cone is centered, from the cone tip to the center of the circular base of the cone, on the casting direction.

10. The method of claim 7 ,
wherein the type of the projection is equivalent to a ray type, specifying that the projection includes a ray; and
wherein:
at least one point along the ray corresponds to a hook; and
each particular one of the multiple objects is identified based on the hook intersecting with that particular object when a selection gesture from the user is identified.

11. The method of claim 7 , wherein:
the type of the projection is equivalent to a distorted cone type;
the casting direction is along a line connecting one of the user's eyes to the control point;
the method further comprises:
continuously monitoring, according to movements of the control point and the one of the user's eyes, a path of a ray that extends away from the user, begins at the control point, and falls along the line that intersects the control point and the user's dominant eye; and
identifying a distorted cone formed with a point of the distorted cone at the one of the user's eyes and the sides of the distorted cone extending through the path of the ray; and

the projection becomes the distorted cone and the multiple objects are identified based on their intersection with at least a part of the distorted cone.

12. The method of claim 7 , wherein the type of the projection is equivalent to a ray type, specifying that the projection includes a line extending from the control point.

13. A computing system for performing an action in relation to multiple objects in an artificial reality environment, the computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by a computing system, cause the computing system to perform a process comprising:
determining a control point and casting direction based on one or more tracked positions of one or more body parts of a user;
identifying a target object with a corresponding surface type;
selecting a type of the projection based on mappings between surface types and projection types;
generating a projection, of the selected type, that begins at the control point and that extends along the casting direction;
identifying multiple objects that at least partially intersect with at least part of the projection, either simultaneously or at different times; and
performing an action in relation to the multiple objects.

14. The computing system of claim 13 , wherein the type of the projection is equivalent to a cylinder type, specifying that the projection includes a cylinder that extends, beginning at the control point, outward from the user and is centered on the casting direction.

15. The computing system of claim 13 ,
wherein the type of the projection is equivalent to a cone type, specifying that the projection includes a cone; and
wherein:
the tip of the cone is at the control point;
the cone extends, from the cone tip, outward from the user; and
the cone is centered, from the cone tip to the center of the circular base of the cone, on the casting direction.

16. The computing system of claim 13 ,
wherein the type of the projection is equivalent to a ray type, specifying that the projection in

Assignees

• Meta Platforms Tech Llc

Inventors

• Ravasz Jonathan
• Pinchon Etienne
• Varga Adam Tibor
• Stevens Jasper
• Ellis Robert
• Jones Jonah
• Krivoruchko Evgenii

Classifications

• G02B27/017 Primary

  Head mounted · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06V40/107

  Static hand or arm · CPC title

• G06V40/193

  Preprocessing; Feature extraction · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title