Projection casting in virtual environments

Patent metadata

Field	Value
Publication number	US-11257295-B2
Application number	US-202117211771-A
Country	US
Kind code	B2
Filing date	Mar 24, 2021
Priority date	Sep 20, 2019
Publication date	Feb 22, 2022
Grant date	Feb 22, 2022

Abstract

The present technology relates to artificial reality systems. Such systems provide projections a user can create to specify object interactions. For example, when a user wishes to interact with an object outside her immediate reach, she can use a projection to select, move, or otherwise interact with the distant object. The present technology also includes object selection techniques for identifying and disambiguating between objects, allowing a user to select objects both near and distant from the user. Yet further aspects of the present technology include techniques for interpreting various bimanual (two-handed) gestures for interacting with objects. The present technology further includes a model for differentiating between global and local modes for, e.g., providing different input modalities or interpretations of user gestures.

First claim

We claim:

1. A method for selecting multiple objects in an artificial reality environment, the method comprising:
determining a control point and casting direction based on one or more tracked positions of one or more body parts of a user;
identifying, based on one or more images from one or more cameras that capture both a first hand of the user and a second hand of the user, a distance between the first hand of the user and the second hand of the user;
generating a cone-shaped projection that begins at the control point and that extends along the casting direction, wherein a width of at least part of the cone-shaped projection is based on the identified distance;
identifying multiple objects that at least partially intersect with at least part of the cone-shaped projection, either simultaneously or at different times; and
selecting the multiple objects.

2. The method of claim 1 , wherein,
the tip of the cone-shaped projection is at the control point;
the cone-shaped projection extends, from the cone-shaped projection tip, outward from the user; and
the cone-shaped projection is centered, from the cone-shaped projection tip to the center of a circular base of the cone-shaped projection, on the casting direction.

3. The method of claim 1 , wherein the width of the at least part of the cone-shaped projection that is based on the identified distance is the diameter of a circular base of the cone-shaped projection.

4. The method of claim 1 wherein the casting direction is along a line connecting one of the user's eyes to the control point.

5. The method of claim 1 , wherein:
the casting direction is based on a line determined from an origin point to a control point;
the origin point comprises an eye of the user, a shoulder of the user, a hip of the user, or a point along a path connecting two other points on the body of the user; and
the control point is one of: a fist of the user, a palm of the user, one or more fingertips of the user, or a wrist of the user.

6. The method of claim 1 , wherein the method is performed in response to identifying that the user performed a pinch gesture between the user's thumb and middle finger, and wherein the control point is defined in relation to the point where the user's thumb and middle finger touch.

7. The method of claim 1 further comprising:
in response to the selecting the multiple objects, causing a menu to be attached to the first hand of the user;
identifying an interaction, by the user's second hand, with an item on the menu;
in response to the interaction, causing a resize or rotation of at least one of the selected multiple objects.

8. The method of claim 1 further comprising:
monitoring a rotation of the first hand of the user such that rotation of the first hand of the user is mapped to rotation of at least one of the selected multiple objects; and
based on the monitoring, detecting a rotation of the first hand of the user and, in response, causing a corresponding rotation of the at least one of the selected multiple objects.

9. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for selecting multiple objects in an artificial reality environment, the process comprising:
determining a control point and casting direction based on one or more tracked positions of one or more body parts of a user;
identifying, based on one or more images from one or more cameras that capture both a first hand of the user and a second hand of the user, a distance between the first hand of the user and the second hand of the user;
generating a cone-shaped projection that begins at the control point and that extends along the casting direction, wherein a width of at least part of the cone-shaped projection is based on the identified distance;
identifying multiple objects that at least partially intersect with at least part of the cone-shaped projection, either simultaneously or at different times; and
selecting the multiple objects.

10. The computer-readable storage medium of claim 9 , wherein,
the tip of the cone-shaped projection is at the control point;
the cone-shaped projection extends, from the cone-shaped projection tip, outward from the user; and
the cone-shaped projection is centered, from the cone-shaped projection tip to the center of a circular base of the cone-shaped projection, on the casting direction.

11. The computer-readable storage medium of claim 9 , wherein the width of the at least part of the cone-shaped projection that is based on the identified distance is the diameter of a circular base of the cone-shaped projection.

12. The computer-readable storage medium of claim 9 , wherein the casting direction is along a line connecting one of the user's eyes to the control point.

13. The computer-readable storage medium of claim 9 , wherein:
the casting direction is based on a line determined from an origin point to a control point;
the origin point comprises an eye of the user, a shoulder of the user, a hip of the user, or a point along a path connecting two other points on the body of the user; and
the control point is one of: a fist of the user, a palm of the user, one or more fingertips of the user, or a wrist of the user.

14. The computer-readable storage medium of claim 9 , wherein the method is performed in response to identifying that the user performed a pinch gesture between the user's thumb and middle finger, and wherein the control point is defined in relation to the point where the user's thumb and middle finger touch.

15. The computer-readable storage medium of claim 9 , wherein the process further comprises:
in response to the selecting the multiple objects, causing a menu to be attached to the first hand of the user;
identifying an interaction, by the user's second hand, with an item on the menu; and
in response to the interaction, causing a resize or rotation of at least one of the selected multiple objects.

16. The computer-readable storage medium of claim 9 , wherein the process further comprises:
monitoring a rotation of the first hand of the user such that rotation of the first hand of the user is mapped to rotation of at least one of the selected multiple objects; and
based on the monitoring, detecting a rotation of the first hand of the user and, in response, causing a corresponding rotation of the at least one of the selected multiple objects.

17. A computing system method for selecting multiple objects in an artificial reality environment, the computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
determining a control point and casting direction based on one or more tracked positions of one or more body parts of a user;
identifying, based on one or more images from one or more cameras that capture both a first hand of the user and a second hand of the user, a distance between the first hand of the user and the second hand of the user;
generating a cone-shaped projection that begins at the control point and that extends along the casting direction, wherein a width of at least part of the cone-shaped projection is based on the identified distance;
identifying multiple objects that at least partially intersect with at least part of the cone-shaped projection, either simultaneously or at different times; and
selecting the multiple objects.

18. The computing system of claim 17 , wherein,

Assignees

• Facebook Tech Llc

Inventors

• Ravasz Jonathan
• Pinchon Etienne
• Varga Adam Tibor
• Stevens Jasper
• Ellis Robert
• Jones Jonah
• Krivoruchko Evgenii

Classifications

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G02B27/017

  Head mounted · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title