Grasping virtual objects in augmented reality

Patent metadata

Field	Value
Publication number	US-9552673-B2
Application number	US-201213653968-A
Country	US
Kind code	B2
Filing date	Oct 17, 2012
Priority date	Oct 17, 2012
Publication date	Jan 24, 2017
Grant date	Jan 24, 2017

Abstract

An augmented reality system which enables grasping of virtual objects is described such as to stack virtual cubes or to manipulate virtual objects in other ways. In various embodiments a user's hand or another real object is tracked in an augmented reality environment. In examples, the shape of the tracked real object is approximated using at least two different types of particles and the virtual objects are updated according to simulated forces exerted between the augmented reality environment and at least some of the particles. In various embodiments 3D positions of a first one of the types of particles, kinematic particles, are updated according to the tracked real object; and passive particles move with linked kinematic particles without penetrating virtual objects. In some examples a real-time optic flow process is used to track motion of the real object.

First claim

The invention claimed is:

1. A computing device arranged to calculate a virtual object in a virtual reality environment, the computing device comprising:
an input arranged to receive a plurality of images of at least part of a real object in an interaction volume associated with the virtual reality environment;
a 3D tracker arranged to track 3D motion and/or deformation of a surface of the real object in the interaction volume using the received images and approximate the surface of the real object by filling the contour of the real object with a plurality of non-intersecting passive particles, wherein each passive particle is linked to one of a plurality of kinematic particles; and
a physics simulator arranged to calculate the position of the virtual object in the virtual reality environment according to simulated forces between the virtual object and the real object, the simulated forces being determined at least using the tracked 3D motion and/or deformation, the plurality of passive particles, and the plurality of kinematic particles, wherein the simulated forces are determined such that the passive particles cannot penetrate the virtual object or the other passive particles.

2. A device as claimed in claim 1 the physics simulator being arranged to calculate the position of the virtual object in a robust, stable manner when the at least part of the real object penetrates the virtual object in the virtual reality environment.

3. A device as claimed in claim 1 wherein the 3D tracker is arranged to delete at least one of the plurality of kinematic particles.

4. A device as claimed in claim 1 where the images comprise depth maps and RGB images.

5. A device as claimed in claim 1 where the 3D tracker is arranged to calculate a 2D optical flow field from intensity images and to calculate a 3D optical flow field from the 2D optical flow field by looking up depth values of elements of the 2D optical flow field in depth maps associated with the intensity images.

6. A device as claimed in 1 wherein the 3D tracker comprises a graphics processing unit arranged to calculate an optical flow field of the surface of the real object.

7. A device as claimed in claim 1 wherein the 3D tracker comprises a graphics processing unit arranged to calculate a surface of the real object.

8. A device as claimed in claim 1 wherein the 3D tracker comprises a graphics processing unit arranged to calculate a 3D shape of the real object.

9. A device as claimed in claim 1 wherein the 3D tracker is at least partially implemented using hardware logic selected from any one or more of: a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device and a graphics processing unit.

10. A computer-implemented method of calculating a virtual object in a virtual reality environment comprising:
receiving, at a processor, a plurality of images of at least part of a real object in an interaction volume associated with the virtual reality environment;
tracking 3D motion and/or deformation of a surface of the real object in the interaction volume using the received images and approximate the surface of the real object by filling the contour of the real object with a plurality of non-intersecting passive particles, wherein each passive particle is linked to one of a plurality of kinematic particles; and
calculating a position of the virtual object in the virtual reality environment according to simulated forces between the virtual object and the real object, the simulated forces being determined at least using the tracked 3D motion and/or deformation, the plurality of passive particles, and the plurality of kinematic particles, wherein the simulated forces are determined such that the passive particles cannot penetrate the virtual object or the other passive particles.

11. A method as claimed in claim 10 comprising calculating the position of the virtual object in a robust, stable manner when the at least part of the real object penetrates the virtual object in the virtual reality environment.

12. A method as claimed in claim 10 comprising using the tracked 3D motion and/or deformation to delete at least one of the plurality of kinematic particles.

13. A method as claimed in claim 10 comprising calculating a 2D optical flow field from intensity images and calculating a 3D optical flow field from the 2D optical flow field by looking up depth values of elements of the 2D optical flow field in depth maps associated with the intensity images.

14. A computer-implemented method of simulating grasping of a virtual object comprising:
receiving, at a processor, a plurality of images of at least part of a user's hand in an interaction volume having an associated virtual reality environment visible to the user;
tracking 3D motion and/or deformation of a surface of the user's hand in the interaction volume using the received images and approximate the surface of the user's hand by filling the contour of the user's hand with a plurality of non-intersecting passive particles, wherein each passive particle is linked to one of a plurality of kinematic particles; and
calculating a position of the virtual object in the virtual reality environment according to simulated forces between the virtual object and the user's hand, the simulated forces being determined at least using the tracked 3D motion and/or deformation, the plurality of passive particles, and the plurality of kinematic particles in order that when the user grasps the virtual object corresponding simulated grasping in the virtual reality environment is calculated, wherein the simulated forces are determined such that the passive particles cannot penetrate the virtual object or the other passive particles.

15. A method as claimed in claim 14 comprising calculating the position of the virtual object in a robust, stable manner when the at least part of the real object penetrates the virtual object in the virtual reality environment.

16. A method as claimed in claim 14 comprising displaying the virtual object in the virtual reality environment according to the calculated position of the virtual object.

17. A method as claimed in claim 14 comprising displaying the virtual object and the virtual reality environment in a robust, stable manner when the at least part of the real object penetrates the virtual object in the virtual reality environment.

18. A method as claimed in claim 14 at least partially implemented using hardware logic selected from any one or more of: a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device and a graphics processing unit.

Assignees

• Microsoft Corp
• Microsoft Technology Licensing Llc

Inventors

• Hilliges Otmar
• Kim David
• Izadi Shahram
• Weiss Malte Hanno

Classifications

• G06T7/269

  using gradient-based methods · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title