Spatially consistent representation of hand motion

Patent metadata

Field	Value
Publication number	US-2020311396-A1
Application number	US-201916363964-A
Country	US
Kind code	A1
Filing date	Mar 25, 2019
Priority date	Mar 25, 2019
Publication date	Oct 1, 2020
Grant date	—

Abstract

Examples are disclosed that relate to representing recorded hand motion. One example provides a computing device comprising instructions executable by a logic subsystem to receive video data capturing hand motion relative to an object, determine a first pose of the object, and associate a first coordinate system with the object based on the first pose. The instructions are further executable to determine a representation of the hand motion in the first coordinate system, the representation having a time-varying pose relative to the first pose of the object, and configure the representation for display relative to a second instance of the object having a second pose in a second coordinate system, with a time-varying pose relative to the second pose that is spatially consistent with the time-varying pose relative to the first pose.

First claim

1 . A computing device, comprising:
a logic subsystem; and a storage subsystem comprising instructions executable by the logic subsystem to:
receive video data capturing motion of a hand relative to a first instance of a designated object;
determine a first pose of the first instance of the designated object;
associate a first coordinate system with the first instance of the designated object based on the first pose;
determine a geometric representation of the motion of the hand in the first coordinate system, the geometric representation having a time-varying pose relative to the first pose of the first instance of the designated object; and
configure the geometric representation for display relative to a second instance of the designated object having a second pose in a second coordinate system, where the display of the geometric representation relative to the second instance of the designated object is configured with a time-varying pose relative to the second pose that is spatially consistent with the time-varying pose relative to the first pose.

2 . The computing device of claim 1 , further comprising instructions executable to, based on the video data, determine a time-varying representation of an environment in which the motion of the hand is captured.

3 . The computing device of claim 2 , where the geometric representation of the motion of the hand is determined based on a foreground portion of the time-varying representation segmented from a background portion of the time-varying representation.

4 . The computing device of claim 3 , where the background portion is identified based on data obtained from three-dimensionally scanning the environment.

5 . The computing device of claim 1 , where the first pose of the first instance of the designated object varies in time.

6 . The computing device of claim 5 , where the display of the geometric representation varies as the designated object undergoes articulated motion.

7 . The computing device of claim 1 , where the first instance of the designated object includes a first instance of a removable part, further comprising instructions executable to determine a geometric representation of motion of the hand relative to the first instance of the removable part in a third coordinate system associated with the first instance of the removable part.

8 . The computing device of claim 7 , further comprising instructions executable to configure the geometric representation of the motion of the hand relative to the first instance of the removable part for display relative to a second instance of the removable part in a fourth coordinate system associated with the second instance of the removable part.

9 . The computing device of claim 8 , further comprising instructions executable to determine a geometric representation of the first instance of the removable part, and to configure the geometric representation of the first instance of the removable part for display with the second instance of the removable part.

10 . The computing device of claim 1 , where one or more of a relative position, a relative orientation, and a relative scale of the time-varying pose relative to the first pose are substantially equal to a relative position, a relative orientation, and a relative scale of the time-varying pose relative to the second pose, respectively.

11 . A computing device, comprising:
a display; a logic subsystem; and a storage subsystem comprising instructions executable by the logic subsystem to:
receive a geometric representation of motion of a hand, the geometric representation having a time-varying pose determined relative to a first pose of a first instance of a designated object in a first coordinate system;
receive image data obtained by scanning an environment occupied by the computing device and by a second instance of the designated object;
based on the image data, determine a second pose of the second instance of the designated object;
associate a second coordinate system with the second instance of the designated object based on the second pose; and
output, via the display, the geometric representation relative to the second instance of the designated object with a time-varying pose relative to the second pose that is spatially consistent with the time-varying pose relative to the first pose.

12 . The computing device of claim 11 , further comprising instructions executable to receive a geometric representation of motion of the hand determined relative to a first instance of a removable part of the first instance of the designated object in a third coordinate system, and to output, via the display, the geometric representation of the motion of the hand determined relative to the first instance of the removable part relative to a second instance of the removable part in a fourth coordinate system.

13 . The computing device of claim 12 , further comprising instructions executable to receive a geometric representation of the first instance of the removable part, and to output, via the display, the geometric representation of the first instance of the removable part for viewing with the second instance of the removable part.

14 . The computing device of claim 11 , where the second pose of the designated object varies in time.

15 . The computing device of claim 11 , where the display includes an at least partially transparent display configured to present virtual imagery and real imagery.

16 . At a computing device, a method, comprising:
three-dimensionally scanning an environment including a first instance of a designated object; recording video data capturing motion of a hand relative to the first instance of the designated object; based on data obtained by three-dimensionally scanning the environment, determining a static representation of the environment; based on the video data, determining a time-varying representation of the environment; determining a first pose of the first instance of the designated object; based on the first pose, associating a first coordinate system with the first instance of the designated object; based on the static representation and the time-varying representation, determining a geometric representation of the motion of the hand in the first coordinate system, the geometric representation having a time-varying pose relative to the first pose of the first instance of the designated object; and configuring the geometric representation for display relative to a second instance of the designated object having a second pose in a second coordinate system, where the display of the geometric representation relative to the second instance of the designated object is configured with a time-varying pose relative to the second pose that is spatially consistent with the time-varying pose relative to the first pose.

17 . The method of claim 16 , further comprising:
associating a first world coordinate system with the static representation; associating a second world coordinate system with the time-varying representation; and aligning the first world coordinate system and the second world coordinate system to thereby determine an aligned world coordinate system.

18 . The method of claim 17 , wherein determining the geometric representation of the motion of the hand in the first coordinate system includes first determining a geometric representation of the motion of the hand in the aligned world coordinate system, and then transforming the geometric representation of the motion of the hand in the aligned world coordinate system from the aligned wor

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Pollefeys Marc Andre Leon
• Sinha Sudipta Narayan
• Sawhney Harpreet Singh
• Tekin Bugra
• Bogo Federica

Classifications

• G06V40/28 Primary

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06V40/19

  Sensors therefor · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G02B27/017

  Head mounted · CPC title