Motion capture from a mobile self-tracking device

Patent metadata

Field	Value
Publication number	US-10845188-B2
Application number	US-201614987999-A
Country	US
Kind code	B2
Filing date	Jan 5, 2016
Priority date	Jan 5, 2016
Publication date	Nov 24, 2020
Grant date	Nov 24, 2020

Abstract

Methods and apparatus for capturing motion from a self-tracking device are disclosed. In embodiments, a device self-tracks motion of the device relative to a first reference frame while recording motion of a subject relative to a second reference frame, the second reference frame being a reference frame relative to the device. In the embodiments, the subject may be a real object or, alternately, the subject may be a virtual subject and a motion of the virtual object may be recorded relative to the second reference frame by associating a position offset relative to the device with the position of the virtual object in the recorded motion. The motion of the subject relative to the first reference frame may be determined from the tracked motion of the device relative to the first frame and the recorded motion of the subject relative to the second reference frame.

First claim

What is claimed is:

1. A single, portable device comprising:
a plurality of sensors;
at least one processor coupled to the plurality of sensors; and
memory coupled to the at least one processor, the memory including code, which when executed causes the at least one processor to control the device to:
generate first data associated with one or more locations of the device from a first at least one sensor of the plurality of sensors;
track a motion of the device relative to a first reference frame based on the first data, wherein the first reference frame comprises a first three-dimensional coordinate system relative to an environment;
generate second data associated with a subject from a second at least one sensor of the plurality of sensors;
record a motion of the subject in the environment relative to a second reference frame as the subject moves relative to the first reference frame and as the device moves to track at least a portion of the motion of the subject by action of a user, the motion of the subject in the environment relative to the second reference frame being recorded based on the second data, wherein the second reference frame comprises a second three-dimensional coordinate system relative to the device;
create a 3D mesh on a surface in a video of the recorded motion of the subject; and
control a virtual asset in the video to deviate from the recorded motion of the subject based on the 3D mesh.

2. The single, portable device of claim 1 , wherein the subject is a virtual object and the code is executable to cause the at least one processor to control the device to record the motion of the subject by controlling the device to:
record a motion of the virtual object in the environment relative to the second reference frame based on the second data by associating a position that is offset relative to the device with one or more locations of the virtual object in the recorded motion of the virtual object.

3. The single, portable device of claim 1 , wherein the code is executable to cause the at least one processor to further control the device to:
determine a motion of the subject relative to the first reference frame from the tracked motion of the device relative to the first reference frame and the recorded motion of the subject relative to the second reference frame.

4. The single, portable device of claim 1 , wherein the code is executable to cause the at least one processor to further control the device to:
determine a motion of the subject relative to a third reference frame from the tracked motion of the device relative to the first reference frame and the recorded motion of the subject relative to the second reference frame, the third reference frame having a determinable relationship with the first reference frame.

5. The single, portable device of claim 1 , wherein the subject is a real object and the code is executable to cause the at least one processor to control the device to record the motion of the subject by controlling the device to:
record a skeletal motion of the real object in the environment relative to the second reference frame based on the second data.

6. The single, portable device of claim 1 , wherein the subject is a real object and the code is executable to cause the at least one processor to control the device to record the motion of the subject by controlling the device to:
record a motion of the real object in the environment relative to the second reference frame based on the second data by utilizing at least one marker.

7. The single, portable device of claim 1 , wherein the subject is a virtual object and the code is executable to further cause the at least one processor to control the device to:
generate the video by recording of a playback of the recorded motion of the subject with the virtual asset in place of the virtual object.

8. The single, portable device of claim 1 , wherein the plurality of sensors comprises:
a depth camera to generate a depth data signal;
an RGB camera to generate a RGB data signal; and,
an inertial measurement unit to generate an inertial data signal.

9. A method comprising:
generating first data associated with one or more locations of a device from a first at least one sensor of a plurality of sensors on the device, wherein the device is a single, portable device that a person can carry;
tracking, at the device, a motion of the device relative to a first reference frame based on the first data, wherein the first reference frame comprises a first three-dimensional coordinate system relative to an environment;
generating, at the device, second data associated with a subject from a second at least one sensor of the plurality of sensors on the device;
recording, from the device, a motion of the subject in the environment relative to a second reference frame as the subject moves relative to the first reference frame and as the device moves to track at least a portion of the motion of the subject by action of a user, the motion of the subject in the environment relative to the second reference frame being recorded based on the second data, wherein the second reference frame comprises a second three-dimensional coordinate system relative to the device;
determining a motion of the subject relative to the first reference frame from the tracked motion of the device relative to the first reference frame and the recorded motion of the subject relative to the second reference frame;
generating a video of the recorded motion of the subject relative to the second reference frame with a virtual asset in place of the subject;
creating a 3D mesh on a surface in the video; and
initiating a deviation, in the video, of the virtual asset from the recorded motion of the subject relative to the second reference frame based on the 3D mesh.

10. The method of claim 9 wherein the subject is a virtual object and the recording the motion of the subject in the environment relative to the second reference frame comprises:
recording a motion of the virtual object in the environment relative to the second reference frame based on the second data by associating a position offset relative to the device with one or more locations of the virtual object in the recorded motion of the virtual object.

11. The method of claim 9 further comprising:
determining a motion of the subject relative to a third reference frame from the tracked motion of the device relative to the first reference frame and the recorded motion of the subject relative to the second reference frame, the third reference frame having a determinable relationship with the first reference frame.

12. The method of claim 9 wherein the subject is a virtual object and
generating the video comprises recording a playback of the recorded motion of the subject with the virtual asset in place of the virtual object.

13. An apparatus comprising:
an input; and
a circuit in communication with the input, the circuit configured to:
receive first data comprising one or more self-tracked locations of a device relative to a first reference frame, wherein the first reference frame comprises a first three-dimensional coordinate system relative to an environment and wherein the device is a dedicated camera device or a mobile computing device;
receive second data recorded at the device, the second data comprising recorded motion of a subject relative to a second reference frame, wherein the second reference frame comprises a second three-dimensional coordinate system relative to the device, and wherein the recorded motion of the subject was recorded while the device moved to track at least a portion of the recorded motion of the subjec

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Weiss John
• Pradeep Vivek
• Hu Xiaoyan

Classifications

• H04N23/11

  for generating image signals from visible and infrared light wavelengths · CPC title

• G06T7/246 Primary

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06T2207/30241

  Trajectory · CPC title

• G06T2207/30244

  Camera pose · CPC title