Marker based tracking

Patent metadata

Field	Value
Publication number	US-10657704-B1
Application number	US-202016781575-A
Country	US
Kind code	B1
Filing date	Feb 4, 2020
Priority date	Nov 1, 2017
Publication date	May 19, 2020
Grant date	May 19, 2020

Abstract

A tracking system converts images to a set of points in 3D space. The images are of a wearable item that includes markers, and the set of points include representations of the markers. A view is selected from a plurality of views using the set of points, and the selected view includes one or more representations of the representations. A depth map is generated based on the selected view and the set of points, and the depth map includes the one or more representations. A neural network maps labels to the one or more representations in the depth map using a model of a portion of a body that wears the wearable item. A joint parameter is determined using the mapped labels. The model is updated with the joint parameter, and content provided to a user of the wearable item is based in part on the updated model.

First claim

What is claimed is:

1. A method comprising:
mapping of labels, via a neural network, to one or more representations in a depth map using a model of a portion of a body that wears a wearable item, wherein the wearable item includes the markers and the representations in the depth model are of the markers;
determining a joint parameter using the mapped labels; and
updating the model with the joint parameter, wherein content provided to a user of the wearable item is based in part on the updated model.

2. The method of claim 1 , wherein the mapping of labels, via the neural network, to one or more representations in the depth map using the model, further comprises:
mapping positions of joints described in the model to image space positions using the neural network;
matching the image space positions of the joints to image space positions of the markers; and
labeling the matched positions of the joints.

3. The method of claim 2 , wherein the matching the image space positions of the joints to image space positions of the markers is done using minimum-cost bipartite matching.

4. The method of claim 1 , comprising:
converting images to a set of points in 3D space, wherein the images are of the wearable item, and the set of points in 3D space include representations of the markers;
selecting a view from a plurality of views using the set of points, wherein the selected view includes one or more representations of the representations; and
generating the depth map based on the selected view and the set of points, wherein the depth map includes the one or more representations.

5. The method of claim 4 , wherein the images of the wearable item include a plurality of viewpoints of the wearable item.

6. The method of claim 4 , wherein selecting the view from the plurality of views using the set of points comprises:
determining an orientation of a synthetic camera relative to the set of points that includes at least a threshold number of representations; and
generating the view based on the determined orientation and a field of view of the synthetic camera.

7. The method of claim 4 , wherein selecting the view from the plurality of views using the set of points comprises:
determining an orientation, of a plurality of orientations, of a synthetic camera relative to the set of points that includes a largest number of representations; and
generating the view based on the determined orientation and a field of view of the synthetic camera.

8. The method of claim 4 , further comprising:
capturing, via a plurality of cameras, the images of the wearable item, wherein the captured images include at least one of the markers.

9. The method of claim 8 , wherein the plurality of cameras each have a different field of view than any of the plurality of views.

10. The method of claim 1 , wherein the neural network is a convolutional neural network.

11. The method of claim 1 , wherein the wearable item is a glove.

12. A non-transitory computer readable medium configured to store program code instructions, when executed by a processor, cause the processor to perform steps comprising:
mapping of labels, via a neural network, to one or more representations in a depth map using a model of a portion of a body that wears a wearable item, wherein the wearable item includes the markers and the representations in the depth model are of the markers;
determining a joint parameter using the mapped labels; and
updating the model with the joint parameter, wherein content provided to a user of the wearable item is based in part on the updated model.

13. The computer readable medium of claim 12 , wherein the mapping of labels, via the neural network, to one or more representations in the depth map using the model, further comprises:
mapping positions of joints described in the model to image space positions using the neural network;
matching the image space positions of the joints to image space positions of the markers; and
labeling the matched positions of the joints.

14. The computer readable medium of claim 13 , wherein the matching the image space positions of the joints to image space positions of the markers is done using minimum-cost bipartite matching.

15. The computer readable medium of 12 , the steps further comprising:
converting images to a set of points in 3D space, wherein the images are of the wearable item, and the set of points in 3D space include representations of the markers;
selecting a view from a plurality of views using the set of points, wherein the selected view includes one or more representations of the representations; and
generating the depth map based on the selected view and the set of points, wherein the depth map includes the one or more representations.

16. The computer readable medium of 15 , wherein the images of the wearable item include a plurality of viewpoints of the wearable item.

17. The computer readable medium of 15 , wherein selecting the view from the plurality of views using the set of points comprises:
determining an orientation of a synthetic camera relative to the set of points that includes at least a threshold number of representations; and
generating the view based on the determined orientation and a field of view of the synthetic camera.

18. The computer readable medium of 15 , wherein selecting the view from the plurality of views using the set of points comprises:
determining an orientation, of a plurality of orientations, of a synthetic camera relative to the set of points that includes a largest number of representations; and
generating the view based on the determined orientation and a field of view of the synthetic camera.

19. The computer readable medium of 15 , the steps further comprising:
capturing, via a plurality of cameras, the images of the wearable item, wherein the captured images include at least one of the markers.

20. The computer readable medium of 19 , wherein the plurality of cameras each have a different field of view than any of the plurality of views.

Assignees

• Facebook Tech Llc

Inventors

• Han Shangchen
• Twigg Christopher David
• Wang Robert Y

Classifications

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T17/10

  Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes · CPC title

• G06T7/75 Primary

  involving models · CPC title