Providing a 2-dimensional dataset from 2-dimensional and 3-dimensional computer vision techniques

Patent metadata

Field	Value
Publication number	US-2021287433-A1
Application number	US-202016814051-A
Country	US
Kind code	A1
Filing date	Mar 10, 2020
Priority date	Mar 10, 2020
Publication date	Sep 16, 2021
Grant date	—

Abstract

Implementations generally provide a 2-dimensional dataset from 2-dimensional and 3-dimensional computer vision techniques. In some implementations, a method includes obtaining a plurality of 2-dimensional (2D) videos of a subject performing at least one action. The method further includes generating a 3-dimensional (3D) model based on the plurality of 2D videos. The method further includes generating a 3D scene based on the 3D model. The method further includes generating a 2D dataset based on the 3D scene.

First claim

What is claimed is:

1 . A system comprising:
one or more processors; and logic encoded in one or more non-transitory computer-readable storage media for execution by the one or more processors and when executed operable to cause the one or more processors to perform operations comprising: obtaining a plurality of 2-dimensional (2D) videos of a subject performing at least one action; generating a 3-dimensional (3D) model based on the plurality of 2D videos; generating a 3D scene based on the 3D model; and generating a 2D dataset based on the 3D scene.

2 . The system of claim 1 , wherein the plurality of 2D videos is synchronized.

3 . The system of claim 1 , wherein the plurality of 2D videos is obtained from a plurality of physical cameras that are positioned at arbitrary locations in a physical environment.

4 . The system of claim 1 , wherein the logic when executed is further operable to cause the one or more processors to perform operations comprising obtaining one or more annotations associated with the plurality of 2D videos.

5 . The system of claim 1 , wherein the logic when executed is further operable to cause the one or more processors to perform operations comprising:
determining one or more model data modifications to 3D model data; and applying the one or more model data modifications to the 3D model data.

6 . The system of claim 1 , wherein the logic when executed is further operable to cause the one or more processors to perform operations comprising:
determining one or more scene settings; generating one or more virtual cameras; and
adding the one or more virtual cameras to the 3D scene set based on the scene settings.

7 . The system of claim 1 , wherein the logic when executed is further operable to cause the one or more processors to perform operations comprising:
obtaining one or more annotations associated with the plurality of 2D videos; and applying the one or more annotations to the 2D dataset.

8 . A non-transitory computer-readable storage medium with program instructions stored thereon, the program instructions when executed by one or more processors are operable to cause the one or more processors to perform operations comprising:
obtaining a plurality of 2-dimensional (2D) videos of a subject performing at least one action; generating a 3-dimensional (3D) model based on the plurality of 2D videos; generating a 3D scene based on the 3D model; and generating a 2D dataset based on the 3D scene.

9 . The computer-readable storage medium of claim 8 , wherein the plurality of 2D videos is synchronized.

10 . The computer-readable storage medium of claim 8 , wherein the plurality of 2D videos is obtained from a plurality of physical cameras that are positioned at arbitrary locations in a physical environment.

11 . The computer-readable storage medium of claim 8 , wherein the instructions when executed are further operable to cause the one or more processors to perform operations comprising obtaining one or more annotations associated with the plurality of 2D videos.

12 . The computer-readable storage medium of claim 8 , wherein the instructions when executed are further operable to cause the one or more processors to perform operations comprising:
determining one or more model data modifications to 3D model data; and applying the one or more model data modifications to the 3D model data.

13 . The computer-readable storage medium of claim 8 , wherein the instructions when executed are further operable to cause the one or more processors to perform operations comprising:
determining one or more scene settings; generating one or more virtual cameras; and
adding the one or more virtual cameras to the 3D scene set based on the scene settings.

14 . The computer-readable storage medium of claim 8 , wherein the instructions when executed are further operable to cause the one or more processors to perform operations comprising:
obtaining one or more annotations associated with the plurality of 2D videos; and
applying the one or more annotations to the 2D dataset.

15 . A computer-implemented method comprising:
obtaining a plurality of 2-dimensional (2D) videos of a subject performing at least one action; generating a 3-dimensional (3D) model based on the plurality of 2D videos; generating a 3D scene based on the 3D model; and generating a 2D dataset based on the 3D scene.

16 . The method of claim 15 , wherein the plurality of 2D videos is synchronized.

17 . The method of claim 15 , wherein the plurality of 2D videos is obtained from a plurality of physical cameras that are positioned at arbitrary locations in a physical environment.

18 . The method of claim 15 , further comprising obtaining one or more annotations associated with the plurality of 2D videos.

19 . The method of claim 15 , further comprising:
determining one or more model data modifications to 3D model data; and applying the one or more model data modifications to the 3D model data.

20 . The method of claim 15 , further comprising:
determining one or more scene settings; generating one or more virtual cameras; and
adding the one or more virtual cameras to the 3D scene set based on the scene settings.

Assignees

• Sony Corp

Inventors

• Lyons Gary
• Suzuki Kenji
• Faust Brent
• Lee Jong Hwa

Classifications

• G06V20/647

  by matching two-dimensional images to three-dimensional objects · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V20/49

  Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06T2207/30196

  Human being; Person · CPC title