Methods and systems for performing 3D simulation based on a 2D video image

Patent metadata

Field	Value
Publication number	US-11217006-B2
Application number	US-201916666338-A
Country	US
Kind code	B2
Filing date	Oct 28, 2019
Priority date	Oct 29, 2018
Publication date	Jan 4, 2022
Grant date	Jan 4, 2022

Abstract

An exemplary three-dimensional (3D) simulation system accesses a two-dimensional (2D) video image captured by a video capture device and that depicts a bounded real-world scene and a real-world object present within the bounded real-world scene. The 3D simulation system accesses respective 3D models of the bounded real-world scene and the real-world object. Based on the 2D video image, the 3D simulation system tracks a spatial characteristic of the real-world object relative to the bounded real-world scene. Based on the tracked spatial characteristic of the real-world object and the 3D models of the bounded real-world scene and the real-world object, the 3D simulation system generates a 3D simulation of the bounded real-world scene within which the real-world object is simulated in accordance with the tracked spatial characteristic of the real-world object. Corresponding methods and systems are also disclosed.

First claim

What is claimed is:

1. A method comprising:
accessing, by a three-dimensional (3D) simulation system, a two-dimensional (2D) video image captured by a video capture device and depicting a real-world event involving a bounded real-world scene and a person present within the bounded real-world scene, the person having a plurality of body parts including a particular body part;
accessing, by the 3D simulation system, a 3D model of the bounded real-world scene and a 3D model of the person, wherein the 3D model of the bounded real-world scene and the 3D model of the person are both preexisting 3D models generated prior to commencement of the real-world event and accessed for use during the real-world event;
tracking, by the 3D simulation system and based on the 2D video image, a pose of the person, wherein:
the pose comprises a combination of a spatial orientation of the person and a spatial location of the person relative to the bounded real-world scene,
the tracking of the pose is performed using machine learning technology that has been trained based on a plurality of events that are similar to, and that have occurred prior to, the real-world event, and
the tracking of the pose includes at least one of:
tracking the particular body part for a frame of the 2D video image within which the particular body part is not depicted due to being occluded from a vantage point of the video capture device, and
differentiating the person from a body part of a different person present within the bounded real-world scene, the body part of the different person contiguous with the depiction of the person within a frame of the 2D video image due to being colinear with the person and a vantage point of the video capture device; and

generating, by the 3D simulation system and based on the tracked pose of the person and the 3D models of the bounded real-world scene and the person, a 3D simulation of the bounded real-world scene within which the person is simulated in accordance with the tracked pose of the person.

2. The method of claim 1 , wherein the 3D models of the bounded real-world scene and the person are stored in and accessed from a storage repository accessible to the 3D simulation system.

3. The method of claim 1 , further comprising:
identifying, by the 3D simulation system prior to the commencement of the real-world event, a fixed space within which the real-world event is confined;
defining, by the 3D simulation system based on the identified fixed space, a boundary of the bounded real-world scene; and
generating, by the 3D simulation system based on the defined boundary and prior to the commencement of the real-world event, the 3D model of the bounded real-world scene.

4. The method of claim 1 , wherein:
the 2D video image is represented by color data that depicts the bounded real-world scene and the person during the real-world event; and
the tracking of the pose of the person is performed based exclusively on a set of data including the color data representing the 2D video image and excluding any depth data representative of the bounded real-world scene or the person.

5. The method of claim 1 , wherein:
the method further comprises accessing, by the 3D simulation system prior to the commencement of the real-world event, additional 2D video images captured by additional video capture devices disposed at different vantage points in relation to the bounded real-world scene;
the additional 2D video images each depict the bounded real-world scene and the person using color data and depth data representative of the bounded real-world scene and the person; and
the 3D models of the bounded real-world scene and the person that are accessed for use during the real-world event are generated prior to the commencement of the real-world event based on the color data and the depth data from the additional 2D video images.

6. The method of claim 1 , further comprising predicting, by the 3D simulation system at a first time and based on the 2D video image, the pose of the person as the pose of the person will be detected at a second time that occurs after the first time;
wherein the tracking of the pose of the person performed at the second time is based on the predicting of the pose of the person that was performed at the first time.

7. The method of claim 1 , wherein:
the 2D video image captured by the video capture device depicts a plurality of real-world objects each of which is present within the bounded real-world scene and one of which is the person; and
along with the person, the plurality of real-world objects further includes an inanimate object that is used by the person.

8. The method of claim 1 , embodied as computer-executable instructions on at least one non-transitory computer-readable medium.

9. The method of claim 1 , wherein the tracking of the pose of the person further includes tracking, based on a depiction in the 2D video image of the person moving within the bounded real-world scene, a plurality of key points of a skeletal model of the person.

10. The method of claim 9 , wherein the generating of the 3D simulation of the bounded real-world scene includes posing the 3D model of the person within the 3D model of the bounded real-world scene in accordance with the tracking of the plurality of key points.

11. A system comprising:
a memory storing instructions; and
a processor communicatively coupled to the memory and configured to execute the instructions to:
access a two-dimensional (2D) video image captured by a video capture device and depicting a real-world event involving a bounded real-world scene and a person present within the bounded real-world scene, the person having a plurality of body parts including a particular body part;
access a 3D model of the bounded real-world scene and a 3D model of the person, wherein the 3D model of the bounded real-world scene and the 3D model of the person are both preexisting 3D models generated prior to commencement of the real-world event and accessed for use during the real-world event;
track, based on the 2D video image, a pose of the person, wherein:
the pose comprises a combination of a spatial orientation of the person and a spatial location of the person relative to the bounded real-world scene,
the tracking of the pose is performed using machine learning technology that has been trained based on a plurality of events that are similar to, and that have occurred prior to, the real-world event, and
the tracking of the pose includes at least one of:
tracking the particular body part for a frame of the 2D video image within which the particular body part is not depicted due to being occluded from a vantage point of the video capture device, and
differentiating the person from a body part of a different person present within the bounded real-world scene, the body part of the different person contiguous with the depiction of the person within a frame of the 2D video image due to being colinear with the person and a vantage point of the video capture device; and

generate, based on the tracked pose of the person and the 3D models of the bounded real-world scene and the person, a 3D simulation of the bounded real-world scene within which the person is simulated in accordance with the tracked pose of the person.

12. The system of claim 11 , wherein the 3D models of the bounded real-world scene and the person are stored in and accessed from a storage repository accessible to the system.

13. The system of claim 11 , wherein the processor is further configured to execute the instructions to:
identify, prior to the commencement of the real-world event, a fixed space within which the

Assignees

• Verizon Patent & Licensing Inc

Inventors

• Van Hoff Arthur
• Kopeinigg Daniel
• Lee Philip
• Hajmohammadi Solmaz
• Khire Sourabh
• Venshtain Simion

Classifications

• G06T7/248

  involving reference images or patches · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/251

  involving models · CPC title

• G06T19/00

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G06T2207/20081

  Training; Learning · CPC title