Generating three-dimensional content from two-dimensional images

Patent metadata

Field	Value
Publication number	US-11113887-B2
Application number	US-201916428982-A
Country	US
Kind code	B2
Filing date	Jun 1, 2019
Priority date	Jan 8, 2018
Publication date	Sep 7, 2021
Grant date	Sep 7, 2021

Abstract

A method includes receiving two-dimensional video streams from a plurality of cameras, the two-dimensional video streams including multiple angles of a sporting event. The method further includes determining boundaries of the sporting event from the two-dimensional video streams. The method further includes identifying a location of a sporting object during the sporting event. The method further includes identifying one or more players in the sporting event. The method further includes identifying poses of each of the one or more players during the sporting event. The method further includes generating a three-dimensional model of the sporting event based on the boundaries of the sporting event, the location of the sporting object during the sporting event, and the poses of each of the one or more players during the sporting event. The method further includes generating a simulation of the three-dimensional model.

First claim

What is claimed is:

1. A method comprising:
receiving two-dimensional video streams from a plurality of cameras, the two-dimensional video streams including multiple angles of a live event;
determining boundaries of the live event from the two-dimensional video streams;
identifying a location of a live object during the live event;
identifying one or more players in the live event;
identifying poses of each of the one or more players during the live event;
identifying one or more social network profiles that correspond to the one or more players in the live event;
generating a three-dimensional model of the live event based on the boundaries of the live event, the location of the live object during the live event, and the poses of each of the one or more players during the live event;
generating a simulation of the three-dimensional model;
providing a presentation of the simulation of the three-dimensional model as an augmentation of real-world content in a cross-reality view presented to a user, the real-world content including a presentation of the live event that is different from the presentation of the simulation of the three-dimensional model in the cross-reality view; and
providing, together with the presentation of the simulation of the three-dimensional model and as an additional augmentation of the real-world content in the cross-reality view presented to the user, a presentation of one or more links to the one or more social network profiles of the one or more players and a presentation of an additional three-dimensional model, the additional three-dimensional model being of buttons for controlling the presentation of the simulation.

2. The method of claim 1 , further comprising:
providing the simulation to a cross reality system that includes augmented reality glasses, wherein the simulation is overlaid onto real-world elements included in the real-world content.

3. The method of claim 1 , further comprising:
generating a machine learning model, wherein the machine learning model is used to determine the boundaries, identify the location of the live event, identify the poses of each of the one or more players, generate the three-dimensional model, and generate the simulation.

4. The method of claim 3 , wherein the machine learning model is generated by at least one of a deep neural network, a convolutional neural network, and a recurrent neural network.

5. The method of claim 1 , wherein identifying the poses of the one or more players during the live event includes generating a skeletal model of each of the one or more players and the skeletal model predicts a location of corresponding body parts that are obscured in image frames of the two-dimensional video streams.

6. The method of claim 5 , further comprising:
classifying people that are in the two-dimensional video streams including the one or more players;
wherein predicting the location of corresponding body parts that are obscured in the image frames includes applying a classification of the people to determine whether the corresponding body parts are obscured by the one or more players or an other person.

7. The method of claim 6 , further comprising:
identifying one or more additional social network profiles that correspond to additional people that are in the two-dimensional video streams, wherein the additional people include spectators to the live event; and
providing one or more additional links to the one or more additional social network profiles within the simulation.

8. The method of claim 1 , further comprising generating the additional three-dimensional model of buttons based on a type of the live event.

9. The method of claim 8 , wherein the additional three-dimensional model includes at least one button that is specific to the type of the live event and that is for selecting an event in the live event for playback in the presentation of the simulation.

10. The method of claim 1 , wherein the simulation of the three-dimensional model and the additional three-dimensional model are presented in a user interface that includes an option to create a bookmark of part of the live event for later viewing.

11. A system comprising:
one or more processors coupled to a memory;
a processing module stored in the memory and executable by the one or more processors, the processing module operable to receive two-dimensional video streams from a plurality of cameras, the two-dimensional video streams including multiple angles of a live event;
a machine learning module stored in the memory and executable by the one or more processors, the machine learning module operable to generate a machine learning model to determine boundaries of the live event from the two-dimensional video streams and identify a location of a live object during the live event;
a tracking module stored in the memory and executable by the one or more processors, the tracking module operable to, based on the machine learning model, identify one or more players in the live event, identify one or more social network profiles that correspond to the one or more players in the live event, and identify poses of each of the one or more players during the live event;
a simulation module stored in the memory and executable by the one or more processors, the simulation module operable to, based on the machine learning model, generate a three-dimensional model of the live event based on the boundaries of the live event, the location of the live object during the live event, and the poses of each of the one or more players during the live event and generate a simulation of the three-dimensional model; and
a user interface module stored in memory and executable by the one or more processors, the user interface module operable to provide a presentation of the simulation of the three-dimensional model as an augmentation of real-world content in a cross-reality view presented to a user, the real-world content including a presentation of the live event that is different from the presentation of the simulation of the three-dimensional model in the cross-reality view;
wherein the user interface module is operable to provide, together with the presentation of the simulation of the three-dimensional model and as an additional augmentation of the real-world content in the cross-reality view presented to the user, a presentation of one or more links to the one or more social network profiles of the one or more players and a presentation of an additional three-dimensional model, the additional three-dimensional model being of buttons for controlling the presentation of the simulation.

12. The system of claim 11 , wherein the simulation of the three-dimensional model and the additional three-dimensional model are overlaid onto real-world elements included in the real-world content.

13. The system of claim 12 , wherein the machine learning model is generated by at least one of a deep neural network, a convolutional neural network, and a recurrent neural network.

14. The system of claim 11 , wherein identifying the poses of the one or more players during the live event includes generating a skeletal model of each of the one or more players and the skeletal model predicts a location of corresponding body parts that are obscured in image frames of the two-dimensional video streams.

15. A non-transitory computer storage medium encoded with instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving two-dimensional video streams from a plurality of cameras, the two-dimensional video streams including multiple angles o

Assignees

• Verizon Patent & Licensing Inc

Inventors

• Kopeinigg Daniel
• Van Hoff Arthur
• Lee Philip
• Hajmohammadi Solmaz
• Khire Sourabh
• Venshtain Simion
• Walkingshaw Andrew

Classifications

• G06V20/42

  of sport video content · CPC title

• G06V40/103

  Static body considered as a whole, e.g. static pedestrian or occupant recognition · CPC title

• G06V10/82

  using neural networks · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06Q10/40 Primary

  Business processes related to social networking or social networking services · CPC title