Neural network processing for multi-object 3d modeling

Patent metadata

Field	Value
Publication number	US-2019130639-A1
Application number	US-201816234463-A
Country	US
Kind code	A1
Filing date	Dec 27, 2018
Priority date	Aug 10, 2018
Publication date	May 2, 2019
Grant date	—

Abstract

Embodiments are directed to neural network processing for multi-object three-dimensional (3D) modeling. An embodiment of a computer-readable storage medium includes executable computer program instructions for obtaining data from multiple cameras, the data including multiple images, and generating a 3D model for 3D imaging based at least in part on the data from the cameras, wherein generating the 3D model includes one or more of performing processing with a first neural network to determine temporal direction based at least in part on motion of one or more objects identified in an image of the multiple images or performing processing with a second neural network to determine semantic content information for an image of the multiple images.

First claim

What is claimed is:

1 . A non-transitory computer-readable storage medium having stored thereon executable computer program instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
obtaining data from a plurality of cameras, the data comprising a plurality of images; and generating a three-dimensional (3D) model for 3D imaging based at least in part on the data from the plurality of cameras, wherein generating the 3D model includes one or more of the following;
performing processing with a first neural network to determine temporal direction based at least in part on motion of one or more objects identified in an image of the plurality of images; or
performing processing with a second neural network to determine semantic content information for an image of the plurality of images.

2 . The medium of claim 1 , wherein performing processing with the first neural network includes identifying a background and the one or more objects in an image and determining a temporal direction for each of the one or more objects.

3 . The medium of claim 1 , wherein performing processing with the first neural network further includes generating a separate model for the background and each of the one or more objects, the model of each of one or more objects including the respective temporal direction for the object.

4 . The medium of claim 3 , further comprising executable computer program instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
refining a 3D image utilizing the models for the background and one or more objects.

5 . The medium of claim 1 , wherein performing processing with the second neural network includes receiving image data and determining the semantic content information based at least in part on the received image data.

6 . The medium of claim 5 , further comprising executable computer program instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
generating a 3D image based at least in part on the image data and the generated semantic content information.

7 . The medium of claim 5 , further comprising executable computer program instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
performing processing with the second neural network to further determine one or more areas of interest in an image.

8 . The medium of claim 7 , further comprising executable computer program instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
generating a 3D model based at least in part on the one or more areas of interest in an image.

9 . The medium of claim 7 , further comprising executable computer program instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
analyzing camera data from a plurality of cameras utilizing the determined areas of interest, and prioritizing use of the camera data from each the plurality of cameras based at least in part on the analysis.

10 . A system comprising:
one or more processor cores; a memory to store data for three-dimensional (3D) imaging, the data comprising a plurality of images; and inputs from a plurality of cameras for 3D data capture; wherein the system is to provide one or more of the following:
3D modeling including a first neural network to determine temporal direction based at least in part on motion of one or more objects identified in an image of the plurality of images; or
3D model generation including a second neural network to determine semantic content information for an image of the plurality of images.

11 . The system of claim 10 , wherein the first neural network is to identify a background and the one or more objects in an image, and to determine a temporal direction for each of the one or more objects.

12 . The system of claim 11 , wherein the first neural network is further to generate a separate model for the background and each of the one or more objects, the model of each of one or more objects including the respective temporal direction for the object.

13 . The system of claim 12 , wherein the system is to refine a 3D image utilizing the models for the background and one or more objects.

14 . The system of claim 10 , wherein the second neural network is to receive image data and to determine the semantic content information based at least in part on the received image data.

15 . The system of claim 14 , wherein the semantic content information includes information regarding an object present in an image, an activity occurring an image, or both.

16 . The system of claim 14 , wherein the system is to generate a 3D image based at least in part on the image data and the generated semantic content information.

17 . The system of claim 14 , wherein the second neural network is further to determine one or more areas of interest in an image.

18 . The system of claim 17 , wherein the system is to generate a 3D model based at least in part on the one or more areas of interest in an image.

19 . The system of claim 17 , wherein the second neural network is further to analyze camera data from a plurality of cameras utilizing the determined areas of interest, and to prioritize use of the camera data from each the plurality of cameras based at least in part on the analysis.

20 . A head-mounted display (HMD) apparatus comprising:
a three-dimensional (3D) display; a motion detector to generate motion data for the HMD system; and one or more inputs for camera data for the 3D display; wherein the apparatus is to generate an estimate of motion of the apparatus based at least in part on the motion data and the camera data; and wherein the apparatus is to predict a next frame for viewing utilizing a neural network analysis based at least in part on the motion estimation.

21 . The apparatus of claim 20 , wherein the prediction of the next frame for viewing includes a feedback loop to determine real-time accuracy of the prediction and to apply the determined accuracy to improve future accuracy of prediction.

22 . The apparatus of claim 20 , wherein the prediction of the next frame for viewing further includes generation of a range of frames.

23 . The apparatus of claim 22 , wherein the apparatus is to select a frame from the range of frames.

24 . The apparatus of claim 20 , wherein the prediction of the next frame for viewing further utilizes additional data indicative of a user's intent regarding image viewing.

25 . The apparatus of claim 24 , wherein the additional data includes eye tracker data.

Assignees

• Intel Corp

Inventors

• Boyce Jill
• Soe Soethiha
• Panneer Selva
• Lake Adam
• Jain Nilesh
• Vembar Deepak
• Anderson Glen J
• George Varghese
• Marshall Carl
• Janus Scott
• Tangri Saurabh
• Veeramani Karthik
• Surti Prasoonkumar

Classifications

• G06V20/64

  Three-dimensional [3D] objects · CPC title

• G06V10/28

  Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title