Automatic point cloud validation for immersive media

Patent metadata

Field	Value
Publication number	US-11869141-B2
Application number	US-201917436481-A
Country	US
Kind code	B2
Filing date	May 14, 2019
Priority date	May 14, 2019
Publication date	Jan 9, 2024
Grant date	Jan 9, 2024

Abstract

Techniques related to validating an image based 3D model of a scene are discussed. Such techniques include detecting an object within a captured image used to generate the scene, projecting the 3D model to a view corresponding to the captured image to generate a reconstructed image, and comparing image regions of the captured and reconstructed images corresponding to the object to validate the 3D model.

First claim

What is claimed is:

1. An apparatus for validating a 3D model, comprising:
a memory to store a first captured image attained via a first camera of a plurality of cameras trained on a scene; and
a processor coupled to the memory, the processor to:
detect an object within an image region of the first captured image, wherein the first captured image comprises one of a plurality of simultaneously captured images of the scene at a first time instance;
generate, based on the simultaneously captured images, a 3D model of the scene corresponding to the first time instance;
project the 3D model to a view of the first camera relative to the scene to generate a first reconstructed image representative of the scene from the view of the first camera at the first time instance;
evaluate representation of the object in the 3D model based on a difference metric, the difference metric based on a comparison of first image content of the first captured image within the image region and second image content of the first reconstructed image within the image region; and
generate a 3D model error indicator in response to the difference metric comparing unfavorably to a threshold.

2. The apparatus of claim 1 , wherein the difference metric comprises one of a pixel by pixel comparison of pixel values of the first and second image content, a shape comparison of shapes detected within the first and second image content, or a human pose comparison of human poses detected within the first and second image content.

3. The apparatus of claim 1 , wherein the image region comprises a bounding box having coordinates in the first captured image and the processor to determine the difference metric comprises the processor to apply the same bounding box coordinates to the first captured image and the first reconstructed image to determine the corresponding first and second image content.

4. The apparatus of claim 1 , the processor further to:
detect a plurality of second image regions each corresponding to the object as detected in the remaining simultaneously captured images of the scene;
project the 3D model to each view of the remaining plurality of cameras to generate second reconstructed images representative of the scene from the views of the remaining cameras; and
determine a plurality of second difference metrics based on comparisons of each corresponding image content of the second image regions within the captured images and the reconstructed images, wherein the 3D model error indicator is further in response to the plurality of second difference metrics.

5. The apparatus of claim 4 , wherein the 3D model error indicator is generated in response to an average of the difference metric and the second difference metrics exceeding a second threshold.

6. The apparatus of claim 1 , the processor further to:
detect a second object within a second image region of the first captured image;
determine a second difference metric based on a comparison of third image content of the first captured image within the second image region and fourth image content of the first reconstructed image within the second image region; and
generate a second 3D model error indicator in response to the second difference metric being greater than a second threshold, wherein the difference metric comparing unfavorably to the threshold comprises the difference metric being greater than the threshold, and wherein the threshold is less than the second threshold in response to the image region being closer to a center of the first captured image than the second image region or the threshold is greater than the second threshold in response to the image region having a lower image region density than the second image region.

7. The apparatus of claim 6 , wherein the threshold and the second threshold are determined by application of a monotonically increasing function to a distance from image center of the image region and the second image region or by application of a monotonically decreasing function to an image region density of the image region and the second image region.

8. The apparatus of claim 1 , wherein the processor to evaluate representation of the object in the 3D model comprises the processor to determine the object is not represented in the 3D model.

9. The apparatus of claim 1 , wherein the processor to detect the object within the image region comprises the processor to:
perform object detection on the first captured image to detect the object and an image region; and
adjust a location of the image region within the first captured image using a geometric constraint based on detection of the object within one or more of the plurality of simultaneously captured images of the scene.

10. A method for validating a 3D model comprising:
detecting an object within an image region of a first captured image attained via a first camera of a plurality of cameras trained on a scene, wherein the first captured image comprises one of a plurality of simultaneously captured images of the scene at a first time instance;
generating, based on the simultaneously captured images, a 3D model of the scene corresponding to the first time instance;
projecting the 3D model to a view of the first camera relative to the scene to generate a first reconstructed image representative of the scene from the view of the first camera at the first time instance;
evaluating representation of the object in the 3D model based on a difference metric, the difference metric based on a comparison of first image content of the first captured image within the image region and second image content of the first reconstructed image within the image region; and
generating a 3D model error indicator in response to the difference metric comparing unfavorably to a threshold.

11. The method of claim 10 , wherein the difference metric comprises one of a pixel by pixel comparison of pixel values of the first and second image content, a shape comparison of shapes detected within the first and second image content, or a human pose comparison of human poses detected within the first and second image content.

12. The method of claim 10 , further comprising:
detecting a plurality of second image regions each corresponding to the object as detected in the remaining simultaneously captured images of the scene;
projecting the 3D model to each view of the remaining plurality of cameras to generate second reconstructed images representative of the scene from the views of the remaining cameras; and
determining a plurality of second difference metrics based on comparisons of each corresponding image content of the second image regions within the captured images and the reconstructed images, wherein generating the 3D model error indicator is further in response to the plurality of second difference metrics.

13. The method of claim 10 , further comprising:
detecting a second object within a second image region of the first captured image;
determining a second difference metric based on a comparison of third image content of the first captured image within the second image region and fourth image content of the first reconstructed image within the second image region; and
generating a second 3D model error indicator in response to the second difference metric being greater than a second threshold, wherein the difference metric comparing unfavorably to the threshold comprises the difference metric being greater than the threshold, and wherein the threshold is less than the second threshold in response to the image region being closer to a center of the first captured image than the second image region.

14. The method of claim 10

Assignees

• Intel Corp

Inventors

• Tong Xiaofeng
• Li Wenlong

Classifications

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06V10/255

  Detecting or recognising potential candidate objects based on visual cues, e.g. shapes · CPC title

• G06T2200/08

  involving all processing steps from image acquisition to 3D model generation · CPC title

• G06T15/04 Primary

  Texture mapping · CPC title