View-dependent carving of reconstructed data from segmentation masks

Patent metadata

Field	Value
Publication number	US-11250616-B2
Application number	US-202016832466-A
Country	US
Kind code	B2
Filing date	Mar 27, 2020
Priority date	Mar 27, 2020
Publication date	Feb 15, 2022
Grant date	Feb 15, 2022

Abstract

Methods, systems and apparatuses may provide for technology that generates a point cloud model of an object depicted in image content associated with a plurality of physical cameras, generates a color projection of the point cloud model based on viewpoint information associated with a virtual camera, and excludes background color information from the color projection based on one or more segmentation masks associated with the object. The technology may also exclude peripheral information from the color projection based on at least one of the segmentation mask(s) and the viewpoint information.

First claim

We claim:

1. A computing system comprising:
a plurality of physical cameras to generate image content containing peripheral information;
a graphics processor coupled to the plurality of physical cameras; and
a memory including a set of instructions, which when executed by the graphics processor, cause the computing system to:
generate a point cloud model of an object depicted in the image content,
generate a color projection of the point cloud model based on viewpoint information associated with a virtual camera, wherein the peripheral information corresponds to information captured by a pair of cameras in the plurality of physical cameras when the viewpoint information is nearest to a third physical camera in the plurality of physical cameras,
exclude background color information from the color projection, based on one or more segmentation masks associated with the object,
exclude the peripheral information from the color projection based on at least one of the one or more segmentation masks and the viewpoint information, and
conduct an interpolation between the peripheral information and non-peripheral information captured by the third camera in response to the viewpoint information indicating that a perspective of the virtual camera is positioned between the pair of physical cameras, wherein the color projection is to be generated based on the interpolation.

2. The computing system of claim 1 , wherein the instructions, when executed, further cause the computing system to filter the peripheral information from the image content in accordance with the at least one of the one or more segmentation masks.

3. The computing system of claim 1 , wherein the one or more segmentation masks are to distinguish between background content and object content in the image content, and wherein the instructions, when executed, further cause the computing system to filter the background color information from the image content in accordance with the one or more segmentation masks.

4. The computing system of claim 1 , wherein the object is to be a dynamic object, the image content is to be video content and the viewpoint information is to be real-time viewpoint information.

5. A semiconductor apparatus comprising:
one or more substrates; and
logic coupled to the one or more substrates, wherein the logic is implemented at least partly in one or more of configurable logic or fixed-functionality hardware logic, the logic coupled to the one or more substrates to:
generate a point cloud model of an object depicted in image content associated with a plurality of physical cameras, wherein the image content contains peripheral information;
generate a color projection of the point cloud model based on viewpoint information associated with a virtual camera, wherein the peripheral information corresponds to information captured by a pair of physical cameras in the plurality of physical cameras when the viewpoint information is nearest to a third physical camera in the plurality of physical cameras;
exclude background color information from the color projection based on one or more segmentation masks associated with the object;
exclude the peripheral information from the color projection based on at least one of the one or more segmentation masks and the viewpoint information; and
conduct an interpolation between the peripheral information and non-peripheral information captured by the third physical camera in response to the viewpoint information indicating that a perspective of the virtual camera is positioned between the pair of physical cameras, wherein the color projection is to be generated based on the interpolation.

6. The semiconductor apparatus of claim 5 , wherein the logic coupled to the one or more substrates is to filter the peripheral information from the image content in accordance with the at least one of the one or more segmentation masks.

7. The apparatus of claim 5 , wherein the one or more segmentation masks are to distinguish between background content and object content in the image content, and wherein the logic coupled to the one or more substrates is to filter the background color information from the image content in accordance with the one or more segmentation masks.

8. The apparatus of claim 5 , wherein the object is to be a dynamic object, the image content is to be video content and the viewpoint information is to be real-time viewpoint information.

9. At least one non-transitory computer readable storage medium comprising a set of instructions, which when executed by a computing system, cause the computing system to:
generate a point cloud model of an object depicted in image content associated with a plurality of physical cameras, wherein the image content contains peripheral information;
generate a color projection of the point cloud model based on viewpoint information associated with a virtual camera, wherein the peripheral information corresponds to information captured by a pair of physical cameras in the plurality of physical cameras when the viewpoint information is nearest to a third physical camera in the plurality of physical cameras;
exclude background color information from the color projection based on one or more segmentation masks associated with the object;
exclude the peripheral information from the color projection based on at least one of the one or more segmentation masks and the viewpoint information; and
conduct an interpolation between the peripheral information and non-peripheral information captured by the third physical camera in response to the viewpoint information indicating that a perspective of the virtual camera is positioned between the pair of physical cameras, wherein the color projection is to be generated based on the interpolation.

10. The at least one non-transitory computer readable storage medium of claim 9 , wherein the instructions, when executed, further cause the computing system to filter the peripheral information from the image content in accordance with the at least one of the one or more segmentation masks.

11. The at least one non-transitory computer readable storage medium of claim 9 , wherein the one or more segmentation masks are to distinguish between background content and object content in the image content, and wherein the instructions, when executed, further cause the computing system to filter the background color information from the image content in accordance with the one or more segmentation masks.

12. The at least one non-transitory computer readable storage medium of claim 9 , wherein the object is to be a dynamic object, the image content is to be video content and the viewpoint information is to be real-time viewpoint information.

13. A method comprising:
generating a point cloud model of an object depicted in image content associated with a plurality of physical cameras, wherein the image content contains peripheral information;
generating a color projection of the point cloud model based on viewpoint information associated with a virtual camera, wherein the peripheral information corresponds to information captured by a pair of physical cameras in the plurality of physical cameras when the viewpoint information is nearest to a third physical camera in the plurality of physical cameras;
excluding background color information from the color projection based on one or more segmentation masks associated with the object;
excluding the peripheral information from the color projection based on at least one of the one or more segmentation masks and the viewpoint information; and
conducting an interpolation between the peripheral information and non-peripheral information ca

Assignees

• Intel Corp

Inventors

• Kaplan Adam
• Hovers Yuval
• Beer Ilan
• Raziel Ben

Classifications

• G06T15/55 Primary

  Radiosity · CPC title

• G06T2207/10012

  Stereo images · CPC title

• G06T7/10

  Segmentation; Edge detection (motion-based segmentation G06T7/215) · CPC title

• G06T1/20

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• G06F9/5027

  the resource being a machine, e.g. CPUs, Servers, Terminals · CPC title