Producing a segmented image of a scene

Patent metadata

Field	Value
Publication number	US-10586337-B2
Application number	US-201715857821-A
Country	US
Kind code	B2
Filing date	Dec 29, 2017
Priority date	Dec 30, 2016
Publication date	Mar 10, 2020
Grant date	Mar 10, 2020

Abstract

A computer-implemented method of computer vision in a scene that includes one or more transparent objects and/or one or more reflecting objects comprises obtaining a plurality of images of the scene, each image corresponding to a respective acquisition of a physical signal, the plurality of images including at least two images corresponding to different physical signals; and generating a segmented image of the scene based on the plurality of images. This improves the field of computer vision.

First claim

The invention claimed is:

1. A computer-implemented method of computer vision in a scene that includes one or more transparent objects and/or one or more reflecting objects, the method comprising:
obtaining a plurality of images of the scene, each image corresponding to a respective acquisition of a physical signal, the plurality of images including at least two images corresponding to different physical signals, the plurality of images including an infrared image and one or both of an RGB image and a depth image; and
generating a segmented image of the scene based on the plurality of images,
wherein
when the plurality of images includes the depth image captured by a depth sensor, the scene includes at least one transparent object which is crossed by rays emitted by the depth sensor, and the generating of the segmented image includes combining information from the infrared image and information from the depth image to create borders segmenting the transparent object, and
when the plurality of images includes the RGB image, the scene includes at least one reflecting object which has at least one color and a reflecting surface that reflects other colors, and the generating of the segmented image includes combining information from the infrared image and information from the RGB image to create borders segmenting the reflecting object.

2. The method of claim 1 , wherein the infrared image is a thermal image.

3. The method of claim 1 , wherein the scene is a building interior scene or a building exterior scene.

4. The method of claim 3 , wherein the scene comprises at least one biological entity.

5. The method of claim 3 , further comprising:
iterating the obtaining of a plurality of images and the generating of a segmented image of the scene to generate a plurality of segmented images of the scene, and
reconstructing a 3D model of the scene based on corresponding segments of the plurality of segmented images.

6. The method of claim 5 , wherein the corresponding segments of the plurality of segmented images, based on which the reconstructing of a 3D model of the scene is performed, all correspond to non-biological entities.

7. The method of claim 5 , wherein iterating the obtaining of a plurality of images is performed by obtaining a plurality of videos of the scene, each video corresponding to a respective video acquisition of a physical signal.

8. The method of claim 7 , wherein each video acquisition is performed with a camera on which a plurality of sensors are mounted, each sensor corresponding to a respective physical signal.

9. The method of claim 8 , wherein the camera is moved in the scene and simultaneously performs a plurality of video acquisitions each of a respective physical signal.

10. The method of claim 1 , wherein the generating of the segmented image is performed by a Markov Random Field energy minimization.

11. A non-transitory data storage medium having recorded thereon a computer program comprising instructions for performing a computer-implemented method of computer vision in a scene that includes one or more transparent objects and/or one or more reflecting objects, the method comprising:
obtaining a plurality of images of the scene, each image corresponding to a respective acquisition of a physical signal, the plurality of images including at least two images corresponding to different physical signals, the plurality of images including an infrared image and one or both of an RGB image and a depth image; and
generating a segmented image of the scene based on the plurality of images,
wherein
when the plurality of images includes the depth image captured by a depth sensor, the scene includes at least one transparent object which is crossed by rays emitted by the depth sensor, and the generating of the segmented image includes combining information from the infrared image and information from the depth image to create borders segmenting the transparent object, and
when the plurality of images includes the RGB image, the scene includes at least one reflecting object which has at least one color and a reflecting surface that reflects other colors, and the generating of the segmented image includes combining information from the infrared image and information from the RGB image to create borders segmenting the reflecting object.

12. A system comprising:
a processor coupled to a memory having recorded thereon a computer program comprising instructions for performing a computer-implemented method of computer vision in a scene that includes one or more transparent objects and/or one or more reflecting objects, the instructions causing the processor to be configured to:
obtain a plurality of images of the scene, each image corresponding to a respective acquisition of a physical signal, the plurality of images including at least two images corresponding to different physical signals, the plurality of images including an infrared image and one or both of an RGB image and a depth image; and
generate a segmented image of the scene based on the plurality of images,
wherein
when the plurality of images includes the depth image captured by a depth sensor, the scene includes at least one transparent object which is crossed by rays emitted by the depth sensor, and the generation of the segmented image includes combining information from the infrared image and information from the depth image to create borders segmenting the transparent object, and
when the plurality of images includes the RGB image, the scene includes at least one reflecting object which has at least one color and a reflecting surface that reflects other colors, and the generating of the segmented image includes combining information from the infrared image and information from the RGB image to create borders segmenting the reflecting object.

13. The system of claim 12 , further comprising an infrared sensor and one or both of an RGB sensor and a depth sensor.

14. The system of claim 12 , wherein the infrared image is a thermal image.

15. The system of claim 12 , wherein the scene is a building interior scene or a building exterior scene, and/or the scene comprises at least one biological entity.

16. The system of claim 15 , wherein the processor is further configured to
iterate the obtaining of a plurality of images and the generating of a segmented image of the scene to generate a plurality of segmented images of the scene, and
reconstruct a 3D model of the scene based on corresponding segments of the plurality of segmented images, and/or the corresponding segments of the plurality of segmented images based on which the reconstructing of a 3D model of the scene is performed all correspond to non-biological entities.

17. The system of claim 16 , wherein iterating the obtaining of a plurality of images is performed by obtaining a plurality of videos of the scene, each video corresponding to a respective video acquisition of a physical signal.

18. The system of claim 17 , wherein each video acquisition is performed with a camera on which a plurality of sensors are mounted, each sensor corresponding to a respective physical signal, and/or the camera is moved in the scene and simultaneously performs a plurality of video acquisitions each of a respective physical signal.

19. The system of claim 12 , wherein the generating of the segmented image is performed by a Markov Random Field energy minimization.

Assignees

• Dassault Systemes

Inventors

• Sanchez Bermudez Fernando
• Mehr Eloi
• Bonner David
• Guitteny Vincent
• Boufarguine Mourad
• Johnson Patrick

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06T7/174

  involving the use of two or more images · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T7/143 Primary

  involving probabilistic approaches, e.g. Markov random field [MRF] modelling · CPC title