Technologies for thermal enhanced semantic segmentation of two-dimensional images

Patent metadata

Field	Value
Publication number	US-11037312-B2
Application number	US-201916457949-A
Country	US
Kind code	B2
Filing date	Jun 29, 2019
Priority date	Jun 29, 2019
Publication date	Jun 15, 2021
Grant date	Jun 15, 2021

Abstract

Technologies for thermal enhanced semantic segmentation include a computing device having a visible light camera and an infrared camera. The computing device receives a visible light image of a scene from the visible light camera and an infrared image of the scene from the infrared camera. The computing device registers the infrared image to the visible light image to generate a registered image. Registering the infrared image may include increasing resolution of the infrared image. The computing device generates a thermal boundary saliency image based on the registered infrared image. The computing device may generate the thermal boundary saliency image by applying a Gabor jet convolution to the registered infrared image. The computing device performs semantic segmentation on the visible light image, the registered infrared image, and the thermal boundary saliency image to generate a pixelwise semantic classification of the scene. Other embodiments are described and claimed.

First claim

The invention claimed is:

1. A computing device for semantic classification, the computing device comprising: an input enhancer to (i) register an input infrared light image of a scene to a visible light image of the scene to generate a registered infrared light image, wherein the registered infrared light image has a resolution equal to a resolution of the visible light image, and (ii) generate a thermal boundary saliency image based on the registered infrared light image, wherein the thermal boundary saliency image highlights saliency boundaries of the registered infrared light image; and a semantic classifier to (i) perform semantic segmentation on the visible light image, the registered infrared light image, and the thermal boundary saliency image, and (ii) output a pixelwise semantic classification of the scene in response to performance of the semantic segmentation.

2. The computing device of claim 1 , further comprising an input manager to:
receive the visible light image from a visible light camera; and
receive the input infrared light image from an infrared light camera.

3. The computing device of claim 2 , wherein the computing device, the visible light camera, and the infrared light camera are included in an advanced vehicle.

4. The computing device of claim 1 , wherein to register the input infrared light image comprises to increase a resolution of the input infrared light image to generate the registered infrared light image.

5. The computing device of claim 4 , wherein to increase the resolution of the input resolution infrared light image comprises to perform bicubic convolutional interpolation on the input infrared light image.

6. The computing device of claim 1 , wherein to register the input infrared light image comprises to transform a geometry of the input infrared light image based on a kinematic calibration associated with the input infrared light image.

7. The computing device of claim 1 , wherein to generate the thermal boundary saliency image comprises to process the registered infrared light image with a Gabor jet convolution, wherein the thermal boundary saliency image comprises a Gabor magnitude image.

8. The computing device of claim 7 , wherein the Gabor jet convolution comprises a plurality of Gabor kernel convolution layers, wherein each Gabor kernel convolution layer has a different phase angle.

9. The computing device of claim 1 , wherein to perform the semantic segmentation comprises to process the visible light image, the registered infrared light image, and the thermal boundary saliency image with an artificial neural network.

10. The computing device of claim 9 , wherein to perform the semantic segmentation comprises to: input each color of the visible light image as a channel to the artificial neural network, input the registered infrared light image as a channel to the artificial neural network, and input the thermal boundary saliency image as a channel to the artificial neural network.

11. One or more non-transitory computer-readable storage media comprising a plurality of instructions stored thereon that, in response to being executed, cause a computing device to: register an input infrared light image of a scene to a visible light image of the scene to generate a registered infrared light image, wherein the registered infrared light image has a resolution equal to a resolution of the visible light image; generate a thermal boundary saliency image based on the registered infrared light image, wherein the thermal boundary saliency image highlights saliency boundaries of the registered infrared light image; perform semantic segmentation on the visible light image, the registered infrared light image, and the thermal boundary saliency image; and output a pixelwise semantic classification of the scene in response to performing the semantic segmentation.

12. The one or more computer-readable storage media of claim 11 , further comprising a plurality of instructions stored thereon that, in response to being executed, cause the computing device to:
receive the visible light image from a visible light camera; and
receive the input infrared light image from an infrared light camera.

13. The one or more computer-readable storage media of claim 12 , wherein the computing device, the visible light camera, and the infrared light camera are included in an advanced vehicle.

14. The one or more computer-readable storage media of claim 11 , wherein to register the input infrared light image comprises to increase a resolution of the input infrared light image to generate the registered infrared light image.

15. The one or more computer-readable storage media of claim 11 , wherein to register the input infrared light image comprises to transform a geometry of the input infrared light image based on a kinematic calibration associated with the input infrared light image.

16. The one or more computer-readable storage media of claim 11 , wherein to generate the thermal boundary saliency image comprises to process the registered infrared light image with a Gabor jet convolution, wherein the thermal boundary saliency image comprises a Gabor magnitude image.

17. The one or more computer-readable storage media of claim 16 , wherein the Gabor jet convolution comprises a plurality of Gabor kernel convolution layers, wherein each Gabor kernel convolution layer has a different phase angle.

18. The one or more computer-readable storage media of claim 11 , wherein to perform the semantic segmentation comprises to process the visible light image, the registered infrared light image, and the thermal boundary saliency image with an artificial neural network.

19. A method for semantic classification, the method comprising: registering, by a computing device, an input infrared light image of a scene to a visible light image of the scene to generate a registered infrared light image, wherein the registered infrared light image has a resolution equal to a resolution of the visible light image; generating, by the computing device, a thermal boundary saliency image based on the registered infrared light image, wherein the thermal boundary saliency image highlights saliency boundaries of the registered infrared light image; performing, by the computing device, semantic segmentation on the visible light image, the registered infrared light image, and the thermal boundary saliency image; and outputting, by the computing device, a pixelwise semantic classification of the scene in response to performing the semantic segmentation.

20. The method of claim 19 , further comprising:
receiving, by the computing device, the visible light image from a visible light camera; and
receiving, by the computing device, the input infrared light image from an infrared light camera.

21. The method of claim 20 , wherein the computing device, the visible light camera, and the infrared light camera are included in an advanced vehicle.

22. The method of claim 19 , wherein registering the input infrared light image comprises increasing a resolution of the input infrared light image to generate the registered infrared light image.

23. The method of claim 19 , wherein registering the input infrared light image comprises transforming a geometry of the input infrared light image based on a kinematic calibration associated with the input infrared light image.

24. The method of claim 19 , wherein generating the thermal boundary saliency image comprises processing the registered inf

Assignees

• Intel Corp

Inventors

• Gonzalez Aguirre David Israel
• Felip Leon Javier
• Elli Maria Soledad

Classifications

• G06T7/37 Primary

  using transform domain methods · CPC title

• G06V10/895

  the filter being related to phase processing, e.g. phase-only filters · CPC title

• G06V10/803

  of input or preprocessed data · CPC title

• G06V10/449

  Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters · CPC title

• G06T7/174 Primary

  involving the use of two or more images · CPC title