Identification of transparent objects from image discrepancies

Patent metadata

Field	Value
Publication number	US-11094074-B2
Application number	US-201916518791-A
Country	US
Kind code	B2
Filing date	Jul 22, 2019
Priority date	Jul 22, 2019
Publication date	Aug 17, 2021
Grant date	Aug 17, 2021

Abstract

A computing system is provided. The computing system includes a visible light camera, a thermal camera, and a processor with associated storage. The processor is configured to execute instructions stored in the storage to receive, from the visible light camera, a visible light image for a frame of a scene and receive, from the thermal camera, a thermal image for the frame of the scene. The processor is configured to detect image discrepancies between the visible light image and the thermal image and, based on the detected image discrepancies, determine a presence of a transparent object in the scene. The processor is configured to, based on the detected image discrepancies, output an identification of at least one location in the scene that is associated with the transparent object.

First claim

The invention claimed is:

1. A computing system, comprising:
a visible light camera;
a thermal camera; and
a processor and associated storage, the processor being configured to execute instructions stored in the storage to:
receive, from the visible light camera, a visible light image for a frame of a scene;
receive, from the thermal camera, a thermal image for the frame of the scene;
detect image discrepancies between the visible light image and the thermal image, said image discrepancies including image features more discernable in the thermal image and less discernable in the visible light image;
based on the detected image discrepancies, determine a presence of a transparent object in the scene;
based on the detected image discrepancies, identify a plurality of pixels in the frame that are associated with the transparent object and a plurality of pixels in the frame that are associated with image content bordering the transparent object;
determine depth values for the plurality of pixels associated with the image content bordering the transparent object; and
infill, for each of the plurality of pixels in the frame that are associated with the transparent object, depth values corresponding to physical depths of the transparent object in the scene based on the depth values of the plurality of pixels associated with the image content bordering the transparent object.

2. The computing system of claim 1 , the processor further configured to execute the instructions to:
identify surface differentiation of the transparent object in the frame and associate one or more pixels with the surface differentiation;
for the one or more pixels associated with the surface differentiation, determine depth values corresponding to physical depths of the surface differentiation; and
for at least some pixels in the frame that are associated with the transparent object and that are not associated with the surface differentiation, determine, based on at least one depth value of the surface differentiation of the transparent object and the depth values for the plurality of pixels associated with the image content bordering the transparent object, depth values that are transparent object depth values corresponding to physical depths of the transparent object.

3. The computing system of claim 1 , further comprising a depth detection system including components selected from the group consisting of a pair of stereo cameras, a pair of stereo low-level light cameras, a single camera and an inertial measurement unit (IMU), the single camera and a light projector, a pair of cameras and the light projector, and a laser light source and a camera, wherein determining the first and second depth values is based on images received from the depth detection system.

4. The computing system of claim 1 , further comprising a display, the processor further configured to execute the instructions to:
generate, from the depth values for the plurality of pixels associated with the transparent object, a visual representation of the transparent object in the scene; and
output the visual representation of the transparent object to the display.

5. The computing system of claim 1 , wherein the visible light camera and the thermal camera are included in a head-mounted display (HMD) device having a display, the processor further configured to execute the instructions to generate, from the first and second depth values, a three-dimensional representation of the scene that is displayed on the display of the HMD device.

6. The computing system of claim 1 , the instructions further comprising a machine learning (ML) algorithm that has been trained using a training data set and is configured to receive the visible light image and the thermal image, and wherein the ML algorithm is executable by the processor to:
determine the depth values for the plurality of pixels associated with the image content bordering the transparent object; and
infill, for each of the plurality of pixels in the frame that are associated with the transparent object, the depth values corresponding to physical depths of the transparent object in the scene.

7. The computing system of claim 1 , wherein the frame is a first frame, the visible light image is a first visible light image, and the thermal image is a first thermal image, the processor further configured to execute the instructions to:
receive a second visible light image for a second frame of the scene;
receive a second thermal image for the second frame of the scene;
determine parallax values for image content in the first and second frames based on the first and second visible light images and on the first and second thermal images;
for at least some pixels in one or both of the first frame and the second frame, and based on the determined parallax values, determine first depth values corresponding to physical depths of image content not associated with the transparent object; and
for at least some pixels in one or both of the first frame and the second frame, and based on the determined parallax values, determine second depth values corresponding to physical depths of image content associated with the transparent object.

8. The computing system of claim 1 , the instructions further comprising a machine learning (ML) algorithm that has been trained using a training data set as a classifier for transparent objects and is configured to receive the visible light image and the thermal image, wherein determining the presence of the transparent object in the scene includes executing the ML algorithm on the processor to determine, based on the detected image discrepancies between the visible light image and the thermal image, the presence of the transparent object in the scene.

9. The computing system of claim 1 , the processor further configured to execute the instructions to:
determine first depth values corresponding to physical depths for at least one location in the scene that is not associated with the transparent object;
determine second depth values corresponding to physical depths for the at least one location in the scene that is associated with the transparent object;
distribute estimated depth values for a three-dimensional space using calibration parameters of a depth detection system to generate a point cloud for the scene, points of the point cloud associated with the transparent object being identified in the point cloud;
generate a surface mesh of the point cloud by connecting points in the point cloud to form polygons;
filter points and polygons in the surface mesh according to a filtering algorithm; and
output a filtered surface mesh that is a three-dimensional representation of the scene indicating transparent spatial locations in the mesh.

10. The computing system of claim 1 , wherein
the visible light camera is configured to receive and detect light in a range of 350 nm to 1200 nm and/or 1000 nm to 1600 nm, and
the thermal camera is configured to receive and detect light in a range of 5,000 nm to 15,000 nm.

11. A method for use with a computing device including a processor and associated storage, the processor being configured to execute instructions stored in the storage, the method comprising:
at the processor:
receiving, from a visible light camera, a visible light image for a frame of a scene;
receiving, from a thermal camera, a thermal image for the frame of the scene;
detecting image discrepancies between the visible light image and the thermal image, said image discrepancies including image features more discernable in the thermal image and less discernable in the visible light image;
based on the detected image discrepancies, determine a presence of a tran

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Price Raymond Kirk
• Pekelny Yuri
• Bleyer Michael

Classifications

• G06T7/593 Primary

  from stereo images · CPC title

• G06V20/00

  Scenes; Scene-specific elements (control of digital cameras H04N23/60) · CPC title

• G06V10/143

  Sensing or illuminating at different wavelengths · CPC title

• G06T7/55 Primary

  from multiple images · CPC title

• H04N23/45

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title