Pixel classification to reduce depth-estimation error

Patent metadata

Field	Value
Publication number	US-11416998-B2
Application number	US-201916526305-A
Country	US
Kind code	B2
Filing date	Jul 30, 2019
Priority date	Jul 30, 2019
Publication date	Aug 16, 2022
Grant date	Aug 16, 2022

Abstract

A method to process a contributing digital image of a subject in an image-processing computer. The contributing digital image is received in a depth-resolving machine configured to furnish a depth image based at least in part on the contributing digital image. The contributing digital image is also received in a classification machine previously trained to classify a pixel of the contributing digital image as liable to corrupt a depth value of a corresponding pixel of the depth image. A repair value is computed for the depth value of the corresponding pixel of the depth image, which is then corrected based on the repair value and returned to the calling process.

First claim

The invention claimed is:

1. Enacted in an image-processing computer, a method to process a contributing digital image of a subject, the method comprising:
receiving the contributing digital image in a depth-resolving machine, the depth-resolving machine being configured to furnish a depth image based at least in part on the contributing digital image;
receiving the contributing digital image in a classification machine previously trained to classify a pixel of the contributing digital image as having a condition liable to corrupt a depth value of a corresponding pixel of the depth image furnished via the depth-resolving machine, the condition including one or more of subject motion, specular reflection, reflection from a retroreflective surface of the subject, and superposition of light reflected along two or more inequivalent paths, wherein the classification machine comprises an artificial neural network having an output layer comprising a classification layer;
computing, based on the condition, a repair value for the depth value of the corresponding pixel of the depth image;
correcting the depth image based on the repair value; and
returning the depth image as corrected.

2. The method of claim 1 wherein the classification machine is trained using ground-truth training data with labeled pixels that exhibit the condition.

3. The method of claim 1 wherein the contributing digital image is one of a plurality of contributing digital images of the subject received in the depth-resolving machine and in the classification machine.

4. The method of claim 1 wherein correcting the depth image includes replacing the depth value of the corresponding pixel of the depth image by the repair value.

5. The method of claim 1 wherein the contributing digital image includes a raw shutter or phase image from a time-of-flight (ToF) camera, and wherein the depth-resolving machine is configured to furnish the depth image by ToF analysis.

6. The method of claim 1 wherein the depth-resolving machine is configured to compute the depth value based on a disparity between the pixel of the contributing digital image and a corresponding pixel of another contributing digital image.

7. The method of claim 1 wherein the contributing digital image exhibits a reflection of a structured illumination of the subject, and wherein the depth-resolving machine is configured to compute the depth value based on geometric analysis of the reflection.

8. The method of claim 1 wherein the condition includes the reflection from a retroreflective surface of the subject, and wherein computing the repair value for the corresponding pixel includes one or both of bloom mitigation and resolving a superposition of light reflected along two or more inequivalent paths into two or more single-path reflections.

9. The method of claim 1 wherein the condition includes the superposition of light reflected along two or more inequivalent paths, and wherein computing the repair value for the corresponding pixel includes resolving the superposition into two or more single-path reflections.

10. An image-processing computer comprising:
a logic system; and
operatively coupled to the logic system, a computer-memory system holding instructions that cause the logic system to:
instantiate a depth-resolving machine configured to furnish a derived digital image of a subject based on a contributing digital image of the subject;
instantiate a classification machine trained to classify a pixel of the contributing digital image as having a condition liable to corrupt a corresponding pixel of the derived digital image, the condition including one or more of subject motion, specular reflection, reflection from a retroreflective surface of the subject, and superposition of light reflected along two or more inequivalent paths, wherein the classification machine comprises an artificial neural network having an output layer comprising a classification layer and is trained using ground-truth training data with labeled pixels that exhibit the condition;
receive the contributing digital image in the depth-resolving machine;
receive the contributing digital image in the classification machine;
compute, based on the condition, a repair value for the corresponding pixel of the derived digital image;
correct the derived digital image based on the repair value; and
return the derived digital image as corrected.

11. The image-processing computer of claim 10 wherein the contributing digital image is one of a plurality of contributing digital images of the subject received in the depth-resolving machine and in the classification machine.

12. The image-processing computer of claim 10 wherein the repair value includes a property computed based on another pixel of the derived digital image.

13. The image-processing computer of claim 10 wherein the classification machine is an upstream classification machine, and wherein the instructions cause the logic system to instantiate a downstream classification machine configured to classify one or more pixels of the derived digital image according to object type.

14. The image-processing computer of claim 13 wherein the derived digital image is corrected by replacing an object-type classification of the corresponding pixel of the derived digital image.

15. Enacted in an image-processing computer, a method to process a contributing digital image of a subject, the method comprising:
receiving the contributing digital image in a depth-resolving machine configured to furnish a derived digital image of the subject based on the contributing digital image;
receiving the contributing digital image in a classification machine configured to classify a first pixel of the contributing digital image as exhibiting a first condition liable to corrupt a corresponding first pixel of the derived digital image;
computing, based on the first condition and via a first repair engine configured to repair pixels classified as exhibiting the first condition, a repair value for the corresponding first pixel of the derived digital image, wherein the first condition includes a specular reflection, and wherein the repair value for the corresponding first pixel is computed based on a property of another pixel of the contributing digital image;
correcting the derived digital image based on the repair value computed for the corresponding first pixel of the derived digital image; and
returning the derived digital image as corrected.

16. The method of claim 15 wherein the classification machine is further configured to classify a second pixel of the contributing digital image as exhibiting a second condition liable to corrupt a corresponding second pixel of the derived digital image, the method further comprising:
computing, via a second repair engine configured to repair pixels classified as exhibiting the second condition, a repair value for the corresponding second pixel of the derived digital image.

17. The method of claim 16 wherein the second condition includes motion of the subject, and wherein the repair value for the corresponding second pixel is computed based on one or both of a shortened series of contributing digital images and optical-flow analysis of the motion of the subject.

18. The method of claim 16 wherein the second condition includes a superposition of light reflected along two or more inequivalent paths, and wherein computing the repair value for the corresponding second pixel includes resolving the superposition into two or more single-path reflections.

19. The method of c

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Bleyer Michael
• Pekelny Yuri
• Price Raymond Kirk

Classifications

• G06T7/50 Primary

  Depth or shape recovery · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title