Monitoring doe performance using software scene evaluation

Patent metadata

Field	Value
Publication number	US-2016371845-A1
Application number	US-201514742816-A
Country	US
Kind code	A1
Filing date	Jun 18, 2015
Priority date	Jun 18, 2015
Publication date	Dec 22, 2016
Grant date	—

Abstract

A method for projection includes projecting a pattern of structured light with a given average intensity onto a scene. A sequence of images is captured of the scene while projecting the pattern. At least one captured image in the sequence is processed in order to extract a depth map of the scene. A condition is identified in the depth map indicative of a fault in projection of the pattern. Responsively to the identified condition, the average intensity of the projection of the pattern is reduced.

First claim

1 . A method for projection, comprising:
projecting a pattern of structured light with a given average intensity onto a scene; capturing a sequence of images of the scene while projecting the pattern; processing at least one captured image in the sequence in order to extract a depth map of the scene, the depth map comprising an array of pixels with respective depth values; identifying in the depth map a condition indicative of a fault in projection of the pattern; and responsively to the identified condition, reducing the average intensity of the projection of the pattern.

2 . The method according to claim 1 , wherein projecting the pattern comprises generating the pattern by directing a laser beam to impinge on a diffractive optical element (DOE), and wherein identifying the condition comprises detecting a failure of the DOE.

3 . The method according to claim 1 , wherein identifying the condition comprises detecting that a number of the pixels having valid depth values is below a predefined limit.

4 . The method according to claim 1 , wherein identifying the condition comprises detecting that a distribution of the depth values does not satisfy a predefined validity criterion.

5 . The method according to claim 1 , wherein processing the at least one captured image comprises computing confidence scores with respect to the depth values, and wherein identifying the condition comprises detecting that a distribution of the confidence scores does not satisfy a predefined validity criterion.

6 . The method according to claim 1 , wherein the at least one image captured while projecting the pattern with the given average intensity is a first image, and the depth map extracted therefrom is a first depth map, and
wherein the method comprises, after identifying the condition indicative of the fault:
capturing at least a second image while projecting the pattern at the reduced average intensity;
processing at least the second image in order to extract a second depth map;
making a determination, based on the second depth map, that the condition indicative of the fault has been resolved; and
responsively to the determination, increasing the average intensity of the projection of the pattern.

7 . The method according to claim 6 , wherein reducing the average intensity comprises reducing a duty cycle of the projection of the pattern, and wherein increasing the intensity comprises increasing the duty cycle.

8 . The method according to claim 6 , wherein processing at least the second image comprises extracting multiple depth maps from successive images captured while projecting the pattern at the reduced average intensity, and wherein making the determination comprises deciding that the condition indicative of the fault has been resolved only after finding the condition to have been resolved in a predefined number of the extracted depth maps.

9 . Projection apparatus, comprising:
a projection assembly, which is configured to project a pattern of structured light with a given average intensity onto a scene; an image capture assembly, which is configured to capture a sequence of images of the scene while the pattern is projected onto the scene; and a processor, which is configured to process at least one captured image in the sequence in order to extract a depth map of the scene, the depth map comprising an array of pixels with respective depth values, to identify in the depth map a condition indicative of a fault in projection of the pattern, and responsively to the identified condition, to cause the projection assembly to reduce the average intensity of the projection of the pattern.

10 . The apparatus according to claim 9 , wherein the projection assembly comprises a diffractive optical element (DOE) and a laser, which is configured to direct a laser beam to impinge on the DOE, and wherein the identified condition is indicative of a failure of the DOE.

11 . The apparatus according to claim 9 , wherein the processor is configured to identify the condition by detecting that a number of the pixels having valid depth values is below a predefined limit.

12 . The apparatus according to claim 9 , wherein the processor is configured to identify the condition by detecting that a distribution of the depth values does not satisfy a predefined validity criterion.

13 . The apparatus according to claim 9 , wherein the processor is configured to compute confidence scores with respect to the depth values, and to identify the condition by detecting that a distribution of the confidence scores does not satisfy a predefined validity criterion.

14 . The apparatus according to claim 9 , wherein the at least one image captured while projecting the pattern with the given average intensity is a first image, and the depth map extracted therefrom is a first depth map, and
wherein the processor is configured, after identifying the condition indicative of the fault, to process at least a second image captured by the image capture assembly while the projection assembly projects the pattern at the reduced average intensity, to process at least the second image in order to extract a second depth map, to make a determination, based on the second depth map, that the condition indicative of the fault has been resolved, and responsively to the determination, to cause the projection assembly to increase the average intensity of the projection of the pattern.

15 . The apparatus according to claim 14 , wherein the average intensity is reduced by reducing a duty cycle of the projection of the pattern, and the intensity is increased by increasing the duty cycle.

16 . The apparatus according to claim 14 , wherein the processor is configured to extract multiple depth maps from successive images captured while projecting the pattern at the reduced average intensity, and to decide that the condition indicative of the fault has been resolved only after finding the condition to have been resolved in a predefined number of the extracted depth maps.

17 . A computer software product, comprising a non-transitory, computer-readable medium in which program instructions are stored, which instructions, when read by a programmable processor, cause the processor to receive a sequence of images of a scene while a pattern of structured light is projected onto the scene with a given average intensity, to process at least one captured image in the sequence in order to extract a depth map of the scene, the depth map comprising an array of pixels with respective depth values, to identify in the depth map a condition indicative of a fault in projection of the pattern, and responsively to the identified condition, to reduce the average intensity of the projection of the pattern.

18 . The product according to claim 17 , wherein the pattern is generated by directing a laser beam to impinge on a diffractive optical element (DOE), and wherein the identified condition is indicative of a failure of the DOE.

19 . The product according to claim 17 , wherein the instructions cause the processor to identify the condition by detecting that a number of the pixels having valid depth values is below a predefined limit.

20 . The product according to claim 17 , wherein the instructions cause the processor to identify the condition by detecting that a distribution of the depth values does not satisfy a predefined validity criterion.

Assignees

• Apple Inc

Inventors

• Yasovsky Alon
• Deitch Ronen
• Zmora Yohai

Classifications

• G02B27/4205 Primary

  having a diffractive optical element [DOE] contributing to image formation, e.g. whereby modulation transfer function MTF or optical aberrations are relevant · CPC title

• G06K9/52

  Physics · mapped topic

• G06K9/4661

  Physics · mapped topic

• G06T7/0065 Primary

  Physics · mapped topic

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title