Reduced power operation of time-of-flight camera

Patent metadata

Field	Value
Publication number	US-10607352-B2
Application number	US-201815983054-A
Country	US
Kind code	B2
Filing date	May 17, 2018
Priority date	May 17, 2018
Publication date	Mar 31, 2020
Grant date	Mar 31, 2020

Abstract

A time-of-flight (ToF) camera is configured to operate in a manner that reduces power consumption of the ToF camera. For a key frame, a key-frame depth image is generated based on a plurality of sets of key-frame IR images. Each set of key-frame IR images is acquired using a different modulation frequency of active IR light. For a P-frame after the key frame, a P-frame depth image is generated based on a set of P-frame IR images acquired using a single modulation frequency of active IR light.

First claim

The invention claimed is:

1. A time-of-flight (ToF) camera comprising:
a ToF illuminator configured to emit active IR light;
a sensor array including a plurality of sensors each configured to measure active IR light; and
a ToF controller machine configured to:
for a key frame:
repeatedly activate the ToF illuminator to illuminate a scene with active IR light, wherein the ToF illuminator modulates the active IR light in a plurality of different modulation frequencies,
for each of the plurality of different modulation frequencies, repeatedly address the sensor array to acquire a set of key-frame IR images that each represent measurements of the active IR light reflected from the scene back to the sensor array,
generate a key-frame depth image that includes, for each sensor of the sensor array, a depth value based on a plurality of sets of key-frame IR images, each set of key-frame IR images acquired using a different modulation frequency of active IR light, and
identify one or more features of the scene based on the plurality of sets of key-frame IR images,

for a P-frame occurring after the key frame:
activate the ToF illuminator to illuminate the scene with active IR light modulated in a single modulation frequency,
address the sensor array to acquire a set of P-frame IR images that represents measurement of the active IR light reflected from the scene back to the sensor array,
identify the one or more features of the scene based on the set of P-frame IR images,
determine a positional translation of the one or more features from the key-frame to the P-frame, and
generate a P-frame depth image that includes, for each sensor of the sensor array, a depth value based at least on the set of P-frame IR images acquired using the single modulation frequency of active IR light and the positional translation of the one or more features of the scene; and

an output machine operatively connected to the sensor array and configured to output the key-frame depth image and the P-frame depth image.

2. The ToF camera of claim 1 , wherein the ToF controller machine is configured to generate a plurality of P-frame depth images for successive P-frames between key frames.

3. The ToF camera of claim 1 , wherein the ToF controller machine is configured to dynamically adjust a number of successive P-frames between key frames based on an amount of positional translation of the one or more features.

4. The ToF camera of claim 1 , wherein the single modulation frequency is a highest modulation frequency of the plurality of different modulation frequencies.

5. The ToF camera of claim 1 , wherein the ToF controller machine is configured to generate the P-frame depth image also based on the sets key-frame IR images for the modulation frequencies other than the single modulation frequency of the P-frame.

6. The ToF camera of claim 5 , wherein the ToF controller machine is configured to crop the key-frame IR images for the modulation frequencies other than the single modulation frequency based on the positional translation of the one or more features, and wherein the ToF controller machine is configured to generate the P-frame depth image also based on the cropped key-frame IR images.

7. The ToF camera of claim 1 , wherein the ToF controller machine is configured to perform an intensity de-noising operation on the plurality of sets of key-frame IR images and the set of P-frame IR images.

8. The ToF camera of claim 7 , wherein the intensity de-noising operation includes applying a low-pass filter to the plurality of sets of key-frame IR images and the sets of P-frame IR images.

9. The ToF camera of claim 1 , wherein the ToF controller machine is configured to perform a spatial frequency reduction operation on the plurality of sets of key-frame IR images and the set of P-frame IR images.

10. The ToF camera of claim 9 , wherein the ToF controller machine is configured to perform a dynamic range reduction operation on the plurality of sets of key-frame IR images and the set of P-frame IR images.

11. A ToF camera comprising:
a ToF illuminator configured to emit active IR light;
a sensor array including a plurality of sensors; and
a ToF controller machine configured to:
for a key frame, generate a key-frame depth image that includes, for each sensor of the plurality of sensors of the sensor array, a depth value based on a plurality of sets of key-frame IR images, each set of key-frame IR images acquired using a different modulation frequency of active IR light,
for a P-frame occurring after the key frame, generate a P-frame depth image that includes, for each sensor of the plurality of sensors of the sensor array, a depth value based on a single set of P-frame IR images acquired using a single modulation frequency of active IR light; and

an output machine operatively connected to the sensor array and configured to output the key-frame depth image and the P-frame depth image.

12. The ToF camera of claim 11 , wherein the ToF controller machine is configured to:
for the key frame:
repeatedly activate the ToF illuminator to illuminate a scene with active IR light, wherein the ToF illuminator modulates the active IR light in a plurality of different modulation frequencies,
for each of the plurality of different modulation frequencies, repeatedly address the sensor array to acquire a set of key-frame IR images that represent measurements of the active IR light reflected from the scene back to the sensor array,
for each sensor of the plurality of sensors of the sensor array, determine a depth value based on the plurality of sets of IR images,
identify one or more features of the scene based on the plurality of sets of key-frame IR images,

for the P-frame:
activate the ToF illuminator to illuminate the scene with active IR light modulated in the single modulation frequency,
address the sensor array to acquire the set of P-frame IR images that represents measurement of the active IR light reflected from the scene back to the sensor array,
identify the one or more features of the scene based on the set of P-frame IR images,
determine a positional translation of the one or more features from the key-frame to the P-frame, and

wherein each depth value of the P-frame depth image is determined based on the single set of P-frame IR images and the positional translation of the one or more features of the scene.

13. The ToF camera of claim 11 , further comprising:
one or more motion sensors configured to measure a position of the ToF camera;
wherein the ToF controller machine is configured to:
for the key frame:
repeatedly activate the ToF illuminator to illuminate a scene with active IR light, wherein the ToF illuminator modulates the active IR light in a plurality of different modulation frequencies,
for each of the plurality of different modulation frequencies, repeatedly address the sensor array to acquire a set of key-frame IR images that represent measurements of the active IR light reflected from the scene back to the sensor array,
for each sensor of the plurality of sensors of the sensor array, determine a depth value based on the plurality of sets of IR images,

for the P-frame:
activate the ToF illuminator to illuminate the scene with active IR light modulated in the single modulation frequency,
address the sensor array to acquire a set of IR images that represent measurements of the active IR light reflected from the scene back to the sensor array,
determine a changed perspective of the ToF camera from the key frame to the P-frame based on motion data of the one or more motion sensors, and

wherein each depth value of the P-frame depth image

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Ortiz Egea Sergio
• Lin Hung-Ming

Classifications

• G06T2207/10048

  Infrared image · CPC title

• G06T7/74

  involving reference images or patches · CPC title

• G06T5/20

  using local operators · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/10152

  Varying illumination · CPC title