Imaging system using structured light for depth recovery

Patent metadata

Field	Value
Publication number	US-10410366-B2
Application number	US-201615560782-A
Country	US
Kind code	B2
Filing date	Mar 23, 2016
Priority date	Mar 31, 2015
Publication date	Sep 10, 2019
Grant date	Sep 10, 2019

Abstract

An imaging system includes an optical unit that captures, from a scene, sets of first images as well as sets of second images. The scene is illuminated with non-structured light when the sets of first images are captured and illuminated with structured light when the sets of second images are captured. A pattern processing unit generates, from the second images, a recovered shadow pattern from a shadow pattern projected onto the scene during illumination with structured light, wherein at least one motion-compensated reference image is used. A depth processing unit obtains depth information on the basis of the recovered shadow pattern.

First claim

The invention claimed is:

1. An imaging system comprising:
an imaging sensor configured to capture, from a scene, sets of first images and sets of second images, wherein the scene is illuminated with non-structured light when the sets of first images are captured and wherein the scene is illuminated with structured light when the sets of second images are captured;
pattern processing circuitry configured to generate, from the second images, a recovered shadow pattern from a shadow pattern projected onto the scene during illumination with structured light, wherein at least one motion-compensated reference image is used, respectively; and
depth processing circuitry configured to obtain depth information on the basis of the recovered shadow pattern,
wherein:
the pattern processing circuitry comprises motion compensation circuitry configured to obtain the at least one motion-compensated reference image for a set of second images on the basis of successive sets of first images or on the basis of the set of second images and at least one previously or subsequently obtained set of first images,
the motion compensation circuitry comprises (i) phase correlation circuitry configured to estimate a global motion vector and (ii) recursive block matching circuitry configured to estimate local motion vectors on the basis of the global motion vector and the first images, and
the motion compensation circuitry comprises optical flow circuitry configured to estimate displacement vectors on the basis of auxiliary images partially motion-compensated by the local motion vectors.

2. The imaging system according to claim 1 , wherein
the motion compensation circuitry is configured to obtain the reference images for a set of second images on the basis of the set of second images and a previous or subsequent set of first images.

3. The imaging system according to claim 2 , wherein
the motion compensation circuitry is configured to use only not-shadowed sections of the second images outside of the shadow pattern.

4. The imaging system according to claim 3 , wherein
the shadow pattern includes code lines including fields of different grey levels and non-shadowed white lines separating neighbouring code lines.

5. The imaging system according to claim 1 , wherein
the motion compensation circuitry is configured to obtain the reference images for a set of second images on the basis of sets of first images captured before and after the set of second images.

6. The imaging system according to claim 1 , wherein
the imaging sensor further comprises an imaging lens with longitudinal chromatic aberration, wherein the imaging lens images the scene onto the imaging sensor.

7. The imaging system according to claim 6 , further comprising
sharpness processing circuitry is configured to generate an output image based on the depth information and the first images.

8. The imaging system according to claim 7 , wherein
the sharpness processing circuitry is configured to exchange sharpness information among the first images of sets of first images.

9. An imaging system comprising:
an imaging sensor configured to capture, from a scene sets of first images and sets of second images, wherein the scene is illuminated with non-structured light when the sets of first images are captured and wherein the scene is illuminated with structured light when the sets of second images are captured;
pattern processing circuitry configured to generate% from the second images, a recovered shadow pattern from a shadow pattern projected onto the scene during illumination with structured light, wherein at least one motion-compensated reference image is used, respectively; and
depth processing circuitry configured to obtain depth information on the basis of the recovered shadow pattern,
wherein:
the imaging sensor further comprises an imaging lens with longitudinal chromatic aberration, wherein the imaging lens images the scene onto the imaging sensor, and
the imaging sensor is configured to capture the sets of first images in different wavelength ranges and the sets of second images in the different wavelength ranges, and the pattern processing circuitry comprises a motion compensation circuitry configured to estimate the motion-compensated reference image for a set of second images on the basis of successive sets of first images and wherein the motion compensation circuitry comprises (i) a phase correlation circuitry configured to estimate global motion vectors for each of the wavelength ranges and (ii) a recursive block matching circuitry configured to estimate local motion vectors on the basis of a combined global motion vector obtained from the global motion vectors for each of the wavelength ranges.

10. The imaging system according to claim 1 , wherein
the depth processing circuitry is configured to obtain the depth information from the recovered shadow pattern by using optical triangulation.

11. The imaging system according to claim 1 , wherein
the imaging sensor is configured to capture the sets of first images in different wavelength ranges and the sets of second images in the different wavelength ranges.

12. The imaging system according to claim 11 , wherein
one motion-compensated reference image derived from images assigned to the same wavelength range is used for recovering the shadow pattern in a set of second images.

13. The imaging system according to claim 11 , wherein
one motion-compensated reference image derived from images assigned to different wavelength range is used for recovering the shadow pattern in a set of second images.

14. The imaging system according to claim 11 , wherein
a plurality of motion-compensated reference images, each of them derived from images assigned to a same wavelength range, is used for recovering the shadow pattern in a set of second images.

15. An imaging method comprising:
capturing, from a scene, sets of first images when the scene is illuminated with non-structured light and sets of second images when the scene is illuminated with structured light,
generating, from the second images, a recovered shadow pattern from a shadow pattern projected onto the scene during illumination with structured light, wherein at least one motion-compensated reference image is used, respectively; and
generating depth information on the basis of the recovered shadow pattern,
wherein:
the generating the recovered shadow pattern includes obtaining the at least one motion-compensated reference image for a set of second images on the basis of successive sets of first images or on the basis of the set of second images and at least one previously or subsequently obtained set of first images,
the obtaining comprises (i) phase correlation circuitry configured to estimate a global motion vector and (ii) recursive block matching circuitry configured to estimate local motion vectors on the basis of the global motion vector and the first images, and
the obtaining comprises optical flow circuitry configured to estimate displacement vectors on the basis of auxiliary images partially motion-compensated by the local motion vectors.

16. The imaging method according to claim 15 , wherein
the at least one motion-compensated reference image for a set of second images is obtained on the basis of successive sets of first images or on the basis of the set of second images and at least one previously or subsequently obtained set of first images.

Assignees

• Sony Corp

Inventors

• Michielin Francesco
• Springer Paul
• Emmerich Thimo
• Facius Zoltan
• Sartor Piergiorgio

Classifications

• H04N23/56

  provided with illuminating means · CPC title

• H04N23/555

  for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes · CPC title

• G06T2207/30021

  Catheter; Guide wire · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title