Imager for Detecting Visual Light and Projected Patterns

Patent metadata

Field	Value
Publication number	US-2016288330-A1
Application number	US-201514672369-A
Country	US
Kind code	A1
Filing date	Mar 30, 2015
Priority date	Mar 30, 2015
Publication date	Oct 6, 2016
Grant date	—

Abstract

Methods and systems for depth sensing are provided. A system includes a first and second optical sensor each including a first plurality of photodetectors configured to capture visible light interspersed with a second plurality of photodetectors configured to capture infrared light within a particular infrared band. The system also includes a computing device configured to (i) identify first corresponding features of the environment between a first visible light image captured by the first optical sensor and a second visible light image captured by the second optical sensor; (ii) identify second corresponding features of the environment between a first infrared light image captured by the first optical sensor and a second infrared light image captured by the second optical sensor; and (iii) determine a depth estimate for at least one surface in the environment based on the first corresponding features and the second corresponding features.

First claim

What is claimed is:

1 . A system comprising:
a first optical sensor and a second optical sensor, wherein each optical sensor comprises a first plurality of photodetectors configured to capture visible light interspersed with a second plurality of photodetectors configured to capture infrared light within a particular infrared band; a light source configured to project infrared light of a wavelength within the particular infrared band onto an environment; and a computing device configured to:
identify first corresponding features of the environment between a first visible light image captured by the first optical sensor and a second visible light image captured by the second optical sensor;
identify second corresponding features of the environment between a first infrared light image captured by the first optical sensor and a second infrared light image captured by the second optical sensor; and
determine a depth estimate for at least one surface in the environment based on the first corresponding features and the second corresponding features.

2 . The system of claim 1 , wherein determining the depth estimate for at least one surface in the environment comprises:
determining a first depth estimate of at least one surface in the environment based on the first corresponding features; determining a second depth estimate of at least one surface in the environment based on the second corresponding features; and determining, as the depth estimate, a combined depth estimate of at least one surface based on the first depth estimate and the second depth estimate.

3 . The system of claim 2 , wherein determining the combined depth estimate comprises averaging the first depth estimate and the second depth estimate to determine the combined depth estimate.

4 . The system of claim 1 , wherein the light source is a texture projector configured to project a predetermined texture pattern onto the environment.

5 . The system of claim 4 , wherein the computing system is further configured to cause the texture projector to project the predetermined texture pattern onto the environment before the first infrared light image and the second infrared light image are captured.

6 . The system of claim 1 , wherein the first plurality of photodetectors and the second plurality of photodetectors of each optical sensor are arranged in a checkerboard pattern.

7 . The system of claim 1 , further comprising a robotic manipulator, wherein the at least two optical sensors are coupled to the robotic manipulator.

8 . The system of claim 7 , wherein the light source is coupled to the robotic manipulator.

9 . The system of claim 1 , wherein the computing device is further configured to:
determine a length of time since the combined depth estimate was determined; and based on the length of time exceeding a threshold length of time, repeating the steps of (i) identifying the first corresponding features, (ii) identifying the second corresponding features, and (iii) determining the depth estimate.

10 . An optical sensor comprising:
a planar array of photodetectors comprising a first plurality of photodetectors and a second plurality of photodetectors, wherein each photodetector is configured to generate a charge based on an intensity of light incident on the photodetector; a light filter array coupled to the planar array of photodetectors comprising a first plurality of filters configured to pass visible light onto the first plurality of photodetectors and a second plurality of filters configured to pass infrared light within a particular infrared band onto the second plurality of photodetectors, wherein the first plurality of filters is interspersed with the second plurality of filters; and a control circuit configured to:
generate a visible light image based on charges from the first plurality of photodetectors; and
generate an infrared light image based on charges from the second plurality of photodetectors.

11 . The optical sensor of claim 10 , wherein the first plurality of filters and the second plurality of filters are arranged in a checkerboard pattern.

12 . The optical sensor of claim 10 , wherein the first plurality of filters comprises a first subset of filters configured to pass red visible light, a second subset of filters configured to pass green visible light, and a third subset of filters configured to pass blue visible light.

13 . The optical sensor of claim 12 , wherein the first plurality of filters and the second plurality of filters are arranged to form a square tiling pattern comprising a plurality of filter mosaics, wherein each filter mosaic comprises (1) a filter of the first subset that passes red visible light, (2) a filter of the second subset that passes green visible light, (3) a filter of the third subset that passes red visible light, and (4) a filter of the second plurality of filters that passes infrared light within the particular infrared band.

14 . The optical sensor of claim 10 , wherein the control circuit is further configure to:
cause, over a first duration of time, the first plurality of photodetectors to capture visible light to generate the charges that the visible light image is based on; and cause, over a second duration of time, the second plurality of photodetectors to capture infrared light to generate the charges that the infrared light image is based on, wherein the second duration of time is greater than the first duration of time.

15 . The optical sensor of claim 10 , wherein the control circuit is further configured to:
amplify the charges generated by the first plurality of photodetectors by a first gain value; and amplify the charges generated by the second plurality of photodetectors by a second gain value, wherein the second gain value is greater than the first gain value.

16 . The optical sensor of claim 10 , wherein a number of photodetectors in the first plurality of photodetectors is greater than a number of photodetectors in the second plurality of photodetectors.

17 . A method comprising:
receiving, from a first plurality of photodetectors of a first optical sensor, a first visible light image of an environment as perceived from a first viewpoint; receiving, from a first plurality of photodetectors of a second optical sensor, a second visible light image of the environment as perceived from a second viewpoint; receiving, from a second plurality of photodetectors of the first optical sensor, a first infrared light image of the environment as perceived from the first viewpoint, wherein the second plurality of photodetectors of the first optical sensor is configured to capture infrared light within a particular infrared band; receiving, from a second plurality of photodetectors of the second optical sensor, a second infrared light image of the environment as perceived from the second viewpoint, wherein the second plurality of photodetectors of the second optical sensor is configured to capture infrared light within the particular infrared band; identifying, by the computing device, first corresponding features of the environment between the first visible light image and the second visible light image; identifying, by the computing device, second corresponding features of the environment between the first infrared light image and the second infrared light image; and determining, by the computing device, a depth estimate for at least one surface in the environment based on the first corresponding features and the second corresponding features.

18 . The method of claim 1

Assignees

• Google Inc

Inventors

• Konolige Kurt

Classifications

• G01B11/24

  for measuring contours or curvatures · CPC title

• G05B2219/40425

  Sensing, vision based motion planning · CPC title

• G06T2207/30164

  Workpiece; Machine component · CPC title

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title

• G01B11/026

  by measuring distance between sensor and object (G01B11/0608 takes precedence) · CPC title