Determining a virtual representation of an environment by projecting texture patterns

Patent metadata

Field	Value
Publication number	US-9862093-B2
Application number	US-201514961607-A
Country	US
Kind code	B2
Filing date	Dec 7, 2015
Priority date	Mar 15, 2013
Publication date	Jan 9, 2018
Grant date	Jan 9, 2018

Abstract

Example methods and systems for determining 3D scene geometry by projecting patterns of light onto a scene are provided. In an example method, a first projector may project a first random texture pattern having a first wavelength and a second projector may project a second random texture pattern having a second wavelength. A computing device may receive sensor data that is indicative of an environment as perceived from a first viewpoint of a first optical sensor and a second viewpoint of a second optical sensor. Based on the received sensor data, the computing device may determine corresponding features between sensor data associated with the first viewpoint and sensor data associated with the second viewpoint. And based on the determined corresponding features, the computing device may determine an output including a virtual representation of the environment that includes depth measurements indicative of distances to at least one object.

First claim

What is claimed is:

1. A method comprising:
projecting a plurality of different patterns of light during a given time period using a plurality of projectors, wherein projecting the plurality of different patterns of light using the plurality of projectors comprises projecting, using a first projector of the plurality of projectors, multiple texture patterns having a first wavelength and projecting, using a second projector of the plurality of projectors, at least one texture pattern having a second wavelength;
receiving sensor data from a plurality of optical sensors, wherein the plurality of optical sensors are configured to distinguish between the plurality of different patterns of light, and wherein the sensor data comprises a first sequence of infrared images corresponding to the given time period and captured from a first viewpoint of a first optical sensor of the plurality of optical sensors and a second sequence of infrared images corresponding to the given time period and captured from a second viewpoint of a second optical sensor of the plurality of optical sensors;
determining a first spatio-temporal image based on a combination of the first sequence of infrared images;
determining a second spatio-temporal image based on a combination of the second sequence of infrared images;
determining, by a computing device, corresponding features between the first spatio-temporal image and the second spatio-temporal image; and
based on the determined corresponding features, determining, by the computing device, an output including a virtual representation of an environment, wherein the output comprises a depth measurement indicative of a distance from the first optical sensor to at least one object in the environment.

2. The method of claim 1 , wherein the determined corresponding features comprise corresponding segments of pixels.

3. The method of claim 1 , wherein projecting the plurality of different patterns of light using the plurality of projectors comprises projecting, using the first projector, a first texture pattern having the first wavelength while projecting, using the second projector, a second texture pattern having the second wavelength such that at least a portion of the first texture pattern overlaps at least a portion of the second texture pattern on a surface in the environment during a portion of the given time period.

4. The method of claim 3 , wherein projecting the plurality of different patterns of light using the plurality of projectors further comprises projecting, using the first projector, a third texture pattern having the first wavelength while projecting, using the second projector, the second texture pattern having the second wavelength such that at least a portion of the third texture pattern overlaps at least a portion of the second texture pattern on the surface in the environment during another portion of the given time period.

5. The method of claim 3 , wherein projecting the plurality of different patterns of light using the plurality of projectors further comprises projecting, using the first projector, a third texture pattern having the first wavelength while projecting, using the second projector, a fourth texture pattern having the second wavelength such that at least a portion of the third texture pattern overlaps at least a portion of the fourth texture pattern on the surface in the environment during another portion of the given time period.

6. The method of claim 1 , wherein the first optical sensor and the second optical sensor are configured to distinguish between the first wavelength and the second wavelength using pixel filters.

7. The method of claim 1 , wherein the first optical sensor and the second optical sensor together form a first stereo camera, wherein the plurality of optical sensors further comprise a second stereo camera, and wherein the method further comprises:
determining, by the computing device, a first depth image using sensor data received from the first stereo camera and a second depth image using sensor data received from the second stereo camera; and
based on the first depth image, the second depth image, and a distance between a position of the first stereo camera and a position of the second stereo camera, modifying the depth measurement indicative of the distance to the at least one object in the environment.

8. The method of claim 1 , wherein receiving sensor data from the plurality of optical sensors comprises receiving first sensor data corresponding to a first position of the first optical sensor and the second optical sensor and receiving second sensor data corresponding to a second position of the first optical sensor and the second optical sensor, and wherein the method further comprises:
determining, by the computing device, a first depth image using the first sensor data and a second depth image using the second sensor data; and
based on the first depth image, the second depth image, and a distance between the first position and the second position, modifying the depth measurement indicative of the distance to the at least one object in the environment.

9. The method of claim 1 , wherein the plurality of projectors are located at different respective projector positions, and wherein the method further comprises determining a location of an occluding edge of the at least one object based on image segmentation of respective shadows induced by the projection of light from the different projector positions.

10. A system comprising:
a first projector;
a second projector;
a first optical sensor;
a second optical sensor; and
a computing device configured to perform functions comprising:
causing the first projector to project a first texture pattern having a first wavelength while the second projector projects a second texture pattern having a second wavelength such that at least a portion of the first texture pattern overlaps at least a portion of the second texture pattern on a surface in an environment,
receiving sensor data from the first optical sensor and the second optical sensor, wherein the sensor data comprises at least one first image of an overlapping of the first texture pattern and the second texture pattern as perceived from a first viewpoint of the first optical sensor and at least one second image of the overlapping of the first texture pattern and the second texture pattern as perceived from a second viewpoint of the second optical sensor,
determining, based on the received sensor data, corresponding features between the at least one first image and the at least one second image, and
based on the determined corresponding features, determining an output including a virtual representation of an environment of the computing device, wherein the output comprises a depth measurement indicative of a distance from the first optical sensor to at least one object in the environment.

11. The system of claim 10 , wherein the first optical sensor and the second optical sensor are each configured to distinguish between the first texture pattern and the second texture pattern.

12. The system of claim 10 , further comprising a robotic manipulator, wherein at least one of the first projector, second projector, first optical sensor, and second optical sensor is coupled to the robotic manipulator.

13. The system of claim 10 :
wherein the sensor data associated with the first viewpoint comprises a first sequence of infrared images corresponding to a given time period and the sensor data associated with the second viewpoint comprises a second sequence of infrared images corresponding to the given time period, and
wherein determining the corresponding features comprises:
de

Assignees

• Ind Perception Inc
• X Dev Llc

Inventors

• Bradski Gary
• Konolige Kurt
• Rublee Ethan

Classifications

• B25J9/1671 Primary

  characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems · CPC title

• B25J9/163 Primary

  learning, adaptive, model based, rule based expert control · CPC title

• G06T7/55

  from multiple images · CPC title

• Y10S901/09

  Closed loop, sensor feedback controls arm movement · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title