Depth dependent pixel filtering

Patent metadata

Field	Value
Publication number	US-11954877-B2
Application number	US-202017115320-A
Country	US
Kind code	B2
Filing date	Dec 8, 2020
Priority date	Dec 8, 2020
Publication date	Apr 9, 2024
Grant date	Apr 9, 2024

Abstract

Sensors, including time-of-flight sensors, may be used to detect objects in an environment. In an example, a vehicle may include a time-of-flight sensor that images objects around the vehicle, e.g., so the vehicle can navigate relative to the objects. Sensor data generated by the time-of-flight sensor can include returns associated with highly reflective objects that cause glare. In some examples, a depth of a sensed surface is determined from the sensor data and additional pixels at the same depth are identified. The subset of pixels at the depth are filtered by comparing a measured intensity value to a threshold intensity value for the depth. Other threshold intensity values can be applied to subsets of pixels at different depths.

First claim

What is claimed is:

1. A vehicle comprising:
a time-of-flight sensor configured to generate data based on light received at a receiver of the time-of-flight sensor;
one or more processors; and
memory storing processor-executable instructions that, when executed by the one or more processors, configure the vehicle to perform operations comprising:
receiving first sensor data from the time-of-flight sensor, the first sensor data comprising first depth information and first intensity information for a plurality of pixels generated with the time-of-flight sensor in a first configuration having a first modulation frequency;
receiving second sensor data from the time-of-flight sensor, the second sensor data comprising second depth information and second intensity information for the plurality of pixels generated with the time-of-flight sensor in a second configuration having a second modulation frequency higher than the first modulation frequency;
determining, based at least in part on the first depth information and the second depth information, a pixel of the plurality of pixels having a first depth that is greater than a nominal maximum depth of the time-of-flight sensor in the first configuration;
determining, based on the first sensor data and the second sensor data, additional pixels of the plurality of pixels having the first depth, the pixel and the additional pixels comprising a first subset of the plurality of pixels;
comparing at least one of the first intensity information or the second intensity information for the first subset of the plurality of pixels to a first threshold intensity and at least one of the first intensity information or the second intensity information for a second subset of the plurality of pixels to a second threshold intensity, the second subset of the plurality of pixels having a second depth different from the first depth; and
generating filtered data based at least in part on the comparing, the filtered data comprising at least one of one or more of the pixels of the first subset of the plurality of pixels having an intensity greater than the first threshold intensity or one or more of the pixels of the second subset of the plurality of pixels having an intensity greater than the second threshold intensity.

2. The vehicle of claim 1 , wherein the first threshold intensity is higher than the second threshold intensity.

3. The vehicle of claim 1 , wherein at least one of the first threshold intensity is inversely proportional to a power of the first depth or the second threshold intensity is inversely proportional to a power of the second depth.

4. The vehicle of claim 1 , wherein the determining the pixel having the first depth comprises:
determining first candidate depths for the pixel based at least in part on the first depth information;
determining second candidate depths for the pixel based at least in part the second depth information;
determining the first depth based at least in part on a first candidate depth of the first candidate depths and a second candidate depth of the second candidate depths, the first candidate depth and the second candidate depth being the closest of the first candidate depths and the second candidate depths; and
determining that an intensity of the first depth exceeds a third threshold intensity.

5. The vehicle of claim 1 , the operations further comprising:
determining, based at least in part on the filtered data, information about an object in the environment;
determining a trajectory relative to the object; and
controlling the vehicle to execute the trajectory.

6. A method comprising:
receiving sensor data comprising intensity information and depth information for a plurality of pixels;
determining, based at least in part on the depth information, a first subset of the plurality of pixels having a first depth;
determining, based at least in part on the depth information, a second subset of the plurality of pixels having a second depth;
determining, for the first subset of the plurality of pixels, a first threshold intensity;
determining, for the second subset of the plurality of pixels, a second threshold intensity; and
generating filtered data associated with the sensor data, the filtered data including one or more of (i) a first pixel of the first subset of the plurality of pixels having, based at least in part on the intensity information, a first measured intensity equal to or greater than the first threshold intensity or (ii) a second pixel of the second subset of the plurality of pixels having a second measured intensity equal to or greater than the second threshold intensity.

7. The method of claim 6 , wherein the generating the filtered data comprises:
excluding at least one of first pixels of the first subset of the plurality of pixels having, based on the intensity information, measured intensities below the first threshold intensity; or second pixels of the second subset of the plurality of pixels having, based on the intensity information, measured intensities below the second threshold intensity.

8. The method of claim 6 , wherein the receiving the sensor data comprises:
receiving first sensor data from the sensor, the first sensor data comprising first depth information and first intensity information for a plurality of pixels generated in a first sensor configuration, the first sensor configuration having a first modulation frequency; and
receiving second sensor data from the sensor, the second sensor data comprising second depth information and second intensity information for the plurality of pixels generated with the sensor in a second configuration having a second modulation frequency higher than the first modulation frequency.

9. The method of claim 8 , wherein the determining the first subset of the plurality of pixels having the first depth comprises:
determining, based on the first depth information and the second depth information, a disambiguated depth of the surface;
determining, based at least in part on the disambiguated depth being greater than a nominal maximum depth of the sensor in the first configuration, the first depth as the disambiguated depth; and
identifying, from the depth information, the first subset of the plurality of pixels as pixels having depths corresponding to the first depth.

10. The method of claim 9 , wherein the identifying the first subset of the plurality of pixels further comprises identifying the first subset of the plurality of pixels as pixels within a two-dimensional distance of a pixel corresponding to the surface.

11. The method of claim 8 , wherein the determining the second subset of the plurality of pixels having the second depth comprises:
determining, based on the first depth information, an estimated depth of a surface;
determining, based on the second depth information, a plurality of candidate depths of the surface;
determining the second depth as the candidate depth of the plurality of candidate depths of the surface closest to the estimated depth of the surface; and
identifying, from the depth information, the second subset of the plurality of pixels as pixels having the second depth.

12. The method of claim 11 , wherein the estimated depth of the surface is a nominal depth measured by the sensor in the first configuration.

13. The method of claim 6 , wherein:
the sensor data comprises a plurality of intensity pixels having measured intensities and a plurality of depth pixels having measured depths, individual of the plurality of intensity pixels corresponding to one of the plurality of depth pixels; and
the filtered data comprises a subset of the plurality of intensity pixels

Assignees

• Zoox Inc

Inventors

• Subasingha Subasingha Shaminda
• Chen Yongzhe
• Ferdowsi Mehran
• Holladay Samuel
• Karadeniz Turhan
• Moor Robert Nicholas
• Warga Joseph Patrick
• Waschura Harrison Thomas
• Wilkinson Silas Kogure

Classifications

• G06T7/521 Primary

  from laser ranging, e.g. using interferometry; from the projection of structured light · CPC title

• B60W60/00272 Primary

  relying on extrapolation of current movement · CPC title

• G01B11/22

  for measuring depth · CPC title

• G01S17/894

  Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar · CPC title

• G01S17/931

  of land vehicles · CPC title