Sensor validation using semantic segmentation information

Patent metadata

Field	Value
Publication number	US-2020282929-A1
Application number	US-201916297127-A
Country	US
Kind code	A1
Filing date	Mar 8, 2019
Priority date	Mar 8, 2019
Publication date	Sep 10, 2020
Grant date	—

Abstract

This disclosure is directed to validating a calibration of and/or calibrating sensors using semantic segmentation information about an environment. For example, the semantic segmentation information can identify bounds of objects, such as invariant objects, in the environment. Techniques described herein may determine sensor data associated with the invariant objects and compare that data to a feature known from the invariant object. Misalignment of sensor data with the known feature can be indicative of a calibration error. In some implementations, the calibration error can be determined as a distance between the sensor data and a line or plane representing a portion of the invariant object.

First claim

What is claimed is:

1 . An autonomous vehicle comprising:
a sensor disposed on the autonomous vehicle: one or more processors; and
non-transitory computer-readable media storing one or more instructions that, when executed, cause the one or more processors to perform acts comprising:
receiving, from the sensor, sensor data of an environment of the vehicle,
segmenting, as segmented data, the sensor data based at least in part on a representation of an invariant object in the sensor data,
determining, based at least in part on the segmented data, a subset of the sensor data associated with the invariant object, and
determining, based at least in part on the subset of the sensor data, a calibration error associated with the sensor.

2 . The autonomous vehicle of claim 1 , wherein determining the calibration error comprises:
generating, from the sensor data and according to a calibration function associated with the sensor, undistorted sensor data; and comparing the undistorted sensor data to a feature of the invariant object.

3 . The autonomous vehicle of claim 2 , wherein the comparing the undistorted sensor data to the feature of the invariant object comprises comparing the at least a portion of the subset of the sensor data to at least one of a line, a plane, or a reflectivity of the invariant object.

4 . The autonomous vehicle of claim 1 , wherein the determining the calibration error comprises determining a distance between at least a portion of the subset of the sensor data and a line or a plane associated with the invariant object.

5 . The autonomous vehicle of claim 1 , the acts further comprising:
calibrating, based at least in part on the calibration error, the sensor; generating, based at least in part on the calibrated sensor, a trajectory to control the autonomous vehicle; and controlling the autonomous vehicle based at least in part on the trajectory.

6 . A method comprising:
receiving, from a sensor, sensor data of an environment; determining, based at least in part on the sensor data, segmentation information about the environment; determining, based at least in part on the segmentation information, a subset of the sensor data corresponding to an invariant object; identifying, in the subset of the sensor data, at least one feature corresponding to a feature of the invariant object; and determining, based at least in part on the at least one feature, a calibration error of the sensor.

7 . The method of claim 6 , further comprising:
undistorting the at least one feature based at least in part on one or both of sensor intrinsic information or sensor extrinsic information.

8 . The method of claim 7 , wherein the segmentation information comprises associations between portions of the data and a plurality of objects in the environment depicted in the portions, the method further comprising:
determining, the invariant object from the plurality of objects; and determining, based at least in part on the one or more associations, the portion of the sensor data corresponding to the invariant object.

9 . The method of claim 6 , wherein the invariant object comprises an object having a known attribute, the attribute comprising at least one of linearity, planarity, reflectivity, or an orientation.

10 . The method of claim 9 , wherein the invariant object is at least one of a topographical feature, a fixture, a building, or a horizon line.

11 . The method of claim 7 , wherein the invariant object includes at least one of a linear feature or a planar feature.

12 . The method of claim 11 , wherein the determining the calibration error comprises quantifying an error associated with at least a portion of the subset of the sensor data relative to at least one of a line corresponding to the linear feature or a plane corresponding to the planar feature.

13 . The method of claim 6 , wherein:
the sensor comprises at least one of a LiDAR sensor, depth sensor, multi-view image, or a time-of-flight sensor; the invariant object comprises a planar surface; and the determining the calibration error comprises determining a distance between depths measured by the sensor and a plane corresponding to the planar surface.

14 . The method of claim 6 , wherein:
the sensor comprises at least one of a camera, a LiDAR senor, or a time-of-flight sensor; the invariant object comprises a linear feature; and the determining the calibration error comprises determining a distance between points on an edge detected in the sensor data and a line corresponding to the linear feature.

15 . The method of claim 6 , further comprising:
determining, based at least in part on the calibration error, a calibration function for the sensor; and calibrating the sensor using the calibration function.

16 . The method of claim 6 , further comprising:
adjusting, based at least in part on the semantic segmentation information, a setting of the sensor, the setting comprising at least one of an exposure time, an emitted light intensity, or a gain of the sensor.

17 . A non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform operations comprising: receiving, from a sensor, sensor data of an environment;
determining, based at least in part on the sensor data, information about the environment; determining, based at least in part on the information, a subset of the sensor data corresponding to an invariant object; identifying, in the subset of the sensor data, at least one feature corresponding to a feature of the invariant object; and determining, based at least in part on the at least one feature, a calibration error of the sensor.

18 . The non-transitory computer-readable medium of claim 16 , wherein the information comprises associations between portions of the data and a plurality of objects in the environment depicted in the portions, the operations further comprising:
determining, the invariant object from the plurality of objects; and determining, based at least in part on the one or more associations, the portion of the sensor data corresponding to the invariant object.

19 . The non-transitory computer-readable medium of claim 17 , wherein:
the sensor comprises at least one of a camera, a LiDAR senor, or a time-of-flight sensor; the invariant object comprises a linear feature; and the determining the calibration error comprises determining a distance between points on an edge detected in the sensor data and a line corresponding to the linear feature.

20 . The non-transitory computer-readable medium of claim 17 , wherein the information comprises associations between pixels and one or more objects in the environment depicted in the pixels, the method further comprising:
determining, based at least in part on the one or more associations, a semantic segmentation information subset associated with an object of the one or more objects depicted in the pixels; and determining, from the sensor data, a sensor data subset representative of the object, wherein the comparing the sensor data to the information comprises comparing the sensor data subset to the semantic segmentation information subset.

Assignees

• Zoox Inc

Inventors

• Kroeger Till

Classifications

• G01S7/4091

  during normal radar operation · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G01S17/931

  of land vehicles · CPC title

• G01S7/497

  Means for monitoring or calibrating · CPC title

• G06T2207/30204

  Marker · CPC title