Sensor validation using semantic segmentation information

Patent metadata

Field	Value
Publication number	US-11458912-B2
Application number	US-201916297127-A
Country	US
Kind code	B2
Filing date	Mar 8, 2019
Priority date	Mar 8, 2019
Publication date	Oct 4, 2022
Grant date	Oct 4, 2022

Abstract

This disclosure is directed to validating a calibration of and/or calibrating sensors using semantic segmentation information about an environment. For example, the semantic segmentation information can identify bounds of objects, such as invariant objects, in the environment. Techniques described herein may determine sensor data associated with the invariant objects and compare that data to a feature known from the invariant object. Misalignment of sensor data with the known feature can be indicative of a calibration error. In some implementations, the calibration error can be determined as a distance between the sensor data and a line or plane representing a portion of the invariant object.

First claim

What is claimed is:

1. An autonomous vehicle comprising:
a sensor disposed on the autonomous vehicle:
one or more processors; and
non-transitory computer-readable media storing one or more instructions that, when executed, cause the one or more processors to perform acts comprising:
receiving, from the sensor, sensor data of an environment of the vehicle, the sensor data including one or more distortions associated with one or more intrinsic characteristics of the sensor,
generating, from the sensor data and based on the one or more intrinsic characteristics of the sensor, undistorted sensor data free of the one or more distortions,
segmenting, as segmented data, the undistorted sensor data based at least in part on a representation of an invariant object in the undistorted sensor data, the invariant object having a known attribute,
determining, based at least in part on the segmented data, a subset of the undistorted sensor data associated with the invariant object, and
determining, based at least in part on the subset of the undistorted sensor data and the known attribute, a calibration error associated with the sensor.

2. The autonomous vehicle of claim 1 , wherein determining the calibration error comprises:
determining, based at least in part on comparing the undistorted sensor data to the known attribute, a distance between the undistorted sensor data and an expected value associated with the known attribute of the invariant object.

3. The autonomous vehicle of claim 2 , wherein the comparing the undistorted sensor data to the known attribute of the invariant object comprises comparing the at least a portion of the subset of the undistorted sensor data to at least one of a line, a plane, or a reflectivity of the invariant object.

4. The autonomous vehicle of claim 1 , the acts further comprising:
calibrating, based at least in part on the calibration error, the sensor;
generating, based at least in part on the calibrated sensor, a trajectory to control the autonomous vehicle; and
controlling the autonomous vehicle based at least in part on the trajectory.

5. The autonomous vehicle of claim 1 , wherein generating undistorted sensor data comprises:
identifying, in the sensor data, a feature location corresponding to at least one feature associated with the invariant object; and
determining, based on a distortion model of the sensor, a true location corresponding to the feature location.

6. A method comprising:
receiving, from a sensor, sensor data of an environment;
generating, based at least in part on removing one or more distortions from the sensor data, undistorted sensor data, the removing the one or more distortions being based at least in part on one or more intrinsic characteristics of the sensor;
determining, based at least in part on the undistorted sensor data, segmentation information about the environment;
determining, based at least in part on the segmentation information, a subset of the undistorted sensor data corresponding to an invariant object, the invariant object having a known attribute;
identifying, in the subset of the undistorted sensor data, at least one feature corresponding to the known attribute of the invariant object; and
determining, based at least in part on the at least one feature and the known attribute, a calibration error of the sensor.

7. The method of claim 6 , wherein the generating the undistorted sensor data comprises:
undistorting the sensor data based at least in part on sensor extrinsic information.

8. The method of claim 7 , wherein the segmentation information comprises associations between portions of the data and a plurality of objects in the environment depicted in the portions, the method further comprising:
determining, the invariant object from the plurality of objects; and
determining, based at least in part on the associations, the portion of the sensor data corresponding to the invariant object.

9. The method of claim 8 , wherein:
the sensor comprises at least one of a camera, a LiDAR senor, or a time-of-flight sensor;
the invariant object comprises a reflective feature; and
the determining the calibration error comprises determining a difference between points of known reflectivity on the invariant object and corresponding points in the portion of the sensor data corresponding to the invariant object.

10. The method of claim 6 , wherein the invariant object is at least one of a topographical feature, a fixture, a building, or a horizon line.

11. The method of claim 10 , wherein the invariant object includes at least one of a linear feature, a planar feature, or a feature exhibiting a characteristic reflectivity.

12. The method of claim 11 , wherein the determining the calibration error comprises quantifying an error associated with a portion of the subset of the sensor data corresponding to a line of the linear feature or a plane of the planar feature.

13. The method of claim 6 , wherein:
the sensor comprises at least one of a LiDAR sensor, depth sensor, multi-view image, or a time-of-flight sensor;
the invariant object comprises a planar surface; and
the determining the calibration error comprises determining a distance between depths measured by the sensor and a plane corresponding to the planar surface.

14. The method of claim 6 , wherein:
the sensor comprises at least one of a camera, a LiDAR senor, or a time-of-flight sensor;
the invariant object comprises a linear feature; and
the determining the calibration error comprises determining a distance between points on an edge detected in the sensor data and a line corresponding to the linear feature.

15. The method of claim 6 , further comprising:
determining, based at least in part on the calibration error, a calibration function for the sensor; and
calibrating the sensor using the calibration function.

16. The method of claim 6 , further comprising:
adjusting, based at least in part on the segmentation information, a setting of the sensor, the setting comprising at least one of an exposure time, an emitted light intensity, or a gain of the sensor.

17. A non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform operations comprising: receiving, from a sensor, sensor data of an environment;
generating, based at least in part on removing one or more distortions from the sensor data, undistorted sensor data, the removing the one or more distortions being based at least in part on one or more intrinsic characteristics of the sensor;
determining, based at least in part on the undistorted sensor data, segmentation information about the environment;
determining, based at least in part on the segmentation information, a subset of the undistorted sensor data corresponding to an invariant object, the invariant object having a known attribute;
identifying, in the subset of the undistorted sensor data, at least one feature corresponding to the known attribute of the invariant object; and
determining, based at least in part on the at least one feature and the known attribute, a calibration error of the sensor.

18. The non-transitory computer-readable medium of claim 17 , wherein the information comprises one or more associations between portions of the data and a plurality of objects in the environment depicted in the portions, the operations further comprising:
determining, the invariant object from the plurality of objects; and
determining, based at least in part on the one or more associations, the portion of the sensor data

Assignees

• Zoox Inc

Inventors

• Kroeger Till

Classifications

• G01S15/931

  of land vehicles · CPC title

• G01S2013/93271

  in the front of the vehicles · CPC title

• G01S2013/93273

  on the top of the vehicles · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/80 Primary

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title