Automated extraction of semantic information to enhance incremental mapping modifications for robotic vehicles

Patent metadata

Field	Value
Publication number	US-9734455-B2
Application number	US-201514932940-A
Country	US
Kind code	B2
Filing date	Nov 4, 2015
Priority date	Nov 4, 2015
Publication date	Aug 15, 2017
Grant date	Aug 15, 2017

Abstract

Systems, methods and apparatus may be configured to implement automatic semantic classification of a detected object(s) disposed in a region of an environment external to an autonomous vehicle. The automatic semantic classification may include analyzing over a time period, patterns in a predicted behavior of the detected object(s) to infer a semantic classification of the detected object(s). Analysis may include processing of sensor data from the autonomous vehicle to generate heat maps indicative of a location of the detected object(s) in the region during the time period. Probabilistic statistical analysis may be applied to the sensor data to determine a confidence level in the inferred semantic classification. The inferred semantic classification may be applied to the detected object(s) when the confidence level exceeds a predetermined threshold value (e.g., greater than 50%).

First claim

What is claimed is:

1. A method, comprising:
receiving, at a computing system, first data sensed at a first time by a sensor system of an autonomous vehicle, the first data being representative of a first object of a plurality of objects on or proximate a road surface in a region in an environment the autonomous vehicle has autonomously navigated;
comparing the first data with reference data associated with a plurality of reference semantic classifications;
based at least in part on the comparing, determining that the first object does not match any of the plurality of reference semantic classifications;
identifying additional objects having additional object data similar to the first data, the first object and the additional objects comprising a subset of the objects;
receiving, at the computing system, second data sensed subsequent to the first time, the second data representing a behavior of at least one of the additional objects;
based at least in part on determining that the first object does not match any of the plurality of reference semantic classifications, determining, based on the first data and the second data, a probability that the subset of the objects conforms to a behavior;
generating, at the computing system, an inferred semantic classification associated with the subset of the objects when the probability indicates a pattern of objects in the subset of the objects conforming to the behavior;
associating the inferred semantic classification with the plurality of reference semantic classifications, the inferred semantic classification being different from each of the plurality of reference semantic classifications;
updating, at the computing system, map data associated with the environment to include information about the inferred semantic classification; and
transmitting the updated map data to the autonomous vehicle and at least one additional autonomous vehicle.

2. The method of claim 1 , wherein the determining the probability includes analyzing a heat map generated from sensor data included in at least one of the first data or the second.

3. The method of claim 1 , wherein the objects in the subset of the objects includes data representing a pedestrian object type.

4. The method of claim 3 , wherein the inferred semantic classification associated with the subset of the objects comprises an un-marked pedestrian crossing associated with a road network in the region.

5. The method of claim 1 , wherein the first data comprises route data.

6. The method of claim 5 , wherein the route data comprises a route network definition file.

7. The method of claim 1 , wherein the first data comprises map data.

8. The method of claim 7 , wherein the map data comprises at least one map tile.

9. The method of claim 1 further comprising:
updating the reference semantic classifications to include the inferred semantic classification.

10. The method of claim 1 , wherein the determining the probability comprises analyzing data representing heat maps simultaneously generated from frames of sensor data included in the first data from the autonomous vehicle, the frames of sensor data being generated by a plurality of different types of sensors in the sensor system.

11. The method of claim 10 , wherein the plurality of different types of sensors includes a light detection and ranging sensor and the frames of sensor data generated by the light detection and ranging sensor include data representing laser depth information associated with the subset of the objects.

12. The method of claim 10 , wherein the plurality of different types of sensors includes a multispectral image capture sensor and the frames of sensor data generated by the multispectral image capture sensor include data representing near infrared wavelengths of light associated with the subset of the objects.

13. The method of claim 10 , wherein the plurality of different types of sensors includes an image capture sensor and the frames of sensor data generated by the image capture sensor include data representing color intensity of light associated with the subset of the objects.

14. The method of claim 1 , wherein the determining the probability comprises analyzing map data based on simultaneous localization and mapping, the map data being generated by a localizer of the autonomous vehicle, the map data including temporal information associated with the period of time.

15. The method of claim 14 , wherein the generating the data comprising the inferred semantic classification is based on the temporal information.

16. A system comprising:
a bi-directional autonomous vehicle configured to drive forward in a first direction or drive forward in a substantially opposite second direction without turning around the bi-directional autonomous vehicle, the autonomous vehicle configured to drive autonomously on a roadway;
a plurality of sensors on the bi-directional autonomous vehicle configured to sense a plurality of objects on or proximate a roadway in an environment surrounding the bi-directional autonomous vehicle; and
a computing system communicatively coupled to the bi-directional autonomous vehicle to receive data from the plurality of sensors, the computing system being programmed to:
determine first data for a first object of the plurality of objects, the first object being a moving object at a location in the environment;
compare the first data to reference semantic classifications data indicating object types and object behaviors at locations in the environment, the reference semantic classifications data being associated with one or more reference semantic classifications;
determine, based on the comparison, that the first object does not conform to any of the one or more reference semantic classifications;
determine, based at least in part on data acquired at different times, a pattern of behavior of additional moving objects at the location, each of the additional moving objects at the location having additional moving object data similar to the first data;
based at least in part on the determination that the first object does not conform to any of the one or more reference semantic classifications and based at least in part on the pattern of behavior of the additional moving objects at the location, associate an inferred semantic classification with moving objects at the location;
update the reference semantic classifications data to include the inferred semantic classification as an additional reference semantic classification; and
update route data used to navigate the roadway based on the inferred semantic classification.

17. The system of claim 16 , wherein the moving object is a pedestrian, the additional moving objects are additional pedestrians, and the route data is updated to indicate an un-marked pedestrian crossing the roadway.

18. The system of claim 17 , wherein the system is further programmed to generate navigation instructions responsive to the updated route data.

19. The system of claim 18 , wherein the navigation instructions include instructions to navigate the un-marked pedestrian crossing according to predetermined instructions for navigating a marked pedestrian crossing.

20. The system of claim 16 , wherein the location in the environment comprises a position proximate a stationary object of interest.

21. The system of claim 16 , wherein the computing system is further programmed to transmit the updated route data to a plurality of autonomous vehicles comprising a fleet of autonomous vehicles.

Assignees

• Zoox Inc

Inventors

• Levinson Jesse Sol
• Sibley Gabriel Thurston
• Rege Ashutosh Gajanan

Classifications

• G01S17/894

  Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar · CPC title

• G01S13/865

  Combination of radar systems with lidar systems · CPC title

• G01S13/867

  Combination of radar systems with cameras · CPC title

• G01S7/4026

  Antenna boresight · CPC title

• G01C21/3602

  Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera · CPC title