Automated extraction of semantic information to enhance incremental mapping modifications for robotic vehicles

Patent metadata

Field	Value
Publication number	US-2017316333-A1
Application number	US-201715498144-A
Country	US
Kind code	A1
Filing date	Apr 26, 2017
Priority date	Nov 4, 2015
Publication date	Nov 2, 2017
Grant date	—

Abstract

Systems, methods and apparatus may be configured to implement automatic semantic classification of a detected object(s) disposed in a region of an environment external to an autonomous vehicle. The automatic semantic classification may include analyzing over a time period, patterns in a predicted behavior of the detected object(s) to infer a semantic classification of the detected object(s). Analysis may include processing of sensor data from the autonomous vehicle to generate heat maps indicative of a location of the detected object(s) in the region during the time period. Probabilistic statistical analysis may be applied to the sensor data to determine a confidence level in the inferred semantic classification. The inferred semantic classification may be applied to the detected object(s) when the confidence level exceeds a predetermined threshold value (e.g., greater than 50%).

First claim

1 - 20 . (canceled)

21 . A method comprising:
receiving, at a computing system, heat map data, the heat map data generated from sensor log data from one or more environmental sensors associated with an environment and including information associated with one or more classifications associated with the environment; receiving, at the computing system, object information indicative of an object in the environment, the object information being based, at least in part, on sensor data from one or more computing system sensors in communication with the computing system; determining, based at least in part on the object information, an object classification of the object; based at least in part on the heat map data and the object classification, determining a pattern of behavior associated with the object; and based at least in part on the pattern of behavior, associating an additional classification with a region of the environment associated with the object.

22 . The method of claim 21 , wherein the determining the pattern of behavior comprises determining a probability that the object conforms to a behavior.

23 . The method of claim 21 , wherein the heat map data comprises one or more of a time of day or a day of the week.

24 . The method of claim 23 ,
wherein the object classification includes a pedestrian, the pattern of behavior includes transiting the region, and the additional classification includes an un-marked crosswalk.

25 . The method of claim 21 , wherein the computing system is associated with an autonomous vehicle and the environment is an environment in which the autonomous vehicle is traversing.

26 . The method of claim 25 , further comprising commanding the autonomous vehicle to navigate in the environment based at least in part on the additional classification.

27 . The method of claim 28 , wherein the plurality of different types of sensors comprise at least one of a lidar, a radio detecting and ranging sensor, a sound navigation and ranging sensor, a camera, a global positioning system sensor, or an inertial measurement unit.

28 . The method of claim 21 , further comprising:
determining, at the computing system, a position of the computing system relative to the environment, wherein the determining the pattern of behavior is further based, at least in part, on the position of the computing system.

29 . A system comprising:
a computing system communicatively coupled to a vehicle to receive data from a plurality of sensors, the computing system being programmed to:
determine first object data associated with a first object in the environment, the first object data comprising an object classification, a first location in the environment, and a first behavior,
compare the first object data with reference classifications data associated with reference classifications, a reference classification comprising a reference object behavior at a reference location in the environment,
determine a difference between the first object data and the reference classifications data; and
determine an additional classification of the first object.

30 . The system of claim 29 , wherein the computing system is further programmed to:
update the reference classifications data to include the additional classification as an additional reference classification, the additional reference classification being different from the reference classifications; and update route data used to navigate the environment to include the additional reference classification.

31 . The system of claim 29 ,
wherein the reference classifications data comprise heat map data associated with one or more heat maps, and further wherein the first object includes a pedestrian, the additional objects include additional pedestrians, and the first behavior comprises crossing a roadway at a location other than a crosswalk.

32 . The system of claim 29 , the computing system being programmed to further:
calculate a probability of a change of map data based at least upon the difference, the calculation comprising a statistical change point detection; update the map data based at least in part on the probability of the change; and send instructions to control the autonomous vehicle based at least in part on the updated map data, the instructions configured to cause the autonomous vehicle to navigate the environment.

33 . The system of claim 29 , wherein:
the first object data further includes first temporal data comprising at least one of a time of day or a day of the week; and each of the reference classifications further includes reference temporal data comprising at least one of a time of day or a day of the week.

34 . The system of claim 29 , wherein the computing system is further programmed to:
determine additional object data associated with additional objects at the first location, the additional object data including behaviors of the additional objects at the first location; and determine that the additional behaviors comprise a pattern of behavior similar to the first behavior.

35 . The system of claim 34 , wherein the computing system is programmed to determine that the additional behaviors comprise a pattern of behavior similar to the first behavior by determining a probability that the additional objects conform to the pattern of behavior.

36 . The system of claim 34 , wherein the computing system is further programmed to determine the probability that the additional objects conform to the pattern of behavior by analyzing heat map data representing heat maps generated from sensor data and including information about one or more objects in the environment, the sensor data comprising sensor data acquired over time.

37 . A method of generating heat maps, the method comprising:
receiving sensor data from one or more sensors located on a vehicle, the sensor data comprising information about an environment of the vehicle and a time associated with the information; determining, based at least in part on the sensor data, object data for one or more objects proximate to the vehicle, the object data comprising information indicative of a region and a behavior associated with the one or more objects; generating a heat map based at least in part on the object data; and determining a classification of the region based at least in part on the heat map.

38 . The method of claim 37 , further comprising updating map data with the classification of the region.

39 . The method of claim 37 , wherein the classification is a first classification having an associated first time, the method further comprising:
determining a second classification of the region at a second time; and updating map data based, at least in part, on the second classification.

40 . The method of claim 39 , wherein the first classification includes an un-marked crosswalk and the second classification includes a roadway.

41 . The method of claim 37 , wherein the vehicle is an autonomous vehicle, the method further comprising sending a signal to the autonomous vehicle, the signal causing the autonomous vehicle to navigate the environment, the signal based, at least in part, on the heat map.

42 . The method of claim 37 , wherein the heat map comprises a plurality of heat maps, each one of the plurality of heat maps comprising data from a single type of sensor.

Assignees

• Zoox Inc

Inventors

• Levinson Jesse Sol
• Sibley Gabriel Thurston
• Rege Ashutosh Gajanan

Classifications

• B60W40/04 Primary

  Traffic conditions · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G01S17/87

  Combinations of systems using electromagnetic waves other than radio waves · CPC title

• G01S17/931

  of land vehicles · CPC title

• G01S2013/9322

  using additional data, e.g. driver condition, road state or weather data · CPC title