Emergency vehicle detection and response

Patent metadata

Field	Value
Publication number	US-11450205-B2
Application number	US-201916732151-A
Country	US
Kind code	B2
Filing date	Dec 31, 2019
Priority date	Dec 31, 2019
Publication date	Sep 20, 2022
Grant date	Sep 20, 2022

Abstract

Techniques for detecting and responding to an emergency vehicle are discussed. A vehicle computing system may determine that an emergency vehicle based on sensor data, such as audio and visual data. In some examples, the vehicle computing system may determine aggregate actions of objects (e.g., other vehicles yielding) proximate the vehicle based on the sensor data. In such examples, a determination that the emergency vehicle is operating may be based on the actions of the objects. The vehicle computing system may, in turn, identify a location to move out of a path of the emergency vehicle (e.g., yield) and may control the vehicle to the location. The vehicle computing system may determine that the emergency vehicle is no longer relevant to the vehicle and may control the vehicle along a route to a destination. Determining to yield and/or returning to a mission may be confirmed by a remote operator.

First claim

What is claimed is:

1. A system comprising:
a sensor;
one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
receiving, from at least the sensor, sensor data representing an environment of a vehicle, wherein the sensor data comprises at least audio data and visual data, wherein the visual data comprises at least one of image data captured by a camera or lidar data captured by a lidar sensor;
determining, based at least in part on the sensor data, a motion trend associated with a first threshold percentage of objects operating on a drivable surface of the environment;
determining that a difference between the motion trend and an expected motion trend meets or exceeds a second threshold;
determining, based at least in part on the difference meeting or exceeding the second threshold, the audio data, and the visual data, that an emergency vehicle is operating in the environment;
determining, based at least in part on the sensor data, an emergency vehicle location and a predicted trajectory associated with the emergency vehicle;
determining, based at least in part on the emergency vehicle location and the predicted trajectory associated with the emergency vehicle, to yield to the emergency vehicle;
determining, based at least in part on determining to yield, a yielding location for the vehicle to move outside a path associated with the predicted trajectory of the emergency vehicle; and
controlling the vehicle based at least in part on the yielding location.

2. The system of claim 1 , the operations further comprising:
sending at least one of the sensor data or data based on the sensor data to a remote computing device; and
receiving, from the remote computing device, a second indication that the emergency vehicle is operating in the environment,
wherein determining that the emergency vehicle is operating in the environment is based at least in part on the second indication.

3. The system of claim 1 , wherein determining that the emergency vehicle is operating in the environment comprises:
determining that the audio data is associated with the emergency vehicle; and
determining that a direction of arrival associated with the audio data is indicative of an approaching emergency vehicle.

4. The system of claim 1 , wherein determining that the emergency vehicle is operating in the environment comprises:
determining that a classification of a detected object represented in the sensor data comprises the emergency vehicle; and at least one of:
determining that a lighting sequence associated with the detected object is indicative of an emergency state; or
determining that a light associated with the lighting sequence comprises at least one of a blue or a red light.

5. The system of claim 1 , wherein determining to yield is further based on the difference exceeding the second threshold.

6. A method comprising:
receiving, from a sensor, sensor data associated with an environment in which a vehicle operates;
determining an action associated with a detected object in the environment, wherein the action comprises at least one of: a pull-over action, a lane-change action, a yielding action, a slowing action, or a turning action;
determining, based at least in part on the sensor data and the action, that an emergency vehicle is operating in the environment;
determining a likelihood that the emergency vehicle will impact motion of the vehicle;
determining an average direction of movement associated with a threshold number of vehicles operating in the environment;
determining, based on the likelihood, to yield to the emergency vehicle at a yielding location, wherein the yielding location is associated with the average direction of movement; and
controlling the vehicle to the yielding location based at least in part on determining to yield.

7. The method of claim 6 , wherein determining that the emergency vehicle is operating in the environment further comprises at least one of:
determining that audio data of the sensor data is associated with the emergency vehicle;
determining that a direction of arrival associated with the audio data is indicative of an approaching emergency vehicle;
determining that a classification of a first detected object comprises the emergency vehicle;
determining that a light associated with the first detected object is indicative of an emergency state; or
determining that the light comprises at least a red light.

8. The method of claim 6 , further comprising:
determining, based at least in part on the sensor data, a first trajectory associated with the detected object; and
determining a difference between the first trajectory and a first expected trajectory,
wherein determining the likelihood is based at least in part on the difference.

9. The method of claim 8 , wherein the determining an average direction of movement comprises:
determining, based at least in part on the sensor data, trajectories of objects operating on a drivable surface associated with the vehicle in the environment; and
determining directions associated with the trajectories of the objects.

10. The method of claim 6 , further comprising:
sending, based at least on the likelihood, at least one of the sensor data or data based on the sensor data to a remote computing device; and
receiving, from the remote computing device, a confirmation that the emergency vehicle is operating in the environment,
wherein determining that the emergency vehicle is operating in the environment is further based at least in part on the confirmation.

11. The method of claim 6 , further comprising:
receiving additional sensor data from the sensor;
determining, based at least in part on the additional sensor data, that one or more release criteria are satisfied;
sending, based at least in part on determining that the one or more release criteria are satisfied, at least one of the additional sensor data or data based on the additional sensor data to a remote computing device;
receiving, from the remote computing device, confirmation of the one or more release criteria being satisfied; and
controlling the vehicle toward a destination based at least in part on the confirmation.

12. The method of claim 11 , further comprising at least one of:
determining a vehicle trajectory associated with vehicle travel to the destination; or
receiving, from the remote computing device, a waypoint associated with vehicle travel to the destination,
wherein controlling the vehicle is based at least in part on at least one of the vehicle trajectory or the waypoint.

13. The method of claim 6 , wherein determining the yielding location further comprises at least one of:
determining that the yielding location is associated with a drivable surface;
determining that the yielding location is outside a path of the emergency vehicle;
determining that a predicted trajectory associated with the emergency vehicle is associated with a first lane of a road in the environment and that the yielding location is associated with a second lane of the road; or
determining that an object is not located at the yielding location.

14. The method of claim 6 , further comprising:
determining that the emergency vehicle is irrelevant to the vehicle;
determining a vehicle trajectory from the yielding location to a destination associated with the vehicle; and
controlling the vehicle based at least in part on the vehicle trajectory.

15. The method of claim 14 , wherein

Assignees

• Zoox Inc

Inventors

• Tariq Sarah
• Gogna Ravi
• Wimmershoff Marc
• Subasingha Subasingha Shaminda

Classifications

• B60W2756/10

  Involving external transmission of data to or from the vehicle · CPC title

• G08G1/017

  identifying vehicles (G08G1/015, G08G1/054 take precedence) · CPC title

• G08G1/096791

  where the origin of the information is another vehicle · CPC title

• B60W60/001

  Planning or execution of driving tasks · CPC title

• B60W2420/54

  Audio sensitive means, e.g. ultrasound · CPC title