Bandwidth constrained image processing for autonomous vehicles

Patent metadata

Field	Value
Publication number	US-10659991-B2
Application number	US-201716461104-A
Country	US
Kind code	B2
Filing date	Dec 6, 2017
Priority date	Dec 6, 2016
Publication date	May 19, 2020
Grant date	May 19, 2020

Abstract

Methods, apparatuses, systems, and non-transitory computer readable storage media for providing bandwidth constrained image processing are described. The disclosed technology determines a data transfer rate of at least one signal received from a vehicle, the at least one signal including state/status data of the vehicle. In response to determining that the data transfer rate satisfies a data transfer rate criterion, a location of the vehicle and a location of at least one of a plurality of objects that obstruct the at least one signal is determined using the state data and external data associated with the vehicle. The disclosed technology generates optimized state data using the state data and the external data.

First claim

What is claimed is:

1. A method, the method comprising:
determining, by a signal processing system of a controller apparatus, a data transfer rate of a plurality of signals received from a vehicle, the plurality of signals including state data of the vehicle;
in response to determining that the data transfer rate satisfies a data transfer rate criterion, determining, by the signal processing system and using the state data and external data associated with the vehicle, a location of the vehicle and a location of at least one of a plurality of objects that obstruct at least one of the plurality of signals; and
generating, by the signal processing system, optimized state data using the state data and the external data.

2. The method of claim 1 , wherein the state data includes sensor data of a plurality of sensors associated with the vehicle, wherein the plurality of sensors includes any of an optical sensor, a light detection and ranging sensor, a motion sensor, and an audio sensor.

3. The method of claim 2 , further comprising:
sending, by a communication unit of the controller apparatus, a prioritized sensor data message to the vehicle, the prioritized sensor data message including a request for prioritized sensor data from the plurality of sensors, the prioritized sensor data including a prioritized listing of the plurality of sensors using locations of the plurality of sensors; and
receiving, by the communication unit, the sensor data from the vehicle in an order using the prioritized listing.

4. The method of claim 2 , further comprising:
filtering, by a filter of the controller apparatus, the sensor data to remove noise from the sensor data, the optimized state data being generated using the filtered sensor data.

5. The method of claim 2 , further comprising:
generating, by an image processing system of the controller apparatus, image data using the sensor data;
storing, by the image processing system, the image data in a database; and
retrieving, by the image processing system, the image data when the data transfer rate satisfies the data transfer rate criterion, the optimized state data being generated using the retrieved image data.

6. The method of claim 5 , further comprising:
generating, by the image processing system, an image using the image data; and
in response to receiving an input to select a portion of the image, modifying, by the image processing system, the portion of the image that was selected, the modifying including any of magnifying the portion of the image, filtering noise from the portion of the image, and increasing contrast in the portion of the image.

7. The method of claim 1 , further comprising:
modifying, by the signal processing system, any of a type of compression and an amount of compression that is applied to the state data using the data transfer rate.

8. The method of claim 1 , further comprising:
determining, by the signal processing system, a transmission delay using a time interval between the plurality of signals being received; and
in response to determining that the transmission delay satisfies a delay criterion, generating, by a machine learning module of the controller apparatus, predictive data using state data previously received within a predetermined time period, the optimized state data being generated using the predictive data.

9. The method of claim 1 , further comprising:
in response to determining that the data transfer rate satisfies the data transfer rate criterion, sending, by a communication unit, instruction data to the vehicle, the instruction data including any of an instruction for the vehicle to stop and an instruction for the vehicle to change location to eliminate obstruction of the at least one of the plurality of signals by the at least one of the plurality of objects.

10. A controller apparatus, comprising:
a memory; and
a processor configured to execute instructions stored in the memory to:
determine a data transfer rate of a plurality of signals received from a vehicle, the plurality of signals including state data of the vehicle;
in response to determining that the data transfer rate satisfies a data transfer rate criterion, determine, using the state data and external data associated with the vehicle, a location of the vehicle and a location of at least one of a plurality of objects that obstruct at least one of the plurality of signals; and
generate optimized state data using the state data and the external data.

11. The controller apparatus of claim 10 , wherein the state data includes sensor data of a plurality of sensors associated with the vehicle, wherein the plurality of sensors includes any of an optical sensor, a light detection and ranging sensor, a motion sensor, and an audio sensor.

12. The controller apparatus of claim 11 , wherein the processor is further configured to execute instructions stored in the memory to:
send a prioritized sensor data message to the vehicle, the prioritized sensor data message including a request for prioritized sensor data from the plurality of sensors, the prioritized sensor data including a prioritized listing of the plurality of sensors using locations of the plurality of sensors; and
receive the sensor data from the vehicle in an order using the prioritized listing.

13. The controller apparatus of claim 11 , wherein the processor is further configured to execute instructions stored in the memory to:
determine a transmission delay using a time interval between the plurality of signals being received; and
in response to determining that the transmission delay satisfies a delay criterion, generate predictive data using state data previously received within a predetermined time period, the optimized state data being generated using the predictive data.

14. The controller apparatus of claim 11 , wherein the processor is further configured to execute instructions stored in the memory to:
generate image data using the sensor data;
store the image data in a database; and
retrieve the image data when the data transfer rate satisfies the data transfer rate criterion, the optimized state data being generated using the retrieved image data.

15. The controller apparatus of claim 14 , wherein the processor is further configured to execute instructions stored in the memory to:
generate an image using the image data; and
in response to receiving an input to select a portion of the image, modify the portion of the image that was selected, wherein to modify includes any of magnifying the portion of the image, filtering noise from the portion of the image, and increasing contrast in the portion of the image.

16. A non-transitory computer-readable storage medium including program instructions executable by one or more processors of a controller apparatus that, when executed, cause the one or more processors to perform operations, the operations comprising:
determining a data transfer rate of a plurality of signals received from a vehicle, the plurality of signals including state data of the vehicle;
in response to determining that the data transfer rate satisfies a data transfer rate criterion, determining, using the state data and external data associated with the vehicle, a location of the vehicle and a location of at least one of a plurality of objects that obstruct at least one of the plurality of signals; and
generating optimized state data using the state data and the external data.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the state data includes sensor data of a plurality of sensors assoc

Assignees

• Nissan North America Inc
• Nasa

Inventors

• Pedersen Liam
• Sierhuis Maarten
• Utz Hans
• Penna Mauro Della
• Fong Terrence
• Allan Mark
• Bualat Maria
• Schafer Eric

Classifications

• H04W4/46

  for vehicle-to-vehicle communication [V2V] · CPC title

• G06T2200/16

  involving adaptation to the client's capabilities · CPC title

• H04L67/12 Primary

  specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks · CPC title

• H04L43/0894

  Packet rate · CPC title

• G06T2207/20104

  Interactive definition of region of interest [ROI] · CPC title