Smart node for autonomous vehicle perception augmentation

Patent metadata

Field	Value
Publication number	US-2022019225-A1
Application number	US-202016928844-A
Country	US
Kind code	A1
Filing date	Jul 14, 2020
Priority date	Jul 14, 2020
Publication date	Jan 20, 2022
Grant date	—

Abstract

A system for navigation of a vehicle that includes a vehicle computer vision system to receive digital images of an environment along a path. The vision system has a vision range and includes a processor and programming instructions. The processor detects in the digital images a first set of objects of interest (OOIs) and determines motion of each OOI in the first set of OOIs. The system includes a communication device that receives augmented perception data associated with a node along a portion of the path. The received perception data identifies motion of each OOI of a second set of OOIs detected within a vision range of the node. The system includes a navigation controller that uses a fusion of the first set of OOIs and the second set of OOIs to control motion of the vehicle along the path.

First claim

1 . A method of assisting with navigation of a vehicle, the method comprising:
by a vehicle computer vision system of a vehicle, receiving at least one digital image of an environment along a planned route, wherein the vehicle computer vision system has a vision range; by a processor that is associated with the vehicle:
detecting, in the at least one digital image, a first set of objects of interest;
determining motion of each of the objects of interest in the first set of objects of interest;
receiving augmented perception data associated with a node that is located at an intersection along an imminent path of the planned route, wherein the received augmented perception data identifies motion of each object of interest of a second set of objects of interest detected within a vision range of the node, wherein the vision range of the vehicle computer vision system and the vision range of the node are different; and
using a fusion of the first set of objects of interest and the second set of objects of interest to control motion of the vehicle to and along the imminent path.

2 . The method of claim 1 , wherein:
the node is mounted at the intersection with a traffic signal device; the at least one digital image includes image data representative of the traffic signal device, of an imminent intersection; the method further comprises, by the processor, classifying the at least one traffic signal device to create a traffic light classification state; and the controlling of the motion of the vehicle includes using the traffic light classification state to control the motion of the vehicle to and along the imminent path and through the intersection.

3 . The method of claim 1 , wherein the traffic light classification state includes at least of the following one operational states:
a green light state; a yellow light state; a red light state; a circular light state; a left arrow light state; a right arrow light state; a forward arrow light state; a flashing yellow light state; or a flashing red light state.

4 . The method of claim 1 , wherein:
at least one object of interest in the second set of objects of interest is hidden from the vehicle computer vision system; the method further comprises, by the processor, fusing the at least one object of interest of the second set of objects of interest with the first set of objects of interest to create fusion data with at least one hidden object of interest; and the controlling of the motion of the vehicle includes using the fusion data to control the motion of the vehicle.

5 . The method of claim 1 , further comprising, by the processor:
receiving, from a remote server, network traffic information representative of a traffic condition detected within the vision range of the node and within vision ranges of a plurality of additional nodes; determining a traffic congestion condition at at least one node associated with the imminent path based on the received traffic information; and modifying the imminent path in response to the determined traffic congestion condition; and when controlling of the motion of the vehicle, causing the vehicle to use the modified path.

6 . The method of claim 1 , wherein:
the at least one digital image comprises a plurality of digital images; and the method further comprises, by the processor:
performing image processing of raw digital image data of each digital image of the plurality of digital images;
extracting features from the plurality of digital images to identify each object of interest in the plurality of digital images;
classifying each identified object of interest;
determining a location of the each identified object of interest; and
forecasting motion of the each identified object of interest, wherein the motion includes one or more of the following: speed of each identified object of interest; or direction of movement of each identified object of interest.

7 . The method of claim 1 , wherein the augmented perception data of each object of interest in the second set of objects of interest includes data representative of one or more of the following: an object of interest classification; an object of interest location; an object of interest speed; or an object of interest direction of movement.

8 . The method of claim 7 , wherein:
the object of interest location includes global coordinates of a global coordinate system; at least one object of interest in the second set of objects of interest includes location coordinates which are outside of the vision range of the vehicle computer vision system immediately before or at the time the augmented perception data is received from the node associated with the vision range of the node; and the controlling of the motion of the vehicle includes using the augmented perception data of the at least one object of interest in the second set of objects of interest having the location coordinates outside of the vision range of the vehicle computer vision system.

9 . A system for navigation of a vehicle, the system comprising:
a vehicle computer vision system of a vehicle configured to receive at least one digital image of an environment along a planned route, wherein the vehicle computer vision system has a vision range and comprises a processor and a computer-readable storage medium comprising programming instructions that are configured to, when executed, cause the processor to:
detect in the at least one digital image a first set of objects of interest; and
determine motion of each object of interest in the first set of objects of interest;
a communication device configured to receive augmented perception data associated with a node in range of the vehicle along an imminent path of the planned route, wherein:
the received augmented perception data identifies motion of each object of interest of a second set of objects of interest detected within a vision range of the node, and
the vision range of the vehicle computer vision system and the vision range of the node are different; and
a vehicle navigation controller configured to use a fusion of the first set of objects of interest and the second set of objects of interest to control motion of the vehicle to and along the imminent path.

10 . The system of claim 9 , wherein:
the node is mounted at an intersection with a traffic signal device of an intersection that is associated with the imminent path; and the at least one digital image includes image data representative of the traffic signal device; and the programming instructions are configured to, when executed, further cause the processor to classify, by a traffic light classifier, the at least one traffic signal device to create a traffic light classification state; and the vehicle navigation controller is also configured to use the traffic light classification state to control the motion of the vehicle.

11 . The system of claim 10 , wherein:
the communication device is configured to:
communicate to a remote server system a query for traffic control information associated with the imminent path, and
receive, from a remote server system, traffic light states and traffic condition information based the query;
the programming instructions are configured to, when executed, further cause the processor to update the traffic light classifier; and the vehicle navigation controller is also configured to use the traffic light classification state based on the updated traffic light classifier to control the motion of the vehicle.

12 . The system of claim 9 , wherein the traffic light classification state includes at least of the following one operational states of a traffic light cy

Assignees

• Argo Ai Llc

Inventors

• Foley Sean
• Biala Ilan
• Laverne Michel
• Hays James
• Browning Brett

Classifications

• G06F18/251

  of input or preprocessed data · CPC title

• G06F18/2431

  Multiple classes · CPC title

• B60W60/001

  Planning or execution of driving tasks · CPC title

• G06V20/58 Primary

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06V10/806

  of extracted features · CPC title