Cognitive mapping for vehicles

Patent metadata

Field	Value
Publication number	US-10345822-B1
Application number	US-201815881228-A
Country	US
Kind code	B1
Filing date	Jan 26, 2018
Priority date	Jan 26, 2018
Publication date	Jul 9, 2019
Grant date	Jul 9, 2019

Abstract

A system, comprising a processor, and a memory, the memory including instructions to be executed by the processor to acquire the images of the vehicle environment, determine a cognitive map, which includes a top-down view of the vehicle environment, based on the image, and operate the vehicle based on the cognitive map.

First claim

We claim:

1. A method, comprising:
acquiring, from an image sensor, an image of a vehicle environment;
determining, by executing programming in a processor, a cognitive map as output from a convolutional neural network (CNN) that accepts the image as input, the cognitive map including a plurality of objects, including a class, location, and pose of each object in a top-down view of the vehicle environment, wherein the cognitive map includes a plurality of planes, each of the planes including at most a single class of object; and
operating the vehicle based on the cognitive map.

2. The method of claim 1 , wherein the vehicle environment includes a roadway, and the objects include other vehicles and pedestrians.

3. The method of claim 2 , further comprising determining the cognitive map including locations of the objects including at least one of other vehicles and pedestrians, relative to the vehicle.

4. The method of claim 1 , wherein the image is a monocular video frame.

5. The method of claim 1 , further comprising training the convolutional neural network based on ground truth data prior to determining the cognitive map.

6. The method of claim 5 , wherein ground truth data is based on object detection, pixel-wise segmentation, 3D object pose, and relative distance.

7. The method of claim 6 , wherein training the convolutional neural network is based on prediction images included in the convolutional neural network.

8. The method of claim 7 , wherein the prediction images are based on ground truth data.

9. A system, comprising a processor; and a memory, the memory including instructions to be executed by the processor to:
acquire an image of a vehicle environment;
determine a cognitive map as output from a convolutional neural network (CNN) that accepts the image as input, the cognitive map including a plurality of objects, including a class, location, and pose of each object in a top-down view of the vehicle environment, wherein the cognitive map includes a plurality of planes, each of the planes including at most a single class of object; and
operate the vehicle based on the cognitive map.

10. The processor of claim 9 , wherein the vehicle environment includes a roadway, and the objects include other vehicles and pedestrians.

11. The processor of claim 10 , the instructions further including instructions to determine the cognitive map including locations of the objects including at least one of other vehicles and pedestrians, relative to the vehicle.

12. The processor of claim 9 , wherein the image is a monocular video frame.

13. The processor of claim 9 , wherein the convolutional neural network is trained based on ground truth data prior to determining the cognitive map.

14. The processor of claim 13 , wherein ground truth data includes object detection, pixel-wise segmentation, 3D object pose, and relative distance.

15. The processor of claim 14 , wherein training the convolutional neural network is based on prediction images included in the convolutional neural network.

16. The processor of claim 15 , wherein the prediction images are based on ground truth data.

17. A system, comprising:
a video sensor operative to acquire an image of a vehicle environment;
vehicle components operative to operate a vehicle;
a processor; and a memory, the memory including instructions to be executed by the processor to:
acquire the image of the vehicle environment;
determine a cognitive map as output from a convolutional neural network (CNN) that accepts the image as input, the cognitive map including a plurality of objects, including a class, location, and pose of each object in a top-down view of the vehicle environment, wherein the cognitive map includes a plurality of planes, each of the planes including at most a single class of object; and

operate the vehicle based on the cognitive map.

18. The system of claim 17 , wherein the vehicle environment includes a roadway, and the objects include other vehicles and pedestrians.

Assignees

• Ford Global Tech Llc

Inventors

• Parchami Mostafa
• Taimouri Vahid
• Puskorius Gintaras Vincent

Classifications

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/24143

  Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN] · CPC title

• G06N3/045

  Combinations of networks · CPC title