Processing graph representations of tactical maps using neural networks

Patent metadata

Field	Value
Publication number	US-11555706-B1
Application number	US-201816143117-A
Country	US
Kind code	B1
Filing date	Sep 26, 2018
Priority date	Sep 27, 2017
Publication date	Jan 17, 2023
Grant date	Jan 17, 2023

Abstract

A graph representation of a tactical map representing a plurality of static components of an environment of a vehicle is generated. Nodes of the graph represent static components, and edges represent relationships between multiple static components. Different edge types are used to indicate respective relationship semantics among the static components. Individual nodes are represented as having the same number and types of edges in the graph. Using the graph as input to a neural network based model, a set of results is obtained. A motion control directive based at least in part on the results is transmitted to a motion-control subsystem of the vehicle.

First claim

What is claimed is:

1. A method, comprising:
performing, at one or more computing devices:
obtaining a first tactical map associated with an environment of a first vehicle, wherein the first tactical map indicates at least a plurality of static components of the environment;
generating a homogenized graph representation of the first tactical map comprising a plurality of nodes and a plurality of edges, wherein individual ones of the nodes represent respective static components, wherein individual ones of the edges represent respective relationships between multiple static components, wherein an edge type indicative of relationship semantics of a first edge associated with a first node differs from an edge type of a second edge associated with the first node, and wherein the number of edges associated with the first node of the graph is equal to the number of edges associated with individual ones of one or more other nodes of the graph;
obtaining, using the homogenized graph representation as input to a neural network-based machine learning model, at least a first set of reasoning results pertaining to the first vehicle and one or more static components; and
transmitting, to a motion-control subsystem of the first vehicle, one or more motion-control directives based at least in part on the first set of reasoning results.

2. The method as recited in claim 1 , wherein the first neural network-based machine learning model comprises one or more convolution layers.

3. The method as recited in claim 2 , wherein a propagation function of a particular convolution layer of the one or more convolution layers comprises a non-linear function.

4. The method as recited in claim 2 , wherein a propagation function of a particular convolution layer of the one or more convolution layers comprises a summation, over one or more edge types, of a product of at least (a) an adjacency matrix associated with an edge type (b) input received from another layer of the neural network and (c) a vector representing learned weights associated with respective edge types.

5. The method as recited in claim 1 , wherein the first node represents one or more of: (a) a lane segment of a road, (b) an intersection, (c) a traffic sign, (d) a traffic signal or (e) a pedestrian walkway.

6. The method as recited in claim 1 , wherein a particular edge of the homogenized graph representation indicates one or more of: (a) a geometric constraint associated with at least a pair of static components, (b) a topological constraint associated with at least a pair of static components or (c) one or more attributes associated with at least a pair of static components.

7. The method as recited in claim 1 , further comprising performing, by the one or more computing devices:
analyzing, using one or more machine-learning models, a representation of one or more moving objects in the environment of the first vehicle, wherein the one or more motion-control directives are based at least in part on results of said analyzing.

8. The method as recited in claim 1 , wherein the first set of reasoning results includes a probabilistic response to a query pertaining to a movement of the first vehicle from one lane segment to another.

9. A system, comprising:
one or more computing devices;
wherein the one or more computing devices are configured to:
generate a graph representation of a first tactical map, wherein the first tactical map comprises information pertaining to a plurality of static components of an environment of a vehicle, wherein the graph representation comprises a plurality of nodes and a plurality of edges, wherein individual ones of the nodes represent respective static components, wherein individual ones of the edges represent respective relationships between multiple static components, and wherein an edge type indicative of relationship semantics of a first edge associated with a first node differs from an edge type of a second edge associated with the first node;
obtain, using at least a portion of the graph representation as input, a first set of results from a neural network-based machine learning model; and
transmit, to a motion-control subsystem of the first vehicle, one or more motion-control directives based at least in part on the first set of results.

10. The system as recited in claim 9 , wherein the first neural network-based machine learning model comprises one or more of: (a) a graph neural network, or (b) a relational network.

11. The system as recited in claim 9 , wherein the first node represents one or more of: (a) a lane segment of a road, (b) an intersection, (c) a traffic sign, (d) a traffic signal or (e) a pedestrian walkway.

12. The system as recited in claim 9 , wherein the first node represents a first lane segment, wherein the graph representation indicates a plurality of properties of the first lane segment including one or more of: (a) a speed limit, (b) a recommended speed, (c) a proceed-with-caution indicator, (d) a must-stop indicator, (e) a keep-clear indicator, (f) a must-not-enter indicator, (g) a stop-if-able indicator, (h) a bid-directional indicator, (i) a geometry indicator or (j) a lane type.

13. The system as recited in claim 9 , wherein a particular edge of the graph representation links the first node to a second node, wherein the first node represents a first lane segment, wherein the second node represents a second lane segment, and wherein the edge type of the particular edge indicates one or more of: (a) whether the second lane segment is to the left of the first lane segment, (b) whether the second lane segment is to the right of the first lane segment, (c) whether the second lane segment is a previous lane segment along a particular path with respect to the first lane segment, (d) whether the second lane segment is a next lane segment along a particular path with respect to the first lane segment, (e) whether the second lane segment overlaps with the first lane segment, (f) whether the second lane segment is coincident with the first lane segment, (g) whether the second lane segment represents a traffic island with respect to the first lane segment, or (h) a permeability of the second lane segment with respect to the first lane segment.

14. The system as recited in claim 9 , wherein the one or more computing devices are configured to:
analyze, using one or more machine learning models, a representation of one or more moving objects in the environment of the first vehicle, wherein the one or more motion-control directives are based at least in part on results of analyzing the representation of the one or more moving objects.

15. One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors cause the one or more processors to:
generate a graph representation of at least a first tactical map, wherein the first tactical map comprises information pertaining to a plurality of static components of an environment of a vehicle, wherein the graph representation comprises a plurality of nodes and a plurality of edges, wherein individual ones of the nodes represent respective static components, wherein individual ones of the edges represent respective relationships between multiple static components, and wherein an edge type indicative of relationship semantics of a first edge associated with a first node differs from an edge type of a second edge associated with the first node;
obtain, using at least a portion of the graph representation as input, a first set of results from a neural network-based machine learning model; and
transmit, to a motion-con

Assignees

• Apple Inc

Inventors

• Levihn Martin
• Wiest Jürgen
• Raiko Pekka Tapani
• Akametalu Anayo Kenechukwu

Classifications

• G05D1/0221

  involving a learning process · CPC title

• G05D1/0088

  characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00) · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G01C21/32 Primary

  Structuring or formatting of map data · CPC title

• G06N3/08

  Learning methods · CPC title