Hierarchical machine-learning network architecture

Patent metadata

Field	Value
Publication number	US-11450117-B2
Application number	US-202117215938-A
Country	US
Kind code	B2
Filing date	Mar 29, 2021
Priority date	Jan 2, 2019
Publication date	Sep 20, 2022
Grant date	Sep 20, 2022

Abstract

The techniques discussed herein may comprise refining a classification of an object detected as being represented in sensor data. For example, refining the classification may comprise determining a sub-classification of the object.

First claim

What is claimed is:

1. A method comprising:
receiving sensor data;
providing, as input to a first machine-learning (ML) model, at least a portion of the sensor data;
receiving, from the first ML model, a classification of an object associated with the sensor data;
providing, as input to a second ML model, the classification or at least a portion of the sensor data;
receiving, from the second ML model, a sub-classification of the classification of the object; and
controlling operation of a vehicle based at least in part on the sub-classification.

2. The method of claim 1 , wherein at least one of a first probability associated with the classification, a feature map, or a region of interest is received from the first ML model.

3. The method of claim 1 , wherein providing input to the second ML model further comprises providing, as input to the second ML model, a portion of the sensor data associated with the object.

4. The method of claim 1 , wherein:
the classification is a first classification,
the sub-classification is a first sub-classification, and
the method further comprises:
selecting the second ML model from among multiple ML models based at least in part on the first classification, wherein the multiple ML models comprise a third ML model associated with a second classification different from the first classification and wherein the third ML model is trained to output a second sub-classification different from the first sub-classification, the second sub-classification being associated with the second classification.

5. The method of claim 1 , further comprising:
receiving, from the second ML model, a probability associated with the sub-classification; and
outputting the classification in association with the portion of sensor data based at least in part on determining that the probability is less than a probability threshold, wherein controlling operation of the vehicle is based at least in part ceasing to control the vehicle based at least in part on the sub-classification and controlling the vehicle based at least in part on the classification.

6. The method of claim 1 , wherein:
the classification is a first classification;
the sub-classification is a first sub-classification; and
the method further comprises:
receiving, from the first ML model, a first probability associated with the first classification and a second probability associated with a second classification, wherein the second probability is less than the first probability;
receiving, from the second ML model, a third probability associated with the first sub-classification;
determining that the third probability is less than a probability threshold;
providing, as input to a third ML model, the first output based at least in part on determining that the third probability is less than the probability threshold; and
receiving, from the third ML model, a second sub-classification different than the first sub-classification,
wherein the third ML model is associated with the second classification or the third ML model is associated with the first classification and is trained to output one or more sub-classifications excluding the first classification.

7. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving sensor data;
providing, as input to a first machine-learning (ML) model, at least a portion of the sensor data;
receiving, from the first ML model, a classification of an object associated with the sensor data;
providing, as input to a second ML model, the classification or at least a portion of the sensor data;
receiving, from the second ML model, a sub-classification of the classification of the object; and
controlling operation of a vehicle based at least in part on the sub-classification.

8. The non-transitory computer-readable medium of claim 7 , wherein at least one of a first probability associated with the classification, a feature map, or a region of interest is received from the first ML model.

9. The non-transitory computer-readable medium of claim 7 , wherein providing input to the second ML model further comprises providing, as input to the second ML model, a portion of the sensor data associated with the object.

10. The non-transitory computer-readable medium of claim 7 , wherein:
the classification is a first classification,
the sub-classification is a first sub-classification, and
the operations further comprise:
selecting the second ML model from among multiple ML models based at least in part on the first classification, wherein the multiple ML models comprise a third ML model associated with a second classification different from the first classification and wherein the third ML model is trained to output a second sub-classification different from the first sub-classification, the second sub-classification being associated with the second classification.

11. The non-transitory computer-readable medium of claim 7 , wherein the operations further comprise:
receiving, from the second ML model, a probability associated with the sub-classification; and
outputting the classification in association with a portion of the sensor data based at least in part on determining that the probability is less than a probability threshold, wherein controlling operation of the vehicle is based at least in part ceasing to control the vehicle based at least in part on the sub-classification and controlling the vehicle based at least in part on the classification.

12. The non-transitory computer-readable medium of claim 7 , wherein:
the classification is a first classification;
the sub-classification is a first sub-classification; and
the operations further comprise:
receiving, from the first ML model, a first probability associated with the first classification and a second probability associated with a second classification, wherein the second probability is less than the first probability;
receiving, from the second ML model, a third probability associated with the first sub-classification;
determining that the third probability is less than a probability threshold;
providing, as input to a third ML model, the first output based at least in part on determining that the third probability is less than the probability threshold; and
receiving, from the third ML model, a second sub-classification different than the first sub-classification,
wherein the third ML model is associated with the second classification or the third ML model is associated with the first classification and is trained to output one or more sub-classifications excluding the first classification.

13. The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise:
receiving a fourth probability associated with the second sub-classification; and
controlling the vehicle based at least in part on at least one of the second classification or the second sub-classification based at least in part on determining that the fourth probability meets or exceeds the probability threshold.

14. A vehicle comprising:
one or more processors; and
a memory comprising processor-executable instructions that, when executed by one or more processors, cause the vehicle to perform operations comprising:
receiving sensor data;
providing, as input to a first machine-learning (ML) model, at least a portion of the sensor data;
receiving, from the first ML model, a classification of an object associated with the sensor data;
providing, as input to a second ML model, the

Assignees

• Zoox Inc

Inventors

• Goel Kratarth
• Tariq Sarah

Classifications

• G06V10/255

  Detecting or recognising potential candidate objects based on visual cues, e.g. shapes · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06V20/58 Primary

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06N3/09

  Supervised learning · CPC title