Attribute aware zero shot machine vision system via joint sparse representations

Patent metadata

Field	Value
Publication number	US-10908616-B2
Application number	US-201816033638-A
Country	US
Kind code	B2
Filing date	Jul 12, 2018
Priority date	May 5, 2017
Publication date	Feb 2, 2021
Grant date	Feb 2, 2021

Abstract

Described is a system for object recognition. The system generates a training image set of object images from multiple image classes. Using a training image set and annotated semantic attributes, a model is trained that maps visual features from known images to the annotated semantic attributes using joint sparse representations with respect to dictionaries of visual features and semantic attributes. The trained model is used for mapping visual features of an unseen input image to its semantic attributes. The unseen input image is classified as belonging to an image class, and a device is controlled based on the classification of the unseen input image.

First claim

What is claimed is:

1. A system for object recognition, the system comprising:
one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform operations of:
using a training image set and annotated semantic attributes, training a model that maps visual features from known images to the annotated semantic attributes using joint sparse representations with respect to dictionaries of visual features and semantic attributes;
using the trained model for mapping visual features of an unseen input image to its semantic attributes;
classifying the unseen input image as belonging to an image class; and
controlling a device based on the classification of the unseen input image, wherein the device is a vehicle component, and controlling the device results in a vehicle maneuver.

2. The system as set forth in claim 1 , wherein the one or more processors further perform an operation of generating a training image set comprising object images from a plurality of image classes, wherein each object image in the training image set has been annotated with a class label and semantic attributes describing the object image.

3. The system as set forth in claim 1 , wherein for training the model, a visual feature space and a semantic attribute space are modeled as nonlinear spaces that provide an identical sparse representation for visual features and their corresponding semantic attributes.

4. The system as set forth in claim 1 , wherein the one or more processors further perform operations of:
finding a sparse representation for a visual feature extracted from the unseen input image; and
generating a semantic attribute prediction that is resolved in the semantic attribute space of the model, wherein a soft-assignment probability vector identifies a probability of the semantic attribute prediction belonging to a class of unseen images.

5. The system as set forth in claim 4 , wherein a regularization parameter is used to regulate entropy of the soft-assignment probability vector.

6. The system as set forth in claim 4 , wherein, given the semantic attribute prediction, the unseen input image is labeled using a class label of a closest semantic attribute in the semantic attribute space of the model.

7. A computer implemented method for object recognition, the method comprising an act of:
causing one or more processors to execute instructions encoded on a non-transitory computer-readable medium, such that upon execution, the one or more processors perform operations of:
using a training image set and annotated semantic attributes, training a model that maps visual features from known images to the annotated semantic attributes using joint sparse representations with respect to dictionaries of visual features and semantic attributes;
using the trained model for mapping visual features of an unseen input image to its semantic attributes;
classifying the unseen input image as belonging to an image class; and
controlling a device based on the classification of the unseen input image, wherein the device is a vehicle component, and controlling the device results in a vehicle maneuver.

8. The method as set forth in claim 7 , wherein the one or more processors further perform an operation of generating a training image set comprising object images from a plurality of image classes, wherein each object image in the training image set has been annotated with a class label and semantic attributes describing the object image.

9. The method as set forth in claim 7 , wherein for training the model, a visual feature space and a semantic attribute space are modeled as nonlinear spaces that provide an identical sparse representation for visual features and their corresponding semantic attributes.

10. The method as set forth in claim 7 , wherein the one or more processors further perform operations of:
finding a sparse representation for a visual feature extracted from the unseen input image; and
generating a semantic attribute prediction that is resolved in the semantic attribute space of the model, wherein a soft-assignment probability vector identifies a probability of the semantic attribute prediction belonging to a class of unseen images.

11. The method as set forth in claim 10 , wherein a regularization parameter is used to regulate entropy of the soft-assignment probability vector.

12. The method as set forth in claim 10 , wherein, given the semantic attribute prediction, the unseen input image is labeled using a class label of a closest semantic attribute in the semantic attribute space of the model.

13. A computer program product for object recognition, the computer program product comprising:
a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions by one or more processors, the one or more processors perform operations of:
using a training image set and annotated semantic attributes, training a model that maps visual features from known images to the annotated semantic attributes using joint sparse representations with respect to dictionaries of visual features and semantic attributes;
using the trained model for mapping visual features of an unseen input image to its semantic attributes;
classifying the unseen input image as belonging to an image class; and
controlling a device based on the classification of the unseen input image, wherein the device is a vehicle component, and controlling the device results in a vehicle maneuver.

14. The computer program product as set forth in claim 13 , further comprising instructions for causing the one or more processors to further perform an operation of generating a training image set comprising object images from a plurality of image classes, wherein each object image in the training image set has been annotated with a class label and semantic attributes describing the object image.

15. The computer program product as set forth in claim 13 , wherein for training the model, a visual feature space and a semantic attribute space are modeled as nonlinear spaces that provide an identical sparse representation for visual features and their corresponding semantic attributes.

16. The computer program product as set forth in claim 13 , further comprising instructions for causing the one or more processors to further perform operations of:
finding a sparse representation for a visual feature extracted from the unseen input image; and
generating a semantic attribute prediction that is resolved in the semantic attribute space of the model, wherein a soft-assignment probability vector identifies a probability of the semantic attribute prediction belonging to a class of unseen images.

17. The computer program product as set forth in claim 16 , wherein a regularization parameter is used to regulate entropy of the soft-assignment probability vector.

18. The computer program product as set forth in claim 16 , wherein, given the semantic attribute prediction, the unseen input image is labeled using a class label of a closest semantic attribute in the semantic attribute space of the model.

19. The system as set forth in claim 1 , wherein the vehicle maneuver is a collision avoidance maneuver.

20. The system as set forth in claim 1 , wherein the unseen input image is an image of an avoidance object, and wherein an alert is generated when the avoidance object is classified.

Assignees

• Hrl Lab Llc

Inventors

• Kolouri Soheil
• Rostami Mohammad
• Kim Kyungnam
• Owechko Yuri

Classifications

• G06V10/7715 Primary

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06F18/2155

  characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling · CPC title

• G06F18/2136

  based on sparsity criteria, e.g. with an overcomplete basis · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title