Leveraging multi cues for fine-grained object classification

Patent metadata

Field	Value
Publication number	US-10424072-B2
Application number	US-201715418614-A
Country	US
Kind code	B2
Filing date	Jan 27, 2017
Priority date	Mar 1, 2016
Publication date	Sep 24, 2019
Grant date	Sep 24, 2019

Abstract

One embodiment provides a method comprising estimating a camera pose of an input image and aligning the input image to a desired camera pose based on a feature database. The input image comprises an image of a fine-grained object. The method further comprises classifying the object based on the alignment.

First claim

What is claimed is:

1. A method comprising:
estimating a camera pose of an input image, wherein the input image comprises an image of a fine-grained object;
aligning the input image to a desired camera pose by projecting a first multi-dimensional space onto the input image from a second multi-dimensional space based on a feature database comprising a set of multi-dimensional points, wherein a resulting aligned input image comprises the object inside the first multi-dimensional space, and the first multi-dimensional space has fewer dimensions than the second multi-dimensional space and the set of multi-dimensional points; and
classifying the object based on the first multi-dimensional space of the aligned input image.

2. The method of claim 1 , wherein the set of multi-dimensional points comprises a set of sparse multi-dimensional points representing sparse geometry of a shape of the object.

3. The method of claim 2 , wherein the set of sparse multi-dimensional points is based on a set of images including the object, and the set of images are captured from different camera poses to illustrate different illumination changes and backgrounds of the object.

4. The method of claim 3 , wherein a portion of the object in each image of the set of images triangulates to a same multi-dimensional point of the feature database.

5. The method of claim 4 , wherein each multi-dimensional point of the feature database is associated with a corresponding set of local multi-dimensional feature descriptors indicative of a visual appearance of the object about the multi-dimensional point.

6. The method of claim 1 , wherein the classifying the object comprises utilizing a single-layer feature extraction scheme that provides both low-level feature representation and high-level feature representation of the object.

7. The method of claim 1 , wherein the projecting the first multi-dimensional space onto the input image from the second multi-dimensional space comprises:
projecting a second multi-dimensional surface onto the input image, wherein the second multi-dimensional space has a same amount of dimensions as the set of multi-dimensional points and the second multi-dimensional surface; and
transforming the projected second multi-dimensional surface to a first multi-dimensional surface, wherein the first multi-dimensional surface has a same amount of dimensions as the first multi-dimensional space, and the first multi-dimensional surface comprises a portion of the input image that includes the object.

8. The method of claim 7 , wherein the projecting the first multi-dimensional space onto the input image from the second multi-dimensional space comprises applying a manifold learning algorithm.

9. The method of claim 1 , wherein the input image is decomposed as a set of sparse feature maps convolved with one or more learned convolutional kernels.

10. A system, comprising:
at least one processor; and
a non-transitory processor-readable memory device storing instructions that when executed by the at least one processor causes the at least one processor to perform operations including:
estimating a camera pose of an input image, wherein the input image comprises an image of a fine-grained object;
aligning the input image to a desired camera pose by projecting a first multi-dimensional space onto the input image from a second multi-dimensional space based on a feature database comprising a set of multi-dimensional points, wherein a resulting aligned input image comprises the object inside the first multi-dimensional space, and the first multi-dimensional space has fewer dimensions than the second multi-dimensional space and the set of multi-dimensional points; and
classifying the object based on the first multi-dimensional space of the aligned input image.

11. The system of claim 10 , wherein the set of multi-dimensional points comprises a set of sparse multi-dimensional points representing sparse geometry of a shape of the object.

12. The system of claim 11 , wherein the set of sparse multi-dimensional points is based on a set of images including the object, and the set of images are captured from different camera poses to illustrate different illumination changes and backgrounds of the object.

13. The system of claim 12 , wherein a portion of the object in each image of the set of images triangulates to a same multi-dimensional point of the feature database.

14. The system of claim 13 , wherein each multi-dimensional point of the feature database is associated with a corresponding set of local multi-dimensional feature descriptors indicative of a visual appearance of the object about the multi-dimensional point.

15. The system of claim 10 , wherein the classifying the object comprises utilizing a single-layer feature extraction scheme that provides both low-level feature representation and high-level feature representation of the object.

16. The system of claim 15 , wherein the projecting the first multi-dimensional space onto the input image from the second multi-dimensional space comprises:
projecting a second multi-dimensional surface onto the input image, wherein the second multi-dimensional space has a same amount of dimensions as the set of multi-dimensional points and the second multi-dimensional surface; and
transforming the projected second multi-dimensional surface to a first multi-dimensional surface, wherein the first multi-dimensional surface has a same amount of dimensions as the first multi-dimensional space, and the first multi-dimensional surface comprises a portion of the input image that includes the object.

17. The system of claim 16 , wherein the projecting the first multi-dimensional space onto the input image from the second multi-dimensional space comprises applying a manifold learning algorithm.

18. The system of claim 10 , wherein the input image is decomposed as a set of sparse feature maps convolved with one or more learned convolutional kernels.

19. A non-transitory computer readable storage medium including instructions to perform a method comprising:
estimating a camera pose of an input image, wherein the input image comprises an image of a fine-grained object;
aligning the input image to a desired camera pose by projecting a first multi-dimensional space onto the input image from a second multi-dimensional space based on a feature database comprising a set of multi-dimensional points, wherein a resulting aligned input image comprises the object inside the first multi-dimensional space, and the first multi-dimensional space has fewer dimensions than the second multi-dimensional space and the set of multi-dimensional points; and
classifying the object based on the first multi-dimensional space of the aligned input image.

20. The computer readable storage medium of claim 19 , wherein the set of multi-dimensional points comprises a set of sparse multi-dimensional points representing sparse geometry of a shape of the object.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Wang Xiaolong
• Li Robert
• Currey Jon

Classifications

• G06T7/337 Primary

  involving reference images or patches · CPC title

• G06V10/7715 Primary

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06F18/21345

  enforcing sparsity or involving a domain transformation · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06T7/74

  involving reference images or patches · CPC title