Sequence-of-sequences model for 3D object recognition

Patent metadata

Field	Value
Publication number	US-11410439-B2
Application number	US-202016870138-A
Country	US
Kind code	B2
Filing date	May 8, 2020
Priority date	May 9, 2019
Publication date	Aug 9, 2022
Grant date	Aug 9, 2022

Abstract

Systems and methods are disclosed for capturing multiple sequences of views of a three-dimensional object using a plurality of virtual cameras. The systems and methods generate aligned sequences from the multiple sequences based on an arrangement of the plurality of virtual cameras in relation to the three-dimensional object. Using a convolutional network, the systems and methods classify the three-dimensional object based on the aligned sequences and identify the three-dimensional object using the classification.

First claim

What is claimed is:

1. A method comprising:
capturing, using a plurality of virtual cameras, multiple sequences of views of a three-dimensional object, each sequence of the multiple sequences representing a unique order of views;
generating, using one or more processors, aligned sequences from the multiple sequences based on an arrangement of the plurality of virtual cameras in relation to the three-dimensional object;
using a convolutional neural network, classifying the three-dimensional object based on the aligned sequences; and
identifying the three-dimensional object using the classification.

2. The method of claim 1 , wherein the arrangement of the plurality of virtual cameras corresponds to an upright orientation and a known rotation axis.

3. The method of claim 1 , wherein each sequence of the multiple sequences has a different starting view.

4. The method of claim 1 , wherein generating aligned sequences further comprises:
aligning each sequence of the multiple sequences using an alignment function to align each sequence of the multiple sequences to a canonical viewpoint.

5. The method of claim 4 , wherein before aligning each sequence, the method further comprises:
generating the alignment function based on a plurality of different starting views.

6. The method of claim 1 , further comprising:
concatenating the aligned sequences; and
classifying the three-dimensional object based on the concatenated aligned sequences.

7. The method of claim 1 , further comprising:
training the convolutional neural network by updating the convolutional neural network with the aligned sequences.

8. The method of claim 1 , further comprising:
classifying the three-dimensional object based on the aligned sequences using a gated recurrent unit.

9. A system comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the system to perform operations comprising:
capturing, using a plurality of virtual cameras, multiple sequences of views of a three-dimensional object, each sequence of the multiple sequences representing a unique order of views;
generating, using one or more processors, aligned sequences from the multiple sequences based on an arrangement of the plurality of virtual cameras in relation to the three-dimensional object;
using a convolutional neural network, classifying the three-dimensional object based on the aligned sequences; and
identifying the three-dimensional object using the classification.

10. The system of claim 9 , wherein the arrangement of the plurality of virtual cameras corresponds to an upright orientation and a known rotation axis.

11. The system of claim 9 , wherein each sequence of the multiple sequences has a different starting view.

12. The system of claim 9 , wherein generating aligned sequences further comprises:
aligning each sequence of the multiple sequences using an alignment function to align each sequence to a canonical viewpoint.

13. The system of claim 12 , wherein before aligning each sequence, the operations further comprise:
generating the alignment function based on a plurality of different starting views.

14. The system of claim 9 , further comprising:
concatenating the aligned sequences; and
classifying the three-dimensional object based on the concatenated aligned sequences.

15. The system of claim 9 , further comprising:
training the convolutional neural network by updating the convolutional neural network with the aligned sequences.

16. The system of claim 9 , further comprising:
classifying the three-dimensional object based on the aligned sequences using a gated recurrent unit.

17. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:
capturing, using a plurality of virtual cameras, multiple sequences of views of a three-dimensional object, each sequence of the multiple sequences representing a unique order of views;
generating, using one or more processors, aligned sequences from the multiple sequences based on an arrangement of the plurality of virtual cameras in relation to the three-dimensional object;
using a convolutional neural network, classifying the three-dimensional object based on the aligned sequences; and
identifying the three-dimensional object using the classification.

18. The computer-readable storage medium of claim 17 , wherein the arrangement of the plurality of virtual cameras corresponds to an upright orientation and a known rotation axis.

19. The computer-readable storage medium of claim 17 , wherein each sequence of the multiple sequences has a different starting view.

20. The computer-readable storage medium of claim 17 , wherein generating aligned sequences further comprises:
aligning each sequence of the multiple sequences using an alignment function to align each sequence to a canonical viewpoint.

Assignees

• Snap Inc

Inventors

• Li Yuncheng
• Ren Zhou
• Xu Ning
• Yan Enxu
• Yu Tan

Classifications

• G06V20/41

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V20/64 Primary

  Three-dimensional [3D] objects · CPC title