Systems and methods for shape-based object retrieval

Patent metadata

Field	Value
Publication number	US-10691979-B2
Application number	US-201815862512-A
Country	US
Kind code	B2
Filing date	Jan 4, 2018
Priority date	Jan 4, 2017
Publication date	Jun 23, 2020
Grant date	Jun 23, 2020

Abstract

A method for classifying physical objects includes: controlling, by a processor, one or more depth cameras to capture depth images of a query object; controlling, by the processor, one or more color cameras to capture a color images of the query object; computing, by the processor, a three-dimensional (3D) model of the query object using the depth images; combining, by the processor, the color images with the 3D model; computing, by the processor, a descriptor from the 3D model and the color images, the descriptor including: a multi-dimensional shape descriptor space representation of a 3D shape of the query object; a multi-dimensional color descriptor space representation of a texture of the query object; and a one-dimensional size descriptor space representation of a size of query object; supplying, by the processor, the descriptor to a classifier to compute a classification of the query object; and outputting the classification of the query object.

First claim

What is claimed is:

1. A method for classifying physical objects comprising:
controlling, by a processor, one or more depth cameras to capture a plurality of depth images of a query object;
controlling, by the processor, one or more color cameras to capture a plurality of color images of the query object;
computing, by the processor, a three-dimensional (3D) model of the query object using the depth images;
combining, by the processor, the color images with the 3D model;
computing, by the processor, a descriptor from the 3D model and the color images, the descriptor comprising:
a multi-dimensional shape descriptor space representation of a 3D shape of the query object;
a multi-dimensional color descriptor space representation of a texture of the query object; and
a one-dimensional size descriptor space representation of a size of query object;

supplying, by the processor, the descriptor to a classifier to compute a classification of the query object; and
outputting the classification of the query object.

2. The method of claim 1 , further comprising controlling a conveyor system configured to convey the query object to redirect the query object in accordance with the classification of the query object.

3. The method of claim 1 , further comprising displaying the classification of the query object on a display device.

4. The method of claim 1 , wherein the classifier is a neural network.

5. The method of claim 4 , wherein the neural network is trained based on an inventory of objects.

6. The method of claim 5 , wherein the computing the classification of the query object based on the descriptor is performed by identifying a result object from the inventory of objects having a closest distance to the descriptor of the query object in shape descriptor space, color descriptor space, and size descriptor space.

7. The method of claim 1 , wherein the 3D model comprises a 3D mesh model computed from the depth images.

8. The method of claim 7 , further comprising:
rendering a plurality of two-dimensional (2D) views of the 3D mesh model; and
computing the descriptor by supplying the 2D views to a convolutional stage of a neural network.

9. A system for classifying physical objects comprising:
a processor; and
memory storing instructions that, when executed by the processor, cause the processor to:
control one or more depth cameras to capture a plurality of depth images of a query object;
control one or more color cameras to capture a plurality of color images of the query object;
compute a three-dimensional (3D) model of the query object using the depth images;
combine the color images with the 3D model;
compute a descriptor from the 3D model and the color images, the descriptor comprising:
a multi-dimensional shape descriptor space representation of a 3D shape of the query object;
a multi-dimensional color descriptor space representation of a texture of the query object; and
a one-dimensional size descriptor space representation of a size of query object;

supply the descriptor to a classifier to compute a classification of the query object; and
output the classification of the query object.

10. The system of claim 9 , further comprising a conveyor system configured to convey the query object,
wherein the memory further stores instructions that, when executed by the processor, cause the processor to redirect the query object in accordance with the classification of the query object.

11. The system of claim 9 , further comprising a display device,
wherein the memory further stores instructions that, when executed by the processor, cause the processor to display the classification of the query object on the display device.

12. The system of claim 9 , wherein the classifier is a neural network.

13. The system of claim 12 , wherein the neural network is trained based on an inventory of objects.

14. The system of claim 13 , wherein the memory further stores instructions that, when executed by the processor, cause the processor to compute the classification of the query object by identifying a result object from the inventory of objects having a closest distance to the descriptor of the query object in shape descriptor space, color descriptor space, and size descriptor space.

15. The system of claim 9 , wherein the 3D model comprises a 3D mesh model computed from the depth images.

16. The system of claim 15 , wherein the memory further stores instructions that, when executed by the processor, cause the processor to compute the descriptor by:
rendering a plurality of two-dimensional (2D) views of the 3D mesh model; and
computing the descriptor by supplying the 2D views to a convolutional stage of a neural network.

Assignees

• Aquifi Inc

Inventors

• Dal Mutto Carlo
• Memo Alvise

Classifications

• G06V10/764 Primary

  using classification, e.g. of video objects · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06V20/64

  Three-dimensional [3D] objects · CPC title

• G06T7/90

  Determination of colour characteristics · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title