Bilateral convolution layer network for processing point clouds

Patent metadata

Field	Value
Publication number	US-11636668-B2
Application number	US-201815986267-A
Country	US
Kind code	B2
Filing date	May 22, 2018
Priority date	Nov 10, 2017
Publication date	Apr 25, 2023
Grant date	Apr 25, 2023

Abstract

A method includes filtering a point cloud transformation of a 3D object to generate a 3D lattice and processing the 3D lattice through a series of bilateral convolution networks (BCL), each BCL in the series having a lower lattice feature scale than a preceding BCL in the series. The output of each BCL in the series is concatenated to generate an intermediate 3D lattice. Further filtering of the intermediate 3D lattice generates a first prediction of features of the 3D object.

First claim

What is claimed is:

1. A machine-vision method comprising:
filtering a point cloud transformation of a three dimensional object to generate a three dimensional lattice;
processing the three dimensional lattice through a series arrangement of bilateral convolution artificial neural networks (BCL), each BCL in the series having a lower lattice feature scale than a preceding BCL in the series;
concatenating an output of each BCL in the series to generate an intermediate three dimensional lattice; and
further filtering the intermediate three dimensional lattice to generate a first prediction of features of the three dimensional object.

2. The method of claim 1 , wherein the filtering and further filtering are carried out using 1×1 convolution layers.

3. The method of claim 1 , further comprising:
projecting multiple two dimensional images onto the intermediate three dimensional lattice to generate a merged three dimensional lattice.

4. The method of claim 3 , further wherein projecting the two dimensional images comprises:
processing the two dimensional images through a first convolution neural network; and
applying an output of the first convolution neural network to a subsequent layer.

5. The method of claim 4 , further comprising:
transforming the merged three dimensional lattice into a second prediction of features of the three dimensional object using a series arrangement of convolution layers.

6. The method of claim 5 , wherein each of the convolution layers is a 1×1 convolution layer.

7. The method of claim 5 , further comprising:
back-projecting the second prediction of features of the three dimensional object onto the two dimensional images.

8. The method of claim 7 , further comprising:
concatenating the two dimensional images, the output of the first convolution neural network, and the second prediction of features of the three dimensional object into a merged projection; and
transforming the merged projection into a prediction of two dimensional object features.

9. A machine-vision system comprising:
at least one processor;
a memory coupled to the at least one processor, the memory configured with a point cloud representation of an object; and
logic to configure the at least one processor to:
process the point cloud representation through a series arrangement of bilateral convolution artificial neural networks (BCL), each BCL in the series having a lower feature scale than a preceding BCL in the series arrangement;
concatenate outputs of two or more of the BCLs in the series arrangement to generate an intermediate representation of the point cloud; and
process the intermediate representation to generate a first prediction of features of the object.

10. The system of claim 9 , further comprising logic to configure the at least one processor to:
process the intermediate representation using a lx 1 convolution layer.

11. The system of claim 9 , further comprising logic to configure the at least one processor to:
project multiple two dimensional images onto the intermediate representation to generate a three dimensional representation.

12. The system of claim 11 , further comprising logic to configure the at least one processor to:
process the two dimensional images through a first convolution neural network; and
apply an output of the first convolution neural network to a subsequent neural network.

13. The system of claim 12 , further comprising logic to configure the at least one processor to:
transform the three dimensional representation into a second prediction of features of the object using a series arrangement of convolution layers.

14. The system of claim 13 , wherein each of the convolution layers in the series arrangement of convolution layers is a 1×1 convolution layer.

15. The system of claim 13 , further comprising logic to configure the at least one processor to:
back-project the second prediction of features of the object onto the two dimensional images.

16. The system of claim 15 , further comprising logic to configure the at least one processor to:
concatenate the two dimensional images, the output of the first convolution neural network, and the second prediction of features of the object into a merged projection.

17. A machine-vision method comprising:
processing a point cloud lattice through a series arrangement of convolutional neural networks that utilize bilateral filtering;
concatenating outputs of two or more convolutional neural networks in the series arrangement to generate an intermediate lattice;
processing the intermediate lattice to predict features of the object; and
back-project the predicted features of the object onto two-dimensional images of the object.

18. The system of claim 17 , further comprising logic to configure the at least one processor to:
form the intermediate lattice in two dimensions; and
project the two-dimensional images onto the intermediate lattice to generate a three-dimensional lattice.

Assignees

• Nvidia Corp

Inventors

• Jampani Varun
• Su Hang
• Sun Deqing
• Yang Ming-Hsuan
• Kautz Jan

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06T7/11

  Region-based segmentation · CPC title