Object recognition neural network for amodal center prediction

Patent metadata

Field	Value
Publication number	US-11989900-B2
Application number	US-202117357118-A
Country	US
Kind code	B2
Filing date	Jun 24, 2021
Priority date	Jun 24, 2020
Publication date	May 21, 2024
Grant date	May 21, 2024

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for object recognition neural network for amodal center prediction. One of the methods includes receiving an image of an object captured by a camera. The image of the object is processed using an object recognition neural network that is configured to generate an object recognition output. The object recognition output includes data defining a predicted two-dimensional amodal center of the object, wherein the predicted two-dimensional amodal center of the object is a projection of a predicted three-dimensional center of the object under a camera pose of the camera that captured the image.

First claim

What is claimed is:

1. A computer-implemented method, the method comprising:
receiving an image of an object captured by a camera;
processing the image of the object using an object recognition neural network that is configured to generate an object recognition output comprising:
data defining a predicted two-dimensional amodal center of the object, wherein the predicted two-dimensional amodal center of the object is a projection of a predicted three-dimensional center of the object under a camera pose of the camera that captured the image;

obtaining data specifying one or more other predicted two-dimensional amodal centers of the object in one or more other images captured under different camera poses; and
determining, from (i) the predicted two-dimensional amodal center of the object in the image and (ii) the one or more other predicted two-dimensional amodal centers of the object, the predicted three-dimensional center of the object.

2. The method of claim 1 , wherein the object recognition output comprises pixel coordinates of the predicted two-dimensional amodal center.

3. The method of claim 2 , wherein the object recognition neural network comprises a regression output layer that generates the pixel coordinates of the predicted two-dimensional amodal center.

4. The method of claim 1 , wherein the object recognition neural network is a multi-task neural network and the object recognition output also comprises data defining a bounding box for the object in the image.

5. The method of claim 4 , wherein the predicted two-dimensional amodal center is outside of the bounding box in the image.

6. The method of claim 1 , wherein the object recognition output comprises a truncation score that represents a likelihood that the object is truncated in the image.

7. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
receiving an image of an object captured by a camera;
processing the image of the object using an object recognition neural network that is configured to generate an object recognition output comprising:
data defining a predicted two-dimensional amodal center of the object, wherein the predicted two-dimensional amodal center of the object is a projection of a predicted three-dimensional center of the object under a camera pose of the camera that captured the image;

obtaining data specifying one or more other predicted two-dimensional amodal centers of the object in one or more other images captured under different camera poses; and
determining, from (i) the predicted two-dimensional amodal center of the object in the image and (ii) the one or more other predicted two-dimensional amodal centers of the object, the predicted three-dimensional center of the object.

8. The system of claim 7 , wherein the object recognition output comprises pixel coordinates of the predicted two-dimensional amodal center.

9. The system of claim 8 , wherein the object recognition neural network comprises a regression output layer that generates the pixel coordinates of the predicted two-dimensional amodal center.

10. The system of claim 7 , wherein the object recognition neural network is a multi-task neural network and the object recognition output also comprises data defining a bounding box for the object in the image.

11. The system of claim 10 , wherein the predicted two-dimensional amodal center is outside of the bounding box in the image.

12. The system of claim 7 , wherein the object recognition output comprises a truncation score that represents a likelihood that the object is truncated in the image.

13. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving an image of an object captured by a camera;
processing the image of the object using an object recognition neural network that is configured to generate an object recognition output comprising:
data defining a predicted two-dimensional amodal center of the object, wherein the predicted two-dimensional amodal center of the object is a projection of a predicted three-dimensional center of the object under a camera pose of the camera that captured the image;

obtaining data specifying one or more other predicted two-dimensional amodal centers of the object in one or more other images captured under different camera poses; and
determining, from (i) the predicted two-dimensional amodal center of the object in the image and (ii) the one or more other predicted two-dimensional amodal centers of the object, the predicted three-dimensional center of the object.

14. The computer-readable storage media of claim 13 , wherein the object recognition output comprises pixel coordinates of the predicted two-dimensional amodal center.

15. The computer-readable storage media of claim 14 , wherein the object recognition neural network comprises a regression output layer that generates the pixel coordinates of the predicted two-dimensional amodal center.

16. The computer-readable storage media of claim 13 , wherein the object recognition neural network is a multi-task neural network and the object recognition output also comprises data defining a bounding box for the object in the image.

17. The computer-readable storage media of claim 16 , wherein the predicted two-dimensional amodal center is outside of the bounding box in the image.

18. The computer-readable storage media of claim 13 , wherein the object recognition output comprises a truncation score that represents a likelihood that the object is truncated in the image.

Assignees

• Magic Leap Inc

Inventors

• Mahendran Siddharth
• Bansal Nitin
• Sekhar Nitesh
• Gangwar Manushree
• Gupta Khushi
• SINGHAL Prateek

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06T7/579 Primary

  from motion · CPC title

• G06F18/2163 Primary

  Partitioning the feature space · CPC title

• G06F18/24

  Classification techniques · CPC title