Direct thin boundary prediction

Patent metadata

Field	Value
Publication number	US-10977798-B2
Application number	US-201916545158-A
Country	US
Kind code	B2
Filing date	Aug 20, 2019
Priority date	Aug 24, 2018
Publication date	Apr 13, 2021
Grant date	Apr 13, 2021

Abstract

In some implementations a neural network is trained to perform to directly predict thin boundaries of objects in images based on image characteristics. A neural network can be trained to predict thin boundaries of objects without requiring subsequent computations to reduce the thickness of the boundary prediction. Instead, the network is trained to make the predicted boundaries thin by effectively suppressing non-maximum values in normal directions along what might otherwise be a thick predicted boundary. To do so, the neural network can be trained to determine normal directions and suppress non-maximum values based on those determined normal directions.

First claim

What is claimed is:

1. A method, comprising:
at an electronic device having a processor:
obtaining training inputs identifying boundaries in ground truth image data;
training a neural network to determine boundaries in images using the training inputs and a loss function, the neural network trained to determine the boundaries, determine normal directions, and limit boundary thickness based on the normal directions; and
integrating the neural network into an application stored on a non-transitory computer-readable medium.

2. The method of claim 1 , wherein the loss function penalizes boundary inaccuracy by penalizing deviation from a boundary identified in the ground truth image data.

3. The method of claim 2 , wherein the deviation from the boundary identified in the ground truth data corresponds to inaccuracy in limiting boundary thickness.

4. The method of claim 3 wherein the inaccuracy in limiting boundary thickness corresponds to inaccuracy in predicting the normal directions.

5. The method of claim 1 , wherein training the neural network to limit boundary thickness comprises:
training the neural network to perform non-maxima suppression; or
training the neural network to identify pixels on either side of boundary pixels in the predicted normal directions by comparing the boundary pixels with the identified pixels on either side and suppressing non-maximum pixels based on the comparisons.

6. The method of claim 5 , wherein the pixels on either side of the boundary pixels are adjacent in the determined normal direction.

7. The method of claim 1 , wherein the training inputs comprise images having manually identified boundaries between objects in the images.

8. The method of claim 1 , wherein only boundaries are labeled in the training inputs.

9. The method of claim 1 further comprising obtaining an image and determining boundaries within the image using the neural network.

10. The method of claim 9 further comprising displaying the image and the determined boundaries.

11. The method of claim 9 further comprising detecting an object in the image based on the determined boundaries.

12. The method of claim 9 further comprising performing visual inertial odometry (VIO) or simultaneous localization and mapping (SLAM) based on the determined boundaries.

13. The method of claim 9 further comprising predicting a collision based on the determined boundaries.

14. The method of claim 1 , wherein the neural network is trained to output boundaries with sub-pixel precision.

15. The method of claim 1 , wherein the neural network is trained to output boundaries with sub-pixel precision by fitting a parabola to intensities of a boundary pixel and pixels on either side of the boundary pixel based on a determined normal direction.

16. A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
obtaining training inputs identifying boundaries in ground truth image data;
training a neural network to determine boundaries in images using the training inputs and a loss function, the neural network trained to determine the boundaries, determine normal directions, and limit boundary thickness based on the normal directions; and
integrating the neural network into an application stored on a non-transitory computer-readable medium.

17. The system of claim 16 , wherein the loss function penalizes boundary inaccuracy by penalizing deviation from a boundary identified in the ground truth image data.

18. The system of claim 17 , wherein the deviation from the boundary identified in the ground truth data corresponds to inaccuracy in limiting boundary thickness.

19. The system of claim 18 , wherein the inaccuracy in limiting boundary thickness corresponds to inaccuracy in predicting the normal directions.

20. A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising:
obtaining training inputs identifying boundaries in ground truth image data;
training a neural network to determine boundaries in images using the training inputs and a loss function, the neural network trained to determine the boundaries, determine normal directions, and limit boundary thickness based on the normal directions; and
integrating the neural network into an application stored on a non-transitory computer-readable medium.

Assignees

• Apple Inc

Inventors

• Kc Amit Kumar
• Ulbricht Daniel

Classifications

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06T7/13 Primary

  Edge detection · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title