Active image depth prediction

Patent metadata

Field	Value
Publication number	US-10672136-B2
Application number	US-201816120105-A
Country	US
Kind code	B2
Filing date	Aug 31, 2018
Priority date	Aug 31, 2018
Publication date	Jun 2, 2020
Grant date	Jun 2, 2020

Abstract

An active depth detection system can generate a depth map from an image and user interaction data, such as a pair of clicks. The active depth detection system can be implemented as a recurrent neural network that can receive the user interaction data as runtime inputs after training. The active depth detection system can store the generated depth map for further processing, such as image manipulation or real-world object detection.

First claim

What is claimed is:

1. A method comprising:
storing, on a user device, a trained neural network comprising a convolutional neural network that outputs into a recurrent neural network;
identifying, on the user device, an image depicting an environment;
receiving, by the user device, an ordinal pair indicating a direction of depth in the environment depicted in the image;
generating, on the user device, an initial depth map from the image using the convolutional neural network;
generating, on the user device, an updated depth map by inputting the received ordinal pair into the recurrent neural network that is trained to update outputs of the convolutional neural network; and
storing the updated depth map.

2. The method of claim 1 , further comprising:
generating a modified image by modifying the image using the updated depth map.

3. The method of claim 1 , wherein the recurrent neural network is trained to implement an alternating direction method of multipliers (ADMM) scheme.

4. The method of claim 3 , wherein the ADMM scheme is configured to receive the ordinal pair as constraints after training of the trained neural network.

5. The method of claim 1 , further comprising:
receiving, by the user device, a plurality of point pairs, each point pair of the plurality of point pairs indicating an additional direction of depth in the environment depicted in the image.

6. The method of claim 1 , wherein the convolutional neural network and the recurrent neural network are trained end-to-end as a single network.

7. The method of claim 1 , further comprising:
generating, on the user device, the image using an image sensor of the user device.

8. The method of claim 7 , further comprising:
displaying the generated image on a display device of the user device.

9. The method of claim 8 , wherein receiving the ordinal pair comprises receiving a first point on the image and a second point on the image while the image is displayed on the display device.

10. The method of claim 2 , further comprising:
identifying, using the updated depth map, a background area of the image.

11. The method of claim 10 , wherein the modified image is generated by applying an image effect to the background area of the image.

12. The method of claim 2 , further comprising:
publishing the modified image as an ephemeral message on a network site.

13. A system comprising:
one or more processors of a machine; and
a memory storing instructions that, when executed by at least one processor among the one or more processors, cause the machine to perform operations comprising:
storing, in the memory, a trained neural network comprising a convolutional neural network that outputs into a recurrent neural network;
identifying, in the memory, an image depicting an environment;
receiving an ordinal pair indicating a direction of depth in the environment depicted in the image;
generating an initial depth map from the image using the convolutional neural network;
generating an updated depth map by inputting the received ordinal pair into trained the recurrent neural network that is trained to update outputs of the convolutional neural network; and
storing the updated depth map in the memory.

14. The system of claim 13 , wherein the operations further comprise:
generating a modified image by modifying the image using the updated depth map.

15. The system of claim 13 , wherein the trained recurrent neural network is trained to implement an alternating direction method of multipliers (ADMM) scheme.

16. The system of claim 15 , wherein the ADMM scheme is configured to receive the ordinal pair as constraints after training of the trained neural network.

17. The system of claim 13 , wherein the system comprises a display device, and wherein receiving the ordinal pair comprises receiving a first point on the image and a second point on the image while the image is displayed on the throe the display device.

18. A machine-readable storage device embodying instructions that, when executed by a device, cause the device to perform operations comprising:
storing a trained neural network comprising a convolutional neural network that outputs into a recurrent neural network;
identifying, on the device, an image depicting an environment;
receiving, by the device, an ordinal pair indicating a direction of depth in the environment depicted in the image;
generating an initial depth map from the image using the convolutional neural network;
generating, on the device, an updated depth map by inputting the received ordinal pair into the recurrent neural network that is trained to update outputs of the convolutional neural network; and
storing the updated depth map.

Assignees

• Duan Kun
• Ron Daniel
• Ma Chongyang
• Xu Ning
• Wang Shenlong
• Hanumante Sumant Milind
• Sagar Dhritiman
• Snap Inc

Inventors

• Duan Kun
• Ron Daniel
• Ma Chongyang
• Xu Ning
• Wang Shenlong
• Hanumante Sumant Milind
• Sagar Dhritiman

Classifications

• G06T11/60

  Creating or editing images; Combining images with text · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T7/536 Primary

  from perspective effects, e.g. by using vanishing points · CPC title