Active image depth prediction

Patent metadata

Field	Value
Publication number	US-11276190-B2
Application number	US-202016859468-A
Country	US
Kind code	B2
Filing date	Apr 27, 2020
Priority date	Aug 31, 2018
Publication date	Mar 15, 2022
Grant date	Mar 15, 2022

Abstract

An active depth detection system can generate a depth map from an image and user interaction data, such as a pair of clicks. The active depth detection system can be implemented as a recurrent neural network that can receive the user interaction data as runtime inputs after training. The active depth detection system can store the generated depth map for further processing, such as image manipulation or real-world object detection.

First claim

What is claimed is:

1. A method comprising:
storing, on a user device, a first neural network and a second neural network, the first neural network in use outputting into the second neural network;
identifying, on the user device, an image depicting an environment;
receiving, by the user device, an ordinal pair indicating a direction of depth in the environment depicted in the image;
generating, on the user device, an initial depth map from the image using the first neural network;
generating, on the user device, an updated depth map by inputting the received ordinal pair and the initial depth map into the second neural network; and
storing the updated depth map.

2. The method of claim 1 , further comprising:
generating a modified image by modifying the image using the updated depth map.

3. The method of claim 1 wherein the second neural network is a recurrent neural network.

4. The method of claim 3 , wherein the recurrent neural network is trained to implement an alternating direction method of multipliers scheme.

5. The method of claim 3 , wherein the first neural network is a trained neural network and the second neural network is configured to receive the ordinal pair as constraints after training of the trained neural network.

6. The method of claim 1 , further comprising:
receiving, by the user device, at least one additional ordinal pair, each additional ordinal pair indicating an additional direction of depth in the environment depicted in the image.

7. The method of claim 1 , further comprising:
generating, on the user device, the image using an image sensor of the user device; and
displaying the generated image on a display device of the user device.

8. The method of claim 7 , wherein receiving the ordinal pair comprises receiving a first point on the image and a second point on the image while the generated image is displayed on the display device.

9. The method of claim 2 , further comprising:
identifying, using the updated depth map, a background area of the image.

10. The method of claim 9 , wherein the modified image is generated by applying an image effect to the background area of the image.

11. A system comprising:
one or more processors of a machine; and
a memory storing instructions that, when executed by at least one processor among the one or more processors, cause the machine to perform operations comprising:
storing, in the memory, a first neural network and a second neural network, the first neural network in use outputting into the second neural network;
identifying, in the memory, an image depicting an environment;
receiving an ordinal pair indicating a direction of depth in the environment depicted in the image;
generating an initial depth map from the image using the first neural network;
generating an updated depth map by inputting the received ordinal pair and the initial depth map into the second neural network; and
storing the updated depth map in the memory.

12. The system of claim 11 , wherein the operations further comprise:
generating a modified image by modifying the image using the updated depth map.

13. The system of claim 11 wherein the first neural network is a trained neural network and the second neural network is configured to receive the ordinal pair as constraints after training of the trained neural network.

14. The system of claim 11 , wherein the system comprises a display device, and wherein receiving the ordinal pair comprises receiving a first point on the image and a second point on the image while the image is displayed on the display device.

15. A method comprising:
identifying, on a user device, an image depicting an environment;
receiving, by the user device, an ordinal pair indicating a direction of depth in the environment depicted in the image;
generating a depth map by inputting the received ordinal pair into a depth engine running on the user device; and
storing the depth map.

16. The method of claim 15 , wherein the depth engine comprises a neural network.

17. The method of claim 15 , further comprising:
generating a modified image by modifying the image using the depth map; and
displaying the modified image on a display device of the user device.

18. The method of claim 15 , further comprising:
receiving, by the user device, at least one additional ordinal pair, each additional ordinal pair indicating an additional direction of depth in the environment depicted in the image.

19. The method of claim 15 , further comprising:
generating, on the user device, the image using an image sensor of the user device; and
displaying the generated image on a display device of the user device, wherein receiving the ordinal pair comprises receiving a first point on the image and a second point on the image while the image is displayed on the display device, and wherein the order of receipt of the first and second points indicates a relative depth order between the first and second points in the depth direction.

20. The method of claim 15 , further comprising:
identifying, using the depth map, a background area of the image; and
generating a modified image by applying an image effect to the background area of the image.

Assignees

• Snap Inc

Inventors

• Duan Kun
• Ron Daniel
• Ma Chongyang
• Xu Ning
• Wang Shenlong
• Hanumante Sumant Milind
• Sagar Dhritiman

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/08

  Learning methods · CPC title

• G06T2207/20081

  Training; Learning · CPC title