Active image depth prediction

Patent metadata

Field	Value
Publication number	US-11715223-B2
Application number	US-202217589459-A
Country	US
Kind code	B2
Filing date	Jan 31, 2022
Priority date	Aug 31, 2018
Publication date	Aug 1, 2023
Grant date	Aug 1, 2023

Abstract

An active depth detection system can generate a depth map from an image and user interaction data, such as a pair of clicks. The active depth detection system can be implemented as a recurrent neural network that can receive the user interaction data as runtime inputs after training. The active depth detection system can store the generated depth map for further processing, such as image manipulation or real-world object detection.

First claim

The invention claimed is:

1. A method, executed by one or more processors, the method comprising:
displaying, on a display, an image depicting an environment;
receiving user input on the display of a first location and a second location in the image, the first and second locations each comprising image coordinates, and the first and second locations indicating a direction of depth in the environment depicted in the image;
generating a depth map for the image based at least in part on the first and second locations; and
storing the depth map.

2. The method of claim 1 , wherein generating the depth map comprises:
generating an initial depth map from the image; and
generating a modified depth map from the initial depth map and the first and second locations.

3. The method of claim 2 , further comprising:
receiving user input of image coordinates for a third location and a fourth location in the image, the third and fourth locations each comprising image coordinates, and the third and fourth locations indicating a second direction of depth in the environment depicted in the image; and
refining the modified depth map using the third and fourth locations.

4. The method of claim 1 , further comprising:
generating a vector between the first location and the second location, the direction of the vector between the first location and the second location being defined by a sequence of user input of the first location and the second location.

5. The method of claim 1 , further comprising:
displaying an arrow connecting the first location and the second location, the direction of the arrow indicating the direction of depth in an area of the image between the first location and the second location.

6. The method of claim 1 , further comprising:
removing a background area in the image based on the depth map, to generate a modified image.

7. The method of claim 6 , further comprising:
applying an image effect to the removed background area of the image.

8. The method of claim 1 wherein the user input of the first location and the second location in the image is a swipe gesture on a touchscreen between the first location and the second location, the direction of depth being indicated by a direction of the swipe gesture.

9. A system comprising:
one or more processors of a machine; and
a memory storing instructions that, when executed by at least one processor among the one or more processors, cause the machine to perform operations comprising:
displaying, on a display, an image depicting an environment;
receiving user input on the display of a first location and a second location in the image, the first and second locations each comprising image coordinates, and the first and second locations indicating a direction of depth in the environment depicted in the image;
generating a depth map for the image based at least in part on the first and second locations; and
storing the depth map in the memory.

10. The system of claim 9 , wherein generating the depth map comprises:
generating an initial depth map from the image; and
generating a modified depth map from the initial depth map and the first and second locations.

11. The system of claim 10 , wherein the operations further comprise:
receiving user input of image coordinates for a third location and a fourth location in the image, the third and fourth locations each comprising image coordinates, and the third and fourth locations indicating a second direction of depth in the environment depicted in the image; and
refining the modified depth map using the third and fourth locations.

12. The system of claim 9 , wherein the operations further comprise:
generating a vector between the first location and the second location, the direction of the vector between the first location and the second location being defined by a sequence of user input of the first location and the second location.

13. The system of claim 9 wherein the operations further comprise:
displaying an arrow connecting the first location and the second location, the direction of the arrow indicating the direction of depth in an area of the image between the first location and the second location.

14. The system of claim 9 , wherein the operations further comprise:
removing a background area in the image based on the depth map, to generate a modified image.

15. A non-transitory machine-readable medium, the machine-readable medium including instructions that when executed by a computer, cause the computer to perform operations comprising:
displaying, on a display, an image depicting an environment;
receiving user input on the display of a first location and a second location in the image, the first and second locations each comprising image coordinates, and the first and second locations indicating a direction of depth in the environment depicted in the image;
generating a depth map for the image based at least in part on the first and second locations; and
storing the depth map.

16. The machine-readable medium of claim 15 , wherein the operations further comprise:
generating an initial depth map from the image; and
generating a modified depth map from the initial depth map and the first and second locations.

17. The machine-readable medium of claim 16 , wherein the operations further comprise:
receiving user input of image coordinates for a third location and a fourth location in the image, the third and fourth locations each comprising image coordinates, and the third and fourth locations indicating a second direction of depth in the environment depicted in the image; and
refining the modified depth map using the third and fourth locations.

18. The machine-readable medium of claim 15 , wherein the operations further comprise:
generating a vector between the first location and the second location, the direction of the vector between the first location and the second location being defined by a sequence of user input of the first location and the second location.

19. The machine-readable medium of claim 15 , wherein the operations further comprise:
displaying an arrow connecting the first location and the second location, the direction of the arrow indicating the direction of depth in an area of the image between the first location and the second location.

20. The machine-readable medium of claim 15 , wherein the operations further comprise:
removing a background area in the image based on the depth map, to generate a modified image; and
applying an image effect to the removed background area of the image.

Assignees

• Snap Inc

Inventors

• Duan Kun
• Ron Daniel
• Ma Chongyang
• Xu Ning
• Wang Shenlong
• Hanumante Sumant Milind
• Sagar Dhritiman

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/08

  Learning methods · CPC title

• G06T7/536 Primary

  from perspective effects, e.g. by using vanishing points · CPC title