Scaled perspective zoom on resource constrained devices

Patent metadata

Field	Value
Publication number	US-11418704-B2
Application number	US-202016933314-A
Country	US
Kind code	B2
Filing date	Jul 20, 2020
Priority date	Jun 15, 2017
Publication date	Aug 16, 2022
Grant date	Aug 16, 2022

Abstract

A dolly zoom effect can be applied to one or more images captured via a resource-constrained device (e.g., a mobile smartphone) by manipulating the size of a target feature while the background in the one or more images changes due to physical movement of the resource-constrained device. The target feature can be detected using facial recognition or shape detection techniques. The target feature can be resized before the size is manipulated as the background changes (e.g., changes perspective).

First claim

What is claimed is:

1. A method comprising:
generating an image using an image sensor of a user device,
generating, using a convolutional neural network, image feature areas for different image features in the image;
identifying, on the user device, a target image feature from one of the image feature areas, the target image feature corresponding to a physical object depicted in the image;
resizing the target image feature;
generating a zoom video sequence by maintaining the scaling of the resized target image feature in the zoom video sequence without maintaining the scaling of image features areas that are not the target image feature such that the resized target image feature covers the depiction of the physical object in the zoom video sequence as the depiction of the physical object diminishes in size due to the user device moving closer to or away from the physical object; and
storing the zoom video sequence on the user device.

2. The method of claim 1 , further comprising:
receiving, through a touchscreen of the user device, an instruction to change the target image feature from an initial size to the size that is manipulated by increased scaling in the zoom video sequence.

3. The method of claim 1 , wherein the target image feature is identified in response to a user input received through a touchscreen of the user device.

4. The method of claim 1 , wherein generating the target image feature comprises applying, by the user device, the convolutional neural network to the image to identify the target image feature.

5. The method of claim 4 , wherein the target image feature is a human face.

6. The method of claim 4 , wherein the target image feature includes one or more segments of a human body.

7. The method of claim 4 , wherein the target image feature is a shape in the one or more images.

8. The method of claim 7 , further comprising: receiving selection of the shape through an input made through a touchscreen of the user device.

9. The method of claim 1 , further comprising:
stabilizing the target image feature in the zoom video sequence such that the target image feature remains in an initial area in the zoom video sequence.

10. The method of claim 1 , further comprising:
stabilizing the target image feature in the zoom video sequence such that the target image feature remains in an initial area in the zoom video sequence as the user device moves away from the physical object.

11. The method of claim 1 , wherein the image is from a live video feed generated by the image sensor of the user device.

12. The method of claim 1 , further comprising: transmitting the zoom video sequence to a network server.

13. A user device comprising:
one or more processors;
an image sensor; and
a memory storing instructions that, when executed by the one or more processors, cause the user device to perform operations comprising:
generating an image using the image sensor;
generating, using a convolutional neural network, image feature areas for different image features in the image;
identifying, on the user device, a target image feature from one of the image feature areas, the target image feature corresponding to a physical object depicted in the image;
resizing the target image feature;
generating a zoom video sequence by maintaining the scaling of the resized target image feature in the zoom video sequence without maintaining the scaling of image features areas that are not the target image feature such that the resized target image feature covers the depiction of the physical object in the zoom video sequence as the depiction of the physical object diminishes in size due to the user device moving closer to or away from the physical object; and
storing the zoom video sequence.

14. The user device of claim 13 , the operations further comprising:
receiving, through a touchscreen of the user device, an instruction to change the target image feature from an initial size to the size that is manipulated by increased scaling in the zoom video sequence.

15. The user device of claim 13 , wherein the target image feature is identified in response to a user input received through a touchscreen of the user device.

16. The user device of claim 13 , wherein generating the target image feature comprises applying, by the user device, the convolutional neural network to the image to identify the target image feature.

17. The user device of claim 16 , wherein the target image feature is a human face.

18. The user device of claim 16 , wherein the target image feature includes one or more segments of a human body.

19. The user device of claim 16 , wherein the target image feature is a shape in the one or more images.

20. A non-transitory machine-readable storage device embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
generating an image using the image sensor;
generating, using a convolutional neural network, image feature areas for different image features in the image;
identifying, on the user device, a target image feature from one of the image feature areas, the target image feature corresponding to a physical object depicted in the image;

resizing the target image feature;
generating a zoom video sequence by maintaining the scaling of the resized target image feature in the zoom video sequence without maintaining the scaling of image features areas that are not the target image feature such that the resized target image feature covers the depiction of the physical object in the zoom video sequence as the depiction of the physical object diminishes in size due to the user device moving closer to or away from the physical object; and
storing the zoom video sequence.

Assignees

• Snap Inc

Inventors

• Luo Linjie
• Ma Chongyang
• Xue Zehao

Classifications

• H04N23/62 Primary

  Control of parameters via user interfaces · CPC title

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/611

  where the recognised objects include parts of the human body · CPC title

• H04N23/63

  by using electronic viewfinders · CPC title

• H04N23/69

  Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming · CPC title