Scaled perspective zoom on resource constrained devices

Patent metadata

Field	Value
Publication number	US-12238404-B2
Application number	US-202217816223-A
Country	US
Kind code	B2
Filing date	Jul 29, 2022
Priority date	Jun 15, 2017
Publication date	Feb 25, 2025
Grant date	Feb 25, 2025

Abstract

A dolly zoom effect can be applied to one or more images captured via a resource-constrained device (e.g., a mobile smartphone) by manipulating the size of a target feature while the background in the one or more images changes due to physical movement of the resource-constrained device. The target feature can be detected using facial recognition or shape detection techniques. The target feature can be resized before the size is manipulated as the background changes (e.g., changes perspective).

First claim

What is claimed is:

1. A method comprising:
generating an image using an image sensor of a user device;
receiving, at the user device, a user selection indicating a depiction of a physical object in the image;
generating a zoom video sequence by maintaining a scaling of only the depiction of the physical object in the zoom video sequence as the user device moves closer to or away from the physical object;
receiving, through a touchscreen of the user device, an instruction to change the depiction of the physical object from an initial size to a size that is manipulated by increased scaling in the zoom video sequence; and
storing the zoom video sequence on the user device.

2. The method of claim 1 , wherein the image comprises a first region including the depiction of the physical object, and a second region including a remaining area of the image excluding the first region.

3. The method of claim 2 , wherein generating the zoom video sequence comprises:
maintaining the scaling of the first region as the user device moves closer to or away from the physical object; and
adjusting the scaling of the second region as the user device moves closes to or away from the physical object.

4. The method of claim 2 , further comprising:
resizing the first region;
maintaining the scaling of the resized first region as the user device moves closer to or away from the physical object; and
adjusting the scaling of the second region as the user device moves closes to or away from the physical object.

5. The method of claim 1 , further comprising:
generating, using a convolutional neural network, image feature areas for different image features in the image;
identifying, on the user device, a target image feature from one of the image feature areas, the target image feature corresponding to the physical object depicted in the image; and
resizing the target image feature,
wherein generating the zoom video sequence comprises:
maintaining a scaling of the resized target image feature in the zoom video sequence without maintaining the scaling of image features areas that are not the target image feature such that the resized target image feature covers the depiction of the physical object in the zoom video sequence as the user device moves closer to or away from the physical object.

6. The method of claim 1 , wherein the physical object is one of a human face, a segment of a human body, or a shape in the image.

7. The method of claim 1 , further comprising:
stabilizing the depiction of the physical object in the zoom video sequence such that the depiction of the physical object remains in an initial area in the zoom video sequence.

8. The method of claim 1 , wherein the image is from a live video feed generated by a front-facing camera of the user device, the user device having the front-facing camera and a rear-facing camera.

9. The method of claim 1 , further comprising:
generating an ephemeral message comprising the zoom video sequence; and
transmitting the ephemeral message to a network server.

10. A user device comprising:
one or more processors;
an image sensor; and
a memory storing instructions that, when executed by the one or more processors, cause the user device to perform operations comprising:
generating an image using the image sensor;
receiving, at the user device, a user selection indicating a depiction of a physical object in the image;
generating a zoom video sequence by maintaining a scaling of only the depiction of the physical object in the zoom video sequence as the user device moves closer to or away from the physical object;
receiving, through a touchscreen of the user device, an instruction to change the depiction of the physical object from an initial size to a size that is manipulated by increased scaling in the zoom video sequence; and
storing the zoom video sequence on the user device.

11. The user device of claim 10 , wherein the image comprises a first region including the depiction of the physical object, and a second region including a remaining area of the image excluding the first region.

12. The user device of claim 11 , wherein generating the zoom video sequence comprises:
maintaining the scaling of the first region as the user device moves closer to or away from the physical object; and
adjusting the scaling of the second region as the user device moves closes to or away from the physical object.

13. The user device of claim 11 , wherein the operations further comprise:
resizing the first region;
maintaining the scaling of the resized first region as the user device moves closer to or away from the physical object; and
adjusting the scaling of the second region as the user device moves closes to or away from the physical object.

14. The user device of claim 10 , wherein the operations further comprise:
generating, using a convolutional neural network, image feature areas for different image features in the image;
identifying, on the user device, a target image feature from one of the image feature areas, the target image feature corresponding to the physical object depicted in the image; and
resizing the target image feature,
wherein generating the zoom video sequence comprises:
maintaining a scaling of the resized target image feature in the zoom video sequence without maintaining the scaling of image features areas that are not the target image feature such that the resized target image feature covers the depiction of the physical object in the zoom video sequence as the user device moves closer to or away from the physical object.

15. The user device of claim 10 , wherein the operations further comprise:
receiving, through a touchscreen of the user device, an instruction to change the depiction of the physical object from an initial size to a size that is manipulated by increased scaling in the zoom video sequence.

16. The user device of claim 10 , wherein the physical object is one of a human face, a segment of a human body, or a shape in the image.

17. The user device of claim 10 , further comprising:
stabilizing the depiction of the physical object in the zoom video sequence such that the depiction of the physical object remains in an initial area in the zoom video sequence.

18. The user device of claim 10 , wherein the image is from a live video feed generated by the image sensor, the image sensor comprising a front-facing camera of the user device,
wherein the operations further comprise:
generating an ephemeral message comprising the zoom video sequence; and
transmitting the ephemeral message to a network server.

19. A non-transitory machine-readable storage device embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
generating an image using an image sensor of a user device;
receiving, at the user device, a user selection indicating a depiction of a physical object in the image;
generating a zoom video sequence by maintaining a scaling of only the depiction of the physical object in the zoom video sequence as the user device moves closer to or away from the physical object;
receiving, through a touchscreen of the user device, an instruction to change the depiction of the physical object from an initial size to a size that is manipulated by increased scaling in the zoom video sequence; and
storing the zoom video sequence on the user device.

Assignees

• Snap Inc

Inventors

• Luo Linjie
• Ma Chongyang
• Xue Zehao

Classifications

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/611

  where the recognised objects include parts of the human body · CPC title

• H04N23/63

  by using electronic viewfinders · CPC title

• H04N5/2628 Primary

  Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation · CPC title

• H04N5/272

  Means for inserting a foreground image in a background image, i.e. inlay, outlay · CPC title