Virtual lens simulation for video and photo cropping

Patent metadata

Field	Value
Publication number	US-10186012-B2
Application number	US-201615157207-A
Country	US
Kind code	B2
Filing date	May 17, 2016
Priority date	May 20, 2015
Publication date	Jan 22, 2019
Grant date	Jan 22, 2019

Abstract

In a video capture system, a virtual lens is simulated when applying a crop or zoom effect to an input video. An input video frame is received from the input video that has a first field of view and an input lens distortion caused by a lens used to capture the input video frame. A selection of a sub-frame representing a portion of the input video frame is obtained that has a second field of view smaller than the first field of view. The sub-frame is processed to remap the input lens distortion to a desired lens distortion in the sub-frame. The processed sub-frame is the outputted.

First claim

The invention claimed is:

1. A method for simulating a virtual lens when applying a crop or zoom effect to an input image, the method comprising:
receiving, by a processor, the input image, the input image having a first field of view of a scene, the input image depicting the scene with an input lens distortion centered in the first field of view, the input lens distortion caused by lens characteristics of a lens used to capture the input image, the lens centered at the first field of view during capture of the input image;
identifying, by the processor, the input lens distortion within the input image;
obtaining, by the processor, a selection of a sub-frame representing a portion of the input image, the sub-frame having a second field of view of the scene smaller than the first field of view;
determining, by the processor, an input lens distortion effect present in the sub-frame based on the input lens distortion centered in the first field of view, a location of the sub-frame within the first field of view, and a size of the second field of view; and
generating, by the processor, an output image based on the input lens distortion and the input lens distortion effect present in the sub-frame, the output image including the sub-frame remapped from the input lens distortion centered in the first field of view to the input lens distortion centered in the second field of view such that a portion of the scene depicted in the sub-frame appears to have been captured using the lens centered at the second field of view.

2. The method of claim 1 , wherein the input lens distortion caused by the lens characteristics causes straight lines in the scene to appear as curved lines in at least a portion of the input image.

3. The method of claim 1 , wherein the input lens distortion comprises a distortion produced by a conventional lens, a wide angle lens, a fisheye lens, a hemispherical lens, a zoom lens, or a flat lens.

4. The method of claim 1 , wherein obtaining the selection of the sub-frame representing the portion of the input image comprises:
automatically identifying the sub-frame based on metadata associated with the input image or image characteristics of the input image;
automatically obtaining a sub-frame center location, a sub-frame size, and a scaling factor for transforming the input image; and
applying the crop or zoom effect applied to the input image based on the sub-frame center location, the sub-frame size, and the scaling factor to generate the sub-frame.

5. The method of claim 1 , wherein obtaining the selection of the sub-frame representing the portion of the input image further comprises:
receiving a manual selection of the sub-frame from post-processing tools.

6. The method of claim 1 , wherein generating the output image comprises:
obtaining metadata associated with the input image, the metadata specifying the lens characteristics of the lens used to capture the input image; and
processing the sub-frame using the lens characteristics specified in the metadata.

7. The method of claim 1 , wherein generating the output image comprises:
applying an inverse input lens distortion function to the sub-frame to remove the input lens distortion effect present in the sub-frame to generate a rectilinear image, wherein straight lines in the scene within the sub-frame appear as straight lines in the rectilinear image; and
applying an input lens distortion function to apply the input lens distortion to the rectilinear image.

8. The method of claim 1 , wherein the sub-frame is remapped from the input lens distortion centered in the first field of view to the input lens distortion centered in the second field of view by applying a direct function to remap the sub-frame without an intermediate step of removing the input lens distortion effect present in the sub-frame.

9. A non-transitory computer-readable storage medium storing instructions for simulating a virtual lens when applying a crop or zoom effect to an input image, the instructions when executed by one or more processors causing the one or more processors to perform steps including:
receiving the input image, the input image having a first field of view of a scene, the input image depicting the scene with an input lens distortion centered in the first field of view, the input lens distortion caused by lens characteristics of a lens used to capture the input image, the lens centered at the first field of view during capture of the input image;
identifying the input lens distortion within the input image;
obtaining a selection of a sub-frame representing a portion of the input image, the sub-frame having a second field of view of the scene smaller than the first field of view;
determining an input lens distortion effect present in the sub-frame based on the input lens distortion centered in the first field of view, a location of the sub-frame within the first field of view, and a size of the second field of view; and
generating an output image based on the input lens distortion and the input lens distortion effect present in the sub-frame, the output image including the sub-frame remapped from the input lens distortion centered in the first field of view to the input lens distortion centered in the second field of view such that a portion of the scene depicted in the sub-frame appears to have been captured using the lens centered at the second field of view.

10. The non-transitory computer-readable storage medium of claim 9 , wherein the input lens distortion caused by the lens characteristics causes straight lines in the scene to appear as curved lines in at least a portion of the input image.

11. The non-transitory computer-readable storage medium of claim 9 , wherein the input lens distortion comprises a distortion produced by a conventional lens, a wide angle lens, a fisheye lens, a hemispherical lens, a zoom lens, or a flat lens.

12. The non-transitory computer-readable storage medium of claim 9 , wherein obtaining the selection of the sub-frame representing the portion of the input image comprises:
automatically identifying the sub-frame based on metadata associated with the input image or image characteristics of the input image;
automatically obtaining a sub-frame center location, a sub-frame size, and a scaling factor for transforming the input image; and
applying the crop or zoom effect applied to the input image based on the sub-frame center location, the sub-frame size, and the scaling factor to generate the sub-frame.

13. The non-transitory computer-readable storage medium of claim 9 , wherein obtaining the selection of the sub-frame representing the portion of the input image further comprises:
receiving a manual selection of the sub-frame from post-processing tools.

14. The non-transitory computer-readable storage medium of claim 9 , wherein generating the output image comprises:
obtaining metadata associated with the input image, the metadata specifying the lens characteristics of the lens used to capture the input image; and
processing the sub-frame using the lens characteristics specified in the metadata.

15. The non-transitory computer-readable storage medium of claim 9 , wherein generating the output image comprises:
applying an inverse input lens distortion function to the sub-frame to remove the input lens distortion effect present in the sub-frame to generate a rectilinear image, wherein straight lines in the scene within the sub-frame appear as straight lines in the rectilinear image; and
applying an input lens distortion function to apply the input lens distortion to the rectilinear image.

16

Assignees

• Gopro Inc

Inventors

• Newman David A
• Bodinet Joshua Edward
• Sievert Otto Kenneth
• Macmillan Timothy

Classifications

• G06T2210/22

  Cropping · CPC title

• G06T3/00

  Geometric image transformations in the plane of the image · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/10004

  Still image; Photographic image · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title