Virtual lens simulation for video and photo cropping

Patent metadata

Field	Value
Publication number	US-10529051-B2
Application number	US-201916535911-A
Country	US
Kind code	B2
Filing date	Aug 8, 2019
Priority date	May 20, 2015
Publication date	Jan 7, 2020
Grant date	Jan 7, 2020

Abstract

In a video capture system, a virtual lens is simulated when applying a crop or zoom effect to an input video. An input video frame is received from the input video that has a first field of view and an input lens distortion caused by a lens used to capture the input video frame. A selection of a sub-frame representing a portion of the input video frame is obtained that has a second field of view smaller than the first field of view. The sub-frame is processed to remap the input lens distortion to a desired lens distortion in the sub-frame. The processed sub-frame is the outputted.

First claim

The invention claimed is:

1. A camera that simulates image distortion of a virtual lens in an image, the camera comprising:
a lens configured to cause an input lens distortion within an input image;
an image sensor configured to capture the input image, the input image including a first field of view of a scene, the input image depicting the scene with the input lens distortion centered in the first field of view;
one or more processors; and
a non-transitory computer-readable storage medium storing instructions that when executed cause the one or more processors to perform steps including:
selecting a sub-frame representing a portion of the input image, the sub-frame including a second field of view of the scene smaller than the first field of view, the sub-frame including a lens distortion effect as a function of the input lens distortion present in the input image, a position of an area of the input image included within the second field of view, and a size of the area of the input image included within the second field of view; and
generating an output image based on a desired lens distortion and the lens distortion effect in the sub-frame, the desired lens distortion and the input lens distortion exhibiting consistent lens characteristics, wherein the output image includes the sub-frame remapped from the input lens distortion centered in the first field of view to the desired lens distortion centered in the second field of view to transform the lens distortion effect present in the sub-frame to the desired lens distortion such that a portion of the scene depicted in the sub-frame appears to have been captured using the second field of view.

2. The camera of claim 1 , wherein the lens includes a wide angle lens.

3. The camera of claim 2 , wherein lens characteristics of the wide angle lens causes straight lines in the scene to appear as curved lines near an edge of the input image.

4. The camera of claim 3 , wherein generating the output image comprises removing curvature in an area of the sub-frame corresponding to the edge of the input image.

5. The camera of claim 1 , wherein the one or more processors include an image processor configured to perform image stabilization.

6. The camera of claim 5 , wherein the image stabilization is enabled through a user interface of the camera.

7. The camera of claim 1 , wherein:
the camera further comprises a sensor configured to generate metadata associated with capture of the input image; and
the sub-frame is selected based on the metadata associated with the capture of the input image.

8. The camera of claim 7 , wherein the sensor includes an orientation sensor, an accelerometer, a gyroscope, or a magnetometer.

9. The camera of claim 7 , wherein the metadata indicates motion of the camera.

10. The camera of claim 7 , wherein the metadata indicates orientation of the camera.

11. A method for simulating image distortion of a virtual lens in an image, the method performed by a camera comprising a lens, an image sensor, and one or more processors, the lens configured to cause an input lens distortion within an input image, the image sensor configured to capture the input image including a first field of view of a scene, the input image depicting the scene with the input lens distortion centered in the first field of view, the method comprising:
selecting a sub-frame representing a portion of the input image, the sub-frame including a second field of view of the scene smaller than the first field of view, the sub-frame including a lens distortion effect as a function of the input lens distortion present in the input image, a position of an area of the input image included within the second field of view, and a size of the area of the input image included within the second field of view; and
generating an output image based on a desired lens distortion and the lens distortion effect in the sub-frame, the desired lens distortion and the input lens distortion exhibiting consistent lens characteristics, wherein the output image includes the sub-frame remapped from the input lens distortion centered in the first field of view to the desired lens distortion centered in the second field of view to transform the lens distortion effect present in the sub-frame to the desired lens distortion such that a portion of the scene depicted in the sub-frame appears to have been captured using the second field of view.

12. The method of claim 11 , wherein the lens includes a wide angle lens.

13. The method of claim 12 , wherein lens characteristics of the wide angle lens causes straight lines in the scene to appear as curved lines near an edge of the input image.

14. The method of claim 13 , wherein generating the output image comprises removing curvature in an area of the sub-frame corresponding to the edge of the input image.

15. The method of claim 11 , wherein the one or more processors include an image processor configured to perform image stabilization.

16. The method of claim 15 , wherein the image stabilization is enabled through a user interface of the camera.

17. The method of claim 11 , wherein:
the camera further comprises a sensor configured to generate metadata associated with capture of the input image; and
the sub-frame is selected based on the metadata associated with the capture of the input image.

18. The method of claim 17 , wherein the sensor includes an orientation sensor, an accelerometer, a gyroscope, or a magnetometer.

19. The method of claim 17 , wherein the metadata indicates motion of the camera.

20. The method of claim 17 , wherein the metadata indicates orientation of the camera.

Assignees

• Gopro Inc

Inventors

• Newman David A
• Bodinet Joshua Edward
• Sievert Otto Kenneth
• Macmillan Timothy

Classifications

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T3/00

  Geometric image transformations in the plane of the image · CPC title

• G06T2210/22

  Cropping · CPC title

• G06T2207/10004

  Still image; Photographic image · CPC title