Image/video editor with automatic occlusion detection and cropping

Patent metadata

Field	Value
Publication number	US-10366497-B2
Application number	US-201715618967-A
Country	US
Kind code	B2
Filing date	Jun 9, 2017
Priority date	Jun 10, 2016
Publication date	Jul 30, 2019
Grant date	Jul 30, 2019

Abstract

Techniques for cropping images containing an occlusion are presented. A method for image editing is presented comprising, when an occlusion is detected in an original digital image, determining an area occupied by the occlusion, assigning importance scores to different content elements of the original digital image, defining a cropping window around an area of the original digital image that does not include the area occupied by the occlusion and that is based on the importance scores, and cropping the original digital image to the cropping window.

First claim

We claim:

1. A method for image editing, comprising:
when an occlusion is detected in an original digital image, determining an area occupied by the occlusion;
rotating the original digital image to align a detected horizon to a horizontal axis;
assigning importance scores to different content elements of the rotated digital image;
defining a cropping window around an area of the rotated digital image that does not include the area occupied by the occlusion and that is based on the importance scores; and
cropping the rotated digital image to the cropping window.

2. The method of claim 1 , wherein the original digital image was captured by a first camera, and further comprising:
detecting the occlusion based at least in part on a data from a second camera.

3. The method of claim 1 , wherein the content elements include at least one region of interest and one non-region of interest and the importance scores are different for portions of image data within the region of interest and portions of image data within the non-region of interest.

4. The method of claim 3 , wherein the cropping window is sized to retain proportionality between the region of interest's relative position within the original digital image and the region of interest's relative position within an image generated from the cropping window.

5. The method of claim 3 , wherein the cropping window is sized to include an entire size of the region of interest and possesses an aspect ratio that matches an aspect ratio of the original digital image.

6. The method of claim 1 , wherein a region of interest is centered in the cropping window.

7. The method of claim 1 , wherein
the importance scores are assigned on a pixel-by-pixel basis within the original digital image as different content elements, and:
identifying one or more regions of interest within the original digital image; and
assigning relative importance scores to the identified region(s) of interest.

8. The method of claim 1 , wherein the importance scores are assigned to the content elements based on their distance from a vanishing point of the original digital image.

9. The method of claim 1 , wherein the importance scores are assigned to the content elements based on assignments of the content elements to one of a foreground content element or a background content element.

10. The method of claim 1 , wherein the cropping window is defined to maximize an aggregate importance score of content elements contained in the cropping window and to possess an aspect ratio that matches an aspect ratio of the original digital image.

11. The method of claim 1 , wherein the cropping window is defined to maximize a size of the cropping window.

12. An image editing system, comprising a processor and memory, the memory containing instructions, that when executed by the processor, cause at least:
when an occlusion is detected in an original digital image, determining an area occupied by the occlusion;
rotating the original digital image to align a detected horizon to a horizontal axis;
assigning importance scores to different content elements of the rotated digital image,
defining a cropping window around an area of the rotated digital image that does not include the area occupied by the occlusion and that is based on the importance scores; and
cropping the rotated digital image to the cropping window.

13. The system of claim 12 , wherein the content elements include at least one region of interest and one non-region of interest and the importance scores are different for portions of image data within the region of interest and portions of image data within the non-region of interest.

14. The system of claim 13 , wherein the cropping window is sized to retain proportionality between the region of interest's relative position within the original digital image and the region of interest's relative position within an image generated from the cropping window.

15. The system of claim 13 , wherein the cropping window is sized to include an entire size of the region of interest and possesses an aspect ratio that matches an aspect ratio of the original digital image.

16. The system of claim 12 , wherein a region of interest is centered in the cropping window.

17. The system of claim 12 , wherein
the importance scores are assigned on a pixel-by-pixel basis within the original digital image as different content elements, and:
identifying one or more regions of interest within the original digital image; and
assigning relative importance scores to the identified region(s) of interest.

18. The system of claim 12 , wherein the importance scores are assigned to the content elements based on their distance from a vanishing point of the original digital image.

19. A non-transitory computer readable media comprising instructions, that when executed on a processor, cause at least:
when an occlusion is detected in an original digital image, determining an area occupied by the occlusion;
rotating the original digital image to align a detected horizon to a horizontal axis;
assigning importance scores to different content elements of the rotated digital image;
defining a cropping window around an area of the rotated digital image that does not include the area occupied by the occlusion and that is based on the importance scores; and
cropping the rotated digital image to the cropping window.

Assignees

• Apple Inc

Inventors

• Zhang Ke
• Zhai Jiefu
• Zheng Yunfei
• Liu Shujie
• Keinath Albert E
• Zhou Xiaosong
• Chung Chris Y
• Wu Hsi-Jung

Classifications

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06V10/273

  removing elements interfering with the pattern to be recognised · CPC title

• G06T7/194 Primary

  involving foreground-background segmentation · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T2207/20132

  Image cropping · CPC title