Systems and methods for processing content

Patent metadata

Field	Value
Publication number	US-10929979-B1
Application number	US-201816235352-A
Country	US
Kind code	B1
Filing date	Dec 28, 2018
Priority date	Dec 28, 2018
Publication date	Feb 23, 2021
Grant date	Feb 23, 2021

Abstract

Systems, methods, and non-transitory computer-readable media can determine a landscape video to be presented in a vertical orientation through a display screen of a computing device. Frames of the landscape video can be cropped to allow for full screen presentation of the landscape video in the vertical orientation. The cropped frames of the landscape video can be provided for presentation through the display screen of the computing device in the vertical orientation.

First claim

What is claimed is:

1. A computer-implemented method comprising:
determining, by a computing system, a landscape video to be provided for presentation through a display screen of a computing device in a vertical orientation;
determining, by the computing system, one or more frames of the landscape video to be cropped to allow for full screen presentation of the landscape video through the display screen of the computing device in the vertical orientation, the determining further comprising:
determining, by the computing system, a salient region within at least one frame of the landscape video based on a saliency prediction model, wherein the saliency prediction model is trained to identify salient regions based on aggregated heat map data;
determining, by the computing system, a cropped portion of the at least one frame of the landscape video that corresponds to the salient region; and

providing, by the computing system, the cropped portion of the at least one frame of the landscape video that corresponds to the salient region for full screen presentation through the display screen of the computing device in the vertical orientation.

2. The computer-implemented method of claim 1 , wherein determining the one or more frames of the landscape video to be cropped to allow for full screen presentation of the landscape video through the display screen of the computing device in the vertical orientation further comprises:
determining, by the computing system, a region to be cropped from at least one frame of the landscape video based on detection of at least one predefined object being detected in the at least one frame.

3. The computer-implemented method of claim 2 , wherein the region to be cropped encompasses the at least one predefined object.

4. The computer-implemented method of claim 2 , wherein the at least one predefined object corresponds to a person or a human face.

5. The computer-implemented method of claim 1 , wherein determining the one or more frames of the landscape video to be cropped to allow for full screen presentation of the landscape video through the display screen of the computing device in the vertical orientation further comprises:
determining, by the computing system, an object represented in at least one frame of the landscape video;
determining, by the computing system, that the object is associated with an active state; and
determining, by the computing system, a region that includes the object to be cropped from the at least one frame.

6. The computer-implemented method of claim 5 , wherein the object corresponds to a human face and second object corresponds to a second human face, and wherein the first object is determined to be associated with an active state based on a determination that the human face is speaking.

7. The computer-implemented method of claim 5 , wherein the object corresponds to a person and a second object represented in the at least one frame corresponds to a second person, and wherein the object is determined to be associated with an active state based on a determination that the person is speaking at a rate that is faster than the second person.

8. The computer-implemented method of claim 5 , wherein the object is determined to be associated with an active state based on a determination that the object is performing some predefined activity.

9. The computer-implemented method of claim 1 , wherein determining the salient region within the at least one frame of the landscape video based on the saliency prediction model further comprises:
determining, by the computing system, a point of interest represented in the at least one frame of the landscape video that is likely to be of interest to users viewing the landscape video based on the saliency prediction model; and
determining, by the computing system, the salient region based on the point of interest.

10. The computer-implemented method of claim 9 , further comprising:
determining, by the computing system, the point of interest represented in the at least one frame is more likely to be of interest to users viewing the landscape video than a second point of interest represented in the at least one frame based on the saliency prediction model.

11. A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform:
determining a landscape video to be provided for presentation through a display screen of a computing device in a vertical orientation;
determining one or more frames of the landscape video to be cropped to allow for full screen presentation of the landscape video through the display screen of the computing device in the vertical orientation, the determining further comprising:
determining a salient region within at least one frame of the landscape video based on a saliency prediction model, wherein the saliency prediction model is trained to identify salient regions based on aggregated heat map data;
determining a cropped portion of the at least one frame of the landscape video that corresponds to the salient region; and

providing the cropped portion of the at least one frame of the landscape video that corresponds to the salient region for full screen presentation through the display screen of the computing device in the vertical orientation.

12. The system of claim 11 , wherein determining the one or more frames of the landscape video to be cropped to allow for full screen presentation of the landscape video through the display screen of the computing device in the vertical orientation further causes the system to perform:
determining a region to be cropped from at least one frame of the landscape video based on detection of at least one predefined object in the at least one frame.

13. The system of claim 12 , wherein the region to be cropped encompasses the at least one predefined object.

14. The system of claim 12 , wherein the at least one predefined object corresponds to a person or a human face.

15. The system of claim 11 , wherein determining the one or more frames of the landscape video to be cropped to allow for full screen presentation of the landscape video through the display screen of the computing device in the vertical orientation further causes the system to perform:
determining an object represented in at least one frame of the landscape video;
determining that the object is associated with an active state; and
determining a region that includes the object to be cropped from the at least one frame.

16. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform a method comprising:
determining a landscape video to be provided for presentation through a display screen of a computing device in a vertical orientation;
determining one or more frames of the landscape video to be cropped to allow for full screen presentation of the landscape video through the display screen of the computing device in the vertical orientation, the determining further comprising:
determining a salient region within at least one frame of the landscape video based on a saliency prediction model, wherein the saliency prediction model is trained to identify salient regions based on aggregated heat map data;
determining a cropped portion of the at least one frame of the landscape video that corresponds to the salient region; and

providing the cropped portion of the at least one frame of the landscape video that corresponds to the salient region for full screen

Assignees

• Facebook Inc

Inventors

• Dimson Thomas Frederick
• Silber Ian McIntyre
• Olson Ryan Keenan

Classifications

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06V20/40

  in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) · CPC title

• G06V40/103

  Static body considered as a whole, e.g. static pedestrian or occupant recognition · CPC title

• G06V40/161

  Detection; Localisation; Normalisation · CPC title

• H04N21/440272

  for performing aspect ratio conversion · CPC title