Video conference with subtiling

Patent metadata

Field	Value
Publication number	US-12581035-B2
Application number	US-202418430404-A
Country	US
Kind code	B2
Filing date	Feb 1, 2024
Priority date	Jun 13, 2023
Publication date	Mar 17, 2026
Grant date	Mar 17, 2026

Abstract

In one or more embodiments a computer implemented method includes generating, by a sensor, a video stream that comprises a series of frames that include a plurality of objects positioned within an environment, determining the objects captured within at least one frame of the video stream, determining one or more croppings for each of the objects in the at least one frame, determining a ranking of the one or more croppings based on a determined total combined loss value for each the one or more croppings, determining each subtiling combination for each of the one or more croppings, each subtiling combination comprising at least one subtile that includes at least one object; determining a total subtiling penalty value (TSPV) for each subtiling combination, and determining a preferred subtiled combination based on the total combined loss value of each the one or more croppings and the TSPV of each subtiling combination.

First claim

What is claimed is:

1 . A computer implemented method, comprising:
generating, by a sensor, a video stream that comprises a series of frames that each include a plurality of objects positioned within an environment; determining the objects captured within at least one frame of the video stream; determining one or more croppings for each of the objects in the at least one frame of the video stream; determining a ranking of the determined one or more croppings based on a determined total combined loss value for each the one or more croppings; determining each subtiling combination for each of the one or more croppings, each subtiling combination comprising at least one subtile that includes at least one object; determining a total subtiling penalty value (TSPV) for each subtiling combination; and determining a preferred subtiled combination based on the total combined loss value of each the one or more croppings and the TSPV of each subtiling combination.

2 . The method of claim 1 , further comprising:
adjusting each subtiling combination based on the TSPV of each subtiling combinations; and pruning subtiling combinations from each of the subtiling combinations prior to adjusting each of the subtiling combinations.

3 . The method of claim 2 , wherein subtiling combinations that comprise subtiles that do not have a fractional aspect ratio of an aspect ratio of a corresponding cropping are pruned.

4 . The method of claim 2 , wherein subtiling combinations that comprise subtiles that extend outside of the at least one frame are pruned.

5 . The method of claim 2 , wherein subtiling combinations that comprise at least one subtile that includes a quantity of objects that is greater than a maximum quantity of objects are pruned.

6 . The method of claim 1 , further comprising:
adjusting each subtiling combination based on the TSPV of each subtiling combinations, wherein adjusting each subtiling combination comprises: determining subtiling penalty values that each correspond to an attribute of each subtiling combination; determining a total subtiling penalty value (TSPV) for each subtiling combination by adding the subtiling penalty values of each subtiling combination; and adjusting each of the subtiling combinations to minimize the TSPV.

7 . The method of claim 1 , wherein the subtiling penalty values for each subtiling combination are determined by at least one of the following:
analyzing a difference in head-size of objects positioned in each subtile of each subtiling combination; analyzing whether each subtiling combination includes subtiles that include a single object; analyzing whether each subtiling combination includes subtiles that include multiple objects; analyzing an amount of asymmetry between subtiles in each subtiling combination; and analyzing a difference in vertical alignment between objects in a same subtile in each subtiling combination.

8 . The method of claim 1 , wherein determining a TSPV for each subtiling combination comprises:
determining a subtiling coefficient corresponding to each subtiling penalty value; multiplying each subtiling penalty value with the corresponding subtiling coefficient; and computing a sum of each subtiling penalty that is multiplied by each subtiling penalty for each subtiling combination.

9 . The method of claim 8 , wherein the subtiling coefficients and the subtiling penalty values are determined based on at least one of: algorithm presets, a video conferencing application, and user preferences.

10 . The method of claim 9 , wherein the algorithm presets include at least one of: preferring singular subtiles, optimize for a fixed composed grid, maximum number of the most important people only, and separate people into separate streams algorithm presets.

11 . The method of claim 1 , wherein determining the preferred subtiled combination comprises:
summing the TSPVs of each subtiling combination that correspond to each grouping to determine grouping TSPVs (GTSPVs) for each grouping; adding each of the GTSPVs for each combination of groupings to the corresponding total combined grouping loss value for each combination of groupings to determine total subtiled group loss values (TSGLVs) for each combination of groupings; and selecting a grouping from the combination of groupings with subtitled croppings having the lowest TSGLV as the preferred subtiled grouping combination.

12 . A video conferencing system comprising:
a sensor configured to generate a video stream that comprises a series of frames; and a controller that comprises a processor and instructions stored in memory, the instructions, when executed by the processor causes the controller to perform a method comprising:
generating, by the sensor, a video stream that comprises a series of frames that each include a plurality of objects positioned within a conference environment;
determining the objects captured within at least one frame of the video stream;
determining one or more croppings for each of the objects in the at least one frame of the video stream;
determining each combination of groupings of objects within the at least one frame of the series of frames;
determining a total combined grouping loss value for each of the combination of groupings;
determining each subtiling combination for each combination of groupings, each subtiling combination comprising at least one subtile;
determining a total subtiling penalty value (TSPV) for each subtiling combination;
adjusting each subtiling combination based on the TSPV of each subtiling combinations; and
determining a preferred subtiled grouping combination based on the total combined grouping loss value of each combination of groupings and the TSPV of each subtiling combination.

13 . The video conferencing system of claim 12 , wherein the instructions further comprise instructions to prune subtiling combinations from each of the subtiling combinations prior to adjusting each of the subtiling combinations, wherein subtiling combinations that are pruned comprise subtiling combinations that include subtiles that do not have a fractional aspect ratio of an aspect ratio of a corresponding cropping, subtiling combinations that include subtiles that extend outside of the at least one frame, and subtiling combinations that include at least one subtile that includes a quantity of objects that is greater than a maximum quantity of objects.

14 . The video conferencing system of claim 12 , wherein the instructions for adjusting each subtiling combination comprise instructions for:
determining subtiling penalty values that each correspond to an attribute of each subtiling combination; determining a total subtiling penalty value (TSPV) for each subtiling combination by adding the subtiling penalty values of each subtiling combination; and adjusting each of the subtiling combinations to minimize the TSPV.

15 . The video conferencing system of claim 12 , wherein the subtiling penalty values for each subtiling combination are determined by at least the following:
analyzing a difference in head-size of objects positioned in each subtile of each subtiling combination; analyzing whether each subtiling combination includes subtiles that include a single object; analyzing whether each subtiling combination includes subtiles that include multiple objects; analyzing an amount of asymmetry between subtiles in each subtiling combination; and analyzing a difference in vertical alignment between objects in a same subtile in each subtiling combination.

16 . The video conferencing syst

Assignees

• Logitech Europe Sa

Inventors

• Ostap Oleg
• Levak Henry
• Verhun Kostiantyn
• Dutchak Ihor
• Naumov Sergii
• Wrobel Pawel
• Maestre Trueba Miguel Angel
• Bansal Anurag
• Ostap Volodymyr
• Perry Kristopher Glenn

Classifications

• H04N5/278

  Subtitling · CPC title

• H04N5/2628

  Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation · CPC title

• G06V20/49

  Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T2207/20132

  Image cropping · CPC title