Segmentation mask extrapolation

Patent metadata

Field	Value
Publication number	US-12307685-B2
Application number	US-202217931859-A
Country	US
Kind code	B2
Filing date	Sep 13, 2022
Priority date	Sep 13, 2022
Publication date	May 20, 2025
Grant date	May 20, 2025

Abstract

Systems and techniques are described herein for processing video data. For instance, a technique can include receiving a first image after a previous image. The process can further include receiving a first segmentation mask associated with the previous image. The process can also include estimating a first set of forward motion vectors between the previous image and the first image. The process can further include estimating a reliability of the first set of forward motion vectors. The process can also include extrapolating a second segmentation mask associated with the first image using the first set of forward motion vectors and first segmentation mask based on the estimated reliability of the first set of forward motion vectors.

First claim

What is claimed is:

1. A method for processing video data, the method comprising:
receiving a first image after a previous image;
receiving a first segmentation mask associated with the previous image;
estimating a first set of forward motion vectors between the previous image and the first image;
estimating a reliability of the first set of forward motion vectors; and
extrapolating a second segmentation mask associated with the first image using the first set of forward motion vectors and the first segmentation mask based on the estimated reliability of the first set of forward motion vectors.

2. The method of claim 1 , wherein estimating the reliability of the first set of forward motion vectors comprises:
determining one or more heuristic statistics based on the first set of forward motion vectors; and
evaluating the one or more heuristic statistics to estimate the reliability of the first set of forward motion vectors.

3. The method of claim 2 , wherein the one or more heuristic statistics comprise at least one of:
gradients for one or more motion vectors of the first set of forward motion vectors, or
a percent change of a foreground.

4. The method of claim 3 , wherein evaluating the one or more heuristic statistics comprise at least one of:
evaluating the gradients for one or more motion vectors against a gradient threshold value, or
estimating the percent change of a foreground based on a threshold value of foreground motion vectors.

5. The method of claim 1 , further comprising:
receiving a second image, wherein the second image is a non-key frame, wherein a segmentation mask machine learning model is configured to generate segmentation masks for key frames;
estimating a second set of forward motion vectors between the second image and another previous image;
estimating the reliability of the second set of forward motion vectors; and
determining to apply a segmentation mask machine learning model to the second image based on an estimate that the second set of forward motion vectors is not reliable.

6. The method of claim 1 , further comprising labeling one or more portions of the first segmentation mask as part of a foreground or background.

7. The method of claim 6 , further comprising:
detecting an uncovered portion of the first image based on the first set of forward motion vectors; and
filling the uncovered portion based on a neighboring background portion of the first segmentation mask and the previous image.

8. The method of claim 7 , wherein filling the uncovered portion comprises:
identifying a motion vector associated with a foreground portion of the first segmentation mask closest to a portion of the uncovered portion;
copying the motion vector; and
associating the copied motion vector with the portion of the uncovered portion.

9. The method of claim 1 , wherein the first image is a non-key frame, and wherein a segmentation mask machine learning model is configured to generate segmentation masks for key frames.

10. The method of claim 1 , further comprising:
receiving a third image, wherein the third image is a key frame, wherein a segmentation mask machine learning model is configured to generate segmentation masks for key frames; and
applying a segmentation mask machine learning model to the third image.

11. An apparatus for processing video data, the apparatus comprising:
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor being configured to:
receive a first image after a previous image;
receive a first segmentation mask associated with the previous image;
estimate a first set of forward motion vectors between the previous image and the first image;
estimate a reliability of the first set of forward motion vectors; and
extrapolate a second segmentation mask associated with the first image using the first set of forward motion vectors and the first segmentation mask based on the estimated reliability of the first set of forward motion vectors.

12. The apparatus of claim 11 , wherein, to estimate the reliability of the first set of forward motion vectors, the at least one processor is further configured to:
determine one or more heuristic statistics based on the first set of forward motion vectors; and
evaluate the one or more heuristic statistics to estimate the reliability of the first set of forward motion vectors.

13. The apparatus of claim 12 , wherein the one or more heuristic statistics comprise at least one of:
gradients for one or more motion vectors of the first set of forward motion vectors, or
a percent change of a foreground.

14. The apparatus of claim 13 , wherein, to evaluate the one or more heuristic statistics, the at least one processor is further configured to:
evaluate the gradients for one or more motion vectors against a gradient threshold value, or
estimate the percent change of a foreground based on a threshold value of foreground motion vectors.

15. The apparatus of claim 11 , wherein the at least one processor is further configured to:
receive a second image, wherein the second image is a non-key frame, wherein a segmentation mask machine learning model is configured to generate segmentation masks for key frames;
estimate a second set of forward motion vectors between the second image and another previous image;
estimate the reliability of the second set of forward motion vectors; and
determine to apply a segmentation mask machine learning model to the second image based on an estimate that the second set of forward motion vectors is not reliable.

16. The apparatus of claim 11 , wherein the at least one processor is further configured to label one or more portions of the first segmentation mask as part of a foreground or background.

17. The apparatus of claim 16 , wherein the at least one processor is further configured to:
detect an uncovered portion of the first image based on the first set of forward motion vectors; and
fill the uncovered portion based on a neighboring background portion of the first segmentation mask and the previous image.

18. The apparatus of claim 17 , wherein, to fill the uncovered portion, the at least one processor is further configured to:
identify a motion vector associated with a foreground portion of the first segmentation mask closest to a portion of the uncovered portion;
copy the motion vector; and
associate the copied motion vector with the portion of the uncovered portion.

19. The apparatus of claim 17 , wherein the first image is a non-key frame, and wherein a segmentation mask machine learning model is configured to generate segmentation masks for key frames.

20. The apparatus of claim 11 , wherein the at least one processor is further configured to:
receive a third image, wherein the third image is a key frame, wherein a segmentation mask machine learning model is configured to generate segmentation masks for key frames; and
apply a segmentation mask machine learning model to the third image.

21. A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to:
receive a first image after a previous image;
receive a first segmentation mask associated with the previous image;
estimate a first set of forward motion vectors between the previous image and the first image;
estimate a reliability of the first set of forward motion vectors; and
extrapolate a second segmentation mask

Assignees

• Qualcomm Inc

Inventors

• Asma Qureshi
• Gnanapragasam Darren
• Shoa Hassani Lashdan Alireza
• Hansen David

Classifications

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/194

  involving foreground-background segmentation · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/11

  Region-based segmentation · CPC title