Scale image resolution for motion estimation

Patent metadata

Field	Value
Publication number	US-12154278-B2
Application number	US-202217853422-A
Country	US
Kind code	B2
Filing date	Jun 29, 2022
Priority date	Jun 29, 2022
Publication date	Nov 26, 2024
Grant date	Nov 26, 2024

Abstract

Systems and techniques are provided for conserving resources when performing motion estimation. An example process can include determining, based on an input image and a reference image, motion vectors identifying motion between the input image and the reference image; determining whether the motion indicated by the motion vectors is below a first threshold; based on a determination that the motion indicated by the motion vectors is below the first threshold, refraining from determining a local motion between the input image and the reference image; determining a transform matrix based on the motion vectors and without using a local motion between the input image and the reference image; and adjusting the input image based on the transform matrix.

First claim

What is claimed is:

1. An apparatus for processing image data, the apparatus comprising:
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor configured to:
determine, using a higher-power motion estimation pipeline, a dense motion map between a first input image and a first reference image, the dense motion map indicating local motion between pixels in the first input image and pixels in the first reference image;
determine that the local motion indicated by the dense motion map is below a first threshold;
in response to a determination that the local motion indicated by the motion vectors dense motion map is below the first threshold, switch from the higher-power motion estimation pipeline to a lower-power motion estimation pipeline for a second input image and a second reference image, wherein, using the lower-power motion estimation pipeline, the at least one processor is configured to:
determine, using descriptor matching estimation, motion vectors indicating global motion between the second input image and the second reference image;
refrain from determining a dense motion map between the second input image and the second reference image;
refrain from performing image alignment between the second input image and the second reference image; and
warp the motion vectors using a global stabilization matrix to generate a transform matrix without using the dense motion map between the second input image and the second reference image; and

perform image stabilization for the second input image using the transform matrix.

2. The apparatus of claim 1 , wherein the at least one processor is configured to:
determine, based on a third input image and a third reference image, additional motion vectors identifying motion between the third input image and the third reference image;
determine whether the motion indicated by the additional motion vectors is below or above the first threshold;
based on a determination that the motion indicated by the additional motion vectors is above the first threshold, generate an additional dense motion map indicating a local motion between the third input image and the third reference image;
determine an additional transform matrix based on the additional dense motion map; and
adjust the third input image based on the additional transform matrix.

3. The apparatus of claim 2 , wherein, to determine the additional transform matrix based on the additional dense motion map, the at least one processor is configured to warp the additional dense motion map using the global stabilization matrix.

4. The apparatus of claim 2 , wherein the at least one processor is configured to:
prior to generating the additional dense motion map, align the additional motion vectors based on a global motion associated with the apparatus, the additional motion vectors being aligned based on motion information obtained from a motion sensor, the motion sensor comprising at least one of a gyroscope or an inertial measurement unit; and
generate the additional dense motion map based on the aligned additional motion vectors.

5. The apparatus of claim 2 , wherein the at least one processor is configured to determine that the motion indicated by the additional motion vectors is above the first threshold and below a second threshold that is greater than the first threshold.

6. The apparatus of claim 2 , wherein the at least one processor is configured to:
prior to determining the additional motion vectors identifying motion between the third input image and the third reference image, downscale the third input image and the third reference image, wherein the additional motion vectors are determined based on the downscaled third input image and the downscaled third reference image.

7. The apparatus of claim 1 , wherein the at least one processor is configured to:
determine, based on a third input image and a third reference image, additional motion vectors identifying motion between the third input image and the third reference image;
determine whether the motion indicated by the additional motion vectors is above the first threshold and below a second threshold that is greater than the first threshold; and
based on a determination that the motion indicated by the additional motion vectors is above the first threshold and below the second threshold:
downscale the third input image and the third reference image;
generate an additional local motion map indicating a local motion between the downscaled third input image and the downscaled third reference image;
determine an additional transform matrix based on the additional local motion map; and
adjust the third input image based on the additional transform matrix.

8. The apparatus of claim 7 , wherein, to determine the additional transform matrix based on the additional local motion map, the at least one processor is configured to warp the additional local motion map using the global stabilization matrix.

9. The apparatus of claim 7 , wherein the at least one processor is configured to:
prior to generating the additional local motion map, align the additional motion vectors based on a global motion associated with the apparatus, the additional motion vectors being aligned based on motion information obtained from a motion sensor, the motion sensor comprising at least one of a gyroscope and an inertial measurement unit.

10. The apparatus of claim 9 , wherein the at least one processor is configured to generate the additional local motion map based on the aligned additional motion vectors.

11. The apparatus of claim 1 , further comprising a camera configured to capture a video sequence comprising the first input image, the first reference image, the second input image, and the second reference image.

12. The apparatus of claim 1 , wherein the apparatus comprises a mobile device.

13. A method of processing image data, the method comprising:
determining, by a computing device using a higher-power motion estimation pipeline, a dense motion map between a first input image and a first reference image, the dense motion map indicating local motion between pixels in the first input image and pixels in the first reference image;
determining that the local motion indicated by the dense motion map is below a first threshold;
in response to a determination that the local motion indicated by the dense motion map is below the first threshold, switch from the higher-power motion estimation pipeline to a lower-power motion estimation pipeline for a second input image and a second reference image, wherein the lower-power motion estimation pipeline includes:
determining motion vectors indicating global motion between the second input image and the second reference image;
refraining from determining a dense motion map between the second input image and the second reference image;
refraining from performing image alignment between the second input image and the second reference image; and
warping the motion vectors using a global stabilization matrix to generate a transform matrix without using the dense motion map between the second input image and the second reference image; and

performing image stabilization for the second input image using the transform matrix.

14. The method of claim 13 , further comprising:
determining, based on a third input image and a third reference image, additional motion vectors identifying motion between the third input image and the third reference image;
determining whether the motion indicated by the additional motion vectors is below or above the first threshold;
based on a deter

Assignees

• Qualcomm Inc

Inventors

• Bandwar Nitin
• Zobel Pia
• Hardoon Roee
• Lee Sungwon
• Kazi Tauseef
• Han Bing

Classifications

• G06T7/215

  Motion-based segmentation · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/207 Primary

  for motion estimation over a hierarchy of resolutions (multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53) · CPC title