Temporal filtering weight computation

Patent metadata

Field	Value
Publication number	US-12277681-B2
Application number	US-202318211376-A
Country	US
Kind code	B2
Filing date	Jun 19, 2023
Priority date	Nov 11, 2021
Publication date	Apr 15, 2025
Grant date	Apr 15, 2025

Abstract

Techniques to temporally filter images via a filtering weight computation are disclosed. A first image having a first timestamp and a second image having a second timestamp are acquired. These images are generated by a camera, and the first timestamp is before the second timestamp. A motion compensation (MC) operation is performed on the first image to produce an MC image. A difference image is generated using the MC image and the second image. The difference image reflects differences in intensities that exist between the two images. A local weight map is generated based on those differences. A global weight map is generated based on certain IMU data. A final weight map is generated based on a combination of the local weight map and the global weight map. The final weight map is used to generate a filtered image.

First claim

What is claimed is:

1. A method for performing temporal filtering, said method comprising a filtering weight computation that includes:
acquiring a first image having a first timestamp, the first image being generated by a camera;
acquiring a second image having a second timestamp, the second image also being generated by the camera, wherein the first timestamp is before the second timestamp such that the first image is a previously acquired image relative to the second image;
performing a motion compensation (MC) operation on the first image to account for motion that might have occurred during a time between the first timestamp and the second timestamp such that an MC image is generated;
generating a difference image using the MC image and the second image, the difference image reflecting differences in intensities that exist between corresponding pixels in both the MC image and the second image;
generating a final weight map based on a local weight map that is based on the differences in intensities in the difference image and a global weight map that is based on movement data that exists for the camera, wherein generating the final weight map further includes performing an upscaling operation to upscale the local weight map to match an original resolution of the first image;
using the final weight map to generate a filtered image; and
during a subsequent iteration of the filtering weight computation, causing the filtered image to be used as the previously acquired image.

2. The method of claim 1 , wherein the first image is a low light image.

3. The method of claim 1 , wherein the movement data is inertial measurement unit (IMU) data.

4. The method of claim 3 , wherein the IMU data includes data describing an angular position of the camera and an orientation of the camera.

5. The method of claim 4 , wherein the IMU data further includes data describing an acceleration of the camera.

6. The method of claim 1 , wherein the camera operates at a rate between 30 frames per second and 120 frames per second.

7. The method of claim 1 , wherein the camera operates at a rate between 60 frames per second and 90 frames per second.

8. The method of claim 1 , wherein the camera moved after generating the first image and before the second image is generated.

9. The method of claim 1 , wherein the MC image is down sampled.

10. The method of claim 1 , wherein the second image is down sampled.

11. A computer system that performs temporal filtering via a filtering weight computation, said computer system:
at least one processor; and
at least one hardware storage device that stores instructions that are executable by the at least one processor to cause the computer system to:
acquire a first image having a first timestamp, the first image being generated by a camera;
acquire a second image having a second timestamp, the second image also being generated by the camera, wherein the first timestamp is before the second timestamp such that the first image is a previously acquired image relative to the second image;
perform a motion compensation (MC) operation on the first image to account for motion that might have occurred during a time between the first timestamp and the second timestamp such that an MC image is generated;
generate a difference image using the MC image and the second image, the difference image reflecting differences in intensities that exist between corresponding pixels in both the MC image and the second image;
generate a final weight map based on a local weight map that is based on the differences in intensities in the difference image and a global weight map that is based on movement data that exists for the camera, wherein generating the final weight map includes an upscaling operation to upscale the local weight map to match an original resolution of the first image;
use the final weight map to generate a filtered image; and
during a subsequent iteration of the filtering weight computation, cause the filtered image to be used as the previously acquired image.

12. The computer system of claim 11 , wherein the first and second images are down sampled.

13. The computer system of claim 12 , wherein the first and second images are down sampled to a same resolution level.

14. The computer system of claim 12 , wherein down sampling the first and second images removes, at least in part, a ghosting effect.

15. The computer system of claim 12 , wherein said down sampling is a 4 pixel by 4 pixel down sampling.

16. The computer system of claim 11 , wherein the first image and the second image are low light images.

17. A computer system that performs temporal filtering via a filtering weight computation, said computer system:
at least one processor; and
at least one hardware storage device that stores instructions that are executable by the at least one processor to cause the computer system to:
acquire a first image having a first timestamp, the first image being generated by a camera;
acquire a second image having a second timestamp, the second image also being generated by the camera, wherein the first timestamp is before the second timestamp such that the first image is a previously acquired image relative to the second image;
perform a motion compensation (MC) operation on the first image to account for motion that might have occurred during a time between the first timestamp and the second timestamp such that an MC image is generated;
generate a difference image using the MC image and the second image, the difference image reflecting differences that exist between corresponding pixels in both the MC image and the second image;
generate a final weight map based on a local weight map that is based on the differences in the difference image and a global weight map that is based on movement data that exists for the camera, wherein the local weight map is structured to include analog gain data and digital gain data for generating the first and second images;
use the final weight map to generate a filtered image; and
during a subsequent iteration of the filtering weight computation, cause the filtered image to be used as the previously acquired image.

18. The computer system of claim 17 , wherein the first and second images are low light images.

19. The computer system of claim 17 , wherein the movement data is inertial measurement unit (IMU) data.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Price Raymond Kirk
• Edmonds Christopher Douglas
• Bleyer Michael
• Maekelae Christian Markus

Classifications

• G06T5/70 Primary

  Denoising; Smoothing · CPC title

• G06T2207/20224

  Image subtraction · CPC title

• G06T2207/20216

  Image averaging · CPC title

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title