Temporal filtering weight computation

Patent metadata

Field	Value
Publication number	US-11756177-B2
Application number	US-202117524274-A
Country	US
Kind code	B2
Filing date	Nov 11, 2021
Priority date	Nov 11, 2021
Publication date	Sep 12, 2023
Grant date	Sep 12, 2023

Abstract

Techniques to temporally filter images via a filtering weight computation are disclosed. A first image having a first timestamp and a second image having a second timestamp are acquired. These images are generated by a camera, and the first timestamp is before the second timestamp. A motion compensation (MC) operation is performed on the first image to produce an MC image. A difference image is generated using the MC image and the second image. The difference image reflects differences in intensities that exist between the two images. A local weight map is generated based on those differences. A global weight map is generated based on certain IMU data. A final weight map is generated based on a combination of the local weight map and the global weight map. The final weight map is used to generate a filtered image.

First claim

What is claimed is:

1. A method for performing temporal filtering, said method comprising a filtering weight computation that includes:
acquiring a first image having a first timestamp, the first image being generated by a camera;
acquiring a second image having a second timestamp, the second image also being generated by the camera, wherein the first timestamp is before the second timestamp such that the first image is a previously acquired image relative to the second image;
performing a motion compensation (MC) operation on the first image to account for motion that might have occurred during a time between the first timestamp and the second timestamp such that an MC image is generated;
generating a difference image using the MC image and the second image, the difference image reflecting differences in intensities that exist between corresponding pixels in both the MC image and the second image;
generating a local weight map based on the differences in intensities in the difference image;
generating a global weight map based on inertial measurement unit (IMU) data that was generated by an IMU, the IMU data reflecting both an angular movement of the camera and an acceleration of the camera during a period between the first timestamp and the second timestamp;
generating a final weight map based on a combination of the local weight map and the global weight map;
using the final weight map to generate a filtered image; and
during a subsequent iteration of the filtering weight computation, causing the filtered image to be used as the previously acquired image.

2. The method of claim 1 , wherein, subsequent to performing the MC operation and prior to generating the difference image, the filtering weight computation further includes:
down-sampling the MC image to a selected resolution level;
down-sampling the second image to the selected resolution level,
wherein generating the difference image is then performed using the down-sampled MC image and the down-sampled second image.

3. The method of claim 2 , wherein down-sampling the MC image and the second image operates to reduce noise in the MC image and the second image.

4. The method of claim 2 , wherein generating the local weight map based on the differences in intensities in the difference image includes:
converting the differences in intensities to weight parameters by applying a negative exponential function to an absolute of the differences in intensities;
upscaling a number of the weight parameters in the local weight map to a number that matches an original resolution of the first image and the second image, wherein said upscaling includes performing one or more of a linear upscaling or a bilinear upscaling operation to smooth transitions between the weight parameters included in the local weight map.

5. The method of claim 1 , wherein the local weight map includes one or more of an analog gain setting of the camera or a digital gain setting of the camera.

6. The method of claim 1 , wherein the global weight map includes one or more of an analog gain setting of the camera or a digital gain setting of the camera.

7. The method of claim 1 , wherein the final weight map is generated by combining the global weight map with the local weight map by multiplying weights included in the global weight map against weights included in the local weight map.

8. The method of claim 1 , wherein values included in the global weight map are uniform across the entire global weight map.

9. The method of claim 1 , wherein relatively larger differences in intensities that are reflected by the difference image are afforded relatively lower weights in the local weight map while relatively smaller differences in intensities that are reflected by the difference image are afforded relatively higher weights in the local weight map.

10. The method of claim 1 , wherein:
relatively higher amounts of angular movement and acceleration of the camera result in relatively lower weight values being included in the global weight map, said weight values reflecting an amount of influence the first image will have in generating the filtered image; and
relatively lower amounts of angular movement and acceleration of the camera result in relatively higher weight values being included in the global weight map.

11. A computer system configured to perform temporal filtering via a filtering weight computation, said computer system comprising:
one or more processors; and
one or more computer-readable hardware storage devices that store instructions that are executable by the one or more processors to cause the computer system to perform the filtering weight computation by causing the computer system to:
acquire a first image having a first timestamp, the first image being generated by a camera;
acquire a second image having a second timestamp, the second image also being generated by the camera, wherein the first timestamp is before the second timestamp such that the first image is a previously acquired image relative to the second image;
perform a motion compensation (MC) operation on the first image to account for motion that might have occurred during a time between the first timestamp and the second timestamp such that an MC image is generated;
generate a difference image using the MC image and the second image, the difference image reflecting differences in intensities that exist between corresponding pixels in both the MC image and the second image;
generate a local weight map based on the differences in intensities in the difference image;
generate a global weight map based on inertial measurement unit (IMU) data that was generated by an IMU, the IMU data reflecting both an angular movement of the camera and an acceleration of the camera during a period between the first timestamp and the second timestamp;
generate a final weight map based on a combination of the local weight map and the global weight map;
use the final weight map to generate a filtered image; and
during a subsequent iteration of the filtering weight computation, cause the filtered image to be used as the previously acquired image.

12. The computer system of claim 11 , wherein the final weight map provides pixel-based weights that regulate generation of the filtered image.

13. The computer system of claim 11 , wherein the camera is a low light camera.

14. The computer system of claim 11 , wherein, as a result of generating the difference image, the filtering weight computation temporally averages multiple images together.

15. The computer system of claim 11 , wherein the global weight map includes a single weight value for the entirety of the global weight map.

16. The computer system of claim 11 , wherein the MC image is subjected to a 4 pixel by 4 pixel down-sampling.

17. The computer system of claim 16 , wherein the second image is subjected to the same 4 pixel by 4 pixel down-sampling.

18. The computer system of claim 17 , wherein the local weight map is subjected to a 4 pixel by 4 pixel upscaling.

19. One or more hardware storage devices that store instructions that are executable by one or more processors of a computer system to cause the computer system to perform a filtering weight computation by causing the computer system to:
acquire a first image having a first timestamp, the first image being generated by a camera;
acquire a second image having a second timestamp, the second image also being generated by the camera, wherein the first timestamp is before the second timestamp such that the first image is a previo

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Price Raymond Kirk
• Edmonds Christopher Douglas
• Bleyer Michael
• Maekelae Christian Markus

Classifications

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/20182

  Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering · CPC title

• G06T7/254

  involving subtraction of images · CPC title

• H04N23/682

  Vibration or motion blur correction · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title