Device motion aware temporal denoising

Patent metadata

Field	Value
Publication number	US-11902661-B2
Application number	US-202217569768-A
Country	US
Kind code	B2
Filing date	Jan 6, 2022
Priority date	Jan 6, 2022
Publication date	Feb 13, 2024
Grant date	Feb 13, 2024

Abstract

An apparatus for device motion aware temporal denoising includes a processor and a memory that stores code executable by the processor to receive video data comprising a sequence of video frames that includes a current frame and one or more preceding frames captured by a video capture device that functions as a camera of a portable electronic device. In various examples, the code causes the processor to determine one or more camera motion compensation adjustments to apply to the one or more preceding frames of the sequence based on one or more non-imaging device motion sensor measurements for the video capture device and to generate a sequence of camera-motion compensated preceding frames by applying the one or more camera motion compensation adjustments to the preceding frames of the sequence. A method and a computer program may perform the functions of the apparatus.

First claim

What is claimed is:

1. An apparatus for device motion aware temporal denoising, the apparatus comprising:
a processor; and
a memory that stores code executable by the processor to:
receive video data comprising a sequence of video frames that includes a current frame and one or more preceding frames captured by a video capture device that functions as a camera of a portable electronic device;
determine one or more camera motion compensation adjustments to apply to the one or more preceding frames of the sequence based on one or more non-imaging device motion sensor measurements for the video capture device;
generate a sequence of camera-motion compensated preceding frames by applying the one or more camera motion compensation adjustments to the preceding frames of the sequence; and
apply a first denoising process comprising video temporal denoising and/or video spatial denoising to first portions of the current frame which have an overlapping temporal history with the corresponding preceding frames and apply a second denoising process comprising video spatial denoising, rather than temporal denoising, to second portions of the current frame which do not have an overlapping temporal history with the corresponding preceding frames.

2. The apparatus of claim 1 , further comprising an inertial measurement unit that generates the one or more non-imaging device motion sensor measurements in response to motion of the video capture device.

3. The apparatus of claim 1 , wherein the code is further executable by the processor to apply the one or more camera motion compensation adjustments to the preceding frames of the sequence based on differences in a sequence of camera pose measurements corresponding respectively to the preceding frames relative to a camera pose measurement of the current frame.

4. The apparatus of claim 1 , further comprising a connected component motion sensor that generates the one or more non-imaging device motion sensor measurements based on motion of first portion of the portable electronic device that includes the video capture device relative to a second portion of the portable electronic device.

5. The apparatus of claim 1 , wherein the one or more camera motion compensation adjustments comprise an x-shift and a y-shift of image pixels of the preceding frames of the sequence to improve alignment of the image pixels of the preceding frames relative to corresponding image pixels of the current frame based on the non-imaging device motion sensor measurements.

6. The apparatus of claim 1 , further comprising a depth sensor that enables the processor to determine a measured distance between the camera and a target object within one or more video frames of the sequence to facilitate performing the one or more camera compensation adjustments to the preceding frames.

7. The apparatus of claim 1 , wherein the code is further executable by the processor to use a predetermined range of estimated distances between the camera and a target object to perform one or more camera motion compensation adjustments to the preceding frames, in response to determining the that that the camera motion is rotation about a predetermined axis.

8. The apparatus of claim 1 , wherein the processor is further configured to:
receive one or more non-imaging derived object motion measurements of an object within a field of view for the video capture device; and
apply one or more object motion compensation adjustments to respective portions of the preceding frames of the sequence corresponding to image pixels for the object in the current frame in response to determining that the one or more non-imaging derived object motion measurements are below a predetermined motion threshold.

9. The apparatus of claim 1 , wherein the one or more camera-motion compensation adjustments applied to the preceding frames are determined by the processor independently from image pixel values of the preceding frames.

10. A method for device motion aware temporal denoising, comprising:
receiving video data comprising a sequence of video frames that includes a current frame and one or more preceding frames captured by a video capture device that functions as a camera of a portable electronic device;
determining one or more camera motion compensation adjustments to apply to the one or more preceding frames of the sequence based on one or more non-imaging device motion sensor measurements for the video capture device;
generating a sequence of camera-motion compensated preceding frames by applying the one or more camera motion compensation adjustments to the preceding frames of the sequence; and
applying a first denoising process comprising video temporal denoising and/or video spatial denoising to first portions of the current frame which have an overlapping temporal history with the corresponding preceding frames and applying a second denoising process comprising video spatial denoising, rather than temporal denoising, to second portions of the current frame which do not have an overlapping temporal history with the corresponding preceding frames.

11. The method of claim 10 , further comprising obtaining the one or more non-imaging device motion sensor measurements from an inertial measurement unit in response to motion for the video capture device.

12. The method of claim 11 , further comprising applying the one or more camera motion compensation adjustments to the preceding frames of the sequence based on differences in a sequence of camera pose measurements corresponding respectively to the preceding frames relative to a camera pose measurement corresponding to the current frame.

13. The method of claim 10 , further comprising generating the one or more non-imaging device motion sensor measurements based on motion of a first portion of the portable electronic device that includes the video capture device relative to a second portion of the portable electronic device.

14. The method of claim 10 , wherein the one or more camera motion compensation adjustments comprise an x-shift and a y-shift of image pixels of the preceding frames of the sequence to improve alignment of the image pixels of the preceding frames relative to corresponding image pixels of the current frame based on the non-imaging device motion sensor measurements.

15. The method of claim 10 , further comprising determining a measured distance between the camera and a target object within one or more video frames of the sequence to facilitate performing the one or more camera compensation adjustments to the preceding frames.

16. The method of claim 10 , further comprising performing one or more camera rotation compensation adjustments to the one or more preceding frames based on based on a predetermined range of estimated distances between the camera and a target object.

17. The method of claim 10 , further comprising:
receiving one or more non-imaging derived object motion measurements of an object within a field of view for the video capture device; and
applying one or more object motion compensation adjustments to respective portions of the preceding frames of the sequence corresponding to image pixels for the object in the current frame in response to determining that the one or more non-imaging derived object motion measurements are below a predetermined motion threshold.

18. The method of claim 10 , wherein applying the second denoising process does not include image cropping.

19. The method of claim 10 , wherein the one or more non-imaging device motion sensors are hinge angle sensors.

Assignees

• Lenovo Singapore Pte Ltd

Inventors

• Bryson Barrett
• Nicholson John W

Classifications

• H04N23/683 Primary

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/6811 Primary

  based on the image signal · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title