Adaptive sampling of images

Patent metadata

Field	Value
Publication number	US-11645761-B2
Application number	US-202016993781-A
Country	US
Kind code	B2
Filing date	Aug 14, 2020
Priority date	Aug 14, 2020
Publication date	May 9, 2023
Grant date	May 9, 2023

Abstract

In one embodiment, a method includes determining characteristics of one or more areas in an image by analyzing pixels in the image, computing a sampling density for each of the one or more areas in the image based on the characteristics of the one or more areas, generating samples corresponding to the image by sampling pixels in each of the one or more areas according to the associated sampling density, and providing the samples to a machine-learning model as an input, where the machine-learning model is configured to reconstruct the image by processing the samples.

First claim

What is claimed is:

1. A method comprising, by a computing device:
determining characteristics of one or more areas in an image by analyzing pixels in the image;
computing, based on the characteristics of the one or more areas, a sampling density for each of the one or more areas in the image, wherein computing the sampling density for the area comprises:
identifying a darkest pixel and a brightest pixel in the area;
measuring a distance between the darkest pixel and the brightest pixel; and
computing the sampling density for the area based on the distance such that a longer distance yields a higher sampling density;

generating samples corresponding to the image by sampling pixels in each of the one or more areas according to the associated sampling density; and
providing the samples to a machine-learning model as an input, wherein the machine-learning model is configured to reconstruct the image by processing the samples.

2. The method of claim 1 , wherein the characteristic associated with an area comprises a determined amount of information within the area.

3. The method of claim 2 , wherein the amount of information within the area is determined based on content in the area.

4. The method of claim 3 , wherein computing the sampling density for the area further comprises:
identifying a darkest pixel and a brightest pixel in the area;
measuring a brightness difference between the darkest pixel and the brightest pixel; and
computing the sampling density for the area based on the brightness difference such that a larger brightness difference yields a higher sampling density.

5. The method of claim 3 , wherein computing the sampling density for the area further comprises:
calculating a statistical variance of brightness of pixels in the area; and
computing the sampling density for the area based on the calculated statistical variance such that a larger statistical variance yields a higher sampling density.

6. The method of claim 2 , wherein the image corresponds to a frame of a video stream, and wherein the amount of information within the area is determined based on images corresponding to previous frames of the video stream.

7. The method of claim 6 , wherein computing a sampling density for each of the one or more areas in the image further comprises:
generating a predicted image for the current frame based on the images corresponding to the previous frames of the video stream;
determining the predicted amount of information within the area based on the predicted image, wherein the amount of information within the area depends on objects located within the area of the predicted image; and
computing the sampling density for the area based on the determined amount of information within the area.

8. The method of claim 7 , wherein an object movement tracking is used for generating the predicted image for the current frame based on the images corresponding to the previous frames of the video stream.

9. The method of claim 7 , wherein an optical flow technique is used for generating the predicted image for the current frame based on the images corresponding to the previous frames of the video stream.

10. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
determine characteristics of one or more areas in an image by analyzing pixels in the image;
compute, based on the characteristics of the one or more areas, a sampling density for each of the one or more areas in the image, wherein computing the sampling density for the area comprises:
identifying a darkest pixel and a brightest pixel in the area;
measuring a distance between the darkest pixel and the brightest pixel; and
computing the sampling density for the area based on the distance such that a longer distance yields a higher sampling density;

generate samples corresponding to the image by sampling pixels in each of the one or more areas according to the associated sampling density; and
provide the samples to a machine-learning model as an input, wherein the machine-learning model is configured to reconstruct the image by processing the samples.

11. The media of claim 10 , wherein the characteristic associated with an area comprises a determined amount of information within the area.

12. The media of claim 11 , wherein the amount of information within the area is determined based on content in the area.

13. The media of claim 12 , wherein computing the sampling density for the area further comprises:
identifying a darkest pixel and a brightest pixel in the area;
measuring a brightness difference between the darkest pixel and the brightest pixel; and
computing the sampling density for the area based on the brightness difference such that a larger brightness difference yields a higher sampling density.

14. The media of claim 12 , wherein computing the sampling density for the area further comprises:
calculating a statistical variance of brightness of pixels in the area; and
computing the sampling density for the area based on the calculated statistical variance such that a larger statistical variance yields a higher sampling density.

15. The media of claim 11 , wherein the image corresponds to a frame of a video stream, and wherein the amount of information within the area is determined based on images corresponding to previous frames of the video stream.

16. The media of claim 15 , wherein computing a sampling density for each of the one or more areas in the image further comprises:
generating a predicted image for the current frame based on the images corresponding to the previous frames of the video stream;
determining the predicted amount of information within the area based on the predicted image, wherein the amount of information within the area depends on objects located within the area of the predicted image; and
computing the sampling density for the area based on the determined amount of information within the area.

17. The media of claim 16 , wherein an object movement tracking is used for generating the predicted image for the current frame based on the images corresponding to the previous frames of the video stream.

18. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
determine characteristics of one or more areas in an image by analyzing pixels in the image;
compute, based on the characteristics of the one or more areas, a sampling density for each of the one or more areas in the image, wherein computing the sampling density for the area comprises:
identifying a darkest pixel and a brightest pixel in the area;
measuring a distance between the darkest pixel and the brightest pixel; and
computing the sampling density for the area based on the distance such that a longer distance yields a higher sampling density;

generate samples corresponding to the image by sampling pixels in each of the one or more areas according to the associated sampling density; and
provide the samples to a machine-learning model as an input, wherein the machine-learning model is configured to reconstruct the image by processing the samples.

Assignees

• Meta Platforms Tech Llc

Inventors

• Patney Anjul
• Kaplanyan Anton S
• Goodall Todd

Classifications

• G06T9/004 Primary

  Predictors, e.g. intraframe, interframe coding · CPC title

• G06T9/002

  using neural networks · CPC title

• H04N19/17

  the unit being an image region, e.g. an object · CPC title

• G06N20/00

  Machine learning · CPC title

• G06T3/4023

  based on decimating pixels or lines of pixels; based on inserting pixels or lines of pixels · CPC title