System and method to process images of a video stream

Patent metadata

Field	Value
Publication number	US-11538136-B2
Application number	US-202017083146-A
Country	US
Kind code	B2
Filing date	Oct 28, 2020
Priority date	Oct 28, 2020
Publication date	Dec 27, 2022
Grant date	Dec 27, 2022

Abstract

A device includes a memory configured to store an adapted network that is configured to generate a modified image based on a single image. The device includes a processor configured to obtain, from a stream of video data, a first distorted image depicting an object, and to provide the first distorted image to the adapted network to generate a first modified image. The processor is configured to obtain, from the stream of video data, a second distorted image depicting the object, and to provide the second distorted image to the adapted network to generate a second modified image. The object is distorted differently in the second distorted image than in the first distorted image. The processor is configured to generate a video output including the first modified image and the second modified image without visible artifacts due to distortion differences between the first distorted image and the second distorted image.

First claim

What is claimed is:

1. A device configured to process images of a stream of video data, the device comprising:
a memory configured to store an adapted network that is configured to generate a modified image based on a single input image; and
a processor configured to:
obtain, from a stream of video data, a first distorted image depicting an object;
provide the first distorted image as input to the adapted network to generate a first modified image;
obtain, from the stream of video data, a second distorted image depicting the object, wherein the object is distorted differently in the second distorted image than in the first distorted image;
provide the second distorted image as input to the adapted network to generate a second modified image; and
generate a video output including the first modified image and the second modified image without visible artifacts due to distortion differences between the first distorted image and the second distorted image.

2. The device of claim 1 , wherein the distortion differences are based on spatial aliasing differences between the first distorted image and the second distorted image.

3. The device of claim 1 , wherein the distortion differences are based on downscaling aliasing artifacts caused by a sub-pixel shift of the depiction of the object in the second distorted image relative to the depiction of the object in the first distorted image.

4. The device of claim 1 , wherein the distortion differences are based on compression artifacts caused by compression used to generate the first distorted image and the second distorted image.

5. The device of claim 1 , wherein the distortion differences are caused due to hand jitter between capturing a first image corresponding to the first distorted image and capturing a second image corresponding to the second distorted image.

6. The device of claim 1 , wherein the distortion differences are caused to due to movement of the object between capturing a first image corresponding to the first distorted image and capturing a second image corresponding to the second distorted image.

7. The device of claim 1 , wherein the memory is configured to store a set of adapted networks that includes the adapted network, and wherein the processor is configured to select the adapted network from the set of adapted networks based on a selection criterion.

8. The device of claim 7 , wherein the selection criterion is based on a user, a location, a purpose, an event, or a combination thereof.

9. The device of claim 7 , wherein the selection criterion is based on default data, a configuration setting, a user input, a sensor input, or a combination thereof.

10. The device of claim 1 , wherein the first distorted image is generated by downscaling a first image to generate a downscaled image, compressing the downscaled imaged to generate a compressed image, and decompressing the compressed image.

11. The device of claim 1 , further comprising a receiver configured to receive a compressed image from a second device, wherein the processor is configured to generate the first distorted image by decompressing the compressed image.

12. The device of claim 1 , wherein the adapted network is used to generate the second modified image independently of data associated with the first distorted image.

13. The device of claim 1 , wherein the first distorted image includes first channel data corresponding to one or more first channels and second channel data corresponding to one or more second channels, and wherein the processor is further configured to:
provide the first channel data and the second channel data to the adapted network to generate first modified data, the first modified data corresponding to the one or more first channels;
generate, based on the second channel data, second modified data independently of the adapted network, the second modified data corresponding to the one or more second channels; and
generating the first modified image based on the first modified data and the second modified data.

14. The device of claim 13 , wherein the first channel data includes luma information of the first distorted image, and wherein the second channel data includes chrominance information of the first distorted image.

15. The device of claim 1 , wherein the adapted network includes a convolutional neural network (CNN), a multi-layer perceptron (MLP) neural network, or a recurrent neural network (RNN).

16. The device of claim 1 , wherein the processor is further configured to update the adapted network based at least in part on a comparison of the first modified image and a first target image, wherein the first target image is based on a first image, and wherein the first distorted image is generated by applying a distortion to the first image.

17. The device of claim 1 , wherein the adapted network has been trained using a batch of training image pairs including a particular image pair having a first target image and a particular distorted image of a plurality of distorted images, wherein the first target image is based on a first image, and wherein the plurality of distorted images is generated based on distortions applied to the first image.

18. The device of claim 1 , wherein the adapted network has been trained using a particular image pair that includes a first target image and a particular distorted image, wherein the particular distorted image is generated by applying a distortion to a first image, wherein the first target image is based on the first image, and wherein the adapted network has been trained using the particular image pair by:
providing the particular distorted image as input to the adapted network to generate a particular modified image; and
updating the adapted network based at least in part on a comparison of the particular modified image and the first target image.

19. The device of claim 1 , wherein the adapted network has been trained using a particular image pair that includes a first target image and a particular distorted image, wherein the particular distorted image is generated by applying a particular compression level to compress a first image, and wherein the first target image is based on the first image.

20. The device of claim 19 , wherein the adapted network has been trained using a batch of training image pairs that includes at least the particular image pair and a second particular image pair, wherein the second particular image pair includes a second target image and a second particular distorted image, wherein the second target image is based on the first image, wherein the second particular distorted image is generated by applying a second compression level to compress the first image, and wherein the second compression level is distinct from the particular compression level.

21. The device of claim 1 , wherein the adapted network has been trained using a particular image pair that includes a first target image and a particular distorted image, wherein the first target image is generated by applying a particular pixel shift to a first image, and wherein the particular distorted image is generated by downscaling the first target image.

22. The device of claim 21 , wherein the adapted network has been trained using a batch of training image pairs that includes at least the particular image pair and a second particular image pair, wherein the second particular image pair includes a second target image and a second particular distorted image, wherein the second target image is generated b

Assignees

• Qualcomm Inc

Inventors

• Yun Stone
• Shoa Hassani Lashdan Alireza
• Gnanapragasam Darren
• Hansen David

Classifications

• G06V20/49

  Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes · CPC title

• G06T3/4053

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title