Automated pixel error detection using an inpainting neural network

Patent metadata

Field	Value
Publication number	US-11210774-B2
Application number	US-202016836453-A
Country	US
Kind code	B2
Filing date	Mar 31, 2020
Priority date	Mar 31, 2020
Publication date	Dec 28, 2021
Grant date	Dec 28, 2021

Abstract

According to one implementation, a pixel error detection system includes a hardware processor and a system memory storing a software code. The hardware processor is configured to execute the software code to receive an input image, to mask, using an inpainting neural network (NN), one or more patch(es) of the input image, and to inpaint, using the inpainting NN, the masked patch(es) based an input image pixels neighboring each of the masked patch(es). The hardware processor is configured to further execute the software code to generate, using the inpainting NN, a residual image based on differences between the inpainted masked patch(es) and the patch(es) in the input image and to identify one or more anomalous pixel(s) in the input image using the residual image.

First claim

What is claimed is:

1. An automated pixel error detection system comprising:
a hardware processor; and
a system memory storing a software code;
the hardware processor configured to execute the software code to:
receive an input image;
mask, using an inpainting neural network (NN), one or more patches of the input image;
inpaint, using the inpainting NN, the one or more masked patches based on a plurality of input image pixels neighboring each of the one or more masked patches;
generate, using the inpainting NN, a residual image based on differences between the inpainted one or more masked patches and the one or more patches in the input image; and
identify at least one anomalous pixel in the input image using the residual image.

2. The automated pixel error detection system of claim 1 , wherein the one or more masked patches comprise a plurality of masked patches, and wherein the inpainting NN is configured to inpaint the plurality of masked patches concurrently.

3. The automated pixel error detection system of claim 1 , wherein the plurality of input image pixels neighboring each of the one or more masked patches surrounds each of the one or more masked patches.

4. The automated pixel error detection system of claim 1 , wherein the hardware processor is configured to further execute the software code to:
detect at least one anomaly candidate in the input image using the residual image,
determine a residual value associated with the at least one anomaly candidate; and
identify the at least one anomaly candidate as the at least one anomalous pixel based on comparing the residual value with a predetermined threshold residual value.

5. The automated pixel error detection system of claim 1 , wherein the input image is one of a plurality of video frames received by the automated pixel error detection system, and wherein the hardware processor is configured to further execute the software code to:
detect at least one anomaly candidate in the input image using the residual image,
perform a comparison of a location of the at least one anomaly candidate in the video frame including the input image with corresponding locations in at least one of a previous frame or a next frame of the plurality of video frames; and
identify the at least one anomaly candidate as the at least one anomalous pixel based on the comparison.

6. The automated pixel error detection system of claim 1 , wherein the hardware processor is configured to further execute the software code to:
detect a plurality of anomaly candidates in the input image using the residual image, cluster the plurality of anomaly candidates based on a location of each of the plurality of anomaly candidates in the input image, resulting in at least one anomaly candidate cluster; and
identify the at least one anomalous pixel based on a geometry of the at least one anomaly candidate cluster.

7. The automated pixel error detection system of claim 1 , wherein the hardware processor is configured to further execute the software code to:
detect a first plurality of anomaly candidates in the input image using the residual image,
determine a residual value associated with each of the first plurality of anomaly candidates;
disregard any anomaly candidates associated with a residual value less than a predetermined threshold residual value, resulting in a second plurality of anomaly candidates;
cluster the second plurality of anomaly candidates based on a location of each of the second plurality of anomaly candidates in the input image, resulting in at least one anomaly candidate cluster; and
identify the at least one anomalous pixel based on a geometry of the at least one anomaly candidate cluster.

8. The automated pixel error detection system of claim 1 , wherein the inpainting NN comprises a plurality of partial convolutional encoder layers and a plurality of partial convolutional decoder layers.

9. The automated pixel error detection system of claim 1 , wherein the hardware processor is configured to further execute the software code to generate an error markup image corresponding to the input image, the error markup image indicating a location of the at least one anomalous pixel in the error markup image.

10. The automated pixel error detection system of claim 1 , wherein the hardware processor is configured to further execute the software code to correct the input image using a respective one of the inpainted one or more masked patches corresponding to a location of the at least one anomalous pixel in the input image.

11. A method for use by an automated pixel error detection system including a hardware processor and a system memory storing a software code, the method comprising:
receiving, by the software code executed by the hardware processor, an input image;
masking, by the software code executed by the hardware processor and using an inpainting neural network (NN), one or more patches of the input image;
inpainting, by the software code executed by the hardware processor and using the inpainting NN, the one or more masked patches based on a plurality of input image pixels neighboring each of the one or more masked patches;
generating, by the software code executed by the hardware processor and using the inpainting NN, a residual image based on differences between the inpainted one or more masked patches and the one or more patches in the input image; and
identifying, by the software code executed by the hardware processor, at least one anomalous pixel in the input image using the residual image.

12. The method of claim 11 , wherein the one or more masked patches comprise a plurality of masked patches, and wherein the inpainting NN is configured to inpaint the plurality of masked patches concurrently.

13. The method of claim 11 , wherein the plurality of input image pixels neighboring each of the one or more masked patches surrounds each of the one or more masked patches.

14. The method of claim 11 , further comprising:
detecting, by the software code executed by the hardware processor, at least one anomaly candidate in the input image using the residual image,
determining, by the software code executed by the hardware processor, a residual value associated with the at least one anomaly candidate; and
identifying the at least one anomaly candidate as the at least one anomalous pixel based on comparing the residual value with a predetermined threshold residual value.

15. The method of claim 11 , wherein the input image is one of a plurality of video frames received by the automated pixel error detection system, the method further comprising:
detecting, by the software code executed by the hardware processor, at least one anomaly candidate in the input image using the residual image,
performing a comparison, by the software code executed by the hardware processor, of a location of the at least one anomaly candidate in the video frame including the input image with corresponding locations in at least one of a previous frame or a next frame of the plurality of video frames; and
identifying the at least one anomaly candidate as the at least one anomalous pixel based on the comparison.

16. The method of claim 11 , the method further comprising:
detecting, by the software code executed by the hardware processor, a plurality of anomaly candidates in the input image using the residual image,
clustering, by the software code executed by the hardware processor, the plurality of anomaly candidates based on a location of each of the plurality of anomaly candidates in the input image, resulting in at least one anomaly candidate cl

Assignees

• Disney Entpr Inc
• Eth Zuerich
• Eth Zuerich Eidgenoessische Technische Hochschule Zuerich

Inventors

• Schroers Christopher Richard
• Djelouah Abdelaziz
• Wang Sutao
• Doggett Erika Varis

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06T9/002

  using neural networks · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title