Sensing light through a surface with a residue

Patent metadata

Field	Value
Publication number	US-9965869-B1
Application number	US-201715655155-A
Country	US
Kind code	B1
Filing date	Jul 20, 2017
Priority date	Jun 2, 2017
Publication date	May 8, 2018
Grant date	May 8, 2018

Abstract

In one embodiment, a light sensor, such as a camera, records an image through the surface with the residue to produce a stained image. A processor associated with the camera identifies object outlines within the image using a machine learning model, and smooth the colors within the object outlines. In another embodiment, the light sensor is placed beneath a dual-mode region of a display containing the residue. The dual-mode region can be opaque and function as part of the display, or can be transparent and allow environment light to reach the light sensor. Initially, the processor determines the position of the residue by causing the dual-mode region to display a predetermined pattern, while the light sensor records the predetermined pattern. Using the determined position of the residue, the processor corrects the pixels within the residue in the recorded image, by interpolating the values of the pixels outside of the residue.

First claim

The invention claimed is:

1. A method comprising:
placing a light sensor beneath a dual-mode region of a display, the dual-mode region comprising a transparent mode allowing environment light to reach the light sensor and an opaque mode blocking environment light from reaching the light sensor, wherein the dual-mode region contains a residue;
displaying a predetermined pattern on the dual-mode region above the light sensor;
recording a first image of the predetermined pattern by the light sensor;
determining a position of the residue by detecting a pixel in the first image that is different from a corresponding pixel in the predetermined pattern, and including a position of the pixel in the first image into the position of the residue;
recording a stained image of environment surrounding the light sensor through the dual-mode region containing the residue; and
calculating a pixel in a resulting image of environment whose position is within the position of the residue based on a pixel in the stained image whose position is outside the position of the residue.

2. The method of claim 1 , said calculating the pixel in the resulting image comprising:
determining a pixel neighborhood associated with the pixel in the stained image;
given the pixel in the stained image whose position is within the position of the residue, calculating the pixel in the resulting image by combining each pixel in the pixel neighborhood whose position is outside of the position of the residue.

3. The method of claim 2 , said combining each pixel in the pixel neighborhood comprising averaging each pixel in the pixel neighborhood.

4. The method of claim 2 , said determining the pixel neighborhood comprising:
identifying an outline associated with one or more objects in the stained image; and
determining the pixel neighborhood for the pixel in the stained image, wherein the pixel neighborhood does not contain the outline associated with the one or more objects in the stained image.

5. A system comprising:
a dual-mode region of a display containing a residue, the dual-mode region comprising a transparent mode and an opaque mode;
a light sensor disposed beneath the dual-mode region of the display, the transparent mode allowing environment light to reach the light sensor, and the opaque mode blocking environment light from reaching the light sensor;
a processor creating a resulting image of environment surrounding the light sensor, the processor:
sending a first control signal to the dual-mode region to display a predetermined pattern on the dual-mode region above the light sensor;
sending a second control signal to the light sensor to record a first image of the predetermined pattern;
sending a third control signal to the dual-mode region to become transparent;
sending a fourth control signal to the light sensor to record a stained image of environment surrounding the light sensor through the transparent dual-mode region;
recording the stained image of environment surrounding the light sensor through the dual-mode region containing the residue; and
calculating a pixel in the resulting image of environment based on a pixel in the stained image whose position is outside a position of the residue, wherein a position of the pixel in the resulting image is within the position of the residue.

6. The system of claim 5 , the predetermined pattern comprising a checkerboard pattern.

7. The system of claim 5 , the light sensor comprising at least one of a camera, an ambient light sensor, or a proximity sensor.

8. The system of claim 5 , comprising oleophobic coating applied to the dual-mode region.

9. The system of claim 5 , said processor calculating the pixel in the resulting image comprising:
said processor determining a pixel neighborhood associated with the pixel in the stained image; and
given the pixel in the stained image whose position is within the position of the residue, said processor calculating the pixel in the resulting image by combining each pixel in the pixel neighborhood whose position is outside of the position of the residue.

10. The system of claim 9 , said processor combining each pixel in the pixel neighborhood comprising:
said processor averaging each pixel in the pixel neighborhood whose position is outside of the position of the residue.

11. The system of claim 9 , said processor determining the pixel neighborhood comprising:
said processor identifying an outline associated with one or more objects in the stained image; and
said processor determining the pixel neighborhood for the pixel in the stained image, wherein the pixel neighborhood does not contain the outline associated with the one or more objects in the stained image.

Assignees

• Essential Products Inc

Inventors

• Evans V David John
• Jiang Xinrui
• Rubin Andrew E
• Hershenson Matthew
• Miao Xiaoyu
• Tate Joseph Anthony
• Gagne-Keats Jason Sean

Classifications

• G06T7/62 Primary

  of area, perimeter, diameter or volume · CPC title

• G01N21/01

  Arrangements or apparatus for facilitating the optical investigation · CPC title

• G06N99/005

  Physics · mapped topic

• G01N2021/0137

  with PC or the like · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title