Intelligent camera

Patent metadata

Field	Value
Publication number	US-10937216-B2
Application number	US-201916265871-A
Country	US
Kind code	B2
Filing date	Feb 1, 2019
Priority date	Nov 1, 2017
Publication date	Mar 2, 2021
Grant date	Mar 2, 2021

Abstract

Presented here is technology to efficiently process camera images to generate lead part artistic images and videos using an artificial intelligence module receiving inputs from multiple sensors. Multiple sensors can include a depth sensor, a conventional camera, and a motion tracker providing inputs to the artificial intelligence module. Based on the inputs, the artificial intelligence module can segment the received image and/or video into a foreground image and a background image to produce portrait imagery by blurring the background image and/or video. The artificial intelligence module can select the most aesthetically pleasing image from a video. In addition, the artificial intelligence module can adjust lighting in an image or video to create artistic lighting effects. All the processing can be done in real time due to efficient combination of artificial intelligence modules, traditional image processing techniques, and use of specialized hardware.

First claim

The invention claimed is:

1. A method comprising:
receiving an image from a camera and a depth information associated with the image;
identifying an entity in the image and a shadow associated with the entity in the image;
determining a quality of a lighting of the image based on the entity and the shadow associated with the entity in the image;
when the quality of the lighting of the image is low, adjusting the shadow associated with the entity in the image, said adjusting comprising:
constructing a depth representation of the image having a depth based on the depth information;
based on the entity obtaining a surfacing software associated with the entity, wherein the surfacing software describes interaction of light with a surface of the entity;
creating a light source illuminating the depth representation of the image and producing light interacting with the surfacing software; and
producing a new image based on the depth representation of the image, the surfacing software and the light source interacting with the depth representation of the image and the surfacing software.

2. A method comprising:
receiving an image from an imaging device;
identifying an entity in the image and a shadow associated with the entity in the image;
determining a quality of a lighting of the image based on the entity and the shadow associated with the entity in the image; and
when the quality of the lighting of the image is low, adjusting the shadow associated with the entity in the image by:
obtaining a depth information associated with the image;
constructing a depth representation of the image having a depth based on the depth information;
based on the entity obtaining a surfacing software associated with the entity, wherein the surfacing software describes interaction of light with a surface of the entity;
creating a light source illuminating the depth representation of the image and producing light interacting with the surfacing software; and
producing a new image based on the depth representation of the image, the surfacing software and the light source interacting with the depth representation of the image and the surfacing software.

3. The method of claim 2 , said determining the quality of the image comprising:
determining whether the shadow associated with the entity is sharp; and
when the shadow associated with the entity is sharp, creating a transition region between a non-shadow region and the shadow associated with the entity.

4. The method of claim 3 , said determining whether the shadow associated with the entity is sharp comprising:
determining a size of a darkest region of the shadow and a size of a lighter region of the shadow;
when the size of the lighter region of the shadow compared to the size of the darkest region of the shadow is below a predetermined threshold, determining that the shadow associated with the entity is sharp.

5. The method of claim 2 , said determining the quality of the image comprising:
determining whether the shadow associated with the entity indicates a shape of the entity; and
when the shadow associated with the entity does not indicate that the shape of the entity, adjusting the shadow associated with the image.

6. The method of claim 2 , said creating the light source comprising:
determining a second entity within the image proximate to a visual focus of the image;
creating a plurality of lights surrounding the second entity.

7. The method of claim 2 , said creating the light source comprising:
obtaining a type of lighting to apply to the image;
obtaining the light source associated with the type of lighting and a position of the light source; and
creating the light source at the obtained position.

8. The method of claim 7 , said obtaining the type of lighting comprising:
automatically determining the type of lighting based on the entity contained in the image, a location of the imaging device, or a time of day when the image was recorded.

9. The method of claim 2 , wherein the image comprises a time-dependent visual representation, the method comprising:
performing a low-pass filter of time varying attributes of the time-dependent visual representation.

10. A system comprising:
an imaging device to record a visual representation of a scene;
a processor configured to:
receive the visual representation from the imaging device;
identify an entity in the visual representation and a shadow associated with the entity in the visual representation;
determine whether a quality of a lighting of the visual representation is satisfactory based on the entity and the shadow associated with the entity in the visual representation; and
when the quality of the lighting of the visual representation is not satisfactory, adjust the shadow associated with the entity in the visual representation by:
obtaining a depth information associated with the visual representation;
based on the depth information, constructing a depth representation of the visual representation having a depth;
based on the entity, obtaining a surfacing software associated with the entity, wherein the surfacing software describes interaction of light with a surface of the entity;
creating a light source illuminating the depth representation of the visual representation and producing light interacting with the surfacing software; and
producing a new visual representation based on the depth representation of the visual representation, the surfacing software and the light source interacting with the depth representation of the visual representation and the surfacing software.

11. The system of claim 10 , the processor configured to:
determine whether the shadow associated with the entity is sharp; and
when the shadow associated with the entity is sharp, create a transition region between a non-shadow region and the shadow associated with the entity.

12. The system of claim 11 , the processor configured to determine whether the shadow associated with the entity is sharp comprising the processor configured to:
determine a size of a darkest region of the shadow and a size of a lighter region of the shadow;
when the size of the lighter region of the shadow compared to the size of the darkest region of the shadow is below a predetermined threshold, determine that the shadow associated with the entity is sharp.

13. The system of claim 10 , the processor configured to determine the quality of the visual representation comprising the processor configured to:
determine whether the shadow associated with the entity indicates a shape of the entity; and
adjust the shadow associated with the visual representation.

14. The system of claim 10 , the processor configured to create the light source comprising the processor configured to:
determine a second entity within the visual representation proximate to a visual focus of the visual representation; and
create a plurality of lights surrounding the second entity.

15. The system of claim 10 , the processor configured to create the light source comprising the processor configured to:
obtain a type of lighting to apply to the visual representation;
obtain the light source associated with the type of lighting and a position of the light source; and
create the light source at the obtained position.

16. The system of claim 15 , the processor configured to obtain the type of lighting comprising the processor configured to:
automatically determine the type of lighting based on the entity contained in the visual representation, a location of the imaging device, or a time of day when the visual representa

Assignees

• Essential Products Inc

Inventors

• Miao Xiaoyu
• Rubin Andrew E

Classifications

• G06V10/82 Primary

  using neural networks · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title