Intelligent camera

Patent metadata

Field	Value
Publication number	US-10339689-B2
Application number	US-201816177013-A
Country	US
Kind code	B2
Filing date	Oct 31, 2018
Priority date	Nov 1, 2017
Publication date	Jul 2, 2019
Grant date	Jul 2, 2019

Abstract

Presented here is technology to efficiently process camera images to generate artistic images and videos using an artificial intelligence module receiving inputs from multiple sensors. Multiple sensors can include a depth sensor, a conventional camera, and a motion tracker providing inputs to the artificial intelligence module. Based on the inputs, the artificial intelligence module can segment the received image and/or video into a foreground image and a background image to produce portrait imagery by blurring the background image and/or video. The artificial intelligence module can select the most aesthetically pleasing image from a video. In addition, the artificial intelligence module can adjust lighting in an image or video to create artistic lighting effects. All the processing can be done in real time due to efficient combination of artificial intelligence modules, traditional image processing techniques, and use of specialized hardware.

First claim

The invention claimed is:

1. A method to create aesthetically pleasing images in real time with a cell phone camera, the method comprising:
obtaining from a camera a plurality of images from a plurality of viewpoints of an environment surrounding the camera, the plurality of images obtained by continually moving the camera through the plurality of viewpoints;
selecting within a centisecond an aesthetically pleasing image with a highest aesthetic score from the plurality of images by using a neural network trained to assign an aesthetic score to each image in the plurality of images, the neural network running on a processor optimized to execute operations associated with the neural network, and the neural network comprising a plurality of layers arranged sequentially, each layer in the plurality of layers comprising a plurality of nodes performing a plurality of computations in parallel said selecting within the centisecond comprising:
measuring an amount of time associated with selecting the aesthetically pleasing image with the highest aesthetic score;
when the amount of time exceeds a predetermined amount of time, distributing a first plurality of nodes associated with a first layer in the plurality of layers across multiple processors associated with a cell phone until the amount of time is below the predetermined amount of time; and

displaying a visual notification along with the aesthetically pleasing image on a viewfinder associated with the camera, the visual notification indicating to a user to record the aesthetically pleasing image.

2. A method comprising:
obtaining from a light sensor a plurality of images from a plurality of viewpoints of an environment surrounding the light sensor, the plurality of images obtained by continually moving the light sensor through the plurality of viewpoints;
selecting within a specified amount of time an aesthetically pleasing image with a highest aesthetic score from the plurality of images by using an artificial intelligence module trained to assign an aesthetic score to each image in the plurality of images, wherein the artificial intelligence module comprises a plurality of layers arranged sequentially, each layer in the plurality of layers comprising a plurality of nodes performing a plurality of computations in parallel;
measuring an amount of time associated with selecting the aesthetically pleasing image with the highest aesthetic score; and
when the amount of time exceeds a predetermined amount of time, distributing a first plurality of nodes associated with a first layer in the plurality of layers across multiple processors associated with a cell phone until the amount of time is below the predetermined amount of time.

3. The method of claim 2 , wherein the plurality of images comprises an ordered sequence of images from an ordered sequence of viewpoints;
obtaining an amount of time for the artificial intelligence module to assign the aesthetic score to an initial image in the ordered sequence of images; and
when a number of images in the ordered sequence of images combined with the amount of time exceeds the specified amount of time, achieving the selection within the specified amount of time by dropping a subset of images in the ordered sequence of images to obtain faster processing.

4. The method of claim 2 , wherein the plurality of images comprises an ordered sequence of images from an ordered sequence of viewpoints;
obtaining an amount of time for the artificial intelligence module to assign the aesthetic score to an initial image in the ordered sequence of images; and
when a number of images in the ordered sequence of images combined with the amount of time exceeds the specified amount of time, achieving the selection within the specified amount of time by utilizing an additional processor associated with the light sensor.

5. The method of claim 2 , comprising:
determining a presence of a vanishing point associated with an image in the plurality of images by detecting converging lines in the image;
determining a presence of a foreground object associated with the image; and
assigning a high aesthetic score to the image where the foreground object and the vanishing point are proximate to each other and to a center of the image.

6. The method of claim 2 , comprising:
tracking a motion of a plurality of objects associated with the plurality of images;
detecting an object in the plurality of objects with a least amount of motion; and
assigning a high aesthetic score to an image in the plurality of images where the object with the least amount of motion is proximate to a center of the image.

7. The method of claim 2 , comprising:
segmenting the plurality of images into a foreground object and a background object;
determining a location of the foreground object within each image in the plurality of images; and
assigning a high aesthetic score to a first image in the plurality of images where the foreground object is proximate to a center of the image or to a second image in the plurality of images were the foreground object is proximate to an edge of the image and substantially symmetric about the center of the image.

8. The method of claim 2 , comprising:
obtaining from a depth sensor substantially collocated with the light sensor a plurality of depth measurements from the plurality of viewpoints of the environment surrounding the depth sensor, the plurality of viewpoints obtained by continually moving the depth sensor through the plurality of viewpoints, wherein each depth measurement in the plurality of depth measurements corresponds to an image in the plurality of images;
providing the plurality of depth measurements in the plurality of images to the artificial intelligence module running on a dedicated processor and trained to assign the aesthetic score to each image in the plurality of images based on the plurality of depth measurements in the plurality of images; and
selecting within the specified amount of time the aesthetically pleasing image with the highest aesthetic score from the plurality of images using the artificial intelligence module.

9. The method of claim 2 , comprising:
obtaining from a motion tracking sensor substantially collocated with the light sensor a plurality of motion tracking data associated with a plurality of objects in the plurality of images;
providing the plurality of motion tracking data and the plurality of images to the artificial intelligence module running on a dedicated processor and trained to assign the aesthetic score to each image in the plurality of images based on the plurality of motion tracking data and the plurality of images; and
selecting within the specified amount of time the aesthetically pleasing image with the highest aesthetic score from the plurality of images using the artificial intelligence module.

10. The method of claim 2 , wherein the plurality of images comprises a live feed from the light sensor;
selecting the aesthetically pleasing image from the live feed; and
providing a visual notification along with the aesthetically pleasing image on a display associated with the light sensor, the visual notification indicating to a user to record the aesthetically pleasing image.

11. A system comprising:
a light sensor to record a plurality of images from a plurality of viewpoints of an environment surrounding the light sensor, the plurality of images obtained by continually moving the light sensor through the plurality of viewpoints;
an artificial intelligence module trained to assign an aesthetic score to each image in the plurality of images, the artificial intelligence module to receive the plurality of images and to select within a specifie

Assignees

• Essential Products Inc

Inventors

• Miao Xiaoyu
• Rubin Andrew E

Classifications

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06F18/24143

  Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN] · CPC title