Systems and methods that leverage deep learning to selectively store images at a mobile image capture device

Patent metadata

Field	Value
Publication number	US-9836484-B1
Application number	US-201514984683-A
Country	US
Kind code	B1
Filing date	Dec 30, 2015
Priority date	Dec 30, 2015
Publication date	Dec 5, 2017
Grant date	Dec 5, 2017

Abstract

The present disclosure provides an image capture, curation, and editing system that includes a resource-efficient mobile image capture device that continuously captures images. The mobile image capture device is operable to input an image into at least one neural network and to receive at least one descriptor of the desirability of a scene depicted by the image as an output of the at least one neural network. The mobile image capture device is operable to determine, based at least in part on the at least one descriptor of the desirability of the scene of the image, whether to store a second copy of such image in a non-volatile memory of the mobile image capture device or to discard a first copy of such image from a temporary image buffer without storing the second copy of such image in the non-volatile memory.

First claim

What is claimed is:

1. A continuous capture mobile image capture device designed for resource efficiency, the mobile image capture device comprising:
a network interface;
a power source;
an image sensor;
at least one processor; and
a memory that stores a plurality of deep neural networks usable to determine a desirability of a scene depicted by an image, the memory comprising at least a temporary image buffer and a non-volatile memory;
wherein the plurality of deep neural networks comprise one or more of:
a face detection deep neural network that detects a presence of one or more faces in the scene of each input image;
a face recognition deep neural network that matches one or more faces in the scene of each input image to one or more other faces;
a face attributes deep neural network that detects various facial characteristics of one or more faces in the scene of each input image;
an image content deep neural network that outputs one or more semantic labels that describe the scene of each input image; and
a photo quality deep neural network that outputs a photo score that describes various photographic quality characteristics of each input image; and

wherein the mobile image capture device is operable to:
capture a first image that depicts a scene;
maintain a first copy of the first image in the temporary image buffer;
input the first image into at least one of the plurality of deep neural networks;
receive at least one descriptor of the desirability of the scene depicted by the first image as an output from the at least one of the plurality of deep neural networks into which the first image is input; and
determine, based at least in part on the at least one descriptor of the desirability of the scene of the first image, whether to store a second copy of the first image in the non-volatile memory of the mobile image capture device or to discard the first copy of the first image without storing a second copy of the first image.

2. The mobile image capture device of claim 1 , wherein the mobile image capture device is further operable to:
operate the image sensor in a plurality of different capture modes that respectively correspond to a plurality of different resolutions and frame rates,
select, based at least in part on the at least one descriptor of the desirability of the scene of the first image, one of the plurality of different capture modes; and
switch operation of the image sensor to the selected capture mode.

3. The mobile image capture device of claim 1 , wherein the mobile image capture device is further operable to select the at least one of the plurality of deep neural networks into which the first image is input.

4. The mobile image capture device of claim 1 , wherein the plurality of deep neural networks comprise a plurality of feed-forward deep neural networks.

5. The mobile image capture device of claim 1 , wherein the plurality of deep neural networks comprise at least one convolutional neural network.

6. The mobile image capture device of claim 1 , wherein each of the plurality of deep neural networks outputs at least one annotation for each input image, the at least one annotation for each input image indicative of the desirability of the scene depicted by such image.

7. The mobile image capture device of claim 1 , wherein the plurality of deep neural networks comprise a multi-headed deep neural network that receives a single set of inputs and provides a plurality of outputs, wherein the plurality of outputs respectively include a plurality of descriptors of the desirability of the scene of each input image.

8. The mobile image capture device of claim 1 , wherein the mobile image capture device is further operable to, prior to inputting the first image into the at least one of the plurality of deep neural networks:
input the first image into a recurrent deep neural network that analyzes only a portion of the first image;
receive as output from the recurrent deep neural network an indication of whether the first image should be input into the at least one of the plurality of deep neural networks, for further analysis, such that die recurrent deep neural network operates as a prefilter to the plurality of deep neural networks; and
receive as output from the recurrent deep neural network a description of which portion of a next image the recurrent deep neural network should analyze.

9. The mobile image capture device of claim 1 , wherein the mobile image capture device is further operable to receive data that describes a set of entities having an elevated importance to a user of the mobile image capture device, wherein the at least descriptor output by the at least one of the plurality of deep neural networks indicates whether a member of the set of entities is depicted in the scene of the first image.

10. The mobile image capture device of claim 1 , wherein the mobile image capture device is further operable to:
receive, from another mobile image capture device that is proximately located, an excitement signal that indicates that the other mobile image capture device has recently captured an image that depicts a desirable scene; and
determine, based at least in part on the excitement signal, Whether to store a second Copy of a recently captured image in the non-volatile memory of the mobile image capture device or to discard a first copy of such image from the temporary image buffer without storing a second copy of such image in the non-volatile memory.

11. A resource-efficient mobile image capture device that, at least in operation, continuously captures imagery, the mobile image capture device comprising:
a network interface;
a power source;
an image sensor;
at least one processor;
a memory; and
a scene analyzer that includes:
at least one neural network that receives a first image captured by the image sensor and outputs at least one descriptor of a desirability of a scene depicted by the first image;
a save controller that determines, based at least in part on the at least one descriptor of the desirability of the scene of the first image, whether to store a second copy of the first image in the memory of the mobile image capture device or to discard the first image without storing a second copy of the first image; and
a mode controller that:
selects, based at least in part on the at least one descriptor of the desirability of the scene of the first image, one of a plurality of different capture modes of the mobile image capture device, wherein the plurality of different capture modes respectively correspond to a plurality of different resolutions and frame rates; and
switches operation of the image sensor to the selected capture mode.

12. The mobile image capture device of claim 11 , wherein the scene analyzer further comprises an attention model that:
analyzes only a portion of the first image;
outputs an indication, based at least in part on the analysis of the portion of the first image, whether the first image should be further analyzed in its entirety; and
outputs a description of which portion of a second image the attention model should analyze, the second image temporally subsequent to the first image.

13. The mobile image capture device of claim 11 , wherein:
the scene analyzer comprises a plurality of multi-layer non-linear models; and
the scene analyzer further comprises a model selector that selects at least one of the multi-layer non-linear models into which the first image is input.

14. The mobile image capture device of claim 11 , wherein the scene analyzer comprises a multi-headed deep neural network that receives a single set of inputs a

Assignees

• Google Inc
• Google Llc

Inventors

• Bialynicka-Birula Iwona
• Aguera-Arcas Blaise
• Ramage Daniel
• Mcmahan Hugh Brendan
• Lange Oliver Fritz
• Fortuna Emily Anne
• Tyamagundlu Divya
• Holbrook Jess
• Kohlhepp Kristine
• Payne Juston
• Duleba Krzysztof
• Vanik Benjamin
• Lentz Alison
• Clapper Jon Gabriel
• Lovejoy Joshua Denali
• Donsbach Aaron Michael

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V40/167 Primary

  using comparisons between temporally consecutive images · CPC title

• G06F16/51

  Indexing; Data structures therefor; Storage structures · CPC title

• H04N23/632

  for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title