Systems and methods for selective retention and editing of images captured by mobile image capture device

Patent metadata

Field	Value
Publication number	US-10732809-B2
Application number	US-201815913793-A
Country	US
Kind code	B2
Filing date	Mar 6, 2018
Priority date	Dec 30, 2015
Publication date	Aug 4, 2020
Grant date	Aug 4, 2020

Abstract

The present disclosure provides an image capture, curation, and editing system that includes a resource-efficient mobile image capture device that continuously captures images. The mobile image capture device is operable to input an image into at least one neural network and to receive at least one descriptor of the desirability of a scene depicted by the image as an output of the at least one neural network. The mobile image capture device is operable to determine, based at least in part on the at least one descriptor of the desirability of the scene of the image, whether to store a second copy of such image and/or one or more contemporaneously captured images in a non-volatile memory of the mobile image capture device or to discard a first copy of such image from a temporary image buffer without storing the second copy of such image in the non-volatile memory.

First claim

What is claimed is:

1. A system for selective retention of images, the system comprising:
a mobile image capture device comprising a network interface, a power source, an image sensor, at least one processor, and a memory that stores a machine-learned face recognition model usable to recognize faces depicted by images, the memory comprising at least a temporary image buffer and a non-volatile memory, the memory storing data descriptive of a set of faces that have an elevated importance to a user of the mobile image capture device, the mobile image capture device configured to:
capture a first image that depicts one or more faces;
maintain a first copy of the first image in the temporary image buffer;
input the first image into the machine-learned face recognition model;
receive an output of the machine-learned face recognition model that describes the one or more faces depicted by the first image and recognized by the machine-learned face recognition model;
determine, based at least in part on the output of the machine-leafed face recognition model, whether at least one of the one or more faces depicted by the first image is included in the set of faces that have elevated importance to the user; and
determine, based at least in part on whether at least one of the one or more faces depicted by the first image is included in the set of faces that have elevated importance to the user, whether to store a second copy of the first image in the non-volatile memory of the mobile image capture device or to discard the first copy of the first image from the temporary image buffer without storing a second copy of the first image in the non-volatile memory.

2. The system of claim 1 , wherein:
the output of the machine-learned face recognition model comprises one or more embeddings respectively associated with the one or more faces depicted by the first image; and
to determine whether at least one of the one or more faces depicted by the first image is included in the set of faces that have elevated importance to the user, the mobile image capture device is configured to respectively compare the embedding associated with each of the one or more faces to each of a set of embeddings respectively associated with the set of faces that have elevated importance to the user.

3. The system of claim 1 , wherein the machine-learned face recognition model comprises a deep neural network.

4. The system of claim 1 , wherein:
the memory of the mobile image capture device further stores a machine-learned face tracking model usable to perform face tracking across a plurality of input images and provide a bounding box descriptive of a location of a tracked face within a given input image of the plurality of input images; and
to input the first image into the machine-learned face recognition model, the mobile image capture device is configured to:
extract one or more face crops from the first image based on one or more first bounding boxes output by the machine-learned face tracking model for the first image; and
input the one or more face crops into the machine-learned face recognition model.

5. The system of claim 1 , wherein the mobile image capture device is further configured to automatically populate the set of faces that have elevated importance to the user with one or more additional faces that are frequently recognized by the machine-learned face recognition model within images captured by the mobile image capture device.

6. The system of claim 1 , wherein at least some of the faces included in the set of faces that have elevated importance to the user have been selected by the user of the mobile image capture device from existing images.

7. The system of claim 1 , wherein the mobile image capture device is further configured to:
receive a plurality of existing images associated with the user;
identify one or more additional faces that are frequently recognized by the machine-learned face recognition model within the plurality of existing images; and
automatically add the one or more additional faces to the set of faces that have elevated importance to the user.

8. The system of claim 1 , wherein, when at least one of the one or more faces depicted by the first image is included in the set of faces that have elevated importance to the user, the mobile image capture device is configured to increase a probability that the second copy of the first image is stored in the non-volatile memory.

9. The system of claim 1 , wherein, when at least one of the one or more faces depicted by the first image is included in the set of faces that have elevated importance to the user, the mobile image capture device is configured to:
automatically determine a recommended composition for the first image based at least in part on a location associated with the at least one face that is included in the set of faces, wherein the recommended composition for the first image comprises a suggested crop of the first image that includes the at least one face that is included in the set of faces.

10. The system of claim 1 , wherein, when at least one of the one or more faces depicted by the first image is not included in the set of faces that have elevated importance to the user, the mobile image capture device is further configured to:
automatically determine a recommended composition for the first image, wherein the recommended composition comprises at least one of a blurring of or a cropping to exclude the at least one face that is not included in the set of faces.

11. The system of claim 1 , further comprising a user computing device communicatively coupled to the mobile image capture device, wherein the user computing device is configured to provide an interactive user interface that enables the user to edit the set of faces that have an elevated importance to the user.

12. The system of claim 1 , further comprising a user computing device, the user computing device configured to:
selectively communicatively connect to the mobile image capture device;
select one or more of a plurality of images stored in the non-volatile memory of the user computing device for transfer, wherein images that depict faces included in the set of faces that have elevated importance to the user are preferentially selected for transfer relative to images that do not depict faces included in the set of faces;
receive from the mobile image capture device the selected one or more images; and
provide a user interface that enables a user to edit the one or more images received from the mobile image capture device.

13. The system of claim 1 , wherein the mobile image capture device is configured to:
maintain data sufficient to generate a high resolution copy of the first image in a temporary raw data buffer;
input a low resolution copy of the first image into the machine-learned face recognition model; and
determine, based at least in part on the output of the machine-learned face recognition model, whether to store a high resolution copy of the first image in the non-volatile memory of the mobile image capture device or to discard the first copy of the first image from the temporary image buffer without storing a high resolution copy of the first image in the non-volatile memory.

14. An image capture device, comprising:
one or more processors;
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the image capture device to perform operations, the operations comprising:
capturing a first image that depicts one or more faces;
storing the first image in a volatile memory of the image captur

Assignees

• Google Llc

Inventors

• Bialynicka-Birula Iwona
• Aguera-Arcas Blaise
• Ramage Daniel
• Mcmahan Hugh Brendan
• Lange Oliver Fritz
• Fortuna Emily Anne
• Tyamagundlu Divya
• Holbrook Jess
• Kohlhepp Kristine
• Payne Juston
• Duleba Krzysztof
• Vanik Benjamin
• Lentz Alison
• Clapper Jon Gabriel
• Lovejoy Joshua Denali
• Donsbach Aaron Michael

Classifications

• G06V40/161

  Detection; Localisation; Normalisation · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06V20/35

  Categorising the entire scene, e.g. birthday party or wedding scene · CPC title

• G06V20/30

  in albums, collections or shared content, e.g. social network photos or video · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title