Low power framework for processing, compressing, and transmitting images at a mobile image capture device

Patent metadata

Field	Value
Publication number	US-9838641-B1
Application number	US-201514984825-A
Country	US
Kind code	B1
Filing date	Dec 30, 2015
Priority date	Dec 30, 2015
Publication date	Dec 5, 2017
Grant date	Dec 5, 2017

Abstract

The present disclosure provides an image capture, curation, and editing system that includes a resource-efficient mobile image capture device that continuously captures images. In particular, the present disclosure provides low power frameworks for processing, compressing, and transmitting images at a mobile image capture device. One example low power framework includes a scene analyzer that analyzes a scene depicted by a first image and determines whether to store the first image in a non-volatile memory or to discard the first image from a temporary image buffer without storing the first image in the non-volatile memory.

First claim

What is claimed is:

1. An image capture device that has a low-power framework for processing imagery, the image capture device comprising:
an image sensor that provides high resolution raw image data;
a temporary raw data buffer that temporarily stores the high resolution raw image data;
a down sampler that down samples the high resolution raw image data to output low resolution raw image data;
an imaging pipeline that streams the low resolution raw image data from the down sampler and processes the low resolution raw image data to generate low resolution versions of one or more first images;
a temporary image buffer that temporarily stores the low resolution versions of the one or more first images;
a scene analyzer that analyzes at least a portion of a scene depicted by at least one of the low resolution versions one or more first images and determines whether to store the one or more first images in a non-volatile memory or to discard the low resolution versions of the one or more first images from the temporary image buffer without storing the one or more first images in the non-volatile memory;
an image compression component that compresses the one or more first images when the scene analyzer determines that the one or more first images should be stored in the non-volatile memory; and
the non-volatile memory to which the one or more first images are written after compression.

2. The image capture device of claim 1 , wherein, when the scene analyzer determines that the one or more first images should be stored in the non-volatile memory:
the image compression component compresses the low resolution versions of the one or more first images; and
the image capture device writes the low resolution versions of the one or more first images to the non-volatile memory after compression.

3. The image capture device of claim 1 , wherein, in a base capture mode of operation, the image sensor periodically transitions between providing high resolution raw image data and low resolution raw image data.

4. The image capture device of claim 1 , wherein, when the scene analyzer determines that the one or more first images should be stored in the non-volatile memory:
the imaging pipeline receives the high resolution raw image data from the temporary raw data buffer and processes the high resolution raw image data to generate high resolution versions of the one or more first images;
the image compression component compresses the high resolution versions of the one or more first images; and
the image capture device writes the high resolution versions of the one or more first images to the non-volatile memory after compression.

5. The image capture device of claim 4 , wherein, when the scene analyzer determines that the one or more first images should be stored in the non-volatile memory, the low resolution versions of the one or more first images are also compressed and written to the non-volatile memory.

6. The image capture device of claim 1 , wherein, in at least a second mode of operation:
the imaging pipeline streams the raw image data from the image sensor and processes the raw image data to generate a second image;
the second image is provided to the image compression component for compression and then storage at the non-volatile memory without being analyzed by the scene analyzer.

7. The image capture device of claim 6 , wherein the second mode of operation comprises a burst mode of operation in which the image capture device captures high resolution images at a high frame rate and performs imaging pipeline operations, compression, and storage at the non-volatile memory for all of the captured high resolution images.

8. The image capture device of claim 6 , wherein the second mode of operation comprises a hyper-lapse mode of operation in which the image capture device captures mid resolution images at a high frame rate and performs imaging pipeline operations, compression, and storage at the non-volatile memory for all of the captured mid resolution images.

9. The image capture device of claim 1 , wherein the scene analyzer comprises:
at least one neural network that receives the at least one of the one or more first images as an input and outputs at least one descriptor of a desirability of the scene depicted by at least one of the one or more first images; and
a save controller that determines, based at least in part on the at least one descriptor, whether to store the one or more first images in the non-volatile memory or to discard the one or more first images from the temporary image buffer without storing the one or more first images in the non-volatile memory.

10. The image capture device of claim 1 , wherein the image capture device comprises a mobile image capture device that, at least in operation, continuously captures and processes imagery.

11. A method for processing imagery, the method comprising:
outputting, by an image sensor of an image capture device, high resolution raw image data;
temporarily storing, by a temporary raw data buffer of the image capture device, the high resolution raw image data; and
down sampling the high resolution raw image data to output low resolution raw image data;
processing, by the imaging pipeline, the low resolution raw image data to generate a low resolution version of a first image;
temporarily storing, by a temporary image buffer of the image capture device, the low resolution version of the first image;
analyzing a scene depicted by the low resolution version of the first image;
determining based at least in part on the analysis of the scene depicted by the low resolution version of the first image whether to store the first image in the non-volatile memory or to discard the low resolution version of the first image from the temporary image buffer without storing the first image in the non-volatile memory; and
when it is determined that the first image should be stored in the non-volatile memory:
compressing the first image; and
writing the first image to the non-volatile memory after compression.

12. The method of claim 11 , wherein, when it is determined that the first image should be stored in the non-volatile memory:
compressing the first image comprises compressing the low resolution version of the first image; and
writing the first image to the non-volatile memory comprises writing the low resolution version of the first image to the non-volatile memory after compression.

13. The method of claim 11 , further comprising, when it is determined that the first image should be stored in the non-volatile memory:
receiving, by the imaging pipeline; the high resolution raw image data from the temporary raw data buffer;
processing, by the imaging pipeline, the high resolution raw image data to generate a high resolution version of the first image;
wherein compressing the first image comprises compressing the high resolution version of the first image; and
wherein writing the first image to the non-volatile memory comprises writing the high resolution version of the first image to the non-volatile memory after compression.

14. The method of claim 11 , wherein; in at least one mode of operation, the method comprises:
outputting, by the image sensor of an image capture device, low resolution raw image data rather than high resolution raw image data.

15. The method of claim 11 , further comprising:
streaming, by the imaging pipeline, a second set of raw image data from the image sensor;
processing, by the imaging pipeline, the second set of raw image data to generate a second image; and
without providing the second image to the sce

Assignees

• Google Inc
• Google Llc

Inventors

• Lim Suk Hwan
• Aguera-Arcas Blaise

Classifications

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82

  using neural networks · CPC title

• H04N23/667

  Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes · CPC title

• H04N23/65

  Control of camera operation in relation to power supply · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title