Image processing techniques for self-captured images

Patent metadata

Field	Value
Publication number	US-9911237-B1
Application number	US-201615072910-A
Country	US
Kind code	B1
Filing date	Mar 17, 2016
Priority date	Mar 17, 2016
Publication date	Mar 6, 2018
Grant date	Mar 6, 2018

Abstract

Image processing techniques for self-captured images are disclosed. An image can be captured during activation of an illumination element of a mobile device. Presence of a representation of the mobile device can be determined in the captured image based at least in part upon locating a reflection of an illumination produced by the illumination element. Dimensions of the representation of the mobile device can be determined and compared to actual dimensions of the mobile device to provide a scaling factor. The scaling factor can be used to determine the size of various objects, including articles of clothing, in the image.

First claim

What is claimed is:

1. A computer-implemented method for processing images, comprising:
under the control of one or more computer systems configured with executable instructions,
activating a flash element of a mobile device;
capturing, during activation of the flash element, an image with the mobile device, the image including a representation of the mobile device reflected in a mirror;
locating, in the image, a reflection in the mirror of a flash produced by the activation of the flash element;
identifying, using an edge detection algorithm, candidate edges of one or more objects captured in the image;
identifying, among the candidate edges, mobile device edges corresponding to edges of the representation of the mobile device captured in the image, based at least in part upon a proximity of one or more candidate edges to a location of the reflection of the flash in the image;
determining one or more representation dimensions of the mobile device based at least in part upon a measurement in pixels of the mobile device edges in the representation of the mobile device in the image, the representation dimensions of the mobile device including at least a mobile device representation length and a mobile device representation width;
retrieving, from a database including mobile device specification information, one or more actual dimensions of the mobile device, the actual dimensions of the mobile device including a mobile device actual length and a mobile device actual width;
determining a scaling factor for the image, the scaling factor corresponding to a proportionality between one or more representation dimensions of the mobile device and one or more actual dimensions of the mobile device;
determining that a first article of clothing is present in the image; and
determining a size of the first article of clothing based at least in part upon the scaling factor determined for the image.

2. The computer-implemented method of claim 1 , further comprising:
generating description data during capture of the image, the description data including information capable of identifying the mobile device; and
retrieving, from the database including mobile device specification information, the one or more actual dimensions of the mobile device based on the description data.

3. The computer-implemented method of claim 1 , wherein activating the flash element of the mobile device is performed a plurality of times to produce a particular flash pattern, and further comprising:
locating, in the image, a reflection of the particular flash pattern.

4. The computer-implemented method of claim 1 , further comprising:
determining that a size of a second article of clothing corresponds to the size of the first article of clothing; and
generating an augmented reality overlay on the image, the augmented reality overlay depicting the second article of clothing in place of the first article of clothing in the image.

5. The computer-implemented method of claim 1 , wherein detecting mobile device boundaries in the image further comprises:
determining that one or more portions of the representation of the mobile device are concealed in the image; and
determining an estimation of the one or more concealed portions of the representation of the mobile device based on one or more visible portions of the representation of the mobile device.

6. A computer-implemented method for image processing, comprising:
under the control of one or more computer systems configured with executable instructions,
activating an illumination element of a mobile device while a first image is being captured by the mobile device, the first image including a reflective surface;
locating, in the first image, a reflection of an illumination produced by the activation of the illumination element;
identifying, using an edge detection algorithm, candidate edges of objects captured in the first image;
identifying, among the candidate edges, mobile device edges corresponding to edges of the representation of the mobile device captured in the first image based at least in part upon proximity of one or more of the candidate edges to a location of the reflection of the illumination in the first image; and
determining that a representation of the mobile device is present in the first image based at least in part upon (i) locating the reflection of the illumination and (ii) identifying the mobile device edges.

7. The computer-implemented method of claim 6 , wherein the illumination produced by the activation of the illuminating element comprises a flash, and wherein locating the reflection of the illumination produced by activation of the illumination element comprises locating a reflection of the flash in the reflective surface.

8. The computer-implemented method of claim 6 , wherein activating the illumination element comprises activating the illumination element for a first time, and further comprising:
determining in a first frame of image data that the representation of the mobile device is present at a first location in the image;
determining in a second frame of image data that the representation of the mobile device cannot be located at the first location;
causing, in response to determining in the second frame of image data that the representation of the mobile device cannot be located at the first location, the illumination element to activate for a second time;
locating a second reflection of the illumination produced by the activation of the illumination element; and
determining in a third frame of image data that the representation of the mobile device is present at a second location in the image based at least in part upon locating the second reflection of the illumination produced by the activation of the illumination element.

9. The computer-implemented method of claim 6 , wherein the illumination produced by activation of the illumination element comprises a flash, and wherein locating the illumination produced by activation of the illumination element further comprises:
generating a first binary mask based on the first image;
capturing a second image without activation of the illumination element;
generating a second binary mask based on the second image;
generating a difference image by subtracting the second binary mask from the first binary mask; and
identifying the illumination in the difference image.

10. The computer-implemented method of claim 6 , wherein determining that the mobile device is present in the first image based at least in part upon location of the illumination further comprises:
identifying a substantially rectangular shape in the first image within a threshold distance of the located reflection of the illumination;
determining an aspect ratio corresponding to the identified substantially rectangular shape in the first image;
determining, based on a random sample consensus, a correlation between the determined aspect ratio and a known aspect ratio, the known aspect ratio corresponding to an aspect ratio of a known mobile device; and
determining that the representation of the mobile device is present in the first image based at least in part upon the correlation between the determined aspect ratio of the identified substantially rectangular shape in the first image and the known aspect ratio corresponding to the known mobile device.

11. A system for image processing, comprising:
at least one processor;
memory including instructions that, when executed by the at least one processor, cause the system to:
locate, in an image, a reflection of a flash produced by a mobile device;
identify a substantially rectangular shape in the image within a threshold distance of the located refl

Assignees

• A9 Com Inc

Inventors

• Gray Douglas Ryan

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04N7/185

  from a mobile camera, e.g. for remote control · CPC title

• G06T7/13

  Edge detection · CPC title

• G06T7/12

  Edge-based segmentation · CPC title

• G06F16/5838

  using colour · CPC title