Techniques for providing virtual light adjustments to image data

Patent metadata

Field	Value
Publication number	US-10740959-B2
Application number	US-201816032418-A
Country	US
Kind code	B2
Filing date	Jul 11, 2018
Priority date	Sep 9, 2017
Publication date	Aug 11, 2020
Grant date	Aug 11, 2020

Abstract

Embodiments of the present disclosure can provide systems, methods, and computer-readable medium for providing virtual lighting adjustments to image data. A user interface for presenting and/or modifying image data may be provided via an electronic device. User input may be received that indicates a selection of a virtual lighting mode. Landmark points corresponding to a set of pixels of the image data may be identified based, at least in part, on depth measurement values of the set of pixels. One or more masks may be generated from the landmark points. One or more virtual lighting adjustments associated with the selected virtual lighting mode may be made to the image data using these masks (or the landmark points and an implied geometry of the landmark points). The adjusted/modified image may be presented to the user via the user interface at the electronic device.

First claim

What is claimed is:

1. A method, comprising:
presenting, by an electronic device, a user interface for modifying image data defining an image, the image data including headshot image data of a subject of the image;
receiving, at the user interface, user input indicating selection of a virtual lighting mode;
identifying a plurality of landmark points from the headshot image data, each of the plurality of landmark points corresponding to a set of pixels of the headshot image data, the plurality of landmark points individually being identified based, at least in part, on depth measurement values associated with the set of pixels;
generating a plurality of masks for the headshot image data, each mask corresponding to a set of pixel locations of an area of the headshot image data, and each mask being determined based, at least in part, on the locations of the plurality of landmark points; and
modifying the headshot image data utilizing a subset of virtual lighting adjustments from a plurality of virtual lighting adjustments to areas of the headshot image data corresponding to the plurality of masks, the subset of virtual lighting adjustments utilized being modified based, at least in part, on the virtual lighting mode selected and the depth measurement values, and wherein each of the subset of virtual lighting adjustments utilizes a different mask of the plurality of masks.

2. The method of claim 1 , wherein the presenting further comprises:
capturing by an image capture device of the electronic device, the image data including the headshot image data; and
displaying, on a display of the electronic device, a preview of the image data including the headshot image data.

3. The method of claim 1 , wherein the modifying further comprising:
displaying, on the display of the electronic device, a preview of the headshot image data as modified.

4. The method of claim 1 , wherein generating a plurality of masks further comprises:
determining, for at least one mask of the plurality of masks, at least one feature area of the headshot image data, the feature area corresponding to a plurality of pixels of the image data, the feature area corresponding to at least one of: a nose area of the subject, at least one eye area of the subject, a mouth area of the subject, a dental area of the subject, or at least one facial hair area of the subject; and
excluding the at least one feature area from the at least one mask.

5. The method of claim 1 , further comprising:
darkening a sub-area of at least one mask of the plurality of masks utilizing a virtual light removal technique.

6. The method of claim 1 , wherein the plurality of virtual lighting adjustments correspond to a plurality of virtual lights, wherein the subset of the plurality of virtual light adjustments utilize at least one of:
a virtual spotlight that increases luminance of a first group of pixels associated with a first sub-area of at least one mask of the plurality of masks;
a virtual strobe light that increases luminance of a second group of pixels associated with a second sub-area of at least one mask of the plurality of masks; and
a virtual kick light that increases luminance of a third group of pixels associated with a side portion of at least one mask of the plurality of masks.

7. The method of claim 6 , wherein the luminance of the first group of pixels is increased based, at least in part, comparing an area size corresponding to the first sub-area of the at least one mask to a total area of an image defined by the image data.

8. The method of claim 6 , further comprising:
identifying an orientation and a tilt of the plurality of masks based, at least in part, on the headshot image data; and
determining dimensions of a plurality of lighting adjustment areas corresponding to sub-areas of the plurality of masks based, at least in part, on the orientation and the tilt, wherein the plurality of lighting adjustment areas are utilized to apply the subset of the plurality of virtual lights to the headshot image data utilizing the plurality of masks.

9. The method of claim 1 , wherein the plurality of landmark points are further identified based, at least in part, on providing the depth measurement values associated with the set of pixels to a machine-learning model, the machine-learning model being trained on historical images in which landmark points are known.

10. A method, comprising:
presenting, by an electronic device, a user interface for modifying image data, the image data including headshot image data;
receiving, at the user interface, user input indicating selection of a virtual contour lighting mode;
identifying a plurality of landmark points from the headshot image data, each of the plurality of landmark points corresponding to a set of pixels of the headshot image data, the plurality of landmark points individually being identified based, at least in part, on depth measurement values associated with the set of pixels;
determining a plurality of masks for the headshot image data, the plurality of masks each corresponding to a set of pixel locations of different areas of the headshot image data, the plurality of masks each being determined based, at least in part, on the plurality of landmark points and the depth measurement values;
increasing, according to the selection of the virtual contour lighting mode, luminance of a first plurality of pixels associated with a forehead area, an under-eye area, and a chin area identified by a first mask;
decreasing, according to the selection of the virtual contour lighting mode, luminance of a second plurality of pixels associated with a side area identified by a second mask;
increasing, according to the selection of the virtual contour lighting mode, luminance of a third plurality of pixels associated with a side nose area identified by a third mask;
increasing, according to the selection of the virtual contour lighting mode, luminescence of a fourth plurality of pixels associated with a nose area identified by a fourth mask; and
decreasing, according to the selection of the virtual contour lighting mode, luminance of a fifth plurality of pixels associated with a neck area of the headshot image data, the neck area being identified based, at least in part, on a relation to a fifth mask of the plurality of masks.

11. The method of claim 10 , further comprising:
modifying the headshot image data by executing at least one contrast reduction algorithm, wherein executing the contrast reduction algorithm causes image noise to be reduced with the headshot image data.

12. The method of claim 11 , wherein the at least one contrast reduction algorithm comprises at least one of: a Gaussian mask, a smoothing filter, a linear filter, a non-linear filter, or a median filter.

13. The method of claim 10 , wherein the neck area of the headshot image data is defined in relation to the plurality of masks.

14. The method of claim 10 , wherein at least one of the plurality of masks is further determined by:
identifying two eye-areas of the at least one mask and a mouth area of the at least one mask; and
excluding the two eye-areas and the mouth area from the at least one mask.

15. A method, comprising:
presenting, by an electronic device, a user interface for modifying image data, the image data including headshot image data;
receiving, at the user interface, user input indicating selection of a virtual studio lighting mode;
identifying a plurality of landmark points from the headshot image data, each of the plurality of landmark points corresponding to a set of pixels of the headshot image data, the plurality o

Assignees

• Apple Inc

Inventors

• Johnson Garrett
• Chen Chong
• Cao Frederic

Classifications

• H04N23/633

  for displaying additional information relating to control or operation of the camera · CPC title

• H04N23/631

  Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• G06V40/171 Primary

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title