Camera system and method for hair segmentation

Patent metadata

Field	Value
Publication number	US-2016014392-A1
Application number	US-201414553912-A
Country	US
Kind code	A1
Filing date	Nov 25, 2014
Priority date	Jul 11, 2014
Publication date	Jan 14, 2016
Grant date	—

Abstract

A method for operating an image processing device coupled to a color camera and a depth camera is provided. The method includes receiving a color image of a 3-dimensional scene from a color camera, receiving a depth map of the 3-dimensional scene from a depth camera, generating an aligned 3-dimensional face mesh from a plurality of color images received from the color camera indicating movement of a subject's head within the 3-dimensional scene and form the depth map, determining a head region based the depth map, segmenting the head region into a plurality of facial sections based on both the color image, depth map, and the aligned 3-dimensional face mesh, and overlaying the plurality of facial sections on the color image.

First claim

1 . A method for operating an image processing device coupled to a color camera and a depth camera, comprising:
receiving a color image of a 3-dimensional scene from a color camera; receiving a depth map of the 3-dimensional scene from a depth camera; generating an aligned 3-dimensional face mesh from a plurality of color images received from the color camera indicating movement of a subject's head within the 3-dimensional scene and from the depth map; determining a head region based on the depth map; segmenting the head region into a plurality of facial sections based on both the color image, depth map, and the aligned 3-dimensional face mesh; and overlaying the plurality of facial sections on the color image.

2 . The method of claim 1 , where determining the head region includes determining a head map having foreground pixels and background pixels from the depth map and converting the head map to color coordinates corresponding to the color image to create the head region.

3 . The method of claim 1 , where determining the head region and segmenting the head region are automatically implemented independent of user interaction with the image processing device.

4 . The method of claim 1 , where the plurality of facial sections are segmented based on an energy function configured to determine the energy cost of pixels for each of the facial sections.

5 . The method of claim 1 , where determining the head region includes using a nose-tip as a seed point in the depth map and applying a head outline algorithm to the depth map.

6 . The method of claim 1 , where the facial sections include a hair section and a skin section.

7 . The method of claim 6 , where the skin section is automatically inferred based on a fringe detector, a prior skin red green blue (RGB) color threshold, and the aligned 3-dimensional face mesh.

8 . The method of claim 7 , where the fringe detector is a Sobel edge detector.

9 . The method of claim 6 , where the skin section includes a nose sub-region, an eye sub-section, a mouth sub-section, and an ear sub-section.

10 . The method of claim 6 , where a border between the hair section and the skin section is determined based on angular orientations of pixels in the color image.

11 . The method of claim 1 , where the head region is a binary map.

12 . The method of claim 1 , where the head region is calculated based on a Moore-Neighborhood algorithm.

13 . The method of claim 1 , where the color image and depth map are generated in response to user input received via the image processing device.

14 . The method of claim 1 , where the color image is a red green blue (RGB) image and where segmenting the head region includes converting the color image to a YCbCr image and determining a blue difference chroma (Cb) and a red different chroma (Cr) joint histogram.

15 . The method of claim 1 , where the pixel resolution of the color image is greater than the pixel resolution of the depth image.

16 . A camera system comprising:
a color camera; a depth camera; and an image processing device in electronic communication with the color camera and the depth camera configured to:
receive a color image of a 3-dimensional scene from a color camera;
receive a depth map of the 3-dimensional scene from a depth camera;
generate an aligned 3-dimensional face mesh from movement of a subject's head within the 3-dimensional scene detected via the color camera and from the depth map;
automatically determine a head region based on the depth map in response to generating the color image and depth map;
segment the head region into a first section and a second section based on the color image, the depth map, and aligned 3-dimensional face mesh in response to determining the head region; and
overlay the first and second sections on the color image.

17 . The camera system of claim 16 , where automatically determining the head outline includes using a nose-tip as a seed point in the depth map and applying a head outline algorithm to the depth map.

18 . The camera system of claim 16 , where the first section is a skin section and the second section is a hair section.

19 . A method for operating an image processing device coupled to a color camera and a depth camera, comprising:
receiving a color image of a 3-dimensional scene from a color camera; receiving a depth map of the 3-dimensional scene from a depth camera; generating an aligned 3-dimensional face mesh from movement of a subject's head within the 3-dimensional scene detected via the color camera and from the depth map; automatically determining a head region based on the depth map in response to generating the color image and the depth map; segmenting the head region into a skin section based on the color image, the depth map, and aligned 3-dimensional face mesh in response to determining the head outline; segmenting the head region into a hair section based on the color image, the depth map, and the aligned 3-dimensional face mesh in response to segmenting the skin region; and overlaying the skin section and the hair section on the color image in response to segmenting the head region into the skin section and the hair section.

20 . The method of claim 19 , where segmenting the head region into the skin section includes implementing a fringe detector configured to determine if a pixel is within the skin section based on the edge orientation of the pixel.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Liang Lin
• Huitema Christian F
• Simari Matthew Adam
• Anderson Sean Eron

Classifications

• H04N13/239 Primary

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title

• G06V10/449

  Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters · CPC title

• H04N13/0257

  Electricity · mapped topic

• H04N5/23219

  Electricity · mapped topic