Constructing 3D Surfaces for Multi-Color Objects

Patent metadata

Field	Value
Publication number	US-2016140753-A1
Application number	US-201414540493-A
Country	US
Kind code	A1
Filing date	Nov 13, 2014
Priority date	Nov 13, 2014
Publication date	May 19, 2016
Grant date	—

Abstract

This document describes techniques and apparatuses for constructing three dimensional (3D) surfaces for multi-colored objects. In some aspects, these techniques determine, from a color image and coarse depth information, an illumination model and albedo for a multi-color object. The coarse depth information may then be refined based on the illumination model and combined with the albedo to provide a 3D surface of the multi-color object.

First claim

1 . A computer-implemented method comprising:
receiving a color image of a multi-color object; receiving coarse depth information for the multi-color object; determining, based on the color image and the coarse depth information, an illumination model and albedo for the multi-color object, the albedo determined at least in part by estimating chromaticity information for the color image and regularizing the albedo with pairwise weights defined using the estimated chromaticity information; determining, based on the illumination model and the coarse depth information, refined depth information for the multi-color object; and constructing, based on the albedo and the refined depth information, a relightable three-dimensional (3D) surface of the multi-color object.

2 . The computer-implemented method of claim 1 , further comprising:
determining, based on the color image and the refined depth information, an updated illumination model and updated albedo of the multi-color object; and determining, based on the updated illumination model and the refined depth information, more-refined depth information for the multi-color object by which to construct the relightable 3D surface of the multi-color object.

3 . The computer-implemented method of claim 1 , wherein regularizing the albedo includes determining the pairwise weights with a Gaussian function that operates on the estimated chromaticity information.

4 . The computer-implemented method of claim 1 , wherein determining the refined depth information for the multi-color object includes regularizing or smoothing the refined depth information based on the coarse depth information.

5 . The computer-implemented method of claim 1 , wherein determining the illumination model for the multi-color object includes rendering the coarse depth information with spherical harmonics.

6 . The computer-implemented method of claim 5 , wherein the rendering is performed using a lighting function that operates on lighting coefficients based on the coarse depth information.

7 . The computer-implemented method of claim 6 , wherein the lighting function operates directly in a log domain effective to avoid exponentiation during the determining of the illumination model.

8 . The computer-implemented method of claim 1 , wherein the coarse depth information is received from a depth sensing system that includes one of an optical depth sensor, an infrared depth sensor, a known-geometry modeling system, or a multi-image depth estimation system.

9 . The computer-implemented method of claim 1 , wherein the color image is received from an image sensor and the color image of the multi-color object is a red-blue-green (RGB) image.

10 . One or more hardware-based computer-readable storage devices having instructions stored thereon that, responsive to execution by one or more computer processors, perform operations comprising:
receiving, from an image sensor, a red-green-blue (RGB) image of a multi-color object; receiving, from a depth sensing system, coarse depth information for the multi-color object; decomposing, via a constrained image optimization, the RGB image into albedo, shading, and lighting information for the multi-color object, the albedo decomposed at least in part by estimating chromaticity information for the color image and regularizing the albedo with pairwise weights defined by a Gaussian function applied to the estimated chromaticity information; refining, via a regularized shape optimization, the coarse depth information based on the shading, lighting, and coarse depth information; and providing, based on the albedo and refined depth information, a relightable three dimensional (3D) mesh of the multi-color object.

11 . The one or more hardware-based computer-readable storage devices of claim 10 , wherein the operations of decomposing and refining are performed iteratively prior to providing the relightable 3D mesh of the multi-color object.

12 . The one or more hardware-based computer-readable storage devices of claim 10 , wherein the image optimization is constrained based on the coarse depth information.

13 . The one or more hardware-based computer-readable storage devices of claim 10 , wherein the shape optimization is regularized based on the coarse depth information.

14 . The one or more hardware-based computer-readable storage devices of claim 13 , wherein the operations further comprise, for the constrained image optimization, pre-computing base functions for a lighting model on which the optimization is based.

15 . The one or more hardware-based computer-readable storage devices of claim 10 , wherein the depth sensing system includes one of an optical depth sensor, an infrared depth sensor, a known-geometry modeling system, or a multi-image depth estimation system.

16 . A multi-color three-dimensional (3D) scanning system comprising:
an image sensor configured to capture color images; a depth sensor configured to provide coarse depth information; a surfacing engine configured to:
receive, from the image sensor, a color image of a multi-color object;
receive, from the depth sensor, coarse depth information for the multi-color object;
determine, based on the color image and the coarse depth information, an illumination model and albedo for the multi-color object, the albedo determined at least in part by estimating chromaticity information for the color image and regularizing the albedo with weights defined, based on the estimated chromaticity information, for pixel pairs of the color image;
determine, based on the illumination model and the coarse depth information, refined depth information for the multi-color object; and
construct, based on the albedo and the refined depth information, a three-dimensional relightable surface of the multi-color object.

17 . The multi-color 3D scanning system of claim 16 , wherein the surfacing engine is further configured to, prior to constructing the relightable surface, iteratively perform the operation of determining the illumination model and the albedo and the operation of determining the refined depth information.

18 . The multi-color 3D scanning system of claim 16 , wherein regularizing the albedo includes determining, for a given pixel of the color image, at least two of the weights for the given pixel and at least two other pixels that are adjacent to the given pixel.

19 . The multi-color 3D scanning system of claim 16 , wherein determining the refined depth information for the multi-color object includes regularizing or smoothing the refined depth information based on the coarse depth information.

20 . The multi-color 3D scanning system of claim 16 , wherein the system is implemented as a gaming device, a set-top box, a laptop computer, a table computer, a smart phone, a camera, a video camera, a television, or a monitor.

Assignees

• Adobe Systems Inc

Inventors

• Langguth Fabian
• Sunkavalli Kalyan K
• Hadap Sunil

Classifications

• G06T2207/20112

  Image segmentation details · CPC title

• G06T15/506 Primary

  Illumination models · CPC title

• G06T5/002

  Physics · mapped topic

• G06T15/80

  Shading · CPC title

• G06T2207/10024

  Color image · CPC title