Processing of light fields by transforming to scale and depth space

Patent metadata

Field	Value
Publication number	US-9569853-B2
Application number	US-201615048742-A
Country	US
Kind code	B2
Filing date	Feb 19, 2016
Priority date	Oct 25, 2013
Publication date	Feb 14, 2017
Grant date	Feb 14, 2017

Abstract

Light field images of a three-dimensional scene are transformed from an (image,view) domain to an (image,scale,depth) domain. Processing then occurs in the (image,scale,depth) domain.

First claim

What is claimed is:

1. A method for processing light field images of a three-dimensional scene, the method implemented on a computer system and comprising:
accessing an (image,view) domain representation of the light field images of the three-dimensional scene, wherein the (image,view) domain representation of the light field images is a representation of the light field images as a function of (image) and (view) dimensions;
applying a scale-depth transform to transform the (image,view) domain representation to an (image, scale, depth) domain representation, wherein the (image,scale,depth) domain representation is a representation of the light field images as a function of (image), (scale) and (depth) dimensions and the scale-depth transform is based on a kernel that is two-dimensional in the (image) domain; and
processing the (image,scale,depth) domain representation of the three-dimensional scene, wherein processing the (image,scale,depth) domain representation comprises at least one of (a) estimating depth in the three-dimensional scene based on processing the (image,scale,depth) domain representation or (b) extracting three-dimensional features in the three-dimensional scene based on processing the (image, scale, depth) domain representation.

2. The method of claim 1 wherein the kernel for the scale-depth transform is one-dimensional in the (depth) domain.

3. The method of claim 1 wherein a (scale) portion of the scale-depth transform is based on a Gaussian kernel or one of its derivatives.

4. The method of claim 1 wherein a (depth) portion of the scale-depth transform is based on points at different depths in the three-dimensional scene creating different curves in the (image,view) domain.

5. The method of claim 4 wherein the (depth) portion of the scale-depth transform is based on points at different depths in the three-dimensional scene creating rays at different angles in the (image,view) domain.

6. The method of claim 5 wherein applying the scale-depth transform comprises:
convolving the (image,view) domain representation with the Ray-Gaussian kernel or its derivative for σ x ∈{σ 1 , . . . , σ n }, σ y ∈{σ 1 , . . . , σ n } and for φ∈{φ 1 , . . . , φ m }; and
repeating (k−1) times the step of downsampling the (image,view) domain representation by p and convolving with the Ray-Gaussian kernel or its derivative for σ x ∈{σ 1 , . . . , σ n }, σ y ∈{σ 1 , . . . , σ n } and for φ∈{φ 1 , . . . , φ m }; where n is the number of samples per downsampling range of scale, m is the number of samples in the depth domain, and p is the downsampling factor.

7. The method of claim 5 wherein applying the scale-depth transform comprises:
convolving the (image,view) domain representation with the Ray-Gaussian kernel or its derivative for σ x ∈{σ 1 , . . . , σ n }, σ y ∈{σ 1 , . . . , σ n } and for φ∈{φ 1 , . . . , φ m };
repeating (k−1) times the step of downsampling an image portion of the (image,view) domain representation by p and convolving with the Ray-Gaussian kernel or its derivative for σ x ∈{σ 1 , . . . , σ n }, σ y ∈{σ 1 , . . . , σ n } and for {(φ′ 1 , . . . , φ′ m }; where n is the number of samples per downsampling range of scale, m is the number of samples in the depth domain, and p is the downsampling factor.

8. The method of claim 7 wherein the scale-depth transform is based on second-order partial derivative Ray-Gaussian transforms, and estimating depth in the three-dimensional scene comprises finding extrema of a determinant of a Hessian of normalized second-order partial derivative Ray-Gaussian transforms.

9. The method of claim 8 wherein the scale-depth transform is based on second-order partial derivative Ray-Gaussian transforms, and extracting three-dimensional features comprises finding extrema of a determinant of a Hessian of normalized second-order partial derivative Ray-Gaussian transforms and estimating blobs in the three-dimensional scene based on the extrema of the determinant of the Hessian.

10. The method of claim 1 wherein the scale-depth transform is based on a Ray-Gaussian kernel

σ
x

,

σ
y

,
φ

⁢

(

x
,
y
,
u

)

=

1

2
⁢
π
⁢

⁢

σ
x

⁢

σ
y

⁢

ⅇ

-

(

x
-

u
⁢

⁢
tan
⁢

⁢
φ

)

2

2
⁢

σ
x
2

-

y
2

2
⁢

σ
y
2

or one of its derivatives, wherein x and y are (image) coordinates, u is a (view) coordinate, σ x and σ y are (scale) coordinates, and φ is a (depth) coordinate.

11. The method of claim 1 wherein processing the (image,scale,depth) domain representation of the three-dimensional scene comprises estimating depth in the three-dimensional scene based on processing the (image,scale,depth) domain representation.

12. The method of claim 1 wherein processing the (image,scale,depth) domain representation of the three-dimensional scene comprises extracting features in the three-dimensiona

Assignees

• Ricoh Innovations Corp
• Ricoh Co Ltd

Inventors

• To{Hacek Over (S)}Ić Ivana
• Berkner Kathrin

Classifications

• G06F17/14

  Fourier, Walsh or analogous domain transformations {, e.g. Laplace, Hilbert, Karhunen-Loeve, transforms (for correlation function computation G06F17/156; spectrum analysers G01R23/16)} · CPC title

• G06T2207/10052

  Images from lightfield camera · CPC title

• H04N13/0048

  Electricity · mapped topic

• G06T7/0051 Primary

  Physics · mapped topic

• H04N13/0232

  Electricity · mapped topic