Method for processing a light field image delivering a super-rays representation of a light field image

Patent metadata

Field	Value
Publication number	US-2020082539-A1
Application number	US-201716466025-A
Country	US
Kind code	A1
Filing date	Nov 30, 2017
Priority date	Dec 1, 2016
Publication date	Mar 12, 2020
Grant date	—

Abstract

The present disclosure concerns a method for processing a light field image comprising a set of image views. The method is remarkable in that it comprises: selecting ( 101 ) a set of pixels, named centroids, within a given image view in said light field image, each centroid being associated with a unique label and a unique light ray; de-projecting said centroids, that delivers a set of points in a 3D space; re-projecting said points of said set of points in a 3D space in at least one other image view in said light field image, which is different from said reference view, that delivers a set of re-projected centroids; and determining ( 102 ), for each pixel, being associated with a light ray, within a neighborhood of a re-projected centroid, a label related to one of the centroids, said determining relying on a proximity criterion in term of color between pixels and centroids, and a proximity criterion in term of spatial distances between pixels and re-projected centroids; and updating ( 103 ), for a given centroid, a color value associated with said given centroid, and spatial coordinates of said given centroid within said given image view, wherein said updating comprises: de-projecting pixels in said at least one other image view that are associated with a same label of said given centroid, delivering a set of obtained points; and re-projecting said obtained points into said given image view, delivering a set of projected points; and averaging both color values associated with said projected points and spatial coordinates associated with said projected points for defining updated information related to said given centroid.

First claim

1 . A method for processing a light field image comprising a set of image views, the method being characterized in that it comprises:
selecting ( 101 ) a set of pixels, named centroids, within a given image view in said light field image, each centroid being associated with a unique label and a unique light ray; de-projecting said centroids, that delivers a set of points in a 3D space; re-projecting said points of said set of points in a 3D space in at least one other image view in said light field image, which is different from said reference view, that delivers a set of re-projected centroids; and determining ( 102 ), for each pixel, being associated with a light ray, within a neighborhood of a re-projected centroid, a label related to one of the centroids, said determining relying on a proximity criterion in term of color between pixels and centroids, and a proximity criterion in term of spatial distances between pixels and re-projected centroids; and updating ( 103 ), for a given centroid, a color value associated with said given centroid, and spatial coordinates of said given centroid within said given image view, wherein said updating comprises: de-projecting pixels in said at least one other image view that are associated with a same label of said given centroid, delivering a set of obtained points; and re-projecting said obtained points into said given image view, delivering a set of projected points; and averaging both color values associated with said projected points and spatial coordinates associated with said projected points for defining updated information related to said given centroid.

2 . The method for processing according to claim 1 , wherein said de-projecting and re-projecting are done according to intrinsic and extrinsic calibration parameters.

3 . The method for processing according to any claims 1 to 2 , wherein said updating is done for all of the centroids.

4 . The method for processing according to any claims 1 to 3 , wherein said given image view is a central image view in said set of image views.

5 . The method for processing according to any claims 1 to 4 , wherein said determining said label is done according to a minimization process that comprises the determination of

A


(
r
)

=

argmin
c



{

Δ
Lab



(

r
,

r
c

)

+

m
·

Δ
xy



(

r
,

r
c
′

)

}

,

wherein c is a label associated to a centroid, r c corresponds to a centroid ray in said given image view having for coordinates (s c ,t c ,x c ,y c ) and associated with a depth d c ,r′ c corresponds to a projection of said centroid ray r c in said at least one another image view, having for coordinates (s′,t′,P s′,t′ d c ,(x c ,y c )), and wherein r corresponds to said light ray associated with a point, having for coordinates (s′,t′,x,y), and where Δ Lab (r,r c )=∥Lab r −Lab r c ∥ 2 , and Δ xy (r,r′ c )=∥(x,y)−P s′,t′ d c ,(x c ,y c )∥ 2 , wherein Lab r c corresponds to a color value of said centroid ray r c ,Lab r corresponds to a color value of said light ray r, and m is a weighting parameter.

6 . The method for processing according to any claims 1 to 5 , wherein at least two reference images views are used, and said at least two reference image views are far from each other's from an angular point of view.

7 . The method for processing according to any claims 1 to 6 , wherein said light field image has been acquired by a cameras array.

8 . The method for processing according to any claims 1 to 6 , wherein said light field image has been acquired by a plenoptic camera.

9 . An electronic device configured to process a light field image comprising a set of image views, the electronic device comprising at least one processor and at least one memory unit associated with said at least one processor, said at least one processor being characterized in that it is configured to:
select a set of pixels, named centroids, within a given image view in said light field image, each centroid being associated with a unique label and a unique light ray; de-project said centroids, that delivers a set of points in a 3D space; re-project said points of said set of points in a 3D space in at least one other image view in said light field image, which is different from said reference view, that delivers a set of re-projected centroids; and determine, for each pixel, being associated with a light ray, within a neighborhood of a re-projected centroid, a label related to one of the centroids, said determining relying on a proximity criterion in term of color between pixels and centroids, and a proximity criterion in term of spatial distances between pixels and re-projected centroids; and update, for a given centroid, a color value associated with said given centroid, and spatial coordinates of said given centroid within said given image view, wherein said update comprises: de-project pixels in said at least one other image view that are associated with a same label of said given centroid, delivering a set of obtained points; and re-project said obtained points into said given image view, delivering a set of projected points; and determine an average of both color values associated with said projected points and spatial coordinates associated with said projected points for defining updated information related to said given centroid.

10 . The electronic device according to claim 9 , wherein actions consisting in de-projecting and re-projecting are done according to intrinsic and extrinsic calibration parameters.

11 . The electronic device according to any claims 9 to 10 , wherein the update is done for all of the centroids.

12 . The electronic device according to any claims 9 to 11 , wherein said given image view is a central image view in said set of image views.

13 . The electronic device according to any claims 9

Assignees

• Interdigital Ce Patent Holdings

Inventors

• Hog Matthieu
• Sabater Neus
• Guillemot Christine

Classifications

• G06T7/174 Primary

  involving the use of two or more images · CPC title

• G06T2207/10052

  Images from lightfield camera · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title