Real-time Photorealistic 3D Holography With Deep Neural Networks

Patent metadata

Field	Value
Publication number	US-2023205133-A1
Application number	US-202117919842-A
Country	US
Kind code	A1
Filing date	Apr 21, 2021
Priority date	Apr 21, 2020
Publication date	Jun 29, 2023
Grant date	—

Abstract

A method for producing a hologram representative of a subject three-dimensional scene includes receiving and storing input digital data characterizing a first image of the subject three-dimensional scene. The method further includes processing the data in a neural network that has been trained to transform the input digital data into a holographic representation of the subject three-dimensional scene, the representation containing phase information characterizing depth and parallax of the scene. The method also includes providing an output of the holographic representation of the subject three-dimensional scene.

First claim

1 . A method for producing a hologram representative of a subject three-dimensional scene, the method comprising:
receiving and storing input digital data characterizing a first image of the subject three-dimensional scene; processing the data in a neural network that has been trained to transform the input digital data into a holographic representation of the subject three-dimensional scene, the representation containing phase information characterizing depth and parallax of the scene; and providing an output of the holographic representation of the subject three-dimensional scene.

2 . A method according to claim 1 , wherein the holographic representation encodes color information of the scene.

3 . A method according to claim 1 , wherein the neural network is a convolutional neural network.

4 . A method according to claim 1 , wherein the digital data characterizing the first image includes color and depth information, and the holographic representation including amplitude and phase information by color.

5 . A method according to claim 1 , wherein processing the data is configured to produce a holographic representation that reproduces occlusion effects.

6 . A method according to claim 1 , wherein the neural network has been trained to simulate Fresnel diffraction.

7 . A method according to claim 6 , wherein the neural network has been additionally trained to cause the holographic representation to exhibit a desired depth of field effect.

8 . A method according to claim 1 , wherein the neural network has been trained on training data representing scenes constructed from samples of random three-dimensional meshes having random textures.

9 . A method according to claim 8 , wherein the training data are configured to follow a probability density function in such a manner as to have a uniform pixel distribution across a range of depths.

10 . (canceled)

11 . A method according to claim 1 , further comprising, before providing the output of the holographic representation, performing anti-aliasing processing of the holographic representation to render it displayable with reduced artifacts.

12 . (canceled)

13 . (canceled)

14 . (canceled)

15 . A method for producing a hologram representative of a subject three-dimensional scene, the method comprising:
receiving and storing input digital data characterizing a set of images of the subject three-dimensional scene; processing the data in a neural network that has been trained to transform the input digital data into a phase-only holographic representation of the subject three-dimensional scene, the representation containing phase information characterizing depth and parallax of the scene, wherein the representation is computed to take into account effects of wave-based occlusion with respect to the complete scene; and providing an output of the holographic representation of the subject three-dimensional scene.

16 . A method according to claim 15 , wherein the holographic representation encodes color information of the scene.

17 . A method according to claim 15 , wherein the neural network is a convolutional neural network.

18 . A method according to claim 15 , wherein processing the data further includes performing aberration correction.

19 . (canceled)

20 . A method according to claim 15 , wherein the neural network has been additionally trained to cause the holographic representation to be focused on any desired focal plane within the subject three-dimensional scene so as to exhibit a desired depth of field.

21 . A method according to claim 15 ,
wherein the neural network has received additional training in two stages to directly optimize the phase-only hologram (with anti-aliasing processing) by incorporating a complex to phase-only conversion into the training, wherein in a first stage the neural network is trained to predict a midpoint hologram propagated to a center of the subject three-dimensional scene and to minimize a difference between a target focal stack and a predicted focal stack, and in a second stage a phase-only target hologram is generated from the predicted midpoint hologram and refined by calculating a dynamic focal stack loss, between a post-encoding focal stack and the target focal stack, and a regularization loss associated therewith.

22 . A method according to claim 15 , wherein the digital data characterize a stream of images occurring at an average frame rate, and the processing is configured to occur in real time.

23 . A method according to claim 15 , wherein the set of images of the subject three-dimensional scene includes a plurality of layered depth images.

24 . A method according to claim 15 , wherein the set of images of the subject three-dimensional scene is a single RGBD image.

25 . A method for producing a hologram representative of a subject three-dimensional scene, the method comprising:
receiving and storing input digital data characterizing a set of images of the subject three-dimensional scene; processing the data in a neural network that has been trained to transform the input digital data into a phase-only holographic representation of the subject three-dimensional scene and has been further trained in two stages to directly optimize the phase-only hologram (with anti-aliasing processing) by incorporating a complex to phase-only conversion into the training, wherein in a first stage the neural network is trained to predict a midpoint hologram propagated to a center of the subject three-dimensional scene and to minimize a difference between a target focal stack and a predicted focal stack, and in a second stage a phase-only target hologram is generated from the predicted midpoint hologram and refined by calculating a dynamic focal stack loss, between a post-encoding focal stack and the target focal stack, and a regularization loss associated therewith, the representation containing phase information characterizing depth and parallax of the scene; and providing an output of the holographic representation of the subject three-dimensional scene.

26 . (canceled)

Assignees

• Massachusetts Inst Technology

Inventors

• Matusik Wojciech
• Shi Liang

Classifications

• G03H1/04 Primary

  Processes or apparatus for producing holograms (G03H1/26 takes precedence) · CPC title

• G03H1/0808 Primary

  Methods of numerical synthesis, e.g. coherent ray tracing [CRT], diffraction specific · CPC title

• G03H1/2294

  Addressing the hologram to an active spatial light modulator · CPC title

• G03H2210/454

  into planes · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title