Method and system for near-eye focal plane overlays for 3D perception of content on 2D displays

Patent metadata

Field	Value
Publication number	US-11689709-B2
Application number	US-201917256983-A
Country	US
Kind code	B2
Filing date	Jul 1, 2019
Priority date	Jul 5, 2018
Publication date	Jun 27, 2023
Grant date	Jun 27, 2023

Abstract

Some embodiments of a method may include: identifying two-dimensional (2D) content present in an image of a real-world scene; retrieving metadata comprising depth information associated with the 2D content; generating a plurality of focal plane images using the metadata, the plurality of focal plane images comprising depth cues for the 2D content; and displaying the plurality of focal plane images as a see-through overlay synchronized with the 2D content.

First claim

What is claimed:

1. A method comprising:
identifying, using a camera coupled to a head-mounted display (HMD), two-dimensional (2D) content displayed on a screen external to the HMD;
obtaining depth information associated with the 2D content;
generating a plurality of focal plane images using the depth information, the plurality of focal plane images comprising depth cues for the 2D content; and
adapting to display the plurality of focal plane images as a see-through overlay synchronized with the 2D content.

2. The method of claim 1 , wherein the screen is part of a real-world scene.

3. The method of claim 1 , wherein the depth cues for the 2D content comprise information regarding at least one of distance and texture.

4. The method of claim 1 , wherein each of the plurality of focal plane images comprises high-spatial-frequency image information for an associated image depth.

5. The method of claim 4 , wherein the high-spatial-frequency image information comprises accommodation cues for focusing at varying distances.

6. The method of claim 1 , further comprising:
low-pass-filtering the 2D content; and
adapting to display the low-pass-filtered 2D content,
wherein adapting to display the plurality of focal plane images displays the plurality of focal plane images as an overlay over the low-pass-filtered 2D content.

7. The method of claim 1 , further comprising capturing the 2D content with the camera.

8. The method of claim 1 , further comprising:
identifying a spatial position of the screen,
wherein adapting to display the plurality of focal plane images comprises aligning the plurality of focal plane images with the spatial position of the screen.

9. The method of claim 1 ,
wherein obtaining the depth information comprises retrieving metadata comprising the depth information,
wherein the metadata comprises timing information to enable synchronously aligning the displayed plurality of focal plane images with the 2D content, and
wherein adapting to display the plurality of focal plane images comprises synchronously aligning the plurality of focal plane images with the 2D content using the timing information.

10. The method of claim 1 ,
wherein obtaining the depth information comprises retrieving metadata comprising the depth information,
wherein the metadata comprises three-dimensional (3D) depth information for the 2D content, and
wherein the 3D depth information for the 2D content comprises a time sequence of depth maps synchronized to the 2D content.

11. The method of claim 10 ,
further comprising converting a resolution of the depth maps to match a resolution of the 2D content,
wherein the resolution of the depth maps is different than the resolution of the 2D content.

12. The method of claim 1 , further comprising:
detecting an asymmetry of the 2D content displayed on the screen,
wherein adapting to display the plurality of focal plane images comprises adjusting the plurality of focal plane images based on the asymmetry of the 2D content.

13. The method of claim 1 , wherein adapting to display the see-through overlay enables a user to view the screen via a direct optical path.

14. An apparatus comprising:
a camera;
a multi-focal plane display;
a processor; and
a non-transitory computer-readable medium storing instructions that are operative, when executed by the processor, to cause the apparatus to:
identify, using a camera coupled to a head-mounted display (HMD), two-dimensional (2D) content displayed on a screen external to the HMD;
obtain depth information associated with the 2D content;
generate a plurality of focal plane images using the depth information, the plurality of focal plane images comprising depth cues for the 2D content; and
adapt to display the plurality of focal plane images as a see-through overlay synchronized with the 2D content.

15. The apparatus of claim 14 , further comprising:
an optical lens structure configured to adjust direct optical viewing of the screen; and
an optical low-pass filter.

16. A method comprising:
detecting, using a camera coupled to a head-mounted display (HMD), presence, spatial position, and orientation information relating to 2D video content displayed on a 2D display external to the HMD;
receiving 3D video information corresponding to the 2D video content;
synchronizing in time the 3D video information with the 2D video content;
tracking the spatial position information and orientation information relating to the 2D video content;
decomposing the 3D video information into a plurality of focal plane images;
filtering one or more of the plurality of focal plane images to remove one or more respective low frequency representations from the plurality of focal plane images; and
adapting to display the filtered focal plane images.

17. The method of claim 16 , wherein filtering one or more of the plurality of focal plane images comprises high-pass-filtering at least one of the plurality of focal plane images.

18. The method of claim 16 , wherein decomposing the 3D video information into the plurality of focal plane images comprises:
determining a depth of the 3D video information;
forming a plurality of 2D weighting planes by processing the depth of the 3D video information with one or more depth-blending functions; and
forming the plurality of focal plane images by weighting the 2D video content with the plurality of 2D weighting planes.

19. The method of claim 16 , wherein the 3D information comprises a plurality of high-frequency focal plane images and positions in a common axial coordinate system of the plurality of high-frequency focal plane images.

20. The method of claim 16 ,
wherein detecting presence, spatial position, and orientation information relating to 2D video content comprises detecting presence, spatial position, and orientation information relating to the 2D display, and
wherein tracking the spatial position information and orientation information relating to the 2D video content comprises tracking the spatial position information and orientation information relating to the 2D display.

Assignees

• Interdigital Vc Holdings Inc

Inventors

• Valli Seppo T
• Siltanen Pekka K
• Kerofsky Louis
• Neff Ralph

Classifications

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G02B2027/0134

  of stereoscopic type · CPC title

• G02B2027/0127

  comprising devices increasing the depth of field · CPC title

• G02B30/52

  the three-dimensional [3D] volume being constructed from a stack or sequence of two-dimensional [2D] planes, e.g. depth sampling systems · CPC title