Interactive three dimensional displays on handheld devices

Patent metadata

Field	Value
Publication number	US-9704299-B2
Application number	US-201514919324-A
Country	US
Kind code	B2
Filing date	Oct 21, 2015
Priority date	Apr 8, 2010
Publication date	Jul 11, 2017
Grant date	Jul 11, 2017

Abstract

Techniques are disclosed for rendering an anamorphic projection of 3D scene geometry on a handled device using a correct asymmetric perspective geometry projection. Once pose of the handheld device is determined, a relative eye position may be inferred when the device is tilted away from an initial or default pose, based on data supplied by accelerometers. Thus, embodiments of the invention result in a holographic style display without the need for glasses or external sensing attachments.

First claim

What is claimed is:

1. A computer-implemented method for rendering a set of three-dimensional (3D) geometry on a display of a handheld device, the computer-implemented method comprising:
determining, based on accelerometer data, a current pose of the display of the handheld device;
determining, based on the current pose of the display of the handheld device, a current position of a plane within the 3D geometry; and
in response to the current position of the plane within the 3D geometry, rendering an anamorphic 3D image of the 3D geometry on the display of the handheld device, wherein the anamorphic 3D image presents a holographic style image on the display of the handheld device, wherein the holographic style image creates an illusion of at least a first portion of the 3D geometry appearing in 3D space either above or in front of the plane by distorting the first portion of the 3D geometry using an anamorphic projection.

2. The computer-implemented method of claim 1 , wherein the holographic style image further creates an illusion of at least a second portion of the 3D geometry appearing in 3D space either below or behind the plane by distorting the second portion of the 3D geometry using the anamorphic projection.

3. The computer-implemented method of claim 1 , further comprising, determining a lighting response to apply in rendering the 3D geometry on the display of the handheld device.

4. The computer-implemented method of claim 1 , wherein the anamorphic 3D image presents the holographic style image on the display of the handheld device relative to a default eye position viewing the display of the handheld device at an expected viewing distance.

5. The computer-implemented method of claim 1 , further comprising:
capturing an image of a scene using a camera, wherein the display of the handheld device presents the image of the scene, wherein at least a first element of the 3D geometry is generated from the captured image.

6. The computer-implemented method of claim 1 , wherein the anamorphic 3D image presents the holographic style image when viewed from a relative eye position corresponding to the current pose determined based on the accelerometer data, wherein rendering the anamorphic 3D display of the 3D geometry on the display of the handheld device comprises:
generating an asymmetric anamorphic projection matrix used to distort the rendering;
generating a view matrix used to account for an offset viewing direction; and
transforming, including distorting, the 3D geometry using the asymmetric anamorphic projection matrix and the view matrix.

7. The computer-implemented method of claim 6 , further comprising:
determining, based on a current ambient lighting state, a lighting response to apply in rendering the 3D geometry on the display of the handheld device;
determining a lighting response to apply in rendering the 3D geometry on the display screen of the handheld device, wherein the lighting response is determined based on the current ambient lighting state;
wherein the anamorphic 3D image presents the holographic style image when viewed from a relative eye position corresponding to the current pose determined based on the accelerometer data, lighted based on the current ambient lighting state.

8. The computer-implemented method of claim 7 , wherein the holographic style image further creates an illusion of at least a second portion of the 3D geometry appearing in 3D space either below or behind the plane by distorting the second portion of the 3D geometry using the anamorphic projection;
capturing an image of a scene using a first camera, wherein the display screen presents the image of the scene and wherein at least a first element of the 3D geometry is generated from the captured image; and
overlying one or more virtual 3D objects over at least a portion of the first element of the 3D geometry generated from the captured image.

9. The computer-implemented method of claim 8 , further comprising:
filtering the accelerometer data using a low-pass filter, wherein each of the one or more accelerometers measure acceleration of the handheld device relative to freefall in a distinct dimension;
determining a distortion present in the display based on the current pose;
generating an image warp to transform the rendering of the anamorphic 3D display of the 3D geometry to correct a foreshortening effect resulting from the distortion;
tracking an eye position of a viewer using a second camera, wherein the current pose and a position of the handheld device are determined based on the tracked eye position; and
determining whether the default pose of the handheld device is being viewed in one of a portrait viewing mode and a landscape viewing mode.

10. The computer-implemented method of claim 9 , wherein the asymmetric anamorphic projection matrix distorts the rendered anamorphic 3D display to account for a relative tilting of the handheld device away from a default pose;
wherein the view matrix generally provides a transformation applied to world space geometry coordinates of 3D geometry to generate a location and orientation relative to an offset eye position in view space;
wherein the lighting response applies the current ambient lighting state in rendering the 3D geometry, wherein the computer-implemented method further comprises:
recovering an approximate light probe from red, green, and blue (RGB) intensity values recorded by the image, wherein the lighting state is determined from the approximate light probe.

11. A non-transitory computer readable medium storing instructions, which, which executed on a processor, perform an operation for rendering a set of three-dimensional (3D) geometry on a display of a handheld device, the operation comprising:
determining, based on accelerometer data, a current pose of the display of the handheld device;
determining, based on the current pose of the display of the handheld device, a current position of a plane within the 3D geometry; and
in response to the current position of the plane within the 3D geometry, rendering an anamorphic 3D image of the 3D geometry on the display of the handheld device, wherein the anamorphic 3D image presents a holographic style image on the display of the handheld device, wherein the holographic style image creates an illusion of at least a first portion of the 3D geometry appearing in 3D space either above or in front of the plane by distorting the first portion of the 3D geometry using an anamorphic projection.

12. The non-transitory computer readable medium of claim 11 , wherein the holographic style image further creates an illusion of at least a second portion of the 3D geometry appearing in 3D space either below or behind the plane by distorting the second portion of the 3D geometry using the anamorphic projection.

13. The non-transitory computer readable medium of claim 11 , wherein the operation further comprises, determining a lighting response to apply in rendering the 3D geometry on the display of the handheld device.

14. The non-transitory computer readable medium of claim 11 , wherein the anamorphic 3D image presents the holographic style image on the display of the handheld device relative to a default eye position viewing the display of the handheld device at an expected viewing distance.

15. The non-transitory computer readable medium of claim 11 , wherein the operation further comprises:
capturing an image of a scene using a camera, wherein the display of the handheld device presents the image of the scene, wherein at least a first element of the 3D geometry is generated from the captured image.

Assignees

• Disney Entpr Inc

Inventors

• Mitchell Kenny

Classifications

• G06F2200/1614

  Image rotation following screen orientation, e.g. switching from landscape to portrait mode · CPC title

• G06F1/1613

  for portable computers (cooling arrangements therefor G06F1/203; constructional details or arrangements for pocket calculators, electronic agendas or books G06F15/0216; constructional details of portable telephone sets: with several bodies H04M1/0202) · CPC title

• G06F2200/1637

  Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer · CPC title

• G06T2219/2021

  Shape modification · CPC title

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title