Methods, systems, and computer readable media for shader-lamps based physical avatars of real and virtual people

Patent metadata

Field	Value
Publication number	US-9538167-B2
Application number	US-201013254837-A
Country	US
Kind code	B2
Filing date	Mar 8, 2010
Priority date	Mar 6, 2009
Publication date	Jan 3, 2017
Grant date	Jan 3, 2017

Abstract

Methods, systems, and computer readable media for shader lamps-based avatars of real and virtual people are disclosed. According to one method, shader lamps-based avatars of real and virtual objects are displayed on physical target objects. The method includes obtaining visual information of a source object and generating at least a first data set of pixels representing a texture image of the source object. At least one of a size, shape, position, and orientation of a 3D physical target object are determined. A set of coordinate data associated with various locations on the surface of the target object are also determined. The visual information is mapped to the physical target object. Mapping includes defining a relationship between the first and second sets of data, wherein each element of the first set is related to each element of the second set. The mapped visual information is displayed on the physical target object using a display module, such as one or more projectors located at various positions around the target object.

First claim

What is claimed is:

1. A method for projecting shader lamps-based avatars onto physical target objects, the method comprising:
using a processor:
obtaining, at a capture site, visual information of a human source object, wherein the human source object is a human user's head and wherein obtaining the visual information further comprises determining a source position and orientation of the human user's head;
generating, from the visual information of the human source object, a virtual 3D model of the human source object;
generating a texture map for the virtual 3D model of the human source object by projecting the visual information onto the virtual 3D model of the human source object;
determining a target position and orientation of a 3D physical target object and generating a virtual 3D model of the physical target object, wherein the 3D physical target object comprises an avatar of a human head and animatronics for controlling movement of the avatar of the human head;
rendering a textured head model for the human source object from a projector perspective using the texture map for the virtual 3D model of the human source object and the virtual 3D model of the physical target object; and
displaying, at a display site remote from the capture site, the rendering of the textured head model for the human source object directly on a surface of the physical target object by controlling movement of the avatar of the human head using the animatronics and the source position and orientation of the human user's head at the capture site, and by projecting light onto the avatar of the human head using the target position and orientation to mimic the appearance and motion of the human user's head at the capture site.

2. The method of claim 1 comprising, in real time, continuously and repeatedly obtaining the visual information of the human source object and controlling movement of the avatar of the human user's head using the animatronics to mimic the appearance and motion of the human user's head in real time.

3. The method of claim 1 wherein:
generating the virtual 3D model of the human source object includes providing an initial mapping between coordinate data associated with features of the source and target objects; and
rendering the textured head model includes determining the shape of the target object, morphing the initial mapping based on the determined shape of the target object, and re-mapping the morphed data set to the target object.

4. The method of claim 1 comprising decoupling shape, pose, and/or appearance of the human source object from shape, pose, and/or appearance of the physical target object.

5. The method of claim 1 wherein displaying the rendering of the textured head model includes using a flexible or shapeable surface-based display that approximately corresponds to the shape of the physical target object.

6. The method of claim 1 wherein displaying the rendering of the textured head model includes one of projection, emission, and transmission of the rendering of the textured head model.

7. The method of claim 6 wherein projection of the rendering of the textured head model includes using one or more projectors located at various positions around the target object.

8. The method of claim 6 wherein projection of the rendering of the textured head model includes using one of front projection and inside/rear projection.

9. The method of claim 6 wherein emission of the rendering of the textured head model includes using one of a plasma display panel (PDP), a light emitting diode (LED)-based display, an organic LED (OLED) display, and an active matrix OLED (AMOLED) display.

10. The method of claim 6 wherein transmission of the rendering of the textured head model includes using a liquid crystal display (LCD).

11. The method of claim 1 wherein displaying the rendering of the textured head model includes projecting context-sensitive visual information for providing prosthetic presence where an image of an individual as the individual appeared prior to trauma is projected onto a robotic surrogate, through which the individual is enabled to interact with an environment.

12. The method of claim 1 wherein the size and shape of the physical target object is determined based on at least one of the size of the human source object, the shape of the human source object, and the position of a viewer relative to the target object.

13. The method of claim 1 comprising capturing audio information from the human source object and synchronizing the presentation of the audio information with the rendering of the textured head model.

14. A system for projecting shader lamps-based avatars onto physical target objects, the system comprising:
an input module for:
obtaining, at a capture site, visual information of a human source object, wherein the human source object is a human user's head and wherein obtaining the visual information further comprises determining a source position and orientation of the human user's head;
generating, from the visual information of the human source object, a virtual 3D model of the human source object;
generating a texture map for the virtual 3D model of the human source object by projecting the visual information onto the virtual 3D model of the source object;
determining a target position and orientation of a 3D physical target object and generating a virtual 3D model of the physical target object, wherein the 3D physical target object comprises an avatar of a human head and animatronics for controlling movement of the avatar of the human head;

a mapping module for rendering a textured head model for the human source object from a projector perspective using the texture map for the virtual 3D model of the human source object and the virtual 3D model of the physical target object; and
a display module for displaying, at a display site remote from the capture site, the rendering of the textured head model for the human source object directly on a surface of the physical target object by controlling movement of the avatar of the human head using the animatronics and the source position and orientation of the human user's head at the capture site, and by projecting light onto the avatar of the human head using the target position and orientation to mimic the appearance and motion of the human user's head at the capture site.

15. The system of claim 14 wherein the mapping module continuously and in real time repeatedly obtains the visual information of the human source object and controls movement of the avatar of the human user's head using the animatronics to mimic the appearance and motion of the human user's head in real time.

16. The system of claim 14 wherein the mapping module is configured to:
provide an initial mapping between coordinate data associated with features of the human source and target objects;
render the textured head model by determining the shape of the target object, morphing the initial mapping based on the determined shape of the target object, and re-mapping the morphed data set to the target object.

17. The system of claim 14 wherein the mapping module is configured to decouple the shape, pose, and/or appearance of the human source object from the shape, pose, and/or appearance of the target object.

18. The system of claim 14 wherein the display module includes a flexible or shapeable surface-based display that approximately corresponds to the shape of the physical target object.

19. The system of claim 14 wherein the display module includes one or more project

Assignees

• Welch Gregory Francis
• Fuchs Henry
• Lincoln Peter
• Nashel Andrew
• State Andrei
• Univ North Carolina Chapel Hill

Inventors

• Welch Gregory Francis
• Fuchs Henry
• Lincoln Peter
• Nashel Andrew
• State Andrei

Classifications

• H04N13/388 Primary

  Volumetric displays, i.e. systems where the image is built up from picture elements distributed through a volume · CPC title

• G06T15/04

  Texture mapping · CPC title

• H04N13/0488 Primary

  Electricity · mapped topic