View rendering for the provision of virtual eye contact using special geometric constraints in combination with eye-tracking

Patent metadata

Field	Value
Publication number	US-9521362-B2
Application number	US-201414287264-A
Country	US
Kind code	B2
Filing date	May 27, 2014
Priority date	Nov 30, 2011
Publication date	Dec 13, 2016
Grant date	Dec 13, 2016

Abstract

A virtual camera pose determiner is configured to determine a position and an orientation of a virtual camera. The position of the virtual camera is determined on the basis of a display position of a displayed representation of a remote participant on a display. The orientation of the virtual camera is determined on the basis of a geometrical relation between the display position of the remote participant on the display, and a position of a local participant. The virtual camera is configured to transmit an image or a sequence of images to the remote participant, so that an image provided by the virtual camera has the view on the local participant as is if viewed from the display position. Further embodiments provide a video communication system having a virtual camera pose determiner for providing a virtual camera pose on basis of the display position and the position of the local participant.

First claim

The invention claimed is:

1. A virtual camera pose determiner, configured to determine a position and an orientation of a virtual camera,
wherein the position of the virtual camera is determined on the basis of a display position of a displayed representation of a remote participant on a display,
wherein the orientation of the virtual camera is determined on the basis of a geometrical relation between the display position of the remote participant on the display, and a position of a local participant;
wherein the virtual camera is configured to transmit an image or a sequence of images to the remote participant, so that an image provided by the virtual camera comprises the view on the local participant as is if viewed from the display position;
wherein the display position is determined as one of a mid-eye position, a left eye position and a right eye position of the displayed representation of the remote participant; and
wherein the position of the local participant is determined as one of a mid-eye position, a left eye position, and a right eye position of the local participant.

2. The virtual camera pose determiner according to claim 1 , wherein the orientation of the virtual camera is determined on the basis of a vector extending substantially from the display position to the position of the local participant.

3. The virtual camera pose determiner according to claim 1 , wherein the position of the virtual camera substantially coincides with the display position.

4. The virtual camera pose determiner according to claim 1 , wherein the display position is located in a display plane in which the displayed representation of the remote participant is located.

5. The virtual camera pose determiner according to claim 1 , further comprising:
a display position determiner configured to determine the display position by identifying an image feature in the displayed representation of the remote participant.

6. The virtual camera pose determiner according to claim 5 , wherein the image feature is assumed to be one of a face, a head, an eye or a pair of eyes, and wherein the display position determiner comprises at least one of a face detection and an eye detection configured to detect a face, an eye or a pair of eyes of the remote participant in the displayed representation of the remote participant.

7. The virtual camera pose determiner according to claim 1 , further comprising a local participant position determiner configured to determine the position of the local participant using at least one of
a set of images showing the local participant from different perspectives for 3D reconstruction of an at least partial 3D shape of the local participant,
at least one image and a 3D model of the local participant, and
an eye tracker for tracking an eye or a pair of eyes of the local participant.

8. The virtual camera pose determiner according to claim 1 , further comprising an input for the displayed representation of the remote participant, an internal or external interface for the position of the local participant, and an output for the determined pose of the virtual camera.

9. The virtual camera pose determiner according to claim 1 , wherein the virtual camera pose determiner further comprises a gaze tracker configured to determine a gaze direction of the local participant.

10. The virtual camera pose determiner according to claim 1 , further comprising a calibrator for calibrating a determination of the display position and a determination of the position of the local participant so that the display position and the position of the local participant can expressed using a common reference coordinate system.

11. A video communication system comprising:
an interface for receiving an inbound image or a sequence of inbound images;
an inbound image analyzer for analyzing the inbound image or the sequence of inbound images and configured to estimate a location of a sought image feature within the inbound image or the sequence of inbound images;
a display position determiner for determining a display position of the sought image feature when the inbound image or the sequence of inbound images is displayed on a display for a local participant of a video communication;
a local participant image interface for receiving a local participant image or a sequence of local participant images depicting at least a part of the local participant;
a local participant position determiner for determining a position of the local participant on the basis of the local participant image or the sequence of local participant images;
a virtual camera pose determiner for providing a virtual camera pose on basis of the display position and the position of the local participant;
a virtual view renderer for rendering a virtual view showing at least the local participant on the basis of the local participant image or the sequence of local participant images and the virtual camera pose, so that the virtual view shows the local participant from a perspective according to the virtual camera pose;
at least one further local participant image interface for receiving a further local participant image or a sequence of further local participant images showing the local participant from a different perspective than the local participant image or the sequence of local participant images; and
a 3D reconstruction component that analyzes the local participant image or the sequence of local participant images and the further local participant image or the sequence of further local participant images with respect to 3D information regarding the local participant.

12. The video communication system according to claim 11 , wherein the sought image feature is one of an eye area, a face area, a mid-eye area, a left eye and a right eye of a remote participant of the video communication depicted in the inbound image or the sequence of inbound images.

13. The video communication system according to claim 11 , further comprising a local image analyzer for analyzing the local participant image and for estimating a local image feature position of a sought image feature within the local participant image to be used by the local participant position determiner for determining the position of the local participant.

14. The video communication system according to claim 11 , wherein the virtual camera pose determiner is configured to determine the orientation of the virtual camera on the basis of a vector extending substantially from the display position to the position of the local participant.

15. The video communication system according to claim 11 , further comprising a calibrator for calibrating the virtual camera pose determiner with respect to at least one real camera connectable to the local participant image interface.

16. The video communication system according to claim 11 , further comprising an eye tracker for tracking an eye or an eye area of the local participant, wherein a resulting eye position is used by the virtual camera pose determiner to determine the position of the local participant.

17. The video communication system according to claim 11 , wherein the virtual view renderer is configured to use the 3D information for rendering the virtual view.

18. The video communication system according to claim 11 , further comprising an outbound image interface,
wherein the inbound image or the sequence of inbound images interface and the outbound image interface are connectable, via a connection, to a remote outbound image interface and a remote inbound image interface of a remote video communication syst

Assignees

• Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E V

Inventors

• Atzpadin Nicole
• Feldmann Ingo
• Kauff Peter
• Schreer Oliver
• Waizenegger Wolfgang

Classifications

• H04N7/144 Primary

  camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact · CPC title