Hybrid visual communication

Patent metadata

Field	Value
Publication number	US-9325936-B2
Application number	US-201314052803-A
Country	US
Kind code	B2
Filing date	Oct 14, 2013
Priority date	Aug 9, 2013
Publication date	Apr 26, 2016
Grant date	Apr 26, 2016

Abstract

Exemplary embodiments for visual communication between a first device and a second device, comprising: using data from a camera system to create a 3D mesh model of a first device user, wherein the 3D mesh model is made available for storage on the second device for subsequent display on the second device; during the visual communication session between the first device and the second device, receiving sensor data from a sensor array, including image data capturing changing facial expressions and motion of the first device user; using the image data to determine 3D mesh model updates; transmitting the 3D mesh model updates to the second device for the second device to update display of the 3D mesh model of the first device user; and receiving from the second device for display on the first device at least one of a 2D video stream or updates to a 3D mesh model of a second device user.

First claim

I claim:

1. A method for visual communication between a first device and a second device, comprising:
using data from a camera system to create a 3D mesh model of a first device user, wherein the 3D mesh model is made available for storage on the second device for subsequent display on the second device;
during the visual communication session between the first device and the second device, receiving sensor data from a sensor array, including image data capturing changing facial expressions and motion of the first device user;
using the image data to determine 3D mesh model updates;
transmitting the 3D mesh model updates to the second device for the second device to update display of the 3D mesh model of the first device user, wherein the 3D mesh model updates are represented as at least one of a) changes to relative vertex positions of the 3D mesh model calculated from the detected position changes of the first device user, and b) selected blend shape coefficients or other enumeration of one or more selected blend shapes; and
receiving from the second device for display on the first device at least one of a 2D video stream or updates to a 3D mesh model of a second device user.

2. The method of claim 1 , wherein receiving sensor data from the sensor array further comprises: receiving other sensor data relevant to a context of the visual communication session, including activity data of the first device user, and ambient conditions.

3. The method of claim 2 , further comprising: downsampling and aggregating the other sensor data to provide sensor data updates.

4. The method of claim 3 , wherein a total of M data samples are received over a time period, and wherein the downsampling and aggregating reduce the M data samples to N sensor data updates to provide an MxN data conversion, wherein N is significantly less than M.

5. The method of claim 3 , further comprising: interleaving the sensor data updates with the 3D mesh model updates during transmission of the 3D mesh model updates to the second device.

6. The method of claim 3 , further comprising: augmenting the 3D mesh model updates with the sensor data updates for display of additional graphic elements on the 3D mesh model prior to transmitting the 3D mesh model updates to the second device.

7. A method for visual communication between a first device and a second device, comprising:
using data from a camera system to create a 3D mesh model of a first device user, wherein the 3D mesh model is made available for storage on the second device for subsequent display on the second device;
during the visual communication session between the first device and the second device, receiving sensor data from a sensor array, including image data capturing changing facial expressions and motion of the first device user;
using the image data to determine 3D mesh model updates;
transmitting the 3D mesh model updates to the second device for the second device to update display of the 3D mesh model of the first device user, wherein the image data may include an image of at least portions of a background of the first device user, wherein the background is transmitted to the second device so that 3D mesh model is superimposed over the background when displayed on the second device during the visual communication session; and
receiving from the second device for display on the first device at least one of a 2D video stream or updates to a 3D mesh model of a second device user.

8. The method of claim 1 , further comprising: sending the 3D mesh model updates during a 3D visual mode and sending a stream of 2D video to the second device during a 2D video mode.

9. The method of claim 8 , further comprising: allowing the first user to selectively switch back and forth between the 3D visual mode and the 2D video mode.

10. The method of claim 8 , further comprising: at least one of automatically suggesting the 2D video mode or the 3D visual mode to the user based upon available bandwidth, and dynamically changing video modes based on changes to the bandwidth during the visual communication session.

11. The method of claim 1 , further comprising:
responsive to determining that the second device supports only 2D video communication or is in 2D video mode, displaying a 2D video stream received from the second device on a display of the first device.

12. The method of claim 1 , further comprising:
responsive to determining the second device supports 3D visual communication or is in 3D visual mode:
displaying a 3D mesh model of the second device user on the first device;
updating display of the 3D mesh model of the second device user in response to receiving 3D mesh model updates from the second device; and
tracking movement of the first device user's face relative to the first device and reorienting display of the 3D mesh model of the second device user to provide a different perspective of the 3D mesh model on a display of the first device.

13. The method of claim 12 , wherein displaying a 3D mesh model of the second device user on the first device further comprises: rendering the 3D mesh model on the first device so that a face of a second device user appears to look directly at the first device user.

14. The method of claim 13 , further comprising: receiving on the first device a color image of the second device user's face from the second device and selectively updating the color image at a granularity of a color texture within a polygon of the 3D mesh model.

15. A method for visual communication between a first device and a second device, comprising:
using data from a camera system to create a 3D mesh model of a first device user, wherein the 3D mesh model is made available for storage on the second device for subsequent display on the second device;
during the visual communication session between the first device and the second device, receiving sensor data from a sensor array, including image data capturing changing facial expressions and motion of the first device user;
using the image data to determine 3D mesh model updates;
transmitting the 3D mesh model updates to the second device for the second device to update display of the 3D mesh model of the first device user, wherein responsive to a switch from 3D visual mode to 2D video mode on the first device, gradually blending and transmitting a stream of the 3D mesh model updates with a 2D video stream to the second device until only the 2D video stream is transmitted; and
receiving from the second device for display on the first device at least one of a 2D video stream or updates to a 3D mesh model of a second device user.

16. A method for visual communication between a first device and a second device, comprising:
using data from a 3D camera system comprising a structured light source and a 3D sensor that together collect a depth map of the first user, to create a 3D mesh model of a first device user, the 3D mesh model created by prompting the first device user to move the first device around a head of the first device user and for the first device user to make different expressions and to capture different angles of the first device user's head and expressions of the first device user's face, and wherein the 3D mesh model is made available for storage on the second device for subsequent display on the second device;
during the visual communication session between the first device and the second device, receiving sensor data from a sensor array, including image data capturing changing facial expressions and motion of the first device user;
using the image data to determine 3D mesh model updates;
t

Assignees

• Samsung Electronics Ltd
• Samsung Electronics Co Ltd

Inventors

• Cullen Schuyler

Classifications

• H04N13/128

  Adjusting depth or disparity · CPC title

• H04N13/275

  from three-dimensional [3D] object models, e.g. computer-generated stereoscopic image signals · CPC title

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title

• H04N7/147 Primary

  Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q) · CPC title

• H04N13/00 Primary

  Stereoscopic video systems; Multi-view video systems; Details thereof · CPC title