System and method for synthesizing and preserving consistent relative neighborhood position in multi-perspective multi-point tele-immersive environments

Patent metadata

Field	Value
Publication number	US-10360808-B2
Application number	US-201715850214-A
Country	US
Kind code	B2
Filing date	Dec 21, 2017
Priority date	Aug 30, 2013
Publication date	Jul 23, 2019
Grant date	Jul 23, 2019

Abstract

An e-learning system has a classroom having a seating arrangement for students, a plurality of depth-sensing cameras, a session controller coupled to the depth-sensing cameras, executing software from a non-transitory medium, a presenter station, and a plurality of augmented-reality headsets, one each associated with each of the seats in the fixed-seating arrangement, and one associated with the presenter station. The software operates in one of an instruction mode, an interaction mode, and a discussion mode. The viewpoints and audio feeds to all of the headsets are controlled such that the students directly interacting see and hear one another directly, and the remainder of the headsets see and hear one or the other of the directly-engaged students according to which student is speaking.

First claim

The invention claimed is:

1. An e-learning system comprising:
a classroom having a fixed seating arrangement for a plurality of students and a plurality of first depth-sensing cameras positioned to capture 3D video images of each student from a variety of viewpoints;
a computerized session controller coupled to the first depth-sensing cameras, and executing software from a non-transitory medium;
a presenter station providing a position for a presenter, and a plurality of second depth-sensing cameras positioned to capture video images of a presenter at the presenter station from a variety of viewpoints, the plurality of second depth-sensing cameras coupled to the computerized session controller; and
a plurality of augmented-reality headsets, including video display and audio feedback circuitry, one each associated with each of the seats in the fixed-seating arrangement, and one associated with the presenter station;
wherein the software executing at the computerized session controller operates in one of an instruction mode, wherein a straight-on view and audio of the presenter station is streamed to each of the augmented-reality headsets at each of the seats in the fixed seating arrangement for the plurality of students, an interaction mode, wherein video and audio of interaction between the presenter station and a particular student at one of the seats in the fixed seating arrangement is streamed, such that the particular student views and hears the presenter station head on, and the presenter station views and hears the particular student head-on, and the remainder of the students view and hear the presenter and the student alternately as each participates in an audio exchange, and a discussion mode, wherein video and audio streaming takes place between individual students, the viewpoints and audio feeds to all of the AR headsets controlled such that the students directly interacting see and hear one another directly, and the remainder of the headsets see and here one or the other of the directly-engaged students according to which student is speaking.

2. The e-learning system of claim 1 wherein the classroom and the presenter station are in the same physical, geographic location, and the first and second depth-sensing cameras, and the augmented reality headsets are coupled to the computerized session controller on a local area network.

3. The e-learning system of claim 1 wherein the classroom and the presenter station are in different physical, geographic locations, and the first and second depth-sensing cameras, and the augmented reality headsets are coupled to the computerized session controller through the Internet network.

4. The e-learning system of claim 1 further comprising a plurality of classrooms, each having a fixed seating arrangement for a plurality of students and a plurality of first depth-sensing cameras positioned to capture 3D video images of each student from a variety of viewpoints, a plurality of augmented-reality headsets, including video display and audio feedback circuitry, one each associated with each of the seats in the fixed-seating arrangement at each classroom.

5. The e-learning system of claim 1 wherein the software executing at the computerized session controller provides for switching mode-to-mode based on gesture recognition in video feeds from individual ones of the depth-sensing cameras, or key words in audio feeds.

6. The e-learning system of claim 5 wherein the presenter is enabled to override mode switches software executing at the computerized session controller provides for switching mode-to-mode based on override input from an authorized source.

7. The e-learning system of claim 6 wherein the presenter is enabled to override mode switches.

8. The e-learning system of claim 1 wherein a viewpoint is determined by interpolation of frames from two different depth-sensing cameras.

Assignees

• Amrita Vishwa Vidyapeetham

Inventors

• Rangan P Venkat
• Hariharan Balaji
• Narayanankutty Ramkumar
• Gopalakrishnan Uma

Classifications

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• H04N7/142

  Constructional details of the terminal equipment, e.g. arrangements of the camera and the display · CPC title

• H04N7/15

  Conference systems · CPC title

• G06K9/00355

  Physics · mapped topic

• G09B5/067 Primary

  Combinations of audio and projected visual presentation, e.g. film, slides · CPC title