Personalized meeting event capture using egocentric tracking in smart spaces

Patent metadata

Field	Value
Publication number	US-9503687-B2
Application number	US-201514667504-A
Country	US
Kind code	B2
Filing date	Mar 24, 2015
Priority date	Mar 24, 2015
Publication date	Nov 22, 2016
Grant date	Nov 22, 2016

Abstract

A system to determine viewer attention to presented content. Markers are applied to a presentation. Head orientation information is received from a body mounted camera worn by a viewer and is used to determine a sequence of head orientations of the viewer. The sequence of head orientations is associated with an identifier of the viewer and a corresponding sequence of first time stamps. A sequence of images is captured by the body mounted camera worn by the viewer. The sequence of images is associated with an identifier of the viewer and the body mounted camera and a corresponding sequence of second time stamps. Respective members of the sequences of images and head orientations are associated, and the presentation content is identified by evaluating information from visible markers in the captured sequence of images. Viewer attention to different elements can then be determined.

First claim

What is claimed is:

1. A system to determine viewer attention to presented content, comprising:
one or more processors; and
memory storing one or more programs to be executed by the one or more processors, the one or more programs comprising instructions for:
applying first markers to a first set of content to be presented to a plurality of viewers, wherein each of the first markers includes first metadata that identifies at least a content ID associated with the first set of content and a respective presentation unit of a presentation;
displaying the presentation on the presentation unit to the plurality of viewers with the applied first markers;
receiving head orientation information from a body mounted camera worn by a first viewer of a plurality of viewers and determining from the head orientation information a sequence of head orientations of the first viewer and associating each of the sequence of head orientations with an identifier associated with the first viewer and a corresponding sequence of first time stamps;
receiving a sequence of images captured by the body mounted camera worn by the first viewer, wherein the sequence of images are of in-room visuals/information presented to and viewable by the first viewer, and to associate the sequence of images with an identifier associated with the first viewer and the body mounted camera and a corresponding sequence of second time stamps;
storing and associating respective members of the sequence of images and the sequence of head orientations;
identifying presentation content by evaluating the metadata of a marker in the captured sequence of images; and
measuring viewing activities of the first viewer, wherein the viewing activity measurements include when the first viewer is or is not attending to the displayed content.

2. The system of claim 1 , wherein the first markers include:
physical markers to identify presentation surfaces of the presentation where the presentation is presented, wherein the presentation includes non-digital content and/or digital content.

3. The system of claim 1 , wherein the presentation with virtual markers embedded is displayed concurrently on personal screens as well as presentation screens.

4. The system of claim 1 , wherein the image receiving module receives the sequence of images captured by two or more body mounted cameras worn by two or more of the plurality of viewers.

5. The system of claim 1 , wherein the one or more programs further comprise instructions for:
receiving in-room audio recorded by a microphone coupled to the body mounted camera, wherein the in-room audio includes audio from the first set of content presented and audio from speakers in the room, and the in-room audio is associated with the first viewer.

6. The system of claim 1 , wherein the one or more programs further comprise instructions for:
identifying which viewer of the plurality of viewers is speaking based on which microphone coupled to the body mounted camera has the strongest audio signal at any given point in time.

7. The system of claim 1 , wherein the one or more programs further comprise instructions for:
using voice fingerprinting to verify the identity of a speaker of the plurality of viewers.

8. The system of claim 1 , wherein the one or more programs further comprise instructions for:
based on the viewing activities of the plurality of viewers, identifying a subset of the plurality of viewers, wherein members of the subset of the plurality of viewers are conversing with each other at a given moment of time, and identifies dominant or passive speakers in the subset of the plurality of viewers.

9. The system of claim 1 , wherein the one or more programs further comprise instructions for:
preparing a report on overall audience interest and on individual interest in the presentation.

10. The system of claim 1 , wherein the one or more programs further comprise instructions for:
based on the viewing activities stored at the system, sending additional content relevant to a viewer for display on one or more devices associated with the viewer.

11. A method to determine viewer attention to presented content, comprising:
at a computer system with a processor and a memory:
applying first markers to a first set of content to be presented to a plurality of viewers, wherein each of the first markers includes first metadata that identifies at least a content ID associated with the first set of content and a respective presentation unit of a presentation;
displaying the presentation on the presentation unit to the plurality of viewers with the applied first markers;
receiving head orientation information from a body mounted camera worn by a first viewer of a plurality of viewers and determining from the head orientation information a sequence of head orientations of the first viewer and associating each of the sequence of head orientations with an identifier associated with the first viewer and a corresponding sequence of first time stamps;
receiving a sequence of images captured by the body mounted camera worn by the first viewer, wherein the sequence of images are of in-room visuals/information presented to and viewable by the first viewer, and to associate the sequence of images with an identifier associated with the first viewer and the body mounted camera and a corresponding sequence of second time stamps;
storing and associating respective members of the sequence of images and the sequence of head orientations;
identifying presentation content by evaluating the metadata of a marker in the captured sequence of images; and
measuring viewing activities of the first viewer, wherein the viewing activity measurements include when the first viewer is or is not attending to the displayed content.

12. The method of claim 11 , wherein measuring the viewing activities of the first viewer further comprises:
for each of a plurality of times, wherein the respective members of the sequence of images and the sequence of head orientations are associated at the plurality of times based on the first and second timestamps:
computing a difference of direction between the sequence of head orientations and a relative position of the identified content in the sequence of images; and
determining that the first viewer is attending to the identified content when the difference of direction is within a predefined angular difference at the respective time.

13. The method of claim 11 , wherein the first markers include:
physical markers to identify presentation surfaces of the presentation where the presentation is presented, wherein the presentation includes non-digital content and/or digital content.

14. The method of claim 11 , wherein the presentation with virtual markers embedded is displayed concurrently on personal screens as well as presentation screens.

15. The method of claim 11 , wherein receiving a sequence of images includes: receiving the sequence of images captured by two or more body mounted cameras worn by two or more of the plurality of viewers.

16. The method of claim 11 , further comprising:
receiving in-room audio recorded by a microphone coupled to the body mounted camera, wherein the in-room audio includes audio from the first set of content presented and audio from speakers in the room, and the in-room audio is associated with the first viewer.

17. The method of claim 11 , further comprising:
identifying which viewer of the plurality of viewers is speaking based on which microphone coupled to the body mounted camera has an audio signal that is strongest at any given point

Assignees

• Fuji Xerox Co Ltd

Inventors

• Kratz Sven
• Shingu Jun

Classifications

• G06T2207/30204

  Marker · CPC title

• H04N7/15 Primary

  Conference systems · CPC title

• G06T7/004

  Physics · mapped topic

• G10L17/00

  Speaker identification or verification techniques · CPC title

• G06T7/246

  using feature-based methods, e.g. the tracking of corners or segments · CPC title