Display with built in 3D sensing capability and gesture control of tv

Patent metadata

Field	Value
Publication number	US-10831278-B2
Application number	US-201615001657-A
Country	US
Kind code	B2
Filing date	Jan 20, 2016
Priority date	Mar 7, 2008
Publication date	Nov 10, 2020
Grant date	Nov 10, 2020

Abstract

Information from execution of a vision processing module may be used to control a 3D vision system.

First claim

What is claimed is:

1. A system comprising:
a display device;
a three-dimensional vision system comprising:
stereo cameras;
one or more illuminators configured to create light with a spatially varying texture pattern, the three-dimensional vision system being coupled to the display in a manner that enables the display to provide interactive virtual content to a user being tracked by the three-dimensional vision system,
wherein the illuminators and stereo cameras are mounted within an enclosure of the display device; and

a computer comprising:
a vision processing module that is programmed to perform stereo processing and that maps a virtual model of fingers of the user to three-dimensional points associated with the user, tracks the fingers over time, and uses the tracked three-dimensional points to detect a gesture of the user;
an interactive content engine that receives output from the vision processing module to drive interactive graphical content; and
graphics hardware that is coupled to the interactive content engine and that renders the interactive graphical content for display on the display device.

2. The system of claim 1 , wherein the vision processing module is programmed to track the fingers over time by mapping a skeletal model to the three-dimensional points.

3. The system of claim 1 , wherein the vision processing module is programmed to detect the user modifying a virtual object that is displayed via the display device.

4. The system of claim 3 , wherein detecting the user modifying the virtual object comprises detecting the user picking up the virtual object.

5. The system of claim 1 , wherein the interactive content engine drives interactive graphical content by instructing the graphics hardware to display a representation of the fingers of the user on the display device.

6. The system of claim 5 , wherein the interactive content engine is programmed to instruct the graphics hardware to display one or more markers on the representation of the fingers of the user when the fingers are in position to interact with a virtual object displayed on the display device.

7. The system of claim 5 , wherein the representation of the fingers of the user comprises cursors representing positions of the fingers of the user.

8. The system of claim 1 , wherein the gesture of the user comprises at least one of:
a thumbs up gesture; or
a thumbs down gesture, and wherein the interactive content engine is responsive to the gesture to at least one of:
navigate a menu of media content; or
alter play of media content.

9. The system of claim 1 , wherein the interactive content engine is programmed to:
detect an emotion of the user; and
direct the graphics hardware to alter, based on the emotion of the user, the interactive graphical content displayed via the display device.

10. The system of claim 1 , wherein the spatially varying texture pattern has enough contrast to be seen by the stereo cameras over ambient light, and has a spatial frequency that provides the vision processing module detailed texture information, and one or more lenses of the illuminators has at least one of:
a fixed focal length that causes the pattern to be nearly in focus when an illuminator of the illuminators is pointed at a distant object; or
an adjustable focal length that allows the pattern to be focused at a variety of distances.

11. The system of claim 8 , wherein the interactive content engine is responsive to the gesture to both:
navigate the menu of media content; and
alter play of media content by stopping play of the media content.

12. The system of claim 1 , wherein the light with the spatially varying texture pattern includes light that is invisible or close to invisible to a human user.

13. A method comprising:
creating, via one or more illuminators of a three-dimensional vision system comprising stereo cameras and a display device, light with a spatially varying texture pattern,
wherein the illuminators and stereo cameras are mounted within an enclosure of the display device;
detecting positions of fingers of a user via the three-dimensional vision system, the three-dimensional vision system being coupled to a display device in a manner that enables the display device to provide interactive graphical content to the user being tracked by the three-dimensional vision system;
mapping, via a vision processing module that is programmed to perform stereo processing, a virtual model of the fingers of the user to three-dimensional points associated with the user;
tracking the fingers over time;
using the tracked three-dimensional points to detect a gesture of the user;
driving, based on the gesture of the user, the interactive graphical content; and
rendering the interactive graphical content for display on the display device.

14. The method of claim 13 , wherein using the tracked three-dimensional points to detect the gesture of the user comprises detecting the user modifying a virtual object that is displayed via the display device.

15. The method of claim 14 , wherein detecting the user modifying the virtual object comprises detecting the user picking up the virtual object.

16. The method of claim 13 , wherein driving the interactive graphical content comprises instructing graphics hardware to display a representation of the fingers of the user on the display device.

17. The method of claim 16 , further comprising instructing the graphics hardware to display one or more markers on the representation of the fingers of the user.

18. The method of claim 13 , wherein tracking the fingers over time comprises mapping a skeletal model to the three-dimensional points.

19. The method of claim 13 , further comprising:
detecting an emotion of the user; and
altering, based on the emotion of the user, the interactive graphical content displayed via the display device.

20. A non-transitory computer-readable storage having executable instructions stored thereon, the executable instructions configured to cause a computing system having one or more hardware processors to perform operations comprising:
creating, via one or more illuminators of a three-dimensional vision system comprising stereo cameras and a display device, light with a spatially varying texture pattern,
wherein the illuminators and stereo cameras are mounted within an enclosure of the display device;
detecting positions of fingers of a user via a three-dimensional vision system, the three-dimensional vision system being coupled to a display device in a manner that enables the display device to provide interactive graphical content to the user being tracked by the three-dimensional vision system;
mapping, via a vision processing module that is programmed to perform stereo processing, a virtual model of the fingers of the user to three-dimensional points associated with the user;
tracking the fingers over time;
using the tracked three-dimensional points to detect a gesture of the user;
driving, based on the gesture of the user, the interactive graphical content; and
rendering the interactive graphical content for display on the display device.

Assignees

• Facebook Inc

Inventors

• Bell Matthew

Classifications

• H04N13/239

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title

• A63F13/213

  comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) · CPC title

• A63F13/00

  Video games, i.e. games using an electronically generated display having two or more dimensions · CPC title

• H04N13/366

  using viewer tracking · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title