Real-time smart display detection system

Patent metadata

Field	Value
Publication number	US-9531999-B2
Application number	US-201414546994-A
Country	US
Kind code	B2
Filing date	Nov 18, 2014
Priority date	Apr 14, 2014
Publication date	Dec 27, 2016
Grant date	Dec 27, 2016

Abstract

A smart display detection system is disclosed. The smart display system is configured to allow mobile users to connect with display screens through mobile device cameras. The system can detect the display screens within images captured by the mobile device camera in real time by matching the camera query image with screen images dynamically updated from the display devices. A synchronized time-stamped matching strategy is applied to achieve high performance in detecting the screens playing motion intensive video contents and an aggressive feature selection method is applied to minimize the system uploading bandwidth. The system can accurately find display screens playing either static or dynamic content within an image captured on a mobile device.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, at one or more processors, a data stream including a series of screen images of a data screen displayed by and captured from a display device while data is being presented;
receiving, at the one or more processors, an image query comprising an image captured from a user device camera while the data screen is being displayed;
extracting feature points from the image and a screen image of the series of screen images;
building a randomized k-d tree using feature points of the screen image in the series of screen images;
identifying, with the one or more processors, the data stream including the series of screen images of the data screen corresponding to the data screen in the image captured from the user device camera using the randomized k-d tree; and
transmitting the data stream for presentation to a user of the user device.

2. The method of claim 1 , comprising:
registering the display device displaying the data screen; and
associating the series of screen images of the data screen displayed by the display device with the display device.

3. The method of claim 1 , wherein identifying the data stream including the series of screen images of the data screen corresponding to the data screen in the image captured from the user device camera comprises detecting a region in the image from the user device camera that matches an image in the series of screen images of the data screen.

4. The method of claim 1 , wherein the image query includes location information of the user device and wherein identifying the data stream corresponding to the data screen comprises searching only for data screens near the user device.

5. The method of claim 1 , comprising:
time-stamping the image query and screen images of the series of screen images; and
wherein identifying the data stream corresponding to the data screen further comprises:
comparing a time stamp of the image query to time stamps of the series of screen images to identify a starting screen image with a time stamp prior to the image query; and
identifying the data screen in the image, captured from the user device camera, from screen images after the starting screen image using the randomized k-d tree.

6. The method of claim 1 , wherein the image query comprises a plurality of images from the user device camera, the plurality of images captured at intervals coordinated with receiving the series of screen images.

7. A computer program product comprising a non-transitory computer usable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:
receive a data stream including a series of screen images of a data screen displayed by and captured from a display device while data is being presented;
receive an image query comprising an image captured from a user device camera while the data screen is being displayed;
extract feature points from the image and a screen image of the series of screen images;
build a randomized k-d tree using feature points of the screen image in the series of screen images;
identify the data stream including the series of screen images of the data screen corresponding to the data screen in the image captured from the user device camera using the randomized k-d tree; and
transmit the data stream for presentation to a user of the user device.

8. The computer program product of claim 7 , wherein the computer readable program further causes the computer to:
register the display device displaying the data screen; and
associate the series of screen images of the data screen displayed by the display device with the display device.

9. The computer program product of claim 7 , wherein to identify the data stream including the series of screen images of the data screen corresponding to the data screen in the image captured from the user device camera the computer readable program further causes the computer to detect a region in the image from the user device camera that matches an image in the series of screen images of the data screen.

10. The computer program product of claim 7 , wherein the image query includes location information of the user device and wherein to identify the data stream corresponding to the data screen the computer readable program further causes the computer to search only for data screens near the user device.

11. The computer program product of claim 7 , wherein the computer readable program further causes the computer to:
time-stamp the image query and screen images of the series of screen images; and
wherein to identify the data stream corresponding to the data screen, the computer readable program causes the computer to:
compare a time stamp of the image query to time stamps of the series of screen images to identify a starting screen image with a time stamp prior to the image query; and
identify the data screen in the image, captured from the user device camera, from screen images after the starting screen image using the randomized k-d tree.

12. The computer program product of claim 7 , wherein the image query comprises a plurality of images from the user device camera, the plurality of images captured at intervals coordinated with receiving the series of screen images.

13. A system comprising:
one or more hardware processors; and
a memory storing instructions, which when executed by the one or more hardware processors cause the system to:
receive a data stream including a series of screen images of a data screen displayed by and captured from a display device while data is being presented;
receive an image query comprising an image captured from a user device camera while the data screen is being displayed;
extract feature points from the image and a screen image of the series of screen images;
build a randomized k-d tree using feature points of the screen image in the series of screen images;
identify the data stream including the series of screen images of the data screen corresponding to the data screen in the image captured from the user device camera using the randomized k-d tree; and
transmit the data stream for presentation to a user of the user device.

14. The system of claim 13 wherein the instructions further cause the server to:
register the display device displaying the data screen; and
associate the series of screen images of the data screen displayed by the display device with the display device.

15. The system of claim 13 , wherein to identify the data stream including the series of screen images of the data screen corresponding to the data screen in the image captured from the user device camera, the instructions further cause the server to detect a region in the image from the mobile device camera that matches an image in the series of screen images of the data screen.

16. The system of claim 13 , wherein the image query includes location information of the mobile device and to detect the data screen, the instructions further cause the server to search only for data screens near the user device.

17. The system of claim 13 , wherein the instructions further cause the server to:
time-stamp the image query and screen images of the series of screen images; and
wherein to identify the data stream corresponding to the data screen, the instructions further cause the server to:
compare a time stamp of the image query to time stamps of the series of screen images to identify a starting screen image with a time stamp prior to the image query; and
identify the data screen in the

Assignees

• Shi Shu
• Barrus John
• Ricoh Co Ltd

Inventors

• Shi Shu
• Barrus John

Classifications

• G06K9/00758

  Physics · mapped topic

• H04N7/147

  Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q) · CPC title

• H04N7/15 Primary

  Conference systems · CPC title

• G06V20/48

  Matching video sequences · CPC title