Object recognition and navigation from ongoing video

Patent metadata

Field	Value
Publication number	US-9760778-B1
Application number	US-201514644354-A
Country	US
Kind code	B1
Filing date	Mar 11, 2015
Priority date	Mar 11, 2015
Publication date	Sep 12, 2017
Grant date	Sep 12, 2017

Abstract

A system for performing object recognition on images in an image series, for example from a broadcast television signal or video-on-demand source. The system buffers images from the image series and accesses the buffer when an indication is received to start object recognition. The buffered images are sent to a recognition server, which returns object data for objects found in the image. The objects are then navigable using a user interface, through which the user can obtain additional information about the objects, or perform additional functions, such as adding the object to a shopping cart, etc.

First claim

What is claimed is:

1. A method comprising:
receiving a plurality of video frames of a television program from a broadcast source;
displaying the television program on a television, comprising displaying a portion of the television program on the television;
storing, in a memory buffer, at least thirty seconds worth of video frames associated with the portion of the television program;
receiving, during a first time period, a first indication from a remote control to identify objects represented in the portion of the television program previously displayed;
pausing the displaying of the television program;
determining a first video frame from the video frames stored in the memory buffer, the first video frame being determined based on at least one of the first video frame satisfying in-focus criteria or the first video frame being received more than a threshold amount of time prior to receipt of the first indication;
determining image data associated with the first video frame;
determining, using the image data, that the first video frame comprises a first item for sale in an online store;
determining, using the image data, that the first video frame comprises a second item for sale in an online store;
determining a first identification number associated with the first item, the first identification number associated with the online store;
determining a second identification number associated with the second item, the second identification number associated with the online store;
displaying a first display on the television, the first display comprising the first video frame, a visually enhanced representation of the first item and a visually enhanced representation of the second item;
receiving a second indication from the remote control to display further information about the first item;
determining first information about the first item using the first identification number; and
displaying the first information on the television.

2. The method of claim 1 , the method further comprising:
receiving the first indication from a dedicated button on the remote control;
determining a user identification number, the user identification number associated with the online store and associated with the remote control; and
associating the user identification number with the first item and the second item.

3. The method of claim 1 , the method further comprising:
determining, prior to the first time period and using contents of the memory buffer, that the first video frame comprises a first object associated with a first coordinate within the first video frame, the first coordinate comprising a pixel location within the first video frame overlapping with the first object; and
determining, after the first time period and using the image data and the first coordinate, that the first object is the first item.

4. A method comprising:
receiving a video signal representing a plurality of images;
storing data corresponding to the plurality of images in at least one memory;
sending, during a first time period, the plurality of images to a display component;
receiving a first indication to identify one or more objects in the video signal, the first indication not specifying a particular object;
determining, in response to receiving the first indication, first image data corresponding to a first image of the plurality of images, the first image data being determined based on at least one of the first image satisfying in-focus criteria or the first image data being received at a threshold time prior to receipt of the first indication;
determining that a first object is represented in the first image data;
determining first data describing the first object; and
outputting the first data.

5. The method of claim 4 , wherein determining that the first object is represented in the first image data comprises:
processing the first image data to identify a location of the first object in the first image;
sending, to a remote device, the first image data and the location; and
receiving, from the remote device, a second indication that the first object is represented in the first image data.

6. The method of claim 4 , further comprising:
receiving an identification number associated with the first object;
sending the identification number to a remote device; and
receiving the first data from the remote device.

7. The method of claim 6 , further comprising:
determining a user account associated with the first indication; and
associating the first identification number with the user account.

8. The method of claim 4 , further comprising:
sending the plurality of images to a display; and
receiving the first indication from a remote control.

9. The method of claim 4 , further comprising:
receiving a second indication to display further information about the first object;
determining second data describing the first object; and
outputting the second data.

10. The method of claim 4 , further comprising outputting the first data in a manner overlaying the first image.

11. The method of claim 4 , further comprising, prior to outputting the first data, displaying a visually enhanced representation of the first object.

12. The method of claim 4 , further comprising displaying information regarding purchasing the first object.

13. A computing system comprising:
at least one processor;
at least one memory including instructions operable to be executed by the at least one processor to cause the computing system to perform a set of actions comprising:
receiving a video signal representing a plurality of images;
storing data corresponding to the plurality of images in at least one memory;
sending, during a first time period, the plurality of images to a display component;
receiving a first indication to identify one or more objects in the video signal, the first indication not specifying a particular object;
determining, in response to receiving the first indication, first image data corresponding to a first image of the plurality of images, the first image data being determined based on at least one of the first image satisfying in-focus criteria or the first image data being received at a threshold time prior to receipt of the first indication;
determining that a first object is represented in the first image data;
determining first data describing the first object; and
outputting the first data.

14. The computing system of claim 13 , wherein determining that the first object is represented in the first image data comprises:
processing the first image data to identify a location of the first object in the first image;
sending, to a remote device, the first image data and the location; and
receiving, from the remote device, a second indication that the first object is represented in the first image data.

15. The computing system of claim 13 , the set of actions further comprising:
receiving an identification number associated with the first object;
sending the identification number to a remote device; and
receiving the first data from the remote device.

16. The computing system of claim 15 , the set of actions further comprising:
determining a user account associated with the first indication; and
associating the first identification number with the user account.

17. The computing system of claim 13 , the set of actions further comprising:
sending the plurality of images to a display; and
receiving the first indication from a remote control.

18. The computing system of claim 13 , the set of actio

Assignees

• Amazon Tech Inc

Inventors

• Wakim Peter Anthony

Classifications

• G06V20/20 Primary

  in augmented reality scenes · CPC title

• G06F16/51

  Indexing; Data structures therefor; Storage structures · CPC title

• G06V10/235

  based on user input or interaction · CPC title

• G06K9/00711 Primary

  Physics · mapped topic

• G06K9/468

  Physics · mapped topic