Synchronizing playback of digital content with captured physical content

Patent metadata

Field	Value
Publication number	US-9317486-B1
Application number	US-201313913143-A
Country	US
Kind code	B1
Filing date	Jun 7, 2013
Priority date	Jun 7, 2013
Publication date	Apr 19, 2016
Grant date	Apr 19, 2016

Abstract

A computing device may provide a visual cue to items of content (for example, words in a book) synchronized with the playback of companion content (for example, audio content corresponding to the book). Embodiments of the present disclosure are directed to a content playback synchronization system for use with physical books (or other physical media). In an embodiment, the computing device captures images of the physical book and may display a visual cue (for example, an underline, box, dot, cursor, or the like) to identify a current location in textual content of the captured and displayed images of the physical book corresponding to a current output position of companion audio content. As the audio content is presented (i.e., as it “plays back”), the highlight and/or visual cue may be advanced to maintain synchronization between the output position within the audio content and a corresponding position in the displayed physical textual content.

First claim

What is claimed is:

1. A device for synchronizing output of an audio book with a corresponding physical book, the device comprising:
an image capture device configured to capture an image of a section of a physical book;
a processor configured to:
adjust an alignment of text in the image;
determine a boundary that encloses a word of the text in the image;
identify a portion of the image within the boundary; and
apply an emphasis to the portion of the image;

a display device configured to display the image;
an audio output device configured to output an audio book corresponding to the physical book; and
a data store configured to store synchronization information for associating the word with a corresponding portion of the audio book,
wherein the processor is in communication with the image capture device, the display device, the audio output device, and the data store, and wherein the processor is further configured to:
cause the display device to display the image, including the portion to which the emphasis has been applied, as the corresponding portion of the audio book is being audibly output by the audio output device, based at least in part on the synchronization information.

2. The device of claim 1 , wherein the processor is further configured to adjust the alignment of the text in the image by:
adjusting at least one of: an orientation, an angle, or a skew of the image.

3. The device of claim 1 , wherein the processor is further configured to:
determine a second boundary that encloses a second word of the text in the image;
identify a second portion of the image within the second boundary;
apply an emphasis to the second portion of the image; and
cause the display device to display the image, including the second portion to which the emphasis has been applied, as a second corresponding portion of the audio book associated with the second word is being audibly output by the audio output device, based at least in part on the synchronization information indicating an advancing position in the audio book.

4. The device of claim 1 , wherein the processor is further configured to apply the emphasis to the portion of the image by least one of: re-rendering the portion of the image, emphasizing the portion of the image, underlining the portion of the image, boxing the portion of the image, circling the portion of the image, pointing to the portion of the image, illuminating the portion of the image, or obscuring another portion of the image.

5. The device of claim 2 , wherein the alignment of text in the image is adjusted by adjusting the orientation of the image, and wherein the processor is further configured to adjust an orientation of the image by:
determining an offset in an orientation of the text in the image;
reorienting the image to correct the offset.

6. The device of claim 2 , wherein the alignment of text in the image is adjusted by adjusting the angle of the image, the processor is further configured to adjust an angle of the image by:
determining an offset in an angle of the text in the image;
adjusting the image to correct the offset.

7. The device of claim 2 , wherein the alignment of text in the image is adjusted by adjusting the skew of the image, the processor is further configured to adjust a skew of the image by:
determining an offset in a skew of the text in the image;
deskewing the image to correct the offset.

8. The device of claim 1 , wherein the processor is further configured to:
detect an end of the section of the physical book based on the image; and
suspend output of the audio book by the audio output device when the portion of the image is at the end of the section.

9. The device of claim 1 , wherein the processor is further configured to:
cause the display device to display the image, including the portion to which the emphasis has been applied, by providing a tactile cue of the portion that is emphasized.

10. The device of claim 1 , wherein the processor is further configured to:
determine an identity of the physical book by at least:
capturing a second image of the physical book including identifying information, wherein the identifying information includes at least one of: a cover, a title page, an ISBN, a barcode, an embedded electronic device, a format, or a unique identifier;
extracting the identifying information from the second image; and
analyzing the identifying information to identify the physical book; and

requesting the audio book corresponding to the physical book based on the identity of the physical book.

11. The device of claim 1 , wherein the processor is further configured to:
cause overlay on the image of an item of supplemental content associated with the physical book,
wherein the item of supplemental content includes at least one of: author information, publisher information, edition information, a number of pages, character information, handwritten markings, or user-produced data.

12. The device of claim 1 , wherein the processor is further configured to:
detect user-produced data from the image of the section of the physical book, wherein the user-produced data includes at least one of handwritten content or other markings made by a user in the physical book.

13. The device of claim 1 , wherein the processor is further configured to determine the boundary that encloses the word by at least one of:
recognizing the word using optical character recognition and determining a location and outer boundary of the word in the image, or
determining spatial coordinates of the word in the image and determining a location and outer boundary of the word in the image.

14. A computer-implemented method comprising:
under control of one or more computing devices configured with specific computer executable instructions,
capturing, by an image capture device, an image of a section of a physical book;
adjusting an alignment of text in the image;
determining a boundary that encloses a word of the text in the image;
identifying a portion of the image within the boundary;
applying an emphasis to the portion of the image;
outputting, by an audio output device, an audio book corresponding to the physical book;
accessing a data store configured to store synchronization information for associating the word with a corresponding portion of the audio book; and
causing a display device to display the image, including the portion to which the emphasis has been applied, as the corresponding portion of the audio book is being audibly output by the audio output device, based at least in part on the synchronization information.

15. The computer-implemented method of claim 14 , wherein adjusting the alignment of the text in the image further comprises:
adjusting at least one of: an orientation, an angle, or a skew of the image.

16. The computer-implemented method of claim 14 further comprising:
determining a second boundary that encloses a second word of the text in the image;
identifying a second portion of the image within the second boundary;
applying an emphasis to the second portion of the image; and
cause the display device to display the image, including the second portion to which the emphasis has been applied, as a second corresponding portion of the audio book associated with the second word is being audibly output by the audio output device, based at least in part on the synchronization information indicating an advancing position in the audio book.

17. The computer-implemented method of claim 14 , wherein applying the emphasis to the portion of the image com

Assignees

• Audible Inc

Inventors

• Story Jr Guy Ashley
• Brothers Nathan Garret
• Arora Ajay
• Goldstein Douglas Scott

Classifications

• G06F3/0304 Primary

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06V10/225

  based on a marking or identifier characterising the area · CPC title

• G06F2203/0381

  Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer · CPC title

• G06F3/0487

  using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser · CPC title

• G06F3/0425

  using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386) · CPC title