Providing vehicle owner's manual information using object recognition in a mobile device

Patent metadata

Field	Value
Publication number	US-9552519-B2
Application number	US-201414293656-A
Country	US
Kind code	B2
Filing date	Jun 2, 2014
Priority date	Jun 2, 2014
Publication date	Jan 24, 2017
Grant date	Jan 24, 2017

Abstract

There is provided a communication system that includes a vehicle and a mobile device, and a method uses that system to provide vehicle owner's manual information. The method includes the steps of: receiving a digital image of a portion of a vehicle; identifying an object of the vehicle in the digital image; associating the object with at least one reference object image stored in an object library comprising a plurality of reference object images; displaying owner's manual information based on the association; and storing at least a portion of the digital image in the object library as a new reference object image for future identifications.

First claim

The invention claimed is:

1. A computer program product, comprising:
a non-transitory computer readable medium having stored thereon program instructions for use by a camera-enabled mobile device, wherein, upon execution by the mobile device, the program instructions stored on the computer readable medium cause the mobile device to carry out the steps of:
receiving a first digital image of a portion of a vehicle, wherein the portion of the vehicle is in a stationary state at a time the first digital image is received;
identifying a primary object of the vehicle in the first digital image;
associating the primary object with at least one reference object image previously stored in an object library comprising a plurality of reference object images;
displaying owner's manual information based on the association;
in response to associating the primary object with the at least one reference object image, adding at least a portion of the first digital image to the object library by storing the at least a portion of the first digital image in the object library as one of the plurality of reference object images, wherein the at least a portion includes the identified primary object; and
thereafter, receiving a second digital image and repeating the identifying, associating, and displaying steps with respect to the second digital image,
wherein repeating the associating step includes:
parsing a plurality of reference object images in the object library to associate at least one of the plurality of reference object images with a primary object in the second digital image, wherein the parsed plurality of reference object images includes the added at least a portion of the first digital image; and
using one or more of the plurality of reference object images as a basis for displaying owner's manual information associated with the second digital image, wherein the used one or more of the plurality of reference object images includes the added at least a portion of the first digital image.

2. The computer program product of claim 1 , wherein the identifying step includes determining the identity of the primary object based on the primary object's location within a centralized region of the first digital image.

3. The computer program product of claim 1 , wherein the object library includes more than one reference object image for at least some vehicle primary objects.

4. The computer program product of claim 3 , wherein for at least some vehicle primary objects, the more than one reference object image includes one or more of the reference object image captured at a first photographic angle, the reference object image captured at a second photographic angle different than the first photographic angle, and a non-photographic version of the reference object image.

5. The computer program product of claim 1 , wherein the storing step further comprises one of: storing the entire first digital image in the object library, or storing only a portion of the first digital image in the object library.

6. The computer program product of claim 1 , wherein the added at least a portion of the first digital image stored in the object library as one of the plurality of reference object images has at least one different lighting or color characteristic than the previously stored reference object images.

7. The computer program product of claim 1 , wherein the displaying step further comprises displaying one or more menu items directly-associated with the primary object, or displaying one or more predictive menu items, wherein the predictive menu items are indirectly-associated with the primary object, or both.

8. The computer program product of claim 1 , further comprising automatically highlighting at least one object in the digital scan to suggest owner's manual information that is available to the user.

9. The computer program product of claim 1 , wherein the program instructions further comprise identifying a secondary object of the vehicle in the first digital image while concurrently identifying the primary object.

10. The computer program product of claim 9 , wherein the program instructions further comprise:
associating the secondary object with at least one reference object image stored in an object library,
wherein the displaying step includes:
displaying information directly-associated with both the first and second objects, or
displaying information pertaining to a classification of objects to which both the first and second objects are associated.

11. The computer program product of claim 10 , wherein the step of identifying the secondary object includes determining the identity of the secondary object based the secondary object's location outside the centralized region of the first digital image.

12. The computer program product of claim 1 , wherein the program instructions further comprise:
receiving speech audio containing one or more identifying terms associated with at least one object identified in the first digital image;
determining the one or more identifying terms using an automatic speech recognition (ASR) engine; and
determining the owner's manual information to be displayed based on the one or more identifying terms and the identified and associated primary object of the first digital image.

13. A computer program product, comprising:
a non-transitory computer readable medium having stored thereon program instructions for use by a camera-enabled mobile device, wherein, upon execution by the mobile device, the program instructions stored on the computer readable medium cause the mobile device to carry out the steps of:
receiving a digital scan containing at least one vehicle object;
receiving speech audio containing one or more words associated with vehicle object(s);
processing the received speech audio using an automatic speech recognition (ASR) engine in the mobile device;
identifying the at least one vehicle object based on the digital scan and based on the one or more words of the processed speech audio, wherein the identifying step further comprises associating the at least one vehicle object with at least one reference object image stored in an object library of the mobile device that comprises a plurality of reference object images; and
in response to the identification, displaying owner's manual information via a display of the mobile device, wherein the owner's manual information is associated with the at least one vehicle object while displaying the at least one vehicle object in the digital scan,
wherein, when the received and processed speech audio includes a query associated with the digital scan, the displaying step includes displaying refined owner's manual information that is pertinent to the query.

14. The computer program product of claim 13 , further comprising storing the digital scan in the object library as a new reference object image for future identifications, or storing the speech audio in an audio library for future identifications, or both.

15. The computer program product of claim 13 , the identifying step further comprising automatically highlighting at least one object in the digital scan to suggest owner's manual information that is available to the user.

16. The computer program product of claim 13 , wherein the at least one reference object image includes one or more of a reference object image captured at a first photographic angle, the same reference object image captured at a second photographic angle different than the first photographic angle, and a graphic-design version of the same reference object image.

17. The co

Assignees

• Gen Motors Llc

Inventors

• Gurovich Danilo E
• Fry Keith A
• Mccutchen Shane M
• Rudman Daniel E
• Goree Ii Joseph M

Classifications

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title

• G06K9/00677

  Physics · mapped topic

• G06K9/00671 Primary

  Physics · mapped topic