Template-based recognition of food product information

Patent metadata

Field	Value
Publication number	US-9292565-B2
Application number	US-82813910-A
Country	US
Kind code	B2
Filing date	Jun 30, 2010
Priority date	Jun 30, 2010
Publication date	Mar 22, 2016
Grant date	Mar 22, 2016

Abstract

A method for template based recognition of food product information includes capturing an image of food product packaging and extracting an attribute from the image. The attribute is used to find a matching template in a template-database. The matching template is then used to extract food product information from the image.

First claim

What is claimed is:

1. A method for template based recognition of food product information comprises:
capturing an image of food product packaging;
extracting an attribute from the image;
finding, in a template database, a template matching the attribute, in which the matching template identifies data redundancies in the image;
extracting, by referencing the matching template, food product information from the image;
using the data redundancies to cross check the extracted food product information;
transforming the food product information according to user defined preferences; and
displaying the transformed information to a user.

2. The method of claim 1 , in which capturing an image comprises taking a picture of the food product packaging using a mobile device and recording GPS data that identifies a location where the image of the food product packaging was taken.

3. The method of claim 2 , further comprising adjusting the user defined preferences according to the location where the image of the food product packaging was taken.

4. The method of claim 3 , in which adjusting the user defined preferences comprises displaying purchasing information when the image of the food product packaging is taken at a retail location and displaying inventory information when the image of the food product is taken in a user's home.

5. The method of claim 1 , in which extracting an attribute from the image comprises optical character recognition and feature recognition of graphical elements of a nutrition fact table on the food product packaging.

6. The method of claim 1 , in which finding a template matching the attribute comprises a comparison of graphical elements of the food product packaging with graphical elements of templates in the template database and a comparison of optically recognized alpha numeric text with elements of the templates in the template database.

7. The method of claim 1 , further comprising using the matching template to validate food product information by comparing optically recognized text strings and values with expected text strings and value ranges contained within the matching template.

8. The method of claim 7 , further comprising writing validated food product information to a data structure.

9. The method of claim 1 , in which the user defined preferences comprise at least one of: analyzing the data for user allergens, language translation, a nutrient content filter, defining uncommon terms, calculating user specific values, recording a scanning history of the user, recording a history of user consumption, searching for product recall notices, searching for product reviews, presenting reminders to the user, and setting accessibility options.

10. The method of claim 1 , in which the food product information comprises ingredient and nutritional information translated into a language specified in the user defined preferences.

11. The method of claim 1 , further comprising, if a matching template is not found, displaying a warning to the user and updating the template database with new templates.

12. The method of claim 11 , in which updating the template database comprises the creation of new templates from crowd sourced data collected when a predetermined number of users have not been able to access a template from the template database for a given product label.

13. The method of claim 1 , further comprising recognizing errors in templates using crowd sourcing.

14. The method of claim 1 , further comprising:
analyzing the food product information for potential drug interactions with medications taken by the user; and
displaying the potential drug interactions to the user.

15. The method of claim 1 , further comprising:
tracking a history of food product information imaged by the user to measure progress toward a health goal; and
displaying the history to the user.

16. The method of claim 1 , in which capturing an image comprises taking a picture of natural food item, the method further comprising:
identifying potential natural food items which may match the natural food item in the picture;
presenting options to the user to confirm which of the potential natural food items is correct; and
retrieving data related to the natural food item from a database.

17. The method of claim 1 , in which using the data redundancies to cross check the extracted food product information comprises determining if fat, carbohydrates and protein amounts contained in the extracted food product information are consistent with values expected from ingredients in the extracted food product information.

18. The method of claim 1 , in which using the data redundancies to cross check the extracted food product information comprises determining if the total amount fat, carbohydrates and protein in the extracted food product information are less than or equal to the mass of the servings in the extracted food product information.

19. The method of claim 1 , further comprising:
exchanging scientific names of ingredients in the extracted food product information with the common names of the ingredients; and
displaying the common names of the ingredients.

20. A system comprising:
a mobile device with a camera;
a template database comprising templates of food product packaging;
a template matching module to match an image of a food product taken with the camera with a template in the template database in which the matching template identifies data redundancies in the image;
an analytics engine to extract information from the image according to the template and to use the data redundancies to cross check the extracted food product information; and
a personal filter to manipulate the extracted information according to user preferences to produce personalized data;
in which the personalized data is presented to a user of the mobile device.

21. A computer program product for template based recognition of food product information, the computer program product comprising:
a computer readable non-transitory storage medium having computer readable program code embodied therewith, the computer readable program code comprising:
computer readable program code configured to capture an image of food product packaging;
computer readable program code configured to extract at least one attribute from the image;
computer readable program code configured to use the at least one attribute to identify a matching template in a template database in which the matching template identifies data redundancies in the image;
computer readable program code configured to use the matching template to extract food product information from the image; and
computer readable program code configured to use the data redundancies in the image to cross check the extracted food product information.

Assignees

• Bhagwan Varun
• Ormont Justin
• Underwood Heather
• IBM

Inventors

• Bhagwan Varun
• Ormont Justin
• Underwood Heather

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06F17/30879

  Physics · mapped topic

• G06F17/30389 Primary

  Physics · mapped topic

• G06F16/242 Primary

  Query formulation · CPC title

• G06F16/5866

  using information manually generated, e.g. tags, keywords, comments, manually generated location and time information · CPC title