Real-time diet assessment and food identification using thermal imaging

Patent metadata

Field	Value
Publication number	US-10074028-B2
Application number	US-201615298581-A
Country	US
Kind code	B2
Filing date	Oct 20, 2016
Priority date	Oct 20, 2015
Publication date	Sep 11, 2018
Grant date	Sep 11, 2018

Abstract

Systems and methods are described for automatically identifying a food item. A color image and a thermal image are received by an electronic processor with a first food item in the field of view of both the color image and the thermal image. The electronic processor identifies a region of pixels in the color image that corresponds to the first food item based at least in part on a temperature intensity of the pixels in the identified region of pixels relative to other pixels in the thermal image. At least one feature is extracted from the identified region of pixels in the color image corresponding to the first food item and the electronic processor automatically identifies a type of food corresponding to the first food item based at least in part on the at least extracted feature.

First claim

What is claimed is:

1. A method of identifying a food item, the method comprising:
positioning two identifiable calibration targets near a first food item, the two identifiable calibration targets include a first calibration target and a second calibration target;
capturing a color image with the first food item and the two identifiable calibration targets within a field of view of the color image;
capturing a thermal image with the first food item and the two identifiable calibration targets within a field of view of the color image;
receiving the color image with the first food item in the field of view of the color image;
receiving the thermal image with the first food item in the field of view of the thermal image;
defining, by an electronic processor, a line in the thermal image extending from a defined location on the first calibration target to a defined location on the second calibration target;
defining, by the electronic processor, a line in the color image extending from the defined location on the first calibration target to the defined location on the second calibration target;
adjusting an orientation and a scale of the thermal image or the color image until a slope and size of the line in the thermal image matches a slope and size of the line in the color image;
identifying, by the electronic processor, a region of pixels in the color image corresponding to the first food item based at least in part on a temperature intensity of the pixels in the identified region of pixels relative to other pixels in the thermal image;
extracting at least one feature from the identified region of pixels in the color image corresponding to the first food item; and
identifying, by the electronic processor, a type of food corresponding to the first food item based at least in part on the at least one extracted feature.

2. The method of claim 1 , further comprising:
analyzing, by the electronic processor, the thermal image to identify a thermal threshold indicative of a difference between a temperature of one or more food items on a plate and a temperature of the plate, wherein the one or more food items on the plate includes the first food item; and
identifying at least one region of pixels corresponding to the one or more food items on the plate by identifying pixels in the thermal image with a temperature intensity above the identified thermal threshold.

3. The method of claim 2 , wherein a scale and orientation of the field of view of the thermal image is calibrated to align with the field of view of the color image, and wherein identifying the region of pixels in the color image corresponding to the first food item includes identifying the region of pixels in the color image corresponding to the at least one identified region of pixels corresponding to the one or more food items on the plate in the thermal image.

4. The method of claim 2 , wherein analyzing the thermal image to identify the thermal threshold includes applying a window median method routine to the thermal image.

5. The method of claim 1 , further comprising:
identifying, by the electronic processor, a first region of pixels in the color image corresponding to a potential food item based at least in part on an edge finding routine applied to the color image by the electronic processor; and
identifying, by the electronic processor, a second region of pixels in the thermal image corresponding to a potential food item based at least in part on the temperature intensity of the pixels in the second region of pixels relative to other pixels in the thermal image,
wherein a scale and orientation of the field of view of the thermal image is calibrated to align with the field of view of the color image, and
wherein identifying the region of pixels in the color image corresponding to the first food item based at least in part on the temperature intensity of the pixels in the identified region of pixels relative to other pixels in the thermal image includes
determining, by the electronic processor, whether the first region of pixels in the color image corresponds to the first food item based at least in part on an alignment of at least some of the pixels in the first region of pixels in the color image with the second region of pixels in the thermal image.

6. The method of claim 1 , further comprising applying an iterative segmentation refining routine to the identified region of pixels until a refined identified region of pixels output by the iterative segmentation refining routine converges.

7. The method of claim 6 , wherein applying the iterative segmentation refining routine includes applying a GrabCub routine to the color image with a region of pixels identified based on temperature intensity in the thermal image as an initial region of interest.

8. The method of claim 1 , wherein positioning two identifiable calibration targets near the food item includes placing a first bottle cap filled with cold water near the food item and placing a second bottle cap filled with cold water near the food item on a side of the food item opposite the first bottle cap.

9. The method of claim 1 , further comprising:
analyzing, by the electronic processor, the identified region of pixels in the color image corresponding to the first food item to extract a plurality of feature vectors including at least one selected from a group consisting of a color feature vector, a texture feature vector, and a histogram of gradients feature vector;
applying, by the electronic processor, a machine learning routine to identify the type of food based on the plurality of extracted feature vectors and previously stored feature vector information corresponding to previously identified food items in a food item database; and
updating the food item database based on the plurality of extracted feature vectors for the first food item.

10. A food identification system comprising:
a color camera;
a thermal camera; and
an electronic processor configured to
receive a color image from the color camera with a first food item in a field of view of the color image,
receive a thermal image from the thermal camera with the first food item in a field of view of the thermal image,
identify a region of pixels in the color image corresponding to the first food item based at least in part on a temperature intensity of the pixels in the identified region of pixels relative to other pixels in the thermal image,
extract at least one feature from the identified region of pixels in the color image corresponding to the first food item, and
identify a type of food corresponding to the first food item based at least in part on the at least one extracted feature,

wherein the electronic processor is further configured to
identify a first calibration target and a second calibration target positioned in the field of view of the color image and in the field of view of the thermal image,
define a line in the thermal image extending from a defined location on the first calibration target to a defined location on the second calibration target,
define a line in the color image extending from the defined location on the first calibration target to the defined location on the second calibration target, and
adjust an orientation and a scale of the thermal image or the color image until a slope and size of the line in the thermal image matches a slope and size of the line in the color image.

11. The food identification system of claim 10 , wherein the electronic processor is further configured to
analyze the thermal image to identify a thermal threshold indicative of a difference between a temperature of one or more food items on a plate and a temperature of the plate, wh

Assignees

• Gupta Sandeep
• Banerjee Ayan
• Univ Arizona State

Inventors

• Gupta Sandeep
• Banerjee Ayan

Classifications

• G06V20/68

  Food, e.g. fruit or vegetables · CPC title

• G06V10/806

  of extracted features · CPC title

• G06V10/803

  of input or preprocessed data · CPC title

• G06V10/56

  relating to colour · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title