Systems and methods for cookware detection

Patent metadata

Field	Value
Publication number	US-9449220-B2
Application number	US-201414257267-A
Country	US
Kind code	B2
Filing date	Apr 21, 2014
Priority date	Apr 21, 2014
Publication date	Sep 20, 2016
Grant date	Sep 20, 2016

Abstract

Systems and methods for cookware detection are provided. One example system includes a vision sensor positioned so as to collect a plurality of images of a cooktop. The system includes a classifier module implemented by one or more processors. The classifier module is configured to calculate a cookware score for each of the plurality of images and to use the cookware score for each of the plurality of images to classify such image as either depicting cookware or not depicting cookware. The system includes a classifier training module implemented by the one or more processors. The classifier training module is configured to train the classifier module based at least in part on a positive image training dataset and a negative image training dataset.

First claim

What is claimed is:

1. A method for detecting cookware, the method comprising:
identifying, by one or more computing devices, one or more locations respectively associated with one or more burners included in a cooktop, wherein identifying, by one or more computing devices, the one or more locations respectively associated with the one or more burners comprises:
obtaining, by the one or more computing devices, a reference frame of imagery depicting the cooktop without any objects placed thereon;
obtaining, by the one or more computing devices, one or more calibration frames of imagery depicting the cooktop with one or more items of cookware respectively positioned at the one or more locations respectively associated with the one or more burners, wherein the one or more calibration frames of imagery are captured when motion is not detected at the cooktop;
performing, by the one or more computing devices, background subtraction for at least one of the one or more calibration frames of imagery with respect to the reference frame of imagery to identify new imagery; and
segmenting, by the one or more computing devices, the new imagery to identify the one or more locations respectively associated with the one or more burners included in the cooktop;
obtaining, by the one or more computing devices, a frame of imagery captured by a vision sensor, wherein the frame of imagery either depicts cookware on the cooktop, or does not depict cookware on the cooktop;
segmenting, by the one or more computing devices, the frame of imagery into one or more image segments based at least in part on the one or more locations respectively associated with the one or more burners included in the cooktop;
using, by the one or more computing devices, a classifier to provide an initial classification for each of the one or more image segments of the frame of imagery, the initial classification for each of the one or more image segments classifying the one or more image segments into either a first class of images depicting cookware or a second class of images not depicting cookware, wherein classification of one of the one or more image segments into the first class of images corresponds to detection of cookware on the cooktop;
providing, by the one or more computing devices for at least one of the one or more image segments, an indication to a user of whether the initial classification for such image segment comprises the first class or the second class;
after providing the indication for at least one of the one or more image segments, receiving, by the one or more computing devices, a user input indicating whether the initial classification for such image segment is correct; and
when the user input indicates that the initial classification for one of the one or more image segments is not correct, changing, by the one or more computing devices, the initial classification for such image segment to a subsequent classification, wherein the subsequent classification comprises the second class when the initial classification comprises the first class, and wherein the subsequent classification comprises the first class when the initial classification comprises the second class.

2. The method of claim 1 , wherein:
the first class of images comprises a positive image training dataset upon which the classifier was trained, the positive image training dataset comprising Images that depict cookware; the second class of images comprises a negative image training dataset upon which the classifier was trained, the negative image training dataset comprising images that do not depict cookware; and the method further comprises:
adding, by the one or more computing devices, each of the one or more image segments for which the subsequent classification comprises the first class to the positive image training dataset; and
adding, by the one or more computing devices, each of the one or more image segments for which the subsequent classification comprises the second class to the negative image training dataset.

3. The method of claim 2 , wherein the method further comprises, after adding each of the one or more image segments to either the positive image training dataset or the negative image training dataset, re-training the classifier based at least in part on the positive image training dataset and the negative image training dataset.

4. The method of claim 3 , wherein:
the classifier provides the initial classification for each of the one or more image segments based on a plurality of planes respectively associated with a plurality of image features; and re-training the classifier based at least in part on the positive image training dataset and the negative image training dataset comprises recomputing the plurality of planes.

5. The method of claim 1 , wherein receiving, by the one or more computing devices, the user input indicating whether the initial classification is correct comprises:
after providing the indication, pausing, by the one or more computing devices, for a response period of time; interpreting, by the one or more computing devices, a receipt of user input during the response period of time as indicating that the initial classification is not correct; and
interpreting, by the one or more computing devices, an absence of user input during the response period of time as indicating that the initial classification is correct.

6. The method of claim 1 , wherein receiving, by the one or more computing devices, a user input comprises receiving, by the one or more computing devices, data describing a voice command provided by the user, data describing one or more buttons pressed by the user, or data describing a gesture performed by the user.

7. The method of claim 1 , further comprising, prior to using, by the one or more computing devices, the classifier to provide the initial classification for each of the one or more image segments:
obtaining, by the one or more computing devices, a first plurality of images captured by the vision sensor, wherein the first plurality of images depict cookware upon the cooktop;
storing, by the one or more computing devices, the first plurality of images in a memory as a positive image training dataset;
obtaining, by the one or more computing devices, a second plurality of images captured by the vision sensor, wherein the second plurality of images do not depict cookware upon the cooktop;
storing, by the one or more computing devices, the second plurality of images in the memory as a negative image training dataset; and
training, by the one or more computing devices, the classifier based at least in part on the positive image training dataset and the negative image training dataset.

8. A system for detecting cookware, the system comprising:
a vision sensor positioned so as to collect imagery depicting a cooktop;
one or more processors; and
one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
obtaining a first plurality of images, wherein the first plurality of images depict cookware upon the cooktop;
storing the first plurality of images in a memory as a positive image training
dataset;
obtaining a second plurality of images, wherein the second plurality of images do not depict cookware upon the cooktop;
storing the second plurality of images in the memory as a negative image training
dataset;
training a classifier based on the positive image training dataset and the negative image training dataset;
identifying one or more locations respectively associated with one or more burners included in the cooktop, wherein identifying, by one or more computing devices, the one or more

Assignees

• Gen Electric
• Haier Us Appliance Solutions Inc

Inventors

• Guan Li

Classifications

• G06V30/1914

  Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries, e.g. user dictionaries · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V40/20 Primary

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title

• G06F18/28

  Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries · CPC title