Methods for viewing and tracking stored items

Patent metadata

Field	Value
Publication number	US-11335010-B2
Application number	US-201916541512-A
Country	US
Kind code	B2
Filing date	Aug 15, 2019
Priority date	Aug 15, 2019
Publication date	May 17, 2022
Grant date	May 17, 2022

Abstract

A method of operating a refrigerator, as provided herein, may include initiating a first image capture sequence at a camera module and identifying a first stored item based on a first two-dimensional image of the first image capture sequence. The method may further include determining an internal location of the first stored item within a chilled chamber or generating an expanded image of the first stored item and a second stored item spaced apart from the first stored item.

First claim

What is claimed is:

1. A method of operating a refrigerator appliance comprising a camera module mounted within a cabinet at a top portion of a chilled chamber, the method comprising:
initiating a first image capture sequence at the camera module;
identifying a first stored item based on a first two-dimensional image of the first image capture sequence;
identifying a fixed fiducial marker of one or more shelves mounted within the cabinet to establish a vertical coordinate system;
determining an internal location of the first stored item within the chilled chamber based on a second two-dimensional image of the first image capture sequence and the fixed fiducial marker by using the vertical coordinate system, the second two-dimensional image of the first image capture sequence being captured subsequent to the first two-dimensional image of the first image capture sequence;
recording a descriptor of the first stored item and the internal location of the first stored item;
initiating a third image capture sequence following the first image capture sequence;
identifying a user appendage based on a first two-dimensional image of the third image capture sequence;
determining removal of the first stored item from the chilled chamber based on a second two-dimensional image of the third image capture sequence, the second two-dimensional image of the third image capture sequence being captured subsequent to the first two-dimensional image of third first image capture sequence; and
discarding the descriptor of the first stored item and the internal location of the first stored item.

2. The method of claim 1 , wherein the descriptor of the first stored item comprises at least a portion of the first two-dimensional image of the first image capture sequence.

3. The method of claim 1 , wherein the descriptor of the first stored item comprises a received two-dimensional image captured separately from the refrigerator appliance.

4. The method of claim 1 , wherein determining the internal location of the first stored item comprises
identifying a user appendage within the second two-dimensional image of the first image capture sequence, and
determining the user appendage crosses a preset fiducial edge of at least one shelf mounted within the chilled chamber
wherein the fiducial marker comprises the preset fiducial edge.

5. The method of claim 1 , further comprising:
initiating a second image capture sequence following the first image capture sequence;
identifying a second stored item based on a first two-dimensional image of the second image capture sequence;
determining an internal location of the second stored item within the chilled chamber based on a second two-dimensional image of the second image capture sequence, the second two-dimensional image of the second image capture sequence being captured subsequent to the first two-dimensional image of the second image capture sequence; and
recording a descriptor of the second stored item and the internal location of the second stored item.

6. The method of claim 5 , generating an expanded image of the first stored item and the second stored item spaced apart from the first stored item.

7. A method of operating a refrigerator appliance comprising a camera module mounted within a cabinet at a top portion of a chilled chamber, the method comprising:
initiating a first image capture sequence at the camera module;
identifying a first stored item based on a first two-dimensional image of the first image capture sequence;
identifying a fixed fiducial marker of one or more shelves mounted within the cabinet to establish a vertical coordinate system;
determining an internal location of the first stored item within the chilled chamber based on a second two-dimensional image of the first image capture sequence and the fixed fiducial marker by using the vertical coordinate system, the second two-dimensional image of the first image capture sequence being captured subsequent to the first two-dimensional image of the first image capture sequence;
recording a descriptor of the first stored item and the internal location of the first stored item;
initiating a second image capture sequence following the first image capture sequence;
identifying a second stored item based on a first two-dimensional image of the second image capture sequence;
determining an internal location of the second stored item within the chilled chamber based on a second two-dimensional image of the second image capture sequence, the second two-dimensional image of the second image capture sequence being captured subsequent to the first two-dimensional image of the second image capture sequence; and
recording a descriptor of the second stored item and the internal location of the second stored item,
wherein determining the internal location of the second stored item comprises
identifying the opened state of the drawer based on one or more two-dimensional images of the second image capture sequence, and
detecting the second stored item within the predetermined bounding area corresponding to the opened state of the drawer,

wherein determining the internal location of the first stored item comprises
identifying an opened state of a drawer based on one or more two-dimensional images of the first image capture sequence, and
detecting the first stored item within a predetermined bounding area corresponding to the opened state of the drawer, and

wherein determining the internal location of the second stored item further comprises
determining the second stored item covers a minimum captured area of the first layer based on the second two-dimensional image of the second image capture sequence, and
selecting the internal location of the second stored item as a second vertical layer of the drawer in response to determining the second stored item covers the minimum captured area of the first layer.

8. The method of claim 7 , wherein the internal location of the first stored item comprises a first vertical layer within the drawer.

9. The method of claim 7 , wherein the minimum captured area of the first vertical layer comprises at least 30% of a total captured area of the first stored item in the second two-dimensional image of the first image capture sequence.

10. The method of claim 7 , further comprising:
generating an expanded image of the first stored item and the second stored item spaced apart from the first stored item.

11. A method of operating a refrigerator comprising a camera module mounted within a cabinet at a top portion of a chilled chamber, the method comprising:
initiating a first image capture sequence at the camera module;
identifying a first stored item based on a first two-dimensional image of the first image capture sequence;
recording an image of the first stored item;
initiating a second image capture sequence following the first image capture sequence;
identifying a second stored item based on a first two-dimensional image of the second image capture sequence;
recording an image of the second stored item; and
generating an expanded image of the first stored item and the second stored item spaced apart from the first stored item.

12. The method of claim 11 , wherein the image of the first stored item comprises at least a portion of the first two-dimensional image of the first image capture sequence.

13. The method of claim 11 , wherein the image of the first stored item comprises a received two-dimensional image captured separately from the refrigerator.

14. The method of claim 11 , further comprising:
initiating a third image capture sequence following the first image capture se

Assignees

• Haier Us Appliance Solutions Inc

Inventors

• Ryu Choon Jae
• Schroeder Michael Goodman
• Poos Andrew

Classifications

• G06T2207/30196

  Human being; Person · CPC title

• G06V40/28 Primary

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06T7/74

  involving reference images or patches · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06T7/248 Primary

  involving reference images or patches · CPC title