Methods and systems employing image sensing and 3D sensing to identify shelved products

Patent metadata

Field	Value
Publication number	US-12437542-B2
Application number	US-202117476608-A
Country	US
Kind code	B2
Filing date	Sep 16, 2021
Priority date	Dec 14, 2018
Publication date	Oct 7, 2025
Grant date	Oct 7, 2025

Abstract

Inventory on a rack of store shelves is monitored by a camera-equipped system that senses when items have been removed. Image data is desirably sensed at plural spectral bands, to enhance item identification by digital watermark and/or other image recognition techniques. The system can be alert to the presence of nearby shoppers, and change its mode of operation in response, e.g., suppressing flash illumination or suspending image capture. The system may self-calibrate to the geometry of shelving in its field of view, and affine-correct captured imagery based on the camera's viewpoint. A great many other features and arrangements are also detailed.

First claim

The invention claimed is:

1. In a product distribution facility that has racks of shelving, the racks of shelving comprising plural shelves spaced above each other for storing products, an ambient inventorying system comprising:
a sensing subsystem including an image sensor and a 3D sensor, said sensing subsystem being mounted to a ceiling of said distribution facility or to one of said racks of shelving;
one or more processors and one or more memories, the one or more memories containing software instructions that configure the system to perform acts including:
processing data from the 3D sensor to identify a first shelf, and a second shelf next-above the first shelf;
identifying—in imagery captured by the image sensor—a storage region between said identified first and second shelves; and
processing an image excerpt that depicts part of said identified storage region, to thereby identify a product stored on said first shelf;

wherein said instructions further configure the system to (A) operate a flash to illuminate imagery captured by the image sensor and to suppress operation of the flash when a person is within view of said image sensor, and (B) to capture said imagery with image sensor exposures of longer than a tenth of a second.

2. The system of claim 1 , in which the software instructions further configure the system to perform acts including:
discerning, from data captured by the sensing subsystem, two essentially-parallel edges that extend up, substantially vertically, from an identified first shelf; and
selecting said image excerpt to include a region between said two essentially-parallel edges.

3. The system of claim 1 , in which the software instructions further configure the system to perform acts including:
estimating a viewpoint distortion function for said image excerpt; and
applying a counter-distortion to the image excerpt to reduce viewpoint distortion, prior to processing said image excerpt to identify a product stored on said first shelf.

4. The system of claim 1 including a flash with LEDs of plural different colors, said instructions configuring the system (i) to operate said flash to emit a first color of illumination at a first time during which the image sensor captures first imagery; (ii) to operate said flash to emit a second, different, color of illumination at a second time after the first time, during which second time the image sensor captures second imagery; and (iii) to process excerpts of said first and second imagery to thereby identify a product stored on said first shelf.

5. The system of claim 1 in which said instructions configure the system to determine, from a shelf identified by processing data from the 3D sensor, a sub-region within the captured imagery that should be ignored.

6. The system of claim 1 in which the sensing subsystem comprises a gimbal arrangement on which the image sensor is mounted, the gimbal arrangement being motor-driven to enable pivoting of the image sensor about at least one axis.

7. The system of claim 1 in which said instructions configure the system to determine, from a shelf identified by processing data from the 3D sensor, a sub-region within the captured imagery that should be ignored.

8. The system of claim 1 in which the sensing subsystem comprises a gimbal arrangement on which the image sensor is mounted, the gimbal arrangement being motor-driven to enable pivoting of the image sensor about at least one axis.

9. The system of claim 1 including a flash comprising one or more LEDs.

10. The system of claim 9 in which said flash comprises LEDs of plural different colors, said instructions configuring the system (i) to operate said flash to emit a first color of illumination at a first time during which the image sensor captures first imagery; (ii) to operate said flash to emit a second, different, color of illumination at a second time after the first time, during which second time the image sensor captures second imagery; and (iii) to process excerpts of said first and second imagery to thereby identify a product stored on said first shelf.

11. The system of claim 9 in which said instructions configure the system to determine, from a shelf identified by processing data from the 3D sensor, a sub-region within the captured imagery that should be ignored.

12. The system of claim 1 in which said excerpt is less than an entirety of an image frame captured by the image sensor.

13. In a product distribution facility that has racks of shelving, the racks of shelving comprising plural shelves spaced above each other for storing products, an ambient inventorying method comprising the acts:
operating a sensing subsystem, comprising an image sensor and a 3D sensor mounted to a ceiling of said distribution facility or to one of said racks of shelving, to capture imagery and 3D data;
processing the 3D data to identify a first shelf, and a second shelf next-above the first shelf;
identifying, in imagery captured by the image sensor, a storage region between said identified first and second shelves; and
processing an image excerpt that depicts part of said identified storage region, to thereby identify a product stored on said first shelf;
the method further including (A) operating a flash to illuminate imagery captured by the image sensor and to suppress operation of the flash when a person is within view of said image sensor, and (B) capturing said imagery with image sensor exposures of longer than a tenth of a second.

14. The method of claim 13 in which the sensing subsystem includes a flash.

15. The method of claim 13 further comprising:
discerning, from data captured by the sensing subsystem, two essentially-parallel edges that extend up, substantially vertically, from an identified first shelf; and
selecting said image excerpt to include a region between said two essentially-parallel edges.

16. The method of claim 15 further comprising:
estimating a viewpoint distortion function for said image excerpt; and
applying a counter-distortion to the image excerpt to reduce viewpoint distortion, prior to processing said image excerpt to identify a product stored on said first shelf.

17. The method of claim 13 further comprising:
(i) operating a flash with LEDs of plural different colors to emit a first color of illumination at a first time during which the image sensor captures first imagery;
(ii) operating said flash to emit a second, different, color of illumination at a second time after the first time, during which second time the image sensor captures second imagery; and
(iii) processing excerpts of said first and second imagery to thereby identify a product stored on said first shelf.

18. The method of claim 13 further comprising:
determining, from a shelf identified by processing data from the 3D sensor, a sub region within the captured imagery that should be ignored.

Assignees

• Digimarc Corp

Inventors

• Evans Eric D
• Olmstead Bryan

Classifications

• G06Q10/087

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• H04N23/72

  Combination of two or more compensation controls · CPC title

• H04N23/58

  Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors · CPC title

• G06K7/1413

  1D bar codes · CPC title

• G06K19/06028

  using bar codes · CPC title