Restocking shelves based on image data

Patent metadata

Field	Value
Publication number	US-2017193430-A1
Application number	US-201514985626-A
Country	US
Kind code	A1
Filing date	Dec 31, 2015
Priority date	Dec 31, 2015
Publication date	Jul 6, 2017
Grant date	—

Abstract

To identify spaces on shelves which are not full of product and need restocking, markers having a pattern are placed on or near shelves containing product. A camera obtains an image of the shelf, products, and markers. A processor executing software is used to analyze the image to identify the markers and the shelf. Next, storage areas on the shelf are analyzed to identify vacant areas represented by a lower light intensity relative to adjacent areas. The processor determines a location of the vacant areas, and determines the size of the vacant areas using the known marker size and the size of the marker in the image, to generate a scale which is applied to the vacant area. This information and stored product information is used to determine which products should be restocked. The marker pattern can be selected to be readily recognizable by an image processing algorithm.

First claim

What is claimed is:

1 . A method of identifying spaces on shelves which are not full of objects, comprising:
identifying a marker having known dimensions, the marker located proximate a shelf containing objects; obtaining an image of the shelf including the marker, using a device having a camera; executing software stored on non-transitory media using a processor of at least one of the device having a camera and a computer, the software configured to
analyze the image to identify the marker;
analyze the image to identify the shelf;
analyze the storage area of the shelf to identify vacant areas of the shelf represented by a lower light intensity relative to adjacent areas;
determine a location of the vacant areas with respect to the shelf;
determine the size of the vacant areas using the actual size of the marker and the size of the marker in the image, to generate a scale which can be applied to the vacant area; and
process the determined information and stored information pertaining to the objects to determine what objects are missing from the vacant areas.

2 . The method of claim 1 , wherein the software is executed upon a server, and further including transmitting the image from the device having a camera to the server.

3 . The method of claim 1 , wherein a plurality of markers are positioned proximate the shelf.

4 . The method of claim 3 , wherein the markers are positioned at regularly spaced intervals.

5 . The method of claim 4 , wherein a distance between markers proximate the shelf is known, and the known information is used to develop a scale for determining distances in the image.

6 . The method of claim 1 , wherein the marker includes a pattern, and the pattern is selected to be distinct from patterns found on the objects.

7 . The method of claim 1 , wherein the marker includes markings which can be analyzed by the software to determine information pertaining to at least one of a location of the marker, an identity of the marker, information pertaining to the shelf, and information pertaining to the objects.

8 . The method of claim 1 , wherein the software is further configured to determine the size of an area of actual vacant space on the shelf that is represented by a determined vacant space in the image.

9 . The method of claim 1 , wherein the software is further configured to apply a transformation algorithm to a representation of the marker within the image, to cause the representation to accurately represent a known true shape of the marker, and to then apply values obtained from the transformation algorithm to a representation of a vacant area to determine a true shape of the vacant area.

10 . The method of claim 9 , wherein the software is further configured to determine the size of an area of vacant space on the shelf that is represented by the determined true shape of the vacant area in the image.

11 . The method of claim 1 , wherein the software is further configured to:
determining the sizes of vacant areas in the image; applying a transformation algorithm to a representation of the marker within the image, to cause the representation to accurately represent a known true shape of the marker, in order to obtain values corresponding to the required transformation; and using the obtained values to alter the sizes of the vacant areas to obtain the actual sizes of the vacant areas.

12 . The method of claim 1 , wherein the image is obtained from a stream of images that the camera has captured in a video.

13 . The method of claim 1 , wherein the software is further configured to determine which objects to place in the vacant areas.

14 . The method of claim 13 , wherein determining which objects to place includes assigning a priority to the order in which objects are placed, priority being given to objects corresponding to vacant areas which are large relative to other vacant areas.

15 . The method of claim 13 , wherein data corresponding to a determination of which objects to place in the vacant areas is used to calculate future demand for objects.

16 . The method of claim 1 , wherein a plurality of images are obtained, and further including using a drone to capture the images.

17 . The method of claim 1 , wherein the marker is placed proximate the shelf in order to be visible within the image.

18 . The method of claim 1 , wherein the processor is further configured to de-skew the image using a known shape of the marker.

19 . A method of identifying spaces on shelves which are not full of objects, comprising:
placing a marker bearing a pattern, the marker having known dimensions, proximate a shelf containing objects; obtaining an image of the shelf including the marker, using a device having a camera; executing software stored on non-transitory media using a processor of at least one of the device having a camera and a computer, the software configured to
analyze the image to identify the marker;
analyze the image to identify the shelf;
identify areas of the shelf in relation to a planogram;
analyze the storage area of the shelf to identify vacant areas of the shelf by comparing adjacent areas of the shelf with reference to the planogram, and if the planogram indicates adjacent areas are intended to contain the same product but the corresponding adjacent areas of the image appear different, determining that at least one row includes a vacant area;
communicate the location of the vacant areas for subsequent restocking.

20 . A method of restocking product on store display shelves, comprising:
identifying a marker proximate a shelf containing objects; at predetermined time intervals, obtaining an image of the shelf including the marker, using a device having a camera; executing software stored on non-transitory media using a processor, the software configured to
analyze the image to identify the marker;
use the marker location to identify areas corresponding to a planogram;
compare stored image data of a product at an identified area that is fully stocked with image data obtained at a time interval, and if the stored image and the image data obtained at a the time interval are not the same, designating the identified area and product as requiring restocking.

Assignees

• IBM

Inventors

• Barreira Avegliano Priscilla
• Cardonha Carlos Henrique
• Mattos Lima Andrea Britto
• Sanchez Gallo Diego

Classifications

• G06V20/17

  taken from planes or by drones · CPC title

• B64U2101/30

  for imaging, photography or videography · CPC title

• G06V20/00

  Scenes; Scene-specific elements (control of digital cameras H04N23/60) · CPC title

• G06T5/006

  Physics · mapped topic

• G06K9/52

  Physics · mapped topic