Automatic mapping of store layout using soft object recognition

Patent metadata

Field	Value
Publication number	US-10733661-B1
Application number	US-201615158464-A
Country	US
Kind code	B1
Filing date	May 18, 2016
Priority date	May 22, 2015
Publication date	Aug 4, 2020
Grant date	Aug 4, 2020

Abstract

A method for automatically mapping a store layout includes identifying a path for traversing a retail area and capturing images of the retail area at various points along the path. The images may be analyzed to identify visual characteristics which may be compared to a template of retail products in a template library. When an object depicted in the image matches with a retail product in the template library, the object may correspond to the retail product. Additionally, a retail department for the object may also be identified. The retail department may be compared to the retail product corresponding to the object, and when the retail product is not associated with the retail department, another retail product which is associated with the retail department may be identified as corresponding to the object. A map of the store layout may be generated based on the identified retail products.

First claim

We claim:

1. A computer-implemented method for automatically mapping a store layout, the method executed by one or more processors programmed to perform the method, the method comprising:
receiving, at the one or more processors, a plurality of three-dimensional images captured by an autonomous vehicle within a retail area, including receiving, for each of the plurality of three-dimensional images, a location of the three-dimensional image within the retail area, wherein each of the plurality of three-dimensional images depicts one or more merchandizing fixtures within the retail area;
for each of the plurality of three-dimensional images:
identifying, by the one or more processors, visual characteristics within each of one or more objects on the one or more merchandizing fixtures, wherein each of the one or more objects is a retail product; and
for each of the one or more objects:
determining, by the one or more processors, likelihoods that the object corresponds to a plurality of retail products by comparing the visual characteristics of the object to visual characteristics for each of the plurality of retail products including:
comparing, by the one or more processors, a text string in the object to text strings corresponding to templates of each of the plurality of retail products; and
determining, by the one or more processors, the likelihoods that the object corresponds to the plurality of retail products based on the comparison;
determining, by the one or more processors, a retail department for the object by:
determining distances from the object to retail products neighboring the object;
identifying retail departments corresponding to the retail products neighboring the object; and
determining the retail department for the object based on the distances from the object to the retail products neighboring the object;

adjusting, by the one or more processors, the likelihoods that the object corresponds to the plurality of retail products by comparing the retail department for the object to retail departments for the plurality of retail products;
determining, by the one or more processors, a retail product for the object having a highest adjusted likelihood of the adjusted likelihoods that the object corresponds to the plurality of retail products; and

generating, by the one or more processors, a map of a store layout for the retail area including indications of each of the determined retail products and locations of each of the determined retail products within the retail area.

2. The method of claim 1 , wherein determining a retail department for the object includes:
determining, by the one or more processors, the retail department for the object based on a most frequently occurring retail department of the retail departments.

3. The method of claim 1 , wherein determining a retail department for the object includes:
identifying, by the one or more processors within the three-dimensional image, one or more tags corresponding to objects neighboring the object; and
analyzing, by the one or more processors, the one or more tags to identify retail departments.

4. The method of claim 1 , wherein identifying visual characteristics within each of one or more objects includes:
identifying, by the one or more processors, boundaries of the one or more objects within the three-dimensional image; and
identifying, by the one or more processors, the visual characteristics within the boundaries of each of the one or more objects.

5. The method of claim 4 , wherein identifying visual characteristics within the boundaries of each of the one or more objects includes:
identifying, by the one or more processors, text characters within the object using stroke width transform (SWT) techniques; and
identifying, by the one or more processors, the text string based on the text characters using optical character recognition (OCR) techniques.

6. The method of claim 4 , wherein identifying visual characteristics within the boundaries of each of the one or more objects includes:
identifying, by the one or more processors, a size and shape of the object;
wherein determining likelihoods that the object corresponds to the plurality of retail products based on the visual characteristics of the object includes:
comparing, by the one or more processors, the size and shape of the object to templates of each of the plurality of retail products; and
determining, by the one or more processors, the likelihoods that the object corresponds to the plurality of retail products based on the comparison.

7. The method of claim 1 , wherein determining likelihoods that the object corresponds to the plurality of retail products based on the visual characteristics of the object includes:
determining, by the one or more processors, a first likelihood that the object corresponds to one of the plurality of retail products by comparing a size and shape of the object to a size and shape for a template of the retail product;
determining, by the one or more processors, a second likelihood that the object corresponds to the retail product by comparing the text string in the object to a text string for the template of the retail product;
determining, by the one or more processors, a third likelihood that the object corresponds to the retail product by comparing style parameters for the object to style parameters for the template of the retail product; and
combining, by the one or more processors, the first, second, and third likelihoods to determine an overall likelihood that the object corresponds to the retail product.

8. The method of claim 1 , wherein determining a retail product for the object based on the likelihoods that the object corresponds to the plurality of retail products includes:
determining that the object corresponds to one of the plurality of retail products when at least one of: (i) the likelihood for the retail product exceeds a predetermined likelihood threshold or (ii) the likelihood for the retail product is a highest likelihood of the likelihoods for the plurality of retail products.

9. The method of claim 1 , further comprising:
filtering, by the one or more processors, the one or more merchandizing fixtures from the plurality of three-dimensional images including:
identifying depth within the plurality of three-dimensional images; and
filtering out one or more portions of the plurality of three-dimensional images having a depth which is less than a predetermined depth threshold.

10. The method of claim 1 , further comprising:
displaying, by the one or more processors, the map of the store layout on a user interface.

11. A system for automatically mapping a store layout, the system comprising:
one or more processors,
a non-transitory computer-readable memory coupled to the one or more processors, and storing thereon instructions that, when executed by the one or more processors, cause the system to:
receive a plurality of three-dimensional images captured by an autonomous vehicle within a retail area, including receiving, for each of the plurality of three-dimensional images, a location of the three-dimensional image within the retail area, wherein each of the plurality of three-dimensional images depicts one or more merchandizing fixtures within the retail area;
for each of the plurality of three-dimensional images:
identify visual characteristics within each of one or more objects on the one or more merchandizing fixtures, wherein each of the one or more objects is a retail product; and
for each of the one or more objects:
determine likelihoods that the object corresponds to a plurality of retail products by comparing the visual characteristics of the object to vis

Assignees

• Walgreen Co
• Univ Pennsylvania

Inventors

• Bergstrom John M
• Cleveland Jonas
• Thakur Dinesh
• Dames Philip
• Philips Cody
• Kientz Terry
• Daniilidis Kostas
• Kumar Vijay

Classifications

• G06V30/18

  Extraction of features or characteristics of the image · CPC title

• G06V30/10

  Character recognition · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06F18/24147

  Distances to closest patterns, e.g. nearest neighbour classification · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title