Method for automatically generating waypoints for imaging shelves within a store

Patent metadata

Field	Value
Publication number	US-11130239-B2
Application number	US-201916513228-A
Country	US
Kind code	B2
Filing date	Jul 16, 2019
Priority date	May 19, 2016
Publication date	Sep 28, 2021
Grant date	Sep 28, 2021

Abstract

One variation of a method for automatically generating waypoints for imaging shelves within a store includes: dispatching a robotic system to autonomously generating a map of a floor space within the store; accessing an architectural metaspace defining target locations and addresses of the set of shelving structures within the store; distorting the architectural metaspace into alignment with the map to generate a normalized metaspace representing real locations and addresses of the set of shelving structures in the store; defining a set of waypoints distributed longitudinally along and offset laterally from a first shelving structure represented in the normalized metaspace based on a known position of an optical sensor in the robotic system; and dispatching the robotic system to record optical data while occupying the set of waypoints during an imaging routine.

First claim

We claim:

1. A method for automatically mapping a store comprising:
accessing a map of a floor space within the store generated based on map data collected by a robotic system while navigating through the store during a mapping routine;
accessing an architectural plan defining target locations of a set of shelving structures within the store;
distorting features in the architectural plan into alignment with like features in the map to generate a normalized metaspace representing real locations of the set of shelving structures;
identifying the set of shelving structures within the normalized metaspace based on target locations of the set of shelving structures defined in the architectural plan and distorted into alignment with like features in the map;
defining a route along the set of shelving structures identified within the normalized metaspace, the route comprising a first route segment along a first shelving structure, in the set of shelving structures, represented in the normalized metaspace; and
dispatching the robotic system to scan shelving structures in the set of shelving structures in the store while traversing the route during a scan cycle succeeding the mapping routine.

2. The method of claim 1 , further comprising:
accessing a second map of the floor space of the store generated by the robotic system while navigating along the route during the scan cycle;
accessing a set of images recorded by the robotic system while navigating through the store during the scan cycle, each image in the set of images associated with a position, along the route, occupied by the robotic system during recordation of the image during the scan cycle;
for each image in the set of images:
detecting a shelving structure, in the set of shelving structures, depicted in the image based on a position, along the route, associated with the image;
detecting a set of features in the image;
identifying a set of products located on the shelving structure based on the set of features; and
generating a stock condition of the shelving structure based on the set of products; and

aggregating stock conditions of the set of shelving structures into a realogram of the store.

3. The method of claim 2 , further comprising:
detecting a difference between a planogram of the store and the realogram; and
generating a restocking prompt to restock a particular shelving structure in the set of shelving structures with a particular product based on the difference; and
serving the restocking prompt to a computing device affiliated with an associate of the store.

4. The method of claim 1 :
further comprising dispatching the robotic system to autonomously collect map data of the floor space within the store during the mapping routine in response to receipt of confirmation of a change to a shelving structure layout within the store;
wherein generating the normalized metaspace comprises generating the normalized metaspace representing the change to the shelving structure layout within the store; and
wherein defining the route comprises defining the set of route segments according to the change to the shelving structure layout represented in the normalized metaspace.

5. A method for automatically mapping shelving structures within a store comprising:
accessing a map of a floor space of the store generated based on map data collected by a robotic system while navigating through the store during a scan cycle;
accessing a set of images recorded by the robotic system during the scan cycle;
detecting a set of shelving structures represented in the map;
for each image in the set of images:
detecting a shelving structure, in the set of shelving structures, depicted in the image based on a location within the map associated with the image;
detecting a set of features in the image;
identifying a set of products located on the shelving structure based on the set of features; and
generating a stock condition of the shelving structure based on the set of products;

aggregating stock conditions of the set of shelving structures into a realogram of the store representing locations of the set of shelving structures and products throughout the store during the scan cycle; and
in response to receipt of confirmation of a preferred stock state of the store during the scan cycle, storing the realogram as a planogram defining target stock conditions of shelving structures and products throughout the store.

6. The method of claim 5 , further comprising:
accessing a second set of images recorded by the robotic system during a second scan cycle succeeding the scan cycle;
for each image in the second set of images:
detecting a shelving structure, in the set of shelving structures, depicted in the image;
detecting a set of features in the image;
identifying a set of products located on the shelving structure during the second scan cycle based on the set of features; and
generating a second stock condition of the shelving structure during the second scan cycle based on the set of products;

aggregating stock conditions of the set of shelving structures into a second realogram of the store representing locations of the set of shelving structures and products throughout the store during the second scan cycle;
detecting a difference between the planogram and the second realogram;
generating a restocking prompt to restock a particular shelving structure in the set of shelving structures with a particular product based on the difference; and
serving the restocking prompt to a computing device affiliated with an associate of the store.

7. The method of claim 5 , further comprising:
dispatching the robotic system to execute a second scan cycle, succeeding the scan cycle, within the store;
accessing a first image comprising visual data recorded by the robotic system during the second scan cycle, the first image associated with a first location and a first orientation of the robotic system within the store;
detecting a first shelf represented proximal a first region of the first image;
identifying an address of the first shelf based on the first location, the first orientation, and a vertical position of the first shelf within the first image;
based on the address of the first shelf, retrieving a first list of products associated with the first shelf by the planogram;
retrieving a first set of template images from a database of template images, each template image in the first set of template images comprising visual features of a product specified in the first list of products;
extracting a first set of features from the first region of the first image;
detecting an understock condition of a first product, in the first list of products, on the first shelf in response to deviation between features in the first set of features and features in a first template image, in the first set of template images, representing the first product; and
in response to detecting the understock condition of the first product on the first shelf, generating a first restocking prompt for the first shelf.

8. The method of claim 5 , further comprising, for each image in the set of images:
scanning the image for an aisle address; and
projecting the aisle address onto the map based on a location within the map associated with the image.

9. The method of claim 5 , further comprising:
dispatching the robotic system to execute the scan cycle in the store;
at the robotic system, during the scan cycle:
navigating autonomously along a set of aisles within the store;
constructing the map of the floor space of the store; and
intermittently recording photographic images while navigating along aisles in the store; and

for each sequence of photographic images record

Assignees

• Simbe Robotics Inc

Inventors

• Shah Mirza Akbar
• Bogolea Bradley
• Gee Jeffrey
• Safi Jariullah
• Fraser Luke

Classifications

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• B25J11/008 Primary

  Manipulators for service tasks · CPC title

• B64U2201/20

  Remote controls · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• B25J13/089

  Determining the position of the robot with reference to its environment · CPC title