Method for automatically generating waypoints for imaging shelves within a store

Patent metadata

Field	Value
Publication number	US-10399230-B2
Application number	US-201715600556-A
Country	US
Kind code	B2
Filing date	May 19, 2017
Priority date	May 19, 2016
Publication date	Sep 3, 2019
Grant date	Sep 3, 2019

Abstract

One variation of a method for automatically generating waypoints for imaging shelves within a store includes: dispatching a robotic system to autonomously generating a map of a floor space within the store; accessing an architectural metaspace defining target locations and addresses of the set of shelving structures within the store; distorting the architectural metaspace into alignment with the map to generate a normalized metaspace representing real locations and addresses of the set of shelving structures in the store; defining a set of waypoints distributed longitudinally along and offset laterally from a first shelving structure represented in the normalized metaspace based on a known position of an optical sensor in the robotic system; and dispatching the robotic system to record optical data while occupying the set of waypoints during an imaging routine.

First claim

We claim:

1. A method for automatically generating waypoints for imaging shelves within a store comprises:
dispatching a robotic system to autonomously collect map data of a floor space within the store during a mapping routine;
generating a map of the floor space from map data collected by the robotic system during the mapping routine;
accessing an architectural metaspace defining target locations and addresses of the set of shelving structures within the store;
distorting features in the architectural metaspace into alignment with like features in the map to generate a normalized metaspace representing real locations and addresses of the set of shelving structures in the store;
defining a coordinate system within the normalized metaspace;
defining a set of waypoints within the normalized metaspace relative to the coordinate system, the set of waypoints comprising a first subset of waypoints distributed longitudinally along and offset laterally from a real location of a first shelving structure, in the set of shelving structures, represented in the normalized metaspace, each waypoint in the first subset of waypoints defining an orientation relative to the coordinate system based on a known position of an optical sensor in the robotic system and the real location of the first shelving structure in the normalized metaspace; and
dispatching the robotic system to record optical data while occupying each waypoint in the first subset of waypoints during an imaging routine.

2. The method of claim 1 :
further comprising:
accessing, from a planogram of the store, a first elevation map associated with a first address of the first shelving structure, the first elevation map specifying locations of product facings assigned to a first side of the first shelving structure;
extracting locations of a first set of shelving segments in the first shelving structure from the first elevation map; and
projecting locations of the first set of shelving segments onto the location of the first shelving structure in the normalized metaspace to locate shelving segments in the first shelving structure in the normalized metaspace;

wherein defining the set of waypoints within the normalized metaspace comprises:
defining waypoints longitudinally centered between each shelving segment in the first shelving structure represented in the normalized metaspace; and
calculating an angular orientation normal to a longitudinal face of the first shelving structure, relative to the coordinate system, in the normalized metaspace; and
writing the angular orientation to each waypoint in the first subset of waypoints.

3. The method of claim 2 , further comprising:
for each shelving segment in each shelving structure represented in the normalized metaspace, calculating a linear combination of values of products assigned to the shelving segment by the planogram;
for each waypoint in the set of waypoints, assigning a value to the waypoint based on a linear combination of values of products assigned to a shelving segment adjacent the waypoint;
defining an order for the set of waypoints based on values assigned to the set of waypoints; and
serving the order for the set of waypoints to the robotic system for execution during the imaging routine.

4. The method of claim 3 , wherein defining the order for the set of waypoints comprises calculating the order for the set of waypoints that minimizes total distance traversed by the robotic system from a docking station within the store, to a first waypoint adjacent a first shelving segment associated with a greatest linear combination of values of assigned products, to a second waypoint adjacent a second shelving segment associated with a lowest linear combination of values of assigned products, and back to the docking location.

5. The method of claim 2 , wherein generating the normalized metaspace comprises generating the normalized metaspace representing real gaps between shelving structures, misalignment of adjacent shelving structures, shelving structures in the store omitted from the architectural plan, shelving structures in the store added over the architectural plan, and positions of shelving segments within the store.

6. The method of claim 2 :
further comprising extracting a height of the first shelving structure from the first elevation map; and
wherein defining the set of waypoints within the normalized metaspace comprises:
calculating a target lateral offset distance from the first shelving structure based on the height of the first shelving structure and optical capabilities of a camera integrated into the robotic system; and
defining the first subset of waypoints offset laterally from the longitudinal face of the first shelving structure by the target lateral offset distance.

7. The method of claim 2 , further comprising:
for each waypoint in the set of waypoints, labeling the waypoint with an address of a particular shelving structure and a particular shelving segment predicted to fall within a field of view of the optical sensor when the robotic system occupies a location and an orientation defined by the waypoint;
during the imaging routine, receiving a first image recorded by the robotic system while occupying a first waypoint, in the set of waypoints, adjacent a first shelving segment in the first shelving structure;
accessing a first list of identifiers of products assigned to the first shelving segment based on a first address of the first shelving segment stored with the first waypoint;
retrieving a first set of template images, from a database of template images, representing products in the first list of products stored with the first waypoint; and
identifying products on shelves in the first shelving segment based on similarities between features detected in the first image and template images in the first set of template images.

8. The method of claim 2 , further comprising:
during the imaging routine, receiving a first image recorded by a first camera in the robotic system while occupying a first waypoint adjacent a first shelving segment in the first shelving structure;
projecting heights of a first set of shelves assigned to the first shelving segment by the first elevation map onto the first image based on a known position of the first camera on the robotic system;
identifying a first region in the image corresponding to a first shelf in the first shelving segment;
extracting a first list of identifiers of products assigned to the first shelf by the first elevation map;
retrieving a first set of template images, from a database of template images, representing products in the first list of products; and
identifying products on the first shelf based on similarities between features detected in the first region of the first image and template images in the first set of template images.

9. The method of claim 1 , wherein generating the normalized metaspace comprises:
detecting shelving structures represented in the map;
detecting shelving structures represented in the architectural metaspace;
calculating a transform that approximately centers shelving structures detected in the map onto shelving structures detected in the architectural metaspace; and
applying the transform to shelving structures and to locations of shelving structure addresses in the architectural metaspace to generate the normalized metaspace.

10. The method of claim 9 ,
wherein accessing the architectural metaspace comprises accessing a standardized store layout for a retail brand associated with the store, the standardized store layout defining generic target locations of shelving structures in stores associated with the retail brand;
wherein detecting shelving structures

Assignees

• Simbe Robotics Inc

Inventors

• Shah Mirza Akbar
• Bogolea Bradley
• Gee Jeffrey
• Safi Jariullah
• Fraser Luke

Classifications

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• B25J11/008 Primary

  Manipulators for service tasks · CPC title

• B64U2201/20

  Remote controls · CPC title

• G06Q20/00

  Payment architectures, schemes or protocols (apparatus for performing or posting payment transactions G07F7/08, G07F19/00; electronic cash registers G07G1/12) · CPC title

• B25J9/1664

  characterised by motion, path, trajectory planning · CPC title