Method for locating a warehousing robot, a method of constructing a map, robot and storage medium

Patent metadata

Field	Value
Publication number	US-12466653-B2
Application number	US-202118026105-A
Country	US
Kind code	B2
Filing date	Jul 30, 2021
Priority date	Sep 14, 2020
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

Disclosed are a method for locating a warehousing robot, a method for constructing a map, a robot and a storage medium. In a specific embodiment, a semantic map of a warehouse environment is constructed in advance, and the semantic map comprises a plurality of objects existing in the warehouse environment and semantic information of the objects. In the localization process, a warehousing robot uses its own image sensor to acquire an image or video data of a surrounding environment (11), identifies target objects in the image or video data and semantic information of the target objects (12) to obtain the relative position relationship between each target object and the warehousing robot (13), and then determines the location of the warehousing robot in the semantic map based on the relative position relationship and the semantic information of each target object (14). The method for constructing a map is based on visual semantic localization. Because the method directly detects specific targets, the detection speed is fast, semantic information is rich, and the method is not easily influenced by other interference factors. The method gets rid of the dependence on signs in the warehouse environment and has high localization flexibility.

First claim

What is claimed is:

1 . A method for locating a warehousing robot, comprising:
acquiring an image or video data of a surrounding warehouse environment by using an image sensor on a warehousing robot during movement of the warehousing robot; performing target detection and semantic recognition on the image or video data of the surrounding warehouse environment to obtain target objects in the surrounding warehouse environment and semantic information of the target objects; calculating a relative position relationship between each target object and the warehousing robot based on a transformation relationship between a sensor coordinate system and a robot coordinate system; and determining a location of the warehousing robot in a semantic map according to the relative position relationship between each target object and the warehousing robot and the semantic information of each target object, wherein the semantic map comprises a plurality of objects in the warehouse environment and semantic information of the objects.

2 . The method according to claim 1 , wherein calculating the relative position relationship between each target object and the warehousing robot based on the transformation relationship between a sensor coordinate system and a robot coordinate system comprises:
calculating three-dimensional coordinates of pixels on the target object in the sensor coordinate system; transforming the three-dimensional coordinates of the pixels on the target object from the sensor coordinate system to the robot coordinate system based on the transformation relationship between the sensor coordinate system and the robot coordinate system; and calculating the relative position relationship between the target object and the warehousing robot according to the three-dimensional coordinates of the pixels on the target object in the robot coordinate system.

3 . The method according to claim 2 , wherein
the image sensor is a monocular camera and the target object is an object intersecting a ground plane or an object located on the ground plane, and calculating the three-dimensional coordinates of the pixels on the target object in the sensor coordinate system comprises: calculating the three-dimensional coordinates, on the ground plane, of the pixels intersecting the ground plane on the target object in combination with a photographic geometry of the monocular camera; or the image sensor is a binocular camera, and calculating the three-dimensional coordinates of the pixels on the target object in the sensor coordinate system comprises: calculating the three-dimensional coordinates of the pixels on the target object in the sensor coordinate system by using a binocular stereo matching algorithm; or the image sensor is an RGBD camera, and calculating the three-dimensional coordinates of the pixels on the target object in the sensor coordinate system comprises: calculating the three-dimensional coordinates of the pixels on the target object in the sensor coordinate system according to the matching relationship between an acquired RGB image containing the target object and a depth image containing the target object.

4 . The method according to claim 1 , wherein determining the location of the warehousing robot in a semantic map according to the relative position relationship between each target object and the warehousing robot and the semantic information of each target object comprises:
determining a location of the target object in the semantic map according to the semantic information of the target object; and determining the location of the warehousing robot in the semantic map according to the relative position relationship between the target object and the warehousing robot and the location of the target object in the semantic map.

5 . The method according to claim 4 , wherein determining the location of the target object in the semantic map according to the semantic information of the target object comprises:
calculating an initial location of the warehousing robot in the semantic map based on other sensor data on the warehousing robot; and searching for the location of the target object around the initial location in the semantic map according to the semantic information of the target object.

6 . The method according to claim 5 , wherein calculating the initial location of the warehousing robot in the semantic map based on other sensor data on the warehousing robot comprises:
calculating the initial location of the warehousing robot in the semantic map based on at least one of laser sensor data, IMU data and odometer data on the warehousing robot.

7 . The method according to claim 1 , further comprising:
acquiring the image or video data of the warehouse environment by using the image sensor on the warehousing robot during process of the warehousing robot traversing the warehouse environment; performing target detection and semantic recognition on the image or video data of the warehouse environment to obtain a plurality of objects existing in the warehouse environment and semantic information of the objects; calculating relative position relationships between the plurality of objects and the warehousing robot respectively based on the transformation relationship between the sensor coordinate system and the robot coordinate system; and adding the semantic information of the plurality of objects to a basic environmental map according to the relative position relationships between the plurality of objects and the warehousing robot and a location of the warehousing robot in the basic environmental map when acquiring the image or video data of the warehouse environment, so as to obtain the semantic map of the warehouse environment.

8 . The method according to claim 7 , wherein when acquiring the image or video data of the warehouse environment, the method further comprises:
constructing the basic environmental map of the warehouse environment by using the SLAM technology, and determining the location of the warehousing robot in the basic environmental map when acquiring the image or video data of the warehouse environment.

9 . The method according to claim 1 , wherein the target objects at least comprise: shelves, intersecting lines between the shelves and a ground plane, and intersections formed by the shelves.

10 . The method according to claim 9 , wherein the target objects further comprise: at least one of shelf bases, fire hydrants and escape signs.

11 . A method for constructing a semantic map, comprising:
acquiring an image or video data of a warehouse environment by using an image sensor on a warehousing robot during process of the warehousing robot traversing the warehouse environment; performing target detection and semantic recognition on the image or video data of the warehouse environment to obtain a plurality of objects existing in the warehouse environment and semantic information of the objects; calculating relative position relationships between the plurality of objects and the warehousing robot respectively based on a transformation relationship between a sensor coordinate system and a robot coordinate system; and adding the semantic information of the plurality of objects to a basic environmental map according to the relative position relationships between the plurality of objects and the warehousing robot and a location of the warehousing robot in the basic environmental map when acquiring the image or video data of the warehouse environment, so as to obtain the semantic map of the warehouse environment.

12 . A warehousing robot, comprising a device body including:
a memory for storing computer programs; and a processor coupl

Assignees

• Lingdong Tech Beijing Co Ltd

Inventors

• Zhang Guangpeng

Classifications

• G01C21/206

  specially adapted for indoor navigation · CPC title

• G01C21/3848

  Data obtained from both position sensors and additional sensors · CPC title

• G01C21/383

  Indoor data · CPC title

• B65G1/1375 Primary

  the orders being assembled on a commissioning stacker-crane or truck · CPC title

• G01C21/005 Primary

  with correlation of navigation data from several sources, e.g. map or contour matching (G01C21/30 takes precedence) · CPC title