Intelligent forklift and method for detecting pose deviation of container

Patent metadata

Field	Value
Publication number	US-11625854-B2
Application number	US-202117508784-A
Country	US
Kind code	B2
Filing date	Oct 22, 2021
Priority date	Apr 25, 2019
Publication date	Apr 11, 2023
Grant date	Apr 11, 2023

Abstract

An intelligent forklift and a method for detecting a pose deviation of a container are provided. The intelligent forklift includes a working state monitor, an image sensor, and a processor. The working state monitor is configured to monitor a working state of the intelligent forklift while the intelligent forklift is carrying and moving a stock container and send a trigger to the image sensor based on the working state. The image sensor is configured to capture an RGBD image frame containing the stock container in response to the trigger and to transmit the RGBD image frame to the processor. The processor is configured to receive the RGBD image frame, detect a pose deviation degree of the stock container and provide an alarm and/or a prompt based on a detection result of the pose deviation degree for adjusting the pose of the stock container.

First claim

What is claimed is:

1. An intelligent forklift, comprising: a working state monitor, at least one image sensor, and a processor; wherein:
the at least one image sensor is electrically connected with the working state monitor and the processor respectively;
the working state monitor is configured to monitor a working state of a stock container while the intelligent forklift is carrying and moving the stock container, and to control the at least one image sensor to capture an image based on the working state;
the working state monitor is configured to trigger the at least one image sensor to capture an image in response to monitoring that an action of lifting up the stock container by the intelligent forklift is finished; and stop an image capturing action of the at least one image sensor in response to monitoring an action of placing the stock container by the intelligent forklift is finished;
the at least one image sensor is configured to capture a red-green-blue-depth (RGBD) image frame containing the stock container in response to a trigger from the working state monitor, and to transmit the RGBD image frame to the processor;
the processor is configured to receive the RGBD image frame captured by the at least one image sensor during a process of moving the stock container, detect a pose deviation degree of the stock container based on the RGBD image frame; provide an alarm and/or a prompt based on a detection result of the pose deviation degree for adjusting the pose of the stock container on the intelligent forklift, wherein the pose deviation degree comprises a relative deviation degree between the stock container and a fork of the intelligent forklift, and a relative deviation degree between an item on the stock container and the stock container; and
wherein the processor is configured to detect the pose deviation degree of the stock container based on a depth feature and an area feature of a target region of interest containing the stock container, a depth feature and an area feature of a preset template to be compared, wherein the preset template to be compared at least comprises a standard container template and a previous RGBD image frame.

2. The intelligent forklift of claim 1 , wherein the at least one image sensor is installed above the fork of the intelligent forklift.

3. The intelligent forklift of claim 1 , wherein the RGBD image frame comprises an RGB image and a depth map, and an image used to represent the stock container is located within a designated region of the RGBD image frame.

4. The intelligent forklift of claim 3 , wherein the processor is configured to:
determine, from the RGB image of the RGBD image frame, a position of the stock container in the RGB image based on a predetermined container recognition model;
cut the target region of interest containing the stock container from the depth map of the RGBD image frame based on the position of the stock container in the RGB image;
determine a depth feature of the target region of interest and an area feature of the target region of interest, and determine a depth feature difference and an area feature difference based on the depth feature of the target region of interest, the area feature of the target region of interest, a depth feature of a preset template to be compared, and an area feature of the preset template to be compared; and
detect the pose deviation degree of the stock container based on the depth feature difference, the area feature difference, a preset depth feature threshold and a preset area feature threshold.

5. The intelligent forklift of claim 4 , wherein the processor is configured to:
obtain a down-sampled region of interest image by performing down-sampling on the target region of interest based on a preset number of layers for pyramid images constructed for the target region of interest;
obtain the depth feature of the target region of interest by summing up depth values of respective pixels contained in the down-sampled region of interest image; and
determine an area of the down-sampled region of interest image, and obtain the area feature of the target region of interest.

6. The intelligent forklift of claim 1 , wherein the standard container template is obtained based on a mean value of a preset number of initially captured RGBD image frames.

7. The intelligent forklift of claim 1 , wherein the processor is configured to:
determine that there is a deviation in a pose of the stock container on the intelligent forklift, in response to detecting at least one of:
a depth feature difference between the RGBD image frame and the standard container template being greater than a preset standard depth feature threshold;
an area feature difference between the RGBD image frame and the standard container template being greater than a preset standard area feature threshold;
the depth feature difference between the RGBD image frame and the previous RGBD image frame of the RGBD image frame being greater than a preset adjacent depth feature threshold; or
the area feature difference between the RGBD image frame and the previous RGBD image frame of the RGBD image frame being greater than the preset adjacent area feature threshold.

8. A method for detecting a pose deviation of a container, comprising:
capturing a red-green-blue-depth (RGBD) image frame containing a stock container during a process of moving the stock container in response to monitoring an action of lifting up the stock container by an intelligent forklift is finished, wherein the RGBD image frame comprises a red-green-blue (RGB) image and a depth map;
determining, from the RGB image of the RGBD image frame, a position of the stock container in the RGB image based on a predetermined container recognition model;
cutting, from the depth map of the RGBD image frame, a target region of interest containing the stock container based on the position of the stock container in the RGB image;
detecting a pose deviation degree of the stock container based on a depth feature of the target region of interest, an area feature of the target region of interest, a depth feature of a preset template to be compared, an area feature of the preset template to be compared, a preset depth feature threshold, and a preset area feature threshold; wherein the pose deviation degree comprises a relative deviation degree between the stock container and a fork of the intelligent forklift, and a relative deviation degree between an item on the stock container and the stock container; the preset template to be compared at least comprises a standard container template and a previous RGBD image frame; and
stopping an image capturing action in response to monitoring that an action of placing the stock container by the intelligent forklift is finished.

9. The method of claim 8 , wherein detecting the pose deviation degree of the stock container comprises:
determining the depth feature of the target region of interest and the area feature of the target region of interest;
determining a depth feature difference and an area feature difference based on the depth feature of the target region of interest, the area feature of the target region of interest, the depth feature of the preset template to be compared, and the area feature of the preset template to be compared; and
detecting the pose deviation degree of the stock container based on the depth feature difference, the area feature difference, and the preset depth feature threshold and the preset area feature threshold.

10. The method of claim 9 , wherein determining the depth feature of the target region of interest and the area feature of the target region of interest comprises:
obtaining a down-sampled region of interest image by

Assignees

• Beijing Geekplus Tech Co Ltd

Inventors

• Guo Xiaoli
• Yang Ningning

Classifications

• B66F9/0755 Primary

  Position control; Position detectors · CPC title

• G01C11/00

  Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying · CPC title

• G06V10/22

  by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition · CPC title

• G01C11/04

  Interpretation of pictures · CPC title

• G06T7/62

  of area, perimeter, diameter or volume · CPC title