Method, apparatus, device and system for processing commodity identification and storage medium

Patent metadata

Field	Value
Publication number	US-11023717-B2
Application number	US-201916354054-A
Country	US
Kind code	B2
Filing date	Mar 14, 2019
Priority date	Jun 29, 2018
Publication date	Jun 1, 2021
Grant date	Jun 1, 2021

Abstract

The present application provides a method, an apparatus, a device and a system for processing commodity identification and a storage medium, where the method includes: receiving image information transmitted by a camera apparatus and a distance signal transmitted by a distance sensor corresponding to the camera apparatus; determining a start frame and an end frame for a pickup behavior of a user according to the image information and the distance signal; and determining, according to the start frame and the end frame for the pickup behavior of the user, information of a commodity taken by the user. By performing a commodity identification on the start frame and the end frame for the pickup behavior of the user, and determining the information of the commodity taken by the user, commodity identification efficiency is effectively improved.

First claim

What is claimed is:

1. A method for processing commodity identification, comprising:
receiving image information transmitted by a camera apparatus and a distance signal transmitted by a distance sensor corresponding to the camera apparatus;
determining a start frame and an end frame for a pickup behavior of a user according to the image information and the distance signal; and
determining, according to the start frame and the end frame for the pickup behavior of the user, information of a commodity taken by the user;
wherein the determining the start frame and the end frame for the pickup behavior of the user according to the image information and the distance signal comprises:
performing difference identification between a next frame and a preceding frame for the image information to determine whether there is a target object extends in or extends out; and determining, according to the distance signal, whether there is a target object extends in or extends out, wherein the next frame and the preceding frame are adjacent frames, or the next frame is spaced apart from the preceding frame by a preset number of frames;
if it is identified according to the image information that a target next frame has a target object extending in relative to a target preceding frame, then determining the target preceding frame as the start frame for the pickup behavior of the user;
if it is determined according to the distance signal that there is a target object extends in, then determining a frame in the image information corresponding to a time of the distance signal as the start frame for the pickup behavior of the user; and
if it is determined according to the image information that a target next frame has a target object extending out relative to a target preceding frame and it is determined according to the distance signal that there is a target object extends out, then determining the target next frame as the end frame for the pickup behavior of the user.

2. The method according to claim 1 , wherein the performing the difference identification between the next frame and the preceding frame for the image information comprises:
performing the difference identification between the next frame and the preceding frame for the image information using a preset first convolutional neural network (CNN) model, wherein the first CNN model is trained using multiple image frames in training data and inter-frame difference annotated data.

3. The method according to claim 1 , wherein the determining, according to the distance signal, whether there is a target object extends in or extends out comprises:
comparing the distance signal with a currently stored background distance signal, wherein the currently stored background distance signal is a stable signal transmitted by the distance sensor when there is no pickup from the user;
if the distance signal has a significant jump relative to the currently stored background distance signal, then determining that there is a target object extends in;
after it is determined that there is a target object extends in, if the distance signal is restored to coincide with the currently stored background distance signal, then determining that the target object extends out.

4. The method according to claim 1 , wherein the determining, according to the start frame and the end frame for the pickup behavior of the user, the information of the commodity taken by the user comprises:
entering the start frame and the end frame for the pickup behavior of the user into a preset commodity identification model to obtain a corresponding relationship between a commodity identifier and a commodity position in the start frame and the end frame, wherein the commodity position is a position of a commodity corresponding to the commodity identifier in the image; and
determining, according to the corresponding relationship between the commodity identifier and the commodity position in the start frame and the end frame, an identifier of the commodity taken by the user.

5. The method according to claim 4 , before the determining, according to the corresponding relationship between the commodity identifier and the commodity position in the start frame and the end frame, the identifier of the commodity taken by the user, further comprising:
performing a different region detection on the start frame and the end frame with a different region detection algorithm to obtain positional information of a different region in the end frame relative to the start frame;
correspondingly, the determining, according to the corresponding relationship between the commodity identifier and the commodity position in the start frame and the end frame, the identifier of the commodity taken by the user comprises:
determining, according to the positional information of the different region in the end frame relative to the start frame and the corresponding relationship between the commodity identifier and the commodity position in the start frame and the end frame, the identifier of the commodity taken by the user.

6. The method according to claim 5 , wherein the performing the different region detection on the start frame and the end frame with the different region detection algorithm to obtain the positional information of the different region in the end frame relative to the start frame comprises:
extracting a first high-dimensional feature map of the start frame and a second high-dimensional feature map of the end frame;
comparing the first high-dimensional feature map with the second high-dimensional feature map to obtain a feature point having a difference;
determining boundary coordinates of a different region in the second high-dimensional feature map relative to the first high-dimensional feature map according to the feature point having the difference, wherein the boundary coordinates are coordinates in a high-dimensional feature map coordinate system; and
determining the positional information of the different region of the end frame relative to the start frame according to the boundary coordinates of the different region in the second high-dimensional feature map relative to the first high-dimensional feature map.

7. The method according to claim 6 , wherein the extracting the first high-dimensional feature map of the start frame and the second high-dimensional feature map of the end frame comprises:
extracting the first high-dimensional feature map of the start frame and the second high-dimensional feature map of the end frame using a preset second CNN model.

8. The method according to claim 6 , wherein the comparing the first high-dimensional feature map with the second high-dimensional feature map to obtain the feature point having the difference comprises:
for any first feature point in the first high-dimensional feature map and a second feature point in the second high-dimensional feature map having same coordinates as the first feature point, calculating a distance between the first feature point and the second feature point, if the distance between the first feature point and the second feature point is greater than a preset threshold, then determining that there is a difference between the second feature point and the first feature point, and taking the second feature point as the feature point having the difference.

9. The method according to claim 1 , wherein after the determining the start frame and the end frame for the pickup behavior of the user according to the image information and the distance signal, further comprising:
storing the start frame and the end frame for the pickup behavior of the user.

10. A system for processing commodity identification, comprising: an artificial intelligence (AI) chip, one or m

Assignees

• Baidu online network technology beijing co ltd

Inventors

• Fu Peng
• Wang Kuipeng
• Hu Yuexiang
• Zhou Qiang
• Fan Yanwen
• Kou Haofeng
• He Shengyi
• Zhou Renyi
• Fang Yanghua
• Bao Yingze

Classifications

• G06V20/52 Primary

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V40/20 Primary

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06F2218/00

  Aspects of pattern recognition specially adapted for signal processing · CPC title

• G06F18/2111

  by using evolutionary computational techniques, e.g. genetic algorithms · CPC title

• G06N3/045

  Combinations of networks · CPC title