Method of processing image, method of training model, and electronic device

Patent metadata

Field	Value
Publication number	US-12450869-B2
Application number	US-202218089709-A
Country	US
Kind code	B2
Filing date	Dec 28, 2022
Priority date	Dec 29, 2021
Publication date	Oct 21, 2025
Grant date	Oct 21, 2025

Abstract

A method of processing an image, a method of training a multi-task processing model, and an electronic device, which relate to a field of an automatic driving technology, in particular to a field of high-definition map technology. The method of processing an image includes: processing a to-be-processed image to obtain a feature point of the to-be-processed image, a feature point descriptor map of the to-be-processed image, and a dense descriptor map of the to-be-processed image; determining a pair of matched feature points between the to-be-processed image and a reference image based on the feature point and the feature point descriptor map; and determining a pair of matched pixels between the to-be-processed image and the reference image based on the dense descriptor map.

First claim

What is claimed is:

1. A method of processing an image, comprising:
processing a to-be-processed image to obtain a feature point of the to-be-processed image, a feature point descriptor map of the to-be-processed image, and a dense descriptor map of the to-be-processed image;
determining a pair of matched feature points between the to-be-processed image and a reference image, based on the feature point and the feature point descriptor map; and
determining a pair of matched pixels between the to-be-processed image and the reference image, based on the dense descriptor map,
wherein the processing a to-be-processed image to obtain a feature point of the to-be-processed image, a feature point descriptor map of the to-be-processed image and a dense descriptor map of the to-be-processed image comprises:
inputting the to-be-processed image into a multi-task processing model to obtain the feature point of the to-be-processed image, the feature point descriptor map of the to-be-processed image and the dense descriptor map of the to-be-processed image,
wherein the multi-task processing model comprises a feature point classification branch, and the method further comprises:
inputting the to-be-processed image into the feature point classification branch to obtain a feature point category result, so as to determine the pair of matched feature points between the to-be-processed image and the reference image based on the feature point, the feature point descriptor map and the feature point category result,
wherein the determining a pair of matched feature points between the to-be-processed image and a reference image based on the feature point and the feature point descriptor map comprises:
screening the feature point based on the feature point category result, so as to determine a target feature point; and
determining the pair of matched feature points between the to-be-processed image and the reference image based on the target feature point and the feature point descriptor map, and
wherein the determining the pair of matched feature points between the to-be-processed image and the reference image based on the target feature point and the feature point descriptor map comprises:
extracting, from the feature point descriptor map, a feature point descriptor matched with the target feature point based on the target feature point; and
determining, by using a feature point matching method, the pair of matched feature points between the to-be-processed image and the reference image based on the target feature point and the feature point descriptor matched with the target feature point.

2. The method according to claim 1 , wherein the multi-task processing model further comprises a feature point extraction branch, a feature point descriptor map calculation branch and a dense descriptor map calculation branch, and
the inputting the to-be-processed image into a multi-task processing model to obtain the feature point of the to-be-processed image, the feature point descriptor map of the to-be-processed image and the dense descriptor map of the to-be-processed image comprises:
inputting the to-be-processed image into the feature point extraction branch to obtain the feature point;
inputting the to-be-processed image into the feature point descriptor map calculation branch to obtain the feature point descriptor map; and
inputting the to-be-processed image into the dense descriptor map calculation branch to obtain the dense descriptor map.

3. The method according to claim 2 , wherein the feature point extraction branch comprises a plurality of feature point extraction sub branches, and
the inputting the to-be-processed image into the feature point extraction branch to obtain the feature point comprises:
inputting the to-be-processed image into the plurality of feature point extraction sub branches, so as to obtain a plurality of feature point scoring maps, wherein the plurality of feature point scoring maps correspond to the plurality of feature point extraction sub branches one by one;
performing a weighted summation on the plurality of feature point scoring maps to obtain a target feature point scoring map; and
performing an interpolation processing on the target feature point scoring map to obtain the feature point.

4. The method according to claim 2 , wherein the dense descriptor map comprises a high-resolution dense descriptor map and a low-resolution dense descriptor map, and the dense descriptor map calculation branch comprises a high-resolution dense descriptor map calculation sub branch and a low-resolution dense descriptor map calculation sub branch, and
wherein the determining a pair of matched pixels between the to-be-processed image and the reference image based on the dense descriptor map comprises:
determining a candidate pair of matched pixels between the to-be-processed image and the reference image based on the low-resolution dense descriptor map, wherein the low-resolution dense descriptor map is obtained by processing the to-be-processed image using the low-resolution dense descriptor map calculation sub branch; and
determining, from the candidate pair of matched pixels, the pair of matched pixels between the to-be-processed image and the reference image based on the high-resolution dense descriptor map, wherein the high-resolution dense descriptor map is obtained by processing the to-be-processed image using the high-resolution dense descriptor map calculation sub branch.

5. The method according to claim 1 , wherein the feature point classification branch comprises a plurality of feature point classification sub branches, a fusion module, and a classifier, and each of the plurality of feature point classification sub branches comprises a feature point classification extraction module, an encoder, and a fusion module, and
the inputting the to-be-processed image into the feature point classification branch to obtain a feature point category result comprises:
inputting the to-be-processed image into the plurality of feature point classification extraction modules, so as to obtain a plurality of feature point category feature maps, wherein the plurality of feature point category feature maps correspond to the plurality of feature point classification extraction modules one by one;
inputting, for each feature point category feature map of the plurality of feature point category feature maps, the feature point category feature map into the encoder so as to obtain a plurality of encoded sub feature point category feature maps, wherein the plurality of encoded sub feature point category feature maps correspond to a plurality of encoders one by one;
processing the plurality of encoded sub feature point category feature maps by using the fusion module, so as to obtain a fused feature point category map; and
processing the fused feature point category map by using the classifier, so as to obtain the feature point category result.

6. The method according to claim 1 , further comprising:
inputting the reference image into the multi-task processing model to obtain a reference feature point, a reference feature point descriptor map and a reference dense descriptor map, so as to determine the pair of matched feature points between the to-be-processed image and the reference image based on the feature point, the feature point descriptor map, the reference feature point and the reference feature point descriptor map, and determine the pair of matched pixels between the to-be-processed image and the reference image based on the dense descriptor map and the reference dense descriptor map.

7. A method of training the a multi-task processing model implemented in the method according to claim 1 , comprising:
training an initial multi-task processing model by using a trainin

Assignees

• Apollo Intelligent Driving Technology Beijing Co Ltd

Inventors

• He Yuzhe
• Zhou Yao
• Hou Shenhua
• Peng Liang
• Wan Guowei

Classifications

• G06T2207/20081

  Training; Learning · CPC title

• G06V10/46

  Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features (colour feature extraction G06V10/56) · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title