Key points detection using multiple image modalities

Patent metadata

Field	Value
Publication number	US-11967102-B2
Application number	US-202117378495-A
Country	US
Kind code	B2
Filing date	Jul 16, 2021
Priority date	Jul 16, 2021
Publication date	Apr 23, 2024
Grant date	Apr 23, 2024

Abstract

Image-based key points detection using a convolutional neural network (CNN) may be impacted if the key points are occluded in the image. Images obtained from additional imaging modalities such as depth and/or thermal images may be used in conjunction with RGB images to reduce or minimize the impact of the occlusion. The additional images may be used to determine adjustment values that are then applied to the weights of the CNN so that the convolution operations may be performed in a modality aware manner to increase the robustness, accuracy, and efficiency of key point detection.

First claim

The invention claimed is:

1. A method for detecting key points of an object, the method comprising:
obtaining a first image of the object, wherein the first image comprises a first imaging property of the object;
obtaining a second image of the object, wherein the second image comprises a second imaging property of the object; and
processing the first image through a convolutional neural network (CNN) based on at least the first imaging property and the second imaging property to determine one or more key points of the object, wherein:
the CNN comprises a first convolution layer that includes multiple kernels each associated with a respective set of weights;
processing the first image through the CNN comprises performing, via the first convolution layer of the CNN, a first set of convolution operations on the first image based on the first imaging property of the object; and
performing the first set of convolution operations on the first image comprises, for a pixel that belongs to a patch of the first image:
identifying a corresponding patch in the second image;
adjusting a weight of the first convolution layer to be applied to the pixel, wherein the adjustment is made based on the second imaging property associated with two or more pixels of the corresponding patch in the second image; and
performing the first set of convolution operations based on the adjusted weight of the first convolution layer.

2. The method of claim 1 , wherein the first image comprises a red/green/blue (RGB) image of the object and the second image comprises a depth or thermal image of the object.

3. The method of claim 1 , wherein the one or more key points are occluded in the first image.

4. The method of claim 1 , wherein the CNN further comprises a second convolution layer that includes multiple kernels each associated with a respective second set of weights, the method further comprises performing a second set of convolution operations on an output of the first convolution layer, and wherein the second set of weights associated with each of the multiple kernels of the second convolution layer is adjusted based on the second imaging property of the object during the performance of the second set of convolution operations.

5. The method of claim 1 , wherein the CNN further comprises a second convolution layer that includes multiple kernels each associated with a respective second set of weights, the method further comprises performing a second set of convolution operations on an output of the first convolution layer, and wherein the second set of weights associated with each of the multiple kernels of the second convolution layer is not adjusted based on the second imaging property of the object during the performance of the second set of convolution operations.

6. The method of claim 1 , wherein the CNN comprises multiple additional convolution layers each associated with a respective set of weights, the method further comprises performing convolution operations on the first image via the multiple additional convolution layers, and wherein the respective set of weights associated with each of the multiple additional convolution layers is adjusted based on the second imaging property of the object during the performance of the convolution operations.

7. The method of claim 1 , wherein the one or more key points of the object are determined using a single branch of the CNN configured to process the first image in conjunction with the second image.

8. The method of claim 1 , wherein the one or more key points include a joint location.

9. An apparatus configured to detect key points of an object, the apparatus comprising:
one or more processors configured to:
obtain a first image of the object, wherein the first image comprises a first imaging property of the object;
obtain a second image of the object, wherein the second image comprises a second imaging property of the object; and
process the first image through a convolutional neural network (CNN) based on at least the first imaging property and the second imaging property to determine one or more key points of the object, wherein:
the CNN comprises a first convolution layer that includes multiple kernels each associated with a respective set of weights;
the one or more processors being configured to process the first image through the CNN comprises the one or more processors being configured to perform, via the first convolution layer of the CNN, a first set of convolution operations on the first image based on the first imaging property; and
the one or more processors being configured to perform the first set of convolution operations on the first image comprises the one or more processors being configured to, for a pixel that belongs to a patch of the first image:
identify a corresponding patch in the second image;
adjust a weight of the first convolution layer to be applied to the pixel, wherein the adjustment is made based on the second imaging property associated with two or more pixels of the corresponding patch in the second image; and
perform the first set of convolution operations based on the adjusted weight of the first convolution layer.

10. The apparatus of claim 9 , wherein the first image comprises a red/green/blue (RGB) image of the object and the second image comprises a depth or thermal image of the object.

11. The apparatus of claim 9 , wherein the one or more key points are occluded in the first image.

12. The apparatus of claim 9 , wherein the CNN further comprises a second convolution layer that includes multiple kernels each associated with a respective second set of weights, the one or more processors are further configured to perform a second set of convolution operations on an output of the first convolution layer, and wherein the second set of weights associated with each of the multiple kernels of the second convolution layer is adjusted based on the second imaging property of the object during the performance of the second set of convolution operations.

13. The apparatus of claim 9 , wherein the CNN further comprises a second convolution layer that includes multiple kernels each associated with a respective second set of weights, the one or more processors are further configured to perform a second set of convolution operations on an output of the first convolution layer, and wherein the second set of weights associated with each of the multiple kernels of the second convolution layer is not adjusted based on the second imaging property of the object.

14. The apparatus of claim 9 , wherein the CNN comprises multiple additional convolution layers each associated with a respective set of weights, the one or more processors are further configured to perform convolution operations on the first image via the multiple additional convolution layers, and wherein the respective set of weights associated with each of the multiple additional convolution layers is adjusted based on the second imaging property of the object during the performance of the convolution operations.

15. The apparatus of claim 9 , wherein the one or more key points of the object are determined using a single branch of the CNN configured to process the first image in conjunction with the second image.

16. The apparatus of claim 9 , wherein the one or more key points include a joint location.

Assignees

• Shanghai United Imaging Intelligence Co Ltd

Inventors

• Sharma Abhishek
• Innanje Arun
• Wu Ziyan

Classifications

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06T7/0012

  Biomedical image inspection · CPC title

• G06V40/103

  Static body considered as a whole, e.g. static pedestrian or occupant recognition · CPC title

• G06T2207/10024

  Color image · CPC title