Semantically aware keypoint matching

Patent metadata

Field	Value
Publication number	US-11830253-B2
Application number	US-202117230947-A
Country	US
Kind code	B2
Filing date	Apr 14, 2021
Priority date	Apr 14, 2020
Publication date	Nov 28, 2023
Grant date	Nov 28, 2023

Abstract

A method for keypoint matching includes receiving an input image obtained by a sensor of an agent. The method also includes identifying a set of keypoints of the received image. The method further includes augmenting the descriptor of each of the keypoints with semantic information of the input image. The method also includes identifying a target image based on one or more semantically augmented descriptors of the target image matching one or more semantically augmented descriptors of the input image. The method further includes controlling an action of the agent in response to identifying the target.

First claim

What is claimed is:

1. A method for keypoint matching performed by a semantically aware keypoint matching model, the method comprising:
receiving an input image obtained by a sensor of an agent;
identifying a set of keypoints of the received image, each of the keypoints corresponding to a different descriptor;
augmenting, for each keypoint of the set of keypoints, the descriptor associated with the keypoint with semantic information associated with one or more respective pixels corresponding to the keypoint, the semantic information indicating a semantic label associated with the one or more respective pixels;
identifying a target image based on one or more semantically augmented descriptors of the target image matching one or more semantically augmented descriptors of the input image; and
controlling an action of the agent in response to identifying the target image.

2. The method of claim 1 , further comprising identifying the one or more semantically augmented descriptors of the target image matching the one or more semantically augmented descriptors of the input image based on a nearest neighbor matching function.

3. The method of claim 2 , further comprising biasing the matching of the one or more semantically augmented descriptors of the target image and the one or more semantically augmented descriptors of the input image toward descriptors with matching semantic labels.

4. The method of claim 1 , further comprising obtaining the semantic information from a semantic segmentation model trained to label one or more pixels of the input image.

5. The method of claim 1 , further comprising identifying a current location of the agent based on the identified target image, wherein controlling the action of the agent comprises navigating to a location based on the identified current location.

6. The method of claim 1 , further comprising identifying the target image from a plurality of stored images, each of the stored images associated with one or more semantically augmented descriptors.

7. The method of claim 1 , wherein the agent is an autonomous vehicle.

8. The method of claim 1 , further comprising embedding the semantic information into a target space; and
identifying a matching descriptor based on nearest neighbor matching.

9. An apparatus for keypoint matching performed at a semantically aware keypoint matching model, the apparatus comprising:
a processor;
a memory coupled with the processor; and
instructions stored in the memory and operable, when executed by the processor, to cause the apparatus:
to receive an input image obtained by a sensor of an agent;
to identify a set of keypoints of the received image, each of the keypoints corresponding to a different descriptor;
to augment, for each keypoint of the set of keypoints, the descriptor associated with the keypoint with semantic information associated with one or more respective pixels corresponding to the keypoint, the semantic information indicating a semantic label associated with the one or more respective pixels;
to identify a target image based on one or more semantically augmented descriptors of the target image matching one or more semantically augmented descriptors of the input image; and
to control an action of the agent in response to identifying the target image.

10. The apparatus of claim 9 , wherein execution of the instructions further cause the apparatus to identify the one or more semantically augmented descriptors of the target image matching the one or more semantically augmented descriptors of the input image based on a nearest neighbor matching function.

11. The apparatus of claim 10 , wherein execution of the instructions further cause the apparatus to bias the matching of the one or more semantically augmented descriptors of the target image and the one or more semantically augmented descriptors of the input image toward descriptors with matching semantic labels.

12. The apparatus of claim 9 , wherein execution of the instructions further cause the apparatus to obtain the semantic information from a semantic segmentation model trained to label one or more pixels of the input image.

13. The apparatus of claim 9 , wherein execution of the instructions further cause the apparatus to identify a current location of the agent based on the identified target image, wherein controlling the action of the agent comprises navigating to a location based on the identified current location.

14. The apparatus of claim 9 , wherein execution of the instructions further cause the apparatus to identify the target image from a plurality of stored images, each of the stored images associated with one or more semantically augmented descriptors.

15. The apparatus of claim 9 , wherein the agent is an autonomous vehicle.

16. The apparatus of claim 9 , wherein execution of the instructions further cause the apparatus to embed the semantic information into a target space.

17. A non-transitory computer-readable medium having program code recorded thereon for keypoint matching performed at a semantically aware keypoint matching model, the program code executed by a processor and comprising:
program code to receive an input image obtained by a sensor of an agent;
program code to identify a set of keypoints of the received image, each of the keypoints corresponding to a different descriptor;
program code to augment, for each keypoint of the set of keypoints, the descriptor associated with the keypoint with semantic information associated with one or more respective pixels corresponding to the keypoint, the semantic information indicating a semantic label associated with the one or more respective pixels;
program code to identify a target image based on one or more semantically augmented descriptors of the target image matching one or more semantically augmented descriptors of the input image; and
program code to control an action of the agent in response to identifying the target image.

18. The non-transitory computer-readable medium of claim 17 , wherein the program code further comprises program code to identify the one or more semantically augmented descriptors of the target image matching the one or more semantically augmented descriptors of the input image based on a nearest neighbor matching function.

19. The non-transitory computer-readable medium of claim 18 , wherein the program code further comprises program code to bias the matching of the one or more semantically augmented descriptors of the target image and the one or more semantically augmented descriptors of the input image toward descriptors with matching semantic labels.

20. The non-transitory computer-readable medium of claim 17 , wherein the program code further comprises program code to obtain the semantic information from a semantic segmentation model trained to label one or more pixels of the input image.

Assignees

• Toyota Res Inst Inc

Inventors

• Tang Jiexiong
• Ambrus Rares Andrei
• Guizilini Vitor
• Gaidon Adrien David

Classifications

• G05D1/243

  Means capturing signals occurring naturally from the environment, e.g. ambient optical, acoustic, gravitational or magnetic signals (using passive navigation aids external to the vehicle G05D1/244; using signals from positioning sensors located off-board the vehicle G05D1/249) · CPC title

• G05D2101/10

  using artificial intelligence [AI] techniques · CPC title

• G06V20/56 Primary

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G05D1/0221

  involving a learning process · CPC title

• G05D1/0246

  using a video camera in combination with image processing means · CPC title