Methods and apparatuses for corner detection using neural network and corner detector

Patent metadata

Field	Value
Publication number	US-12007564-B2
Application number	US-202318315766-A
Country	US
Kind code	B2
Filing date	May 11, 2023
Priority date	May 17, 2019
Publication date	Jun 11, 2024
Grant date	Jun 11, 2024

Abstract

An apparatus configured to be head-worn by a user, includes: a screen configured to present graphics for the user; a camera system configured to view an environment in which the user is located; and a processing unit coupled to the camera system, the processing unit configured to: obtain locations of features for an image of the environment, wherein the locations of the features are identified by a neural network; determine a region of interest for one of the features in the image, the region of interest having a size that is less than a size of the image; and perform a corner detection using a corner detection algorithm to identify a corner in the region of interest.

First claim

What is claimed is:

1. A method of training and using a neural network for image interest point detection, comprises:
generating a reference dataset comprising a plurality of reference sets, wherein each of the plurality of reference sets comprises:
an image, and
a set of reference interest points corresponding to the image; and

for each reference set of the plurality of reference sets:
generating a warped image by applying a homograph to the image,
generating a warped set of reference interest points by applying the homograph to the set of reference interest points,
calculating, by the neural network receiving the image as input, a set of calculated interest points and a calculated descriptor,
calculating, by the neural network receiving the warped image as input, a set of calculated warped interest points and a calculated warped descriptor,
calculating a loss based on the set of calculated interest points, the calculated descriptor, the set of calculated warped interest points, the calculated warped descriptor, the set of reference interest points, the warped set of reference interest points, and the homograph, and
modifying the neural network based on the loss; wherein the method further comprises: obtaining locations of features in an input image, wherein the locations of the features are identified by the neural network; determining a region of interest for one of the features in the input image, the region of interest having a size that is less than a size of the input image; and performing a corner detection using a corner detection algorithm to identify a corner in the region of interest.

2. The method of claim 1 , wherein the neural network comprises an interest point detector subnetwork and a descriptor subnetwork,
wherein the interest point detector subnetwork is configured to receive the image as input and calculate the set of calculated interest points based on the image, and
wherein the descriptor subnetwork is configured to receive the image as input and calculate the calculated descriptor based on the image.

3. The method of claim 1 , wherein modifying the neural network based on the loss comprises modifying one or both of the interest point detector subnetwork and the descriptor subnetwork based on the loss.

4. The method of claim 1 , further comprising prior to generating the reference dataset, training the interest point detector subnetwork using a synthetic dataset comprising a plurality of synthetic images and a plurality of sets of synthetic interest points,
wherein generating the reference dataset comprises generating the reference dataset using the interest point detector subnetwork.

5. The method of claim 1 , wherein generating the reference dataset comprises:
for each reference set of the plurality of reference sets:
obtaining the image from an unlabeled dataset comprising a plurality of unlabeled images,
generating a plurality of warped images by applying a plurality of homographs to the image,
calculating, by the neural network receiving the plurality of warped images as input, a plurality of sets of calculated warped interest points,
generating a plurality of sets of calculated interest points by applying a plurality of inverse homographs to the plurality of sets of calculated warped interest points, and
aggregating the plurality of sets of calculated interest points to obtain the set of reference interest points.

6. The method of claim 1 , wherein each of the plurality of reference sets further includes a reference descriptor corresponding to the image, and wherein generating the reference dataset includes:
for each reference set of the plurality of reference sets:
obtaining the image from an unlabeled dataset comprising a plurality of unlabeled images,
generating a plurality of warped images by applying a plurality of homographs to the image,
calculating, by the neural network receiving the plurality of warped images as input, a plurality of calculated warped descriptors,
generating a plurality of calculated descriptors by applying a plurality of inverse homographs to the plurality of calculated warped descriptors, and
aggregating the plurality of calculated descriptors to obtain the reference descriptor.

7. The method of claim 1 , wherein the set of reference interest points is a two-dimensional map having values corresponding to a probability that a particular pixel of the image has an interest point is located at the particular pixel.

8. A method comprising:
capturing a first image;
capturing a second image;
calculating, by a neural network receiving the first image as input, a first set of calculated interest points and a first calculated descriptor;
calculating, by the neural network receiving the second image as input, a second set of calculated interest points and a second calculated descriptor; and
determining a homograph between the first image and the second image based on the first and second sets of calculated interest points and the first and second calculated descriptors,
wherein the neural network includes: an interest point detector subnetwork configured to calculate the first set of calculated interest points and the second set of calculated interest points, and a descriptor subnetwork configured to calculate the first calculated descriptor and the second calculated descriptor, and
wherein the method further comprises:
obtaining locations of features in an input image, wherein the locations of the features are identified by the neural network;
determining a region of interest for one of the features in the input image, the region of interest having a size that is less than a size of the input image; and
performing a corner detection using a corner detection algorithm to identify a corner in the region of interest.

9. The method of claim 8 , wherein:
the interest point detector subnetwork is configured to calculate the first set of calculated interest points concurrently with the descriptor subnetwork calculating the first calculated descriptor; and
the interest point detector subnetwork is configured to calculate the second set of calculated interest points concurrently with the descriptor subnetwork calculating the second calculated descriptor.

10. The method of claim 8 , further comprising training the neural network by generating a reference dataset comprising a plurality of reference sets, wherein each of the plurality of reference sets includes an image and a set of reference interest points corresponding to the image; and
for each reference set of the plurality of reference sets:
generating a warped image by applying a homograph to the image,
generating a warped set of reference interest points by applying the homograph to the set of reference interest points,
calculating, by the neural network receiving the image as input, a set of calculated interest points and a calculated descriptor,
calculating, by the neural network receiving the warped image as input, a set of calculated warped interest points and a calculated warped descriptor,
calculating a loss based on the set of calculated interest points, the calculated descriptor, the set of calculated warped interest points, the calculated warped descriptor, the set of reference interest points, the warped set of reference interest points, and the homograph, and
modifying the neural network based on the loss.

11. The method of claim 10 , wherein modifying the neural network based on the loss includes modifying one or both of the interest point detector subnetwork and the descriptor subnetwork based on the loss.

12. The method of claim 10 , further comprising prior to generating the reference dataset, trainin

Assignees

• Magic Leap Inc

Inventors

• Mohan Anush
• Rosenthal Guy
• Perry Adi
• Sheveki Gilboa

Classifications

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06T3/18

  Image warping, e.g. rearranging pixels individually · CPC title