Systems and methods for determining environment dimensions based on environment pose

Patent metadata

Field	Value
Publication number	US-12165352-B2
Application number	US-202217734399-A
Country	US
Kind code	B2
Filing date	May 2, 2022
Priority date	Jun 15, 2021
Publication date	Dec 10, 2024
Grant date	Dec 10, 2024

Abstract

Disclosed herein are systems and method for determining environment dimensions based on environment pose. In one aspect, the method may include training, with a dataset including a plurality of images featuring an environment and labelled landmarks in the environment, a neural network to identify a pose of an environment. The method may comprise receiving an input image depicting the environment, generating an input tensor based on the input image, and inputting the input tensor into the neural network, which may be configured to generate an output tensor including a position of each identified landmark, a confidence level associated with each position, and a pose confidence score. The method may include calculating a homography matrix between each position in the output tensor along a camera plane and a corresponding position in an environment plane in order to output an image that visually connects each landmark along the environment plane.

First claim

The invention claimed is:

1. A method for determining environment dimensions based on environment pose, the method comprising:
training, with a dataset comprising a plurality of images featuring an environment and labelled landmarks in the environment, a neural network to identify a pose of an environment in an arbitrary image, wherein the pose comprises connected labelled landmarks of the environment;
receiving an input image depicting the environment;
generating an input tensor based on the received input image;
inputting the input tensor into the neural network, wherein the neural network is configured to generate an output tensor comprising a position of each identified landmark, a confidence level associated with each position, and a pose confidence score;
calculating a homography matrix between each position in the output tensor along a camera plane and a corresponding position in an environment plane, based on a pre-built model of the environment; and
outputting an image that visually connects each landmark along the environment plane based on the homography matrix.

2. The method of claim 1 , wherein a camera perspective of the input image does not match any of the camera perspectives of the plurality of images in the dataset.

3. The method of claim 1 , wherein the neural network comprises:
a convolutional backbone with feature extraction layers, and
a segmentation head.

4. The method of claim 1 , wherein the pre-built model of the environment is indicative of distances between each landmark in the environment.

5. The method of claim 1 , wherein the neural network is further configured to determine a heat map for each position of each identified landmark, wherein the heat map represents an area in which the identified landmark may be in the input image.

6. The method of claim 5 , wherein the neural network optimizes a loss using stochastic gradient descent.

7. The method of claim 1 , wherein the input image is a video frame of a livestream, and wherein the neural network determines environment dimensions in real-time.

8. The method of claim 1 , wherein the environment is a sports field and the labelled landmarks are locations on the sports field.

9. A system for determining environment dimensions based on environment pose, the system comprising:
a hardware processor configured to:
train, with a dataset comprising a plurality of images featuring an environment and labelled landmarks in the environment, a neural network to identify a pose of an environment in an arbitrary image, wherein the pose comprises connected labelled landmarks of the environment;
receive an input image depicting the environment;
generate an input tensor based on the received input image;
input the input tensor into the neural network, wherein the neural network is configured to generate an output tensor comprising a position of each identified landmark, a confidence level associated with each position, and a pose confidence score;
calculate a homography matrix between each position in the output tensor along a camera plane and a corresponding position in an environment plane, based on a pre-built model of the environment; and
output an image that visually connects each landmark along the environment plane based on the homography matrix.

10. The system of claim 9 , wherein a camera perspective of the input image does not match any of the camera perspectives of the plurality of images in the dataset.

11. The system of claim 9 , wherein the neural network comprises:
a convolutional backbone with feature extraction layers, and
a segmentation head.

12. The system of claim 9 , wherein the pre-built model of the environment is indicative of distances between each landmark in the environment.

13. The system of claim 9 , wherein the neural network is further configured to determine a heat map for each position of each identified landmark, wherein the heat map represents an area in which the identified landmark may be in the input image.

14. The system of claim 13 , wherein the neural network optimizes a loss using stochastic gradient descent.

15. The system of claim 9 , wherein the input image is a video frame of a livestream, and wherein the neural network determines environment dimensions in real-time.

16. The system of claim 9 , wherein the environment is a sports field and the labelled landmarks are locations on the sports field.

17. A non-transitory computer readable medium storing thereon computer executable instructions for determining environment dimensions based on environment pose, including instructions for:
training, with a dataset comprising a plurality of images featuring an environment and labelled landmarks in the environment, a neural network to identify a pose of an environment in an arbitrary image, wherein the pose comprises connected labelled landmarks of the environment;
receiving an input image depicting the environment;
generating an input tensor based on the received input image;
inputting the input tensor into the neural network, wherein the neural network is configured to generate an output tensor comprising a position of each identified landmark, a confidence level associated with each position, and a pose confidence score;
calculating a homography matrix between each position in the output tensor along a camera plane and a corresponding position in an environment plane, based on a pre-built model of the environment; and
outputting an image that visually connects each landmark along the environment plane based on the homography matrix.

Assignees

• Acronis Int Gmbh

Inventors

• Ulasen Sergey
• Snorkin Alexander
• Adaschik Andrey
• Shapiro Artem
• Shandyba Vasyl
• Bell Serg
• Protasov Stanislav

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/30228

  Playing field · CPC title