Systems and methods for determining environment dimensions based on landmark detection

Patent metadata

Field	Value
Publication number	US-12423858-B2
Application number	US-202217681785-A
Country	US
Kind code	B2
Filing date	Feb 27, 2022
Priority date	Mar 12, 2021
Publication date	Sep 23, 2025
Grant date	Sep 23, 2025

Abstract

Disclosed herein are systems and method for determining environment dimensions based on landmark detection, the method including: training, with a dataset including a plurality of images featuring an environment and labelled landmarks in the environment, a neural network to identify the labelled landmarks in an arbitrary image of the environment; receiving an input image depicting the environment; generating an input tensor based on the received input image; inputting the input tensor into the neural network, wherein the neural network is configured to generate an output tensor including a position of each identified landmark and a visibility score associated with each position; calculating a homography matrix between each position in the output tensor along a camera plane and a corresponding position in an environment plane, based on a pre-built model of the environment; and outputting an image that visually connects each landmark along the environment plane based on the homography matrix.

First claim

The invention claimed is:

1. A method for determining environment dimensions based on landmark detection, the method comprising:
training, with a dataset comprising a plurality of images featuring an environment and labelled landmarks in the environment, a neural network to identify the labelled landmarks in an arbitrary image of the environment;
receiving an input image depicting the environment;
generating an input tensor based on the received input image;
inputting the input tensor into the neural network, wherein the neural network is configured to generate an output tensor comprising a position of each identified landmark and a visibility score associated with each position, wherein the visibility score is a probability of whether a respective landmark is shown in a frame of the input image, and wherein a respective visibility score of a respective identified landmark that is less than a predetermined threshold score is indicative of the respective identified landmark being outside a visible portion of the environment as captured in the frame, and wherein the neural network is configured to label relative positions outside of the input image for landmarks with visibility scores less than the predetermined threshold score;
calculating a homography matrix between each position in the output tensor along a camera plane and a corresponding position in an environment plane, based on a pre-built model of the environment and visibility scores in the output tensor; and
outputting an image that visually connects each landmark along the environment plane based on the homography matrix.

2. The method of claim 1 , further comprising:
identifying an object in the input image;
determining pixel coordinates of the object in the input image; and
applying the homography matrix to convert the pixel coordinates to physical coordinates of the object in the environment.

3. The method of claim 1 , wherein the neural network comprises:
a convolutional backbone with feature extraction layers, and
a regression head with fully connected layers.

4. The method of claim 1 , wherein the pre-built model of the environment is indicative of distances between each landmark in the environment.

5. The method of claim 1 , wherein the neural network is configured to determine a loss that is a linear combination of a mean absolute error for each identified landmark position and a cross entropy for point presence on the frame.

6. The method of claim 5 , wherein the neural network optimizes the loss using stochastic gradient descent.

7. The method of claim 1 , wherein the input image is a video frame of a livestream, and wherein the neural network determines environment dimensions in real-time.

8. The method of claim 1 , wherein the environment is a sports field and the labelled landmarks are locations on the sports field.

9. A system for determining environment dimensions based on landmark detection, the system comprising:
a hardware processor configured to:
train, with a dataset comprising a plurality of images featuring an environment and labelled landmarks in the environment, a neural network to identify the labelled landmarks in an arbitrary image of the environment;
receive an input image depicting the environment;
generate an input tensor based on the received input image;
input the input tensor into the neural network, wherein the neural network is configured to generate an output tensor comprising a position of each identified landmark and a visibility score associated with each position, wherein the visibility score is a probability of whether a respective landmark is shown in a frame of the input image, and wherein a respective visibility score of a respective identified landmark that is less than a predetermined threshold score is indicative of the respective identified landmark being outside a visible portion of the environment as captured in the frame, and wherein the neural network is configured to label relative positions outside of the input image for landmarks with visibility scores less than the predetermined threshold score;
calculate a homography matrix between each position in the output tensor along a camera plane and a corresponding position in an environment plane, based on a pre-built model of the environment and visibility scores in the output tensor; and
output an image that visually connects each landmark along the environment plane based on the homography matrix.

10. The system of claim 9 , wherein the hardware processor is further configured to:
identify an object in the input image;
determine pixel coordinates of the object in the input image; and
apply the homography matrix to convert the pixel coordinates to physical coordinates of the object in the environment.

11. The system of claim 9 , wherein the neural network comprises:
a convolutional backbone with feature extraction layers, and
a regression head with fully connected layers.

12. The system of claim 9 , wherein the pre-built model of the environment is indicative of distances between each landmark in the environment.

13. The system of claim 9 , wherein the neural network is configured to determine a loss that is a linear combination of a mean absolute error for each identified landmark position and a cross entropy for point presence on the frame.

14. The system of claim 13 , wherein the neural network optimizes the loss using stochastic gradient descent.

15. The system of claim 9 , wherein the input image is a video frame of a livestream, and wherein the neural network determines environment dimensions in real-time.

16. The system of claim 9 , wherein the environment is a sports field and the labelled landmarks are locations on the sports field.

17. A non-transitory computer readable medium storing thereon computer executable instructions for determining environment dimensions based on landmark detection, including instructions for:
training, with a dataset comprising a plurality of images featuring an environment and labelled landmarks in the environment, a neural network to identify the labelled landmarks in an arbitrary image of the environment;
receiving an input image depicting the environment;
generating an input tensor based on the received input image;
inputting the input tensor into the neural network, wherein the neural network is configured to generate an output tensor comprising a position of each identified landmark and a visibility score associated with each position, wherein the visibility score is a probability of whether a respective landmark is shown in a frame of the input image, and wherein a respective visibility score of a respective identified landmark that is less than a predetermined threshold score is indicative of the respective identified landmark being outside a visible portion of the environment as captured in the frame, and wherein the neural network is configured to label relative positions outside of the input image for landmarks with visibility scores less than the predetermined threshold score;
calculating a homography matrix between each position in the output tensor along a camera plane and a corresponding position in an environment plane, based on a pre-built model of the environment and visibility scores in the output tensor; and
outputting an image that visually connects each landmark along the environment plane based on the homography matrix.

18. The non-transitory computer readable medium of claim 17 , further comprising instructions for:
identifying an object in the input image;
determining pixel coordinates of the object

Assignees

• Acronis Int Gmbh

Inventors

• Ulasen Sergey
• Shandyba Vasyl
• Snorkin Alexander
• Shapiro Artem
• Adaschik Andrey
• Beloussov Serguei
• Protasov Stanislav

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06T2207/30228

  Playing field · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title