Method for determining correct scanning distance using augmented reality and machine learning models

Patent metadata

Field	Value
Publication number	US-2020125871-A1
Application number	US-201916694609-A
Country	US
Kind code	A1
Filing date	Nov 25, 2019
Priority date	Oct 23, 2018
Publication date	Apr 23, 2020
Grant date	—

Abstract

A smart device is provided with an application program for displaying a video feed received from the smart device's camera. The application can determine the coordinates for an intersection point, which is a point on the ground where the smart device is pointing at. The application can display a target on the visual representation of the intersection point. Based on whether the smart device is at an appropriate distance from the intersection point, the user interface can superimpose an indicator on the video feed received from the camera. This can inform the user whether the smart device is at an optimal scan distance from the intersection point (or an object) so that the object can be identified by a machine learning model.

First claim

1 - 20 . (canceled)

21 . A method comprising:
displaying a video feed and an indicator over the video feed in a user interface of a device, wherein the indicator corresponds to a visual representation of an intersection point; determining a distance between the device and the intersection point; displaying an attribute for the indicator in the user interface, wherein the attribute is determined based on the distance between the device and the intersection point; receiving a command from a user of the device to detect an object in a frame of the video feed; identifying, using an object recognition model, the object shown in the video feed; receiving information about the object shown in the video feed; and displaying the information in the user interface of the device.

22 . The method of claim 21 , wherein the intersection point is a point on a ground plane where a perpendicular hypothetical line coming out of the device would hit the ground plane.

23 . The method of claim 22 , wherein the indicator is displayed over the visual representation of the intersection point.

24 . The method of claim 22 , wherein the intersection point corresponds to a centerpoint of a field of view of a camera of the device.

25 . The method of claim 22 , wherein the device is configured to display the indicator coplanar with a visual representation of the ground plane.

26 . The method of claim 21 , wherein the indicator has at least one circle.

27 . The method of claim 26 , wherein the attribute of the indicator is a color or size of the at least one circle.

28 . The method of claim 21 , wherein the user interface is configured to change the attribute for the indicator when the distance between the device and the intersection point changes.

29 . The method of claim 21 , further comprising:
comparing the distance between the device and the intersection point with a threshold distance; and displaying the attribute for the indicator based on the comparison.

30 . The method of claim 29 , wherein:
displaying a first attribute as the attribute for the indicator if the distance between the device and the intersection point exceeds the threshold distance; and displaying a second attribute as the attribute for the indicator if the distance between the device and the intersection point does not exceed the threshold distance.

31 . The method of claim 29 , wherein:
the threshold distance is predetermined, or threshold distance is calculated by determining a probability value for identification of the object shown in the frame.

32 . The method of claim 29 , wherein the threshold distance is calculated using the following steps:
receiving at least two frames from a camera of the device, wherein each frame is associated with a frame distance which is the distance between device and the intersection point at the time of capturing the frame; providing the at least two frames to a machine learning model; determining a probability value for identification of the object in each of the at least two frames; comparing the probability values to a threshold probability; and if at least one of the probability values exceeds the threshold probability, setting the threshold distance as the frame distance of the frame associated with the probability value that exceeds the threshold probability.

33 . The method of claim 32 , further comprising:
if at least one of the probability values exceeds the threshold probability, displaying a third attribute to inform the user that the user is within an appropriate distance from the object; and if the probability values do not exceed the threshold probability, displaying a fourth attribute to inform the user that the user needs to move either closer to the object or away from the object.

34 . A non-transitory computer-accessible medium having stored thereon computer-executable instructions for an application for identifying objects executable on a device, wherein, when a computing hardware arrangement executes the instructions, the computing arrangement is configured to perform procedures comprising:
displaying a video feed and an indicator over the video feed in a user interface of the application, wherein the indicator corresponds to a visual representation of an intersection point; determining a distance between the device and the intersection point; comparing the distance between the device and the intersection point with a threshold distance; and displaying an attribute for the indicator in the user interface, wherein the attribute is determined based on the comparison.

35 . The non-transitory computer-accessible medium of claim 34 , wherein the threshold distance is calculated using the following steps:
receiving at least two frames from a camera of the device, wherein each frame is associated with a frame distance which is the distance between device and the intersection point at the time of capturing the frame; providing the at least two frames to a machine learning model; determining a probability value for identification of an object in each of the at least two frames; comparing the probability values to a threshold probability; and if at least one of the probability values exceeds the threshold probability, setting the threshold distance as the frame distance of the frame associated with the probability value that exceeds the threshold probability.

36 . The non-transitory computer-accessible medium of claim 35 , further comprising:
if at least one of the probability values exceeds the threshold probability, displaying a third attribute to inform the user that the user is within an appropriate distance from the object; and if the probability values do not exceed the threshold probability, displaying a fourth attribute to inform the user that the user needs to move either closer to the object or away from the object.

37 . The non-transitory computer-accessible medium of claim 34 , further comprising:
capturing a frame of the video feed when a condition is met; conducting a search query for a product shown in the frame; and displaying a listing including the product in the user interface of the application.

38 . The non-transitory computer-accessible medium of claim 37 , wherein the condition is met when the device is within an optimal scan distance of the product.

39 . The non-transitory computer-accessible medium of claim 38 , wherein the optimal scan distance is a predetermined distance, a range of distance learned by a model over time, or a threshold percentage of a field of view of a camera being occupied by the product.

40 . The non-transitory computer-accessible medium of claim 34 , further comprising identifying, using an object recognition model, a product shown in the video feed.

Assignees

• Capital One Services Llc

Inventors

• Duckworth Staevan
• Martinez Daniel
• Hardin William
• Castaneda Jonathan
• Lee Susie

Classifications

• G06T2207/20081

  Training; Learning · CPC title

• G01B11/026

  by measuring distance between sensor and object (G01B11/0608 takes precedence) · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• G06F16/9535

  Search customisation based on user profiles and personalisation · CPC title

• G01B11/03

  by measuring coordinates of points · CPC title