Method for determining correct scanning distance using augmented reality and machine learning models

Patent metadata

Field	Value
Publication number	US-10380440-B1
Application number	US-201816168633-A
Country	US
Kind code	B1
Filing date	Oct 23, 2018
Priority date	Oct 23, 2018
Publication date	Aug 13, 2019
Grant date	Aug 13, 2019

Abstract

A smart device is provided with an application program for displaying a video feed received from the smart device's camera. The application can determine the coordinates for an intersection point, which is a point on the ground where the smart device is pointing at. The application can display a target on the visual representation of the intersection point. Based on whether the smart device is at an appropriate distance from the intersection point, the user interface can superimpose an indicator on the video feed received from the camera. This can inform the user whether the smart device is at an optimal scan distance from the intersection point (or an object) so that the object can be identified by a machine learning model.

First claim

The invention claimed is:

1. A method comprising:
displaying a user interface on a smart device, wherein the user interface includes a page for displaying a video feed received from a camera of the smart device;
detecting a ground plane in the video feed;
displaying a target over a point in the video feed, wherein the target corresponds to an intersection point where the plane at which the smart device is directed intersects with the ground plane;
determining a distance between the smart device and the intersection point;
comparing the distance with a threshold distance value;
displaying an indicator in the user interface, wherein the indicator reflects the comparison of the distance with the threshold distance value;
receiving a command from a user of the smart device to capture a photo;
providing the photo to a machine learning model to identify information about at least one object in the photo; and
displaying the information about the at least one object in the user interface of the smart device.

2. The method of claim 1 , wherein the detecting the ground plane includes feature extraction through visual processing.

3. The method of claim 1 , wherein the smart device is configured to determine coordinates of the smart device and coordinates of the intersection point in a 3D space, the distance can be calculated based on the coordinates of the smart device and the intersection point.

4. The method of claim 1 , wherein the threshold distance value is a predetermined number calculated based on an average size of a typical object to be identified by the smart device.

5. The method of claim 1 , wherein the threshold distance value is calculated by determining a probability value for identification of the at least one object.

6. The method of claim 1 , wherein the threshold distance value is calculated using the following steps:
receiving at least two photos from the camera of the smart device;
providing the photos to the machine learning model;
determining a probability value for identification of the at least one object in each photo;
comparing the probability values to a threshold probability value; and
if at least one of the probability values exceeds the threshold probability value, setting the threshold probability value as the distance between the smart device and the intersection point associated with the photo including the probability value that exceeds the threshold probability value.

7. The method of claim 6 , further comprising:
if at least one of the probability values exceeds the threshold probability value, displaying an indicator to inform the user that the user is within an appropriate distance from the at least one object; and
if the probability values do not exceed the threshold probability value, display an indicator to inform the user that the user needs to move either closer to the at least one object or away from the at least one object.

8. The method of claim 1 , further comprising:
conducting an online search for objects with the same information as the information of the at least one object; and
displaying results of the online search in the user interface of the smart device, wherein the results are filtered by a distance to a current location of the smart device and a piece of information obtained from a social media application of the smart device.

9. The method of claim 1 , wherein the at least one object is a vehicle.

10. The method of claim 9 , wherein the details of the at least one object includes a year, make and model of the vehicle.

11. The method of claim 10 , wherein the machine learning model was trained using photos of vehicles.

12. The method of claim 1 , further comprising:
taking a photo of the video feed;
determining an orientation of the at least one object;
comparing the orientation of the at least one object to a predetermined orientation; and
displaying a second indicator in the user interface based on the comparison of the orientation of the at least one object to the predetermined orientation.

13. The method of claim 12 , wherein the at least one object is a vehicle and the predetermined orientation is a front corner of the vehicle.

14. The method of claim 1 , wherein the indicator is a reticle including at least one ellipse, wherein the reticle contains the target.

15. The method of claim 14 , wherein in response to the smart device moving away from the intersection point, the user interface displays more ellipses.

16. The method of claim 14 , wherein in response to the smart device approaching the intersection point, the user interface displays fewer ellipses.

17. The method of claim 1 , wherein the indicator pulses if the distance between the smart device and the intersection point is more or less than the threshold distance value.

18. The method of claim 1 , wherein the indicator changes color based on the distance of the smart device from the intersection point.

19. The method of claim 1 , wherein a shape of the indicator changes based on the distance of the smart device from the intersection point.

Assignees

• Capital One Services Llc

Inventors

• Duckworth Staevan
• Martinez Daniel
• Hardin William
• Castaneda Jonathan
• Lee Susie

Classifications

• G06V20/20

  in augmented reality scenes · CPC title

• G06V20/64

  Three-dimensional [3D] objects · CPC title

• G06V10/17

  using hand-held instruments · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V20/584

  of vehicle lights or traffic lights · CPC title