Augmented reality wayfinding in rideshare applications

Patent metadata

Field	Value
Publication number	US-11900815-B2
Application number	US-201916589068-A
Country	US
Kind code	B2
Filing date	Sep 30, 2019
Priority date	Sep 30, 2019
Publication date	Feb 13, 2024
Grant date	Feb 13, 2024

Abstract

Technologies for providing augmented reality wayfinding experiences in ridesharing applications are provided. In some examples, a method for providing augmented reality wayfinding experiences can include determining a first location of an autonomous vehicle (AV) relative to a second location of a client device associated with a user that requested a ride from the AV; based on the first location of the AV relative to the second location of the client device, determining a direction from the second location of the client device to the first location of the AV; presenting, at the client device, a feed from a camera sensor associated with the client device, the feed including a local scene captured by the camera sensor; and presenting a virtual content overlay on the feed, the virtual content overlay including an indication of the direction from the second location of the client device to the first location of the AV.

First claim

What is claimed is:

1. A method comprising:
determining a first location of an autonomous vehicle (AV) relative to a second location of a client device associated with a user that requested a ride from the AV;
based on the first location of the AV relative to the second location of the client device, determining a direction from the second location of the client device to the first location of the AV;
presenting, at the client device, an audio cue informing the user that a user action can be performed to activate an augmented reality (AR) wayfinder feature wherein the user action comprises holding the client device vertically;
in response to performance of the user action of holding the client device vertically, presenting, at the client device, a real-time feed from a camera sensor associated with the client device, the real-time feed comprising a local scene captured by the camera sensor and the real-time feed being from a viewpoint of the camera sensor;
presenting a virtual content overlay on the real-time feed presented at the client device, the virtual content overlay comprising a first indication of the direction from the second location of the client device to the first location of the AV;
presenting, at the client device, one or more selectable interface controls for remotely triggering activating a horn on the AV and unlocking the doors on the AV;
determining that the AV is outside of a field-of-view (FOV) of the camera sensor;
in response to determining that the AV is outside of the FOV of the camera sensor, presenting, at the client device, a virtual pointer instructing the user to point the camera sensor in a different direction determined to place the AV within the FOV of the camera sensor, wherein a size of the virtual pointer is based on a distance between the AV and the client device, and continuing, after a handoff between the AV and the client device, to provide, on the client device, wayfinding information from a drop-off location associated with the ride to a final destination.

2. The method of claim 1 , wherein the virtual content overlay comprises at least one of, a digital message, and a virtual object, and wherein the first indication of the direction from the second location of the client device to the first location of the AV comprises at least one of a path from the second location of the client device to the first location of the AV, a direction for walking from the second location of the client device to the first location of the AV, a description of the first location of the AV, a distance between the second location of the client device and the first location of the AV, and wayfinding instructions.

3. The method of claim 1 , further comprising:
presenting, at the client device, one or more selectable interface controls for remotely triggering an action at the AV, the action comprising flashing a light on the AV.

4. The method of claim 1 , further comprising:
determining a particular direction in which the camera sensor is pointed; and
presenting, at the client device, an instruction to point the camera sensor in a different direction, the instruction comprising one or more indicators identifying the different direction to point the camera sensor, the different direction being towards at least one of the different location of the AV and an estimated future location of the AV.

5. The method of claim 4 , further comprising:
determining that the AV is outside of the FOV of the camera sensor based on the determining the particular direction in which the camera sensor is pointed; and
determining that pointing the camera sensor in the different direction places the AV within the FOV of the camera sensor.

6. The method of claim 1 , wherein the first indication comprises a first visual indication, the method further comprising, based on a determination that the AV has moved from the first location to a different location, modifying at least one of a first size and a first shape of a second visual indication relative to at least one of a second size and a second shape of the first visual indication, the at least one of the first size and the first shape of the second visual indication being modified based on a change in a relative distance between the client device and the AV resulting from the AV moving from the first location to the different location, the second visual indication comprising an indication of an updated direction from the second location of the client device to the different location of the AV.

7. The method of claim 1 , further comprising:
presenting, via the client device, a digital map identifying the first location of the AV and the second location of the client device, the digital map displaying at least one of a distance between the AV and the client device and a route from a first current location of the AV to a second current location of the client device.

8. The method of claim 1 , further comprising:
detecting that the client device has moved from the second location; and
based on a determination that the AV is within a field-of-view of the camera sensor from a current location of the client device, presenting a modified virtual content overlay on a current real-time feed from the camera sensor presented at the client device, the modified virtual content overlay comprising an AV indicator identifying the AV within the current real-time feed presented at the client device.

9. The method of claim 1 , further comprising:
detecting that the AV has moved from the first location to a different location;
determining, based on the different location of the AV relative to the second location of the client device, an updated direction from the different location of the AV to the second location of the client device; and
based on a determination that the AV has moved from the first location to the different location, presenting, at the client device, an updated virtual content overlay on an updated real-time feed from the camera sensor, the updated virtual content overlay comprising a second indication of the updated direction from the different location of the AV to the second location of the client device, wherein the updated real-time feed comprises an updated local scene captured by the camera sensor.

10. The method of claim 1 , further comprising:
determining that a view of the AV from a current location of the client device is obstructed by one or more objects; and
presenting one or more indicators identifying an AV location within a current feed from the camera sensor, the one or more indicators comprising at least one of a visual representation of the AV superimposed over a rendering of the one or more objects, a description of where the AV is located, and a visual indicator identifying where the AV is located.

11. The method of claim 1 , wherein presenting the cue is further in response to the AV arriving near the pick-up location.

12. The method of claim 1 , wherein the handoff between the AV and the client device occurs after the AV arrives at the drop-off location associated with the ride.

13. A system comprising:
one or more processors; and
memory comprising instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to:
determine a first location of an autonomous vehicle (AV) relative to a second location of a computing device, wherein the computing device is associated with a user that requested a ride from the AV;
based on the first location of the AV relative to the second location of the computing device, determine a direction from the second location of the computing device to the first location of the AV;
present, at the

Assignees

• Gm Cruise Holdings Llc

Inventors

• Leary Jessica
• Staff Jonathon
• Patel Pratikbhai
• Ryan Bradley
• Berlin Josh
• Anderson John Logan
• Magnoli Michael
• Shiao Stanley

Classifications

• G08G1/205 Primary

  Indicating the location of the monitored vehicles as destination, e.g. accidents, stolen, rental · CPC title

• B60W10/30

  including control of auxiliary equipment, e.g. air-conditioning compressors or oil pumps · CPC title

• G05D1/0038

  by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246) · CPC title

• G05D1/0248

  in combination with a laser (lasers per se H01S) · CPC title

• G08G1/202

  Dispatching vehicles on the basis of a location, e.g. taxi dispatching · CPC title