Virtual models for communications between user devices and external observers

Patent metadata

Field	Value
Publication number	US-12360588-B2
Application number	US-202318450640-A
Country	US
Kind code	B2
Filing date	Aug 16, 2023
Priority date	May 10, 2019
Publication date	Jul 15, 2025
Grant date	Jul 15, 2025

Abstract

Systems and methods for interactions between an autonomous vehicle and one or more external observers include virtual models of drivers the autonomous vehicle. The virtual models may be generated by the autonomous vehicle and displayed to one or more external observers, and in some cases using devices worn by the external observers. The virtual models may facilitate interactions between the external observers and the autonomous vehicle using gestures or other visual cues. The virtual models may be encrypted with characteristics of an external observer, such as the external observer's face image, iris, or other representative features. Multiple virtual models for multiple external observers may be simultaneously used for multiple communications while preventing interference due to possible overlap of the multiple virtual models.

First claim

What is claimed is:

1. A user device, comprising:
an image sensor;
a display visible to a first external observer;
a memory; and
a processor configured to:
obtain a first one or more images from the image sensor, the first one or more images including at least a portion of the first external observer relative to an exterior of the user device;
determine, based on one or more features in the first one or more images, that the first external observer is in a presence of an external environment of the user device;
output, to the display based on the determination that the first external observer is in the presence of the external environment of the user device, a first virtual model observable by the first external observer, the first virtual model depicting a user of the user device;
after the first virtual model is output to the display based on the determination that the first external observer is in the presence of the external environment of the user device, obtain a second one or more images from the image sensor, the second one or more images not including at least a portion of the first external observer;
determine, based on the second one or more images, that the first external observer is not in the presence of the external environment of the user device; and
disable the output of the first virtual model based on the determination that the first external observer is not in the presence of the external environment of the user device.

2. The user device of claim 1 , wherein the processor is further configured to:
identify an input associated with the first external observer; and
determine that the first external observer is in the presence of the external environment of the user device based on the input.

3. The user device of claim 2 , wherein the input includes one or more gestures from the first external observer.

4. The user device of claim 2 , wherein the input includes a voice input from the first external observer.

5. The user device of claim 1 , wherein, to determine that the first external observer is in the presence of the external environment of the user device, the processor is configured to:
track, based on the one or more features in the first one or more images, an eye gaze of the first external observer.

6. The user device of claim 5 , wherein the processor is configured to:
determine a field of view of the first external observer based on tracking the eye gaze; and
determine that the field of view includes at least a portion of the user device.

7. The user device of claim 1 , wherein the image sensor includes at least one of a camera or a depth sensor.

8. The user device of claim 1 , wherein the processor is further configured to:
determine that a second external observer is in the presence of the external environment of the user device; and
output, to the display based on the determination that the second external observer is in the presence of the external environment of the user device, the first virtual model observable by the second external observer.

9. The user device of claim 1 , wherein the processor is further configured to:
determine that a second external observer is in the presence of the external environment of the user device; and
output, to the display based on the determination that the second external observer is in the presence of the external environment of the user device, a second virtual model observable by the second external observer.

10. The user device of claim 1 , wherein the first virtual model is a visual depiction of a portion of the user of the user device.

11. The user device of claim 1 , wherein, to output the first virtual model, the processor is further configured to output audio associated with the first virtual model.

12. The user device of claim 1 , wherein the processor is configured to:
communicate with the first external observer using the first virtual model in response to a communication from the first external observer.

13. The user device of claim 12 , wherein the communication from the first external observer includes one or more gestures.

14. The user device of claim 1 , wherein the user device includes a head mounted display, a virtual reality device, an augmented reality device, or a vehicle.

15. A method of communicating with a first external observer, comprising:
obtaining, by a user device, a first one or more images including at least a portion of the first external observer relative to an exterior of the user device;
determining, based on one or more features in the first one or more images, that the first external observer is in a presence of an external environment of the user device;
displaying, based on the determination that the first external observer is in the presence of the external environment of the user device, a first virtual model observable by the first external observer, the first virtual model depicting a user of the user device;
after the first virtual model is displayed based on the determination that the first external observer is in the presence of the external environment of the user device, obtaining a second one or more images, the second one or more images not including at least a portion of the first external observer;
determining, based on the second one or more images, that the first external observer is not in the presence of the external environment of the user device; and
disabling the display of the first virtual model based on the determination that the first external observer is not in the presence of the external environment of the user device.

16. The method of claim 15 , further comprising:
identifying an input associated with the first external observer; and
determining that the first external observer is in the presence of the external environment of the user device based on the input.

17. The method of claim 16 , wherein the input includes one or more gestures from the first external observer.

18. The method of claim 16 , wherein the input includes a voice input from the first external observer.

19. The method of claim 15 , wherein determining that the first external observer is in the presence of the external environment of the user device comprises:
tracking, based on the one or more features in the first one or more images, an eye gaze of the first external observer.

20. The method of claim 19 , further comprising:
determining a field of view of the first external observer based on tracking the eye gaze; and
determining that the field of view includes at least a portion of the user device.

21. The method of claim 15 , wherein the first one or more images are obtained from an image sensor, and wherein the image sensor includes at least one of a camera or a depth sensor.

22. The method of claim 15 , further comprising:
determining that a second external observer is in the presence of the external environment of the user device; and
displaying, based on the determination that the second external observer is in the presence of the external environment of the user device, the first virtual model observable by the second external observer.

23. The method of claim 15 , further comprising:
determining that a second external observer is in the presence of the external environment of the user device; and
displaying, based on the determination that the second external observer is in the presence of the external environment of the user device, a second virtual model observable by the second exter

Assignees

• Qualcomm Inc

Inventors

• Banerjee Debdeep
• Kandhadai Ananthapadmanabhan Arasanipalai

Classifications

• B60Q1/543

  for indicating other states or conditions of the vehicle (indicating taxis' state of hire or fare G07B13/045) · CPC title

• B60Q1/507

  specific to autonomous vehicles · CPC title

• B60Q1/5037

  the display content changing automatically, e.g. depending on traffic situation · CPC title

• G06V10/40

  Extraction of image or video features · CPC title

• G06V40/18

  Eye characteristics, e.g. of the iris · CPC title