Facial activity detection for virtual reality systems and methods

Patent metadata

Field	Value
Publication number	US-12333640-B2
Application number	US-202318092728-A
Country	US
Kind code	B2
Filing date	Jan 3, 2023
Priority date	Jan 4, 2022
Publication date	Jun 17, 2025
Grant date	Jun 17, 2025

Abstract

In an embodiment, a virtual reality ride system includes a display to present virtual reality image content to a first rider, an audio sensor to capture audio data associated with a second rider, and an image sensor to capture image data associated with the second rider. The virtual reality ride system also includes at least one processor communicatively coupled to the display and configured to (i) receive the audio data, the image data, or both, (ii) generate a virtual avatar corresponding to the second rider, wherein the virtual avatar includes a set of facial features, (iii) update the set of facial features based on the audio data, the image data, or both, and (iv) instruct the display to present the virtual reality image content including the virtual avatar and the updated set of facial features.

First claim

The invention claimed is:

1. A virtual reality ride system, comprising:
a first virtual reality device comprising:
a first display configured to present virtual reality image content to a first rider;
an audio sensor configured to capture audio data associated with the first rider;
an image sensor configured to capture image data associated with the first rider; and
at least one processor, the at least one processor configured to:
receive the audio data, the image data, or both;
generate a virtual avatar corresponding to the first rider, wherein the virtual avatar comprises a set of facial features;
animate the set of facial features based on the audio data, the image data, or both; and
transmit the virtual avatar and the animated set of facial features to a second virtual reality device; and

the second virtual reality device comprising a second display configured to present the virtual avatar and the animated set of facial features to a second rider.

2. The virtual reality ride system of claim 1 , wherein the at least one processor is configured to:
generate a facial gesture based on the image data as part of the animated set of facial features.

3. The virtual reality ride system of claim 1 , wherein the at least one processor is configured to:
determine a sequence of visemes based on the audio data; and
animate the set of facial features based on the visemes.

4. The virtual reality ride system of claim 1 , comprising a ride vehicle sensor configured to capture vehicle sensor data indicative of movement characteristics of a ride vehicle.

5. The virtual reality ride system of claim 4 , wherein the at least one processor is configured to:
receive the vehicle sensor data;
determine a predicted movement profile associated with the ride vehicle; and
alter the virtual avatar based on the predicted movement profile.

6. The virtual reality ride system of claim 5 , wherein altering the virtual avatar based on the predicted movement profile comprises altering the set of facial features based on the predicted movement profile.

7. The virtual reality ride system of claim 1 , comprising a rider sensor configured to capture sensor data indicative of a set of movement characteristics associated with the second rider.

8. The virtual reality ride system of claim 7 , wherein the set of movement characteristics comprises an orientation of the second rider, a position of the second rider, a speed of the second rider, an acceleration of the second rider, or any combination thereof.

9. The virtual reality ride system of claim 7 , wherein the at least one processor is configured to:
receive the sensor data; and
update the virtual avatar based on the sensor data.

10. The virtual reality ride system of claim 9 , wherein the at least one processor is configured to:
alter a pose of the virtual avatar based on the sensor data;
alter the set of facial features based on the sensor data; and
transmit the virtual avatar and the altered set of facial features to the second virtual reality device, wherein the second display is configured to present the virtual avatar and the altered set of facial features to the second rider.

11. The virtual reality ride system of claim 9 , wherein the at least one processor is configured to alter a pose of the virtual avatar based on the sensor data.

12. The virtual reality ride system of claim 11 , wherein the at least one processor is configured to transmit the virtual avatar after being altered into the pose to the second virtual reality device, wherein the second display is configured to present the virtual avatar in the pose.

13. The virtual reality ride system of claim 9 , wherein the at least one processor is configured to alter the set of facial features based on the sensor data.

14. The virtual reality ride system of claim 13 , wherein the at least one processor is configured to transmit the altered set of facial features to the second virtual reality device, wherein the second display is configured to present the altered set of facial features to the second rider.

15. A virtual reality device, comprising:
an audio sensor configured to capture audio data indicative of speech of a user;
an image sensor configured to capture image data indicative of facial characteristics of the user; and
at least one processor communicatively coupled to the audio sensor, and the image sensor, wherein the at least one processor is configured to:
determine a set of facial characteristics based on the image data;
determine a set of facial movements associated with the set of facial characteristics based on the audio data; and
transmit the set of facial characteristics and the set of facial movements to a second virtual reality device, the second virtual reality device configured to display virtual reality image content based on the set of facial characteristics and the set of facial movements.

16. The virtual reality device of claim 15 , comprising:
a display configured to display virtual reality image content to the user;
wherein the at least one processor is configured to:
receive, from the second virtual reality device, second audio data, second image data, or both;
generate a model of a second user based on the second image data;
animate the model based on the second audio data; and
instruct the display to present the virtual reality image content including the animated model.

17. The virtual reality device of claim 16 , comprising an audio playback device configured to playback the second audio data.

18. The virtual reality device of claim 16 , wherein the at least one processor is configured to:
receive user sensor data associated with the second user; and
animate the model based on the user sensor data.

19. The virtual reality device of claim 16 , wherein the at least one processor is configured to:
receive vehicle sensor data associated with a ride vehicle; and
animate the model based on the vehicle sensor data.

20. The virtual reality device of claim 16 , wherein the at least one processor is configured to:
determine text data associated with the second audio data;
determine a set of visemes associated with the text data; and
animate the model based on the set of visemes.

21. The virtual reality device of claim 16 , wherein the second audio data corresponds to speech of the second user.

22. The virtual reality device of claim 21 , wherein the at least one processor is configured to alter a set of facial features of the model based on the speech.

23. A method, comprising:
receiving audio data, image data, or both;
generating a virtual avatar based on the image data, wherein the virtual avatar comprises a set of facial features;
determining a set of facial characteristics associated with the image data;
comparing the set of facial characteristics with a set of facial gesture profiles, each facial gesture profile of the set of facial gesture profiles comprising a corresponding set of stored facial characteristics;
selecting, based on the comparison, a facial gesture profile of the set of facial gesture profiles;
animating the set of facial features based on the selected facial gesture profile, the audio data, or both; and
presenting virtual reality image content comprising the virtual avatar and the animated set of facial features.

24. The method of claim 23 , comprising:
receiving a set of vehicle sensor data indicative of a movement profile associated with

Assignees

• Universal City Studios Llc

Inventors

• Jordan Robert Michael
• Traynor Mark James
• Goergen Patrick John

Classifications

• G06V40/171

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title

• A63G31/16 Primary

  creating illusions of travel · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T13/205

  driven by audio data · CPC title