Systems and methods for resource-optimized mixed reality using user-centric adaptive object detection

Patent metadata

Field	Value
Publication number	US-12387489-B2
Application number	US-202418809564-A
Country	US
Kind code	B2
Filing date	Aug 20, 2024
Priority date	Oct 25, 2023
Publication date	Aug 12, 2025
Grant date	Aug 12, 2025

Abstract

System and method for reducing latency and bandwidth usage in reality devices include a reality device and one or more processors. The reality device includes a camera to operably capture a set of consequent frames of views external to a vehicle. The one or more processors are operable to monitor movements of one or more eyes of a user to obtain eye-tracking data using the reality device, identify an area of interest (AoI) in the frame based on the eye-tracking data, AoI history, or driving statistics of the user, crop the frame to obtain a reduced-size frame including the AoI in the frame, and transmit the reduced-size frame to an edge server for performing a task on behalf of the vehicle.

First claim

What is claimed is:

1. A system for reducing latency and bandwidth usage comprising:
a reality device comprising a camera to operably capture a frame of a view external to a vehicle; and
one or more processors operable to:
monitor movements of one or more eyes of a user to obtain eye-tracking data using the reality device;
identify an area of interest (AoI) in the frame based on the eye-tracking data, AoI history, or driving statistics of the user;
crop the frame to obtain a reduced-size frame including the AoI in the frame; and
transmit the reduced-size frame to an edge server for performing a task on behalf of the vehicle.

2. The system of claim 1 , wherein the movements of the one or more eyes comprise positions, angles, or pupil sizes of the one or more eyes of the user.

3. The system of claim 1 , wherein the reality device further comprises an eye-tracking sensor operable to track the movements of the one or more eyes of the user and generate the eye-tracking data.

4. The system of claim 3 , wherein the eye-tracking sensor comprises an infrared laser source operable to illuminate the one or more eyes of the user and an eye camera operable to capture the eye-tracking data.

5. The system of claim 1 , wherein the eye-tracking data comprise user focus points, gaze fixations of each corresponding focus point, saccades between gaze fixations, scan paths of gaze fixations and saccades, and pupil dilation.

6. The system of claim 5 , wherein the identification of the AoI in the frame is based on gaze-based interaction with virtual objects, wherein the gaze-based interaction comprises duration and frequency of gaze to the virtual objects.

7. The system of claim 1 , wherein identifying the AoI based on driving statistics of the user comprises:
collecting driving sensory data to determine driving events; and
correlating the movements of the one or more eyes with the driving events to identify the AoI in the frame.

8. The system of claim 1 , wherein the one or more processors are operable to further receive object detection data from the edge server, wherein the object detection data comprises box cords of detected objects in the frame, confidence of each corresponding box cord, and object information of each detected object.

9. The system of claim 8 , wherein the one or more processors are further operable to autonomously drive the vehicle based on the object detection data.

10. The system of claim 8 , wherein the one or more processors are further operable to superimpose the object detection data onto a real-world view.

11. The system of claim 10 , wherein the object detection data are superimposed onto a vision of the user or a current real-world frame.

12. A method for reducing latency and bandwidth usage comprising:
monitoring movements of one or more eyes of a user to obtain eye-tracking data using a reality device;
identifying an area of interest (AoI) in a frame of a view external to a vehicle based on the eye-tracking data, AoI history, or driving statistics of the user, wherein the frame is captured by the reality device;
cropping the frame to obtain a reduced-size frame including the AoI in the frame; and
transmitting the reduced-size frame to an edge server for performing a task on behalf of the vehicle.

13. The method of claim 12 , wherein the movements of the one or more eyes comprise positions, angles, or pupil sizes of the one or more eyes of the user.

14. The method of claim 12 , wherein the reality device comprises an eye-tracking sensor operable to track the movements of the one or more eyes of the user and generate the eye-tracking data, the eye-tracking sensor comprising an infrared laser source operable to illuminate the one or more eyes of the user and an eye camera operable to capture the eye-tracking data.

15. The method of claim 12 , wherein the eye-tracking data comprise user focus points, gaze fixations of each corresponding focus point, saccades between gaze fixations, scan paths of gaze fixations and saccades, and pupil dilation.

16. The method of claim 15 , wherein the identification of the AoI in the frame is based on gaze-based interaction with virtual objects, wherein the gaze-based interaction comprises duration and frequency of gaze to the virtual objects.

17. The method of claim 12 , wherein identifying the AoI based on driving statistics of the user comprises:
collecting driving sensory data to determine driving events; and
correlating the movements of the one or more eyes with the driving events to identify the AoI in the frame.

18. The method of claim 12 , wherein the method further comprises receiving object detection data from the edge server, wherein the object detection data comprises box cords of detected objects in the frame, confidence of each corresponding box cord, and object information of each detected object.

19. The method of claim 18 , wherein the method further comprises autonomously driving the vehicle based on the object detection data.

20. The method of claim 18 , wherein the method further comprises superimposing the object detection data onto a real-world view.

Assignees

• Toyota Eng & Mfg North America
• Toyota Motor Co Ltd

Inventors

• Doe Daniel
• Chen Dawei
• Han Kyungtae

Classifications

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V10/32 Primary

  Normalisation of the pattern dimensions · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06V10/235

  based on user input or interaction · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title