Systems and methods for managing occupant interaction using depth information

Patent metadata

Field	Value
Publication number	US-12450941-B2
Application number	US-202318122317-A
Country	US
Kind code	B2
Filing date	Mar 16, 2023
Priority date	Mar 16, 2023
Publication date	Oct 21, 2025
Grant date	Oct 21, 2025

Abstract

A method for managing occupant interaction with components of a vehicle includes generating, via a time-of-flight sensor, a point cloud representing a compartment of the vehicle, the point cloud including three-dimensional positional information about the compartment. The method further includes identifying at least one body segment of a user based on the point cloud. The method further includes determining, via processing circuitry in communication with the time-of-flight sensor, a position of at least one feature of the at least one body segment. The method further includes calculating an orientation of the at least one body segment based on the at least one feature. The method further includes estimating an intended task of the user based on the orientation of the at least one body segment. The method further includes generating, via the processing circuitry, a response signal in response to estimation of the intended task.

First claim

What is claimed is:

1. A method for managing occupant interaction with components of a vehicle, the method comprising:
generating, via a time-of-flight sensor, a point cloud representing a compartment of the vehicle, the point cloud including three-dimensional positional information about the compartment;
identifying at least one body segment of a user based on the point cloud;
determining, via processing circuitry in communication with the time-of-flight sensor, a position of at least one feature of the at least one body segment;
calculating an orientation of the at least one body segment based on the at least one feature;
estimating an intended task of the user based on the orientation of the at least one body segment; and
generating, via the processing circuitry, a response signal in response to estimation of the intended task.

2. The method of claim 1 , wherein the at least one body segment includes a head, the at least one feature is a facial feature, the orientation is a tilt of the head, and the intended task is a gaze direction of the user, the method further comprising:
calculating the tilt of the head based on the facial feature; and
estimating the gaze direction based on the tilt of the head.

3. The method of claim 2 , further comprising:
identifying, based on the point cloud, eyes of the user;
calculating a frontal plane based on the eyes of the user; and
determining a vector normal to the frontal plane, wherein the vector corresponds to the estimation of the gaze direction.

4. The method of claim 3 , further comprising:
communicating, via the processing circuitry, a signal to align an adjustable component of the vehicle based on the estimation of the gaze direction.

5. The method of claim 4 , wherein the adjustable component is configured to adjust a visual feedback element.

6. The method of claim 5 , wherein the visual feedback element is a display, and wherein the adjustable component is configured to adjust a visual presentation of the display to align the display with the estimation of the gaze direction.

7. The method of claim 5 , wherein the visual feedback element is a mirror, and wherein the adjustable component is configured to adjust an angle of the mirror to align the mirror with a target viewing angle.

8. The method of claim 4 , wherein the adjustable component is configured to adjust a position of the user.

9. The method of claim 1 , wherein the at least one body segment includes an appendage, the at least one feature is a longitudinal extension, the orientation is an indication direction, and the intended task is an input selection of the user, the method further comprising:
calculating the indication direction of the appendage based on the longitudinal extension; and
estimating the input selection based on the indication direction.

10. The method of claim 9 , further comprising:
predicting a system function of the vehicle to be adjusted based on the estimation of the input selection; and
presenting an option to activate the system function.

11. The method of claim 10 , further comprising:
identifying, based on the point cloud, a gesture by the user; and
selecting, via the processing circuitry, the option based on the gesture.

12. The method of claim 11 , wherein the input selection is manipulation of a mechanical switch.

13. The method of claim 11 , wherein the input selection is a digital indicator presented at a user interface for the vehicle.

14. The method of claim 13 , further comprising:
communicating a signal to adjust a size of the digital indicator at the user interface based on the identification of the gesture.

15. A system for managing occupant interaction with components of a vehicle, the system comprising:
a time-of-flight sensor configured to generate a point cloud representing a compartment of the vehicle, the point cloud including three-dimensional positional information about the compartment; and
processing circuitry in communication with the time-of-flight sensor configured to:
identify at least one body segment of a user based on the point cloud;
determine a position of at least one feature of the at least one body segment;
calculate an orientation of the at least one body segment based on the at least one feature;
estimate an intended task of the user based on the orientation of the at least one body segment; and
generate a response signal in response to the estimation of the intended task.

16. The system of claim 15 , wherein the at least one body segment includes a head, the at least one feature is a facial feature, the orientation is a tilt of the head, and the intended task is a gaze direction of the user, the processing circuitry further configured to:
calculate the tilt of the head based on the facial feature; and
estimate the gaze direction based on the tilt of the head.

17. The system of claim 16 , further comprising:
an adjustable component of the vehicle in communication with the processing circuitry, the processing circuitry further configured to communicate a signal to align the adjustable component.

18. The system of claim 15 , wherein the at least one body segment includes an appendage, the at least one feature is a longitudinal extension, the orientation is an indication direction, and the intended task is an input selection of the user, the processing circuitry further configured to:
calculate the indication direction of the appendage based on the longitudinal extension; and
estimate the input selection based on the indication direction.

19. The system of claim 18 , wherein the processing circuitry is further configured to:
predict a system function of the vehicle to be adjusted based on the estimation of the input selection;
present an option to activate the system function;
identify, based on the point cloud, a gesture by the user; and
select the option based on the gesture.

20. A system for managing occupant interaction with components of a vehicle, the system comprising:
a time-of-flight sensor configured to generate a point cloud representing a compartment of the vehicle, the point cloud including three-dimensional positional information about the compartment;
a visual feedback element;
an adjustable component configured to adjust the visual feedback element; and
processing circuitry in communication with the time-of-flight sensor and the adjustable component, the processing circuitry configured to:
identify a head of a user based on the point cloud;
determine a position of facial features of the head;
calculate a tilt of the head based on the facial features;
estimate a gaze direction of the user based on the tilt; and
communicate a signal to align the adjustable component based on the estimation of the gaze direction.

Assignees

• Ford Global Tech Llc

Inventors

• Gorski Ryan Joseph
• Van Wiemeersch John Robert
• Ghannam Mahmoud Yousef

Classifications

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06T2207/30201

  Face · CPC title

• G01S17/894 Primary

  Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title