Enhanced sensor operation

Patent metadata

Field	Value
Publication number	US-11930302-B2
Application number	US-202117171352-A
Country	US
Kind code	B2
Filing date	Feb 9, 2021
Priority date	Feb 9, 2021
Publication date	Mar 12, 2024
Grant date	Mar 12, 2024

Abstract

A two-dimensional image of a vehicle occupant in a vehicle is collected. The collected two-dimensional image is input to a machine learning program trained to output one or more reference points of the vehicle occupant, each reference point being a landmark of the vehicle occupant. One or more reference points of the vehicle occupant in the two-dimensional image is output from the machine learning program. A location of the vehicle occupant in an interior of the vehicle is determined based on the one or more reference points. A vehicle component is actuated based on the determined location. For each of the one or more reference points, a similarity measure is determined between the reference point and a three-dimensional reference point, the similarity measure based on a distance between the reference point and the three-dimensional reference point.

First claim

The invention claimed is:

1. A system, comprising:
a three-dimensional image sensor;
a two-dimensional image sensor; and
a computer communicatively coupled to the three-dimensional image sensor and the two-dimensional image sensor, the computer including a processor and a memory, the memory storing instructions executable by the processor to:
receive a two-dimensional image of an object from the two-dimensional image sensor;
output, from a machine learning program trained to identify one or more reference points in an input two-dimensional image, a reference point of the object in the two-dimensional image;
when a similarity measure based on a distance between the reference point and a three-dimensional reference point is below a threshold, collect three-dimensional image data of the object from the three-dimensional image sensor;
output a new three-dimensional reference point based on the collected three-dimensional image data; and
deactivate the three-dimensional image sensor upon outputting the new three-dimensional reference point.

2. The system of claim 1 , wherein the three-dimensional image sensor is one of a stereo image sensor or a time-of-flight image sensor.

3. The system of claim 1 , wherein the instructions further include instructions to identify the three-dimensional reference point from a reference three-dimensional image captured by the three-dimensional image sensor before the two-dimensional image sensor captured the two-dimensional image.

4. The system of claim 1 , wherein the image includes an occupant, and the reference point in the two-dimensional image is associated to a joint of the occupant.

5. The system of claim 1 , wherein the reference point is an object keypoint, and the similarity measure is an object keypoint similarity measure.

6. The system of claim 1 , wherein the distance is a Euclidean distance between the reference point and the three-dimensional reference point.

7. The system of claim 1 , wherein the instructions further include instructions to actuate a vehicle component based on the output reference point.

8. A system, comprising a computer including a processor and a memory, the memory storing instructions executable by the processor to:
collect a two-dimensional image of a vehicle occupant in a vehicle;
input the collected two-dimensional image to a machine learning program trained to output one or more reference points of the vehicle occupant, each reference point being a landmark of the vehicle occupant;
output, from the machine learning program, one or more reference points of the vehicle occupant in the two-dimensional image;
determine a location of the vehicle occupant in an interior of the vehicle based on the one or more reference points;
actuate a vehicle component based on the determined location;
for each of the one or more reference points, determine a similarity measure between the reference point and a three-dimensional reference point, the similarity measure based on a distance between the reference point and the three-dimensional reference point;
actuate a three-dimensional image sensor to collect three-dimensional image data; and
deactivate the three-dimensional image sensor upon generating a new three-dimensional reference point based on the three-dimensional image data.

9. The system of claim 8 , wherein the three-dimensional image sensor is one of a stereo image sensor or a time-of-flight image sensor.

10. The system of claim 8 , wherein the instructions further include instructions to identify a plurality of reference points in the two-dimensional image and to identify an action performed by the vehicle occupant based on the plurality of reference points.

11. The system of claim 10 , wherein the instructions further include instructions to actuate one or more vehicle components based on the identified action.

12. The system of claim 8 , wherein the instructions further include instructions to identify the three-dimensional reference point from a previously collected three-dimensional image.

13. The system of claim 8 , wherein at least one of the one or more reference points in the two-dimensional image is associated to a joint of the vehicle occupant.

14. The system of claim 8 , wherein each reference point is an object keypoint, and the similarity measure is an object keypoint similarity measure.

15. The system of claim 8 , wherein the instructions further include instructions to actuate a safety device based on the location of the vehicle occupant in the vehicle.

16. The system of claim 8 , wherein the instructions further include instructions to rotate an air vent toward the vehicle occupant based on the location of the vehicle occupant in the vehicle.

17. A system, comprising:
a three-dimensional image sensor;
a two-dimensional image sensor; and
a computer communicatively coupled to the three-dimensional image sensor and the two-dimensional image sensor, the computer including a processor and a memory, the memory storing instructions executable by the processor to:
receive a two-dimensional image of an object from the two-dimensional image sensor;
output, from a machine learning program trained to identify one or more reference points in an input two-dimensional image, a reference point of the object in the two-dimensional image;
when a similarity measure based on a distance between the reference point and a three-dimensional reference point is below a threshold, collect three-dimensional image data of the object from the three-dimensional image sensor;
output a new three-dimensional reference point based on the collected three-dimensional image data; and
reactivate the three-dimensional image sensor when the similarity measure is below the threshold.

18. The system of claim 17 , wherein the three-dimensional image sensor is one of a stereo image sensor or a time-of-flight image sensor.

19. The system of claim 17 , wherein the image includes an occupant, and the reference point in the two-dimensional image is associated to a joint of the occupant.

20. The system of claim 17 , wherein the reference point is an object keypoint, and the similarity measure is an object keypoint similarity measure.

21. The system of claim 17 , wherein the instructions further include instructions to actuate a vehicle component based on the output reference point.

22. A system, comprising a computer including a processor and a memory, the memory storing instructions executable by the processor to:
collect a two-dimensional image of a vehicle occupant in a vehicle;
input the collected two-dimensional image to a machine learning program trained to output one or more reference points of the vehicle occupant, each reference point being a landmark of the vehicle occupant;
output, from the machine learning program, one or more reference points of the vehicle occupant in the two-dimensional image;
determine a location of the vehicle occupant in an interior of the vehicle based on the one or more reference points;
actuate a vehicle component based on the determined location;
for each of the one or more reference points, determine a similarity measure between the reference point and a three-dimensional reference point, the similarity measure based on a distance between the reference point and the three-dimensional reference point; and
reactivate a three-dimensional image sensor when a similarity measure between a new three-dimensional reference point and one of the one or more reference points of the vehi

Assignees

• Ford Global Tech Llc
• Univ Michigan State

Inventors

• Parenti Robert
• Siddiqui Adil
• Ghannam Mahmoud Yousef
• Boddeti Yasodekshna

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• H04N7/188 Primary

  Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position (signal generation from motion picture films H04N5/253) · CPC title

• B60H1/00742

  by detection of the vehicle occupants' presence; by detection of conditions relating to the body of occupants, e.g. using radiant heat detectors · CPC title

• B60H1/00871

  Air directing means, e.g. blades in an air outlet (construction of nozzles or air diffusers B60H1/34) · CPC title