In-cabin hazard prevention and safety control system for autonomous machine applications

Patent metadata

Field	Value
Publication number	US-11485308-B2
Application number	US-202016915577-A
Country	US
Kind code	B2
Filing date	Jun 29, 2020
Priority date	Jun 29, 2020
Publication date	Nov 1, 2022
Grant date	Nov 1, 2022

Abstract

In various examples, systems and methods are disclosed that accurately identify driver and passenger in-cabin activities that may indicate a biomechanical distraction that prevents a driver from being fully engaged in driving a vehicle. In particular, image data representative of an image of an occupant of a vehicle may be applied to one or more deep neural networks (DNNs). Using the DNNs, data indicative of key point locations corresponding to the occupant may be computed, a shape and/or a volume corresponding to the occupant may be reconstructed, a position and size of the occupant may be estimated, hand gesture activities may be classified, and/or body postures or poses may be classified. These determinations may be used to determine operations or settings for the vehicle to increase not only the safety of the occupants, but also of surrounding motorists, bicyclists, and pedestrians.

First claim

What is claimed is:

1. A method comprising:
applying, to a deep neural network (DNN), image data representative of an image of at least one occupant of a vehicle;
computing, using the DNN, key point data indicative of key point locations corresponding to the at least one occupant;
based at least in part on the key point locations, reconstructing a shape and a volume corresponding to the at least one occupant;
estimating a position of the at least one occupant and a size of the at least one occupant based at least in part on the shape and the volume; and
performing one or more actions based at least in part on the position and the size of the at least one occupant.

2. The method of claim 1 , wherein the key point data is further indicative of one or more angles of one or more appendages of the at least one occupant at the key point locations, and the reconstructing the shape and the volume is further based at least in part on the one or more angles.

3. The method of claim 1 , further comprising:
comparing the position of the occupant, in image space, to predetermined locations of one or more seats within the vehicle; and
based at least in part on the comparing, determining that the occupant is in a seat of the one or more seats;
wherein an action of the one or more actions includes activating an airbag corresponding to the seat based at least in part on the determining that the occupant is in the seat.

4. The method of claim 1 , further comprising:
determining, based at least in part on the key point data, a first angle corresponding to a first line from the reconstructed shape extending between a first key point associated with the left wrist of the at least one occupant and a second key point associated with the left elbow of the at least one occupant;
based at least in part on the first angle, cropping the image to generate a first cropped image corresponding to the left hand of the at least one occupant;
determining, based at least in part on the key point data, a second angle corresponding to a second line from the reconstructed shape extending between a third key point associated with the left wrist of the at least one occupant and a fourth key point associated with the left elbow of the at least one occupant;
based at least in part on the second angle, cropping the image to generate a second cropped image corresponding to the right hand of the at least one occupant;
applying second data representative of the first cropped image and the second cropped image to another DNN; and
computing, using the another DNN, third data indicative of a first activity associated with the left hand and a second activity associated with the right hand,
wherein the performing the one or more actions is based at least in part on at least one of the first activity or the second activity.

5. The method of claim 4 , further comprising:
comparing priority values corresponding to the first activity and the second activity;
wherein the performing the one or more actions is further based at least in part on the priority values.

6. The method of claim 1 , wherein the key point data corresponds to an instance of the DNN, and the method further comprises:
computing, using the DNN, additional key point data representative of respective key point locations of the at least one occupant over a plurality of instances of the DNN and corresponding to a plurality of sequential images, wherein the plurality of instances includes the instance and the plurality of sequential images includes the image;
reconstructing one or more shapes of the at least one occupant for each of the plurality of sequential images based at least in part on the additional key point data;
generating a tensor corresponding to the one or more shapes;
applying, to another DNN, second data representative of the tensor; and
computing, using the another DNN, an activity associated with the at least one occupant,
wherein the performing the one or more actions is further based at least in part on the activity.

7. The method of claim 1 , wherein the one or more actions include one or more of: switching to manual control of the vehicle from autonomous control, switching from autonomous control of the vehicle to manual control, autonomously executing a safety procedure of the vehicle, generating an audible notification, generating a tactile notification, generating a visual notification, generating a textual notification, activating an air bag, deactivating an air bag, or adjusting actuation levels corresponding to one or more of a brake or an accelerator of the vehicle.

8. The method of claim 1 , wherein the DNN is trained using multi-view input images from multiple cameras within the vehicle.

9. A method comprising:
applying, to a first deep neural network (DNN), image data representative of an image of a field of view of an interior of a vehicle including a person;
computing, using the first DNN, first data representative of a first location and a first angle associated with a left wrist of the person and a second location and a second angle associated with a right wrist of the person;
generating, based at least in part on the first location and the first angle, a first cropped image from the image that correspond to the left hand of the person;
generating, based at least in part on the second location and the second angle, a second cropped image from the image that corresponds to the right hand of the person;
applying, to a second DNN, second data representative of the first cropped image and the second cropped image;
determining, using the second DNN, a first activity corresponding to the left hand and a second activity corresponding to the right hand; and
based at least in part on at least one of the first activity or the second activity, performing an action.

10. The method of claim 9 , further comprising:
determining a first priority value corresponding to the first activity and a second priority value corresponding to the second activity; and
based at least in part on the first priority value and the second priority value, selecting an activity from the first activity and the second activity having the highest priority value;
wherein the action corresponds to the selected activity.

11. The method of claim 9 , wherein the first data is representative of locations of a plurality of key points, and the method further comprises:
generating a skeletal model of the person based at least in part on the locations of the plurality of key points;
determining a position of the person based at least in part on the skeletal model; and
determining a seat occupancy corresponding to the person within the vehicle based at least in part on the position.

12. The method of claim 11 , further comprising:
generating a volumetric reconstruction of the person based at least in part on the locations of the plurality of key points;
determining a size of the person based at least in part on the volumetric reconstruction; and
determining an activation status of an airbag based at least in part on the seat occupancy and the size.

13. The method of claim 9 , wherein the first angle is formed by a line extending from a first key point corresponding to a left wrist of the person and a second key point corresponding to a left elbow of the person, and the second angle is formed by a third key point corresponding to a right wrist of the person and a fourth key point corresponding to a right elbow of the person.

14. The method of claim 9 , wherein:
the generating the first cropped image from the image includes generating a first bounding box corresponding to the left hand, and c

Assignees

• Nvidia Corp

Inventors

• Torabi Atousa
• Sivaraman Sakthivel
• Avadhanam Niranjan
• Sah Shagan

Classifications

• G06N3/048

  Activation functions · CPC title

• G06N3/045

  Combinations of networks · CPC title

• B60W2050/0062

  Adapting control system settings · CPC title

• B60W2540/223

  Posture, e.g. hand, foot, or seat position, turned or inclined · CPC title

• G06V20/597

  Recognising the driver's state or behaviour, e.g. attention or drowsiness · CPC title