Operation control device, operation control method, and storage medium

Patent metadata

Field	Value
Publication number	US-11220181-B2
Application number	US-202016991064-A
Country	US
Kind code	B2
Filing date	Aug 12, 2020
Priority date	Aug 22, 2019
Publication date	Jan 11, 2022
Grant date	Jan 11, 2022

Abstract

An operation control device includes a line-of-sight detector configured to detect a line of sight of a driver of a vehicle on the basis of an image of an interior of the vehicle captured by an imaging device provided in the vehicle, a hand state detector configured to detect a hand state of the driver on the basis of the image, a driving state determiner configured to determine whether the detected hand state is a first state in which it is estimated that a driving operation of the vehicle is being performed or a second state in which it is estimated that the driving operation is not being performed, a traveling state detector configured to detect a traveling state of the vehicle, and an operation restrictor configured to restrict reception of an operation on an operator including a display device arranged within the vehicle.

First claim

What is claimed is:

1. An operation control device comprising:
a line-of-sight detector configured to detect a line of sight of a driver of a vehicle on the basis of an image of an interior of the vehicle captured by an imaging device provided in the vehicle;
a hand state detector configured to detect a hand state of the driver on the basis of the image;
a driving state determiner configured to determine whether the detected hand state is a first state in which it is estimated that a driving operation of the vehicle is being performed or a second state in which it is estimated that the driving operation is not being performed;
a traveling state detector configured to detect a traveling state of the vehicle; and
an operation restrictor configured to restrict reception of an operation on an operator including a display device arranged within the vehicle,
wherein the operation restrictor is configured to not restrict the reception of the operation in a case where the vehicle is in a stopped state,
wherein the operation restrictor is configured to not restrict the reception of the operation in a case where the vehicle is in the traveling state, a direction of the detected line of sight is facing a predetermined region on a side in front of the driver, and the detected hand state is the first state,
wherein the operation restrictor is configured to restrict the reception of the operation in a case where the vehicle is in the traveling state, and the direction of the detected line of sight is not facing the predetermined region or the detected hand state is the second state,
wherein the operation control device further comprises:
a traveling stability detector configured to detect stability of the traveling state of the vehicle; and
a disturbance detector configured to detect a fourth state in which detection of one or both of the direction of the line of sight and the hand state is disabled by an occupant other than the driver on the basis of the image, and
wherein the operation restrictor is configured to continue a state in which the reception of the operation is not restricted in a case where the disturbance detector detects the fourth state, and the traveling stability detector detects that the traveling state of the vehicle is stable in the state in which the reception of the operation is not restricted.

2. The operation control device according to claim 1 ,
wherein the predetermined region includes at least a region of a part or all of a front windshield of the vehicle and a region near a side mirror.

3. The operation control device according to claim 1 ,
wherein the operation restrictor is configured to not restrict the reception of the operation during a first predetermined time period from a point in time when a third state in which detection of one or both of the direction of the line of sight and the hand state is impossible is reached in a state in which the reception of the operation is not restricted.

4. The operation control device according to claim 3 ,
wherein the operation restrictor is configured to initialize a timer for measuring the first predetermined time period at a point in time when the direction of the detected line of sight is facing the predetermined region and the detected hand state is the first state after the third state is lost.

5. The operation control device according to claim 1 ,
wherein the disturbance detector is configured to detect a fifth fourth state in which detection of one or both of the direction of the line of sight and the hand state is disabled due to a factor other than the driver on the basis of the image,
wherein the operation restrictor is configured to continue a state in which the reception of the operation is not restricted in a case where the disturbance detector detects the fifth state and the traveling stability detector detects that the traveling state of the vehicle is stable in the state in which the reception of the operation is not restricted.

6. The operation control device according to claim 1 ,
wherein the operation restrictor is configured to continue a state in which the reception of the operation is not restricted during a second predetermined time period after a state in which the reception of the operation is not restricted, the disturbance detector detects the fourth state, and the traveling stability detector detects that the traveling state of the vehicle is stable is reached.

7. The operation control device according to claim 6 ,
wherein the operation restrictor is configured to initialize a timer for measuring the second predetermined time period at a point in time when the direction of the detected line of sight is facing the predetermined region and the detected hand state is the first state after the fourth state is no longer detected by the disturbance detector.

8. An operation control method using a computer comprising:
detecting a line of sight of a driver of a vehicle on the basis of an image of an interior of the vehicle captured by an imaging device provided in the vehicle;
detecting a hand state of the driver on the basis of the image;
determining whether the detected hand state is a first state in which it is estimated that a driving operation of the vehicle is being performed or a second state in which it is estimated that the driving operation is not being performed;
detecting a traveling state of the vehicle;
releasing a restriction on reception of an operation on an operator including a display device arranged within the vehicle in a case where the vehicle is in a stopped state;
releasing the restriction on the reception of the operation in a case where the vehicle is in the traveling state, a direction of the detected line of sight is facing a predetermined region on a side in front of the driver, and the detected hand state is the first state; and
restricting the reception of the operation in a case where the vehicle is in the traveling state, and the direction of the detected line of sight is not facing the predetermined region or the detected hand state is the second state;
wherein the operation control method further comprises:
detecting stability of the traveling state of the vehicle;
detecting a fourth state in which detection of one or both of the direction of the line of sight and the hand state is disabled by an occupant other than the driver on the basis of the image, and
continuing a state in which the reception of the operation is not restricted in a case where the fourth state is detected, and it is detected that the traveling state of the vehicle is stable in the state in which the reception of the operation is not restricted.

9. A computer-readable non-transitory storage medium storing a program for causing a computer to:
detect a line of sight of a driver of a vehicle on the basis of an image of an interior of the vehicle captured by an imaging device provided in the vehicle;
detect a hand state of the driver on the basis of the image;
determine whether the detected hand state is a first state in which it is estimated that a driving operation of the vehicle is being performed or a second state in which it is estimated that the driving operation is not being performed;
detect a traveling state of the vehicle;
release a restriction on reception of an operation on an operator including a display device arranged within the vehicle in a case where the vehicle is in a stopped state;
release the restriction on the reception of the operation in a case where the vehicle is in the traveling state, a direction of the detected line of sight is facing a predetermined region on a side in front of the driver, and the detected hand state is the first state; and
restrict the reception of the operation in

Assignees

• Honda Motor Co Ltd

Inventors

• Okuma Kenji
• Zhang Weifeng
• Kusaka Takashi
• Ikenobu Katsuya
• Sato Hiromi

Classifications

• G09G5/00 Primary

  Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators (image data processing or generation, in general G06T) · CPC title

• B60K35/28

  characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver · CPC title

• B60K35/60

  Instruments characterised by their location or relative disposition in or on vehicles (arrangements of lighting devices on dashboards B60Q3/10) · CPC title

• B60K35/81

  for controlling displays · CPC title

• B60K35/22

  Display screens · CPC title