Visual Line Analysis Apparatus, Visual Line Analysis Method, and Visual Line Analysis System

Patent metadata

Field	Value
Publication number	US-2022122290-A1
Application number	US-202117450723-A
Country	US
Kind code	A1
Filing date	Oct 13, 2021
Priority date	Oct 16, 2020
Publication date	Apr 21, 2022
Grant date	—

Abstract

Provided are: an extraction unit extracting a plurality of sets of combinations of planes and coordinates corresponding to a plurality of objects from among images of the respective objects belonging to image information, based on the image information including the respective images of the plurality of objects which are images in a front direction of a worker; a plane selection unit selecting a designated set from among the plurality of sets of combinations of planes and coordinates extracted by the extraction unit according to a rule in which a priority for a plane corresponding to each object is defined; and an analysis unit calculating a gaze point coordinate indicating a gaze point of the worker based on visual line information indicating a visual line position of the worker and information on a combination of a plane and a coordinate belonging to the designated set selected by the plane selection unit.

First claim

What is claimed is:

1 . A visual line analysis apparatus comprising:
an extraction unit that extracts a plurality of sets of combinations of planes and coordinates corresponding to a plurality of objects from among images of the respective objects belonging to image information, based on the image information including the respective images of the plurality of objects, the images being images in a front direction of a worker; a plane selection unit that selects a designated set from among the plurality of sets of combinations of planes and coordinates extracted by the extraction unit according to a rule in which a priority for a plane corresponding to each of the objects is defined; and an analysis unit that calculates a gaze point coordinate indicating a gaze point of the worker based on visual line information indicating a visual line position of the worker and information on a combination of a plane and a coordinate belonging to the designated set selected by the plane selection unit.

2 . The visual line analysis apparatus according to claim 1 , wherein
when the designated set is selected, the plane selection unit adopts any one rule among a first rule in which a plane located at a center of the image belonging to the image information is set as a plane with a highest priority, a second rule in which a plane present at a position closest to a visual line position of the worker is set as a plane with a high priority, and a third rule in which priority levels for the plurality of planes corresponding to the plurality of objects are defined by one priority table or two or more priority tables.

3 . The visual line analysis apparatus according to claim 2 , wherein
in each of the two or more priority tables, the priority levels are set to different values based on work situation information indicating work situations of the worker, respectively, for the plurality of planes, and the plane selection unit selects one priority table from among the two or more priority tables according to the work situation information.

4 . The visual line analysis apparatus according to claim 1 , wherein
the extraction unit stores, as an access target, a plane and coordinate management table in which a plurality of planes belonging to an image of the image information, coordinates of each of the plurality of planes, and a plurality of markers added to each of the planes are recorded in association with a plurality of data paths for identifying each of the plurality of markers, and refers to the plane and coordinate management table based on the data path added to the image information when receiving the image information to extract a plurality of sets of a combination of the plane and coordinate corresponding to the marker identified by the data path.

5 . The visual line analysis apparatus according to claim 1 , wherein
the analysis unit calculates a transformation matrix for transforming a coordinate on an image in the image belonging to the image information into a coordinate on a plane in the plane of the designated set selected by the plane selection unit based on the image information and the information selected by the plane selection unit, and calculates the gaze point coordinate based on the calculated transformation matrix and the visual line information.

6 . A visual line analysis method comprising:
an extraction step of extracting a plurality of sets of combinations of planes and coordinates corresponding to a plurality of objects from among images of the respective objects belonging to image information, based on the image information including the respective images of the plurality of objects, the images being images in a front direction of a worker; a plane selection step of selecting a designated set from among the plurality of sets of combinations of planes and coordinates extracted in the extraction step according to a rule in which a priority for a plane corresponding to each of the objects is defined; and an analysis step of calculating a gaze point coordinate indicating a gaze point of the worker based on visual line information indicating a visual line position of the worker and information on a combination of a plane and a coordinate belonging to the designated set selected in the plane selection step.

7 . The visual line analysis method according to claim 6 , wherein
in the plane selection step, when the designated set is selected, any one rule is adopted among a first rule in which a plane located at a center of the image belonging to the image information is set as a plane with a highest priority, a second rule in which a plane present at a position closest to a visual line position of the worker is set as a plane with a high priority, and a third rule in which priority levels for the plurality of planes corresponding to the plurality of objects are defined by one priority table or two or more priority tables.

8 . The visual line analysis method according to claim 7 , wherein
in each of the two or more priority tables, the priority levels are set to different values based on work situation information indicating work situations of the worker, respectively, for the plurality of planes, and in the plane selection step, one priority table is selected from among the two or more priority tables according to the work situation information.

9 . The visual line analysis method according to claim 6 , wherein
in the extraction step, a plane and coordinate management table in which a plurality of planes belonging to an image of the image information, coordinates of each of the plurality of planes, and a plurality of markers added to each of the planes are recorded in association with a plurality of data paths for identifying each of the plurality of markers is stored as an access target, and the plane and coordinate management table is referred to based on the data path added to the image information when receiving the image information to extract a plurality of sets of a combination of the plane and coordinate corresponding to the marker identified by the data path.

10 . The visual line analysis method according to claim 6 , wherein
in the analysis step, a transformation matrix for transforming a coordinate on an image in the image belonging to the image information into a coordinate on a plane in the plane of the designated set selected in the plane selection step is calculated based on the image information and the information selected in the plane selection step, and the gaze point coordinate is calculated based on the calculated transformation matrix and the visual line information.

11 . A visual line analysis system comprising:
a visual line sensor that outputs visual line information indicating a visual line position of a worker; a camera that outputs image information including images of a plurality of objects as subjects, the images being images in a front direction of the worker; and an information processing apparatus that receives and processes the visual line information output from the visual line sensor and the image information output from the camera, wherein the information processing apparatus includes: an extraction unit that extracts a plurality of sets of combinations of planes and coordinates respectively corresponding to the plurality of objects from among images belonging to the image information based on the image information; a plane selection unit that selects a designated set from among the plurality of sets of combinations of planes and coordinates extracted by the extraction unit according to a rule in which a priority for a plane corresponding to each of the objects is defined; and an analysis unit that calculates a gaze p

Assignees

• Hitachi Ltd

Inventors

• Kumon Taku
• Hirai Riu
• Nakamura Keiki

Classifications

• G06V40/18

  Eye characteristics, e.g. of the iris · CPC title

• G06V10/225 Primary

  based on a marking or identifier characterising the area · CPC title

• G06T2207/30204

  Marker · CPC title

• G06V40/19

  Sensors therefor · CPC title

• G06V20/20

  in augmented reality scenes · CPC title