Processing system, processing method, and storage medium

Patent metadata

Field	Value
Publication number	US-2025069439-A1
Application number	US-202418605592-A
Country	US
Kind code	A1
Filing date	Mar 14, 2024
Priority date	Aug 23, 2023
Publication date	Feb 27, 2025
Grant date	—

Abstract

According to one embodiment, a processing system generates first graph data based on a pose of a worker. The pose is estimated based on a first image of the worker. The first graph data includes a plurality of first nodes corresponding respectively to a plurality of joints of the worker, and a plurality of first edges corresponding respectively to a plurality of skeletal parts of the worker. The processing system inputs the first graph data to a neural network including a graph neural network (GNN). The processing system estimates a task being performed by the worker, by using a result output from the neural network.

First claim

What is claimed is:

1 . A processing system, configured to:
generate first graph data based on a pose of a worker, the pose being estimated based on a first image of the worker, the first graph data including
a plurality of first nodes corresponding respectively to a plurality of joints of the worker, and
a plurality of first edges corresponding respectively to a plurality of skeletal parts of the worker; and
by inputting the first graph data to a neural network including a graph neural network (GNN) and by using a result output from the neural network, estimate a task being performed by the worker.

2 . The system according to claim 1 , further configured to:
generate second graph data based on a pose of the worker estimated based on a second image acquired after the first image, the second graph data including the plurality of first nodes and the plurality of first edges; and estimate the task by inputting the second graph data to the neural network after the inputting of the first graph data, and by using the result output from the neural network.

3 . The system according to claim 2 , wherein
the neural network includes the GNN, and a long short-term memory (LSTM) network to which an output from the GNN is input.

4 . The system according to claim 1 , further configured to:
generate second graph data based on a pose of the worker estimated based on a second image acquired after the first image, the second graph data including a plurality of nodes and a plurality of edges; and estimate the task by inputting graph data to the neural network and by using the result output from the neural network, the plurality of first nodes of the first graph data and the plurality of nodes of the second graph data being respectively connected by a plurality of edges in the graph data.

5 . The system according to claim 1 , further configured to:
generate the first graph data based on a state of an article in addition to the pose, the article being visible in the first image, the state of the article being estimated based on the first image, and the first graph data including:
the plurality of first nodes;
the plurality of first edges; and
a plurality of second nodes corresponding respectively to a plurality of the states that the article may be in.

6 . The system according to claim 1 , further configured to:
generate the first graph data based on a work location on an article in addition to the pose, the article being visible in the first image, the work location on the article being estimated based on the first image, the first graph data including:
the plurality of first nodes;
the plurality of first edges; and
a plurality of third nodes corresponding respectively to a plurality of locations of the article.

7 . The system according to claim 1 , wherein
the plurality of first nodes included in the first graph data represents coordinates of the plurality of joints of the first image.

8 . The system according to claim 1 , further configured to:
cause a display device to display
a time at which the first image is acquired, and
an estimation result of the task at the time.

9 . A processing method, comprising:
causing a processing device to
generate first graph data based on a pose of a worker, the pose being estimated based on a first image of the worker, the first graph data including
a plurality of first nodes corresponding respectively to a plurality of joints of the worker, and
a plurality of first edges corresponding respectively to a plurality of skeletal parts of the worker, and

by inputting the first graph data to a neural network including a graph neural network (GNN) and by using a result output from the neural network, estimate a task being performed by the worker.

10 . A non-transitory computer-readable storage medium storing a program,
the program, when executed by the processing device, causing the processing device to perform the method according to claim 9 .

Assignees

• Toshiba Kk

Inventors

• Oshima Hirotomo
• Yoshii Takanori
• Kato Takehiro
• Namioka Yasuo

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V40/20 Primary

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06T2207/20072

  Graph-based image processing · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/30196

  Human being; Person · CPC title