Translation of sign language in a virtual environment

Patent metadata

Field	Value
Publication number	US-12594496-B2
Application number	US-202318296054-A
Country	US
Kind code	B2
Filing date	Apr 5, 2023
Priority date	Apr 5, 2023
Publication date	Apr 7, 2026
Grant date	Apr 7, 2026

Abstract

A method is provided, including the following operations: receiving video of a first user; processing the video to identify signed communications in a first sign language made by the first user; translating the signed communications in the first sign language into signed communications in a second sign language; rendering an avatar of the user performing the translated signed communications in the second sign language; presenting the avatar on a display for viewing by a second user.

First claim

The invention claimed is:

1 . A computer-implemented method, comprising:
receiving video of a first user of a video game; processing the video to identify gestures made by the first user while the first user plays the video game; receiving context data indicating actions occurring in the video game when the gestures are made by the first user; determining, using the context data, whether the gestures made by the first user include signed communications in a first sign language; in response to determining that the gestures made by the first user include signed communications in the first sign language, translating the signed communications in the first sign language into signed communications in a second sign language; rendering an avatar of the first user performing the translated signed communications in the second sign language; and presenting the avatar on a display for viewing by a second user.

2 . The computer-implemented method of claim 1 , wherein the translating includes translating the signed communication in the first sign language into text in a first written language.

3 . The computer-implemented method of claim 2 , wherein the translating further includes translating the text in the first written language into text in a second written language, and translating the text in the second written language into the signed communications in the second sign language.

4 . The computer-implemented method of claim 1 , wherein the avatar is rendered in a virtual environment, and wherein presenting the avatar includes presenting a view of the virtual environment on the display.

5 . The computer-implemented method of claim 4 , wherein the virtual environment is defined for gameplay of the video game.

6 . The computer-implemented method of claim 1 , wherein the method enables substantial real-time communication between the first user and the second user.

7 . The computer-implemented method of claim 1 , comprising:
determining, using the context data, that the identified gestures made by the first user further include non-signing gestures made by the first user; when the non-signing gestures are made by the first user, rendering the avatar performing the non-signing gestures in a substantially similar manner to the first user; and when the signed communications in the first sign language are made by the first user, rendering the avatar performing the translated signed communications in the second sign language.

8 . A non-transitory computer-readable medium having program instructions embodied thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising:
receiving video of a first user of a video game; processing the video to identify gestures made by the first user while the first user plays the video game; receiving context data indicating actions occurring in the video game when the gestures are made by the first user; determining, using the context data, whether the gestures made by the first user include signed communications in a first sign language; in response to determining that the gestures made by the first user include signed communications in the first sign language, translating the signed communications in the first sign language into signed communications in a second sign language; rendering an avatar of the first user performing the translated signed communications in the second sign language; and presenting the avatar on a display for viewing by a second user.

9 . The non-transitory computer-readable medium of claim 8 , wherein the translating includes translating the signed communication in the first sign language into text in a first written language.

10 . The non-transitory computer-readable medium of claim 9 , wherein the translating further includes translating the text in the first written language into text in a second written language, and translating the text in the second written language into the signed communications in the second sign language.

11 . The non-transitory computer-readable medium of claim 8 , wherein the avatar is rendered in a virtual environment, and wherein presenting the avatar includes presenting a view of the virtual environment on the display.

12 . The non-transitory computer-readable medium of claim 8 , the operations comprising:
determining, using the context data, that the identified gestures made by the first user further include non-signing gestures made by the first user; when the non-signing gestures are made by the first user, rendering the avatar performing the non-signing gestures in a substantially similar manner to the first user; and when the signed communications in the first sign language are made by the first user, rendering the avatar performing the translated signed communications in the second sign language.

13 . A system comprising at least one computing device, the at least one computing device having logic configured to implement operations comprising:
receiving video of a first user of a video game; processing the video to identify gestures made by the first user while the first user plays the video game; receiving context data indicating actions occurring in the video game when the gestures are made by the first user; determining, using the context data, whether the gestures made by the first user include signed communications in a first sign language; in response to determining that the gestures made by the first user include signed communications in the first sign language, translating the signed communications in the first sign language into signed communications in a second sign language; rendering an avatar of the first user performing the translated signed communications in the second sign language; and presenting the avatar on a display for viewing by a second user.

14 . The system of claim 13 , wherein the translating includes translating the signed communication in the first sign language into text in a first written language.

15 . The system of claim 14 , wherein the translating further includes translating the text in the first written language into text in a second written language, and translating the text in the second written language into the signed communications in the second sign language.

16 . The system of claim 13 , wherein the avatar is rendered in a virtual environment defined for gameplay of the video game.

17 . The system of claim 13 , the operations comprising:
determining, using the context data, that the identified gestures made by the first user further include non-signing gestures made by the first user; when the non-signing gestures are made by the first user, th rendering the avatar performing the non-signing gestures in a substantially similar manner to the first user; and when the signed communications in the first sign language are made by the first user, rendering the avatar performing the translated signed communications in the second sign language.

18 . The computer-implemented method of claim 1 , wherein:
the context data indicates a gaze direction of the first user in a virtual environment of the video game when the gestures are made by the first user; and determining whether the gestures made by the first user include signed communication in the first sign language is based at least in part on the gaze direction of the first user in the virtual environment of the video game when the gestures are made by the first user.

19 . The computer-implemented method of claim 1 , wherein:

Assignees

• Sony Interactive Entertainment Inc

Inventors

• Bean Celeste
• Tokubo Todd

Classifications

• G06T11/00

  Two-dimensional [2D] image generation · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06V20/40

  in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06F40/58

  Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation · CPC title