Electronic device, method, and computer readable storage medium for obtaining video sequence including visual object with postures of body independently from movement of camera

Patent metadata

Field	Value
Publication number	US-12400356-B2
Application number	US-202217738394-A
Country	US
Kind code	B2
Filing date	May 6, 2022
Priority date	May 7, 2021
Publication date	Aug 26, 2025
Grant date	Aug 26, 2025

Abstract

The electronic device according to various embodiments include a memory configured to store instructions and at least one processor configured, when executing the instructions, to obtain a first video sequence including a visual object corresponding to a body; obtain a local posture sequence of the visual object that indicates postures of at least one visual element of the visual object in the first video sequence, the at least one visual element of the visual object corresponding to at least one joint of the body; obtain a global motion sequence of the visual object, based on a difference between the postures of the at least one visual element, the postures of the at least one visual object obtained by using the local posture sequence; and obtain a second video sequence, based on the local posture sequence and the global motion sequence.

First claim

What is claimed is:

1. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions to, when executed by an electronic device, cause the electronic device to:
obtain a first video sequence including a visual object corresponding to a body;
obtain a local posture sequence of the visual object that indicates postures of at least one visual element of the visual object in the first video sequence, the at least one visual element of the visual object corresponding to at least one joint of the body;
obtain a global motion sequence of the visual object, based on a difference between the postures of the at least one visual element obtained by using the local posture sequence, wherein the global motion sequence is configured with data on a difference between positions in the first video sequence of a part of the visual object, the part of the visual object corresponds to a top node of nodes which are respectively corresponds to joints of the body including the at least one joint, the nodes are associated with each other in a hierarchical manner; and
obtain a second video sequence including a new visual object representing movement of the body indicated by the first video sequence, the movement is obtained based on compensation for movement of a camera which was used to take the first video sequence using the global motion sequence.

2. The non-transitory computer readable storage medium of claim 1 , wherein the local posture sequence indicating the postures of the at least one visual element is configured with data for orientations of the at least one visual element in the first video sequence.

3. The non-transitory computer readable storage medium of claim 2 , wherein the global motion sequence is configured with data for a difference between orientations of the part of the visual object in the first video sequence.

4. The non-transitory computer readable storage medium of claim 1 , wherein the non-transitory computer readable storage medium further store instructions to, when executed by at least one processor of the electronic device individually or collectively, cause the electronic device to obtain the global motion sequence by inputting the local posture sequence to a model including gated recurrent units (GRUs).

5. The non-transitory computer readable storage medium of claim 1 , wherein the non-transitory computer readable storage medium further store instructions to, when executed by at least one processor of the electronic device individually or collectively, cause the electronic device to:
obtain a global posture sequence converted from the global motion sequence; and
obtain the second video sequence, based on the local posture sequence and the global posture sequence.

6. The non-transitory computer readable storage medium of claim 1 , wherein the first video sequence includes a plurality of images obtained, while a camera is moved, through the camera, and
wherein the non-transitory computer readable storage medium further store instructions to, when executed by at least one processor of the electronic device individually or collectively, cause the electronic device to:
identify at least one image obtained through the camera while the camera being moved has a predetermined posture, from among the plurality of images included in the first video sequence; and
obtain the local posture sequence by analyzing the plurality of images with respect to the identified at least one image.

7. The non-transitory computer readable storage medium of claim 1 , wherein the first video sequence is a video sequence obtained, while a camera is moved, through the camera, and
wherein the new visual object included in the second video sequence corresponds to the body having postures independently from the movement of the camera.

8. The non-transitory computer readable storage medium of claim 1 , wherein the non-transitory computer readable storage medium further store instructions to, when executed by at least one processor of the electronic device individually or collectively, cause the electronic device to obtain the global motion sequence by at least partially applying a skinned multi-person linear (SMPL) model to the local posture sequence.

9. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions to, when executed by an electronic device, cause the electronic device to:
obtain, using a camera being moved according to a first state, a first video sequence including a first image and a second image subsequent to the first image, the first image and the second image obtained through the camera;
identify a visual object in the first image corresponding to a body and the visual object in the second image, the visual object in the first video sequence having a shape corresponding to a representation of the body;
determine postures of the body independently from the movement of the camera, based on a difference between a posture of at least one visual element of the visual object in the first image corresponding to at least one joint of the body and a posture of at least one visual element of the visual object in the second image corresponding to the at least one joint; and
obtain a second video sequence indicating an appearance of a new visual object, which is viewed from another camera that is fixed according to a second state, having the determined postures; and
wherein the new visual object in the second video sequence has a shape distinct from the representation of the body.

10. The non-transitory computer readable storage medium of claim 9 , wherein the non-transitory computer readable storage medium further store instructions to, when executed by at least one processor of the electronic device individually or collectively, cause the electronic device to identify the difference between the posture of the at least one visual element of the visual object in the first image and the posture of the at least one visual element of the visual object in the second image, based on a difference between an orientation of the at least one visual element of the visual object in the first image and an orientation of the at least one visual element of the visual object in the second image.

11. The non-transitory computer readable storage medium of claim 9 , wherein the non-transitory computer readable storage medium further store instructions to, when executed by at least one processor of the electronic device individually or collectively, cause the electronic device to:
identify, from among a plurality of images included in the first video sequence, the first image obtained through the camera while the camera being moved has a predetermined posture; and
determine the postures of the body, based on the difference between the posture of the at least one visual element of the visual object in the identified first image and the posture of the at least one visual element of the visual object in the identified second image.

12. The non-transitory computer readable storage medium of claim 9 , wherein a movement of the visual object in the first video sequence reflects the movement of the camera and a movement of the body, and
wherein a movement of the new visual object in the second video sequence only reflects the movement of the body among the movement of the camera and the movement of the body.

13. The non-transitory computer readable storage medium of claim 9 , wherein each of a plurality of images in the first video sequence including the first image and the second image only includes a part occupied by the visual object and a background part

Assignees

• Ncsoft Corp
• Univ Kwangwoon Ind Acad Collab

Inventors

• Kim Seonghyun
• Park Sungbum
• Ahn Sangjun
• Chang Juyong
• Jeong Sunwon

Classifications

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T7/246

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title