Sequence stabilization of 3D point cloud frames using motion information

Patent metadata

Field	Value
Publication number	US-12469154-B2
Application number	US-202217574139-A
Country	US
Kind code	B2
Filing date	Jan 12, 2022
Priority date	Jan 12, 2022
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

An electronic apparatus and method for sequence stabilization of point cloud frames using motion information is disclosed. The electronic apparatus receives image data that includes images of objects. The image data corresponds to a duration in which in the objects are in a dynamic state. Based on the image data, the electronic apparatus generates a point cloud sequence and extracts motion information associated with the objects. The electronic apparatus determines a first set of 3D points of a first point cloud frame that is in a static state with respect to a second set of 3D points of a second point cloud frame, based on the motion information. The first and second point cloud frames are consecutive frames of point cloud sequence. The electronic apparatus further determines a difference between the first and the second set of 3D points and updates the first point cloud frame based on the difference.

First claim

What is claimed is:

1 . An electronic apparatus, comprising:
circuitry configured to:
receive image data that includes images of at least one object, wherein the image data corresponds to a duration in which the at least one object is in a dynamic state;
generate a three-dimensional (3D) point cloud sequence based on the image data;
extract motion information associated with the at least one object, based on the image data;
control a plurality of image-capture devices to capture portions of the image data at discrete time instants from a plurality of viewpoints in a 3D space;
determine, from the plurality of image-capture devices, an image-capture device as out-of-sync with respect to remaining image-capture devices of the plurality of image-capture devices;
determine a first set of 3D points of a first point cloud frame that is in a static state with respect to a second set of 3D points of a second point cloud frame, based on the extracted motion information and the remaining image-capture devices,
wherein the first point cloud frame and the second point cloud frame are consecutive frames of the 3D point cloud sequence;

determine a difference between the first set of 3D points, that is in the static state with respect to the second set of 3D points, and the second set of 3D points, wherein the image-capture device is determined as out-of-sync based on the determined difference; and
update the first point cloud frame based on the determined difference.

2 . The electronic apparatus according to claim 1 , wherein
the circuitry is further configured to sample 3D points of each point cloud frame of the 3D point cloud sequence into a plurality of clusters, and the motion information comprises a set of motion vectors corresponding to the plurality of clusters.

3 . The electronic apparatus according to claim 2 , wherein a size of each of the plurality of clusters is associated with a count of visual features or a spatial arrangement of the visual features in the at least one object.

4 . The electronic apparatus according to claim 2 , wherein a size of each of the plurality of clusters is associated with a point density parameter associated with the 3D point cloud sequence.

5 . The electronic apparatus according to claim 1 , wherein the dynamic state corresponds to:
a global motion of the at least one object with reference to a ground plane, and a local motion of at least one part of the at least one object.

6 . The electronic apparatus according to claim 1 , wherein
the circuitry is further configured to execute a point cloud registration operation to minimize the determined difference between the first set of 3D points and the second set of 3D points, and the first point cloud frame is updated based on the minimization.

7 . The electronic apparatus according to claim 6 , wherein the point cloud registration operation is an Iterative Closest Point (ICP) registration operation.

8 . The electronic apparatus according to claim 1 , wherein the circuitry is further configured to:
update the 3D point cloud sequence based on the updated first point cloud frame; and generate a 3D video that includes a temporal sequence of 3D models based on a sequence of point cloud frames of the updated 3D point cloud sequence.

9 . A method, comprising:
in an electronic apparatus:
receiving image data that includes images of at least one object, wherein the image data corresponds to a duration in which the at least one object is in a dynamic state;
generating a three-dimensional (3D) point cloud sequence based on the image data;
extracting motion information associated with the at least one object, based on the image data;
controlling a plurality of image-capture devices to capture portions of the image data at discrete time instants from a plurality of viewpoints in a 3D space;
determining, from the plurality of image-capture devices, an image-capture device as out-of-sync with respect to remaining image-capture devices of the plurality of image-capture devices;
determining a first set of 3D points of a first point cloud frame that is in a static state with respect to a second set of 3D points of a second point cloud frame, based on the extracted motion information and the remaining image-capture devices,
wherein the first point cloud frame and the second point cloud frame are consecutive frames of the 3D point cloud sequence;

determining a difference between the first set of 3D points, that is in the static state with respect to the second set of 3D points, and the second set of 3D points, wherein
the image-capture device is determined as out-of-sync based on the determined difference; and

updating the first point cloud frame based on the determined difference.

10 . The method according to claim 9 , further comprising sampling 3D points of each point cloud frame of the 3D point cloud sequence into a plurality of clusters, wherein the motion information comprises a set of motion vectors corresponding to the plurality of clusters.

11 . The method according to claim 10 , wherein a size of each of the plurality of clusters is associated with a count of visual features or a spatial arrangement of the visual features in the at least one object.

12 . The method according to claim 10 , wherein a size of each of the plurality of clusters is associated with a point density parameter associated with the 3D point cloud sequence.

13 . The method according to claim 9 , wherein the dynamic state corresponds to:
a global motion of the at least one object with reference to a ground plane, and a local motion of at least one part of the at least one object.

14 . The method according to claim 9 , further comprising
executing a point cloud registration operation to minimize the determined difference between the first set of 3D points and the second set of 3D points, wherein the first point cloud frame is updated based on the minimization.

15 . The method according to claim 14 , wherein the point cloud registration operation is an Iterative Closest Point (ICP) registration operation.

16 . A non-transitory computer-readable medium having stored thereon, computer-executable instructions that when executed by a computer, causes the computer to execute operations, the operations comprising:
receiving image data that includes images of at least one object, wherein the image data corresponds to a duration in which the at least one object is in a dynamic state; generating a three-dimensional (3D) point cloud sequence based on the image data; extracting motion information associated with the at least one object, based on the image data; controlling a plurality of image-capture devices to capture portions of the image data at discrete time instants from a plurality of viewpoints in a 3D space; determining, from the plurality of image-capture devices, an image-capture device as out-of-sync with respect to remaining image-capture devices of the plurality of image-capture devices; determining a first set of 3D points of a first point cloud frame that is in a static state with respect to a second set of 3D points of a second point cloud frame, based on the extracted motion information and the remaining image-capture devices,
wherein the first point cloud frame and the second point cloud frame are consecutive frames of the 3D point cloud sequence;
determining a difference between the first set of 3D points, that is in the static state with respect to the second set of 3D points, and the second set of 3D points, wherein
the image-capture device is determined as out-of-

Assignees

• Sony Group Corp

Inventors

• Georgis Nikolaos
• Omori Keisuke
• Sassa Kiyoharu

Classifications

• G06T2207/20224

  Image subtraction · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06T2207/30196

  Human being; Person · CPC title