Systems and methods for annotating 3d data

Patent metadata

Field	Value
Publication number	US-2024135684-A1
Application number	US-202217969876-A
Country	US
Kind code	A1
Filing date	Oct 20, 2022
Priority date	Oct 20, 2022
Publication date	Apr 25, 2024
Grant date	—

Abstract

Described herein are systems, methods, and instrumentalities associated with automatically annotating a 3D image dataset. The 3D automatic annotation may be accomplished based on a 2D annotation provided by an annotator and by propagating the 2D annotation through multiple images of a sequence of 2D images associated with the 3D image dataset. The automatically annotated 3D image dataset may then be used to annotate other 3D image datasets based on similarities between the first 3D image dataset and the other 3D image datasets. The automatic annotation of the first 3D image dataset and/or the other 3D image datasets may be conducted based on one or more machine-learning models trained for performing those tasks.

First claim

What is claimed is:

1 . An apparatus, comprising:
at least one processor configured to:
obtain a first sequence of two-dimensional (2D) images associated with a first three-dimensional (3D) image dataset;
receive an annotation associated with a first 2D image in the first sequence of 2D images; and
generate a 3D annotation for the first 3D image dataset based at least on the annotation associated with the first 2D image, wherein the 3D annotation is generated by automatically annotating, based on the annotation associated with the first 2D image and a first machine-learned (ML) data annotation model, multiple other 2D images of the first sequence of 2D images.

2 . The apparatus of claim 1 , wherein the first sequence of 2D images is obtained along an axis of the first 3D image dataset.

3 . The apparatus of claim 1 , wherein the annotation associated with the first 2D image is created by a user of the apparatus, and wherein the at least one processor is further configured to provide a graphical user interface for the user to create the annotation.

4 . The apparatus of claim 1 , wherein the annotation associated with the first 2D image indicates an object of interest in the first 2D image, and wherein the first ML data annotation model is trained for detecting features associated with the object of interest in the multiple other 2D images of the first sequence of 2D images and automatically annotating the multiple other 2D images based on the detected features.

5 . The apparatus of claim 1 , wherein the 3D annotation for the first 3D image dataset is generated further based on a user input that modifies the 3D annotation.

6 . The apparatus of claim 5 , wherein the at least one processor is further configured to adjust parameters of the first ML data annotation model based on the user input.

7 . The apparatus of claim 1 , wherein the at least one processor is further configured to obtain a second 3D image dataset and generate a 3D annotation for the second 3D image dataset based on the 3D annotation of the first 3D dataset.

8 . The apparatus of claim 7 , wherein the at least one processor being configured to generate the 3D annotation for the second 3D image dataset comprises the at least one processor being configured to:
obtain a second sequence of 2D images along an axis of the second 3D image dataset; identify a second 2D image in the second sequence of 2D images based on a similarity between the first 3D image dataset and the second 3D image dataset; annotate, automatically, the second 2D image based on the annotation associated with the first 2D image; and generate the 3D annotation for the second 3D image dataset by automatically annotating multiple other 2D images of the second sequence of 2D images based at least on the automatically annotated second 2D image and the first ML data annotation model.

9 . The apparatus of claim 8 , wherein the at least one processor is further configured to receive a user input that modifies the automatic annotation of the second 2D image and generate the 3D annotation for the second 3D image dataset further based on the user input.

10 . The apparatus of claim 8 , wherein the at least one processor is configured to identify and annotate the second 2D image based on a second ML data annotation model.

11 . The apparatus of claim 7 , wherein the first 3D image dataset includes medical images associated with a first patient and wherein the second 3D image dataset includes medical images associated with a second patient.

12 . A method of automatic image annotation, comprising:
obtaining a first sequence of two-dimensional (2D) images associated with a first three-dimensional (3D) image dataset; receiving an annotation associated with a first 2D image in the first sequence of 2D images; and generating a 3D annotation for the first 3D image dataset based at least on the annotation associated with the first 2D image, wherein the 3D annotation is generated by automatically annotating, based on the annotation associated with the first 2D image and a first machine-learned (ML) data annotation model, multiple other 2D images of the first sequence of 2D images.

13 . The method of claim 12 , wherein the first sequence of 2D images is obtained along an axis of the first 3D image dataset.

14 . The method of claim 12 , wherein the annotation associated with the first 2D image is created by an annotator, and wherein the method further comprises providing a graphical user interface for the annotator to create the annotation.

15 . The method of claim 12 , wherein the annotation associated with the first 2D image indicates an object of interest in the first 2D image, and wherein the first ML data annotation model is trained for detecting features associated with the object of interest in the multiple other 2D images of the first sequence of 2D images and automatically annotating the multiple other 2D images based on the detected features.

16 . The method of claim 12 , wherein the 3D annotation for the first 3D image dataset is generated further based on a user input that modifies the 3D annotation.

17 . The method of claim 16 , further comprising adjusting parameters of the first ML data annotation model based on the user input.

18 . The method of claim 12 , further comprising obtaining a second 3D image dataset and generating a 3D annotation for the second 3D image dataset based on the 3D annotation of the first 3D dataset.

19 . The method of claim 18 , wherein generating the 3D annotation for the second 3D image dataset comprises:
obtaining a second sequence of 2D images along an axis of the second 3D image dataset; identifying a second 2D image in the second sequence of 2D images based on a similarity between the first 3D image dataset and the second 3D image dataset; annotating, automatically, the second 2D image based on the annotation associated with the first 2D image; and generating the 3D annotation for the second 3D image dataset by automatically annotating multiple other 2D images of the second sequence of 2D images based at least on the automatically annotated second 2D image and the first ML data annotation model.

20 . The method of claim 19 , wherein the identification and annotation of the second 2D image are conducted based on a second ML data annotation model.

Assignees

• Shanghai United Imaging Intelligence Co Ltd

Inventors

• Zheng Meng
• Karanam Srikrishna
• Wu Ziyan
• Innanje Arun
• Chen Terrence

Classifications

• G06V10/235

  based on user input or interaction · CPC title

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06V10/774 Primary

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06T7/0012 Primary

  Biomedical image inspection · CPC title

• G06T2207/20081

  Training; Learning · CPC title