Method, system and computer readable storage media for registering intraoral measurements

Patent metadata

Field	Value
Publication number	US-11250580-B2
Application number	US-201916580084-A
Country	US
Kind code	B2
Filing date	Sep 24, 2019
Priority date	Sep 24, 2019
Publication date	Feb 15, 2022
Grant date	Feb 15, 2022

Abstract

A method, system and computer readable storage media for segmenting individual intra-oral measurements and registering said individual intraoral measurements to eliminate or reduce registration errors. An operator may use a dental camera to scan teeth and a trained deep neural network may automatically detect portions of the input images that can cause registration errors and reduce or eliminate the effect of these sources of registration errors.

First claim

What is claimed is:

1. A computer implemented method for three-dimensional (3D) registration, the method comprising:
receiving, by one or more computing devices, individual images configured as depth and corresponding color images of a patient's dentition the depth and corresponding color images being mapped together;
providing pixels of the individual images as input to a trained deep neural network;
automatically identifying sources of registration errors in the individual images using one or more output label values of the trained deep neural network, wherein the output label values are obtained by segmenting the individual images into regions corresponding to one or more object categories;
registering the individual images together based on the one or more output label values to form a registered 3D image that has no registration errors or substantially no registration errors,
wherein the individual images are depth and corresponding color images that are mapped together by mapping the depth images to the corresponding color images or mapping the corresponding color images to the depth images
wherein the method further comprises:
generating a point cloud from the depth images by projecting pixels of the depth images into space:
assigning, responsive to the generating, color values and label values to each point in the point cloud using the corresponding color images and the output label values of the trained deep neural network respectively; and
based on the assigned label values, discarding or partially including one or more points in the point cloud using predetermined weights, such that the contributions of the discarded or partially included one or more points to registration is eliminated or reduced.

2. The method according to claim 1 , wherein the individual images are individual three dimensional optical images.

3. The method according to claim 1 , wherein the individual images are received as a temporal sequence of images.

4. The method according to claim 1 , wherein the one or more object categories include hard gingiva, soft tissue gingiva, tongue, cheek, tooth and tooth-like objects.

5. The method according to claim 1 , wherein an indication of a relevance of an identified source of registration error is based on its surrounding geometry.

6. The method according to claim 1 , wherein the deep neural network is a network chosen from the group consisting of a Convolutional Neural Network (CNN), a Fully Convolutional Neural Network (FCN), a Recurrent Neural Network (RNN) and a Recurrent Convolutional Neural Network (Recurrent-CNN).

7. The method according to claim 1 , further comprising:
training the deep neural network using the one or more computing devices and a plurality of individual training images, to map one or more tissues in at least one portion of each training image to one or more label values,
wherein the training is done on a pixel level by classifying the individual training images, pixels of the individual training images, or super pixels of the individual training images into one or more classes corresponding to semantic data types and/or error data types.

8. The method according to claim 7 wherein the training images include 3D meshes and registered pairs of depth and color images.

9. The method according to claim 8 , wherein the 3D meshes are labelled and the labels are transferred to the registered pairs of 3D and color images using a transformation function.

10. A non-transitory computer-readable storage medium storing a program which, when executed by a computer system, causes the computer system to perform a procedure comprising:
receiving, by one or more computing devices, individual images configured as depth and corresponding color images of a patient's dentition, the depth and corresponding color images being mapped together,
providing pixels of the individual images as input to a trained deep neural network:
automatically identifying sources of registration errors in the individual images using one or more output label values of the trained deep neural network, wherein the output label values are obtained by segmenting the individual images into regions corresponding to one or more object categories:
wherein the individual images are depth and corresponding color images that are mapped together by mapping the depth images to the corresponding color images or mapping the corresponding color images to the depth images;
the procedure comprising registering the individual images together based the one or more output label values to form a registered 3D image having no registration errors or substantially no registration errors; and the procedure further comprising:
generating a point cloud from the depth images by projecting pixels of the depth images into space;
assigning, responsive to the generating, color values and label values to each point in the point cloud using the corresponding color images and the output label values of the trained deep neural network respectively; and
based on the assigned label values, discarding or partially including one or more points in the point cloud using predetermined weights, such that the contributions of the discarded or partially included one or more points to registration is eliminated or reduced.

11. A system for three-dimensional (3D) registration, comprising a processor configured to:
receive, by one or more computing devices, individual images configured as depth and corresponding color images of a patient's dentition, the depth and corresponding color images being mapped together;
providing pixels of the individual images as input to a trained deep neural network;
automatically identify sources of registration errors in the individual images using one or more output label values of the trained deep neural network, wherein the output label values are obtained by segmenting the individual images into regions corresponding to one or more object categories:
wherein the individual images are depth and corresponding color images that are mapped together by mapping the depth images to the corresponding color images or mapping the corresponding color images to the depth images;
wherein the processor is configured to register the individual images together based the one or more output label values to form a registered 3D image having no registration errors or substantially no registration errors
wherein the processor is further configured to:
generate a point cloud from the depth images by projecting pixels of the depth images into space;
assign, responsive to the generating, color values and label values to each point in the point cloud using the corresponding color images and the output label values of the trained deep neural network respectively; and
based on the assigned label values, discard or partially include one or more points in the point cloud using predetermined weights, such that the contributions of the discarded or partially included one or more points to registration is eliminated or reduced.

12. The system according to claim 11 , wherein the deep neural network is a network chosen from the group consisting of a Convolutional Neural Network (CNN), a Fully Convolutional Neural Network (FCN), a Recurrent Neural Network (RNN) and a Recurrent Convolutional Neural Networks (Recurrent-CNN).

Assignees

• Dentsply Sirona Inc

Inventors

• Meyer Marcel
• Schneider Sascha
• Adamson Anders
• Schnabel Ruwen

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title