Photograph-based assessment of dental treatments and procedures

Patent metadata

Field	Value
Publication number	US-11042774-B2
Application number	US-201916370788-A
Country	US
Kind code	B2
Filing date	Mar 29, 2019
Priority date	Aug 20, 2015
Publication date	Jun 22, 2021
Grant date	Jun 22, 2021

Abstract

The current document is directed to methods and systems for monitoring a dental patient's progress during a course of treatment. A three-dimensional model of the expected positions of the patient's teeth can be projected, in time, from a three-dimensional model of the patient's teeth prepared prior to beginning the treatment. A digital camera is used to take one or more two-dimensional photographs of the patient's teeth, which are input to a monitoring system. The monitoring system determines virtual-camera parameters for each two-dimensional input image with respect to the time-projected three-dimensional model, uses the determined virtual-camera parameters to generate two-dimensional images from the three-dimensional model, and then compares each input photograph to the corresponding generated two-dimensional image in order to determine how closely the three-dimensional arrangement of the patient's teeth corresponds to the time-projected three-dimensional arrangement.

First claim

What is claimed is:

1. A dental-treatment monitoring system comprising a computing device with a non-transitory computer-readable data-storage having instructions which can be executed by one or more processors to cause the computing device to:
receive two-dimensional image data of a patient's teeth taken at a particular time during a course of dental treatment, the two-dimensional image data including associated metadata; and
using the one or more processors:
compare the two-dimensional image data with a three-dimensional model representing an expected configuration of the patient's teeth at the particular time;
determine a set of virtual-camera parameters that describe an estimated position and orientation of a virtual camera that produces a generated image from the three-dimensional model;
determine an iteration variable N based on the metadata associated with the two-dimensional image data;
iteratively modify the set of virtual-camera parameters N times to modify the generated image, wherein iterative modification includes:
a first adjustment of a current set of virtual-camera parameters for the generated image;
a second adjustment of the current set of virtual-camera parameters for the generated image, wherein the second adjustment includes: for each of two jaws of the patient, masking out the other of the two jaws and associated teeth from the generated image, and optimizing the set of virtual-camera parameters for the generated image with respect to a pixel-associated cost for the generated image and a next corresponding generated image;
a first refinement of the current set of virtual-camera parameters for the generated image; and
a second refinement of the current set of virtual-camera parameters for the generated image; and

generate a comparison value for the two-dimensional image compared to the three-dimensional model.

2. The system of claim 1 , wherein the pixel-associated cost for the generated image and the next corresponding generated image is computed as a linear combination of mutual information for the generated image and the next corresponding generated image and a sum of pixel-associated costs.

3. The system of claim 1 , wherein optimizing the set of virtual-camera parameters for the generated image comprises: for each optimization iteration:
generating the next corresponding generated image from the three-dimensional model of the patient's teeth using the current set of virtual-camera parameters for the generated image;
computing a cost for the generated image and the next corresponding generated image; and
perturbing the set of virtual-camera parameters for the generated image in a direction that minimizes the cost for the generated image and the next corresponding generated image.

4. The system of claim 1 , wherein optimizing the set of virtual-camera parameters for the generated image comprises application of a Nelder-Mead downhill simplex optimization method, with seven dimensions, including three rotations, three translations, and a virtual-camera view angle.

5. The system of claim 1 , wherein the first refinement of the current set of virtual-camera parameters for the generated image comprises: for each tooth group of a currently considered jaw:
masking the generated image to leave the tooth group of the currently considered jaw, and
optimizing the set of virtual-camera parameters for the generated image with respect to a gradient-vector-based cost for the generated image and the next corresponding generated image.

6. The system of claim 1 , wherein the first refinement of the current set of virtual-camera parameters for the generated image comprises: for each tooth group of a currently considered jaw:
masking the generated image to leave the tooth group of the currently considered jaw, and
optimizing the set of virtual-camera parameters for the generated image with respect to a Laplacian-vector-based cost for the generated image and the next corresponding generated image.

7. The system of claim 1 , wherein the metadata include one or more of: a text label, an indication of a standard type of view represented by the two-dimensional image, an image size, a date and time information, an indication of a camera model and make, an indication of a camera orientation, an indication of an aperture, an indication of a shutter speed, an indication of a focal length, an indication of a metering mode, and an indication of an image speed.

8. The system of claim 1 , wherein the instructions further cause the computing device to determine, from the comparison value, whether a configuration of the patient's teeth is within a threshold level of correspondence to the expected configuration of the patient's teeth.

9. The system of claim 1 , wherein the instructions further cause the computing device to remove teeth from the two-dimensional image that are not considered for producing the generated image.

10. The system of claim 1 , wherein iteratively modifying the set of virtual-camera parameters comprises:
applying a color and intensity threshold value to each pixel of the generated image;
determining whether each pixel corresponds to a tooth region or a non-tooth region based on the color and intensity threshold value; and
generating a teeth mask for the generated image.

11. The system of claim 1 , wherein the instructions further cause the computing device to generate multiple comparison values for multiple two-dimensional images.

12. The system of claim 1 , wherein iteratively modifying the set of virtual-camera parameters further comprises:
thresholding the generated image to generate a first teeth mask;
thresholding a next corresponding image generated from the three-dimensional model of the patient's teeth using the current set of virtual-camera parameters for the generated image to generate a next second corresponding teeth mask,
generating a distance transform of the first teeth mask,
searching over scalings, rotations, and translations of the next second corresponding teeth mask to identify a next minimum-cost overlay of the next second corresponding teeth mask over the distance transform of the first teeth mask, and
adjusting the set of virtual-camera parameters for the generated image corresponding to the next minimum-cost overlay; and
selecting, as an adjusted set of virtual-camera parameters for the generated image, the set of virtual-camera parameters associated with lowest-cost overlay of any of the generated next second corresponding teeth mask over the distance transform of the first teeth mask.

13. The system of claim 12 , wherein a cost of an overlay of the next second corresponding teeth mask over the distance transform of the first teeth mask is a sum of distances associated with pixels in the distance transform of the first teeth mask underlying a tooth region within the next second corresponding teeth mask.

14. The system of claim 12 , wherein adjusting the set of virtual-camera parameters for the generated image corresponding to the next minimum-cost overlay includes:
adjusting the set of virtual-camera parameters to move a position of a center of the next corresponding image generated from a time-projected three-dimensional model of the patient's teeth;
adjusting the set of virtual-camera parameters to rotate the next corresponding image generated;
from the time-projected three-dimensional model of the patient's teeth in an image plane;
adjusting the set of virtual-camera parameters to rotate the next corresponding image generated from the time-projected three-dimensional model of the patient's teeth about an axis parallel to the image plane; and
adjusting the set

Assignees

• Align Technology Inc

Inventors

• Borovinskih Artem
• Derakhshan Mitra
• Koppers Carina
• Meyer Eric
• Tolstaya Ekaterina
• Brailov Yury

Classifications

• A61C7/002 Primary

  Orthodontic computer assisted systems · CPC title

• G06V30/1831

  using gradient analysis · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T7/269

  using gradient-based methods · CPC title

• G06T2207/30232

  Surveillance · CPC title