Foreign object identification and image augmentation for intraoral scanning

Patent metadata

Field	Value
Publication number	US-2020281702-A1
Application number	US-202016809457-A
Country	US
Kind code	A1
Filing date	Mar 4, 2020
Priority date	Mar 8, 2019
Publication date	Sep 10, 2020
Grant date	—

Abstract

In a method of generating a virtual 3D model of a dental site, scan data comprising a plurality of images of a dental site is received during an intraoral scan. An analysis of an image is performed. A representation of a reference object with known properties is identified in the image based on the analysis. A virtual 3D model of the dental site is generated based on the plurality of images. The image and/or the virtual 3D model of the dental site is modified by adding additional data about the reference object to the intraoral image and/or the virtual 3D model based on the one or more known properties of the reference object. If the image is modified, it may be modified prior to generation of the virtual 3D model, and the virtual 3D model may be generated using the modified image.

First claim

What is claimed is:

1 . A method comprising:
receiving intraoral scan data comprising a plurality of images of a dental site; generating a virtual three-dimensional (3D) model of the dental site based on the plurality of images; performing an analysis on an image of the dental site; identifying a representation of a reference object in the image, wherein the reference object has one or more known properties; and modifying at least one of the image or the virtual 3D model of the dental site by adding additional data about the reference object to at least one of the image or the virtual 3D model based on the one or more known properties of the reference object.

2 . The method of claim 1 , wherein the image is one of the plurality of images of the dental site used to generate the virtual 3D model, wherein the image is modified by adding the additional data about the reference object to the image, and wherein the image, as modified, is used to generate the virtual 3D model of the dental site.

3 . The method of claim 2 , further comprising:
performing image processing on the image to determine a probable object represented in the image, wherein the probable object comprises one or more physical properties; comparing the one or more physical properties of the probable object to the one or more known properties of the reference object; determining that the one or more physical properties of the probable object match the one or more known properties the reference object as a result of the comparing; and determining that the probable object is the reference object.

4 . The method of claim 3 , wherein the one or more physical properties comprise at least one of a reflectivity, a diffraction, a reactivity to a wavelength of light, a texture, a surface pattern, a color or a shape.

5 . The method of claim 1 , wherein performing the analysis of the image comprises inputting the image into a machine learning model trained to identify one or more types of reference objects, wherein the machine learning model outputs an indication of the representation of the reference object in the image.

6 . The method of claim 1 , further comprising:
generating the image on which the analysis is performed by projecting the virtual 3D model onto a plane.

7 . The method of claim 6 , further comprising:
during an intraoral scan, generating a view of the dental site based on the modified image, wherein the view comprises an outline of the reference object based on the additional data about the reference object; receiving a plurality of additional images of the dental site during the intraoral scan; replacing one or more parts of a shape of the reference object based on the plurality of additional images; and updating the view of the dental site, the updated view showing those parts of the shape of the reference object that have been replaced.

8 . The method of claim 1 , wherein identifying the representation of the reference object comprises:
processing the image using a machine learning model that has been trained to identify one or more foreign objects at dental sites; and receiving an output of the machine learning model, wherein the output comprises a binary mask that has a number of entries that is equal to a number of pixels or voxels in the image, wherein entries associated with pixels or voxels that are part of the reference object have a first value and wherein entries associated with pixels or voxels that are not part of the reference object have a second value.

9 . The method of claim 1 , further comprising performing the following before receiving the intraoral scan data:
entering a training mode; receiving a plurality of images of the reference object during the training mode; generating a virtual model of the reference object based on the plurality of images of the reference object; and adding at least one of the virtual model of the reference object or the plurality of images of the reference object to a reference object library that comprises entries for a plurality of reference objects.

10 . The method of claim 1 , further comprising:
receiving an indication that the reference object is at the dental site, the indication comprising an identification of the reference object; querying a reference object library comprising entries for a plurality of reference objects using the identification of the reference object; receiving a response to the query, the response comprising data associated with the reference object; and using the data to identify the reference object in the image.

11 . The method of claim 1 , wherein the one or more known properties comprise at least one of a known reflectivity of the reference object, a known diffraction of the reference object, a known reactivity of the reference object to a wavelength of light, a known texture of the reference object, a known surface pattern of the reference object, a known color of the reference object or a known shape of the reference object

12 . The method of claim 1 , further comprising:
determining a confidence value associated with the reference object; determining that the confidence value is below a confidence threshold; presenting an option to a) add the additional data about the reference object to at least one of the image or the virtual 3D model or b) leave a representation of the reference object in at least one of the image or the virtual 3D model unchanged; and receiving a user selection to add the additional data about the reference object to at least one of the image or the virtual 3D model.

13 . The method of claim 1 , wherein the reference object comprises at least one of a shape or a material that causes intraoral scans of the reference object to have a reduced accuracy.

14 . A non-transitory computer readable medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising:
receiving intraoral scan data comprising an intraoral image of a dental site; performing an analysis of the intraoral image to a) identify of a foreign object represented in the intraoral image and b) determine that the foreign object is an instance of a reference object having one or more known properties; modifying the intraoral image by adding additional data about the foreign object to the intraoral image based on the one or more known properties of the reference object; receiving additional intraoral scan data comprising a plurality of additional intraoral images of the dental site; and generating a virtual three-dimensional (3D) model of the dental site based on the modified intraoral image and the plurality of additional intraoral images.

15 . The non-transitory computer readable medium of claim 14 , wherein performing the analysis of the intraoral image comprises:
performing image processing on the intraoral image to determine a plurality of probable objects in the intraoral image, wherein each of the plurality of probable objects comprises one or more physical properties; comparing, for each of the plurality of probable objects, the one or more physical properties of the probable object to a plurality of known properties of reference objects; determining that one or more physical properties of a probable object of the plurality of probable objects match the one or more known properties the reference object as a result of the comparing; and determining that the probable object is the foreign object at the dental site.

16 . The non-transitory computer readable medium of claim 15 , wherein the one or more physical

Assignees

• Align Technology Inc

Inventors

• Kopelman Avi
• Sabina Michael

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• A61B1/00009

  of image signals during a use of endoscope · CPC title