Systems and methods for electron cryotomography reconstruction

Patent metadata

Field	Value
Publication number	US-2024412377-A1
Application number	US-202318542803-A
Country	US
Kind code	A1
Filing date	Dec 18, 2023
Priority date	Jul 26, 2021
Publication date	Dec 12, 2024
Grant date	—

Abstract

Described herein are methods and non-transitory computer-readable media of a computing system configured to obtain a plurality of images of an object from a plurality of orientations at a plurality of times. A machine learning model is encoded to represent a continuous density field of the object that maps a spatial coordinate to a density value. The machine learning model comprises a deformation module configured to deform the spatial coordinate in accordance with a timestamp and a trained deformation weight. The machine learning model further comprises a neural radiance module configured to derive the density value in accordance with the deformed spatial coordinate, the timestamp, a direction, and a trained radiance weight. The machine learning model is trained using the plurality of images. A three-dimensional structure of the object is constructed based on the trained machine learning model.

First claim

What we claim is:

1 . A computer-implemented method comprising:
obtaining, by a computing system, a plurality of images of an object from a plurality of orientations at a plurality of times; encoding, by the computing system, a machine learning model to represent a continuous density field of the object, wherein the continuous density field maps a spatial coordinate to a density value, and the machine learning model comprises:
a deformation module configured to deform the spatial coordinate in accordance with a timestamp and a trained deformation weight and to obtain a deformed spatial coordinate; and
a neural radiance module configured to derive the density value in accordance with the deformed spatial coordinate, the timestamp, a direction, and a trained radiance weight;
training, by the computing system, the machine learning model using the plurality of images to obtain a trained machine learning model; and constructing, by the computing system, a three-dimensional structure of the object based on the trained machine learning model.

2 . The computer-implemented method of claim 1 , wherein each image of the plurality of images comprises an image identification, and the image identification is encoded into a high dimension feature using positional encoding.

3 . The computer-implemented method of claim 1 , wherein the spatial coordinate, the direction, and the timestamp are encoded into a high dimension feature using positional encoding.

4 . The computer-implemented method of claim 1 , wherein obtaining the plurality of images of the object from the plurality of orientations at the plurality of times comprises:
obtaining a plurality of cryo-ET images by mechanically tilting the object at different angles.

5 . The computer-implemented method of claim 1 , wherein the deformation module comprises a first multi-layer perceptron (MLP).

6 . The computer-implemented method of claim 5 , wherein the first MLP comprises an 8-layer MLP with a skip connection at a fourth layer.

7 . The computer-implemented method of claim 1 , wherein the neural radiance module comprises a second multi-layer perceptron (MLP).

8 . The computer-implemented method of claim 7 , wherein the second MLP comprises an 8-layer multi-layer perceptron (MLP) with a skip connection at a fourth layer.

9 . The computer-implemented method of claim 1 , wherein training the machine learning model using the plurality of images comprises:
partitioning the plurality of images into a plurality of bins; selecting a plurality of first sample images from the plurality of bins, wherein each of the plurality of first sample images is selected from a bin of the plurality of bins; and training the machine learning model using the plurality of first sample images.

10 . The computer-implemented method of claim 9 , further comprising:
producing, by the computing system, a piecewise-constant probability distribution function (PDF) for the plurality of images based on the machine learning model; selecting, by the computing system, a plurality of second sample images from the plurality of images in accordance with the piecewise-constant PDF; and further training, by the computing system, the machine learning model using the plurality of second sample images.

11 . A non-transitory computer-readable media of a computing system storing instructions, wherein when the instructions are executed by one or more processors of the computing system, the computing system performs a method comprising:
obtaining a plurality of images of an object from a plurality of orientations at a plurality of times; encoding a machine learning model represent a continuous density field of the object, wherein the continuous density field maps a spatial coordinate to a density value, and the machine learning model comprises:
a deformation module configured to deform the spatial coordinate in accordance with a timestamp and a trained deformation weight and to obtain a deformed spatial coordinate; and
a neural radiance module configured to derive the density value in accordance with the deformed spatial coordinate, the timestamp, a direction, and a trained radiance weight;
training the machine learning model using the plurality of images to obtain a trained machine learning model; and constructing a three-dimensional structure of the object based on the trained machine learning model.

12 . The non-transitory computing medium of claim 11 , wherein each image of the plurality of images comprises an image identification, and the image identification is encoded into a high dimension feature using positional encoding.

13 . The non-transitory computing medium of claim 11 , wherein the spatial coordinate, the direction, and the timestamp are encoded into a high dimension feature using positional encoding.

14 . The non-transitory computing medium of claim 11 , wherein obtaining the plurality of images of the object from the plurality of orientations at the plurality of times comprises:
obtaining a plurality of cryo-ET images by mechanically tilting the object at different angles.

15 . The non-transitory computing medium of claim 11 , wherein the deformation module comprises a first multi-layer perceptron (MLP).

16 . The non-transitory computing medium of claim 15 , wherein the first MLP comprises an 8-layer MLP with a skip connection at a fourth layer.

17 . The non-transitory computing medium of claim 11 , wherein neural radiance module comprises a second multi-layer perceptron (MLP).

18 . The non-transitory computing medium of claim 17 , wherein the second MLP comprises an 8-layer multi-layer perceptron (MLP) with a skip connection at a fourth layer.

19 . The non-transitory computing medium of claim 11 , wherein training the machine learning model using the plurality of images comprises:
partitioning the plurality of images into a plurality of bins; selecting a plurality of first sample images from the plurality of bins, wherein each of the plurality of first sample images is selected from a bin of the plurality of bins; and training the machine learning model using the plurality of first sample images.

20 . The non-transitory computing medium of claim 19 , wherein the instructions, when executed, further causes the computing system to perform:
producing a piecewise-constant probability distribution function (PDF) for the plurality of images based on the machine learning model; selecting a plurality of second sample images from the plurality of images in accordance with the piecewise-constant PDF; and further training the machine learning model using the plurality of second sample images.

Assignees

• Univ Shanghai Technology

Inventors

• Wang Peihao
• Zhang Jiakai
• Liu Xinhang
• Liu Zhijie
• Yu Jingyi

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T15/10

  Geometric effects · CPC title

• G06T9/002

  using neural networks · CPC title

• G06T2207/10061

  from scanning electron microscope · CPC title