Fish biomass, shape, size, or health determination

Patent metadata

Field	Value
Publication number	US-11475689-B2
Application number	US-202016734661-A
Country	US
Kind code	B2
Filing date	Jan 6, 2020
Priority date	Jan 6, 2020
Publication date	Oct 18, 2022
Grant date	Oct 18, 2022

Abstract

Methods, systems, and apparatuses, including computer programs encoded on a computer-readable storage medium for estimating the shape, size, mass, and health of fish are described. A pair of stereo cameras may be utilized to obtain off-axis images of fish in a defined area. The images may be processed, enhanced, and combined. Object detection may be used to detect and track a fish in images. A pose estimator may be used to determine key points and features of the detected fish. Based on the key points, a model of the fish is generated that provides an estimate of the size and shape of the fish. A regression model or neural network model can be applied to the fish model to determine characteristics of the fish.

First claim

What is claimed is:

1. A computer-implemented method comprising:
obtaining, by one or more processors, one or more images of a fish;
determining one or more key points associated with one or more features of the fish in the one or more images;
generating, by the one or more processors, a model of the fish based on the one or more key points associated with the one or more features of the fish;
determining, using the model of the fish, a characteristic of the fish from among: a shortened abdomen, a shortened tail, scoliosis, lordosis, kyphosis, a deformed upper jaw, a deformed lower jaw, a shortened operculum, runting or cardiomyopathy syndrome (CMS); and
outputting a representation of the characteristic of the fish for display or storage at a device connected to the one or more processors.

2. The computer-implemented method of claim 1 , further comprising:
generating a single image from the one or more images;
generating a depth map for the single image; and
identifying the fish and one or more regions of interest in the single image by performing object detection using a recurrent convolutional neural network.

3. The computer-implemented method of claim 2 , wherein:
the one or more key points associated with the one or more features of the fish are determined for each of the one or more regions of interest using pose estimation;
the one or more images are obtained using one or more image acquisition devices; and
the one or more images include an image from one image acquisition device and another image from a different image acquisition device.

4. The computer-implemented method of claim 1 , wherein:
the determined one or more key points include one or more two-dimensional key points; and
generating the model of the fish comprises:
generating a 3D model of the fish.

5. The computer-implemented method of claim 4 , wherein generating a 3D model of the fish comprises:
determining three-dimensional key points for the fish by using the determined one or more two-dimensional key points and the depth map.

6. The computer implemented method of claim 1 , wherein:
generating the model of the fish comprises:
determining a truss network comprised of length values, wherein the length values indicate distances between key points.

7. The computer-implemented method of claim 1 , wherein determining a characteristic of the fish using the model of the fish comprises:
applying a linear regression model to the model of the fish.

8. The computer-implemented method of claim 1 , further comprising:
obtaining one or more secondary images of the fish;
determining a characteristic of the fish based on the obtained one or more secondary images of the fish; and
determining a characteristic based on the characteristic determined using the model of the fish and the characteristic determined based on the obtained one or more secondary images of the fish.

9. The computer-implemented method of claim 1 , further comprising:
training a neural network classifier using a pose estimation model to predict likely key points of the fish.

10. A system comprising:
one or more computing devices and one or more storage devices storing instructions which when executed by the one or more computing devices, cause the one or more computing devices to perform operations comprising:
obtaining one or more images of a fish;
determining one or more key points associated with one or more features of the fish in the one or more images;
generating a model of the fish based on the one or more key points associated with the one or more features of the fish;
determining, using the model of the fish, a characteristic of the fish from among: a shortened abdomen, a shortened tail, scoliosis, lordosis, kyphosis, a deformed upper jaw, a deformed lower jaw, a shortened operculum, runting or cardiomyopathy syndrome (CMS); and
outputting a representation of the characteristic of the fish for display or storage at a device connected to the one or more computing devices.

11. The system of claim 10 , wherein the operations further comprise:
generating a single image from the one or more images;
generating a depth map for the single image; and
identifying the fish and one or more regions of interest in the single image by performing object detection using a recurrent convolutional neural network.

12. The system of claim 11 , wherein:
the one or more key points associated with the one or more features of the fish are determined for each of the one or more regions of interest using pose estimation;
the one or more images are obtained using one or more image acquisition devices; and
the one or more images include an image from one image acquisition device and another image from a different image acquisition device.

13. The system of claim 10 , wherein:
the determined one or more key points include one or more two-dimensional key points; and
generating the model of the fish comprises:
generating a 3D model of the fish.

14. The computer-implemented method of claim 13 , wherein generating a 3D model of the fish comprises:
determining three-dimensional key points for the fish by using the determined one or more two-dimensional key points and the depth map.

15. The system of claim 10 , wherein:
generating the model of the fish comprises:
determining a truss network comprised of length values, wherein the length values indicate distances between key points.

16. The system of claim 10 , wherein determining a characteristic of the fish using the model of the fish comprises:
applying a linear regression model to the model of the fish.

17. The system of claim 11 , wherein the operations further comprise:
obtaining one or more secondary images of the fish;
determining a characteristic of the fish based on the obtained one or more secondary images of the fish; and
determining a characteristic based on the characteristic determined using the model of the fish and the characteristic determined based on the obtained one or more secondary images of the fish.

18. One or more non-transitory computer-readable storage media comprising instructions, which, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:
obtaining one or more images of a fish;
determining one or more key points associated with one or more features of the fish in the one or more images;
generating a model of the fish based on the one or more key points associated with the one or more features of the fish;
determining, using the model of the fish, a characteristic of the fish from among: a shortened abdomen, a shortened tail, scoliosis, lordosis, kyphosis, a deformed upper jaw, a deformed lower jaw, a shortened operculum, runting or cardiomyopathy syndrome (CMS); and
outputting a representation of the characteristic of the fish for display or storage at a device connected to the one or more computing devices.

19. The one or more non-transitory computer-readable storage media of claim 18 , wherein the operations further comprise:
generating a single image from the one or more images;
generating a depth map for the single image; and
identifying the fish and one or more regions of interest in the single image by performing object detection using a recurrent convolutional neural network.

20. The one or more non-transitory computer-readable storage media of claim 19 , wherein:
the one or more key points associated with the one or more features of the fish are determined for each o

Assignees

• X Dev Llc

Inventors

• Young Grace Calvert
• James Barnaby John
• Kimball Peter
• Messana Matthew
• Legros Ferdinand

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06V40/10 Primary

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/10012

  Stereo images · CPC title